I think….I can maybe work in Python. And I do see the value of it to a Data Engineer.
My last posts were somewhat (very?) self serving as I went through trying to learn more software engineering practices and specifically Python for my job with a web app startup. So, I wanted to circle back with how it was going learning.
I spent the last 8 months embedded with a software engineering team with some data scientists, most of both groups could write Python. As I’ve said before, I had taken numerous EdX and DataCamp sessions on Python, but nothing had stuck. So there was a big learning curve going from all SQL and GUI ETL workload to using Python as core language.
The biggest takeaways from the experience for me are:
- There is no substitute for working in a language or technology. When you do it for your own education or personal projects, it just doesn’t stick like it does when you are “forced” to do it. It really focuses your mind.
- Social Learning is a big help. Having a team that you can go and ask (bug?) with questions when you get stuck (or your StackOverflow searching doesn’t help) is invaluable. The fact that I had smart people who were patient, and often told me more about the context or tips and tricks was a huge help.
- Data Engineering Education still isn’t great. DE’s habitat a world in the in between. We aren’t software engineers with all the mature practices from there, but we also aren’t solely living in a data analyst/scientist world, we are a hybrid. And education for the hybrid really isn’t the best, there is lots of good app dev and data science training, but not the middle (getting better though).
Since I think the training out there not great, I did want to share how I think I learned.
I did do some formal training in DataCamp (still more geared towards data scientists) and on MS Learn (which I am a big fan of, even if more toward specific Azure tech, but still helped). Beyond just bugging team members, the biggest helpful resources I found were RealPython.com and StackOverflow. (The caveat with S/O is often you fall down a rabbit hole of competing package methods, but it’s always educational.)
Being honest, a few months ago I thought I’d be writing that I felt like I had not learned much at all, but the past few weeks it has finally come together. I don’t know that it would be my automatic go to for all DE projects, but definitely if the need lives more on the programming/automation side I can see the big benefits.
