Since I last posted, I have changed jobs and done a considerable amount of reading, especially on the Data Engineering Subreddit. Both, driving me to think about the existential question of…”What is a Data Engineer.”
As I pondered, I came to frame my thoughts as such.
Is a Data Engineer:
- A Software Engineer who Focuses on Data?
- A Data Developer who Writes a lot of Code?
This may be a difference without a distinction, but maybe that distinction is fundamental to how we view and perform our jobs.
I’ll start with what I see as the definition of each archetype.
Software Engineer-
Data Developer-
Perhaps, I’m over simplifying, yet I see it the difference in what they are expected to deliver.
Many software engineers are dealing with data, maybe massive amounts of it. They may be using databases and other data stack technology, but to help them deliver better applications. For example, many may not see SQL as a core knowledge requirement. Or they may not feel the need to understand the database they using to deliver application.
Data Developers mostly started out on the data side of the fence. Whether that be as a DBA, a reporting/BI developer, or in ETL; for them SQL literacy is core to their knowledge base. They need to learn more software engineer techniques. That is why, for example, you see more using Git for database schemas; and they are using more traditional software languages such as Python or even JavaScript. ๐ฑ
To be a successful data engineer we need a bit of both skill sets. We can’t really succeed with only one side. How you approach your craft (or think of yourself) may be strongly biased by where you came from.
For me…I’ll probably always think of myself as a Data person who can Code, and will never claim to be an app dev, no matter how much code I write.
Thanks Destin Centala for proofreading and being a sounding board as I worked through my thoughts.
One thought on “What is the Engineering Part of Data Engineering”