What should we learn from the short-lived career of Data Scientists?
Data Scientists will merge into the Software Engineer role, similar to how Front-end and Back-end specializations exist. Before this convergence occurs, it's important to examine the distinct histories, practices, and cultures these professions developed.
A Path-dependent Process
The concept of "path dependence" from economic history explains how two groups following different historical trajectories develop divergent outcomes. Software engineers and machine learning engineers similarly experienced path-dependent development, creating distinct professional cultures.
By the late 20th century, the software industry matured and created the product manager role. Product managers bridge business needs and development teams through user stories, which enabled software engineers to focus purely on engineering excellence. This specialization produced mastery in code reviews, unit testing, and DevOps practices.
Machine learning engineers followed a different trajectory, emerging from data analytics backgrounds. These practitioners prioritized business outcomes over code quality and scalability. They embraced MVPs and experimentation—traits reflecting their original "data scientist" designation rather than "engineer."
The two cultures diverge fundamentally in their approaches:
- Data collection: Software engineers store only immediately necessary data, while machine learning practitioners collect data they might need in future projects
- Testing philosophy: Software engineers validate code against specifications, while machine learning engineers measure algorithmic impact on users and iterate based on feedback
- Risk tolerance: Some machine learning practitioners accept deploying imperfect code if it achieves desired business results
The Merge Ahead
Noel Kippers' metaphor about machine learning professionals "putting on big-boy pants" reflects the expectation for data scientists to adopt stricter engineering standards. Within five years, the distinction between machine learning engineers and software engineers will likely disappear, with the former becoming the latter who use specialized libraries like TensorFlow, HuggingFace, and Scikit-Learn.
However, a concern emerges about what the broader software engineering community will learn from this merger. Most companies prioritize output volume over outcome quality. Software engineers rarely participate in product discovery, despite being positioned as "typically the best single source of innovation."
"The little secret in product is that engineers are typically the best single source of innovation; yet, they are not even invited to the party" — Marty Cagan
The danger lies in companies reverting to waterfall processes dressed in agile language rather than adopting the build-measure-learn methodology that characterized data science practice.
Tarek Amr, February 12, 2021