5 Reasons Why I left Data Science for Software Development
Caveats to keep in mind
After having a deep dive into data science I decided to move on and become a passionate software developer.
Once you have been in the field of data science, you might have come across the following reasons, which might have lead you to doubting the current state of your career.
1. It’s not about the state-of-the-art research
When you go through the data science job description you might fall for the keywords like Big Data, Data Mining or Deep Learning. In the end it is the good old regression, which will provide the best or at least any presentable results.
I guess that every data scientist has a certain amount of interest towards new technologies and state-of-the-art techniques. However, the bitter truth is that this interest will be degraded at some point by your supervisor. Anyone who is leading a project needs to know where the time is being spent and how to optimise the costs. Often the time on research will be shortened in order to meet the deadlines.
The state-of-the-art techniques and models are developed by teams of researchers, not data scientists. As a data scientists might use them, still this is unfortunately a rare case.
2. Preprocessing consumes a lot of time
You already might have heard that data preprocessing the most important part of your analysis. Feeding in unfiltered data might yield misconception and meaningless results.
In the case you think of the NaN values or incompatible data formats and shapes, this is not what I am talking about. Such data cleaning is done within a few hours and is a routine, which can be easily automated. So by saying preprocessing I mean the preparation of data as inputs and outputs. What do you want to pass into the model and what should be predicted? There is no general recipe, which will answer this question for you.
In the process of preparing the data for any kind of analysis there is a lot of communication that needs to be done. If you skip this communication, you might end up with a lot of lost work. Therefore, clarification of the way how data needs to be preprocessed is crucial.
3. Customers only want to see good results
As a data scientist you will have contact with customers. One way or another they will ask you what results you have achieved so far or how are you planning to proceed with the provided information.
So, what are you going to answer to keep the customer at it and not make him look for other solutions in the market? My advice is to keep the communication short and provide only results which are existing and reproducible. Do not promise any certain analysis, which you are intending to perform. This will set a time constraint on your valuable work and decrease the quality of it.
Furthermore, customers are disappointed if the presented results do not have an insight, which they can exploit. They simply do not accept that any result is helping to move forward. Sometimes a good explanation can diminish the disappointment, but that depends on the analytical skills of the customer who you are talking to.
In a few companies which I worked for they used Scrum to deliver a valuable increment every few weeks. Every project which used this method failed miserably. First, estimations of the amount of time for data preparation, analysis and visualisations will always be too low because of unseen impediments. Second, data science is not software development where you usually can show some new features and discuss user experience. Therefore Scrum is not the preferable method of choice.
4. Project augmentation is tedious
Data scientists do not only prepare and analyse data. They also participate in training courses, read research papers and, depending on their experience, they search for calls for tenders, i.e. they land new customers.
In ideal circumstances, customers detect a need for optimising their business where data science might be the answer. The operation’s team defines the need with the expected outcome and places a call for tenders. Data scientists find these calls and contact the potential customers.
However, this is not how it happens in practise. Relationships between decision makers govern the data science market. If your supervisor identifies a lack of incoming projects, you will get instructions to become active. That means you might have to become creative and look for new customers. This implies sometimes a fast set-up of knowledge of businesses you never heard of.
This permanent gaining of knowledge in a tremendous amount of topics is painful and exhausting. Of course, you might say that is exiting and gaining knowledge is an important key of success, but I can assure you that every open-minded curiosity will reach its limit point if the build-up of know-how is just for marketing purposes.
5. Never-ending story
Maybe the most weighted reason why you might turn against data science is that a project is never done. A project might fail, but it will never be finished.
You might have spent months doing analysing the data and finally, the customer is happy when you show him the results and provide the models. Again, after a few weeks later the models are deployed in the customer’s environment and you celebrate your success excessively. Well, but months or years later the customer urgently tries to reach out to you because the model is not working anymore or is delivering corrupt results.
Now you have to search in the back of your mind what this project was all about. You spend hours and long hours implementing an environment where you can reproduce the issue of the customer. Then you might realise that some packages or the software was updated by the IT of the customer in order to fix security issues or something. Well, no problem, you can fix it. That is the slightest evil.
However, more often the reason for the issue will be that the models you delivered were not intended to be used for the new unseen data. Still, going into the discussion and arguing why the model was not intended to do what is now expected, is not the best choice. You will have to provide support in order to keep the customer at it.
Words in conclusion
I am sure those reasons above will not stop you from loving your data science job. Data science can still be exciting and a way to a great career. Just be aware of the points explained in this article.
On my path I decided to move on and leave data science. Due to ongoing interest and free time background in programming, I took up a job in software development. Although in the beginning such a new job position might be tough, I can encourage you to risk the step. Software development is fairly more fun, even more after years of experience.
Thank you for reading!
I hope you enjoyed this article. Please leave a comment if you have any thoughts or critics about the explained reasons. I am also happy to answer any questions.
