7 answers
5
Questions
Asked
1166 views
What is the most frustrating part of a data scientist's job?
#datascience #data
Login to comment
7 answers
Updated
Julia’s Answer
Data science is an overall very rewarding career path! However, as with any jobs, there are ups and downs. Each company / experience is a different, but here are a few challenges I've encountered in my data science career:
- Access to data. Doing data science well, of course, is dependent on the amount and quality of data. If the data isn't available, isn't accessible (e.g., locked down by another group in/outside of your organization), doesn't contain the features you need, or isn't large enough, your job gets a lot harder. Often much of this is outside your control, which can be frustrating.
- Explaining what you do to non-technical audiences. Data science/predictive modeling is hard to understand! Clarifying the timeline/process of your work, limitations of your analyses, how to interpret results, etc. can be challenging when working with key stakeholders who are not data scientists.
If you're going into data science, chances are that you're excited by diving into data, writing code, and analyzing results. There's a lot more to being a data scientist in industry than that so it's important to anticipate these additional tasks and practice how to address them!
- Access to data. Doing data science well, of course, is dependent on the amount and quality of data. If the data isn't available, isn't accessible (e.g., locked down by another group in/outside of your organization), doesn't contain the features you need, or isn't large enough, your job gets a lot harder. Often much of this is outside your control, which can be frustrating.
- Explaining what you do to non-technical audiences. Data science/predictive modeling is hard to understand! Clarifying the timeline/process of your work, limitations of your analyses, how to interpret results, etc. can be challenging when working with key stakeholders who are not data scientists.
If you're going into data science, chances are that you're excited by diving into data, writing code, and analyzing results. There's a lot more to being a data scientist in industry than that so it's important to anticipate these additional tasks and practice how to address them!
Updated
Adit’s Answer
One of the most frustrating aspects of being in Data science would be the phase in which you are grappling with a problem and trying to find a solution and yet you do not seem to be getting anywhere. Either the data is not of good quality or the method you are using for prediction does not get you the results that you are looking for. All of this wears on your patience and you seem to be making no progress. However, it's an illusion that you are not making any progress, and eventually when you arrive at a solution that satisfies you, then you realize that all that waffling that you did on the problem was worth it.
Updated
Muneer’s Answer
Data science is a very rewarding career. You can work independently and provide additional insights for business leaders to make game changing decisions. However, with any job, there are some frustrations.
1. Cleaning up bad data.
2. Properly defining the questions that other people want us to answer
3. Understanding unclear business processes and manual processes
4. Working with non-integrated systems
5. Data science is more than just modeling but its also engineering and analysis (ie extracting data and transforming data)
6. The need to have business acumen
1. Cleaning up bad data.
2. Properly defining the questions that other people want us to answer
3. Understanding unclear business processes and manual processes
4. Working with non-integrated systems
5. Data science is more than just modeling but its also engineering and analysis (ie extracting data and transforming data)
6. The need to have business acumen
Updated
Sean’s Answer
Data science is an amazing career and in high demand right now. However, as with all jobs, there are definitely some challenges you will face. Three of the largest frustrations I have come across:
1) To be a data scientist, you also need to be able to data engineer - many people don't realize, but the majority of work a data scientist needs to do when solving a problem or building a model is actually preparing the data, blending data sources, creating features, etc. You will need to make sure you set expectations upfront with your team on how long this portion will take.
2) Identifying the actual business ask - a lot of times you will need to develop a model for a non-technical audience that might not be exactly clear on what the output will tell them. Spend the time upfront with your business counterparts to ensure the problem/question you are trying to solve is clearly laid out. Try to think about how will someone be able to action the results of the model.
3) Not all data is readily available and you will need to make some assumptions - ensure to assess the mission critical data upfront and understand if there are any pieces missing or that can be filled in using assumptions. Make sure those assumptions are aligned with the business team and clearly documented.
With all of these frustrations, the biggest piece of advice I would have is communicate. Communication is key - upfront, throughout, and at the end - the more collaboration you have, the better answer you will get to, and the more impact your work can have.
1) To be a data scientist, you also need to be able to data engineer - many people don't realize, but the majority of work a data scientist needs to do when solving a problem or building a model is actually preparing the data, blending data sources, creating features, etc. You will need to make sure you set expectations upfront with your team on how long this portion will take.
2) Identifying the actual business ask - a lot of times you will need to develop a model for a non-technical audience that might not be exactly clear on what the output will tell them. Spend the time upfront with your business counterparts to ensure the problem/question you are trying to solve is clearly laid out. Try to think about how will someone be able to action the results of the model.
3) Not all data is readily available and you will need to make some assumptions - ensure to assess the mission critical data upfront and understand if there are any pieces missing or that can be filled in using assumptions. Make sure those assumptions are aligned with the business team and clearly documented.
With all of these frustrations, the biggest piece of advice I would have is communicate. Communication is key - upfront, throughout, and at the end - the more collaboration you have, the better answer you will get to, and the more impact your work can have.
Updated
Matt’s Answer
I agree with most answers that having access to quality data can be very frustrating. I would also add that it can be challenging when the outcome of a data science project is not acted on by the business. Part of this is the responsiblity of the data scientist; Our jobs don't end with the model or the analysis, we need to make sure that the business acts on our deliverables. At the same time, sometimes this is outside of our control and a great project ends up with minimal impact. To avoid this from happening, I suggest that 1) you strive for clarity on the business problem that you are trying to solve upfront, 2) you align with stakeholders on how your project will be acted on, 3) you hone your communication skills, and 4) stay persistent and postive!
James Constantine Frangos
Consultant Dietitian & Software Developer since 1972 => Nutrition Education => Health & Longevity => Self-Actualization.
6342
Answers
Updated
James Constantine’s Answer
Dear Jessica,
The aspects of a data scientist's job that can cause frustration are diverse and can differ based on personal experiences. Some of the most common challenges that data scientists frequently grapple with include:
Tackling unorganized and incomplete data: Data scientists often come across data that is not fully complete, consistent, or error-free. This can pose a challenge in deriving accurate conclusions and may require substantial time and effort to clean and prepare the data for analysis.
Translating complex findings for non-technical audiences: It is a key part of a data scientist's role to convey their findings to individuals who may not have a deep understanding of data analysis or statistics. This can be a complex task as it involves simplifying technical concepts into easily comprehensible language without losing the core meaning of the findings.
Keeping pace with fast-changing technologies and methods: The world of data science is continuously evolving, with new tools, methods, and programming languages appearing regularly. Keeping up-to-date with these developments can be both time-consuming and frustrating, as it demands ongoing learning and adaptation.
Operating with limited computational resources: Data scientists often have to process vast datasets, which can be computationally demanding and require substantial hardware resources. In some situations, they may have to work with restricted computing power, which can hinder the analysis process and lead to frustration.
Balancing project timelines and expectations: Data science projects often come with strict deadlines and high expectations from stakeholders. Juggling the need for detailed analysis with the pressure to produce results quickly can be challenging, particularly when unexpected hurdles occur during the project.
Addressing data privacy and security issues: As data becomes increasingly valuable, data scientists are required to navigate the intricate world of data privacy and security laws and guidelines. Ensuring that data is managed responsibly and securely can be a challenging task, particularly when it involves understanding the subtleties of various legal frameworks.
Working with teams with limited data literacy: In some companies, data scientists may have to work with teams that have a limited understanding of data analysis or statistics. This can result in miscommunication, delays, and frustration when trying to align on project objectives and expectations.
Managing the pressure to uncover significant insights: Data scientists are often under pressure to discover innovative insights from the data they analyze. The stress to deliver these insights can be challenging, especially when the data does not immediately reveal any significant patterns or relationships.
In conclusion, the most frustrating aspects of a data scientist's job can include dealing with disorganized data, simplifying complex findings, staying updated with technological advancements, managing resources and expectations, handling privacy and security concerns, working with teams with limited data literacy, and managing the pressure to uncover significant insights.
Stay blessed!
James Constantine.
The aspects of a data scientist's job that can cause frustration are diverse and can differ based on personal experiences. Some of the most common challenges that data scientists frequently grapple with include:
Tackling unorganized and incomplete data: Data scientists often come across data that is not fully complete, consistent, or error-free. This can pose a challenge in deriving accurate conclusions and may require substantial time and effort to clean and prepare the data for analysis.
Translating complex findings for non-technical audiences: It is a key part of a data scientist's role to convey their findings to individuals who may not have a deep understanding of data analysis or statistics. This can be a complex task as it involves simplifying technical concepts into easily comprehensible language without losing the core meaning of the findings.
Keeping pace with fast-changing technologies and methods: The world of data science is continuously evolving, with new tools, methods, and programming languages appearing regularly. Keeping up-to-date with these developments can be both time-consuming and frustrating, as it demands ongoing learning and adaptation.
Operating with limited computational resources: Data scientists often have to process vast datasets, which can be computationally demanding and require substantial hardware resources. In some situations, they may have to work with restricted computing power, which can hinder the analysis process and lead to frustration.
Balancing project timelines and expectations: Data science projects often come with strict deadlines and high expectations from stakeholders. Juggling the need for detailed analysis with the pressure to produce results quickly can be challenging, particularly when unexpected hurdles occur during the project.
Addressing data privacy and security issues: As data becomes increasingly valuable, data scientists are required to navigate the intricate world of data privacy and security laws and guidelines. Ensuring that data is managed responsibly and securely can be a challenging task, particularly when it involves understanding the subtleties of various legal frameworks.
Working with teams with limited data literacy: In some companies, data scientists may have to work with teams that have a limited understanding of data analysis or statistics. This can result in miscommunication, delays, and frustration when trying to align on project objectives and expectations.
Managing the pressure to uncover significant insights: Data scientists are often under pressure to discover innovative insights from the data they analyze. The stress to deliver these insights can be challenging, especially when the data does not immediately reveal any significant patterns or relationships.
In conclusion, the most frustrating aspects of a data scientist's job can include dealing with disorganized data, simplifying complex findings, staying updated with technological advancements, managing resources and expectations, handling privacy and security concerns, working with teams with limited data literacy, and managing the pressure to uncover significant insights.
Stay blessed!
James Constantine.
Updated
Joe’s Answer
Data Science is a super exciting field to be in, where conceptually data scientists would build machine learning or statistical models that would directly drive business impacts. In reality, building excellent model is essential and is the foundation of the work, but on top of that, the piece of communication and education with partners is equally important. Letting partners understand the model and leverage the model in the right way leads to unlock the full power of the model and thus the ultimate business impacts. The buy-in from partners and accurate execution are required to have successful projects, which can be a tough part of the data scientist lives.