How to Think Like a Lead Data Scientist

Hi there, its Matt! Thank you for reading the Pragmatic Data Scientist! This is a new publication by DataLife360 focused on practical tips for data scientists. Thanks to interest, we’ve decided to split off data scientist focused articles. Please subscribe!

Lately, I've been wondering how to capture the essence of a lead data scientist as I develop my LinkedIn course. A big question kept popping up: “What essential quality is important for a lead data scientist to know?”. The usual answer is its technology, but I’ve found that’s only partially correct.

The real goal is to how to think. We need to think how we approach problems.

A skilled lead data scientist understands the importance of not only focusing on the model but also considering their team's role within the broader problem space.

Thinking like a lead data scientist means recognizing how their models address the business model and the problems they face.

It requires knowing how to break a problem down - first principles mindset.

We need to start with first principle thinking more, in addition to technical foundations. In this article, I’ll break down how data scientists can develop this process at a junior, mid-level, and lead level.

Before diving into how to think at each level, you might be asking, "What's first principle thinking?"

First principal mindset is about dissecting a problem to its core elements: the how, what, and why. Its getting into the active habit of trying to understand the problem, context, and the factors that affect it. This approach helps data scientists, regardless of their experience level, tackle challenges efficiently. For lead data scientists, the 'why' is crucial.

When using first principles, we focus on key questions:

• How will we build this?

• What are we making?

• Why are we doing it?

Every AI and ML model we design stems from these foundational questions.

Data scientists hit roadblocks when they lose sight of these core questions. They're fundamental to creating game-changing solutions. Instead of experimenting using first principles, it can be far too easy to select the first solution we think is right. This can be problematic - we end up building for ourselves rather than the business.

If we want to apply principles first thinking, we need to break it down by level. New data scientists need to get "how" to build down. While mid-level data scientists should deeply understand the "what” they need to build so they can create a working model.

Successful lead data scientists need both to succeed. They need to understand the problem in detail, what it will take to solve, and have a vision for the problem space.

It helps them effectively, answer the question: “Why does the business need this model?”. First principles thinking drives long run business value, working models, and saves time and budget.

We need to set a solid foundation, and that begins from preparing early.

Preparing really begins as a junior data scientist. We’re applying first principles to learn how to build the right way.

The starting point? Perfecting how to build.

It's not just about grokking the code or mastering the tools but really diving deep into the mechanics of crafting a model. Really learn the fundamentals that get it across the finish line. And its not just algorithms. It’s also data ingestion, purification, feature engineering, and the nitty-gritty of model building to meet specific demands.

Never lose that burning curiosity about how to make things better, faster, and more efficient. So ask:

• "What's the best method for rolling out this model?"

• "How should this data stream be treated?"

• "Can I streamline and refine my code?"

• "How do I cross-check these outcomes?"

• "What ripple effects occur if I tweak this feature?"

A junior that not only builds but obsessively asks "how can I build better?" is on the path to becoming a seasoned data scientist. One who can foresee challenges and possesses the dexterity to address them. They’re curious. Uts a habit that’s extremely valuable for your career.

Harnessing this curiosity? That's how you evolve. You'll spot patterns, understand data, and become more adept at model development – but with broader strokes. We need to know the small things first to get great at the large things. Gradually, you'll see the full picture of a project and discern how its pieces mesh together.

You’re really learning the definitions of how the different processes of modeling work. Getting hands on practice building model components helps understand the intricacies of the process. You find out opportunity cost - a crucial skill when you become lead data scientist.

Always remember: Your endgame isn't just flashing your technical wizardry. It's delivering tangible value. Grasping "how to build" lets you realize your contributions in solving real-world business puzzles. At this juncture, it's about following the tasks that you’ve been assigned.

Once you learn basics, you can think about what you can build.

Mid-level DS is where you start taking the “how to build” mentality and applying it. At this point you’ve had building experience and have 2 to 3 years of experience under your belt.

You’re familiar with the pitfalls of building. But now you’re taking a wider role in doing it the right way.

Being successful here is knowing and thinking about what you are building. You need to think about the limitations and strengths of the use case, data, and tools, then build the model. You take more ownership of the models being built.

Questions here take factors into account:

• "What models should we consider?

• ‘What data are we working with, and is it available?"

• “What features will we need to include?”

• “What how will we version this model?”

Thinking like a mid-level data scientist means being proactive, collaborating in problem-solving, and considering the factors that go into the model. Its deciding how to build with the resources available, or even to build the model at all.

Practice hard here. I find so many bad habits that lead data scientists start at this level. This is where your principle-first thinking truly gets put to the test, where you're pondering the broader implications and impact of your model. A lot of people at this level fail when they rush to build a model - rather what the principles behind the model.

When it comes to figuring out what to build, there are a bunch of factors to consider. You've got to look at the resources you have, think about how much time it'll take, and weigh the level of difficulty.

And here's the thing - as a mid-level data scientist, you'll often find yourself making modeling and build decisions in situations that aren't crystal clear, with data that's not perfect.

Once you've got a handle on all that, knowing how to ask the right questions becomes super important. It's all about setting realistic expectations, getting help when you need it, and finding the resources for successful execution. It's all about learning to handle ambiguity and working with it - a crucial skill for lead data scientists.

Thinking like a lead DS means combining insights from both junior and mid-level roles. At this level, you must think strategically and evaluate the necessity of a model.

A surprising challenge in machine learning isn't just the data or algorithms. It's about aligning the technical with the strategy and scope of the project. A lead data scientist's primary responsibility? Bring clarity to the team and business.

You're crafting AI and ML processes to achieve business goals, guiding your mid-level and junior colleagues in the process. By providing clarity on the 'why,' you sharpen their approach, enabling them to probe deeper.

Key questions lead data scientists should ask include:

• "Why this specific model architecture?"

• “Does the business actually require ML?"

• “What's the purpose behind this model?”

Seeing and communicating the larger vision is essential. Principle-first thinking should be your compass, guiding construction, leadership, and delivery. Your ability to navigate the technical and strategic is paramount.

A lead data scientist should also ensure ethical considerations are taken into account when working with data, models, and algorithms. It's not just about what works best from a technical perspective - it's also about considering the implications of your work and its potential impact on data quality, fairness, accuracy, and privacy.

It's imperative for lead data scientists to view projects holistically. Your ML model should serve the business, streamlining existing processes or enhancing efficiency.

I've seen large teams dedicate 10 months to a model, only to question its necessity too late. When this happens misaligned projects, can overshoot budgets, positioning the data science team as a cost center.

Technical expertise is vital, but alignment with business goals is equally crucial. As a lead, anchoring projects to core principles ensures that technical achievements also deliver business value.

Thinking like a lead data scientist involves cutting through ambiguity, translating a vision into technical implementation, and getting the gist of the vision behind an AI product.

Trying to think like a lead is a major mental shift. It’s not possible to do it in one day. And its a lot of struggle - we all struggle with different aspects of it.

A lead data scientist is a navigator. You're not just hands-on with data ingestion, cleaning, or model creation. You're also the compass for your juniors, guiding them to construct the parts of models that align with your bigger picture.

Your creating processes, defining the technical problem space, and translating a AI product vision into technical implementation. This is by no means easy.

But it's one that I think anyone can do with enough practice. So much it is boils down to how, what, and why we build. Each is the foundation for the next level. Adopting principles first thinking is a major asset, and how we apply it at each level matters.