What does it take to transition from Senior to Chief Data Scientist?

I’ve been asked this question by a few highly competent Senior Data Scientists over the years. Even just asking the question is usually a good sign that they have a sense of the answer - a sense that the skills that make you a good Senior Data Scientist may not be the key skills for succeeding at the next level.

To answer this question, let's talk a little bit about what it actually means to become more senior as a data scientist.

The core idea is that as you get more senior, your area of responsibility grows. So, a Junior Data Scientist might be responsible for a single task, usually a well-defined task, that will probably take them hours or at most days, where:

• the goals have already been clearly set,
• success criteria are explicit, and
• we have a good idea of what kinds of problems they might face.

In other words, the task is well-defined and the Junior Data Scientist is responsible for implementing it.

A Senior Data Scientist might be responsible for a much more vaguely defined task:

• It might even be a whole project, where the criteria of success are much less clear.
• They're responsible in part for implementing it, but also for the work of others.
• They’re also responsible for figuring out what good looks like.

A Senior Data Scientist goes beyond just the implementation and the technical - her role encompasses greater complexity and uncertainty.

As you get to Chief Data Scientist, your area of responsibility expands yet more widely:

• You might be responsible for the technical correctness and delivery of multiple projects, and the management of multiple teams.
• Beyond that, you take greater responsibility for making sure that the right projects got picked in the first place, and for the downstream impact of those projects.
• This includes making sure that stakeholders see the value of these projects, and that they end up being adopted by the wider business or end-users.

If seniority is determined by the size, scope and type of responsibility that you're in charge of, then you should be able to figure out what you personally need in order to make the transition. For example, it may be that right now you're confident on the technical side as an individual contributor, and you’re getting the hang of managing a project and checking the work of others. But if your area of responsibility is now going to start to include picking projects, evaluating them, communicating to stakeholders, and making sure the value of the work gets realised, then those become the core skills that you may need to develop.

You can start to do this in many ways, e.g.:

• Offer to take on some project management responsibilities, to get good at estimating, prioritising, juggling and delivering under conditions of uncertainty and complexity.
• Actively practice mentoring your more junior colleagues.
• Seek opportunities to practice presenting and communicating about your work to non-technical stakeholders.
• Learn how to maintain high quality standards for work done by others. This is hard. Tip: focus your attention on providing a clear evaluation framework (e.g. metrics and unit tests) rather than writing or inspecting every single line of code yourself.
• Ask your manager to include you in conversations about future projects, or at least discuss the thinking with you when projects are being kicked off.
• Make sure you run good debriefs at the end of projects with the whole team to learn what could have been done differently.

There’s much more to be said. For example, it’s hard to let go of being the technical expert, as you shift from measuring your contribution in terms of your individual contributions, to the summed output of everyone on your team. The Software Lead Weekly articles are my favourite resource for thinking about these wider questions.