Structuring Tech Teams to deliver ambitious GenAI features

80% of organizations aren’t seeing any tangible enterprise-level impact from their use of Gen AI [1].

At Inato, that’s a pitfall we want to avoid and we believe organizational structure is key to doing so. Our mission is to accelerate drug development by helping pharmaceutical companies and health organizations streamline their clinical trials. As we navigate an industry held back by manual processes and unstructured files, LLMs’ capabilities offer a game-changing opportunity to take a giant leap forward in our mission. Over the past two years, we have successfully transitioned from testing AI supplements to launching sophisticated AI-based products, such as our automated pre-screening tool (read more on that here).

With our growing experience in GenAI, we are now convinced that there are still many untapped opportunities to develop ambitious and impactful products. To fully unlock these exciting new capabilities and overcome their associated challenges, we wanted to take a step back to reassess what an ideal organization looks like for us.

To explore this problematic, I had the pleasure to interview several AI and data leaders from the Modern Data Network:

• Noé Achache (Head of AI Engineering at Theodo)

• Anas El Khaloui (Head of Data at HiPay)

• Emilie Gerardin (AI Lead at DisneyLand Paris)

• Sami Kaddani (ML Engineer at Too Good to Go)

• Kasra Mansouri (Head of Data at FinFrog)

• Thomas Moulin (Lead Data Scientist at Welcome to the Jungle)

Despite different contexts and industries, we all wrestle with the same fundamental questions:

• What are the specific challenges associated with GenAI?

• Where to draw the line between generalist software engineers and AI specialists?

• And ultimately, what is the optimal team organization?

Generative AI models promise plug-and-play innovation. Teams spin up prototypes of AI-based features in days using off-the-shelf LLMs and ready-made Langchain pipelines, then label the work “production-ready.” But this apparent simplicity is deceptive. Embedding an API call in a web form is trivial, but ensuring the output is accurate, robust, and secure is far from it. Setting realistic expectations is vital to avoid rushed releases and preventable pitfalls:

• Prompt engineering is the new feature engineering: Traditional machine learning relies on training datasets with well-crafted input variables, whereas the power of pre-trained LLMs lies in their ability to operate without extensive labelled data. However, because these models can perform so many tasks, prompts must be extremely specific and crystal clear to yield the desired output. Prompt engineering, therefore, isn’t about tricks — it’s about translating unspoken business rules into precise instructions and breaking down complex problems into smaller, more manageable tasks. Mastering this skill has become essential for getting the best results from LLMs without needing to dive into more advanced techniques, such as fine-tuning.

• Evaluation is hard: Unlike traditional ML, where metrics like accuracy, precision, or AUC are well-defined and easily computable, evaluating free-text output from an LLM is complex and often subjective. It requires defining what constitutes a “good” answer and building evaluation datasets, often with manual labelling by domain experts. Once reference answers are gathered, systematically comparing them to actual LLM outputs is a whole new challenge. The most reliable technique, manual evaluation by experts, is quickly cost-prohibitive and not applicable at a large scale. On the contrary, using LLMs as a judge is efficient, but much less reliable. This challenge opens up an entirely new domain requiring specialized expertise.

• Specialized tooling and governance requirements: GenAI applications demand new expertise in areas such as agent orchestration frameworks, data integrations (e.g., MCP servers, RAG pipelines, embeddings), and quality and usage monitoring. Mastering these tools requires skills at the intersection of ML, software engineering, and data engineering. Moreover, new risks emerge, such as prompt injection and hallucinations, necessitating tailored mitigation techniques and best practices. Additionally, regulatory frameworks (e.g., GDPR, HIPAA) need to be considered as they drastically shape how third-party LLMs can be integrated in sensitive domains.

These challenges make it necessary to redefine the boundaries between the traditional roles of ML engineer, data scientist, software engineer, and data engineer. While each discipline brings critical expertise, none alone encompasses the full skill set required to address GenAI’s unique pitfalls. As a result, a new role — the AI engineer — has emerged, unlocking the power of LLMs by tackling their specific challenges.

The ML engineer originally arose to bridge the gap between “notebook-based” data science and production-grade software development. As organizations scaled their use of machine learning, data scientists struggled with building reliable pipelines, while software engineers often lacked domain expertise in model training and evaluation. ML engineers combined coding best practices and system design with a solid grounding in model development and data engineering.

AI engineers build on this foundation, extending the ML engineer’s mission into the GenAI era. They retain a strong emphasis on robust pipelines, testing, and monitoring, while mastering GenAI-specific tools: agent orchestration frameworks, RAG pipelines, embeddings and evaluation frameworks.

Below Venn diagram gives a few examples of specific and shared skills of the Data Scientist, ML Engineer and AI Engineer roles.

Now that we’ve highlighted the challenges associated with LLMs and the necessity of specialized expertise, is the idea of LLMs making AI accessible to all teams ultimately unrealistic? Does every organization require dedicated AI engineers as soon as they want to implement LLM-based features?

First, the answer obviously depends on the size of your organization. For smaller companies, someone with a data or ML background can perfectly ramp-up on those topics and embody the role of an AI Engineer in the context of a given project.

Going further, we think that even for larger organisations and the ones that already have AI engineers, some features can be implemented autonomously by the rest of the tech team.

To identify them, ask yourself these two questions:

1. How reliable does your feature need to be to succeed? The higher the level of reliability required, the more effort you must invest in constructing control datasets and performing advanced evaluations. Conversely, if the feature you want to implement is inherently low-risk or meant to be supplemental for the user, you can likely test it with a small group of users to assess its value and move it into production without investing heavily in evaluations and reliability.

2. How broad is the problem space your feature needs to address? The wider the problem space, the more complex the technical solution behind the feature: it must integrate with multiple data sources and equip agents with diverse tools. As complexity grows, errors compound and the likelihood of low-quality outputs increases. That’s why an all-purpose assistant is rarely a good first LLM project, and you should rather focus on a more focused use case first

It means that implementing low-risk, narrowly scoped projects, can be done without dedicated AI engineers ! In fact, we take it a step further: making the rest of the team autonomous to implement those features is essential for any organization looking to leverage AI capabilities effectively and across the board.

There is a classic debate in data and ML teams: should expertise be centralized in a hub or embedded within specific business or product units? Given the mix of skills required to build successful AI features, this question is equally crucial for AI teams.

There is obviously no one-size-fits-all solution, every context is unique, but some universal trade-offs emerge:

• Centralized teams yield robust, unified infrastructures. This setup is ideal for ambitious, cross-company use cases. The risk, especially with multiple product squads, is that the AI team becomes a bottleneck, causing feature teams to compete for its attention and ultimately under-deliver.

• Distributed teams offer closer proximity to business problems and faster iteration within their domains. It increases availability and encourages proactivity in identifying relevant use cases and addressing issues. The drawback is potential heterogeneity in practices and duplicated work.

How to get the best of both worlds ? You guessed it: we present you the hybrid AI team.

We propose an approach where:

1. Product squads with well-identified, long-term AI initiatives get a dedicated AI engineer responsible for exploring and implementing high-value use cases within the squad

2. Software engineers in all product squads implement low-risk, narrowly scoped projects, with support from the central AI team. This is also true for internal use cases implemented by other teams (support, marketing, ops etc.)

3. A central AI team:

• Defines standards, notably for evaluation, tooling and monitoring

• Explores and implements high-value use cases cross-company and for squads that don’t have dedicated AI engineers

• Equips all software engineers with the support they need to tackle low-hanging fruit autonomously

Harnessing the power of GenAI and LLMs isn’t just a matter of flipping a switch, it demands a deliberate re‑examination of our tech organizations. By combining specialized AI engineers with empowered product squads, you can leverage your whole tech organization to confidently deliver low‑risk features and truly reliable high‑impact products.

For organizations without dedicated budgets for AI initiatives, it’s essential to start small and focus on projects that can demonstrate value quickly. To do so, maximize your chances of success by identifying a low-risk project that addresses a well defined issue. This incremental approach will not only help you prove the worth of AI but also pave the way for securing more funding in the future.