Building an AI development platform to win in the AI era

It has been a bit more than a year now that AI completely transformed how I approach software engineering. One of my main takeaways from this year is that companies should be building internal AI development platforms. During the past months, AI not only transformed how we deliver software but also what we deliver: AI-powered features are on every roadmap, for almost every product.

As a result, all engineering teams had to at least integrate with the API of an AI provider, and most of them built agents. This quickly led to everyone building a chat interface, learning about context window management and building compaction mechanisms, integrating MCP servers, implementing some sort of RAG, setting up tracing and evals and finally facing downtimes of AI providers and implementing fallback mechanisms with multiple providers. None of that is specific to a use-case or a product. Software engineering has been here before with internal development platforms. Tech companies should now apply the same principles to support the surge of AI-powered features.

A brief history of development platforms

In 2006, Werner Vogels said “You build it, you run it”. The ownership of production apps started to move directly to the product & engineering teams developing the application, so did the operational load. In the 2010s, configuration management tools started to appear (Chef, Ansible, etc.) and started reducing the complexity to administrate software infrastructures.

In 2017, Netflix publicly describes “the paved road” concept in several conferences, with a central team building golden standards that other teams are free to adopt, standardizing service discovery, configurations, metrics, logging,… so teams don’t rebuild them per-service. Then in 2019, Team Topologies identifies cognitive load as the key constraint in engineering organizations and suggests a platform team whose job is to reduce part of the load on other teams, providing its work as an opt-in, non mandatory service. In 2020, Spotify publicly named this concept “the Golden Path”.

The Golden Path — as we define it today — is the ‘opinionated and supported’ path to ‘build something’ (for example, build a backend service, put up a website, create a data pipeline). How we use Golden Paths to solve fragmentation in our software ecosystem – August 2020

From there, SaaS vendors start offering such self-service catalogs of hosted products. In 2023, “Gartner predicts that by 2026, 80% of software engineering organizations will establish platform teams as internal providers of reusable services, components and tools for application delivery”. In 2025, the DORA report measured it: “with 90% of organizations reporting the use of an internal developer platform and 76% establishing dedicated platform teams“. And most software engineers experience this daily nowadays.

Everyone is building agents

It has been a while since I or my teams manually configured a database host, but we create new databases every month, and use them every day. And this goes for Kubernetes clusters, tracing, metrics about resource consumption… But I suddenly had to do all of this again when building my first AI agents: there was no self-service gateway to frontier models, we did not have any observability or eval tools for LLM, no catalog of available MCP servers within the company, no standard chat UI, and the list goes on.

After a few days deep into building or setting up all those features, I could finally start working on the AI features I had in mind, so I discovered the available frameworks, the basic features a chat interface needs to have, etc. When it was done, we advertised it internally. Quickly enough, we realized that many teams were also in this process, all of them faced the same challenges and built their own set of features. Some had a great UI, some other had very strong observability, or memory and context management.

Great! We had many motivated engineers who learned about AI and how to integrate it in our products. Going through this phase was key to build expertise across the organization. But then, it was clear that we needed to somehow put things in common: every project had its strenghts that others projects would benefit from. And looking at the predictions, it’s worth investing in sane foundations to sustain what’s coming.

“33% of enterprise software applications will include agentic AI by 2028, up from less than 1% in 2024.” — Gartner press release, June 25, 2025

What would an AI development platform look like?

AI is evolving super rapidly, and its impacts on software engineering are far from being well identified yet. Therefore, the following is not a secret recipe one can apply blindly and get good results; and the findings standing today may go outdated soon.

For now, I identify some key learnings from The Golden Path concept that apply well to AI development platforms:

• A central team, building components that product teams can build upon: properly running AI features in production requires a lot of foundations (chat interface, tracing, specific evals and observability, LLM providers,…). While each team would be able to implement their own flavor of those foundations, they are mostly independent of the use-case and can be provided as a one-size-fits-them-all. Just like any services, they come with maintenance so you would rather have one version of each than many scattered across your organization.
• Everything is optional. The AI development platform is offered to the teams, and they can pick and choose. Teams remain free to go with their own LLM provider or chat interface: maybe they have a very specific use-case that would not be well-served by the golden standard, or they might simply be willing to explore other approaches. The usual deal remains: “You build it, you run it”. This principle is key, as it allows team to explore and innovate, with no risks of impacting negatively the broader organization. It also prevents the unfortunately well-documented failures of internal platforms such as gatekeeping or enforcing “only one way”, which ultimately kills innovation and reduces product quality.
• The platform should focus on commoditizing the most commonly needed features. The most commonly used the feature, the easiest it must be to provision and use it. Product team should never have to hesitate about starting something new because it is too time-consuming to setup. Reducing the frictions to spin up a new state-of-the-art AI-powered service is key to ensure your organization is able to innovate, especially given the pace of evolution of AI.

So we have a team and some guiding principles. What should they work on now? Here are the components I would prioritize in a new AI development platform. All those components have value on their own, but the benefits multiply as those components start interacting together. Investing in a platform with those components available has, in my experience, proved to rapidly scale the potential of in-product AI in companies with several, complementary products.

LLM gateway

AI features eventually need a model, so each team ends up holding API keys, wondering about rate limits, and either building a fallback, or facing errors when their provider has a downtime (and it happens quite often). It is needed by all AI-powered applications. Teams can integrate a provider directly, but it comes with maintenance costs, and while most providers seem to have standard interfaces, many specificities come up once you go in the details (deprecation of temperature and thinking effort management, to just name a few).

Instead, one central service could act as a proxy between your applications and the LLM provider API. This proxy would centralize API key management, cost observability, fallback mechanisms to handle downtime and be the source of tracing. It also provides a unique opportunity to apply cost optimization strategies at company-scale as you could easily apply smart model routing based on prompt classification, in one place, for all your products. It abstracts away the complexity of integrating with multiple providers, which is a huge benefit now as LLMs are evolving quite rapidly and we see new “best in class” models coming from different providers every other week.

Chat UI & streaming protocol

I have worked on quite a few agent integrations lately, and they all ended up implementing very similar user interfaces. And across the industry, we are seeing standard practices emerging when designing an AI chat interface and the related rendering of tool calls: questions, interrupts, consent gates, permission management, etc. It is a lot of work to implement and maintain a user interface like this, so it’s worth considering packaging it and offering it as a JS library. For this library to be fully usable by any backend, its streaming protocol (ie. how it exchanges messages, interrupts, rendering instructions, etc.) with the backend must be standardized and documented. Teams are then free to adopt it, and even extend the protocol with custom events if needed, for their own use case. This heavily reduces the evolution cost of your AI features, and naturally unifies the user experience across your applications.

Observability & evals

Every single agent call needs tracing, cost/token accounting, and some way to judge whether the output was actually good. Observability was already a key pillar of software engineering before LLM, and it is even more critical now as we are putting LLM, which are quite similar to a black box in many cases, at the center of our features. Unlike traditional observability, observability for generative AI is more directly needed by product teams, as a way to iterate on the prompts and optimize the behavior of the AI-powered feature. Therefore, providing easy access to your observability tool, to the traces and the evals to teams outside of engineering is now a key practice to make your team efficient when working on AI-powered features.

Context layer

To perform as expected, agents need product knowledge; for instance, your support agent needs to know all there is to know about your product, its knowledge base, its FAQ, the support team tips and tricks, etc. But stuffing a whole wiki into a system prompt doesn’t scale: it’s expensive, it can reduce the accuracy of the agent, and it goes stale. What agents actually need is a small map first, then the ability to drill into exactly the node relevant to the question. If you browse LinkedIn, it looks like every engineer built a RAG lately: it is not that hard to begin with.

However, building a performant RAG system that scales well, properly ingests the data from your various sources, and evolves as the technology evolves too is quite time-consuming. Keeping the content curated is also quite demanding over time. Therefore, I am a big fan of having a central Context Layer / RAG system, provided as a service, so that the teams working on their products can focus on curating the data that goes into it.

No-code agent development platform

We hear about agents everywhere now, in every SaaS product, in every feature, and in every engineering team. And when looking at those agents and the use cases, a lot of them are fairly basic: an agent with a custom prompt, access to a few tools through MCP, and some memory management (And it’s OK, not everyone needs to build the most complex things with the latest framework!). But then, we can make our engineering organization much more efficient if all those basic features are available and centralized in a single agentic service already. Instead of having many teams re-implementing “My agent 101”, they could just get it out of the box.

In my experience, once the agent is up and running with the right tools, the engineering team is way less needed: many evolutions and iterations can be performed by the owner of the feature directly. For instance, the Product Manager does not need an engineer to tweak the prompt to adjust a certain behavior; they can do it themselves. The same thing goes for the Support team when they spot inaccurate answers from the support agent. So a company can drastically reduce its time-to-market for a majority of its agents if it invests in building a no-code agent development service, an application where non-technical teammates can define agents with a custom prompt and some MCP servers.

To me, this is one of the big changes AI is making in software engineering that I don’t hear or read about enough: through agents, non-technical people can now directly ship value to the users without needing to go through engineers every time. Just like a CMS empowered marketing & content teams to manage their websites directly, a no-code agent development platform can make anyone in the company build their agents, ready to be used.

Agent & tool registry

The number of agents available is rising, and it’s not predicted to stop anytime soon. I see new agents or new tools being made available in my company every week. Every new agent or tool would allow our users to do more things, but every time, we needed to make each agent aware of the latest tools available, and this maintenance can quickly become the bottleneck. This is an N-to-N integration problem: it does not scale.

To solve this, companies including Google, Microsoft, Hugging Face & GoDaddy created the Agentic Resource Discovery (ARD) protocol. This protocol facilitates the creation of catalogs of agents and tools, such that an agent could ask the catalog “What is available for this task?” and get the best-suited agent or tool for it. This changes the integration problem from N-to-N to N-to-1.

Once you start having multiple agents and MCP servers in your company and in your products, building an ARD catalog and giving your agents access to it (through an MCP, or an API) can drastically simplify the connection between your agentic resources. It also allows use cases to surface naturally, as agents become able to adapt themselves to the queries of your users, and figure out the best tools to address the needs.

Getting started

When several teams face the same problem, they will solve it in slightly different ways, in parallel, without knowing about each other’s work. For engineering leaders, this should be a clear sign that the problem shouldn’t belong to any one of those teams, and that a behind-the-scene, horizontal approach would be helpful. That’s exactly the observation that got us databases-as-a-service, Kubernetes platforms, and golden paths a decade ago. It’s happening again, faster, for AI.

I’ve walked through five components, and there could be more we haven’t covered. This is not a blueprint for a perfect AI platform, but I believe it sets a valuable foundation, solving most of the frictions and issues I have encountered working with teams building AI-powered features. Just like traditional internal development platforms, the AI development platforms should then evolve based on the successful and relevant use cases, to better serve the teams using them.

I don’t think anyone has this fully figured out yet, and given how fast the underlying models tools and protocols keep moving, whatever we build this year will change shape pretty soon. As always, what matters is taking the first step, rather than than getting the design right immediately. Find the one piece currently causing the most duplicated pain across your teams, build it as an optional service and let teams adopt it because it’s good.