The gap between a proof of concept and a production system is where a significant proportion of AI initiatives fail to progress. Understanding what that transition involves, and planning for it from the outset, is one of the more important things a technology leader can do before committing investment.
This blog draws on Episode 1 of Technically Speaking and covers the lessons our guests have taken from productionising AI.
Overview
Clear definition of success
In conventional software projects, requirements can be defined in a reasonably deterministic way. There is a clear vision for what the system needs to do, and delivery can be measured against it. However, in AI projects, outputs are non-deterministic: the same input can produce different results depending on context, phrasing and model behaviour.
This changes how teams need to approach scoping, measurement and validation. The question of project success shifts from whether an output meets a specification to how accurately it performs against a defined benchmark. This means that a 'gold standard' set of reference outputs, against which model performance can be measured, needs to be established before development begins. A model averaging a score of 8 out of 10 dropping to 7.5 following a model upgrade is concrete, actionable information. It makes it considerably easier to establish whether a project is ready to proceed to the next stage.
The rationale extends beyond the initial build. Models get upgraded, providers deprecate versions and data changes over time as business processes evolve. Without an evaluation framework in place from the start, there is no reliable basis for understanding whether a change to the model or the underlying data has improved performance, degraded it or introduced different failure modes.
Data readiness
Data readiness is a more front-loaded concern in AI projects than in most software delivery projects. Access to data of sufficient quality and volume needs to be established before a project begins, because without it the project cannot proceed. Finding out early is considerably less costly than finding it out late.
This means data assessment belongs in pre-discovery. Organisations that are mapping out where AI could add value across their operations often identify pockets of data that could support several potential use cases, but which first need to be structured, cleansed or made accessible. Use case road mapping should factor in the data engineering work that may be needed before any AI development can begin.
For more traditional machine learning applications - predictive and classification models, rather than large language model applications - the data dependency is even more pronounced. These models rely on historical data to classify or predict future outcomes, which means the first question is often not whether AI can solve the problem, but whether the organisation is even recording the right data. In some cases, the answer is no, and the prerequisite work is building the capability to capture it. The implication is that the short-term plan may be a data engineering project, not an AI project, which can introduce a lag of months before an AI project can begin.
Planned productionisation
Productionisation is often treated as the final step in an AI project when it is more accurately described as a distinct phase with its own requirements, timelines and risks. In regulated environments in particular, it can be more demanding than the development work that precedes it.
Going live as a service typically requires satisfying formal transition criteria: signoffs from security, testing and operations teams, alongside documentation that may not have been prioritised during a more exploratory proof of concept phase. For AI systems specifically, cyber security testing introduces challenges that have no real equivalent in traditional software projects. AI applications, such as large language models, present an expanded attack surface, including vectors such as prompt injection, data poisoning and exploits that target model behaviour rather than conventional code vulnerabilities. The specialist skills required to assess these risks are still relatively scarce, which means that security review can become a material bottleneck if it is not planned for early in the project timeline.
Adoption and change management
On the human side of productionisation, a technically sound AI system can still fail to deliver value if adoption is handled poorly.
Two practices make a consistent difference to adoption. Firstly, involving sceptics during the design process, rather than waiting until rollout, surfaces the objections that are most likely to obstruct take-up and gives the team an opportunity to address them before they become entrenched. If those sceptics can be converted into advocates through that process, they tend to be considerably more credible and effective champions than people who were enthusiastic from the start.
The second is to think carefully about how users interact with the system. Presenting people with an open-ended prompt interface is not always the most effective approach, particularly for users who are not accustomed to interacting with technology through natural language. Structured inputs, guided forms and clearly defined interaction patterns often produce better outcomes and reduce the amount of training required.
This reflects the broader reality of how AI projects are constituted. A substantial proportion of what gets built in an AI initiative is conventional software engineering and data engineering. The AI model is often a relatively contained component within a larger system, and the quality of that surrounding system matters considerably to whether the overall solution delivers its intended value.
Governance
A common assumption is that governance and guardrails slow down AI adoption. Experience from organisations that have deployed successfully tends to suggest the opposite - clear governance is often what makes adoption possible at all.
Defining clearly what data can be used with AI tools and making explicit that individuals remain accountable for any output they act on, gives people a basis for using the technology with confidence. Without an approved tooling framework, uncertainty about what is permitted can lead many people to avoid the tools entirely, or use publicly available tools, potentially exposing sensitive data to systems outside the organisation's control and contributing to external model training in ways that may not be intended or sanctioned.
For people to remain accountable, the right processes need to be in place. AI systems in most production contexts are not making autonomous decisions. They are reducing the effort involved in reaching a decision or producing a draft output that a human reviews and acts on. Keeping humans in the loop and ensuring that outputs are always editable maintains accountability for AI decisions - and tends to support adoption, because users are more willing to trust a system they can oversee and correct.
Ongoing ownership and operational costs
An AI system needs ongoing ownership, monitoring and support after go-live. Models change, sometimes during active development, and organisations need a process for evaluating and responding to those changes without disrupting production systems. Additionally, data drifts over time and performance can degrade in ways that are not immediately visible.
The cost implications also need to be considered carefully. Infrastructure costs that appear manageable during a proof of concept, running against limited data at low volumes, can scale significantly when the system is handling production-level usage. Token costs for large language model applications are a specific area to watch. Provider pricing has been subsidised during the early adoption phase, and there are reasonable grounds to expect that will change as the market matures. Stress-testing the return on investment against scenarios involving meaningful price increases is a sensible part of any business case.
Prioritised use cases
Early AI deployments, when they go well, can create increased internal demand. That is a positive outcome, but it requires a disciplined approach to prioritisation of use-cases.
It is worth being clear about what kind of value is being pursued. For example, in most cases, the goal is not to reduce headcount but to free up specialist capacity for higher-value activity. In environments where people take years to develop the expertise the organisation depends on, the productivity argument for AI is about removing low-value, time-consuming work from those specialists, not replacing them.
Pre-discovery benchmarking is a useful tool here: measuring how long a task currently takes and using that as the basis for estimating the value of any efficiency gain. Some teams find it difficult to express their work in those terms, but even approximate figures provide a consistent basis for comparing use cases and making a reasoned case for where investment will have the most impact.
It also needs to be considered whether AI is the right tool for the value the organisation is trying to add. It could transpire that a traditional software engineering project is needed, or that a tool already exists, in which case investment is better directed elsewhere.
Ability to stop
The decision to continue an AI project needs to be subjected to sufficient scrutiny at the right points. Insufficient upfront business analysis means teams can find themselves well into a build before discovering that the underlying use case is not well-suited to an AI approach, or that a more straightforward software solution would deliver equivalent value with considerably less complexity.
What protects against this is building genuine decision points into the project structure: stages at which the question of whether to continue is asked honestly, with objective criteria for the answer. Phase-by-phase contracts, rather than large fixed-scope agreements, support this kind of structured reassessment. So does a regular delivery cadence that gives stakeholders something concrete to evaluate at each stage, rather than deferring judgement to a single release event.
It becomes progressively harder to stop a project as investment accumulates, regardless of whether the evidence supports continuing. Teams that handle this well are typically operating with a leadership culture that treats an early, well-evidenced decision to stop as a legitimate and valuable outcome, rather than a failure to be managed.
What's next for AI productionisation?
Agentic AI
Agentic AI, in which systems take sequences of actions autonomously rather than producing outputs for human review, is likely to be a significant area of development and investment over the next few years. The potential value is substantial, but so are the governance and risk management questions it raises. The sensible starting point is human-in-the-loop systems that allow organisations to gather real evidence about how reliably the AI performs at each decision point, and to use that evidence as the basis for decisions about where further automation is appropriate.
LLM Token FinOps
The economics of large language model deployment are also likely to evolve. As provider subsidies are withdrawn, organisations that have not built robust evaluation frameworks will find it difficult to assess whether their existing tools remain cost-effective, or whether alternative approaches such as smaller, locally-hosted models would better serve their needs. Those that can measure performance objectively will be better placed to make those judgements.
Standard practice
Looking further ahead, the expectation is that AI will become a standard part of how organisations design and improve their services, rather than a distinct category of initiative requiring its own governance and planning apparatus. Getting to that point requires working through the current phase carefully, building the patterns and the institutional knowledge that make AI deployment reliable and repeatable, rather than treating each new project as an experiment conducted from scratch.
This was an episode of Technically Speaking, Audacia's podcast series for technology leaders navigating real decisions in complex organisations.
Future episodes will explore the challenges shaping technology leadership today, whether that is navigating the governance questions that come with AI, rethinking what technology leadership looks like or building the infrastructure and culture that makes it all possible.
Listen on your preferred platform here.
If there are topics you'd like to see covered, we'd welcome your input at [email protected]
