Why AI pilots never grow up

An AI pilot can prove that a model performs a task, but that is very different from proving an organisation can use it reliably. Too many projects end before anyone has addressed the data, integration, governance and human change required to put the technology to work.

AI pilots rarely end with an admission of failure. The demonstration works, the presentation records encouraging results and those involved agree that the technology has shown promise, yet months later the system remains confined to the same small group while the organisation moves on to another experiment. This is often treated as evidence that AI is not ready for the business, when the more uncomfortable explanation is that the business was never prepared for what followed.

Most pilots are protected from the conditions that make production difficult. Data can be cleaned, integrations simulated and outputs reviewed by specialists who understand the model’s limitations. Production removes that protection and introduces changing information, security controls, operational exceptions and users who may not have helped design the workflow. The trial has shown that AI can perform under favourable conditions; it has not shown that the organisation can support it as part of everyday work.

The wrong starting point

Pressure to show progress has encouraged businesses to begin with a product and search afterwards for a problem it might solve. Executives see competitors announcing initiatives, vendors promise rapid gains and internal teams are asked to produce something tangible before the purpose has been established. Justin Sharrocks, Managing Director UK and Europe at TrustedTec, sees that urgency creating a gap between what leadership believes it has purchased and what the technical team knows it must still build.

“Most projects stall because they are framed as quick wins,” he says. “They are not necessarily understood or aligned to business goals or problems. Leadership wants to get it in fast, but the technical team is dealing with integration, understanding what data is going into the vendor and what human adoption looks like at the back end. We are not preparing customers for what comes next.”

AI has intensified an old digital transformation mistake: introducing technology as though it will create its own business case. A model may perform effectively, yet the business must still decide who will use the output, which decision it improves and whether the value justifies production. Speed at the beginning can simply postpone the questions that determine whether the investment survives.

At Version 1 AI Labs, Filippo Sassi’s team begins before a model is chosen. Co-creation workshops bring customers, process owners and intended users together to test whether the proposed problem matters and whether AI is the appropriate response. “There are organisations that come to us and say, ‘We have data, let’s do AI,’” Sassi explains. “What is the problem you are trying to solve? The biggest risk is doing AI for the sake of it, not looking at the right problem and not using the right solution.”

What the demonstration leaves out

A contained pilot can avoid much of the work that production demands, which is one reason the economics change as soon as the project begins to scale. The licence or platform fee is only the most visible expense. Organisations may need to clean data, review permissions, redesign workflows, train employees and monitor model behaviour. Technical teams must also decide where the system will run and how it will be secured.

These requirements do not make the project unviable, but they must be understood before the demonstration is presented as a success. Sharrocks argues that a readiness assessment should expose the hidden effort and show where adoption is genuinely likely, because the same product can produce very different results across companies and departments. “We need to define what success looks like from the beginning,” he says. “The vendor should be open about the limitations, the additional work and why it may take longer to reach the adoption metrics.”

Lori MacVittie, Distinguished Engineer at F5, sees readiness as an operating capability rather than a collection of technology assets. Using a managed chatbot may provide useful experience, but it does not mean an organisation knows how to orchestrate models, observe them, apply consistent security or scale them across business processes. Data and cloud infrastructure are necessary, but they are only the beginning.

“If you are going to use AI, you need data and it needs to be labelled and governed, but that is table stakes,” she says. “How are you going to scale it? Where are you going to run inference, which model will you use and how do you secure it? Are you taking advantage of the data in that lake, or are you just turning it into an ocean where you might get something good, but you are not really sure?”

Early success can create confidence before those disciplines are in place. An organisation may have several models in use, yet data labelling remains inconsistent, policies drift and nobody is watching for changes in behaviour. AI does not repair a weak process. The decision to scale must include the workflow around the model, identifying what needs redesigning, what can be automated and where human judgement remains necessary.

The people inside the process

Technical success can conceal a more basic design failure when intended users were not involved early enough. A pilot team may understand the model while knowing far less about the exceptions, informal decisions and workarounds that keep the real process functioning. Those details become critical when the system enters everyday use.

Ian Roberts has seen this particularly clearly in public services. As Managing Director at Granicus, he works with organisations where a simple transaction may cross several departments. “Sometimes the experimentation and the technology work well, but the user experience has not really been worked through,” he says. “Internal stakeholders, staff involved in delivering the service and representation from the community are critical. We have seen good pilots never get out of the gate because the user or the staff delivering that service were not involved early.”

Their involvement is not simply a way to secure acceptance. Frontline employees know where information is missing, which decisions require judgement and what happens when the standard route does not fit the circumstances. An AI assistant may answer a resident’s question accurately, but the service has not improved if the next step sends that person into a fragmented process with no confirmation or progress update. The technology has performed while the outcome remains poor.

Sassi’s co-creation model places users inside development rather than asking them to approve the result at the end. “We try to include users at every point,” he says. “That means the decision-makers, the people currently doing the actions, the testers and the people who will need to use the solution. We design it with them.” This gives the workflow a better chance of reflecting operational reality and reveals where training, trust and human oversight will be needed.

Production is an operating commitment

A demonstration asks whether the technology can perform; production asks whether it can continue performing while the data, systems and organisation around it change. Once deployed, information must remain current, policies maintained and performance monitored against measures that reflect the consequences of an error. A system drafting an internal summary does not require the same controls as one influencing a public service, financial decision or employee outcome.

Sassi argues that evaluation cannot end when the model is launched. The infrastructure must be secure, compliant and auditable, while the data and algorithmic controls need continued attention. “Once it is in production, you need to keep an eye on it,” he says. “You keep monitoring the metrics and use the human in the loop to understand whether the model is still performing as it is supposed to.” Where performance changes, someone must investigate the cause and decide whether the system remains fit for use.

MacVittie reaches a similar conclusion from the enterprise architecture perspective. AI should be treated as a business capability offered securely to employees and applications, not as another isolated app attached to an existing environment. This forces organisations to decide which processes genuinely benefit from AI, which should remain under tighter control and whether the architecture can support automation beyond the pilot without multiplying risk.

A useful pilot should expose the work required for production rather than hide it. It should reveal weaknesses in data, identify integration and governance requirements, involve the people who understand the process and provide enough evidence to judge whether the outcome justifies the investment. Many projects remain in pilot because experimentation creates activity without forcing leaders to accept responsibility for ownership and long-term operation.

The dividing line is not the number of users or the size of the data set. It is the point at which the organisation accepts responsibility for running the system, supporting those who use it and measuring whether it continues to deliver the intended result. Until then, even the most impressive demonstration remains proof that the AI can perform the task, but not that the business is ready to put it to work.