The scoreboard is in
Three years into the copilot era, the scoreboard has stopped being ambiguous.
In McKinsey's August 2026 survey, 80 percent of respondents said AI had made them personally more productive. Thirty-seven percent attributed any enterprise EBIT impact to it, essentially unchanged from the year before. BCG found 60 percent of companies reporting minimal revenue and cost gains from AI. And on 4 November, Gartner reported that 4 percent of the 275 customer service leaders it surveyed had achieved the return they expected from AI.
Customer service is where the chatbot was supposed to pay first. Four percent.
People feel faster. The P&L has not noticed. In commercial functions I think the gap has one main cause, and it is the shape of what we bought: a chat window bolted onto an unchanged process, handed to everyone, owned by no one and measured by usage.
This essay argues for the alternative. It is harder, less photogenic and far better supported by evidence. Take one commercial decision. Give the routine version of it to a model, keep the exceptions with people, and prove the result against a holdout.
One statistic is missing on purpose. MIT NANDA's claim that 95 percent of generative AI pilots return nothing is the most quoted number in this debate. Its sample is small and its data has not been released. The case against chatbot-first does not need it.
Why the chat window fails in commerce
Copilots save minutes nobody collects
The UK Department for Business and Trade ran one of the few independent evaluations of Microsoft 365 Copilot: 1,000 licences, October to December 2024. Satisfaction was high, at 72 percent. The evaluation still found no evidence that the time savings had improved productivity. In observed tasks, Copilot users were slower and less accurate on Excel analysis and produced lower-quality slides. Summarising reports was the one task where they were both faster and more accurate. Australia's Treasury reached a similar verdict after its own 14-week trial: help with basic administration, no clear evidence of better work.
Self-reports flatter. In a randomised trial by METR, experienced developers using AI tools took 19 percent longer, then believed they had been 20 percent faster. Most copilot business cases rest on exactly that kind of self-report.
The minutes that are real leak away. In BCG's June 2026 survey of 11,749 workers, 66 percent of frontline employees said they get limited or no guidance on what to do with the time AI saves them. A minute saved in a calendar becomes a minute with a customer only if someone redesigns the work around it. I made that argument in Your AI Saved an Hour. Who Gets It?, and the commercial evidence keeps confirming it.
Copilots are not useless. They help with drafting and summarising, they lift less experienced people, and broad access builds the literacy a transformation needs. Budget them as capability. Never report them as the transformation.
A customer bot speaks for the company
In February 2024 a Canadian tribunal held Air Canada liable for a refund policy its website chatbot had invented, and rejected the airline's argument that the bot was responsible for its own actions. The same month, Klarna announced that its assistant was doing the work of 700 agents. Fifteen months later its chief executive conceded that cost had carried too much weight and quality had suffered, and Klarna began hiring people again.
Customers saw this coming. In a Gartner survey of 3,566 customers run in early 2026, 87 percent said a company using generative AI in service must offer a way to reach a person. Half said AI made their interactions easier, which is the fair counterpoint. Gartner's advice was blunt: do not make AI a mandatory first step for every issue.
Industrial B2B raises the stakes. A specification, a compatibility claim, a lead time or a price given to a customer is a commercial commitment, at ticket sizes far above an airline fare. And B2B buyers are clear about where they want a person. Gartner's 2025 buyer survey found 61 percent preferring a rep-free experience overall, yet buyers still preferred seller input on whether a product actually fits their needs. Sixty-nine percent had already found inconsistencies between a supplier's website and what its sellers told them. A chatbot adds one more voice that can contradict the other two.
Disclosure is the next problem. In a field experiment with more than 6,200 customers, undisclosed AI sales callers sold as well as proficient humans. Telling customers it was a bot before the conversation cut purchases by 79.7 percent. In the EU, telling them is now the law: Article 50 of the AI Act has applied since 2 August 2026 (I covered what it requires in The Record Behind the Label). Any chatbot business case built on undisclosed performance is already out of date.
Deflection, a chat that ends without reaching a person, is a vendor metric. Resolution, repeat contact and churn are business metrics. Most chatbot business cases are lost in the space between them.
Agents cannot hold the conversation yet
Salesforce's own CRMArena-Pro benchmark covers B2B sales, service and configure-price-quote tasks. Leading agents completed about 58 percent of single-turn tasks and about 35 percent of multi-turn ones, and showed almost no awareness of what information was confidential. Executing defined workflow steps was their strongest skill, above 83 percent.
Read that pattern carefully. Agents are already good at running defined steps and poor at holding an open commercial conversation, which is the opposite of how most of them are being sold. Gartner expects more than 40 percent of agentic AI projects to be cancelled by the end of 2027.
What the controlled trials reward
Strip out the vendor case studies and a smaller, sharper body of evidence remains: randomised and field experiments that measured what happened when AI entered real work.
The gains share a profile. In the best-known study, 5,172 customer support agents given an AI assistant resolved 15 percent more issues per hour on average, and the working paper put the gain for novice and low-skilled agents at 34 percent. At Procter & Gamble, one professional with AI performed as well as a two-person team without it, and AI narrowed the usual gap between technical and commercial proposals. On a large online retail platform, seven randomised deployments produced sales effects from zero to 16.3 percent. The winner was a pre-sale assistant that answered buyers' product questions at the moment of decision. AI-written product descriptions, where many marketing teams start, added about 2 percent.
The losses share a profile too. BCG consultants working on a task just outside the model's capability were 19 percentage points less likely to reach the right answer than colleagues without AI. The most experienced support agents gained little and saw small declines in quality.
AI wins on narrow, frequent, checkable work where it assists a person. It loses on open-ended work beyond the model, and when customers can see it and do not trust it. Most commercial AI has been deployed the other way round.
Pricing has the strongest B2B evidence
If you read one study before your next commercial AI budget meeting, make it Karlinsky-Shichor and Netzer in Marketing Science (2024).
At a B2B aluminium seller, the researchers built a model of each salesperson's own pricing from transaction data and showed that model's price to the salesperson in real time while quoting. Nothing exotic: no new demand model, no autonomous agent. Profit on treated quotes rose 11 percent against a control group. The gain came from removing each seller's inconsistency, the behavioural noise every sales director recognises.
Then comes the finding that matters most. Salespeople still beat the model on out-of-the-ordinary and complex quotes, and a router that sent each quote to whichever of them would price it better outperformed both.
The surrounding research supplies the operating rules. In a field experiment at an auto parts retailer, merchants' free overrides of a pricing tool cut profit by 5.77 percent, although overrides helped on growth-stage products. People use an imperfect algorithm far more readily when they are allowed to adjust it slightly. At Zara, managers followed clearance-price recommendations more once revenue, rather than inventory, was the number in front of them (Caro and Sáez de Tejada Cuenca, 2023).
I call this guided discretion. The model owns routine decisions and people own the exceptions. Every override is logged and reviewed against outcomes, and success is measured against a holdout rather than by model accuracy.
Now put that next to the arithmetic every commercial leader knows. In the classic Harvard Business Review analysis of 2,463 companies, a 1 percent price improvement at constant volume raised operating profit by 11.1 percent on average. Yet in BCG's survey of 400 industrial goods companies, a third still managed pricing in Excel. The biggest profit lever in the business has the best causal evidence behind it and still runs in spreadsheets, while the budget goes to chat windows.
Where the money sits in an industrial commercial engine
Three value pools stand out, and none of them is a conversation.
Price and the quote-to-order flow
Reading RFQs, matching requirements to products, chasing prices and lead times and assembling offers consume a large share of inside sales and application engineering time. Every output in this flow can be checked against an ERP price, a product master and a delivery calendar before a customer sees it. That checkability is the whole point.
The installed base
Across 30 industries, McKinsey found aftermarket services averaging about 25 percent EBIT margin, against about 10 percent for new equipment. Caterpillar has said that service leads generated from component-life data, service invoices and work orders produced around $1 billion in dealer parts sales. That is a company claim reported in the trade press, from around 2023. Its mechanism is still the right one: the value came from joining asset, service and sales records, not from a smarter model.
Routine ordering
Here industrial buyers have already voted, and they did not vote for chat. Fastenal's vending and managed-inventory programmes accounted for 44.6 percent of its sales in mid-2026. The self-service channel that won in industrial distribution is machine to machine, and it largely predates generative AI. That is the benchmark a supplier's chatbot has to beat.
One uncomfortable fact runs across all three pools. In the research behind this essay I found no named industrial manufacturer or distributor that has published an AI-attributed margin or price-realisation figure. The sector's commercial AI story is still told mostly in vendor case studies. The first company to publish a holdout-measured result will set the benchmark for everyone else.
The one chat interface I would defend
I have built a customer-facing AI assistant myself, so the disclosure comes before the example. Before joining ABB, where I am Global Head of Digital Marketing, I led strategy and delivery of Holcim's AI ordering assistant, and I discussed its Spanish pilot in a McKinsey interview published in February 2025. I use only published figures here, graded like every other company claim in this essay, and no internal information from either company.
Construction customers used to order cement by phone. The assistant let them order through WhatsApp. It used order history to propose an order, the customer adjusted and confirmed it, and a person was always reachable. As reported in the interview, digital ordering adoption in the pilot moved from 25 percent to 93 percent, and around 66 percent of the assistant's initial order proposals were accepted. Those figures cover that pilot only. They are not an audit, and they say nothing about revenue.
Testing also showed what happens without guardrails. As Global Cement reported from the interview, testers tricked the assistant into almost ordering a truck of gazpacho soup.
What worked was the bounded transaction behind the chat window: a known customer, a known product list, an order history to propose from, an order checked against the system of record before it is confirmed, and a person one step away. Customers increasingly want exactly that. In Gartner's 2026 survey, 74 percent of B2B customers who used generative AI had used it to complete a task on their behalf. They want AI that gets something done within clear limits.
The full case sets out what another business would need to decide before scaling.
Three ways in
Choose your entry point by where your system of record lives, and start with one domain whose P&L has a named owner.
| Path | Best fit | First release | Measure against a holdout |
|---|---|---|---|
| Quote-to-order with price guidance | Manufacturers and distributors with high RFQ volume and years of quote and win-loss history | AI reads RFQs into structured lines, matches products, pulls ERP price and lead time, and drafts the quote with a model-guided price. The seller approves; unusual quotes go to experts | Cycle time, straight-through rate, rework, price realisation, discount leakage, win rate |
| Installed-base aftermarket | Equipment makers with well-recorded fleets and service or dealer networks | Join asset, service and sales records, then push prioritised parts and service opportunities into seller or dealer CRM with the reason attached | Lead conversion, service revenue per installed asset, renewal and attach rates |
| Bounded repeat ordering | Standard products bought repeatedly, such as building materials, MRO supplies, consumables and spares | Order capture in channels customers already use, validated against the customer master, price list, delivery sites and availability, disclosed as AI and one step from a person | Share of orders captured, order error rate, cost to serve, repeat contact |
For most industrial manufacturers and distributors, my default is the first path. It carries the only causal B2B result, it works on the largest profit lever, and every output can be checked before a customer sees it. Its first release faces employees, which avoids both the disclosure penalty and chatbot liability. The honest caveat: no published study yet shows quote automation lifting industrial win rates or revenue. You will have to generate that evidence yourself, which is one more reason to design the measurement before the software.
Equipment makers whose service margins dwarf their equipment margins should look hard at the second path. Businesses dominated by repeat orders of standard products can start with the third, as long as every order is validated before it is confirmed.
Rewire the decision, not the screen
The redesign that produces value fits on one page.
Write down which decisions the model owns: routine quotes inside a price band, standard reorders, lead-time lookups. Write down which the seller owns: unusual configurations, strategic accounts, growth products and anything without precedent. Then define what triggers a handoff between the two.
Log every override and review overrides against outcomes each month, because in the evidence above unguided overrides cost profit and helped only where products were still growing. Let sellers adjust a recommendation within a band, and show them why the price was recommended. A seller who cannot see the reasoning will discount around it.
Align what sellers see, and what they are paid on, with what the model optimises. No field experiment has yet tested sales-compensation redesign directly, so treat this as reasoned design rather than proven effect. But a model that maximises margin, deployed into a team paid on volume, will be overridden every time it matters.
Deploy first where the measured gains are largest: inside sales, new hires and channel partners, before your best key-account managers. In one AI sales-coaching experiment, middle-ranked agents gained, the weakest were overwhelmed by feedback and the strongest resisted it.
Gates that keep it honest
No published standard sets numerical stage gates for commercial AI. These are my synthesis of the evidence, and they are deliberately strict.
Before the pilot starts, record four to eight weeks of business baseline. Set up a holdout group or a staggered rollout, because self-reported time savings have proved wrong even in direction. Keep a held-back set of real cases to test against, and write down a dated kill criterion.
To scale, require a credible lift on the business metric against the holdout, override and error rates within agreed limits, and consistent results across repeated runs rather than one good demo. The cost per successful outcome, review effort included, has to beat today's cost at production volume.
To give the model more autonomy, for example auto-confirming quotes below a value and discount threshold, require a sustained low override rate, an audit trail for every step and a re-test after every model change. The verification ceiling applies here: never let the model produce commitments faster than someone can stand behind them.
Stop at about six months if the P&L has not moved, if unit economics worsen at volume or if users route around the system. Keep the first wave to one to three domains. Spreading bets thinly produces activity instead of value, which is the argument of AI Pilots Are a Form of Procrastination.
Where chat, copilots and agents still belong
None of this bans the tools that were oversold. It puts them where the evidence supports them.
Copilots belong in the literacy budget. Customer-facing chat belongs where the question or transaction is bounded and checkable: repeat orders, availability, order status and product selection at the point of decision. Disclose it and guarantee a person. Autonomous agents come last, inside a workflow that already works with people, after their consistency is proven, with the gate re-run every six to twelve months as the technology improves.
One more shift deserves a plan. Customers are bringing their own AI. Gartner found them about three times more likely to use third-party tools such as ChatGPT than a company's own chatbot for service issues. Their use of third-party tools nearly doubled in a year, while company chatbot use did not move. If buying follows service, accurate, structured, machine-readable product, price and availability data will matter more than any chatbot a supplier builds. That is my inference, not a measured result. I would act on it anyway.
The asset is underneath
The executive question has changed. It is no longer which assistant to buy. It is which commercial decision to hand to a model, with what rights, measured against what holdout.
Framed that way, the competitive asset is not the model, because every rival can rent the same one. The asset is what sits underneath it: years of quote and win-loss history, a clean customer and product master, a mapped installed base, and a leadership team willing to let a model own the routine while sellers own the exceptions.
Start on Monday. Pull last quarter's quotes. Mark which were routine and which were genuinely unusual, then compare discount and win rate between the two groups. That split is your first map of decision rights, and it already holds more strategy than most chatbot roadmaps.
How this essay was researched. AI research agents helped find and organise sources. I checked the load-bearing ones, chose the argument and wrote the essay. Figures are current to 7 November 2026, and the references below state what each source supports and where it stops.