AI strategy & operational transformation
Back to homeARLAN GREY · STRATEGY & ARTIFICIAL INTELLIGENCE

Arlan Grey perspective

AI agents: set the human oversight budget before scaling up

| 4-minute read

An agent can work correctly and still cost more than the AI-assisted workflow it replaces. Before expanding its use, a leadership team must assess exception-handling costs against the required service level.

The right basis for comparison is the fully loaded cost of an accepted outcome, at comparable quality and turnaround times. Human oversight is part of this calculation, as highlighted by McKinsey in its analysis of agent economics.

A faster agent can overwhelm human review

Consider an entirely hypothetical case: 120 cases per day, of which 20% require an eight-minute human review. That means 24 cases need review. With two hours available, the team can review fifteen. With a constant inflow, no initial backlog and no other constraints, nine additional cases remain pending each day.

Adding 72 minutes per day only keeps pace with the average inflow. Assume an additional capacity buffer of 48 minutes: four hours of human oversight per day in total, including the two hours already available. This hypothetical buffer still needs to be sized for peaks, absences and any backlog.

Calculate the human oversight budget

Over twenty working days, both operating models handle the same 2,400 cases, with accepted outcomes, equivalent quality and equivalent turnaround times assumed. All amounts are hypothetical. The AI-assisted workflow, in which AI prepares the work and a human decides, costs €6,200, including an appropriate capacity buffer. The agent costs €2,000 excluding human oversight; each hour of oversight capacity costs €60.

Comparison of fully loaded costs over twenty days
Operating modelFully loaded cost over twenty days
AI-assisted workflow€6,200
Agent, with 80 hours of human oversight€2,000 + €4,800 = €6,800

These fully loaded costs include setup costs allocated over the chosen time horizon, operations and rework. The economic difference does not automatically translate into cash savings.

Under these assumptions, the agent costs €600 more. Scaling it up to reduce costs is not justified.

The maximum human oversight budget at cost parity is €4,200, or 70 hours. Reviews already consume 64 hours, leaving six hours of capacity buffer, or 18 minutes per day. The capacity buffer needed to maintain the service therefore determines the decision.

If the 16-hour capacity buffer is necessary, 54 hours remain for 480 reviews: 6 minutes 45 seconds per review at cost parity. The process must support this review time without compromising oversight.

At six minutes, the cost would fall to €5,840, with costs outside human oversight unchanged. The favourable difference of €360 must cover any additional improvement costs allocated to the same time horizon.

Choose the option that holds up under testing

Options and decision criteria
OptionDecision criterion
Retain the AI-assisted workflowA baseline that costs less than the estimated expansion.
Scale up the agentDemonstrate reliable capacity and a lower fully loaded cost, or a documented additional benefit that covers the difference.
Target a narrower scopeRecalculate review frequency and duration, including how excluded cases are handled.

Our recommendation: retain the AI-assisted workflow under the current assumptions. The scaling test is successful if, with representative case flows, quality and turnaround requirements are met, the backlog is under control and the fully loaded cost stays within the agreed budget. Otherwise, retain the AI-assisted workflow or reconsider the scope.

Research by BCG on process redesign sheds light on this option: redesigning how work is organised can change the economics of the operating model.

Before approving a budget for scaling up, the Diagnostic de décision IA can be scoped around a decision note: required capacity, a comparison of costs, a recommendation and conditions for reassessment. The scope and required data are agreed before the engagement.