Case 001 · Service operations
Illustrative scenario · Hypothetical figures
The AI answer improved.
The business outcome barely moved.
A service organization wants to reduce the time it takes to resolve customer requests.
The team introduces an AI assistant to help employees find information and formulate better answers.
The pilot appears promising.
- Answer quality
- +22%
- Initial response time
- −38%
- End-to-end resolution time
- −4%
The first two measures improve much more than the end-to-end outcome.
That does not prove where the bottleneck sits.
It tells us where to investigate next.
What would you investigate?
The instinct may be to improve the model, retrieval, or prompts.
But the more important question is:
Where is the work actually getting stuck?
The business ambition
Start with the outcome.
The objective was not to produce a better answer.
It was to resolve customer requests faster without reducing service quality.
Those are different outcomes.
A customer request may move through several steps:
Request → Search → Interpret → Decide → Approve → Execute → Resolve
AI may improve one stage substantially while another stage limits the end-to-end result.
But before concluding that, validate that the measurements are comparable.
Check:
- whether the metrics use the same population of requests,
- whether the measurement periods are comparable,
- whether request complexity or case mix changed,
- how each metric was defined,
- whether the observed differences are practically meaningful,
- and what uncertainty exists in the measurement.
Only then should the workflow diagnosis begin.
A metric gap is a reason to investigate the system — not proof of where the bottleneck sits.
Hypothetical inputs
Hypothetical evidence
Assume the figures above represent relative changes against an agreed baseline.
They are illustrative only.
Without the baseline volumes, measurement method, case mix, cost implications, and uncertainty, a 4% improvement in resolution time cannot be judged as commercially small or unimportant.
At scale, even a modest percentage change may matter.
The relevant observation is simply this:
The end-to-end outcome changed much less than the upstream measures.
Why?
The investigation
Where is the work actually getting stuck?
Several constraints may coexist. Choose one starting hypothesis to see how the evidence, intervention, and system implications change.
Hypothesis 1 — The team cannot find the answer
What may be happening
Employees may be spending too much time locating reliable information across policy documents, customer history, product information, and disconnected systems.
What to test
Measure how much of total resolution time is actually spent:
- searching for information,
- verifying whether information is current,
- resolving conflicting sources,
- requesting information from another team.
Also examine cases where employees already had the correct information but the request still remained unresolved.
What might help
If information access is genuinely the constraint, the intervention may range from:
better source management → improved search → structured retrieval → retrieval-assisted AI
The architecture should follow the information problem.
A large retrieval stack is not automatically necessary.
The relevant question is whether the chosen approach supplies the:
right context, from the right source, at the right time, with sufficient reliability.
What would change the decision?
If information-access measures improve but end-to-end resolution does not move proportionally, this weakens the information-access hypothesis.
Before drawing a conclusion, check:
- downstream waiting time,
- request mix,
- measurement comparability,
- and whether other workflow conditions changed during the same period.
Then investigate the next plausible constraint.
Measuring the outcome
What would count as success?
The original business objective gives us the evaluation frame.
Do not stop at:
answer quality
or
response latency.
Follow the chain:
- Answering-task performance
- Task performance
- Workflow performance
- Business outcome
For this case, useful measures might include:
- End-to-end resolution time
- First-contact resolution
- Repeat contacts
- Incorrect actions
- Human review effort
- Exception rate
- Cost per resolved request
- Customer experience
And interpret them against:
- a clear baseline,
- comparable request populations,
- case complexity,
- operating cost,
- measurement uncertainty,
- and the practical value of the change.
A technically stronger system has created business value only when the improvement survives the rest of the workflow and produces an outcome that matters economically or operationally.
The decision
The decision
The hypothetical evidence does not tell us to build a better model.
It does not tell us to build an agent.
It does not prove that a downstream bottleneck exists.
And it does not tell us that AI failed.
It tells us something more useful:
The relationship between local AI improvement and end-to-end business performance requires investigation.
The next investment should follow the evidence about the constraint.
If the constraint is information, improve access to information.
If it is decision rights, redesign the decision process.
If it is deterministic execution, automate the workflow.
If variable, context-dependent execution remains the constraint, then greater AI autonomy may be justified.
The technology changes because the problem changes.