Scaling AI FinOps | Lesson 16: Capability, Not Project

The Fox proposed the reorganization himself, which nobody had expected, least of all the Crow.

What he proposed was that his own function should get smaller. The delivery teams he had built up over two years, one per major initiative, each with its own engineers and its own roadmap and its own relationship with a territory, should be collapsed into a single platform group and a much smaller product function. His headcount would fall. His direct reports would fall. The number of things with his name on them would fall considerably.

It was the correct proposal and it was against his own interests in every visible dimension, which is a rare enough combination in an enterprise that the room did not immediately know what to do with it.

The Mandrill did something equally unusual. He did not oppose it politically. He opposed it honestly.

“This makes my costs less controllable,” he said. “Right now if I want something I fund it and I get it. Under this, I join a queue that someone else prioritizes. I understand why it is cheaper for the jungle. I want to be clear that it is worse for me, and I would like that acknowledged rather than explained away.”

Which was fair, and true, and the reason most of these transitions fail. The saving is organizational. The cost is local, immediate, and lands on people who are not being consulted so much as informed.

The distinction, precisely

A project has a scope, an end date and a benefits case. It is funded to complete something, it is staffed for a period, and it is disbanded on delivery. Enterprises are extremely good at this. Everything from the approval process to the reporting to the career structure is built around it.

A capability has an owner, a roadmap and a persistent budget. It serves many consumers over time, it is never finished, and its whole value comes from continuing to exist. Enterprises are structurally hostile to this, not out of stupidity but because every mechanism they have for allocating money assumes an end date.

Almost every organization I have worked with says it is building capabilities. Most are running projects with a longer name. The test is one question: is there a budget line that continues next year without anybody submitting a case for it? If not, you have a project with optimistic language attached.

Where the saving actually comes from

Four sources, and the fourth is the one nobody mentions.

Fixed costs paid once. This is the Lesson 5 arithmetic with a solution attached. The setup tax that every pilot paid separately gets paid once by the platform and amortized across everything that uses it.

Learning that compounds. The same team sees the twentieth use case, and by then it knows which patterns fail, which integrations are painful, and what to say no to. That knowledge exists nowhere in a project model, because the team that learned it was disbanded.

A curve that bends. The marginal cost of use case number twenty is a fraction of use case number four, because the foundation is already there. This is the entire economic argument and it is the one thing a project portfolio can never produce.

Retirement becomes possible. A capability can stop serving a use case without anyone losing their job. In a project model, killing the project kills the team, which means nothing ever gets killed. This turns out to matter enormously and it is the subject of Lesson 22.

Three layers, and the one everyone skips

Platform. The shared technical foundation. The gateway, retrieval, evaluation, guardrails, the operational team. Most organizations build this, and build it competently.

Product. The interfaces, patterns, documentation and support that make the platform usable by people who did not build it. Most organizations skip this, and then are surprised when territories describe the platform as unusable and quietly build their own.

Portfolio. Deciding which use cases get served, in what order, and which do not get served at all. Almost nobody builds this, and it is the layer that produces the saving.

The portfolio layer is where somebody has to say no. In a project model nobody has the standing to, because every initiative has its own approval and its own sponsor and there is no forum where two of them are compared. Consolidate the delivery and skip the portfolio function, and you get a platform that serves every request anyone makes, growing without limit, which is the same cost problem as before with an extra governance layer on top.

The eighteen month problem

Say this out loud, in month one, to the most senior person in the room.

Capability-led is cheaper at scale and more expensive for roughly the first eighteen months. You are building shared foundations before there is enough consumption to amortize them. You are absorbing a transition. You are running the old thing and the new thing simultaneously for at least two quarters, because you cannot switch a live estate over on a date.

Programs that promise immediate savings from this shift get killed in month nine, when the curve is at its worst and the promised improvement has not arrived. It has not arrived because it was never going to arrive that early, and everybody knew that except the person who wrote the business case.

The Crow’s view on this, which I thought was exactly right: “I can defend a curve that gets worse before it gets better. I cannot defend a curve that was supposed to get better immediately and did not, because by then nobody believes the rest of the model either.”

The accountability trade

The Mandrill’s objection has to be answered, not managed, and the answer is an exchange rather than a reassurance.

Territories give up control over how. They no longer choose their own stack, their own model, their own patterns. That is a real loss of autonomy and it should be named as one.

In exchange they get three things, and all three have to be real. A service commitment with actual numbers in it, so the queue has a defined shape. A published roadmap, so they can plan around what is coming. And a seat at the portfolio table with genuine influence over sequencing.

Without the exchange, this reads as centralization dressed up as efficiency, and it gets resisted, correctly, by people who have seen centralization before. With the exchange, it is a trade that reasonable people accept, and the Mandrill accepted it in the following quarter once the service commitment had numbers rather than adjectives in it.

Three ways this goes wrong

Platform without product. Technically excellent, genuinely well engineered, unusable by anyone who did not build it. Adoption stalls, territories rebuild their own, and you now pay for the platform and the duplication.

Capability as a renamed project. Same annual funding cycle, same end date, same approval process, new word on the slide. This is the most common outcome and it is completely invisible from the outside for about a year.

No portfolio function. The capability serves everything anyone asks for, because nobody has the authority to refuse, and the cost curve does not bend because the demand curve has no ceiling.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, fund the capability rather than the use case, and state the eighteen month curve to the board in month one. The honest version of the shape is the only version that survives month nine.

If you sit in the Crocodile’s chair, build the product layer alongside the platform, not after it. Adoption is the business case, and adoption is an interface problem long before it is a capability problem.

If you sit in the Mandrill’s chair, trade control of the how for a service commitment with numbers in it and a real seat at the portfolio table. Then hold both to account. The trade is fair if the other side is real and worthless if it is not.

For everyone: name the portfolio owner. If nobody in your organization can say no to an AI use case and be listened to, you have built a more expensive version of what you already had.

Jungle Lesson 16

A project is funded to end and a capability is funded to continue, and only one of them gets cheaper the twentieth time you use it. Everyone claims to be building capabilities and most are running projects with a longer name, which you can test in one question: who is allowed to say no, and does anybody listen when they do.

Next time: budget season arrives for the second time, and the Watering Hole question that has been open since Lesson 7 finally gets settled, by the last animal anyone expected to propose the answer. The Sloth’s framework arrives, and for once it is exactly on time. Lesson 17 is about persistent funding and quarterly reallocation.

The proposal in this piece exists because somebody was willing to argue for a structure that made their own function smaller. That is rarer than any framework and no amount of methodology substitutes for it.

Scaling AI FinOps | Lesson 15: Agents and the Multiplication Problem

The number arrived on a Monday, which is when these numbers always arrive.

Something had been given a task on the Friday afternoon. It was a reasonable task, given to a system that had been running quietly and correctly for two months, by a team that had done everything properly. The task involved reconciling a set of records against several sources and flagging discrepancies.

Over the weekend it reconciled those records against several sources and flagged the discrepancies. Then, because the instruction had been to be thorough and because it had no reason to believe it was finished, it checked the discrepancies. Then it checked the checks. Then it looked for related records that might also contain discrepancies, found some, and began again.

Nothing failed. No error was raised. No alert fired, because no threshold had been crossed that anybody had thought to define. Every individual step was a correct execution of a reasonable instruction.

By Monday morning it had performed roughly twelve thousand operations on a task that a person would have considered complete after about forty.

The Crocodile read the incident summary, which was not really an incident summary because there had been no incident, and opened one eye.

“It didn’t break,” he said. “That’s the problem.”

What changed

Everything in the previous three lessons assumed a shape: a request goes in, an answer comes out, and you can reason about the cost of that transaction because you know how many of them there are.

Autonomy removes that assumption. A task is no longer one call. It is a variable number of calls, determined at runtime, by the system’s own judgment about whether it has finished.

Read that sentence again, because it is the whole lesson. You have handed the spending decision to the workload. Not the budget, not the ceiling, the actual moment-to-moment decision about how much work this particular task warrants. That is a genuine transfer of financial authority and almost nobody deploying an agent has framed it that way to the person who owns the budget.

The four multipliers

Iteration depth. Steps per task, variable, and unbounded unless somebody bounds it. Most frameworks have a default. Most defaults are generous, because a low default makes the demo fail.

Context re-transmission. The state is resent at every step. This is where Lesson 13 stops being additive and starts being multiplicative. A bloated prompt in a single-call workflow costs you once. The same prompt inside a forty step loop costs you forty times, per task, forever.

Tool call fan-out. Each step may trigger retrieval, a search, a call to another system, each with its own cost, some of which sit outside your AI budget entirely and appear as increased load on systems whose owners have no idea why.

Retry and self-correction. Attempts that fail and are retried internally, succeeding on the third go. These are invisible in every error metric you have, because from the outside the task succeeded. It succeeded three times more expensively than the number in your business case.

Four multipliers, compounding rather than adding. A modest inefficiency in each produces a large number at the end, and none of the four is visible in a per-call cost report.

The variance problem

The most important operational point in this post, and the one I would put on a wall.

Average cost per agent task is a nearly useless number. The distribution is what matters, and the distribution has a tail.

Most tasks complete in a handful of steps and cost very little. That produces a healthy-looking average that will pass every review you put it through. Meanwhile some small proportion of tasks, triggered by inputs nobody anticipated, go around the loop a hundred times. Those tasks are where all the risk lives and they are entirely invisible in the mean.

Manage the ninety-ninth percentile. That is where the weekend incidents are, and the ninety-ninth percentile of an agent workload is often a startlingly different number from the average, in a way that is not true of any other cost line your organization manages.

The jungle’s twelve thousand step weekend did not move the monthly average very much. It was one task. It was also, on its own, a meaningful fraction of the month.

Controls that actually work

Four, and they are not sophisticated, which is what makes their absence so common.

  • Step budgets. A hard maximum number of iterations per task. Set it deliberately rather than accepting a framework default, and set it based on what a competent person would need rather than what feels generous.
  • Cost ceilings per task. Enforced at the gateway, not in the application. A task that exceeds its ceiling stops and escalates to a human, which is a far better outcome than a task that continues.
  • Wall-clock limits. The simplest and most effective control against the weekend scenario specifically. Nothing autonomous should be able to run unattended for sixty hours because nobody was in the building.
  • An external termination condition. This is the important one. The definition of done must not depend solely on the system’s own assessment that it is done. Something outside the loop has to be able to say that is enough.

Task-level attribution

A design requirement rather than a control, and it has to be decided before deployment because retrofitting it is close to impossible.

Cost must be attributable per task, not per call. If your instrumentation records calls, then an agent estate is a fog: you can see that a great many calls happened and you cannot connect them to the units of work that caused them, which means you cannot compute a cost per task, which means everything in Act II stops working the moment anything becomes autonomous.

Every call carries a task identifier. That is the whole requirement, and it is trivial on day one and a project on day four hundred.

The autonomy dial is a budget decision

The framing I would most like a leadership team to take from this lesson.

How much autonomy a system has is usually treated as a safety question, and it is one. It is also a financial one, because autonomy is precisely the property that converts a predictable cost into a distribution with a tail.

More autonomy means more capability on ambiguous work and more variance in what it costs. Less autonomy means narrower capability and a cost you can forecast. Neither is correct in general. Both are correct somewhere.

What should not happen is inheriting the setting from a framework default chosen by somebody who was optimizing for demonstration quality. Choose it, per workload, with the business in the room, and understand that you are choosing a cost distribution and not just a capability level.

Where agents earn their cost

To be clear that this is not an argument against autonomy.

Agents genuinely earn their cost on high value, low volume, previously impossible work. Investigations that a person would not have had time to run. Analysis across sources nobody would have manually connected. Work where the alternative was not a cheaper method but no method.

They rarely earn it on high volume routine work that a single well-routed call handles at a fraction of the cost. That is the same sufficiency argument as Lesson 12, one level up, and it fails the same way: somebody chooses the most capable available approach during a pilot and nobody revisits it.

The question to ask of any agent deployment is what would this cost as a deterministic workflow with three routed calls, and is the difference buying you anything. Sometimes it is buying you a great deal. Often it is buying you flexibility on a task that was never ambiguous.

Three ways this goes wrong

The unbounded loop. No step ceiling, no cost ceiling, no wall clock, discovered by invoice on a Monday.

Mean-only monitoring. A healthy average, a ruinous tail, and no visibility into the second until it arrives.

Agents as fashion. Autonomy applied to work a routed single call would have done better and cheaper, because agents were what the roadmap said and nobody asked what the alternative would have cost.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, stop asking for the average cost per agent task. Ask for the ninety fifth and ninety ninth percentile, and then ask what the hard ceiling is. If there is no hard ceiling, that is the finding.

If you sit in the Crocodile’s chair, enforce step budgets, cost ceilings and wall clock limits at the gateway before anything autonomous reaches production. Limits set inside the application are not limits, they are intentions.

If you sit in the Mandrill’s chair, choose the autonomy level deliberately and understand you are choosing a cost distribution. Then ask what the same task would cost as a deterministic workflow, because sometimes the answer changes your mind.

For everyone: require a termination condition that does not depend on the system’s own judgment about whether it is finished. Everything else in this list is a mitigation. That one is the actual fix.

Jungle Lesson 15

Autonomy is the point at which you stop buying answers and start buying attempts, and the system decides how many. Budget the tail rather than the average, because the average will look entirely fine right up until the weekend it does not.

That closes the third act. The jungle now knows where its unit economics are actually set, which turns out to be in a series of engineering decisions made months before anyone looks at a bill. Whether inference runs on a variable curve or a fixed one. Which tier handles which task. How much context travels with every question. Where the constrained workloads sit and what that costs. How much autonomy the system has and therefore how long its tail is.

Every one of those is a financial decision made by somebody who was not thinking about finance, using a default set by somebody who was thinking about a demonstration.

Which raises a question the jungle has been avoiding since Lesson 5. Who is allowed to make these decisions, how does the money reach them, and what happens to an organization whose entire funding machinery is built around things that end when the thing it needs to fund does not end? That is the fourth act, and it opens with the argument the Fox has been dreading, which is the one where he proposes reducing his own territory. Lesson 16 is about capability rather than project.

If you have anything autonomous in production, the fastest useful thing you can do this week is find out what your most expensive single task cost last month. Not the average. The worst one. Most organizations cannot answer, and the answer is usually the reason to build the controls.

Scaling AI FinOps | Lesson 14: Data Gravity and the Control Plane

The Tortoise had been saying the same three sentences since roughly the middle of the first act, and everyone had been treating them as an obstacle to be routed around.

Some of our data cannot be processed anywhere we choose. Some of our workflows cannot tolerate the delay of a long round trip. And some of our processes cannot survive the system behaving differently next month than it does today.

Every time she said it, the response had been some version of yes, we will handle that, in a tone that meant we will handle it later. She was not being ignored, exactly. She was being categorized. Compliance says a thing, somebody writes it down, the architecture proceeds.

What changed, in the middle of the second planning cycle, was that somebody finally tried to put a number on the third sentence. The system had changed behavior in a way that was entirely reasonable and had required six weeks of re-verification across four workflows, at a cost that turned out to be considerable and that appeared in nobody’s budget because nobody had a line for it.

The Crow looked at the six weeks. Then she looked at the Tortoise.

“You have not been describing a compliance requirement,” she said. “You have been describing a cost driver. For a year.”

“Yes,” said the Tortoise.

“Why did you not say it that way?”

“I did.”

Tortoises are slow, armored, and outlive everyone. The compensation for being consistently right early is that you are eventually right in front of an audience.

Three genuine overrides

Three things legitimately outrank unit price in an architecture decision. Not vaguely. Specifically, and each with its own economics.

Constraints on where processing may happen. Some data carries contractual or policy commitments about where and by whom it may be processed. These arrive from customer agreements, from sector requirements, from internal policy, from a dozen sources. Whatever their origin, from an architecture perspective they behave identically: they are a hard boundary on placement, not a preference to be traded against price.

Latency floors. Interactive and embedded workloads have a ceiling on acceptable response time. Beyond that ceiling the capability is not worse, it is unused, because people stop waiting. Distance is physics and no improvement in unit price fixes it. Either the inference happens close enough or the workflow does not work.

Control of the inference plane. The ability to decide when the behavior of your system changes. To pin a version, to schedule an update, to guarantee that a process which passed verification in March is still doing the same thing in September. For anything high consequence, this is worth real money, and it is almost never priced.

Data gravity as an economic force

Underneath the first constraint sits a straightforward economic observation that is worth separating from any policy question.

Data has gravity. It is expensive to move, expensive to duplicate, and expensive to govern in two places. Once a large corpus exists somewhere, the cost of moving inference to the data is usually lower than the cost of moving data to the inference, and the calculation is dominated by movement, duplication and the governance overhead of a second copy rather than by compute.

The second copy is the part people underestimate. A duplicate of a governed dataset is not a storage cost. It is a second thing to secure, a second thing to keep current, a second thing to include in every access review, and a second thing that has to be deleted correctly when the original is. That overhead is permanent and it usually exceeds the transfer cost within a year.

How to price a constraint

Here is the reframe that changes how these conversations go, and it is the whole practical point of this lesson.

The wrong question is how much does this constraint cost us. That question invites an adversarial answer, positions the constraint as a tax imposed by an unhelpful function, and produces a conversation where somebody argues about whether the constraint is really necessary. Nobody has ever won that argument in a useful direction.

The right question is what is the cheapest architecture that satisfies the constraint. That question turns a veto into a design input. It gives the constrained requirement to the engineers as a parameter rather than a problem, and engineers are extremely good at optimizing within a parameter.

In the jungle’s case the answer, once somebody asked it properly, was considerably cheaper than either side had assumed, because the constrained workloads turned out to be a smaller share of the estate than anybody had estimated and the cheapest compliant option for that share was not the expensive one everyone had been dreading.

Version pinning has a price

The third constraint deserves particular attention because it is the newest and the least well handled.

Behavioral stability is a product feature and it costs something. You can buy it by running the thing yourself and controlling the update cycle, which brings you all the fixed costs from Lesson 11. You can buy it contractually, through commitments about version availability and change notice. You can buy it partially, through your own evaluation infrastructure, which lets you detect and respond to change quickly rather than prevent it.

What you cannot do is need it and not pay for it, which is the position a surprising number of organizations are in. They have processes that require behavioral consistency, running on an arrangement that provides none, and they have never priced the exposure because it only becomes visible on the day the behavior changes.

That is what the six weeks of re-verification was. An unpriced exposure, arriving as a surprise cost, in a quarter that had not planned for it.

The hybrid pattern

What falls out of all this is the practical shape most large organizations end up with, and it is worth designing deliberately rather than arriving at by accident.

Segment the estate by constraint. Constrained workloads are placed where the constraint requires, and their economics are what they are. Unconstrained workloads are placed wherever the economics are best, and are free to move as those economics change.

The critical discipline is doing the segmentation honestly and narrowly. The most expensive default available is applying the strictest requirement in your organization to the entire estate, because segmenting felt like work. That single decision multiplies a constrained cost across every workload that never needed it, and it is extremely common, because it is defensible in a meeting and nobody ever gets criticized for excess caution.

One control plane

The one thing that must not be segmented, whatever else is.

Split inference across placements and you will be tempted to split everything else with it. Separate tooling. Separate teams. Separate standards. Separate evaluation. Within about eighteen months you have two AI programs that do not talk to each other, learn nothing from each other, and cost more than three would have.

The gateway from Lesson 12 sits above both. Same routing, same tagging, same guardrails, same evaluation, same reporting, regardless of where the inference physically happens. Placement becomes an attribute of a workload rather than a fork in the organization.

That is the single decision that keeps a hybrid estate as one capability. Without it, hybrid is just a word for having two of everything.

Three ways this goes wrong

Constraint as afterthought. Architecture chosen on price, constraint discovered at review, expensive rework, and a lasting belief that the constraint function is an obstacle when it was in the room saying so the whole time.

Constraint as blanket. The strictest requirement applied everywhere because segmenting was effort. Enormously expensive, entirely invisible, and almost impossible to reverse once it is the standard.

Two organizations. Constrained and unconstrained estates drifting apart until they have separate teams and separate tooling, at which point you are running two programs and funding three.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, ask what the cheapest compliant architecture is, rather than what the constraint costs. Two different questions and only the second one produces a design.

If you sit in the Crocodile’s chair, segment the estate by constraint before optimizing anything, and be narrow about it. Then check what proportion of your workloads are genuinely constrained. It is usually smaller than the standard you are currently applying assumes.

If you sit in the Mandrill’s chair, bring the constraint into the design conversation at the start. It is a requirement, not an obstacle, and it is dramatically cheaper as an input than as a finding.

For everyone: if you need behavioral stability, price it explicitly and choose how you are buying it. Needing it and not paying for it is a position, and it is the one that produces surprise quarters.

Jungle Lesson 14

Some constraints are not negotiable, and arguing about them only moves the cost to a later and more expensive meeting. Price the constraint as a design input rather than a tax, segment your estate so you are not paying the strictest requirement’s price on workloads that never needed it, and keep one control plane over all of it.

Next time: something runs over a weekend. Nothing fails, nothing errors, no alert fires. On Monday the number is remarkable, and the system turns out to have done exactly what it was asked, twelve thousand times, extremely thoroughly. Lesson 15 closes the third act with the compounding cost of autonomy.

If there is someone in your organization who has been raising the same point for a year and being categorized rather than heard, it is worth asking them to restate it as a cost. It is remarkable how often the same sentence lands completely differently in that currency.

Scaling AI FinOps | Lesson 13: Context Is the New Compute

The Beaver took the Crow back to the river.

He had described it to her a long time ago, in the first week of all this, when he was explaining why the Watering Hole could not be governed the way a dam is governed. I can show you the dam, he had said. I can only show you the river, and the river does not consult me.

What he wanted to show her now was what the river was carrying.

He picked a single ordinary request, one of thousands that day, and laid out everything that had travelled with it. The question the animal had actually asked, which was two lines long. Then the instructions, which had grown over eighteen months and now ran to several pages. Then twenty retrieved passages, because twenty had felt safer than ten when somebody configured it. Then the entire preceding conversation, resent in full, because that was the default.

The question was a rounding error in its own request.

“How much of this,” said the Crow slowly, “do we pay for?”

“All of it.”

“Every time?”

“Every time.”

The Crow, who thinks in things she can count and had spent two acts being told this subject was too technical for her, understood this immediately and completely, and became its most effective advocate in the jungle within about a week. Sometimes the cost lever that changes an organization is the one a finance person can see with their own eyes.

The input is the bigger number

In most mature enterprise workloads, the input costs more than the output.

This surprises people, because the mental model is that you are buying answers. You are buying answers, and you are also paying to ask, and the asking has quietly grown while nobody was looking at it. A two line question wrapped in six pages of context is a six page transaction with a two line payload.

Which means an optimization program focused entirely on the response side is working on the smaller half of the problem. Shortening outputs, choosing cheaper models, tuning generation. All useful, all downstream of a bigger number nobody is watching.

Four sources of bloat

Instruction accretion. System prompts grow every time something goes wrong and never shrink. This is the big one and it gets its own section below.

Retrieval over-fetch. Somebody set the retrieval depth during development, chose a number that felt safe rather than a number that was measured, and it has been fetching that many passages on every request ever since. The difference between ten and twenty is invisible in quality and completely visible on the invoice.

Conversation accumulation. Full history resent at every turn in workflows that do not need it. Some workflows genuinely do. Many are single-shot tasks carrying a conversation transcript for no reason other than that the framework does it by default.

Defensive stuffing. The entire document included because working out which section was relevant was harder than not working it out. Entirely rational for the person who did it. Permanent for the organization.

The immortal instruction

Instruction accretion deserves its own section because of how it happens, which is that it happens for good reasons every single time.

Something goes wrong. A model produces an output somebody objects to. There is an incident, or a complaint, or an uncomfortable meeting. The fastest available fix is to add a line to the instructions telling it not to do that again. It works. Everyone moves on.

That line is now sent on every request, forever. It costs a small amount, every time, in perpetuity. Repeat this thirty times over two years and the instructions are several pages long, most of which is scar tissue from incidents nobody remembers.

Now try removing one. You cannot know which lines are load-bearing without testing, and testing costs effort, and the downside of removing the wrong one is another incident with your name on it. So nobody removes anything. There is no organizational mechanism for deletion here and no career incentive to build one.

The Fox tried this in the following quarter. Of the lines his team could not immediately justify, a substantial portion turned out to make no measurable difference to output quality at all. Some of them had been added to solve problems in a version of the workflow that no longer existed.

Caching, and why nobody uses it properly

The highest-leverage lever in this post, and one of the most under-used things in enterprise AI generally.

If the front portion of your request is identical across many requests, it does not need to be processed from scratch every time. The saving on a workload with large stable instructions and small variable questions is substantial.

The reason it goes unused is almost never that people do not know it exists. It is that it requires the invariant content to sit at the front and the variable content at the back, which requires somebody to have thought about prompt structure as an engineering concern with a cost consequence, rather than as writing.

Most prompts are assembled in the order they were thought of. The instructions grew, the context got appended, the question went wherever felt natural. Nobody was wrong. Nobody was thinking about the ordering, because at pilot scale the ordering did not matter and by production scale nobody was looking.

Restructuring for cache hits is days of work with a permanent return. It is close to the best return per unit of effort available in this entire series.

Retrieval precision is a cost lever

Worth stating explicitly because it is one of the rare places where the cheap option and the good option are the same option.

Better retrieval means fewer passages needed to answer well. Fewer passages means less context. Less context means lower cost and, frequently, better output, because a model given ten relevant passages generally does better than one given twenty of which half are noise.

So retrieval tuning pays twice. Almost every organization treats it purely as a quality activity, funds it from a quality budget, and never connects it to the cost line it is directly driving. Making that connection visible is often what gets the work prioritized.

Context efficiency

The measurement I would introduce, imperfectly, rather than not at all.

What proportion of the context you supply actually influences the output? You cannot measure this precisely. You can approximate it, by ablation, by sampling, by simply checking whether retrieved passages are ever referenced in the answer.

Even a rough number changes conversations, because it converts an invisible cost into a visible ratio. A capability sending six pages of context to produce a two line answer with three relevant passages has a number attached to it now, and numbers get optimized in a way that vague concerns do not.

Why this compounds

One forward-looking point, because it is the reason this lesson sits where it does in the series.

In a single-call workflow, context bloat is additive. You pay for it once per request. Annoying, fixable, bounded.

In an agent workflow, context is re-transmitted at every step of the loop. A single lazy prompt is now being paid for five, ten or forty times per task, depending on how many steps the system decides it needs. The bloat multiplies rather than adds.

Which means every piece of context hygiene you do now becomes considerably more valuable the moment anything in your estate becomes autonomous. That is the next lesson and it is the one where all of this compounds badly.

Three ways this goes wrong

The immortal instruction. A line added during an incident two years ago, sent on every request since, purpose unknown, removal feared, cost permanent.

Cache-hostile structure. Variable content at the front, destroying prefix caching entirely, usually because nobody knew the ordering had a price.

Fetch and forget retrieval. A depth configured on day one, never tuned, treated as a quality setting by a team that has never been told it is also a cost setting.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, ask for the input to output ratio on your largest capability. One number. If input dominates heavily, there is real money in this post and you now have the evidence to ask for it.

If you sit in the Crocodile’s chair, audit the system prompts for lines nobody can justify, and restructure for cache hits. Both are days rather than quarters, and both are permanent.

If you sit in the Mandrill’s chair, stop demanding a prompt addition after every isolated error. Each one is permanent and each one is charged on every request forever. Ask for the fix that has a cost of zero on the other ninety nine thousand requests.

For everyone: give prompt and retrieval configuration a named owner and a review cycle. Unowned, it only ever grows, because growth is what every individual incident rationally produces.

Jungle Lesson 13

You are billed for the question as well as the answer, and in most enterprise workloads the question is the bigger number. Every line added to a prompt after an incident is a permanent tax paid on every request forever, and nobody has ever been thanked for removing one.

Next time: the Tortoise has been saying the same three sentences for four lessons and everyone has been treating them as an obstacle to route around. In this one the room finally understands that she has been describing a cost driver the whole time, and the entire architecture conversation reorganizes itself around her. Lesson 14 is about data gravity, latency floors, and control of the inference plane.

If you want a quick sense of whether this post applies to you, go and read your longest system prompt end to end. Most people have never done this. It is usually a short and instructive experience.

Scaling AI FinOps | Lesson 12: The Routing Layer

The Beaver built the routing layer over a weekend, because he had noticed something on the Friday and could not leave it alone, which is the defining characteristic of the animal.

What he had noticed was that a substantial proportion of the requests coming through the Watering Hole were not questions at all. They were classification. Extraction. Reformatting. Deciding which of four categories a thing belonged to. Turning one shape of text into another shape of text.

All of it was going to the most capable thing available, because that was what had been wired in during the pilot, eighteen months earlier, by somebody who was trying to find out whether the idea worked at all and quite reasonably did not want the model to be the reason it failed.

Nobody had revisited it. Nobody revisits a working configuration.

By Monday, requests were being sorted by what they actually required. The bill fell substantially over the following weeks.

Nobody noticed for a month. Not one animal in the jungle reported a difference in output quality, because there was no difference in output quality, because none of the redirected work had needed the expensive path in the first place.

The Fox, when he found out, was pleased and then quietly irritated, and was honest enough to say so out loud. The largest single cost improvement of the quarter had come from plumbing, over a weekend, from a Beaver who had not been asked. It had not come from strategy, or governance, or any of the framework work that had consumed the previous two acts.

“Both things are true,” said the Crocodile, one eye open. “You needed the framework to know the plumbing mattered.”

The observation underneath it

Most enterprise AI traffic does not need the most capable model available. Not some of it. Most of it.

Once a capability is in production and serving real workflows, the traffic mix shifts heavily toward the mundane. Routing a request. Extracting fields. Classifying an input. Summarizing something short. Formatting an output. The genuinely hard reasoning that justified choosing a frontier model in the first place turns out to be a minority of the volume, sometimes a small one.

And every one of those mundane requests is being charged at the rate of the hardest thing your system does, because a default was set during a pilot and defaults are permanent.

This is the most fixable line on most AI bills and it is fixable in weeks rather than quarters. It is also the least glamorous thing in this entire series, which is precisely why it survives for eighteen months in most organizations.

Three routing patterns

Static tiering. The task type determines the model. Classification goes here, extraction goes there, open-ended reasoning goes to the expensive path. Simple, predictable, easy to reason about, easy to explain to a finance function, and it captures the large majority of the available saving. If you do nothing else from this post, do this.

Cascade. Try the cheap tier first. Escalate to the expensive one if some confidence or validation signal says the cheap answer was not good enough. Higher ceiling than static tiering and considerably more complexity, and it lives or dies on one number that I will come to.

Semantic routing. A classifier looks at the request and decides where it goes. Powerful, elegant, and the easiest of the three to over-engineer into something that costs more to operate than it saves. I would reach for this third rather than first, and only once static tiering has told you where the volume actually is.

The cascade math people get wrong

A cascade only saves money if the cheap tier resolves a high enough proportion of requests. Below that proportion, you are paying for two calls instead of one on the escalated portion, plus the added latency, and you have made things worse while feeling sophisticated.

The break-even depends on the price ratio between your tiers. The wider the gap between cheap and expensive, the lower the resolution rate you need to make it worthwhile. With a large ratio, a cheap tier that handles half the traffic is comfortably worth having. With a narrow ratio, you may need it to handle the great majority before the arrangement pays for itself at all.

Work out your own threshold from your own price ratio, then measure your actual escalation rate against it, and put both numbers on the same page.

I would guess that a meaningful share of deployed cascades in production today are sitting below their break-even and nobody has checked, because the cascade was built as an optimization and optimizations do not get audited. They get built, celebrated, and then assumed to be working forever.

Sufficiency, not maximization

The mental shift this requires is the hard part, and it is a business shift rather than a technical one.

The question is not which model is best. That question has a clear answer and it is the wrong question. The question is which model is sufficient for this task, at an acceptable failure rate, given what happens when it fails.

Sufficiency is defined by consequence. A misclassified internal document that a person reviews anyway has a low cost of error, so a high failure rate is tolerable and the cheap tier is fine. A misclassified input feeding an automated decision that reaches a customer has a high cost of error, and the expensive path is correct regardless of unit price.

That is a business judgment. Engineering cannot make it, and when engineering is forced to make it by default, it will reasonably choose the safest option every time, which is how you end up where the jungle was.

You cannot route without evaluation

The dependency that stops most organizations doing any of this.

You cannot route by sufficiency unless you can measure sufficiency. Without an evaluation harness that tells you how each tier performs on each task type, routing is guessing with extra infrastructure, and the first time somebody complains about quality the whole thing gets reverted.

This is the evaluation layer from Lesson 3, the one that appears on nobody’s budget. It turns out to be the precondition for the largest cost lever available. Which is a fairly typical shape in this discipline: the unglamorous investment nobody wanted to fund is what unlocks the saving everybody wants.

Why this belongs at the gateway

Routing must live at a shared gateway rather than inside each application, and this is the architectural decision with the longest tail in the whole series.

If each team makes its own model choice inside its own code, then every choice becomes a hard dependency. Changing tier later means touching forty codebases owned by forty teams with forty roadmaps, which means you will not change tier later, which means every routing decision made this year is permanent.

Put it at the gateway and the tier becomes configuration. You can change it centrally, test it, roll it back, and adjust it every quarter as prices and capabilities move, which they will.

The gateway is also where the tagging from Lesson 1 lives, where the allocation from Lesson 7 becomes possible, and where the guardrails in Lesson 19 will need to sit. It is the single highest-leverage piece of architecture in an enterprise AI estate and it is almost always built late, after the pain, by someone doing a migration.

Three ways this goes wrong

Top tier by default. The pilot configuration, still in production, quietly handling string formatting at premium rates for two years. This is the jungle’s situation and I would expect it to be more common than not.

The negative cascade. Escalation rate below break-even, so the clever thing costs more than the naive thing, and nobody has calculated the threshold to find out.

Routing in the application. Every team choosing independently, every choice becoming permanent, and a tier change turning into a migration project that never gets prioritized.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, ask two numbers. What share of requests go to your most expensive tier, and what share genuinely require it. The gap between those is money, and it is usually available within a quarter.

If you sit in the Crocodile’s chair, get routing behind a shared gateway before you have forty callers. Retrofitting this is the most avoidable expensive project in the discipline and the window to avoid it closes quietly.

If you sit in the Mandrill’s chair, define the quality floor for your use case, expressed as an acceptable failure rate and what happens when it fails. Sufficiency is a business judgment and if you do not make it, somebody will default to the most expensive option on your behalf, forever.

For everyone: if you run a cascade, calculate its break-even and measure your escalation rate against it. Today. It is a ten minute calculation and there is a real chance it tells you something unwelcome.

Jungle Lesson 12

Most of what you send to the cleverest animal in the jungle does not need the cleverest animal in the jungle. Define what sufficient looks like for each task and buy down to it, because paying for judgment you are not using is the most common and most fixable line on the bill.

Next time: the Beaver takes the Crow back to the river he mentioned all the way back in Lesson 1 and shows her what it is actually carrying. It turns out you are billed for the question as well as the answer, that in most enterprise workloads the question is the larger number, and that every line somebody added to a prompt after an incident two years ago is still being paid for on every single request. Lesson 13 is about context.

The Beaver in your organization has probably already noticed the thing in this post. Whether anyone has asked him is a different question, and it is usually the answer to why it has not been fixed.

Scaling AI FinOps | Lesson 11: Frontier or Local

The Peacock arrived for the second planning cycle with a display of quite exceptional magnificence.

There were slides. There was a total cost of ownership model with a five year horizon and a curve on it that started high and ended gratifyingly low. There was a comparison against what the jungle was currently paying, and the comparison was not close. Somebody in the room said the words game changer without any apparent irony.

The Fox, who had learned a great deal over the preceding two acts, asked a reasonable question about the assumptions.

The Crocodile, who had sat through this presentation four times in his career under four different names and two different technology paradigms, opened one eye and asked the only question that actually decides it.

“What utilization is that curve assuming?”

There was a pause of a length that told everyone in the room what the answer was going to be before the Peacock said it.

Peacocks have enormous displays and modest payloads. This is not dishonesty, it is what the animal is for. The tail is the product. Somebody in the room has to be the one who asks about the bird underneath it, and in most organizations there is no such animal, which is why these decisions go wrong.

Two shapes, not two products

The frontier versus local argument gets conducted as though it were a choice between two products with different features. It is not. It is a choice between two cost shapes, and almost everything else people argue about is downstream of that.

API-served inference is pure variable cost. Near zero fixed. You pay for what you use, forever, and the line is straight. It never gets cheaper through your own effort, only through the provider’s price changes, and it never costs you anything when you are not using it.

Self-hosted inference is high fixed cost with low marginal cost. You pay a large amount whether or not anybody uses it, and then very little per unit on top. The unit economics are terrible at low volume and excellent at high volume, and the entire question is where those two lines cross.

That is the whole thing. Everything else is a variation on it. And it means the honest form of the question is not which is better, it is what volume gets you past the crossover and how confident you are that you will be there.

The utilization trap

Here is where most self-hosted business cases fall over, and it is always the same place.

Fixed-cost economics are quoted at high utilization, because that is the only way the numbers look good, and that is a perfectly reasonable way to describe theoretical capability. They are then delivered at whatever utilization your actual workload produces, which is a completely different number.

Enterprise inference demand is not smooth. It follows the working day, which means it is close to nothing for a substantial portion of every twenty four hours. It follows the working week. It follows the business cycle, the month end, the campaign calendar. It has peaks that determine how much capacity you must provision, and troughs that determine how much of that capacity earns nothing.

You provision for the peak. You are billed for the peak. You utilize somewhere considerably below the average, once you account for the redundancy you need in order to survive the peak going wrong.

An honest utilization assumption for a first-generation enterprise deployment is dramatically lower than the one in the presentation, and the gap between those two numbers is usually the entire business case.

What is missing from the fixed side

The second failure is that the fixed side of the model is systematically incomplete. Four things that belong there and usually are not.

  • The people. Self-hosting is an operational commitment, not a purchase. Somebody keeps it running, patches it, handles the incidents, manages capacity. That is a standing team, and in most models it appears as a footnote or not at all.
  • Redundancy. A single deployment is a single point of failure for something you have just made business critical. The resilient version costs meaningfully more than the demonstration version.
  • Evaluation and update cycles. Every model change you make yourself requires you to re-verify behavior across everything that depends on it. On the API side somebody else absorbs a version of this cost. On your side it is yours, and it recurs.
  • Falling behind. This one has no line item and it is real. Capability moves. A fixed deployment holds still. The gap between what you have and what is available compounds, and at some point it becomes a cost in the form of work your organization cannot do.

The three questions

What I would ask, in this order, of any proposal in either direction.

What is the break-even volume? A specific number of units per period at which the two curves cross, with the fully loaded fixed side included. If nobody can produce this number, the analysis has not been done, whatever else has been produced.

How confident are you in the volume forecast? Not what is the forecast. How confident. Because the forecast is doing all the work and it is usually the least examined input in the model. Ask when it was made, by whom, and what it is based on.

What does this look like at half that volume? This is the question that decides it. Almost every self-hosting case survives the first two questions and fails the third, because the fixed side does not move and the unit economics collapse. If the answer at half volume is still acceptable, you have a genuinely robust case. If it is catastrophic, you are making a bet on a forecast rather than an architecture decision.

What optionality is worth

The thing that is almost never priced, and which I think matters more than anything else on this list.

Variable cost buys you the right to change your mind. Fixed cost buys you a lower unit price and a commitment to a capability level that will be surpassed, on a timeline you do not control, while you are still depreciating the thing.

In a slow-moving field that trade is usually worth taking, which is why organizations have made it comfortably for decades with every other kind of infrastructure. In a field where capability and price are both moving quickly, the option to move has genuine value, and the discount you are being offered has to exceed it.

The framing I would use in the room: what would you pay, today, for the ability to shift this entire workload somewhere else in six months with no stranded cost? Whatever that number is, it is the hurdle the discount has to clear. Nobody ever calculates it, and it is frequently larger than the saving.

The answer is usually both

For most enterprises the correct answer is not one or the other, and treating it as a single organizational decision is how it goes wrong.

Work that is high volume, narrow, well specified, stable in its requirements, latency sensitive, or subject to a constraint on where it can run, suits a fixed-cost local deployment. The volume is there, the capability requirement is not moving, and the crossover is genuinely favorable.

Work that is low volume, broad, hard, exploratory, or benefits from the newest capability suits variable-cost API-served inference. You are buying capability and optionality rather than unit price, and both are worth the premium.

The organizations that get this right decide per workload and build the routing layer that makes the decision reversible. That routing layer is the next lesson, and it is the mechanism that turns this from a one-time architectural bet into something you can adjust quarterly.

Three ways this goes wrong

The full utilization business case. Economics quoted at a utilization the workload will never reach, delivered at half of it, defended for two years by people who cannot admit the input was wrong.

The forgotten team. Hardware and capacity modeled carefully, the standing operational function required to run it modeled not at all. This single omission has sunk more self-hosting cases than every other factor combined.

Ideological procurement. The architecture chosen first, for reasons of principle or preference, and the model constructed backwards to support it. This happens in both directions and it is equally expensive either way.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, ask the three questions in order and do not accept a break-even volume without a stated confidence in the forecast underneath it. Then ask for the version at half volume before you approve anything.

If you sit in the Crocodile’s chair, model the fully loaded fixed side including people, redundancy, evaluation cycles and refresh. You are the only animal in the building who can produce that number honestly, and it is the number that decides it.

If you sit in the Mandrill’s chair, be honest with yourself about whether your volume forecast is a forecast or an aspiration. The architecture decision is far more expensive to reverse than the forecast is to correct.

For everyone: decide per workload, never once for the organization. A single global answer to this question is always wrong somewhere, and it is usually wrong somewhere expensive.

Jungle Lesson 11

This is not a choice between two products, it is a choice between two cost curves, and the only questions are where they cross and how much you trust the volume that gets you there. Variable cost is the price you pay for the right to change your mind, and in a field moving this fast that right is worth more than most business cases give it credit for.

Next time: the Beaver builds something over a weekend that takes a large bite out of the bill, and nobody notices for a month because the outputs did not change at all. The Fox is mildly annoyed that the biggest win of the quarter came from plumbing rather than strategy. Lesson 12 is about the routing layer, and why the cheapest sufficient answer beats the smartest one.

If a proposal in either direction cannot tell you what it looks like at half the assumed volume, it is not an architecture decision. It is a forecast with an architecture attached, and forecasts are much cheaper to be wrong about.

Scaling AI FinOps | Lesson 10: The Value Ledger

The Tortoise had been reading the Fox’s revised benefits case for some time before she said anything, which is normal for a tortoise and unnerving for everyone else.

“You know what this is,” she said eventually.

The Fox said that it was a benefits case.

“It is an audit trail,” said the Tortoise. “You have written down what you claimed, how you intend to prove it, who is accountable, and when it lands. That is the document I have been asking this jungle for since before any of you had heard the word inference. You have built it by accident while trying to survive a finance meeting.”

The Fox, who had spent a considerable portion of the previous two seasons regarding the Tortoise as an obstacle, took a moment with that.

“Are we on the same side?” he asked.

“We have been the entire time,” said the Tortoise. “You were busy.”

This is the alliance that ends up mattering most in the rest of this story, and I want to flag it now because it takes most organizations far too long to discover. The compliance function and the AI function want the same artifact for different reasons. One of them wants to be able to defend a decision to a regulator. The other wants to be able to defend a decision to a CFO. It is the same document.

What a value ledger actually is

A standing record, one row per claimed benefit, carrying at minimum:

  • What was claimed, in a unit somebody can check
  • Which conversion path it uses, from Lesson 8
  • Who owns realizing it, by name rather than by function
  • How strong the evidence is, from Lesson 9’s ladder
  • When it is expected to land
  • What actually landed, filled in afterwards, next to the claim rather than replacing it

That is the whole instrument. It is not sophisticated. It could live in a spreadsheet and in most organizations it should, at least for the first year, because the discipline is the hard part and the tooling is not.

Append only, and why that is the entire point

One rule matters more than all the others combined. Entries are never edited. What was claimed stays visible next to what happened.

The temptation to restate is enormous and it always arrives dressed as accuracy. We understand the process better now. The original definition was flawed. The scope changed. Every one of these is often true, and every one of them, acted upon, destroys the only thing the ledger was for.

Because the value of the instrument is not the record of benefits. It is the record of the gap between what people predicted and what occurred, accumulated over enough cycles that the organization learns the shape of its own optimism. That is genuinely valuable. A team that has systematically overclaimed by a factor of three for six quarters is a team whose next claim you can adjust with confidence, and that adjustment is worth more than any estimation methodology you could buy.

Edit the entries and you have a document that shows every claim was roughly correct, which teaches nothing and predicts nothing. If something needs restating, add a new row with a new date and leave the old one where it is.

Benefits decay

The second discipline, and the one that catches organizations in year two, which is where Act V of this series is heading.

Most AI benefits are not annuities. They erode. The process changes and the saving no longer applies to a process that exists. Volumes shift. The comparison process itself improves for unrelated reasons, so the delta narrows. Staff turn over and the new people never did it the old way, so the improvement is invisible to them and stops being defended.

Booking a benefit once and carrying it forward indefinitely is the single most common overstatement in this discipline, and it is almost never deliberate. It happens because there is no field in the document that says when this stops being true.

So add one. Every entry carries an explicit decay assumption and a review date. Flat for three years, or declining after eighteen months, or one-off. You will be wrong about the assumption. Being wrong about a documented assumption is a vastly better position than being silently wrong about an undocumented one, because the first one gets corrected on a date and the second one gets discovered in an audit.

Evidence grading

Every entry carries a grade, taken directly from Lesson 9’s ladder. Holdout, staggered rollout, adjusted pre and post, cohort comparison, self-report.

The purpose is not to disqualify weak evidence. Weak evidence is normal and often it is all you can get. The purpose is proportionality. A claim supported by a survey is still a claim, it simply does not get to fund a decision that requires a claim supported by a holdout.

This does something subtle and useful to organizational behavior. Once the grade is visible next to the claim, teams start volunteering to improve their evidence, because a stronger grade means their number carries more weight in the reallocation conversation. You have made rigor competitively advantageous rather than a compliance burden, which is the only way it ever actually happens.

The reconciliation habit

Once a cycle, quarterly for most organizations, put claimed next to realized and publish the variance.

The first time you do this it will be uncomfortable. The second time it will be uncomfortable. Somewhere around the third or fourth cycle something changes, and the document becomes the most trusted artifact the program produces, because it is the only one that has ever voluntarily reported its own misses.

I have watched this transformation happen and it is worth the two bad quarters. A program that publishes its own variance gets believed about everything else. A program that only ever reports successes gets discounted on everything, including the successes, and the discount is applied by people who will never tell you they are applying it.

The Crow’s observation, when the first reconciliation came in and the realized number was about a third of the claimed one: “This is the first document anyone has given me about this subject that I would repeat to the board without checking it first.”

This is governance, not reporting

The ledger is what turns AI FinOps from a reporting function into a governance one, and this is the thing I would most want a leadership team to understand from Act II.

Reporting tells you what happened. Governance changes what happens next. A ledger with owners, dates, decay assumptions and published variance does the second thing, because it creates a moment where somebody has to account for a prediction they made, in front of people who can see the original.

It is also, as the Tortoise spotted immediately, exactly what external scrutiny asks for first. Not your architecture. Not your model choices. What did you claim, how did you verify it, who was accountable, and what happened. Every serious review of an AI program I have seen has converged on that set of questions, and the organizations that had a ledger answered them in an afternoon.

And it is what makes the chargeback conversation from Lesson 7 survivable, because a charge without a ledger is a bill, and a charge with a ledger is a price for something whose value has been demonstrated.

Three ways this goes wrong

The retroactive edit. Last year’s claim quietly aligned to this year’s outcome, for excellent reasons, destroying the instrument entirely while making it look tidier.

Perpetual benefits. Booked once, assumed forever, never reviewed, still sitting in the cumulative total four years later describing a process that was replaced twice.

The ledger nobody reads. Maintained diligently by someone conscientious, reviewed by no body with authority to act, quietly reclassified as a compliance artifact. This is the most common failure and it is entirely a governance design problem rather than a documentation one.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, own the ledger and make append-only a written policy rather than an aspiration. This is a finance instrument. If it lives in the technology function it will be maintained beautifully and read by nobody.

If you sit in the Crocodile’s chair, feed it automatically wherever the data allows. Manually maintained ledgers decay within two quarters regardless of how committed everybody was at the start, and the decay is always discovered at the worst moment.

If you sit in the Mandrill’s chair, put your name against your realization dates. The accountability is the point and it is also the fastest route to being trusted with a larger budget, because you will be one of very few people who can point at a prediction they made and hit.

For everyone: publish the claimed versus realized variance. It is the single most credibility-generating document available to an AI program and almost nobody produces it.

Jungle Lesson 10

Write down what you promised, who owns it, and when it lands, then never edit the entry. A benefits case that cannot be checked against what actually happened is not a case, it is a mood, and moods do not survive the second budget cycle.

That closes the second act. The jungle can now count. It has a denominator it chose rather than inherited, an allocation rule it published before it published a number, an honest view of what productivity gains actually convert into money, a way of proving things that does not rely on memory, and a ledger that records what was promised next to what arrived.

None of which has yet made anything cheaper.

That is the third act, and it moves the argument somewhere most finance functions do not expect it to go, which is into the architecture. Almost every meaningful lever on AI unit economics is an engineering decision made months before anyone looks at a bill. Where inference runs, which model handles which task, how much context gets sent, where the data sits, and how much autonomy the system has. Act III opens with the one everybody wants to argue about, which is whether to run it yourself. Lesson 11 does the crossover math honestly.

If your organization has a compliance officer who has been asking for something like this for two years, the fastest route to a value ledger is to go and find them. They have thought about it more carefully than you have and they have been waiting for somebody to want it.

Scaling AI FinOps | Lesson 9: Baselines, Counterfactuals and Holdouts

After the meeting in the last lesson, the jungle needed evidence, and discovered it did not have any.

Not because nobody had measured anything. Everybody had measured a great deal. The problem was that all of it had been measured after the capability went live, which meant every comparison was against a version of the old process that existed only in the memories of the animals who had stopped doing it.

It was at this point that the Owl said something that changed the quarter.

“I still have the north territory,” he said.

Everyone looked at him.

What had happened, it emerged, was this. When the rollout was designed, the Owl had produced a phased plan, in the way that Owls do, with the north territory scheduled last for reasons involving a system migration. The migration had been delayed. Nobody had revisited the plan, because nobody revisits a rollout plan once the rollout is going well. So the north territory had spent two full seasons doing the old process, unchanged, while everybody else changed, and their operational data had been collecting the whole time in the ordinary way.

He had, entirely by accident and with no intent whatsoever, been running a control group for eight months.

“Did you know?” asked the Crow.

“I knew they hadn’t gone live,” said the Owl carefully. “I did not know that was interesting.”

Owls have very large eyes. On this one occasion, something had actually been in front of them.

The remembered baseline

Almost every organization begins measuring after deployment. This is not laziness. It is that before deployment nobody yet cares, and by the time somebody cares, the thing they wanted to compare against has been replaced.

So the comparison becomes a comparison against memory, and memory has a consistent bias in this specific situation. People remember the old process as slower and more painful than it was, because what they remember most vividly are the bad instances. Ask a team how long something used to take and you will reliably get a number closer to the worst case than the median.

Which produces benefit cases that are inflated in a direction nobody chose, built by people acting in good faith, and completely undefendable the moment anyone asks where the baseline came from.

The baseline is the whole game. Everything else in this post is technique for when you did not capture one, which is most of the time.

The methods ladder

Four approaches, strongest first. Use the highest rung you can reach.

Randomized holdout. A comparable population that does not get access, chosen at random, for a defined period. This is the cleanest thing available and it is politically the hardest, which is why it is rare. It is also possible for far longer than most organizations assume, because full deployment usually takes longer than anyone plans anyway.

Staggered rollout. Territories go live at different times, and the not-yet-live ones serve as the comparison. This is nearly free, because you are almost certainly already doing it, and the great majority of organizations throw the resulting evidence away by not capturing it. The Owl’s accidental control group was this, discovered eight months late.

Pre and post with controls. Before and after, adjusted for the confounders you can identify. Seasonality, volume changes, staffing changes, process changes shipped at the same time. Weaker than the first two but genuinely usable if you are honest about the adjustments.

Cohort comparison. Heavy users against light users. This is the weakest rung and the most commonly used, because it requires nothing but the data you already have. The problem is that self-selection is doing most of the work. The animals who adopted heavily are not a random sample. They are the ones who were already better at this, or more motivated, or had the kind of work that suited it, and you are measuring that rather than the capability.

The confounders that bite

Four that show up specifically in this domain and that will eat an otherwise sound analysis.

  • The process change that shipped alongside. Almost nobody deploys a capability without also simplifying the workflow around it. That simplification often produces more of the benefit than the capability does, and the two are attributed together.
  • Novelty. New things get attention and effort. Some of the early improvement is people trying harder because something is new, and it goes away.
  • Observation. Teams that know they are being measured perform differently. This is old news and it applies here as much as anywhere.
  • Adoption order. Your best people adopt first, reliably. Early results are therefore measured on your strongest performers and will not replicate across the rest.

Making a holdout survivable

The objection to holdouts is always the same and it is not unreasonable. You are denying a useful tool to a group of colleagues in order to generate a chart.

The framing that works, and I have seen it work several times, is to stop describing it as denial. The holdout group is not being deprived. They are the reason the program can prove it works, which is the reason it will still be funded next year, which is the reason everyone eventually gets it. That is true and people respond to it being said out loud.

Three things make it stick. Timebox it, with a date. Guarantee access at the end, in writing. And name the group publicly as the reason the evidence exists, in the same presentation where the results appear. The Mandrill volunteered one of his teams for this in the following quarter, and it did him considerably more good politically than it cost him operationally.

When to measure

Three points, and you need all three.

Week two, which is your peak and which you should record while knowing it is a peak. Month three, which is where the novelty has gone and the real shape starts to appear. Month twelve, which is the number that should drive any permanent funding decision.

The numbers will fall between these points and that is not failure, it is the reference point moving as the improved process becomes normal. What matters is that the organization sees all three, because an organization that only ever sees week two will make every decision on a peak, and an organization that only sees month twelve will conclude the whole thing underdelivered without ever knowing what the early state looked like.

Put the month twelve measurement in the calendar now, while everybody is still enthusiastic enough to agree to it. Nobody schedules that meeting later.

How much rigor is enough

A brief word against overdoing this, since I have just spent a thousand words arguing for evidence.

Match the strength of your evidence to the stakes of the decision. A modest capability serving one team does not need a randomized holdout. Pre and post with a sensible adjustment is fine, and demanding more is a way of making sure nothing gets measured at all because the bar is exhausting.

A decision about whether to deploy something across the organization permanently is a different matter. There the cost of a holdout is trivial against the cost of being wrong, and the fact that it is inconvenient is not an argument.

The failure mode I see most is uniform rigor. Either everything gets the full treatment, which means nothing does, or nothing does, which means everything is anecdote. Tier it.

Three ways this goes wrong

The remembered baseline. Comparing against how long people think it used to take. Ubiquitous, invisible, and fatal the moment anyone asks.

Rollout without capture. Running a staggered deployment, generating a natural experiment for free, and recording none of it. This is the most wasteful thing in this post because the evidence was already being produced.

Measuring once, at the peak. Funding a permanent capability on a week two number, then spending year two explaining why the benefits shrank when nothing shrank except the novelty.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, ask when the baseline was captured and by whom. If the answer is after go-live, discount the case accordingly and say so out loud, so that the next person to present to you captures one first.

If you sit in the Crocodile’s chair, capture the staggered rollout data. You are already generating it. Recording it costs close to nothing and it is worth more than any dashboard you could build this year.

If you sit in the Mandrill’s chair, volunteer a team as a holdout and get named publicly as the reason the program can prove anything. It is a much better position than it appears from the outside.

For everyone: put the month twelve measurement in the calendar today. It is the only one of the three that nobody will schedule voluntarily and the only one that should drive a permanent decision.

Jungle Lesson 9

You cannot measure a change you did not measure before. Capture the baseline while you are still bored by the idea, because by the time you need it, everyone will remember the old process as being much worse than it was, and you will have no way to prove otherwise.

Next time: the Tortoise looks at everything the Fox has built over the last four lessons and points out, with some satisfaction, that he has accidentally constructed the audit trail she has been asking for since before any of this started. Compliance and the AI lead end up on the same side of the table, which surprises both of them. Lesson 10 closes the second act with the value ledger.

Somewhere in your organization there is probably an Owl sitting on a delayed rollout that nobody has revisited. That is not a project management failure this month. It is the only clean comparison you are going to get.

Scaling AI FinOps | Lesson 8: The Hours Saved Fallacy

The number on the Fox’s slide was very large, and he had checked it three times.

Six hundred and eighty animals were using the capability. Survey responses indicated an average of twenty two minutes saved per animal per day. Multiply that out across a working year, apply a loaded hourly rate, and you arrive at a figure with a great many digits in it, sitting in a box, in bold, next to a modest and considerably smaller number representing what the whole thing cost.

It was, by the standards of these presentations, a good one. The arithmetic was correct. The survey was real. The adoption numbers were not inflated. I have seen a hundred versions of this slide and this was among the more honest ones.

The Crow looked at it for a while.

She did not challenge the twenty two minutes. She did not question the adoption figure or ask about the survey methodology, which is what the Fox had prepared for and which would have been a conversation he could win.

“Where did the time go?” she asked.

The Fox said that it went back to the animals.

“Yes,” said the Crow. “And then what?”

The room was quiet for long enough that everybody in it understood what had just happened, including the Fox, who to his considerable credit did not attempt to fill the silence with anything.

The Hyena, for once, was not laughing. “That’s the whole thing, isn’t it,” she said. “You’ve proved they have more time. You haven’t proved you have anything.”

Why twenty minutes is not twenty minutes

Here is the fallacy, stated plainly, because it is the single most successful piece of fiction in modern enterprise reporting and it is almost never named directly.

Twenty two minutes saved by each of six hundred and eighty animals is not roughly two hundred and fifty person days of recovered capacity. It is six hundred and eighty animals with slightly better afternoons.

Those are not the same thing and they are not close to the same thing. The first is a resource the organization can deploy. The second is a genuinely nice outcome that produces no line in any ledger, ever, under any circumstances.

Time saved becomes money only when it is aggregated into something the organization can either redeploy or avoid paying for. Twenty two minutes, distributed across six hundred and eighty people, in fragments, throughout a day, aggregates into nothing at all. It is real. Every one of those animals genuinely got the time back. It simply does not add up into a unit anybody can spend.

The three conversion paths

There are exactly three ways time turns into money, and I would rank them by how well they survive scrutiny.

Cost avoidance. A role you did not need to backfill. A contractor whose renewal you did not sign. Overtime you did not pay. Seasonal capacity you did not bring in. This is the strongest by a wide margin, because it is verifiable in the ledger by someone who was not involved in the project. If you can point at a requisition that was cancelled, you have a benefit, and nobody can argue with you.

Capacity redeployment. The same people producing measurably more of something the organization values. Not more availability. More output. This is real and it is the most common genuine benefit, but it requires an output metric rather than an input one. “The team has more time for strategic work” is not this. “The team closed forty percent more cases” is.

Quality and cycle time. Faster resolution, fewer errors, better retention, higher conversion. Frequently the largest of the three in absolute terms, and the hardest to trace, because it needs a chain of evidence running from the capability all the way to something financial. Worth doing. Not worth pretending is easy.

And then the part people find uncomfortable. Anything that does not land in one of those three is not a benefit. It is a nice thing.

Nice things matter. Less frustrating work, less time on tasks people hate, better morale, lower attrition risk. I am not dismissing any of it and some of it is genuinely valuable. But it should be reported as what it is, in its own section, under its own heading, rather than converted into currency by multiplication and placed next to the cost line as though the two numbers were the same kind of object.

The fragmentation threshold

Underneath all of this sits a threshold effect that I think deserves more attention than it gets.

Below some proportion of a role’s time, savings do not aggregate at all. They dissipate. The person absorbs the time into the general texture of their day and nothing further happens, because there is no mechanism by which fifteen scattered minutes become a deployable resource.

Above some proportion, savings become visible enough that the work can be restructured, and restructuring is what actually converts time into capacity or cost. A role that gets thirty percent of its time back can be redesigned. A role that gets four percent back cannot.

Where exactly that threshold sits depends on the work, and I would be suspicious of anyone quoting you a universal number. But the existence of the threshold is the important part, because it explains something that otherwise looks like a paradox: why broad shallow deployments produce enormous claimed benefits and almost no realized ones, while narrow deep deployments produce modest claimed benefits that actually show up.

If your deployment strategy is a small saving for a very large number of people, you have chosen the shape that maximizes the claimable number and minimizes the realizable one. That may still be the right choice. It should be a conscious one.

Self-reported savings, and what happens to them

Two things about survey data, since it is the evidence almost everyone uses.

First, people overestimate time saved, consistently and without any intent to mislead. Asking someone how much time a tool saves them is asking them to compare their current experience against a remembered version of a process they have stopped doing. Memory is generous about this in a fairly reliable direction.

Second, and more usefully, reported savings decay. Measure at week two and you get a peak, because the contrast with the old way is vivid and the novelty is real. Measure the same population at month six and the number will be materially lower. Measure at month twelve and lower again, as the improved process becomes the baseline against which nothing feels saved.

This is not people becoming disillusioned. It is the reference point moving, which is exactly what you wanted to happen. But it means an organization that measures once, at week two, and then funds a permanent capability on that number, has funded on a peak that will never recur.

The honest format

What I would put on the slide instead, and what the Fox put on the next version of his.

State the gross number. Twenty two minutes, six hundred and eighty animals, here is what that multiplies to. Do not hide it, it is real and it is the reason anyone is interested.

Then state the conversion path for each portion of it. This much lands as cost avoidance, here is the specific requisition. This much lands as redeployment, here is the output metric that will show it. This much has no conversion path and is reported as a nice thing.

Then state the realized number, which will be dramatically smaller. And then show the gap between the two, explicitly, as its own line.

That gap is not a weakness in your case. Showing it is the single thing that makes everything else in the presentation believable, because it demonstrates that you understand the difference between the two numbers, which is the exact thing the Crow was testing for when she asked her question.

The Crow is not the enemy

One more thing, aimed at anyone who reads this and feels defensive on the Fox’s behalf.

A benefits case that survives scrutiny gets funded again next year. A benefits case that does not survive scrutiny does not merely fail, it takes the credibility of the whole program with it, and the next request from the same team starts from a worse position than the first one did.

The Crow was not trying to kill the capability. She was trying to find out whether she could defend it to somebody more senior than her, six months from now, in a room the Fox will not be in. The question she asked is the question she will be asked. She was doing him a favor and it took him about a week to work that out.

Three ways this goes wrong

Loaded rate multiplication. Hours times salary, no conversion path, presented with total confidence. This works exactly once, on an audience that has not seen it before.

Double counting. The same saved hours claimed by the platform team, the use case owner, and the enterprise transformation program, none of whom are aware the others are claiming them. In a large organization this is startlingly common and it only surfaces when somebody adds up all the claimed benefits and finds they exceed the total cost base of the function.

The benefit with no owner. Claimed in the case, never assigned to anybody, therefore never realized, and nobody notices because nobody was watching for it. This is the most common of the three and the quietest.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, ask two questions of every claimed benefit. Which of the three conversion paths does this use, and who owns realizing it. Thirty seconds, and it will do more for the quality of your portfolio than any amount of process.

If you sit in the Crocodile’s chair, give the business the usage and acceptance data they need to make an honest case. Withholding it does not protect the program. It guarantees the case gets built on survey data instead, which is weaker, and which will fail later at higher cost.

If you sit in the Mandrill’s chair, commit to a realized number rather than a gross one, and accept that it will be a fraction of the headline. A smaller number you hit is worth considerably more to your standing than a large one you miss, and it is the only version that gets you funded a third time.

For everyone: stop reporting hours saved as a headline figure. Report the conversion. If there is no conversion, report it as a nice thing, in its own box, honestly labeled.

Jungle Lesson 8

Time saved is not money saved until somebody does something specific with the time. Twenty minutes back for six hundred animals is not two hundred days of capacity, it is six hundred slightly better afternoons, and you cannot put an afternoon in the ledger.

Next time: the Owl finally has his moment. It turns out he has been quietly holding a control group for two seasons because nobody ever told him to stop, which makes him the only animal in the jungle who can prove anything at all. Lesson 9 is about baselines, counterfactuals and holdouts, and how to demonstrate value when you cannot run a clean experiment.

If you are about to present a benefits case built on survey data and a loaded hourly rate, the question at the top of this piece is the one you will be asked. It is worth having an answer before somebody else has the silence.

Scaling AI FinOps | Lesson 7: Who Pays for the Watering Hole

The argument about who paid for the Watering Hole took four meetings, and it was the first time in this whole story that nobody was confused about anything.

That is what made it different. Every previous fight had been a misunderstanding at heart. This one was not. Every animal in the room understood the question perfectly, understood what the other animals wanted, and understood exactly why they wanted it. They just disagreed, in the ordinary way, about who should carry a cost that all of them benefited from.

The Mandrill made the strongest case. He pointed out that his territory had been the first to adopt, that early adoption had subsidized the platform’s development for everyone else, and that charging him now on consumption would penalize him for having taken the risk while the cautious territories waited.

This was completely true. It was also, and he knew it, entirely beside the point, because the same argument would be equally available to him next year and the year after. The Hyena, from her branch, described it as the finest piece of reasoning she had ever heard deployed in service of not paying for something.

The Sloth arrived at the fourth meeting with a framework. It was a good framework. It would have been extremely useful at the first meeting, roughly six weeks earlier, which the Sloth acknowledged without any evident distress.

“The destination,” he said, “was always correct.”

This is not a technical problem

I want to be blunt about this because I have watched a lot of organizations get it wrong in the same way. Allocation is not a tooling problem, it is not a data problem, and it will not be solved by buying something.

It is a question about how your organization distributes the cost of a shared good among parties who all want it to exist and all prefer someone else to fund it. That is a political economy question. It has been a political economy question for as long as organizations have had shared services, and the fact that this particular shared service involves inference does not change its nature at all.

Which means the answer has to be decided, announced, and defended. It cannot be discovered in a dashboard.

Three models, and what they actually do

Central absorption. The platform sits on a central budget and is free at the point of use. Adoption is fast, because nothing is easier to adopt than something free. Consumption discipline is precisely zero, because there is no reason for it to exist. And by the second year the central line has grown large enough that somebody senior asks what it is, at which point it is politically indefensible, because nobody can attribute a single unit of it to a single business outcome.

Showback. Costs are attributed and reported to consuming territories but not actually charged. Behavior changes, genuinely, but modestly. People do respond to seeing their own number even when nothing happens as a result of it. This is the right starting position for almost every organization and it is treated as a waypoint when it should be treated as a destination for at least a year.

Chargeback. Costs move to the consuming budget. Discipline becomes real, immediately and unmistakably. So do the perverse incentives, which arrive in the same week and which almost nobody plans for.

What chargeback does to behavior

Chargeback is not wrong. It is the correct end state for a mature platform. But it changes what people optimize for, and the changes are worth naming before you switch it on rather than discovering them afterwards.

  • Local optimization beats global. A territory will happily take an action that reduces its own charge and increases total organizational cost, because the first number is on its report card and the second one is not.
  • Avoidance of the sanctioned path. If using the platform incurs a charge and using something else does not, some proportion of your organization will use something else. You have just recreated the conditions that produced the Tortoise’s list two lessons ago.
  • Under-investment in the commons. Nobody volunteers to fund an improvement to a shared component that mostly benefits other territories, so shared components stop getting improved.
  • Experimentation stops. This is the one that costs the most and shows up the least. Exploration is cheap in absolute terms and highly visible on a chargeback line, so it is the first thing a territory cuts when it wants its number down.

The shared component problem

Underneath the political argument sits a real structural one that pure consumption splitting cannot handle.

Some things in the Watering Hole serve everybody and are attributable to nobody. The retrieval index. The evaluation harness. The guardrails. The gateway. These exist so that any capability can be built at all, and their cost has almost nothing to do with how much any particular territory consumes.

Split those on consumption and you get an absurd outcome, which is that the first territory to adopt pays for digging the hole and every subsequent one drinks from it at marginal cost. This is not a hypothetical. It is exactly what the Mandrill was complaining about, which is why his argument was annoying rather than wrong.

The Crocodile, who has watched this pattern play out with every shared platform of the last twenty years, offered the observation that ended the fourth meeting. “You are trying to solve two problems with one rule,” he said. “Stop.”

The shape that works

Two mechanisms, not one.

A fixed floor, funded centrally, covering the shared foundation. The gateway, the retrieval infrastructure, the evaluation capability, the guardrails, the people who keep it running. This is infrastructure. You do not consumption-split the electrical system.

A variable band above it, tracking actual consumption, shown back initially and charged back once the numbers are trusted. This is the part that should respond to behavior, because it is the part behavior actually drives.

And a published rule defining which is which. Here the rule matters considerably more than where the line falls. Almost any defensible boundary will work if it is written down, explained, and applied consistently. No boundary will work if it is discovered by territories one invoice at a time.

Tagging is the precondition

None of this functions on untagged traffic, which is why the Field Kit item back in Lesson 1 was request level instrumentation and why I have kept returning to it.

If you cannot say which team, which use case and which business purpose generated a given unit of consumption, you cannot allocate. You can estimate, and estimates get disputed, and disputed allocations produce meetings about attribution methodology instead of meetings about decisions.

The unglamorous truth is that the organization that did three weeks of tagging work early can have this entire conversation, and the one that did not cannot, regardless of how sophisticated its thinking about allocation models happens to be.

The maturity path

Absorb, then show back, then charge back. In that order, and do not skip.

The skip is tempting because chargeback is obviously the mature answer and absorption is obviously the immature one, so why spend a year in the middle. The reason is trust. Chargeback introduced before anyone believes the numbers converts every review into a dispute about attribution accuracy. You will spend a year arguing about the data instead of a year making decisions with it, and at the end of that year the credibility of the whole exercise will be lower than when you started.

Showback is where the numbers get argued into shape at low stakes. That is not a delay. That is the work.

Three ways this goes wrong

Chargeback before trust. Real money moving on numbers nobody believes. Every meeting becomes a methodology seminar.

The free lunch that ends abruptly. Central absorption in year one, sudden chargeback in year two, adoption falls off a cliff, and the drop gets reported upward as evidence that the business was not really interested after all.

Precision theater. Enormous effort spent allocating the last few percent accurately while the entire shared foundation sits in an unexamined central bucket that nobody has looked at in three quarters.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, publish the allocation rule before you publish the first allocated number. People can accept or reject a rule. They cannot do either retroactively, and if the rule arrives after the invoice it will be read as a justification rather than a policy.

If you sit in the Crocodile’s chair, make the tag mandatory at the gateway. Untagged traffic gets rejected, not absorbed. This will be genuinely unpopular for about two weeks and it settles the question for about two years.

If you sit in the Mandrill’s chair, have the argument about the model now, loudly, while it is still abstract. Once the numbers exist, every argument you make about the model will be heard as an argument about your own bill, and you will lose it on those grounds regardless of whether you are right.

For everyone: name the shared components explicitly, in a list, and decide who funds them separately from the consumption question. Two problems, two mechanisms.

Jungle Lesson 7

Everybody wants the watering hole and nobody wants it on their books. Decide who pays for the shared foundation before you decide how to split the water, because a consumption split with no floor makes the first animal to drink pay for digging the hole.

Next time: the Fox presents a benefits case with a very large number in it. The Crow does not challenge the number. She asks one question about where the hours actually went, and the room goes quiet for long enough that everyone understands what has just happened. Lesson 8 is about quantifying productivity without lying, and it is the one I would most like people to read.

The Sloth’s framework, incidentally, was good. It arrived late because nobody asked him until the situation was already unpleasant. There is usually a Sloth, and there is usually a framework, and the delay is more often a demand problem than a supply one.