Scaling AI FinOps | Lesson 22: Nobody Gets Promoted for Switching Things Off

The last Hummingbird was switched off on a Thursday afternoon, twenty one lessons after it first appeared, and it took about four minutes.

Nobody could establish what it had been for. It had a name that suggested a purpose, but the purpose it suggested had been served by something else for over a year. The animal who built it had moved to another territory two seasons earlier and, when asked, remembered starting it and did not remember why.

It had been running the entire time. It had consumed a small amount of money every single day for two years, quietly, correctly, producing output that went into a location nobody had opened.

The Fox switched it off and waited a fortnight, which was the agreed protocol by that point.

Nothing happened. No complaint, no escalation, no broken workflow. After the fortnight it was removed properly.

The Hyena, who had made a good deal of noise about the Hummingbirds over the preceding two years, was oddly quiet about this one. “It ran for two years,” she said. “Somebody built it because they cared about something. And in the end nobody noticed it stop.”

Which is roughly the correct emotional register for this topic, and the reason it is the least discussed subject in the whole discipline.

The incentive problem, stated plainly

Launching is visible, credited and promotable. There is a demonstration, an announcement, a slide with your name on it and a line in your performance review.

Retiring is invisible, thankless, and carries all of the risk. Nobody is congratulated for it. The upside is a cost reduction that will be attributed to the portfolio rather than to you. The downside is that you broke something, and that will be attributed to you very specifically.

So every enterprise accumulates. This is not a discipline failure and it is not a maturity failure. It is a rational response to an incentive structure, and it will not be fixed by encouragement, by policy, or by anyone writing the word rationalization on a slide. It gets fixed by changing what is rewarded, and by nothing else.

What accumulates in an AI estate

Five categories, in roughly ascending order of how much they cost and how invisible they are.

  • Orphaned pilots. The Lesson 5 population. Individually small, collectively significant, and completely undefended once anybody counts them.
  • Superseded capabilities. Something better was built. The old thing still serves eleven people who never migrated. It costs a full capability’s fixed overhead to serve eleven people.
  • Evaluation harnesses for retired models. Still running, still scheduled, still consuming, testing something that no longer exists.
  • Duplicate capabilities. Two territories built the same thing before the portfolio function existed. Both work. Neither owner will volunteer to be the one that stops.
  • Retrieval indexes nobody queries. The purest case, and the most expensive of the five.

The index nobody reads

The last one deserves its own section because it is the cleanest illustration of why AI waste is different from every other kind, which was the argument that opened this entire series.

A retrieval index costs money every day to keep current. It re-processes content as content changes. It does this whether or not anybody has ever queried it. It appears on no dashboard as waste, because it is running perfectly, doing exactly what it was built to do, without error.

And there is no signal anywhere in your systems that says nobody has asked this anything in fourteen months. Unless somebody specifically goes looking for query counts against index cost, it is invisible forever, and it costs more than any of the orphaned pilots.

This is the Lesson 1 point arriving at its final form. Waste that looks exactly like work, running correctly, indefinitely, because the only thing that would reveal it is a question nobody has thought to ask.

Sunset criteria, written at launch

The intervention that actually works, and the reason it works is timing.

Three things, written into the launch approval, before anybody is attached to anything.

A usage floor. Below this many units per period, this gets reviewed for retirement. Not automatically killed, reviewed, which is a much easier thing to agree to in advance.

A value floor. Below this contribution against the ledger, same.

A named alternative. If this is retired, what do its users do instead. Answering this at launch is trivial. Answering it three years later requires an investigation.

The timing is the entire trick. At launch, nobody is emotionally invested and the criteria feel like sensible hygiene. Three years later, the same conversation is a negotiation with somebody defending their creation, and nobody can objectively assess their own work at end of life. The criteria have to predate the attachment.

Dark launch the shutdown

The single most useful and most under-used technique in this post.

Before removing anything, turn it off quietly and wait. Two weeks is usually enough. Do not announce it. Do not send a consultation email, because a consultation email guarantees that somebody who has not used it in a year will explain that they might need it.

Then see who complains.

The proportion of the time that nobody complains is genuinely surprising the first few times you do it. And when somebody does complain, you have learned something specific and valuable, which is that this thing has exactly one real dependency and you now know who it is and can talk to them directly.

The formal version of the process is identify, notify, migrate, dark launch, remove. But the dark launch step is what turns decommissioning from a negotiation into an observation, and observations are much easier to act on than opinions.

Making retirement promotable

The structural fix, and it is the only one that lasts.

Report decommissioning as a headline achievement, in the same document, with the same prominence as launches. Count capabilities retired alongside capabilities launched, as a matched pair, so a portfolio with twelve launches and zero retirements looks like what it is.

Give the portfolio owner from Lesson 16 explicit credit for reductions. Put it in objectives. Make it the thing somebody is measured on rather than the thing everybody agrees is important.

Organizations that do this find that things get switched off. Organizations that merely encourage it find that nothing does, and then conclude that their people lack discipline, when what their people lack is a reason.

The archaeology problem

One last point, which is really a Lesson 6 problem arriving late.

After eighteen months nobody remembers why something exists. The person who knew has moved. The document that explained it was in a folder that got reorganized. So the decision to keep it defaults to yes, because keeping something you do not understand feels safer than removing something you do not understand.

Documented purpose at launch is what makes retirement possible later. One sentence, written when it was obvious, saying what unit of work this exists to improve and how you would know. That sentence is what a future colleague uses to decide, and its absence is why the Hummingbird ran for two years.

Three ways this goes wrong

Retirement by attrition. Waiting for things to become obviously useless, which they never quite do, because something running correctly never looks useless.

The hostage user. One team still depends on it, so it stays forever at full cost serving four people, and nobody ever does the arithmetic on what those four people are costing.

No sunset criteria. Every retirement becomes a fresh negotiation with somebody emotionally invested, which means retirements happen only during reorganizations, which is not a strategy.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, ask for the list of capabilities retired this year. If the answer is none, your portfolio is growing monotonically and your cost curve is doing the same thing whether or not anyone has noticed.

If you sit in the Crocodile’s chair, find what is still running with no traffic and no owner. Start with the retrieval indexes and compare index cost against query counts. That single comparison finds more money than most optimization projects.

If you sit in the Mandrill’s chair, write sunset criteria into your launch approvals. This is the only moment anybody will agree to them and it takes ten minutes at a point when nobody objects.

For everyone: dark launch the shutdown. Turn it off, wait a fortnight, see who notices. It is remarkable how often the answer is nobody, and it is the cheapest experiment available to you.

Jungle Lesson 22

Every enterprise is structurally incapable of switching things off, because launching is promotable and retiring is invisible. Write the sunset criteria at launch while nobody is attached to it yet, count retirements the way you count launches, and when in doubt, turn it off quietly and see who complains. Surprisingly often, nobody does.

Next time: the Tortoise and the Fox present together, which nobody in the jungle would have predicted two years earlier, and make a case that assurance is not what you pay to satisfy compliance but what you pay for the right to change anything quickly. Lesson 23 is about the cost of trust.

Every organization has its own Hummingbird, running correctly, costing a little every day, serving a purpose nobody remembers. Finding it is not hard. Deciding to look is the hard part, because nobody has ever been thanked for it.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.