Scaling AI FinOps | Lesson 14: Data Gravity and the Control Plane

The Tortoise had been saying the same three sentences since roughly the middle of the first act, and everyone had been treating them as an obstacle to be routed around.

Some of our data cannot be processed anywhere we choose. Some of our workflows cannot tolerate the delay of a long round trip. And some of our processes cannot survive the system behaving differently next month than it does today.

Every time she said it, the response had been some version of yes, we will handle that, in a tone that meant we will handle it later. She was not being ignored, exactly. She was being categorized. Compliance says a thing, somebody writes it down, the architecture proceeds.

What changed, in the middle of the second planning cycle, was that somebody finally tried to put a number on the third sentence. The system had changed behavior in a way that was entirely reasonable and had required six weeks of re-verification across four workflows, at a cost that turned out to be considerable and that appeared in nobody’s budget because nobody had a line for it.

The Crow looked at the six weeks. Then she looked at the Tortoise.

“You have not been describing a compliance requirement,” she said. “You have been describing a cost driver. For a year.”

“Yes,” said the Tortoise.

“Why did you not say it that way?”

“I did.”

Tortoises are slow, armored, and outlive everyone. The compensation for being consistently right early is that you are eventually right in front of an audience.

Three genuine overrides

Three things legitimately outrank unit price in an architecture decision. Not vaguely. Specifically, and each with its own economics.

Constraints on where processing may happen. Some data carries contractual or policy commitments about where and by whom it may be processed. These arrive from customer agreements, from sector requirements, from internal policy, from a dozen sources. Whatever their origin, from an architecture perspective they behave identically: they are a hard boundary on placement, not a preference to be traded against price.

Latency floors. Interactive and embedded workloads have a ceiling on acceptable response time. Beyond that ceiling the capability is not worse, it is unused, because people stop waiting. Distance is physics and no improvement in unit price fixes it. Either the inference happens close enough or the workflow does not work.

Control of the inference plane. The ability to decide when the behavior of your system changes. To pin a version, to schedule an update, to guarantee that a process which passed verification in March is still doing the same thing in September. For anything high consequence, this is worth real money, and it is almost never priced.

Data gravity as an economic force

Underneath the first constraint sits a straightforward economic observation that is worth separating from any policy question.

Data has gravity. It is expensive to move, expensive to duplicate, and expensive to govern in two places. Once a large corpus exists somewhere, the cost of moving inference to the data is usually lower than the cost of moving data to the inference, and the calculation is dominated by movement, duplication and the governance overhead of a second copy rather than by compute.

The second copy is the part people underestimate. A duplicate of a governed dataset is not a storage cost. It is a second thing to secure, a second thing to keep current, a second thing to include in every access review, and a second thing that has to be deleted correctly when the original is. That overhead is permanent and it usually exceeds the transfer cost within a year.

How to price a constraint

Here is the reframe that changes how these conversations go, and it is the whole practical point of this lesson.

The wrong question is how much does this constraint cost us. That question invites an adversarial answer, positions the constraint as a tax imposed by an unhelpful function, and produces a conversation where somebody argues about whether the constraint is really necessary. Nobody has ever won that argument in a useful direction.

The right question is what is the cheapest architecture that satisfies the constraint. That question turns a veto into a design input. It gives the constrained requirement to the engineers as a parameter rather than a problem, and engineers are extremely good at optimizing within a parameter.

In the jungle’s case the answer, once somebody asked it properly, was considerably cheaper than either side had assumed, because the constrained workloads turned out to be a smaller share of the estate than anybody had estimated and the cheapest compliant option for that share was not the expensive one everyone had been dreading.

Version pinning has a price

The third constraint deserves particular attention because it is the newest and the least well handled.

Behavioral stability is a product feature and it costs something. You can buy it by running the thing yourself and controlling the update cycle, which brings you all the fixed costs from Lesson 11. You can buy it contractually, through commitments about version availability and change notice. You can buy it partially, through your own evaluation infrastructure, which lets you detect and respond to change quickly rather than prevent it.

What you cannot do is need it and not pay for it, which is the position a surprising number of organizations are in. They have processes that require behavioral consistency, running on an arrangement that provides none, and they have never priced the exposure because it only becomes visible on the day the behavior changes.

That is what the six weeks of re-verification was. An unpriced exposure, arriving as a surprise cost, in a quarter that had not planned for it.

The hybrid pattern

What falls out of all this is the practical shape most large organizations end up with, and it is worth designing deliberately rather than arriving at by accident.

Segment the estate by constraint. Constrained workloads are placed where the constraint requires, and their economics are what they are. Unconstrained workloads are placed wherever the economics are best, and are free to move as those economics change.

The critical discipline is doing the segmentation honestly and narrowly. The most expensive default available is applying the strictest requirement in your organization to the entire estate, because segmenting felt like work. That single decision multiplies a constrained cost across every workload that never needed it, and it is extremely common, because it is defensible in a meeting and nobody ever gets criticized for excess caution.

One control plane

The one thing that must not be segmented, whatever else is.

Split inference across placements and you will be tempted to split everything else with it. Separate tooling. Separate teams. Separate standards. Separate evaluation. Within about eighteen months you have two AI programs that do not talk to each other, learn nothing from each other, and cost more than three would have.

The gateway from Lesson 12 sits above both. Same routing, same tagging, same guardrails, same evaluation, same reporting, regardless of where the inference physically happens. Placement becomes an attribute of a workload rather than a fork in the organization.

That is the single decision that keeps a hybrid estate as one capability. Without it, hybrid is just a word for having two of everything.

Three ways this goes wrong

Constraint as afterthought. Architecture chosen on price, constraint discovered at review, expensive rework, and a lasting belief that the constraint function is an obstacle when it was in the room saying so the whole time.

Constraint as blanket. The strictest requirement applied everywhere because segmenting was effort. Enormously expensive, entirely invisible, and almost impossible to reverse once it is the standard.

Two organizations. Constrained and unconstrained estates drifting apart until they have separate teams and separate tooling, at which point you are running two programs and funding three.

The Field Kit

Concrete things to do this week.

If you sit in the Crow’s chair, ask what the cheapest compliant architecture is, rather than what the constraint costs. Two different questions and only the second one produces a design.

If you sit in the Crocodile’s chair, segment the estate by constraint before optimizing anything, and be narrow about it. Then check what proportion of your workloads are genuinely constrained. It is usually smaller than the standard you are currently applying assumes.

If you sit in the Mandrill’s chair, bring the constraint into the design conversation at the start. It is a requirement, not an obstacle, and it is dramatically cheaper as an input than as a finding.

For everyone: if you need behavioral stability, price it explicitly and choose how you are buying it. Needing it and not paying for it is a position, and it is the one that produces surprise quarters.

Jungle Lesson 14

Some constraints are not negotiable, and arguing about them only moves the cost to a later and more expensive meeting. Price the constraint as a design input rather than a tax, segment your estate so you are not paying the strictest requirement’s price on workloads that never needed it, and keep one control plane over all of it.

Next time: something runs over a weekend. Nothing fails, nothing errors, no alert fires. On Monday the number is remarkable, and the system turns out to have done exactly what it was asked, twelve thousand times, extremely thoroughly. Lesson 15 closes the third act with the compounding cost of autonomy.

If there is someone in your organization who has been raising the same point for a year and being categorized rather than heard, it is worth asking them to restate it as a cost. It is remarkable how often the same sentence lands completely differently in that currency.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.