The first model provider invoice arrived itemized by day. Thirty-one rows, most of them small, four of them not. The four large ones corresponded to days when we had been testing, which is to say to days when nobody was using the product at all.
What struck me was not the amount, which was trivial. It was the shape. This was a bill that moved with activity, and every previous software bill I had ever seen was a bill that moved with the calendar.
That difference is the whole subject. It is the reason an AI company is not a software company with a model attached, and it is the line item that quietly decides what business you are in.
Inference is cost of goods sold
Classic software runs at eighty to ninety percent gross margin because it is built once and the cost of serving the next customer rounds to nothing. Hosting is largely fixed and spreads across a growing base. That single fact produced two decades of business models, valuation multiples and hiring plans.
An AI product cannot do that trick. Every request spends real money. The marginal cost of the next customer is not near zero, it is a function of how much they use you.
The numbers now in circulation are consistent across sources and worth committing to memory. AI product gross margins are landing in the fifty to sixty percent band rather than the eighty to ninety percent band. One widely cited 2026 survey of several hundred software executives put the average AI product gross margin at 52 percent, up from 41 percent in 2024. Model inference accounted for somewhere around twenty to twenty-three percent of total AI product cost.
Read that last figure carefully, because the direction is the surprising part. Inference share rises as products mature. It does not fall. The intuition from classic software, where unit cost decays as you scale, is exactly inverted. You succeed, usage deepens, cost of goods grows as a proportion of the total.
A founder who assumes the old curve will build a plan in which margin improves automatically with scale. It does not. It improves with engineering, or it does not improve.
Cost per query, and why it is a weekly number
There is one metric that should be on a wall. Total inference spend for the period divided by requests served in the period.
Track it weekly, not monthly, because monthly is slow enough that a bad change ships, propagates and becomes normal before anyone notices. Early-stage products commonly sit anywhere from a fraction of a cent for a lightweight completion to fifteen cents or more for a multi-step agent run that calls tools and reasons across several turns. The spread across that range is enormous and it is entirely determined by choices you make.
When the number rises, there are only two explanations and both are worth knowing about immediately. Either you shipped a more capable feature without optimizing the path underneath it, or the amount of context being consumed per task has grown, usually because somebody added history, retrieved documents or examples to a prompt and it helped.
Neither is wrong. Both are decisions, and a decision you did not know you made is the one that shows up as a margin problem two quarters later.
The levers, in order of size
Routing. This is the largest lever by a wide margin and the one most teams reach for last. Within a single provider’s lineup the price spread between the cheapest capable tier and the most expensive frontier tier commonly runs five times or more per token. Most requests in most products do not need the top tier. Classify the incoming request, send the routine majority to the cheap tier, reserve the expensive one for the cases that genuinely need it, and measure quality on both. Teams that do this well route the large majority of traffic to inexpensive models and see no user-visible degradation.
Caching. Repeated context, whether system instructions, retrieved documents or conversation prefixes, does not need to be paid for at full rate every time. The mechanisms differ by provider and the savings on a workload with stable context are large enough to change the shape of a P&L.
Context discipline. The cheapest token is the one you never send. There is a common pattern where a retrieval step returns twenty documents, all twenty go into the prompt because it is easier, and three of them were relevant. Retrieval quality is a cost lever disguised as a quality lever.
Batching and asynchronous processing. Where the user is not waiting, work that runs on a delayed queue is materially cheaper. A surprising proportion of what teams build synchronously does not need to be.
Self-hosting. Last, deliberately. There is a crossover point where running your own inference beats paying per token, and it exists, but it arrives at volumes higher than most early companies reach and it brings an operational burden that a team of six should think hard about accepting. Do the arithmetic before the conversation, not during it.
Model three futures, not one
Unit prices for inference have fallen dramatically. Capability that cost roughly twenty dollars per million tokens from a frontier model in late 2022 was available at a small fraction of that by early 2026. Hardware rental rates for the relevant accelerators fell substantially across 2025 as well.
It is tempting to plan on that continuing. Do not plan on it exclusively. Build three scenarios.
In the first, prices keep falling at roughly ten percent a quarter and today’s fifty percent gross margin drifts toward seventy without you changing your pricing. That is the optimistic case and it is genuinely plausible.
In the second, prices hold flat. Your margin is whatever your engineering makes it.
In the third, your cost per unit of work rises even as token prices fall, because you shipped more capable features and the number of tokens consumed per completed task grew. This has been the actual experience of a lot of teams. Token consumption per task has risen by one to two orders of magnitude since late 2023 as products moved from single completions to multi-step reasoning with tool use.
If the business only works in the first scenario, that is important information and it is available today rather than in eighteen months.
Three ways this goes wrong
You put inference in operating expenses. It sits in cost of goods sold. Booked as an operating cost it disappears into a line with the design tool subscription, your reported gross margin becomes fiction, and you will not discover the error until somebody in diligence asks a question you cannot answer.
You measure the average and ignore the distribution. Usage in these products is heavily skewed. A small number of users generate a large share of consumption. An average cost per customer that looks healthy can conceal a handful of accounts that are individually unprofitable, and on a flat rate plan those are the accounts most likely to renew enthusiastically.
You optimize before you measure. The instinct to cut costs early is good and it is frequently spent in the wrong place. Instrument first: cost per request, per customer, per feature. Then optimize the thing that is actually large, which in my experience is almost never the thing anyone guessed.
In AI you do not discover your margin at year end. You design it at the start, or somebody else designs it for you.
Monday: the pricing page, and the rule that stops your best customer becoming your worst one.