谁来买单？LLM 网关中的 Token 计量

Once you put a large language model behind a company API gateway, someone asks a simple question. Who is paying for this? The model bill arrives as one number. Finance sees one number. Every team in the company sees one endpoint. Nobody can tell you which team burned forty percent of the budget, which use case is growing, or which experiment someone forgot to shut off six weeks ago. That gap between one bill and one endpoint is where metering lives. We built metering into our LLM gateway before we scaled usage, and it changed how teams behaved. Here is what we learned. Why LLM metering is not like API metering With a normal REST API, metering is easy. One request costs one request. You count requests per subscription key, and you are done. LLM calls do not work that way. One request to a chat endpoint can cost five tokens or five thousand. A prompt injection test script someone ran in a loop overnight can spend more than an entire product team's monthly budget in eight hours. If you count requests, you learn nothing. The unit of cost is the token, and the token count lives in the response, not the request. That detail shapes everything. Your metering code has to read the response body, extract the usage block, and attribute it before the data is gone. { "usage": { "prompt_tokens": 1240, "completion_tokens": 312, "total_tokens": 1552 } } Enter fullscreen mode Exit fullscreen mode Azure OpenAI returns this on every response. If you do not capture it at the gateway, you have to reconstruct it later from logs, and that is a losing fight. Capture it at the gateway, not in the app Every team wanted to do metering in their own application code. We said no. The gateway is the one layer every call passes through, regardless of language, framework, or team. A metering implementation that depends on each team instrumenting their own service is an implementation that covers sixty percent of traffic on a good month. We put a policy on the gateway's inbound path to stamp each request with metadata: the subscription the key belongs to, the product, the environment. On the outbound path, another policy parses the response, pulls the token counts, and ships an event to our analytics pipeline. The policy does not touch the request or response bodies in ways the caller can see. It observes and reports. @{ // parse the usage block and forward a metering event // key: subscription id, product, environment, model // value: prompt_tokens, completion_tokens, latency } Enter fullscreen mode Exit fullscreen mode This is pseudo-code, not a real policy file, because every gateway has its own syntax. The shape matters more than the syntax: stamp metadata on the way in, extract cost on the way out, send it somewhere durable. Attribute everything, including the failures The first version of our metering only counted successful calls. That lasted about two weeks. Someone ran a batch job against a deprecated model deployment that returned 400s all day. The calls were cheap individually, but there were hundreds of thousands of them, and the model provider still charged for the input tokens on each failed request. Failed requests still consume input tokens. Rate-limited requests still consume them. Requests that hit a content filter still consume them. If you only meter the 200s, your bill and your dashboard will disagree, and finance will believe the bill. Our outbound policy records every response code. Failures get their own bucket so teams can see them separately, but they are not excluded from the total. The first time a team saw how much they were spending on 400s, they fixed their client code that afternoon. Give teams their own numbers, not the whole company A company-wide dashboard that says the LLM bill is fifty thousand dollars a month helps nobody. A dashboard that says your team spent nine thousand, up forty percent from last month, on a specific model, changes behavior overnight. We publish per-team views: tokens and estimated cost by team, by model, by day. The estimated cost part matters. Tokens are abstract. Dollars are not. We maintain a small price table per model and multiply. The estimate is not perfect, and we say that on the dashboard. It is close enough to drive decisions. The behavioral change was immediate. Teams started batching requests. They cached repeated prompts. One team switched a summarization job from a large model to a smaller one after seeing the per-run cost. None of that happened when the bill was a single line item. Budgets need teeth, but start with visibility Once teams can see their spend, the next question is limits. We added budget alerts first, hard stops second. The sequence matters. Alerts went to the team's channel when they crossed fifty, eighty, and one hundred percent of their monthly budget. Most teams self-corrected at fifty. For the ones that did not, a hard stop at the gateway rate-limits further calls until someone raises the budget. We deliberately did not lead with hard stops. An engineer running a time-sensitive demo who hits a hard limit learns that the gateway is the enemy. An engineer who gets a warning at fifty percent learns to watch their budget. The hard stop exists for the runaway loops, not for normal work. What we would do differently Three things we would change if we started over. First, we would meter streaming responses from day one. Token counts arrive at the end of a streamed response, and our first implementation missed them. Streaming became popular faster than we expected. Second, we would standardize the metadata stamp earlier. We let teams add custom attributes organically, and we ended up with three different keys for the same project name. Normalizing that retroactively was tedious. Third, we would keep the price table in version control with a review step. Model prices change. When they do, old estimates look wrong. We now treat the price table like code: a pull request, a reviewer, a changelog. The real lesson Metering is not a billing feature. It is a feedback loop. The moment teams could see what their calls cost, they started engineering like the cost mattered. Architecture improved. Waste fell. Nobody needed a policy document telling them to be careful with the expensive model, because the numbers told the story better than any document could. If you are standing up an LLM gateway and nobody has asked about cost attribution yet, they will. Build the metering in before the bill becomes a conversation.