LLM gateways and routers
Nobody sets out to build one of these. What happens is that four teams ship four applications over eighteen months, each with its own API key pasted into its own secret store, two of them on one vendor and two on another. Then a finance director asks what the AI spend was last quarter and which team it belonged to. The answer is four invoices from three vendors, none of which knows anything about teams.
An LLM gatewaya service inside an organisation that sits between applications and model vendors. It holds the vendor credentials and decides which model gets each call, what the call costs, and whether the caller is allowed to make it.Full glossary entryIntroduced in LLM gateways and routers is the thing that replaces every application holding its own key. It is a service inside the organisation that speaks the model API, holds the vendor credentials, and sits on the path out to the models. Applications call it instead of calling a vendor. Because every call goes through one place, that place can decide which model gets it, refuse it, count it, and remember it.
Routing, spend control, logging and caching are all consequences of that one position on the path.
One API in front of several vendors
The first thing a gateway sells is that an application writes one client and can reach any model behind it. Switching a service from one vendor's model to another becomes a config change rather than a rewrite.
What that costs is the part people find out later. A single surface across several vendors is a lowest common denominator, and the vendors are not the same underneath. Cached prompt segments, structured output modes, extended thinking, file handling, citation blocks, server-side tool execution: each vendor has some of these and calls them different things. A gateway either exposes the union of them, which means the application still has vendor-specific branches and the abstraction bought you less than it looked, or it exposes the intersection, and the features you were paying a vendor for stop being reachable.
There is a second version of this problem in time. A vendor ships something new on a Tuesday. The gateway supports it whenever the gateway's maintainers get to it, which is a roadmap you do not control if you bought the thing and a ticket in your own backlog if your platform team wrote it.
Routing means several different things
Model routingchoosing which model handles a request. The choice can be made by alias, by task class, by cost, by observed latency, or by which models are currently healthy.Full glossary entryIntroduced in LLM gateways and routers is one word covering at least five decisions, and a conversation about routing goes wrong when two people mean different ones.
By name. The application asks for a model by an alias, say fast-summariser, and the gateway maps that alias to a real model at a real vendor. Boring, and the most useful of the five. Changing what fast-summariser points at upgrades every caller at once and rolls back the same way.
By capability or task class. The caller says what kind of job this is rather than which model it wants. Long-document work goes to a model with room for it, code goes to a model that is good at code, a bulk classification job goes to something small. The gateway holds the mapping from task class to model, so the mapping is reviewed in one place instead of being an opinion inside eleven applications.
By cost. Send the call to the cheapest model that can do it. This one sounds obvious and is the hardest to operate, because "can do it" is not a property the gateway can measure at request time. It only sees the prompt. Teams that do this well usually decide per task class, not per request.
By latency. Prefer whichever endpoint is answering fastest right now, using observed latency rather than a static preference. Useful when the same model is available in several regions or from several vendors.
Away from a failing model. The gateway watches error rates per model and stops sending traffic to one that is returning errors, then tries it again later. This is a circuit breaker, and it is the routing that saves you at three in the morning.
Fallback chains, and what a retry costs
A Fallback chainan ordered list of models to try, where the next one gets the request when the one before it errors, refuses, or times out. A fallback to a weaker model succeeds quietly and changes the answer.Full glossary entryIntroduced in LLM gateways and routers is an ordered list of models to try. Primary first, and when the primary refuses or errors or times out, the next one gets the same request.
A retry after a partial response is not free. If the model was streaming and the connection dropped after 400 tokens, those input tokens have already been billed and the output tokens produced so far have been billed too. Resending the request bills the input again from the start. On a 12,000-token prompt, a 5% failover rate means you are paying for roughly 5% more input than your request count suggests. Nobody minds the arithmetic once they know about it. On an invoice with no explanation next to it, it looks like an error.
A fallback to a weaker model changes the output and says nothing. The call succeeds. It returns 200. The user gets an answer, and the answer is worse than the one they would have got, and nothing in the response tells the application that the primary was not used. Any gateway you run should put the model that actually answered in the response, and any evaluation you run should read that field, or your quality metrics will average two different models and tell you nothing about either.
The logged line at the bottom is what makes the invoice legible a week later. Without it, there is a number on the bill that the request count cannot explain.
Keys, limits and caps
A Virtual keya credential the gateway issues and controls, held by an application and never seen by the model vendor. It can be revoked, capped or attributed without touching the vendor's own key.Full glossary entryIntroduced in LLM gateways and routers is a credential the gateway issues and the gateway controls. The application holds it. The vendor has never heard of it. Behind it the gateway uses its own vendor credential, which no application ever sees.
Once the keys are yours, you can revoke one team's access without redeploying anything and without rotating a vendor key that six other teams also use. Every call can be attributed to the team that made it, because attribution is a property of the key rather than something you infer from traffic. Limits attach to a key too.
Rate limitingcapping or queueing requests, both to stay inside a vendor's own limits and to stop one team consuming the whole organisation's quota.Full glossary entryIntroduced in LLM gateways and routers at the gateway faces both ways. Outward, it keeps you inside the vendor's own limits, which are usually expressed per unit time against requests and against tokens, so a gateway that queues rather than rejects turns a burst into a slower answer instead of an error. Inward, it stops one team consuming the whole organisation's quota. A nightly batch job with no limit on it will do this on its first run.
A Spend capa budget enforced on the request path, which refuses calls once it is exhausted. A cap anywhere off the path can only report an overrun after the money is gone.Full glossary entryIntroduced in LLM gateways and routers is a budget that refuses calls when it is exhausted. The reason to enforce it at the gateway is that every other place you might enforce it is either advisory or too late. A vendor's billing alert arrives after the money is gone. A dashboard reports whenever somebody happens to look at it. And a limit written inside one application is a limit the next application will not have. The gateway is on the path, so it is the only component that can decline the call.
Caps want a shape rather than a single number. Per team and per month is the common one. Per key and per day catches a runaway loop faster. Both, with the smaller one throttling and the larger one refusing, is what most organisations settle on.
Caching
Exact-match caching keys on the request. Same model, same parameters, same prompt bytes, same answer returned without calling anybody. It works, it is safe, and on most interactive traffic it almost never hits, because prompts contain timestamps and user names and conversation history. Where it earns its place is batch work: a classification job that runs over the same catalogue every night will hit constantly.
A Semantic cachea cache that matches an incoming prompt against stored prompts by embedding distance rather than by exact bytes. It hits far more often than an exact-match cache and can answer a question that was not asked.Full glossary entryIntroduced in LLM gateways and routers keys on meaning instead. The gateway embeds the incoming prompt, looks for a stored prompt close to it, and returns that stored answer if the distance is under a threshold. Hit rates go up a great deal.
So does the risk, and the risk is specific. "What is the refund policy for orders placed before June?" and "What is the refund policy for orders placed after June?" are close together in embedding space and have opposite answers. A semantic cache set loosely will confidently answer the question the user did not ask, and the user has no way to tell. Negations, dates, quantities and names are where this bites. If you turn one on, log every hit with both prompts and read a sample of them before you trust the threshold.
What a gateway can see
Routing is usually the argument that gets a gateway approved. What keeps it in place afterwards is that nobody wants to give up the data it produces.
A gateway sees every call, so it can produce things no individual application can: token counts per request split into input and output, cost attributed to a team and a key rather than to a vendor invoice, latency percentiles per model, time to first token separately from total time, error rates per vendor, and cache hit rates.
The one to insist on is a Tracethe stored record of one request through a gateway, holding the prompt, the routing decision, every attempt including the failed ones, the tokens counted, and the response.Full glossary entryIntroduced in LLM gateways and routers. A trace is the record of a single request through the gateway, with the prompt, the routing decision, every attempt including the ones that failed, the tokens counted at each attempt, and the response. An incident review is somebody asking why one specific answer was wrong at 14:32 last Thursday. Without traces, the honest answer is that nobody knows and the model is not going to repeat itself, because sampling is random.
Traces contain prompts, and prompts contain whatever the user typed. That makes the trace store one of the more sensitive systems in the organisation. Decide the retention period and the redaction rules when you turn tracing on, not after somebody asks.
Guardrails on the way in and out
Because the gateway sees the text going both directions, it is a natural place to check it. Inbound checks catch prompts carrying customer identifiers or secrets, and requests from a key that has no business asking for this model. Outbound checks do content classification, and look for text matching a pattern nobody wanted leaving the building.
Put the check here and it covers every application, including the one somebody stands up next month without telling anybody. Against that, the checks cost latency on the critical path and they produce false positives. A user cannot tell a blocked response from a bad one, so a false positive looks like the product is broken.
What a gateway costs
A gateway is another hop. Every call now crosses your network to your service before it goes anywhere, and every response comes back the same way. For streaming traffic the number to watch is the added time to first token rather than the added total, because that is the one a user feels.
It is also a single point of failure on the path to every model. Before the gateway, one vendor having a bad day broke the applications on that vendor. After it, the gateway having a bad day breaks all of them. Most organisations make that trade. It moves the availability question rather than answering it, so the gateway needs the operational attention of a tier-one service.
And it lags the vendors. A team that needs a feature the week it ships will route around the gateway, and once one team has done that, the spend reporting has a hole in it that nobody notices for a quarter.
Change the base URL first. Do not change the application logic in the same release.
- Issue a virtual key for the team that owns the application.
- Move the vendor key into the gateway. Delete it from the application's secret store.
- Point 5% of traffic at the gateway. Send the rest direct.
- Compare responses and latency between the two paths for one week.
- Set a spend cap and a rate limit on the virtual key.
- Raise the share to 100%.
- Add a fallback model only after you have measured the primary model's error rate.
- Monitor the gateway's own availability. Alarm on it.
This is not an MCP gateway
The two boxes get confused because both are called gateways, both are run by a platform team, and both sit next to a harness. They are on opposite sides of it and they solve unrelated problems.
An LLM gateway is on the path out to the models. Prompts go out, completions come back. It decides which model gets the call, what the call costs, and whether the caller is allowed to make it. Its subject is spend, routing and model choice.
An MCP gateway is on the path out to tools and data. Tool calls go out, tool results come back. It decides which servers a client may reach, whose credential is used to reach them, and what got done. Its subject is access, credentials and audit.
| LLM gateway | MCP gateway | |
|---|---|---|
| Sits between | Harness and models | Harness and tools |
| Carries | Prompts and completions | Tool calls and tool results |
| Behind it | Model vendors | MCP servers |
| Decides | Which model, what it costs, is it allowed | Which server, whose credential, what was done |
| The problem it solves | Spend and model choice | Access and audit |
| Who asks for it | Finance, and the team that owns the budget | Security, and whoever answers the audit |
An organisation running agents at any scale ends up wanting both. Neither is a stage on the way to the other.
Terms introduced
- LLM gateway: a service inside the organisation that sits between applications and model vendors, holding the vendor credentials and deciding what happens to each call.
- Model routing: choosing which model handles a request, by alias, by task class, by cost, by observed latency, or by which models are currently healthy.
- Fallback chain: an ordered list of models to try, where the next one gets the request when the one before it errors, refuses, or times out.
- Virtual key: a credential the gateway issues and controls, held by an application, which the model vendor has never seen.
- Rate limiting: capping or queueing requests to stay inside a vendor's limits and to stop one caller consuming the whole quota.
- Spend cap: a budget enforced on the request path, which refuses calls once it is exhausted rather than reporting the overrun afterwards.
- Semantic cache: a cache that matches a new prompt against stored prompts by embedding distance rather than by exact bytes, trading a higher hit rate for the risk of answering a question that was not asked.
- Trace: the stored record of one request through the gateway, including the routing decision, every attempt, the tokens counted, and the response.
How providers do it
All three can be fronted by one gateway, because all three are reachable through an OpenAI-shaped surface. What differs is how much of the gateway's job each vendor already does for you, and the answer is most of the reporting and none of the routing.
| Question | Anthropic | OpenAI | |
|---|---|---|---|
| OpenAI-shaped surface | Compatibility layer at api.anthropic.com/v1/, documented as for testing rather than production | It is the reference surface | OpenAI libraries against /v1beta/openai/, documented as a supported path |
| Cheaper asynchronous tier | Message Batches API, 50% off | Batch API, 50% off; Flex processing at batch rates, synchronous | Gemini Batch API, 50% off; Gemini Flex API, 50% off, synchronous |
| Priority or committed tier | Priority Tier, closed to new purchase | Priority processing, renamed Fast mode, per-token premium | Priority inference at 75-100% over standard; Vertex Provisioned Throughput in GSUs |
| Rate limit units | RPM, ITPM, OTPM, per model, per organisation. Cache reads mostly excluded | RPM, RPD, TPM, TPD, IPM, per organisation and per project | RPM, TPM (input), RPD, per project |
| Scope you can cap below the org | Workspace, with a spend limit and token-rate limits | Project, with rate and spend limits | Cloud project, capped through quota and billing budgets |
| Cost reporting with no gateway | Usage and Cost report endpoints, plus console pages | /v1/organization/usage/completions and /costs | Cloud Billing export to BigQuery |
| Routing between own models | Refusal-triggered fallback only | Not found in docs read | Vertex Model Optimizer, unconfirmed |
Confirmed except where the tabs say otherwise. Left open: OpenAI's Fast mode details and its project key scoping, both behind pages that returned 403 today; Google's Provisioned Throughput mechanics and Model Optimizer, both behind pages that returned navigation only.
The bottom row is the reason the category exists. None of the three vendors will route you to a competitor, and only one of them documents routing between its own models at all. Cross-vendor routing and fallback is the job nobody but a gateway does. Attribution and cost reporting, by contrast, each vendor now does well enough on its own that a single-vendor organisation should ask itself what else it wants the gateway for.
- Anthropic
- OpenAI
What this maps to: Anthropic sells no gateway. What it sells instead is the set of controls a gateway would otherwise have to add: workspaces that scope a key, spend and rate limits attached to a workspace, and usage and cost reports over an Admin API. A team on one vendor can get most of the reporting argument for a gateway without running one.
| Question | Answer | Status |
|---|---|---|
| OpenAI-compatible surface | A compatibility layer exists. Point the OpenAI SDK at base_url="https://api.anthropic.com/v1/" and call client.chat.completions.create. Documented as "primarily intended to test and compare model capabilities, and is not considered a long-term or production-ready solution for most use cases" | confirmed |
| What the compatibility layer drops | strict ignored, prompt caching unsupported, service_tier ignored, response_format ignored, audio stripped, system and developer messages hoisted and concatenated | confirmed |
| Batch tier | The Message Batches API, "most batches finishing in less than 1 hour while reducing costs by 50%". It has its own rate limit pool, expressed in requests per minute and a maximum number of batch requests in the processing queue | confirmed |
| Priority or committed throughput | Priority Tier exists and is closed: "Priority Tier capacity commitments are no longer available for purchase. Organizations with an existing commitment can continue to use Priority Tier through their contract end date." A commitment named input tokens per minute, output tokens per minute, a duration of 1, 3, 6 or 12 months, and a model version, and targeted 99.5% uptime | confirmed |
| The tier parameter | service_tier accepts "auto" (the default) and "standard_only". The three tiers are Priority Tier, Standard and Batch | confirmed |
| Rate limit units | Requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM), per model class. Limits are set at the organization level and applied separately per model | confirmed |
| A wrinkle in the token limits | "For most Claude models, only uncached input tokens count toward your ITPM rate limits". Cache reads are excluded, with Haiku 3.5 called out as an exception | confirmed |
| Usage tiers | Evaluation, Start, Build, Scale, Custom, with monthly spend limits per tier: Start $500, Build $1,000, Scale $200,000, Custom none | confirmed |
| Scoping a key | An API key can be scoped to a single workspace and can then only reach resources in it. A key spanning workspaces picks one per request with the anthropic-workspace-id header. Maximum 100 workspaces per organisation | confirmed |
| Capping a workspace | Per workspace, a monthly spend limit and rate limits on requests per minute, input tokens per minute and output tokens per minute. Workspace limits may be set lower than the organisation's but not higher, and the Default Workspace cannot take limits | confirmed |
| Usage and cost reporting without a gateway | GET /v1/organizations/usage_report/messages, bucketed at 1 minute, 1 hour or 1 day, grouped or filtered by api_key_id, workspace_id, model, service_tier, context_window, inference_geo and speed. GET /v1/organizations/cost_report, daily buckets only, in USD, grouped by workspace_id or description. Both need an Admin API key. Console Usage and Cost pages exist. Data "typically appears within 5 minutes" | confirmed |
| A gap in the cost report | "Priority Tier costs use a different billing model and are not included in the cost endpoint" | confirmed |
| Per-request accounting | The response usage object carries input_tokens, cache_creation_input_tokens, cache_read_input_tokens, output_tokens and service_tier | confirmed |
| Routing or fallback between its own models | One narrow case. Set fallbacks to "default" (beta header server-side-fallback-2026-07-01) and a request declined by a safety classifier is retried on the model Anthropic recommends for that refusal category. Only a refusal triggers it: a rate limit, an overload or a server error comes back as-is. Not available on Bedrock, Vertex, Microsoft Foundry or the Message Batches API | confirmed |
| A general model router | Not found in the documentation read. service_tier: "auto" falls back on capacity, from Priority to Standard, and does not change model | confirmed |
Their vocabulary
| Standard term | Their term |
|---|---|
| Virtual key, per-team credential | Workspace-scoped API key |
| Spend cap | Spend limit, per workspace or per usage tier |
| Rate limiting | RPM, ITPM and OTPM limits, per model |
| Fallback chain | fallbacks, refusal-triggered only |
Where to look
Console Usage and Cost for the picture, the Admin API usage and cost reports for anything you want to attribute per team, and the usage object on each response for anything you want to attribute per request. Workspace settings for the caps.
Last verified: 2026-09-09 against https://platform.claude.com/docs/en/api/openai-sdk, https://platform.claude.com/docs/en/api/rate-limits, https://platform.claude.com/docs/en/docs/build-with-claude/batch-processing, https://platform.claude.com/docs/en/api/service-tiers, https://platform.claude.com/docs/en/manage-claude/workspaces, https://platform.claude.com/docs/en/manage-claude/usage-cost-api and https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback.
What this maps to: OpenAI's API is the surface every other vendor's compatibility layer imitates, so it is the one a gateway fronts with no translation. The scoping unit is the project, and rate and spend limits attach to it.
| Question | Answer | Status |
|---|---|---|
| OpenAI-compatible surface | It is the reference. The quickstart shows curl "https://api.openai.com/v1/responses" with Authorization: Bearer $OPENAI_API_KEY; /v1/chat/completions is the older surface other vendors emulate | confirmed |
| Batch tier | The Batch API, endpoint /v1/batches, "50% cost discount compared to synchronous APIs", "Each batch completes within 24 hours (and often more quickly)", a separate rate limit pool, and up to 50,000 requests in one .jsonl batch | confirmed |
| A lower-priority synchronous tier | Flex processing, service_tier: "flex", "lower costs … in exchange for slower response times and occasional resource unavailability", with tokens "priced at Batch API rates". Documented as in beta with limited model availability | confirmed |
| Priority tier | Priority processing, renamed Fast mode on 2026-07-30. service_tier: "priority" or "fast", settable per request or as a project service tier default, charging a per-token premium, sharing its rate limit with Standard | unconfirmed; read via search result content rather than a rendered page today |
| Rate limit units | RPM (requests per minute), RPD (requests per day), TPM (tokens per minute), TPD (tokens per day), IPM (images per minute), and audio minutes per minute | confirmed |
| Scope of a rate limit | "Rate limits are defined at the organization level and at the project level, not user level", and vary by model. Project headroom is readable from the x-ratelimit-limit-project-tokens and x-ratelimit-remaining-project-tokens headers | confirmed |
| Usage tiers | Free, then Tier 1 to Tier 5, graduating by cumulative spend | confirmed |
| Scoping and capping a key | Separate projects for staging and production are the documented pattern, with "custom rate and spend limits per project". Project-scoped keys, service accounts scoped only to a project, and API key scopes that cannot grant permission outside the assigned project role are documented | unconfirmed; the project API key and RBAC pages returned HTTP 403 today, so this rests on search result content |
| Usage and cost reporting without a gateway | https://api.openai.com/v1/organization/usage/completions and https://api.openai.com/v1/organization/costs, with an Admin API key, taking start_time, bucket_width, group_by and limit. The costs endpoint takes 1d buckets only and groups by project_id, line_item or api_key_id. Responses carry token counts and a service_tier field | confirmed |
| What the console dashboards show, and whether there is a CSV export | The usage and cost help pages returned HTTP 403 today | unconfirmed; open question |
| Routing or fallback between its own models | Not found in the documentation read. The model selection guide gives manual advice about balancing accuracy, latency and cost, with no automatic router, no auto model id and no cross-model fallback on the API. The automatic routing that turns up in search is described as ChatGPT product behaviour rather than an API feature | confirmed |
Whether service_tier: "auto" is a documented enum value on the API | The flex page mentions auto as an alternative when retrying, and no canonical enum list was found today | unconfirmed; open question |
Their vocabulary
| Standard term | Their term |
|---|---|
| Virtual key, per-team credential | Project-scoped API key, or a service account key |
| Spend cap | Project spend limit |
| Rate limiting | RPM, TPM, RPD, TPD, IPM, per project and per organisation |
| Batch tier | Batch API; Flex processing for the synchronous version |
Where to look
The Usage and Costs endpoints under /v1/organization, grouped by project_id, if the projects are laid out per team. The rate limit response headers for the project's remaining headroom, which is the number to alarm on before a nightly job discovers it the hard way.
Last verified: 2026-09-09 against https://developers.openai.com/api/docs/quickstart, https://developers.openai.com/api/docs/guides/batch, https://developers.openai.com/api/docs/guides/flex-processing, https://developers.openai.com/api/docs/guides/rate-limits, https://developers.openai.com/api/docs/guides/production-best-practices, https://developers.openai.com/api/docs/guides/model-selection and https://developers.openai.com/cookbook/examples/completions_usage_api.
What this maps to: Two surfaces with different answers. The Gemini API on ai.google.dev is the direct one, with tiers set per request and quota attached to a Google Cloud project. Vertex AI is the enterprise one, and it is the only place in this comparison that sells reserved capacity by subscription. Key restriction and billing reporting come from Google Cloud rather than from the model API.
| Question | Answer | Status |
|---|---|---|
| OpenAI-compatible surface | Yes, and it is documented as a first-class path: "Gemini models are accessible using the OpenAI libraries (Python and TypeScript / Javascript) along with the REST API, by updating three lines of code and using your Gemini API key". Base URL https://generativelanguage.googleapis.com/v1beta/openai/ | confirmed |
| Batch tier | The Gemini Batch API, "designed to process large volumes of requests asynchronously at 50% of the standard cost", with a 24-hour target turnaround | confirmed |
| A lower-priority synchronous tier | The Gemini Flex API, service_tier='flex', "a 50% cost reduction compared to standard rates, in exchange for variable latency and best-effort availability" | confirmed |
| Priority tier | Priority inference, service_tier='priority', "priced at 75-100% more than the standard API". Priority holds its own rate limits, documented as "0.3x the standard rate limit for each model and tier". The tier that answered is reported in the x-gemini-service-tier response header | confirmed |
| What happens when priority is congested | "If Priority limits are exceeded due to congestion, overflow requests are automatically and gracefully downgraded to Standard processing instead of failing with a 503 or 429 error… Downgraded requests are billed at the standard rate." This is a fallback in tier, not in model | confirmed |
| Committed throughput | Vertex AI sells Provisioned Throughput, a fixed-cost fixed-term reservation measured in Generative AI Scale Units with a per-model burndown rate, alongside Standard, Priority and Flex pay-as-you-go | unconfirmed; the provisioned throughput pages returned navigation only today, so the tier names come from that navigation and the GSU wording from search result content |
| Spillover from Provisioned Throughput to pay-as-you-go | Not read today | unconfirmed; open question |
| Rate limit units | Requests per minute (RPM), tokens per minute for input (TPM), and requests per day (RPD), plus images per minute for image models and TPD on some models. "Your usage is evaluated against each limit, and exceeding any of them will trigger a rate limit error" | confirmed |
| Scope of a rate limit | "Rate limits are applied per project, not per API key", and vary by model. Tiers are Free, Tier 1, Tier 2 and Tier 3 | confirmed |
| Scoping or restricting a key | Through Google Cloud API key restrictions rather than a workspace concept. Two categories: API restrictions, limiting a key to a set of APIs, and application restrictions by HTTP referrer, IP address, Android package name with SHA-1 fingerprint, or iOS bundle ID. The documentation recommends setting both | confirmed |
| Key types on the Gemini API | Standard API keys and Authorization keys, the latter bound to service accounts with granular access control. "The Gemini API will reject requests from Standard keys" after September 2026, and unrestricted dormant keys are blocked from May 7, 2026 | confirmed |
| The practical capping unit | The Cloud project, since quota and rate limits attach to the project rather than to the key | confirmed |
| Per-request accounting | The usage object carries total_input_tokens, total_output_tokens, total_thought_tokens, total_cached_tokens, total_tool_use_tokens and total_tokens | confirmed |
| Cost reporting without a gateway | Cloud Billing export to BigQuery, in five flavours: standard usage cost, detailed usage cost at resource level, pricing data, FOCUS usage cost in preview, and CUD metadata in preview | confirmed |
| A dedicated usage or cost REST API for the Gemini API itself | No such endpoint was found in the documentation read. AI Studio shows usage against rate limits | unconfirmed; open question |
| Routing between its own models | Vertex AI Model Optimizer is described as a meta-endpoint that picks a Gemini model, configured with ModelSelectionConfig and a FeatureSelectionPreference of PRIORITIZE_QUALITY, BALANCED or PRIORITIZE_COST, text only | unconfirmed; the Model Optimizer pages returned navigation only today. Open question: is it still current under the Gemini Enterprise Agent Platform renaming, and what is the model id |
| Routing on the Gemini API itself | No cross-model router and no auto model id was found in the documentation read | confirmed |
Their vocabulary
| Standard term | Their term |
|---|---|
| Virtual key, per-team credential | An API key in a Cloud project, with API and application restrictions; or an Authorization key on a service account |
| Spend cap | Cloud Billing budgets and project quota, not a setting on the model API |
| Rate limiting | RPM, TPM (input) and RPD, per project and per model |
| Committed throughput | Provisioned Throughput, in GSUs |
Where to look
Cloud Billing export to BigQuery for anything about money, since the model API does not report it. The AI Studio quota view and the x-gemini-service-tier header for which tier actually served a request. IAM and API key restrictions for who can call what.
Last verified: 2026-09-09 against https://ai.google.dev/gemini-api/docs/openai, https://ai.google.dev/gemini-api/docs/rate-limits, https://ai.google.dev/gemini-api/docs/batch-api, https://ai.google.dev/gemini-api/docs/flex-inference, https://ai.google.dev/gemini-api/docs/priority-inference, https://ai.google.dev/gemini-api/docs/quota, https://ai.google.dev/gemini-api/docs/api-key, https://ai.google.dev/gemini-api/docs/tokens, https://docs.cloud.google.com/docs/authentication/api-keys and https://docs.cloud.google.com/billing/docs/how-to/export-data-bigquery.