Skip to main content

Local and cloud inference

Downloading the weights is the easy part. Getting them to answer a question at a speed anyone will tolerate is where the decision gets made.

What running it yourself requires​

means the weights are on hardware you control and no request leaves it. The first constraint is memory. All of the weights have to be resident before a single token comes out, and they have to be resident in the memory attached to whatever does the arithmetic. A CPU will do it out of system RAM and you will wait a long time. A GPU does it out of its own memory, which is much faster and much smaller.

So the first calculation anybody does is parameters times bytes per parameter. A model trained and released in 16-bit precision uses two bytes per parameter, so a 70B model needs about 140 GB just for the weights. Say the card you have has 80 GB of memory on it. That model does not fit on one of those, and you are now buying two or four and dealing with the fact that they have to talk to each other.

Quantization, and what it costs​

is the lever on that calculation. It stores each weight in fewer bits: 8-bit instead of 16, or 4-bit, or lower. The arithmetic follows directly. That 70B model is roughly 70 GB at 8-bit and roughly 35 GB at 4-bit, and the 4-bit version fits on one card a mid-sized company can afford. Smaller weights also move faster from memory into the compute units, and since generating a token is mostly a memory-bandwidth problem rather than a compute problem, quantization usually makes the model faster as well as smaller.

What it costs is accuracy. Each weight is being rounded, and the rounding shows up as degradation that is mild at 8-bit, noticeable at 4-bit, and severe below that. The damage is uneven, which is the annoying part. A heavily quantized model will often chat perfectly well and then fall apart on the things you are relying on it for: long chains of reasoning, exact code, careful instruction following. A benchmark run at 4-bit can look almost unchanged while the model has got measurably worse at your task.

That sets up a trade a practitioner has to make on purpose. Quantizing hard enough brings a 70B model down into roughly the memory a much smaller model at full precision would ask for, so on one card the choice is between the large model with fewer bits per weight and the small one with all of them. Which is better depends on the work, and the only way to know is to run your own evaluation set against both. The general shape is that the larger quantized model wins on breadth of knowledge and the smaller full-precision one wins on precision and consistency, but treat that as a hypothesis to test rather than a rule.

The context window is not free either​

Weights are the fixed cost. On top of them, every request in flight holds a cache of intermediate state for the tokens it has already processed, and that cache grows with the conversation. A short prompt costs almost nothing. A long document, or a coding session forty turns deep, costs real memory. That cost repeats for every concurrent user.

So a machine that loads a model happily can fall over the first time ten people use it with long contexts. The weights fit, the weights plus the cache do not, and the failure looks like the server refusing requests rather than like anything to do with memory sizing. When you plan capacity, budget for the weights and the working memory of the traffic you expect at the same time.

What an inference server does​

Loading a file and running one request at a time is a demo. An is the software that makes it a service. It holds the weights resident, exposes an HTTP API, manages the caches described above, and above all it batches.

Batching is the part that decides what the hardware costs you. Generating one token means moving the entire weights file from memory through the compute units. Doing that for one user wastes almost all of the machine, because the arithmetic finishes long before the next chunk of weights arrives. Do it for thirty users at once and the same pass over the weights serves all thirty. The hardware cost per token drops by something close to that factor. is the refinement that made this practical: rather than waiting for a batch to fill and then running it to completion, the server adds new requests to the running batch as slots free up, so a request that arrives mid-flight does not wait for the previous batch to finish.

The consequence is the one people find counter-intuitive. A single-user local setup has terrible cost per token even though the electricity bill is small, because a GPU serving one person is running at a few percent of what it could do. The idle hardware is the problem. A hosted API is cheap per token partly because the vendor has thousands of requests to pack onto every card.

Speed, and which number you mean​

is the throughput measure, and it means two different things depending on who is asking.

Per-request throughput is what one user feels: how fast text appears once it starts. People read at something like ten tokens per second, so a model producing thirty or forty feels immediate and one producing five feels broken. Aggregate throughput is what the machine does across every request at once, and with a full batch it is many times the per-request figure. A serving setup tuned for aggregate throughput will often make each individual request slightly slower, because bigger batches help the total and hurt the individual. Which one you optimise for is a product decision. An interactive assistant needs per-request speed. An overnight job that classifies two million documents needs aggregate, and does not care if any single one takes a while.

Then there is the . Loading a 40 GB file off disk into GPU memory takes tens of seconds at best, and until it finishes the model answers nothing. So either you keep the model resident and pay for hardware that is idle between requests, or you let it unload and make somebody wait. There is no third option, so "just spin it up when we need it" does not survive contact with a real workload.

What sits on each side of the boundary A heavy vertical boundary line runs down the middle of the drawing, labelled "the boundary: everything to the right is somebody else's machine". On the left, under the heading "Local: hardware you control", four stacked boxes hold GPU hardware you bought or rent, the weights file on disk at roughly 40 gigabytes for a 70B model quantized to 4-bit, the inference server that keeps the weights resident and batches requests, and your documents and prompts. On the right, under the heading "Cloud: the vendor's machine", four stacked boxes hold the vendor's API endpoint, the vendor's models including the frontier ones, metering and rate limits billed per token, and your prompt sitting on their hardware. Below the boxes two tracks run left to right. The upper track is thick and solid, labelled "run locally", and stops at the boundary against a bar marked "the data stops here". The lower track is dashed with an arrowhead, labelled "call the API", and crosses the boundary to the far right, marked "the data crosses". Local: hardware you control Cloud: the vendor's machine The boundary. Everything to the right is somebody else's machine. GPU hardware you bought or rent The weights file, on disk An inference server Your documents and prompts Idle between requests, and still paid for About 40 GB for a 70B model at 4-bit Keeps weights resident, batches requests Never leave the building on this side The vendor's API endpoint The vendor's models Metering and rate limits Your prompt, on their hardware One address, no drivers, no cards to replace Including the frontier ones, which are only here Billed per token, capped per minute Retained or not, according to their terms Run locally prompt and context the data stops here Call the API the same data crosses
Everything else on this page is a cost question. Whether the data crosses that line is usually not.

Which one, then​

Local wins on four things.

The data never leaves. Where a data residency law applies, as it does to a hospital or a defence contractor, that is a requirement rather than a preference to be traded off against cost, and it settles the question before anything else gets considered.

Nothing bills per token. A steady high-volume workload can be dramatically cheaper once the hardware is paid for, because the marginal cost of the ten millionth token is electricity.

No rate limit applies. No vendor decides that the model your product is built on retires in ninety days.

It works with no network. That matters on a ship or in a factory, and in any building where the link is the thing that fails.

The cloud side of the list looks nothing like it. The frontier models are only there, so if your task needs the best available reasoning, local is not on the menu at any price. Capacity is elastic, so a workload that is quiet all week and enormous on Monday morning pays for Monday morning rather than for hardware sized to it. There is nothing to operate: no drivers, no toolkit versions, no card that fails at the weekend, no upgrade to the serving stack. And far less engineering effort stands between a team and a working feature, which is what settles it for most of them.

The cost comparison is where people go wrong. The comparison is total cost against total cost, not the vendor's price per token against your electricity per token, and the local side of it includes the GPUs amortised over the two or three years before they are obsolete, the power and the cooling, the rack or the cloud instance they sit in, the engineer who keeps the serving stack alive, the evaluation work to find out which quantized model is good enough, and the value of whatever quality gap remains. Nobody prices that last item, and it is often the largest. Local becomes cheap at high, steady volume on a task a mid-sized model handles well. At low volume, or on a task that needs the best model available, it costs more than the API and the arithmetic is not close.

Most organisations land on a mix. The usual split puts bulk work on local models: classification, extraction, summarising, embedding, redaction, anything high-volume and well-defined where a good mid-sized model is sufficient and the volume makes per-token pricing hurt. It puts the hard reasoning, the customer-facing product, and anything a harness drives through many turns on a hosted frontier model. Regulated data goes local by default whatever the task. An LLM gateway in front of both is what makes the split manageable, because the calling code then names a capability rather than a machine.

Deciding where a workload runs

Start from the constraint that cannot move.

  1. Check whether the data may leave your control. Run the model locally if it may not.
  2. Check whether the task needs frontier quality. Call a hosted API if it does.
  3. Measure the tokens per day from a real week of traffic. Do not estimate.
  4. Check whether the load is steady or spiky. Buy hardware for steady load only.
  5. Cost the local option in full. Include hardware, power, staff, and the quality gap.
  6. Evaluate a quantized model on your own task before you buy any hardware.
  7. Put a gateway in front of both. Name a capability in your code, not a machine.

Terms introduced​

  • Local inference: running a model on hardware you control, so no request leaves your network.
  • Quantization: storing each weight in fewer bits to cut memory and raise speed, at some cost in quality.
  • Inference server: the software that keeps weights resident, serves an API, and batches requests together.
  • Tokens per second: throughput, meaning either what one request feels like or what the machine does in total.
  • Continuous batching: adding new requests to a batch already running, so a card is never generating for one user.
  • Cold start: the delay while weights are loaded into memory before the model can answer anything.

How providers do it

One of these three publishes a file you can run. The other two answer the question in different ways: by moving the hosting into your cloud account, or by being the runtime.

ProviderCan it run outside the vendor's service?Option short of full localStatus
AnthropicNo weights. Hosted onlyBedrock, Google Cloud, Microsoft Foundry or Claude Platform on AWS, with regional endpoints for data routingconfirmed
MetaYes. Weights download from Meta, Hugging Face or KaggleManaged hosting of Llama on a cloud catalogueconfirmed for download; unconfirmed for which clouds list which generation
OllamaIt is the local runtime. Serves models on localhost:11434Ollama Cloud offloads a larger model while the local tooling stays the sameconfirmed

Still open: which Llama generations each managed cloud currently lists, and what concurrency Ollama's built-in server handles.

The middle column is the one most teams should read first. "Local or cloud" is usually presented as a choice about hardware, and for a regulated workload it is really a choice about where the data goes. A hosted model running on a regional endpoint inside your own cloud account answers that question without anybody buying a GPU.

What this maps to: Claude cannot be run on hardware you own. What Anthropic sells instead is a set of hosted deployments, some of which sit inside a cloud account you already control, which covers a lot of what teams want local inference for.

QuestionAnswerStatus
Can the models run outside Anthropic's own service?Yes, but only on another vendor's managed service. There is no downloadable artefact and no self-hosted optionconfirmed
Where, exactly?Claude API, Amazon Bedrock, Google Cloud (Vertex AI), Microsoft Foundry, and Claude Platform on AWS. Each has its own model IDs, listed per model on the models overviewconfirmed
What is the option short of local?Running inference through Bedrock, Vertex AI or Foundry, so the calls, the billing and the identity sit inside your existing cloud account rather than in a separate vendor relationshipconfirmed
Data residencyBedrock offers global endpoints with dynamic routing and regional endpoints with guaranteed data routing, for Claude Sonnet 4.5 and later. Google Cloud offers global, multi-region and regional endpointsconfirmed
Who controls model retirement?Anthropic publishes retirement commitments for Anthropic-operated platforms, being the Claude API, Claude Platform on AWS and Microsoft Foundry. Amazon Bedrock and Google Cloud set their own lifecycle datesconfirmed
Cold starts, batching, tokens per secondNot exposed. These are the vendor's problem, which is the trade you are makingunconfirmed; no published figures were read today
Cheaper modes for non-interactive workThe Message Batches API is priced at 50% off the base rate, and prompt cache reads cost 10% of base inputconfirmed

Their vocabulary

Standard termTheir term
Running inside your own cloud accountClaude on Amazon Bedrock, Claude on Google Cloud, Claude in Microsoft Foundry, Claude Platform on AWS
Aggregate throughput workMessage Batches API
Cold startNo equivalent. Capacity is the vendor's concern

Where to look

The models overview lists a model ID column per platform, so it doubles as the availability matrix. If the constraint driving you towards local inference is a data residency rule rather than a cost model, the regional endpoint rows on Bedrock and Google Cloud are the thing to read first.

Last verified: 2026-09-09 against https://platform.claude.com/docs/en/about-claude/models/overview.

Check your understanding

0 of 4 answered

  1. A 70B model quantized to 4-bit and a 13B model at full 16-bit precision take about the same GPU memory. How should a team choose between them?
  2. What is the first calculation to do before running a model on your own hardware?
  3. A team plans to save money by unloading its local model when nobody is using it and loading it again on demand. What goes wrong?
  4. A team compares the vendor's price per million tokens against the electricity cost of its own GPUs and concludes local is ten times cheaper. What is wrong with the comparison?