Inference
In shortInference is the act of running a trained AI model to produce an output from your input, such as an answer to a prompt.
In plain words
An AI model lives in two phases. In training, the provider feeds it huge amounts of data and adjusts its parameters. This happens once, takes large computing power, and ends with a fixed set of weights. In inference, the model uses those fixed weights to answer a request. Every time you send a prompt, you trigger inference. Nothing is learned from your request in that moment.
Inference runs token by token: the model predicts one token, adds it to the text, and predicts the next. This is why a long answer takes longer than a short one. Settings such as temperature shape how each token is picked.
Why it matters
- You pay for inference. API prices, quoted per million tokens, cover the compute used to run the model for you.
- Speed has two parts. Anthropic's glossary names two measures: latency, the delay between sending a prompt and receiving output, and time to first token (TTFT), the wait before the first piece of the answer appears. Model size, hardware, network conditions and the size of your prompt all affect them.
- Where it runs matters. In the cloud, the provider's servers run inference and your prompt travels there. With a local model, your own machine does the work, and your hardware sets the limits.
- Not fully repeatable. Anthropic notes that even with temperature set to 0, results are not fully deterministic across calls.
Example
You paste a 500-word text and ask for a summary. The provider's servers read your text and then generate the summary token by token. You are billed for the tokens you sent and the tokens the model wrote. The model itself is unchanged afterwards. A second request starts from the same fixed weights.
Common misconceptions
- "The model learns from my chat as I type." During inference the weights stay fixed. Whether a provider uses your data later for training depends on its terms, which you should read.
- "Inference is cheap, training is the only cost." For a popular service, serving millions of requests is a major, ongoing cost.
- "Inference needs the cloud." Smaller models can run on a laptop.
Questions and answers
What is the difference between training and inference?
Training builds the model once from large data sets. Inference uses the finished model to answer each request.
Why is AI inference slow or costly?
The model produces one token at a time and each step needs a lot of computing. Longer prompts and longer answers add time and cost.
Can I run inference on my own computer?
Yes, for models whose weights you can download and that fit your hardware. See local models.
Read next
- Token
A token is a small chunk of text, often a word piece, that an AI model reads, writes and bills.
- Price per million tokens
Price per million tokens is the usual way AI APIs state their rates: what you pay for each million tokens sent in or generated.
- Local model
A local model is an AI model that runs on your own computer or server instead of a provider's cloud, so prompts need not leave your device.
- Parameters
Parameters are the numbers, also called weights, that a model adjusts during training and then uses to answer; their count is a rough measure of size.
- Temperature
Temperature is a setting that controls how random a model's word choices are: low gives steady answers, high gives more varied ones.