Local model
In shortA local model is an AI model that runs on your own computer or server instead of a provider's cloud, so prompts need not leave your device.
In plain words
With a cloud model, your prompt travels over the internet to a provider's servers, which run the model. With a local model, you download the model file and run it on your own machine. Projects such as Ollama and llama.cpp, both open source on GitHub, make this practical: they load a model and let you chat with it or call it from your own programs. Local models are usually open-weight models.
The key limit is memory. The file holds billions of parameters, and the model must fit in the RAM or graphics memory of your device. Quantisation helps: it stores each weight with fewer bits, so the file shrinks. As rough arithmetic, a model with 7 billion parameters needs about 14 GB at 16 bits per weight and about 3.5 GB at 4 bits, plus room for the working memory of a long chat. Quality drops somewhat as the bits go down.
Why it matters
- Privacy: Your prompts and files stay on your device unless the app you use sends them elsewhere. Still, check the settings and any plug-ins. Read what never to paste into a cloud chatbot.
- Offline and control: It works without a connection, and no provider changes or retires the model under you.
- Cost: No per-token fee. You pay with hardware, power and set-up time.
- Speed: Inference speed depends on your hardware. A laptop is slower than a data-centre server, and long inputs slow it more.
- Quality: Small local models handle summaries, drafts, sorting and simple questions well, but they may fall short on hard reasoning or on very long documents.
- Upkeep: You choose, download and update models, and you judge where they came from. Take weights only from sources you trust.
Example
You have a laptop with 16 GB of memory. You install Ollama, download a small quantised model, and ask it to summarise confidential meeting notes. Nothing goes to a provider. For a long legal analysis, you switch to a larger cloud model, after you remove names and sensitive details.
Common misconceptions
- "Local is always private." The model runs locally, but an app around it may still send data or telemetry. Check.
- "A local model matches the best cloud models." Often not at the same size. Test on your task.
- "Local is free." The cost moves to hardware and time.
Questions and answers
What is a local AI model?
A model that runs on your own computer or server instead of a provider's cloud, so your prompts need not leave your device.
What hardware do I need for a local model?
Enough memory to hold the model. Small quantised models run on an ordinary laptop; larger ones need much more memory or a strong graphics card.
Is a local model more private than ChatGPT-style cloud services?
It can be, since data stays on your device. It depends on the whole setup, so check what the app and its plug-ins send out.
Read next
- Open-weight model
An open-weight model is an AI model whose trained weights are published for download, so you can run and often adapt it yourself.
- Inference
Inference is the act of running a trained AI model to produce an output from your input, such as an answer to a prompt.
- Parameters
Parameters are the numbers, also called weights, that a model adjusts during training and then uses to answer; their count is a rough measure of size.
- Large language model (LLM)
A large language model (LLM) is an AI system trained on huge amounts of text to predict the next token, which lets it write, summarise and answer.
- What not to paste into AI chatbots
Passwords, client data, IDs and secrets do not belong in a chatbot: a traffic-light checklist, provider data rules, Swiss and EU basics, and safer options.