Why graphics card memory matters
In practice a language model is a huge set of numbers (parameters). For it to answer quickly, all of them have to fit in the graphics card's memory. When they don't, part goes to ordinary RAM and the model slows down several to a dozen times. That's why the first question when choosing hardware is: how much VRAM.
The exception is Apple computers with M-series processors, which have memory shared between the processor and the graphics. On them a larger model can run at a reasonable pace, though more slowly than on a powerful card.
Quantisation, or how to fit a bigger model
A model's parameters are normally stored in 16 bits. Quantisation stores them less precisely, for example in 8 or 4 bits. The model becomes two or four times smaller, and quality usually drops only a little, though it can be noticeable on hard tasks and with smaller models.
| Model size | 4 bits (Q4) | 8 bits (Q8) | 16 bits |
|---|---|---|---|
| 3 to 4 billion parameters | approx. 3 GB | approx. 5 GB | approx. 8 GB |
| 7 to 8 billion | approx. 5 to 6 GB | approx. 9 to 10 GB | approx. 16 GB |
| 13 to 14 billion | approx. 9 to 10 GB | approx. 15 to 16 GB | approx. 28 GB |
| 30 to 32 billion | approx. 20 GB | approx. 35 GB | approx. 65 GB |
| 70 billion | approx. 40 to 45 GB | approx. 75 GB | approx. 140 GB |
These are approximations for the model weights alone, with a small margin. On top of that comes memory for the conversation context, described below.
Memory calculator
The formula the calculator uses: the weights are the number of parameters times the bits divided by eight, plus about 10 percent. The context is the conversation cache, which grows with the context length, the size of the model and the number of simultaneous conversations. Plus about 1 GB for the runtime itself. The result is an estimate: specific models differ in their architecture.
Which programs to run the model with
- Ollama
The simplest start: one command downloads and runs a model. Good for testing and small deployments.
- llama.cpp
The engine many tools are built on. Runs on graphics cards, processors and Apple computers. Lots of quantisation options.
- vLLM
A server built for performance with many simultaneous users. Needs a graphics card and more configuration.
- Open WebUI
A browser interface similar to well-known chats, with user accounts, to connect to Ollama or another server.
What you can realistically run in a small company
- A workstation with one 12 to 16 GB card
7 to 14 billion parameter models in 4-bit quantisation. Enough for working with documents, summaries and draft replies for a few people.
- A server with a 24 GB card
Models up to about 32 billion parameters at 4 bits, or smaller models with a longer context and more users.
- Several cards or a 48 GB card and up
Models of around 70 billion parameters. That's an investment that makes sense with heavy, steady use.
Before buying hardware, test the model on your real tasks. A GPU server rented for a few hours costs little and lets you check whether a smaller model is enough.
What to watch out for
- Model licences differ: some allow unrestricted commercial use, some have conditions. Check the licence before deployment.
- Languages: not every model writes well in every language. Test on documents in the language your company works in.
- Response speed depends on the card and the length of the answer. Several people at once on one card means a queue.
- A model server is still a server: updates, configuration backups, access only from the company network.
Terms from this guide
- Local model
A language model running on the company's own hardware rather than in a provider's cloud. The data doesn't have to leave the company.
- VRAM
The graphics card's memory. It decides how large a language model can run locally and how fast it will be.
- Quantisation
Storing a model's parameters with less precision, for example in 4 bits instead of 16. The model takes several times less memory with a small loss of quality.
Questions and answers
Can a local model run without a graphics card?
Yes, on the processor alone, but much more slowly. For testing and occasional use it may be enough; for several people's everyday work, probably not.
Is a local model as good as the largest cloud models?
For complex tasks, usually not. For well-defined tasks on your own documents the difference can be small, especially when the model gets the right passages of the documents.
How many people can one card serve?
It depends on the model, the length of the answers and how often people ask. A few people working in parallel is a reasonable assumption for a single card and a mid-sized model, but it's worth measuring on your own case.
Sources and documentation
- llama.cpp: running quantised modelsgithub.com
- Hugging Face: the GGUF format and quantisation levelshuggingface.co
- Ollama: the model library with sizesollama.com
- vLLM: documentationdocs.vllm.ai