Can your computer run a local AI model?

How Much VRAM Do You Need for a Local AI Model

The GPU's VRAM is the most important resource because the model needs to fit in it to run fast. A rough estimate for models with 7 to 8 billion parameters in 4-bit quantization is around 5 to 6 GB, for 14B models around 9 to 10 GB, and for 70B around 40 GB. Add a buffer for context and working space on top of that, so in practice always count 1 to 2 GB more than the model weights alone.

When a model does not fit in VRAM, part of it spills over into system RAM. Everything still works then, but noticeably slower, because data is constantly being moved across the PCIe bus. That is why the question "how much VRAM" is always also a question of speed, not just whether it can run at all.

A practical order of checks:

1. Check the exact graphics card model and the amount of VRAM (not just the series name).
2. Check the computer's total RAM.
3. Choose the model size and quantization.
4. Leave a buffer for context and the operating system.
5. Only then assess whether it will run smoothly or with occasional spilling into RAM.

On Mac computers with an Apple Silicon chip there is no separate VRAM, because the CPU and GPU share the same memory. In that case count the total memory, but keep in mind that part of it stays reserved for the system and applications.

What Is Quantization and Which One Should You Choose

Quantization is reducing the precision of the numbers a model is stored with. Instead of 16 bits per parameter, the model is stored with 8, 4 or fewer bits. This drastically reduces memory usage, while answer quality drops only slightly, especially with 4-bit variants.

Example: an 8 billion parameter model in 16-bit format needs around 16 GB, in 8-bit around 8 GB, and in 4-bit around 4 to 5 GB. That is why 4-bit quantization (labels like Q4_K_M) practically makes local AI possible on graphics cards with 8 GB of VRAM.

A tip for beginners: start with 4-bit quantization and check the quality on your own tasks. If the answers are not precise enough and you have the memory to spare, try 5 or 6 bits. For translation, summarization and simpler conversations, 4 bits are usually more than enough.

How Context Affects Memory Usage

Context is the amount of text the model keeps "in its head" during a conversation, measured in tokens. A larger context means the model can process a longer document or a longer conversation, but also that it uses more memory. Memory usage grows with every few thousand tokens of context, so a model that barely fit in VRAM easily crosses the limit when you raise the context.

In practice: if you want the model to read long documents, plan for extra memory or reduce the model size. If you are having short conversations, a smaller context is enough and leaves room for speed.

Most Common Estimation Mistakes

The first mistake is looking only at the graphics card name. Two cards with the same name can have different amounts of VRAM, especially laptop versions. The second is forgetting about system RAM: if the model is larger than VRAM, RAM becomes the bottleneck, and 8 GB total is tight for serious work these days.

The third mistake is expecting everything to work the same as in the cloud. Local models are excellent for privacy and working without internet, but speed depends on hardware. The fourth is neglecting drivers and support: without the correct drivers, the graphics card will not be used for acceleration.

If your estimate shows that your computer cannot run the model you need, the solutions are a smaller model, lower quantization or moving to stronger hardware. When it comes to a business environment with multiple users and documents, EVI Solutions can help with a private AI server that runs entirely locally.