Quick answer
You run local AI by installing a program for running models (for example Ollama or LM Studio), downloading a model suited to your amount of RAM and running it in that program or through its interface. Everything stays on the computer, but speed and quality depend on the processor, graphics and memory, so a smaller model is best to start with.
How to recognize the problem
- You want to use AI but are not allowed to send data to the cloud
- The AI tool in the browser does not work or is unavailable
- The company requires document processing without sending to external servers
- The computer is slow when a larger model is started
- The model downloads but does not give answers or crashes
Most common causes
Not enough RAM or VRAM
Models are loaded into working memory, and larger models require several tens of GB. If there is not enough memory, the program switches to disk and operation becomes very slow.
Wrong model choice
A model with more parameters gives better answers, but is often too heavy for a home computer. Smaller models run faster with slightly lower quality.
No graphics acceleration support
Without the appropriate driver or with unsupported graphics, the model runs only on the processor, which is significantly slower.
Not enough disk space
Each model takes up several GB, and programs also store temporary files. Without free space, downloading or operation is interrupted.
Network or download interruption
The first model download is large, so a weak connection or interruption leads to an incomplete file and an error when starting.
Step-by-step fix
- 1
Check your hardware
Open the page with basic system information: on Windows
Settings > System > About, on macOSApple menu > About This Mac, and on Linux check system settings or runfree -hin the terminal. Look at the amount of RAM and the processor type. For a smaller model of about 3 to 4 billion parameters, 8 GB of RAM is recommended, and for models of 7 to 8 billion parameters, 16 GB. Also check free disk space, at least 10 to 20 GB. - 2
Choose a program for local operation
The simplest path is programs that download and run models themselves, such as Ollama (works in the command line) or LM Studio (has a graphical interface). Install one of them from the official site, following the instructions for your system. Do not download installers from unknown sites or links from emails.
- 3
Download a smaller model
In LM Studio, open the model search tab, type the model family name and choose a version with a size label that matches your RAM. In Ollama, you do this with a command in the terminal, for example
ollama pullfollowed by the model name. Start with a smaller model, then try a larger one later. - 4
Run the model and ask your first question
In LM Studio, select the downloaded model and click the load button, then type a short question in the chat field. In Ollama, run the model with the
ollama runcommand and the model name, then type the question on the same line. The first answer may take longer because the model is loading into memory. - 5
Check that it works without the internet
Turn off Wi-Fi or unplug the network cable and repeat the question. If the answer still arrives, the model is running locally and data is not leaving the computer. If the program asks for a connection, check whether you accidentally selected the cloud operation option.
- 6
Set up graphics acceleration
In LM Studio, in the model settings, look for the option for the number of layers sent to the graphics and increase it gradually until you get faster operation without errors. On Windows and macOS, first update the graphics driver or the system. If the computer starts crashing or freezing, reduce the number of layers.
- 7
Adjust the answer length
Long answers take time and memory. In the chat settings, reduce the maximum number of tokens and set the temperature to a moderate value. For summaries and text rewriting, ask for shorter answers and ask specific questions.
- 8
Save your settings and make a backup
Write down which model and settings you are using so you can easily restore them. If the program saves conversations in a local folder, occasionally make a copy of that folder. You can download models again, but not settings and conversation history.
When a local model is not enough
Local models are good for rewriting, summarizing shorter texts, translation and working with sensitive data that must not leave the computer. For complex analyses, long documents and the latest information, they are often weaker than cloud tools, because they do not know about events after the training moment and have no internet access. If you need better quality, first try a larger model on a computer with more RAM, then consider a dedicated server in the company. For business use, define in advance who may run models, where files are stored and what data you may enter, even when everything stays local.
How to prevent it
- Download programs and models only from official sites and official sources
- Before starting a larger model, check free RAM and disk space
- Regularly update graphics drivers and the local operation program itself
- Keep a copy of settings and conversation history that is important to you
When to call a professional
Contact IT support if the model cannot be loaded even after reducing the size, if the computer crashes when working with graphics, or if the company needs to establish rules for local processing of sensitive data.
AI automation · AI agents that process documents, emails and dataFAQ
Does local AI really work without the internet?
Yes, once the model is downloaded and loaded, answers are processed on your computer. The internet is needed only for the first download of the program and model, and for later updates.
How much RAM do I need for a local model?
For a smaller model of about 3 to 4 billion parameters, 8 GB of RAM is enough, and for models of 7 to 8 billion parameters, 16 GB is recommended. Larger models require significantly more memory and stronger graphics.
Is my data safe if I use a local model?
Data stays on the computer and is not sent to a provider, but that does not mean it is automatically protected. Access to the computer, file copies and any sharing of results are still subject to company rules.