Ollama makes local AI easy. The next problem is using it well.

Local models are no longer a weird hobby

Since it's common to find LLM models from companies like Anthropic or OpenAI to be a kind of sorcery, it may be easy to think that using local models is complex and hard to use.

But contrary to what you may think, with the advent of tools like Ollama, spinning up local models has become really easy and doable by everyone.

What is Ollama?

Ollama is an application that runs on your computer and lets you download and interact with open models.

Basically, imagine that instead of going to the Anthropic website and having a conversation with Claude, you do the same by opening a terminal on your computer, and interact with a model called qwen instead.

The task Ollama performs is to download and manage the models for you, and, in particular, to provide you with a channel to communicate with them. This channel can be a simple prompt in your terminal, in the Claude Code application, or in any application via its HTTP REST API.

How to set up Ollama on your computer

Ollama can be installed like any other application from their website.

Once installed, you can set up the directory where models will be stored on your computer and the context's maximum size from the desktop application.

Once started, you can install any local model by running on your terminal:

ollama pull <model_identifier>

And then you can start a chat with it by running:

ollama run <model_identifier>

The default prompt ollama provides works, but it's extremely rudimental if you come from tools such as Claude Code or Codex.

For this reason, Ollama also allows you to use Claude Code or Codex features with Ollama models. To do so, it's enough to run

ollama launch claude
# or
ollama launch codex

And once started, you can select the local model to use, like you would do to switch from Claude Sonnet or Claude Opus.

Once a model is selected, you can do whatever you would normally do in a Claude Code session.

Choosing a model for Ollama

Choosing a model for Ollama implies an understanding of some basic aspects of LLMs.

First, the full list of available models for Ollama can be found on their website.

But once there, what to choose? There are different aspects to consider, which are mainly:

• What do you want to do with the model?
• How much RAM does your computer have?

Some models are good for normal chatting, others for code completion, others for designing software architectures. A good example is:

• General chat and light tasks: llama3.2, gemma3, or qwen
• Coding tasks: qwen2.5-coder, qwen3-coder, or devstral
• Heavier reasoning or agentic coding: larger coding models, when your hardware can handle them

The second aspect, instead, is RAM. LLMs crave memory. They need a lot of it to run, and of course, the more sophisticated and “clever” the model is, the more RAM it consumes.

RAM usage mostly depends on the model size, the quantisation used, and the context length. If you go to the Ollama model catalogue, you'll notice that models have different tags like 24b, 7b, or even 122b. What does that mean?

The b stands for billion of parameters. But what are parameters?

Parameters are, in very simple terms, the internal numbers the model learned during training.

Think of them as billions of tiny knobs. Each knob slightly influences how the model processes your prompt and decides what to generate next. One knob alone means nothing, but all together define the model's behaviour: how well it understands language, code, instructions, facts, and patterns.

So, a 7b model has roughly 7 billion parameters. A 24b model has around 24 billion, and a 122b model has around 122 billion.

In general, more parameters give the model greater capacity to represent complex patterns, which often leads to better answers. The trade-off is that bigger models require more RAM and are usually slower to run.

Why local models matter

Local models are good and should be taken more into account for different reasons:

• They are actually good for most of the basic tasks. How many times have you asked Claude Opus to just write a commit message? Well, if you asked a local model, you would have saved tokens and probably had a similar outcome.
• Keep your conversation and data private: When you send private code or internal data to a cloud model, you are moving that data outside your own perimeter. Even if the provider has strong privacy, retention, and security policies, you are still adding a third party to your threat mode, and data breach can happen.

The trade-off: local does not mean better

Unfortunately, we can't just eliminate mainstream LLMs by replacing them with local models.

While local models look really good on paper, local models are not as good as those implemented by Anthropic or OpenAI.

This means you should expect a real gap in results when using local models or Claude, for instance, on complex tasks.

Another major issue with local models is hardware. Most of us don't have the financial means to afford computers with 128 GB or 256 GB of RAM for use as local AI workstations at home. This means that most of us can't benefit from powerful local models at the moment, and still have to rely on the company's models for complex tasks.

It's still worth saying that local models for simple tasks may require just 20-30 GB and so can run well on your machine if you have, like, 48 or 64 GB of RAM.

Those who can really benefit now from local and self-hosted LLMs are companies that can run models internally to handle sensitive data, reduce API usage, and control costs.

The real problem is choosing when to use local models

Switching between local and remote models would be tedious for developers, who would likely eventually stick with mainstream models for most of their tasks.

Unfortunately, even great local models haven't solved the hassle of switching between local and remote models.

That's exactly what we are trying to achieve at smista.ai, a smart router that, through fully customisable routing policies, can forward tasks to different models based on the rules you provide. Smista.ai works in harmony with local models, thanks to the possibility to specify local-only routes, which keep paths away from remote LLMs.

Conclusions: Hybrid environments are the future

Ollama makes local models accessible. But accessibility is only the first step.

The real value comes when local models become part of a workflow that knows when local is enough, when cloud is better, and when privacy, cost, or quality should decide the route.

That is the future I believe in: not local-only, not cloud-only, but local-first and intelligently routed.