Wiki · Features in depth
Local models
A model running on your own computer: free, nothing leaves the machine, and slower. When it pays off, what it needs and what to watch out for.
When it pays off and when it does not
| Pays off | Does not pay off |
|---|---|
| Data must not leave your computer (contracts, medical notes, internal code) | Complex multi-step work where the agent must use tools reliably |
| Working without internet, or on the move | Long conversations and long documents |
| Repeated simple jobs: summaries, translation, extracting facts, drafts | Speed — when you need the answer right now |
| Trying things out before you start paying | The highest quality on hard reasoning |
The easiest way: inside the app
Open Settings → Providers → Local Models (offered during onboarding as "Run models locally" too).
Click Install runtime. The official runtime for your hardware is downloaded (a few hundred MB), verified and kept up to date.
Pick a model from the catalog and click Download. Before you download anything you can see whether it fits your GPU (green), runs via system memory (amber) or is too big for this machine (red).
Click Use. New chats now run on the local model. Switching back to a cloud provider is one click in the model picker.
Downloads can be paused and resumed — even after closing the app. Large models are tens of gigabytes, so on a slow connection it takes a while.
If you already run Ollama or your own server
When a server is already running (Ollama, your own llama-server, MLX), Hermes finds it and uses it instead of starting its own. You can also point it there by hand:
The managed runtime can be enabled too, with extra ports to probe:
With no server reachable the error names the local runtime ("the local model server isn't running") — no puzzling over a missing API key.
What it needs
| Machine | What gets used |
|---|---|
| NVIDIA (Windows and Linux) | CUDA (on Linux ARM a CUDA 13 driver is required) |
| AMD and Intel | Vulkan; HIP/ROCm on supported targets |
| Mac | Metal |
| No GPU | The CPU — it works, but slowly |
| GPU memory | What fits |
|---|---|
| 8 GB and up | The smaller catalog models comfortably |
| 16 GB and up | 27–35B models at high quality |
How Hermes manages memory
| What it does | What it means for you |
|---|---|
| Starts with a window that fits the GPU and grows it as needed | You may see "context window grown" — that is correct, not an error |
| Every recommended model gets at least a 64K context window | A reasonably long conversation fits |
| Overflow goes to system memory in the least harmful order | Slower answers, but the conversation survives |
| It counts overhead (context, buffers, vision projector) | Less fits in the GPU than the model file's size — Hermes accounts for that |
| When the window cannot grow, the conversation is compressed | Instead of failing silently, older parts get summarised |
More models and your own files
| Where | What is there |
|---|---|
| The app catalog | Curated models with our validation badge |
| Hugging Face search | Far more choice; each result shows whether it fits your GPU, but carries no validation badge |
Your own .gguf file | Can be added to the library without copying — the file stays where it is |
Cost: at home versus in the cloud
| Local | Cloud (model provider) | |
|---|---|---|
| Payment per query | none | by the number of characters |
| Up-front cost | a graphics card and electricity | none |
| Data leaving your machine | none | the text goes to the provider |
| Speed and quality | lower | higher |
| Usage limits | none | set by your plan, occasional slowdowns at peak |
Pitfalls from our systems
| What happened | What to do |
|---|---|
| The agent ran on a server with no graphics card | It runs on the CPU, but so slowly it is unusable. A local model belongs on a machine with a GPU, not on a server |
| A small model could not handle tools | It ignored them or invented the result. For tool work (files, commands, websites) use a larger model or the cloud |
| The model did not fit even though the file was smaller | Context and overhead count too. Go by the colour in the catalog, not the file size |
| Downloading a large model saturated the connection for a day | Use pause and download outside working hours; vision adapters and other files are downloaded as well |
| A longer conversation no longer fit | That is normal — Hermes shortens the conversation rather than crashing. If it bothers you, pick a smaller model or keep long sessions in the cloud |
Our tips
| Tip | Why |
|---|---|
| Start inside the app, not on the command line | You see up front what fits your machine |
| Keep a local model as a fallback | With no internet or no cloud budget the agent still works |
| Local for sensitive things, cloud for hard ones | The combination gets the best of both; switching is one click |
| Check the colour next to a model before downloading | Saves tens of gigabytes downloaded for nothing |