I spent a good twenty minutes looking through the llama.cpp docs for one thing: a way to unload every model at once. A router that can load models on demand has to have a way to kick them all out, right? Turns out it doesn't. Not directly anyway. But what it does have works better than a button ever could.
How I Ended Up Here
I found llama.cpp's router mode about three weeks ago. Before that, switching models meant killing the server, waiting for GPU memory to drain, starting fresh with a different GGUF, and checking nvidia-smi to make sure VRAM actually freed up. Every single time.
Router mode changed that. One server, many models, loaded on demand, no restarts. Turns out this is how everyone else has been running local LLMs all along. I had been killing and restarting like a caveman.
The setup is simple. Instead of --model, you pass --models-dir:
llama-server --models-dir ./models --models-max 4 --port 8080
Drop a handful of GGUFs in the directory. Send a request for qwen3-8b, and the router loads it. Send one for llama-3.2-3b, and it loads that too. Hit --models-max, and the router evicts the least recently used model automatically.
That automatic eviction lulled me into thinking I was safe.
The Moment I Realized I Needed an Unload Button
I was bouncing between models for testing. Qwen for one task. Llama for another. A command-R model I was curious about. A tiny Phi-4 I wanted to benchmark. Before I knew it, four models were loaded and my 16 GB GPU was down to fumes.
I checked VRAM. 14.2 GB used. And I had a larger model I actually wanted to run.
Restarting llama-server would have cleared everything. But I would have lost the cached state, the warm models I still wanted, and a solid thirty seconds of watching the server come back. Not the end of the world. Just annoying.
I wanted one command that said "get everything out of VRAM right now."
The router has a /models/unload endpoint, but it works per model. You need the exact identifier and you call it once for each loaded model. There is no /models/unload-all.
I checked. I grepped the --help output. I looked in the source. Nothing.
The Hack That Works Better Than A Button Ever Would
You do not need an unload-all endpoint. You need to list loaded models, then unload them one at a time. Two operations, and the router already supports both.
The listing endpoint:
curl -s http://localhost:8080/models | jq
Returns something like:
{
"data": [
{ "id": "qwen3-8b", "status": { "value": "loaded" } },
{ "id": "llama-3.2-3b", "status": { "value": "unloaded" } },
{ "id": "phi-4-mini", "status": { "value": "loaded" } }
]
}
The unload call:
curl -s -X POST http://localhost:8080/models/unload \
-H "Content-Type: application/json" \
-d '{"model":"qwen3-8b"}' | jq
Put them together with a pipe:
curl -s http://localhost:8080/models \
| jq -r '.data[] | select(.status.value == "loaded") | .id' \
| while IFS= read -r model; do
echo "Unloading: $model"
curl -s -X POST http://localhost:8080/models/unload \
-H "Content-Type: application/json" \
-d "{\"model\":\"$model\"}" | jq
done
Six lines. A pipe and a loop. Not exactly what I was looking for, but it does the job cleanly.
Why This Pattern Is Actually Good
The per-model approach is safer than a blanket function. If someone else is using the server when the script runs, only loaded models get hit. You cannot accidentally write a command that touches models that were never meant to be touched.
It also gives you visibility. The loop prints each model as it unloads. If one fails, you see which one. If the server is unreachable, curl fails with a clear error. Pipe the output to a log and you have a record.
Is it a hack? Maybe. It is also the right shape for the problem. The router gives you primitives. You compose them yourself.
The Script I Actually Keep Handy
By the third time you type the same pipe, save it. Here is what I keep in my tools directory:
#!/usr/bin/env bash
set -euo pipefail
LLAMA_SERVER_URL="${LLAMA_SERVER_URL:-http://localhost:8080}"
models_json="$(curl -fsS "$LLAMA_SERVER_URL/models")"
loaded_models="$(printf '%s' "$models_json" \
| jq -r '.data[] | select(.status.value == "loaded") | .id')"
if [ -z "$loaded_models" ]; then
echo "No loaded models found."
exit 0
fi
printf '%s\n' "$loaded_models" | while IFS= read -r model; do
[ -z "$model" ] && continue
echo "Unloading: $model"
curl -fsS -X POST "$LLAMA_SERVER_URL/models/unload" \
-H "Content-Type: application/json" \
-d "{\"model\":\"$model\"}" | jq
done
echo "Done. Current model state:"
curl -fsS "$LLAMA_SERVER_URL/models" | jq
curl -fsS makes HTTP errors fail the script. set -euo pipefail catches problems early. The LLAMA_SERVER_URL variable lets you point at any host without editing the file.
Run it locally:
./llama-router-unload-all.sh
Run it against another machine on your network:
LLAMA_SERVER_URL=http://192.168.1.50:8080 ./llama-router-unload-all.sh
The JSON Shape Might Be Different on Your Machine
I learned this the hard way. Not every llama.cpp build returns the same fields. The router is still evolving, and the JSON shape shifts across versions.
If the script reports no loaded models but you know they are loaded, inspect the raw endpoint first:
curl -s http://localhost:8080/models | jq
The filter .data[] | select(.status.value == "loaded") | .id assumes the model identifier is in .id and the loaded flag is in .status.value. Your build might use .name instead of .id. The loaded condition might look different. Adjust the select to match.
The approach stays the same: list models, find the loaded ones, extract identifiers, unload. Field names are just details.
The Gotcha That Wasted 15 Minutes of My Life
I ran the script. It printed "Done." I checked VRAM. Still high. I ran it again. "No loaded models found."
How can VRAM be full if no models are loaded?
Turns out something sent a request between the two runs. Open WebUI was open in a browser tab with a model selected. It pings the inference server every few seconds. The router saw the request, loaded the model back on demand, and served the response. My unload worked. It just did not stay unloaded.
This is the main source of confusion with router mode unloading. The router loads models on demand. If anything keeps asking, the router keeps serving. The unload got overridden by the next incoming request.
The fix is straightforward. If you need VRAM to stay free:
- Close Open WebUI tabs.
- Pause cron jobs or scripts that hit the API.
- Stop benchmark runners.
- Disable health checks that use real model inference.
Unloading is not a firewall. The router is doing what it was designed to do.
When LRU Is Enough and When It Is Not
Router mode has a --models-max flag that caps how many models can be loaded at once. When you hit the limit, the router evicts the least recently used model. This handles normal pressure fine. Three active models and a fourth request comes in? Something gets evicted quietly. Life goes on.
But LRU eviction is reactive. It only kicks in at the limit. You cannot use it to reclaim VRAM before loading a larger model, or reset state for a benchmark, or drain models before maintenance.
My rule: use LRU for daily driving. It handles normal use without intervention. Use explicit unloading when you have something specific in mind, like clearing the deck before you load a much larger model.
Troubleshooting The Dumb Stuff
I hit most of these getting this working.
The /models endpoint returns 404. Your build might not include router support, or you are hitting the wrong port. Run llama-server --help | grep -i models to confirm router flags exist. Also, /models and /v1/models are different endpoints. The first is router management. The second is the OpenAI-compatible list. They return different data.
jq is not installed. The script needs it. On macOS, brew install jq. On Debian or Ubuntu, sudo apt-get install jq. Small dependency, and it pays for itself the first time you need to parse JSON in a shell script.
The unload call returns an error. Wrong model identifier. The identifier you pass must match what /models returns exactly. It might not be the filename. It could be a router alias or a shortened ID. Copy it from the listing output instead of guessing.
Model names with weird characters. Slashes, dots, or spaces in the identifier can break hand-escaped JSON. Use jq to build the request body:
body="$(jq -n --arg model "$model" '{model: $model}')"
curl -s -X POST http://localhost:8080/models/unload \
-H "Content-Type: application/json" \
-d "$body" | jq
This is safer than string interpolation.
VRAM does not go down after unloading. First, confirm the model status actually changed to "unloaded" by listing models again. Then check whether something reloaded it (the gotcha above). Finally, GPU memory reporting tools can lag. nvidia-smi shows allocator-level information, not just what the application intends. Give it a few seconds and check again.
The Part Where I Admit Router Mode Is Still Great
Router mode is one of the best things to happen to llama.cpp. It turns a single-model server into a multi-model layer without needing a separate proxy or management tool.
The unload-all pattern I described is not a workaround. The router gives you building blocks. You list the models, filter for loaded ones, unload them. Boring and debuggable. That is what you want when VRAM is on the line.
Would I take a convenience endpoint? Sure. Am I going to write a strongly worded GitHub issue over it? No. The shell loop works over SSH, from cron, whether Open WebUI is running or not. And it frees VRAM without restarting the server.
Local LLM infrastructure needs reliable operations. Not magic, not a shiny button. A six-line pipe is reliable. I will take that over a button I never ended up writing.