← The X99 Build

kvsnap: mistral.rs vs llama.cpp on an 8GB card

The software side of the 200 Bucks Inference Machine: reusing a long prompt's KV cache on llama.cpp and then mistral.rs, and comparing the two engines on the same models. September 2026. Source · The 17-line patch

X99 build series: The build · mistral.rs vs llama.cpp · Four LLMs compared · Wake on LAN · ReBar BIOS mod · CMP 40HX unlock · V100 + VFIO

A 15.6k-token prompt took 54–85 s to first token on one 8GB card. On mistral.rs with a 17-line patch, reusing its cached prefix brought that to 0.27 s, and 8 of 10 test answers came out token-for-token identical to recomputing. On ordinary requests, Gemma also ran 37% faster on mistral.rs than on llama.cpp, at the cost of 0.9–3.0 GB more GPU memory.

The problem

The machine first hosted OpenClaw, a personal agent for email triage, calendar management, and job application tracking. That turned out to be a poor fit for one small card. OpenClaw sends a ~15.6k-token system prompt on every agent step, and llama.cpp couldn't reuse its cache between steps on Gemma's sliding-window layers. So each step paid about 80 seconds of prompt processing before producing anything, and the agent's own watchdog aborted runs when the GPU clock was stuck.

I dropped the agent and gave the model a narrower job it does well: schema-constrained extraction, benchmarked separately. But the 80-second prompt cost was worth solving directly: process a long shared prefix once, keep its KV cache, and pay only for the new tokens after it.

Serving stack

The machine first served qwen3:8b through mistral.rs behind an OpenAI-compatible endpoint, with context tuned to 32,768 tokens so the model and context stayed resident across two 8GB cards. That build came from the GitHub master branch rather than the published 0.8.1 crate, which had a linking bug in afq.cu, and it needed a small patch to candle's compatibility.cuh: the original guard caused a redefinition collision between hmax_nan and hmin_nan on this hardware, and checking both CUDA_ARCH under 800 and CUDART_VERSION under 12080 resolved it. Current master (0.9.4, September 2026) builds cleanly for sm_75 with CUDA 13.0 and needs no patch; the build takes about 55 minutes on this machine.

With one card left, the everyday model is Gemma 4 E4B through Ollama (llama.cpp). About 4B parameters are active, a large per-layer embedding table stays in system RAM instead of VRAM, and most layers use sliding-window attention, so the full 131k context costs about 2GB of KV cache. Measured on this card with the clock up: about 220 tokens per second of prompt processing and 36 tokens per second of generation.

Attempt 1: llama.cpp slot snapshots

llama-server can save a slot's KV cache to disk and restore it later. The first experiment, on a 15,600-token prefix:

On a normal full-attention model (Qwen2.5 3B) it works out of the box. Gemma is harder: its sliding-window layers keep only the last ~1,024 tokens in a ring buffer, and after a restore the server logs forcing full prompt re-processing ... (likely due to SWA) and throws the cache away. With --swa-full (sliding-window layers keep their full history) the restore holds and time to first token falls from 119 s to 0.3 s. That costs 1.3GB more VRAM at 32k context and slower cold runs, and it gives up the small sliding-window cache that makes Gemma a good fit for this card in the first place.

Attempt 2: mistral.rs

So I moved the experiment to mistral.rs, a Rust inference engine built on Candle. Its prefix cache works in 32-token blocks and handles sliding-window layers itself. Same 15.6k-token prefix, same card, Gemma's default cache layout:

That is a 200× cut in time to first token on Gemma without --swa-full. The cold run was faster too (54 s against 85 s), though the llama.cpp numbers came from a server with different settings, so I wouldn't lean on that comparison.

Four fixes to fit an 8GB Turing card

• A 17 GB activation estimate: the device mapper sized activations for a full attention matrix at the requested context (8 heads × 32,768² × 2 bytes) and refused to load. --max-seq-len only feeds that estimate, so setting it to 4096 and sizing the KV cache separately with --pa-context-len fixed it.
• Half the model on the CPU: auto-mapping then put layers 23–41 on the CPU. Pinning all 42 layers to the GPU (-n 0:42) left no room for the paged KV cache, and without paged attention mistral.rs doesn't chunk prompts, so the 15.6k-token prompt ran out of memory.
• A 1.8 GB lookup table on the GPU: Gemma's per-layer embedding table is only ever read by token ID, and llama.cpp keeps it in system RAM. A small patch (17 lines) does the same in mistral.rs: the lookup runs on the CPU and only the gathered rows are copied to the GPU. That freed 1.85 GB for the KV cache.
• Prefill workspace: without flash attention on sm_75, prefill gathers cached keys and values into a temporary buffer that grows with chunk size. Dropping the chunk from 4096 to 512 tokens made it fit.

Is the cached answer the same answer?

Speed only matters if the cache holds the same state as recomputing it. Comparing final answers is too coarse, so I compared token probabilities: 10 questions about the prefix, 32 greedy tokens each, with the top 5 alternatives and their log-probabilities recorded at every step, run cold twice and warm once.

The cached state is correct but not bit-exact. With the cache, the question tokens attend over stored keys and values in their own chunk; without it, they're part of the prompt's last 512-token chunk. The math is the same, but the summation order changes, so the fp16 rounding does too. The shifts are small, the two divergences happened only where the model was already on a coin flip, and both diverged answers were still correct. A wrong cache would show up as large differences from the very first token, or as flips where the model was confident; neither happened.

Engine comparison: the same models on both engines

KV cache reuse is one reason to switch engines. The other question is whether mistral.rs is any faster or different on ordinary requests, so I reran three of the four models from the extraction benchmark on both engines, back to back on the same card with the same prompt, schema and settings. 69 of the original 70 postings were still online (one IBM posting had closed), so this covers 77 documents: the 69 postings plus the same 8 citizenship test copies. Qwen3 8B was left out to save about two hours of GPU time.

• Speed: Gemma ran 37% faster on mistral.rs (17.5 s against 27.6 s per posting) and Qwen3.5 10% faster; Qwen2.5 was even. mistral.rs wrote 6–24% longer answers but generated them faster, about 26–34 tokens per second end to end against 17–27 on llama.cpp.
• Memory: 0.9–3.0 GB more on mistral.rs, mostly because it reserves its paged KV cache for the full 12k-token context up front instead of growing it. Gemma only fit on the card at all after a small patch moved its per-layer embedding table to system RAM.
• Accuracy: most of the differences don't come from the engine. Qwen2.5 3B ran the exact same weights file on both, and the two engines still disagreed on at least one flag in 34 of 77 postings: at temperature 0, tiny numeric differences change which token wins, and the answer drifts from there. On llama.cpp it raised 16 no-sponsorship false alarms (quoting Figma's housing stipend again); on mistral.rs it raised none, but two answers looped until they hit the length limit. Qwen3.5 and Gemma had to use a different file of the same quantization on mistral.rs, which can't read Ollama's versions, so Qwen3.5's jump to 100% precision may owe as much to the file as to the engine.
• Ranking: Gemma scored 95–97% precision and 95% recall on both engines. It is still the model the pipeline uses, and on mistral.rs it ran within 2 s of the much weaker Qwen2.5 3B.

What's next

The cache lives in GPU memory and is gone when the server restarts. The original plan was a small storage engine for these snapshots: content-addressed files on SSD, tiered between RAM and disk, evicted by recompute cost against size. mistral.rs already hashes each 32-token block of KV cache by content, which is exactly the key such a store needs, so the next step is to build that disk tier inside mistral.rs rather than as a proxy in front of llama-server, and measure time to first token after a restart.