← The X99 Build

Four small LLMs, one 8GB mining card: which one reads a job posting best?

Gemma 4 E4B, Qwen3.5 4B, Qwen3 8B and Qwen2.5 3B, benchmarked on structured extraction over 78 hand-labeled documents, all running on the 200 Bucks Inference Machine. September 2026. The same models rerun on llama.cpp and mistral.rs are in mistral.rs vs llama.cpp.

X99 build series: The build · mistral.rs vs llama.cpp · Four LLMs compared · Wake on LAN · ReBar BIOS mod · CMP 40HX unlock · V100 + VFIO

I tested four small LLMs on pulling eligibility flags out of internship postings, with every flag backed by a quote that code verifies. Gemma 4 E4B won with 95% precision and 97% recall. Qwen3 8B, the biggest model, caught just as much but flagged far too often, ran slowest and nearly filled the card, and Qwen2.5 3B wasn't usable for this task.

The task

Each model reads one internship posting and fills in a fixed JSON schema with five yes/no eligibility flags:

• U.S. citizenship required. A work-authorization, export-control or "U.S. person" line doesn't count.
• Undergrad only. A posting that also takes Master's students doesn't count.
• PhD only.
• No visa sponsorship.
• Security clearance required.

For every flag it sets, the model also has to copy one sentence from the posting, word for word, as evidence. A short Python check then looks for that sentence in the posting text, ignoring whitespace and curly quotes. A flag only counts if its quote is really there, so the model does the extracting and plain code makes the call.

This is a good fit for a small local model: the postings are 1–6k tokens, the output is tightly constrained, and nothing here needs outside knowledge. It's also a good test of whether a model can follow a definition ("a no-sponsorship line is not a citizenship requirement") instead of matching keywords.

Setup

• Hardware: one NVIDIA CMP 40HX (Turing, 8 GB) in the X99 box, clocks locked high. The machine has only 15 GiB of system RAM, which becomes important below.
• Serving: Ollama, with its JSON-schema structured output (format), temperature 0, a 12k-token context, and "thinking" turned off on Gemma and Qwen3/3.5 so the comparison measures one direct answer from each model.
• Data: 70 real postings, rendered in headless Chrome, plus 8 test copies where I added a citizenship sentence in different wordings ("sole U.S. citizenship", "Only United States citizens are eligible…", "requires U.S. citizenship because of government contract requirements", and so on). That's 78 documents in all.
• Answer key: labeled by hand from the posting text, not from any model's output. The positives: 8 citizenship, 14 undergrad-only, 39 no-sponsorship, 1 clearance, 0 PhD-only. The postings also include deliberate traps, such as Waymo's export-control text, Medline's "permanent work authorization" line, and SingleStore's "sponsorship is available".
• Runs: one pass per model over all 78, one model loaded at a time.

Results

Accuracy

Recall is close for three of the four models: Gemma, Qwen3 8B and Qwen3.5 4B each found 59–60 of the 62 requirements. They split on precision, which is what matters for a screening step, since each false alarm is a posting wrongly marked "you can't apply".

Where the false alarms came from

Cost

Every flag

Comparing the four

Gemma 4 E4B: the most careful reader

Gemma was the only model that followed the definitions consistently. It never mistook a sponsorship or work-authorization line for a citizenship requirement. Its two undergrad false alarms were real borderline cases: Amazon's "one quarter/semester remaining" and Fractio's "enrolled in, or graduated within the past year". It also caught three things my first answer key missed: both Genuine Parts postings ("Working on a BS degree") and a "Federal security clearance" line buried in an IBM posting. I corrected the key before scoring.

"E4B" means about 4B effective parameters. The file is 9.6 GB because a large per-layer embedding table sits in system RAM, and most layers use sliding-window attention, so long contexts are cheap on an 8 GB card. That same design makes prompt-cache reuse awkward in llama.cpp, which I dealt with in mistral.rs vs llama.cpp.

Qwen3.5 4B: close behind, and faster

It's the newest model here and the closest competitor, about 25% faster than Gemma, uses less memory, and has a perfect sponsorship score. Its weakness is that it over-flags undergrad-only: it treats "pursuing a university degree" with no level given as bachelor's-only, and once offered Figma's "join us!" as evidence. It also missed the clearance line and two citizenship test copies. If speed mattered more than precision it would be the pick; for a screening step it doesn't.

Qwen3 8B: same recall, far more false alarms

It's twice Gemma's effective size with the same recall, but it flagged almost anything mentioning a bachelor's degree as undergrad-only. That included "Preferred education: Bachelor's Degree" on several IBM postings and "Pursuing a Bachelor's or Master's". It also read application-form questions ("Do you now or will you in the future require immigration sponsorship…") as no-sponsorship statements. At 6.8 GB it nearly fills the card, leaving little room for context, and it was the slowest at 39 s per posting.

Qwen2.5 3B: keyword matching

It called 39 of 70 postings citizenship-only, citing lines like "located in the United States" and "Bachelor's degree or above", and for sponsorship it quoted Figma's housing stipend and Goldman's email policy. It also produced the most quotes that weren't in the posting at all. It's the fastest and smallest model, which makes it fine for cache experiments where answer quality doesn't matter, but not for this.

What I took away

Qwen3 8B lost to a model half its effective size. On a narrow extraction task, following instructions carefully mattered more than raw capacity, and public leaderboards, where Qwen3 8B usually looks strong, didn't predict that. An evaluation built from your own data is worth the hour it takes.

The quote check stops invented evidence, but it can't check meaning. Nearly all of Qwen2.5 3B's false alarms passed it: the sentences it quoted really were in the postings; they just didn't say what the model claimed. So every flag stays a hint that a stronger reviewer confirms, and nothing is dropped on the small model's word alone.

All four models missed "Only United States citizens are eligible for this internship.", the one test sentence without the words "citizenship" or "must be a citizen". A plain text search for "citizen" running alongside the model catches all 8 test copies at no cost.

System RAM turned out to be a bigger problem than VRAM. With only 15 GiB of system RAM, Ollama was killed for running out of memory when Gemma was still loaded as the next model started. It restarted within 5 seconds, but the next run failed on nearly every posting during that gap. Rerunning with one model loaded at a time fixed it. On a machine this size, OLLAMA_MAX_LOADED_MODELS=1 is worth setting.

Caveats

• One run per model at temperature 0, on 78 documents. Differences of one or two flags are within noise; the precision gaps are not.
• I labeled the answer key myself. The models found three mistakes in it, which I fixed before scoring, and there may be others.
• Only 1 clearance and 0 PhD-only positives, so those rows mostly measure false alarms.
• GPU memory figures are approximate readings from nvidia-smi during each run, not peak measurements.
• Thinking mode was off for every model. Qwen3 and Gemma might do better with it on, at several times the cost per posting.