← The X99 Build

Unlocking PCIe 2.0, Resizable BAR and FP16 on the CMP 40HX

Running the cmpunlocker driver patches on the CMP 40HX in the 200 Bucks Inference Machine, and getting PCIe Gen2 to train on an X99 root port where the driver's own retrain fails. October 2026.

X99 build series: The build · mistral.rs vs llama.cpp · Four LLMs compared · Wake on LAN · ReBar BIOS mod · CMP 40HX unlock · V100 + VFIO

The CMP 40HX ships locked to PCIe Gen1, a 64 MiB BAR1 and crippled FP16. The cmpunlocker patches to NVIDIA's open kernel driver lift all three without touching the VBIOS. Compute and ReBAR worked on the first boot. Gen2 didn't: the driver's link retrain failed on the X99 root port. A driver reprobe followed by a second retrain from the root port brought the link up at 5 GT/s. FP16 throughput went up 55×, prompt processing about 12×, token generation 2×, and host-to-GPU bandwidth doubled.

Why

The CMP 40HX is a TU106 (the RTX 2060 Super / 2070 die) sold for mining, and NVIDIA limited it so it couldn't compete with gaming cards. The capacitor mod already got the card to x16, but the link still ran at Gen1 (2.5 GT/s, about 3.1 GB/s in practice). On top of that, FP16 cuBLAS ran at under 1 TFLOPS, which is the path Ollama uses for prompt processing.

cmpunlocker is a set of patches to NVIDIA's open-gpu-kernel-modules 610.57.04. It changes the GSP/SEC2 boot path to restore the SM configuration, opens the GSP PCIe policy for Gen2 and retrains the link, and enables the card's ReBAR capability. Nothing is written to the VBIOS. The unlock is reapplied every time the driver loads, so turning the machine off returns the card to stock.

The compute unlock reproduces the CMP 170HX technique on a Turing card. The 170HX unlocker (pearlfortune/cmpunlocker) plants a crafted payload in the SEC2 Booter during driver initialisation, before GSP boots. The Booter runs signed, so it can write the PLM-gated registers the host isn't allowed to touch (architecture notes). The 40HX patch traces back to that code through sbccc1888's port, and uses the same path to restore the SM configuration. The boot log confirms it ran: FECS_PLM=0xffffff8f.

Installing

1. I measured a baseline on the stock 580 driver first: bandwidth, cuBLAS GEMM, Ollama prompt and generation speed, and model load time.
2. I switched to the open kernel driver at the exact version the patches target: sudo apt install nvidia-driver-610-open, then held the packages so apt can't upgrade them out from under the patches.
3. I ran sudo ./install.sh. It downloads the 610.57.04 source, checks its SHA-256, applies the three patches, builds the modules and installs them to /lib/modules/$(uname -r)/updates/cmpunlocker/.
4. I powered the machine fully off and back on. A warm reboot isn't enough.

One thing to check afterwards: the stock DKMS build of the same driver sits in updates/dkms/ right next to the patched one. modinfo nvidia | grep filename should point at updates/cmpunlocker/, and on my machine it did.

PCIe Gen2: the driver's retrain fails on X99

After the cold boot, dmesg showed the compute unlock passing (SS0=0x88888888 SS1=0x00000008 FECS_PLM=0xffffff8f) and nvidia-smi reported an 8192 MiB BAR1. The link was still at 2.5 GT/s, and the last line of the log explained why:

CMP40_PCIE_GEN2_V2: RETRAIN_FAIL status=1101

The project's diagnostic notes describe this exact failure on VBIOS .04, and fix it by unbinding and rebinding the driver. Its Gen2 retrain was verified on an AMD root port, while an Intel Alder Lake test hit trouble, and I couldn't find any reports from X99. The log also shows the driver had already set a Gen2 target (bits[19:18]=2) on both ends. The retrain itself had failed. So I stopped everything using the GPU, ran the project's reprobe helper, and then asked the root port to retrain once more by setting the Retrain Link bit in its Link Control register:

sudo systemctl stop ollama nvidia-persistenced
sudo ./tools/cmp40hx-driver-reprobe-gen2.sh --confirm-driver-reprobe
sudo setpci -s 00:02.0 CAP_EXP+10.w=0020:0020
sudo lspci -vvv -s 03:00.0 | grep 'LnkSta:'
sudo systemctl start nvidia-persistenced ollama

This time the link trained at 5 GT/s x16 on both ends. The root port's "downgraded" just means it can do 8 GT/s and settled at the card's 5. Loading a model and running the full benchmark set left the link at 5 GT/s throughout, and the compute unlock and 8 GiB BAR1 stayed in place. dmesg showed no AER or Xid errors afterwards.

Correction (October 15). An earlier version of this post said the retrain alone did it, using the unmasked CAP_EXP+10.w=0020. My shell history says otherwise: I had run the project's driver reprobe helper first, and the retrain was the masked form 0020:0020, which sets only the Retrain Link bit and leaves ASPM and Common Clock Configuration alone. Both steps matter. After a cold boot the card advertises Gen1 only (LnkCap2 bit 0x04 clear), and a retrain on its own comes back at 2.5 GT/s, which I confirmed on October 5. The reprobe (a driver unbind and rebind) makes the card advertise Gen2, and then the retrain brings the link up to it. The unlock resets on every power cycle, so both have to run after every cold boot. A systemd oneshot in my Ansible repo now does it before Ollama starts, and it brought the link up to Gen2 by itself on the next cold boot.

Results

Same machine, same models, same scripts. Each number is the median of 3 to 5 runs. Load time is a cold load from Ollama with the model file already in the page cache, so it measures PCIe and setup rather than the disk.

FP16 at 54 TFLOPS is well above the project's own 11.42 cuBLAS figure. Mine is a 4096³ GEMM with FP32 accumulate, which cuBLAS runs on the Tensor Cores, and it lines up with the 63.8 TFLOPS Tensor Core result they report.

Bandwidth doubled exactly as Gen2 should, but load time improved only 11 to 14%. Loading Qwen3 8B averages about 1.2 GB/s, so most of those 4 to 5 seconds is Ollama starting its runner and setting up, not the copy. At Gen1 the 5.2 GB copy alone takes about 1.7 s, and halving it should save about 0.8 s. The measured saving was 0.7 s.

I didn't expect generation to double. Token generation is normally limited by memory bandwidth, and the memory clock didn't change (7001 MHz before and after). It seems the stock SM configuration was holding back decode as well as prefill.

Two caveats. I changed the driver (580 closed to 610 open) and applied the unlock in the same step, and didn't measure stock 610 in between. So the compute numbers belong to driver plus unlock together. The bandwidth and load-time gains come from the link and are clean. Also, the first Gemma prompt run after the model loaded came in at 75 tok/s, so I used the two runs after it.

If you're doing this yourself

Install the exact driver version first, and check with modinfo that the patched module is the one that loads. If dmesg | grep CMP40 shows RETRAIN_FAIL status=1101 on an Intel board, run the driver reprobe and then a masked retrain from the root port. On my X99 neither one works alone. And keep a way back: deleting updates/cmpunlocker, running depmod -a and rebuilding the initramfs returns you to the stock 610 driver. After a kernel update, rerun sudo ./install.sh --no-download, since DKMS rebuilds only the stock module.