The week my ZFS pool ate itself

ZFS mirrors protect you from a dying disk. They do not protect you from yourself. This is the story of how I learned the difference over four increasingly unhinged days, and how the actual culprit turned out to be hiding in a BIOS menu the whole time.

Day zero: the confident years

My home server runs Proxmox with a ZFS pool — two mirrored pairs holding the family’s data, the media library, backups, everything. Mirrors! Redundancy! I had, in the way of homelab people everywhere, allocated 46GB of RAM to VMs and containers on a host that physically contained 32GB. This worked fine, in the way that things work fine right up until the moment they very much don’t.

The moment arrived during a heavy download-and-unpack session. The host ran out of memory. ZFS’s in-memory cache — the ARC — panicked. And here is the sentence that restructured my understanding of storage: the panic wrote identical corrupted metadata to both legs of the mirror.

Both legs. Simultaneously. Because a mirror faithfully replicates whatever it’s told to write, and what it was told to write was garbage. The corruption in RAM happened before the redundancy. The pool now contained an orphaned metadata object — forever after known in my house by its hex address, 0xda4 — and any heavy I/O that touched it would kernel-panic the entire host.

Days one through three: the ghost hunt

What followed was a diagnostic death spiral that I can only describe as educational.

The downloads kept crashing the host, so I blamed the download client — not entirely unfairly, since NZBGet is abandonware with genuine TLS socket hangs, and replacing it with SABnzbd was the one unambiguously correct decision of the week. The crashes continued.

So I upgraded the Proxmox kernel. The new kernel, 6.14.11-6-pve, turned out to have its own bug — a NULL pointer dereference that crashed the machine during package unpacking, with all registers zeroed, which is the CPU’s way of shrugging. So now I had two overlapping crash signatures: ZFS panics during pool I/O, and kernel crashes during any heavy unpack, ZFS loaded or not.

Standard procedure at this point is to isolate: boot a live USB, take your own software out of the equation. The live USB also crashed during unpacking. A fresh, pristine Ubuntu image, crashing the same way. At this point every layer of the stack was a suspect and I was reading kernel oops dumps at 1 a.m. with the specific serenity of a man who has stopped believing in causality.

I ran a full e2fsck on the root filesystem, convinced four days of hard resets must have shredded it. Five passes. Zero errors. The filesystem, insultingly, was fine.

Day four: the villain

The clue was the live USB. If a known-good OS crashes on known-good storage, the problem is below the software. And I was running an Intel i5-13600KF — a 13th-generation chip, a family with a well-documented instability saga that Intel itself acknowledged.

But the chip wasn’t the root cause either. The root cause was my motherboard. ASUS, in its factory-default enthusiasm, had set the board’s power limits to 253W — nearly double the processor’s specification — plus the full buffet of auto-overclocking: Turbo boost tweaks, Adaptive Boost, the works. The CPU had been quietly running outside its design envelope the entire time, and heavy sustained loads (say, unpacking large archives, or a filesystem cache thrashing under memory pressure) pushed it into producing wrong answers. Silently. In RAM.

Which reframes the whole week. The memory overcommit lit the fuse, but the CPU had likely been corrupting data in small, invisible ways for who knows how long — and ZFS, doing its job perfectly, faithfully mirrored one of those corruptions onto both disks.

The fix was almost offensively simple: turn off every BIOS auto-boost feature, set power limits to Intel spec, and upgrade to 64GB of RAM so the overcommit could never recur. The machine has been boringly stable since.

The body count

• One ZFS dataset rebuilt; 0xda4 lives on as a cosmetic scar in zpool status, structurally harmless, permanently humbling.
• One download client replaced (rest in peace, NZBGet, you were broken anyway).
• One kernel version on the do-not-fly list.
• Four days of my life, converted into the following lessons at an hourly rate I decline to calculate.

What I’d tell past me

Mirrors don’t protect against in-memory corruption. RAID and ZFS redundancy defend against disk failure. Corruption that happens in RAM gets replicated with perfect fidelity. If the data matters, the machine needs ECC or, at minimum, a CPU running inside its actual specification.

Never overcommit memory on a ZFS host. The ARC assumes it can have what it needs. When it can’t, it does not fail gracefully; it fails creatively.

Motherboard defaults are marketing, not engineering. Consumer boards ship with power limits and boost settings chosen to win benchmark charts. If a machine’s job is to be correct rather than fast, the first thing to do on day one is set every power limit to the silicon vendor’s spec.

When symptoms span unrelated subsystems, go down the stack, not across it. I spent three days lateral-moving between download clients, kernels, and filesystems. The live-USB crash was the tell: when everything is broken, the thing they share is broken.

And run fsck after hard resets anyway. Mine was clean, but I only knew that after checking, and “unverified” and “fine” feel identical right up until they don’t.

The server is called gufpalace. For four days it was more of a gufsiege. It stands, though — 64GB strong, boosts disabled, quietly serving Plex like nothing happened. Machines don’t hold grudges. That’s my job.