A few days ago I woke up to a homelab server reporting its root filesystem was 100% full — zero bytes available. Nothing obviously wrong, no runaway log file, no giant Docker volume. Just… full. Here’s how I tracked it down, and the lesson it taught me about Linux mount points.
The Setup
The box in question is an Ubuntu Server VM (running on Proxmox) hosting a handful of self-hosted services in Docker — media tools, a photo library app, a few dashboards, that sort of thing. It also mounts a couple of network shares from a NAS on the local network for bulk storage, so large media files live off the VM’s own disk.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/...-lv 124G 124G 0 100% /Chasing Ghosts
The obvious first suspects all came up empty:
Docker — images, containers, and volumes together added up to only a few gigabytes. Not the culprit. A corrupted filesystem — I booted into a rescue environment and ran a full fsck on the volume. It found and fixed some minor inconsistencies, but disk usage didn’t budge afterward. Deleted-but-still-open files — a classic cause of “disk full but I can’t find anything,” where a process holds a file handle open on a file that’s technically been deleted, so the space never gets released. A quick check ruled this out too — nothing significant was open.
The real clue came from comparing two numbers that should always agree but didn’t:
$ du -xh --max-depth=1 /
19G /du, which walks the visible directory tree, could only account for 19G of files. But the filesystem itself — checked directly via its own metadata — reported almost no free space at all. Somewhere, roughly 100G of data was allocated on disk but invisible to a normal directory listing.
The “Aha” Moment
That gap — real disk usage that a directory walk can’t see — has one very specific, very common cause: something is mounted on top of a directory that already has files in it.
Here’s the mechanism, and it’s worth understanding because it can bite anyone running a home server with network storage:
You have a local folder, say /media, which is normally just an empty mount point for a network share. At some point — a network hiccup, a service starting too early at boot, whatever — the network share fails to mount before something tries to write to /media. Since the mount didn’t happen yet, /media is just a regular, empty local folder at that moment. Whatever tries to write there succeeds — but writes to your local disk, not the network share. Later, the network share mounts successfully (maybe on a retry, maybe on the next boot). Linux now shows you the network share’s contents when you look in /media — the local files are still physically on disk, sitting underneath the mount, completely hidden from view. du, ls, file managers — none of them can see past an active mount point to what’s underneath it. The disk blocks stay allocated. Forever, until someone thinks to check.
In my case, one of the automated download tools I run had, on at least one occasion, started before the corresponding network mount was ready — and quietly kept working the whole time, none the wiser that its files were landing on local disk instead of the network share. This had apparently been happening on and off for the better part of a year, a little bit at a time, until it finally filled the disk completely.
Confirming and Fixing It
To check the theory, I temporarily unmounted the network share and looked at what was actually sitting in the local folder underneath it — and there it was: tens of gigabytes of real files that had no business being on the local disk.
From there, the fix was mechanical but required care:
Mount the real network share at a second, separate location so both the “hidden” local copy and the legitimate remote copy are visible at the same time. Compare file-by-file: anything that already exists on the network share is a safe duplicate. Anything that doesn’t exist there yet needs to be copied over first. Copy over anything missing, verify it landed correctly, then delete the local copies. Remount the network share normally.
Root filesystem usage went from 100% full to 16% used in the process — the “missing” ~100G was exactly the hidden local data.
Preventing It From Happening Again
The actual root cause wasn’t the download tool misbehaving — it was a boot-order race condition. The service was starting before the network mount was guaranteed to be ready, so nothing stopped it from writing to an empty local folder instead of failing loudly.
The fix for that is a small systemd configuration change: tell the container runtime to explicitly wait for network filesystems to be mounted before it starts anything:
[Unit]
After=remote-fs.target
Requires=remote-fs.targetThis gets added as a systemd “drop-in override” for the container service, and it ensures the ordering is enforced on every boot going forward, not just this one.
The Takeaway
If your Linux disk usage numbers don’t add up — df says it’s full, but du can’t find the files — don’t assume corruption or a mysterious log leak first. Check whether something’s hiding underneath one of your mount points. It’s an easy trap to fall into with any server that combines local storage and network shares, especially if services start automatically at boot without any guarantee about mount ordering.
And if you’re running anything similar — Docker containers, download clients, media servers — it’s worth double-checking that whatever manages your service startup actually waits for your network mounts to be ready first. It’s a five-minute fix that prevents a very confusing afternoon.