SEARCH THE LAB

What are you looking for?

ScienceReview information not provided

The Same File, Different Disk Usage: Sparse Files and Clones

Review information not provided · 3 min

Two files compare identical with cmp, ls -l shows the same byte count, yet df reports a noticeably different amount of free space. This is not the filesystem…

The Same File, Different Disk Usage: Sparse Files and Clones

Two files compare identical with cmp, ls -l shows the same byte count, yet df reports a noticeably different amount of free space. This is not the filesystem lying — logical length and actual usage were always two separate ledgers. The filesystem records how long a file is; what really lands on disk is physical space allocated in blocks. The gap between the two usually comes from holes, compression, deduplication, and sharing.

Sparse files are the most common example. A program can set a file's length to 1 GiB and write only 4 KiB at the start and 4 KiB at the end, never touching the middle. Filesystems that support sparseness record those never-written ranges as holes and allocate no data blocks. ls -l or stat still reports st_size as 1073741824; du, or stat's st_blocks, may show only a handful of blocks. Reading a hole yields zeros — semantically indistinguishable from actually writing a pile of zeros, but at vastly different cost. truncate, dd's seek parameter, and the preallocated files many logs and databases create all produce this shape.

The trouble starts at the copy step. Plain cp reads and writes byte by byte, turning holes into real zero-filled blocks, so 1 GiB instantly becomes 1 GiB of actual usage. To preserve sparseness you need options such as cp --sparse=always, tar -S, or rsync --sparse, and you must confirm that the destination filesystem itself supports holes. Conversely, packing a sparse file into an archive that does not support sparseness, or forgetting -S while archiving, undoes all of it. The 'it grows every time you copy it' incidents with VM images and container layers mostly come from exactly this.

Cloning takes a different route. Btrfs, XFS with reflink enabled, block cloning in newer ZFS, and APFS's clonefile all let two files share the same physical blocks, separating them only on write — copy-on-write. On Linux, cp --reflink=always can 'copy' a large file instantly within the same filesystem; on macOS, cp -c does the same. At that point both files report full st_size and st_blocks, du counts each of them separately so the total looks doubled, yet df barely moves, because the data is stored only once.

This brings several practical limits. First, cloning usually requires source and destination to be on the same filesystem and the same volume; across devices it degrades into a real copy. Second, shared blocks are only truly released once every reference is gone, so deleting one 'copy' frees no space — easy to misjudge. Third, du adds up block counts per file without recognizing sharing, so directory totals overestimate; to judge real headroom, look at df. Fourth, encryption layers, some network filesystems, older kernels, or volumes without the feature enabled do not support reflink, and cp --reflink=always errors out rather than silently falling back.

There is another layer: block size and tail packing, inline small files, transparent compression, and deduplication also make identical content land on different usage. A filesystem with 4 KiB blocks still consumes a whole block for a 1-byte file unless it supports inline data; compression or deduplication may store two identical copies as one. So when you see a usage discrepancy, first work out which number is being asked about: the length from ls -l, the block count from du, and the free space from df answer different questions.

In practice, you can check sparseness by comparing du with ls -l, or use filefrag and stat to see the block distribution; before copying a large image, confirm whether the destination supports sparseness and reflink; when backing up, be explicit about whether holes must be preserved; and do capacity planning with df, not by extrapolating from du's directory totals. These differences are not the filesystem cheating — it is that 'same content' was never the same as 'same storage cost.'