SEARCH THE LAB

What are you looking for?

ScienceReview information not provided

Why Compressed Backups Sometimes Barely Shrink

Review information not provided · 3 min

You pack a 200 GB directory and compress it, only to find the result is still 190 GB. The first instinct is that the compression tool didn't run. Most of the…

Why Compressed Backups Sometimes Barely Shrink

You pack a 200 GB directory and compress it, only to find the result is still 190 GB. The first instinct is that the compression tool didn't run. Most of the time it did work; the data simply had almost nothing left to give. Compression does exactly one thing: it finds repeated or predictable byte sequences and replaces them with shorter encodings. The closer the data is to random, the more the algorithm can only move bytes around as-is, plus a little framing overhead.

The most common case is data that is already compressed. JPEG, PNG, MP4, MP3, and archives such as zip, gzip, and zstd all apply entropy coding internally, so their byte distribution is close to uniform and a second pass finds few repetitions. Suppose the backup holds twenty thousand JPEG photos totalling 100 GB; tar merely concatenates them, and gzip can at most save a few file headers and alignment padding — a ratio near 1.0 is entirely expected. Re-compressing a batch of MP4 files often shrinks it by only a few percent.

Encrypted data behaves the same way. Ciphers such as AES are designed so that ciphertext is indistinguishable from a random string. Compressing after encryption hands the compressor a bag of noise; the correct order is compress first, then encrypt. If your backup tool encrypts the whole archive by default and an outer compression layer runs afterwards, the payoff is usually tiny.

File count and metadata also eat into the gains. tar writes a 512-byte header for every file and pads to block boundaries; if you gzip files one by one, each member carries its own header and trailer. Imagine a million small files averaging 1 KB each: 1 GB of content, but the headers alone can approach 500 MB after packing. Headers are highly repetitive, so compression removes a good deal of them, yet small-window algorithms cannot match across distant file boundaries well, and the final size can still noticeably exceed the content itself.

The compression window determines how far back an algorithm can look. Classic deflate has only a 32 KB window, so identical blocks far apart never get matched. xz and zstd with long-range mode use much larger dictionaries and catch distant repetitions, at the cost of memory and CPU. If your data repeats a stretch of content every few tens of megabytes, a small-window algorithm will simply miss it.

A frequent illusion is an archive where one part compresses beautifully and another refuses to budge. Say 100 GB of backup contains 80 GB of video and 20 GB of text logs: the text compresses to 4 GB, the video stays 80 GB, total 84 GB — only 16% smaller overall. The incompressible part dilutes the ratio, so the whole thing looks like it 'barely changed'.

Database dumps deserve their own look. Plain-text logs are repetitive and usually compress very well. But tables full of UUIDs, random hashes, timestamps, floating-point numbers, or binary blobs offer little room. Hexadecimal UUID characters look structured but each digit is effectively random; floating-point mantissas are similarly unpredictable. Row-oriented dumps still let you reclaim repeated column names and SQL keywords, but random primary keys push the gains back down.

In practice, don't rush to swap algorithms. Take a representative sample and try gzip, zstd, and xz on it to measure the marginal gain. Store media files directly instead of re-compressing them. Merge many small files into large blocks before compressing. If you need cross-file duplication removal, use a deduplication tool — that is a different job from compression. Raising the level has a ceiling too: level 9 over level 6 often buys a few extra percent at several times the runtime. Decompression speed and restore time belong in the cost of a backup as well.

The limits deserve stating plainly: the compression ratio is set by the data's content, with no universal guarantee; a corrupted compressed archive can damage a wider blast radius than uncompressed files would suffer; and volume splitting, encryption, and deduplication each change how compressible the result is. Understanding this stops you from reading 'it didn't shrink' as a malfunction, and points you instead toward reorganising the data — or accepting that it was always going to be this size.