A Real Format

A compressor and a decompressor that agree with gzip are a closed loop, but gzip is a container, not a use. PNG is a use: it wraps DEFLATE inside a real image format, and reaching it meant implementing everything around the compression that the gzip probe had let it skip. Big-endian chunk lengths and CRCs where gzip is little-endian; a zlib container with an Adler-32 checksum instead of gzip's CRC-32; and scanline filters — the two-dimensional byte prediction, referencing the pixel to the left and the row above, that makes image data compress. Decode and encode both round-tripped against the reference library on the first working build, with zero changes to the language. The compression core carried straight over; everything new was the format around it.

merepngcompressionfile-formatsdogfood

Round-tripping gzip proves the compression works, but gzip is plumbing — a container whose only job is to hold a compressed stream. The more interesting question is whether the same core can serve a format people actually use, one where DEFLATE is a component rather than the point. PNG is that format. Its image data is zlib-wrapped DEFLATE, so the compressor and decompressor from the last two parts drop straight in; what the PNG probe measures is everything else, the format built around the compression, and whether the language’s hands were already full enough to hold it.

Big-endian, for the first time

Gzip stores its lengths and checksums little-endian, low byte first, and every byte-assembly routine so far had been written that way. PNG is big-endian: chunk lengths, the dimensions in the header, the trailing CRC of each chunk all arrive most-significant-byte-first. This is a small thing and a real one — the first place the code had to read and write multi-byte integers in the opposite order — and it surfaced nothing but the need to write the bytes in the other sequence. The chunk structure itself, a length followed by a four-character type followed by data followed by a CRC-32 over the type and data, transcribes directly from the specification into nested loops over byte vectors.

A different container, a different checksum

Inside the image chunks, the pixel data is not raw DEFLATE but zlib-wrapped DEFLATE: a two-byte header, the compressed stream, and an Adler-32 checksum — a different algorithm from the CRC-32 that both gzip and the PNG chunk framing use. The decoder could skip verifying it, since decoding only needs to read the DEFLATE payload, but the encoder cannot: a PNG without a correct Adler-32 is rejected. Implementing Adler-32 — two running sums modulo a prime, one of the bytes and one of the first — was a few lines, and it is worth noting how many distinct checksums this one format demanded: a CRC-32 per chunk in the container, an Adler-32 over the uncompressed pixels in the zlib layer. A real format is a stack of small agreements, each with its own arithmetic.

Scanline filters are two-dimensional byte arithmetic

The part with no analogue in gzip is filtering. Before compression, each row of pixels is transformed by a predictor that subtracts an estimate built from already-seen neighbors — the pixel to the left, the pixel in the row above, the one up and to the left — leaving small residuals that compress far better than raw color values. There are five such filters, and a real encoder chooses per row; the decoder has to invert whichever was chosen, reading the filter byte at the start of each scanline and undoing the prediction pixel by pixel. This is the first genuinely two-dimensional byte computation in the whole arc — every output byte depends on bytes above and to the left of it — and it is exactly the kind of index-heavy loop that is easy to get subtly wrong. The verification was to decode a PNG the reference library wrote, with its adaptive per-row filter choices, and check the pixels against a known checksum; it matched.

Both directions closed. The decoder read a self-made RGB image with the filter cycling through all five types, a self-made RGBA image, and an image the reference library encoded with adaptive filtering — all to matching pixel checksums, on the interpreter and the C backend alike. The encoder wrote a gradient that the self-decoder, the reference library, and the original data all agreed on, and that the operating system recognized as a valid PNG. No part of this required a change to the language: the compression core carried over unaltered, and big-endian bytes, a second checksum, and two-dimensional prediction all fit in what was already there. The gzip arc had reached a real format, encode and decode both in pure Mere, byte-compatible with the tools everyone else uses.

← Back to Mere: Building a Language