Columns, Not Codepoints

Printing a small product table with Japanese cells showed that the Unicode work from earlier was the right first step and the wrong tool for this job. Counting codepoints says a five-character greeting is five wide; a terminal draws it in ten columns. Alignment needs display width — East Asian Width — not codepoint count. The fix was a wcwidth-lite in pure library Mere, decoding UTF-8 with plain division and a table of ranges, landing on all four backends at once because that is where library code lives.

mereunicodestdlibbackendslanguage-design

An earlier episode gave the language a codepoint view of strings, so it could count the characters in a Japanese word instead of the bytes. This probe found the place where that view, correct as it is, is still the wrong measurement — and the distinction is one every text-handling language eventually has to make.

Five characters, ten columns

The probe was small: print a product table with a name column, mixing ASCII and Japanese entries, and line the columns up. Padding with the codepoint-count function produced rows that were visibly ragged — the Japanese rows sat short. The reason is that a terminal does not draw one column per codepoint. It draws CJK characters, fullwidth forms, and emoji at two columns each; it draws combining marks at zero. A five-codepoint greeting occupies ten terminal columns, and the codepoint count — right for “how many characters” — is simply the wrong number for “how many columns will this take.” This is the same lesson as the bytes-versus-string distinction from two episodes ago, one level up: bytes, codepoints, and display columns are three different measurements of the same string, and a program has to ask for the one it actually means. Alignment means columns.

wcwidth, in the library, on every backend

The fix is a display-width function: East Asian Width, in the wcwidth-lite tradition — CJK, fullwidth, and emoji ranges count two, combining marks count zero, everything else counts one — plus left- and right-padding functions built on top of it. What makes it worth an episode is where it lives. All three are ordinary prelude functions, written in the language itself: UTF-8 is decoded with plain division and remainder (a two-byte sequence is one arithmetic expression), and the width table is a dozen range checks against the major blocks. There is no new builtin, no per-backend C or IR or Wasm. And because the prelude has to run everywhere, the moment it compiled it existed identically on the interpreter, C, LLVM, and Wasm — the whole part’s theme of four-way divergence turned inside out, because library code cannot diverge by construction. The width table is written in decimal ranges rather than the hex bounds a Unicode chart would use, which is mildly painful and is the second probe in a row to write a number in decimal that it would rather have written in hex — a papercut now large enough to fix, which is exactly what the final episode does. The aligned-table example renders a mix of ASCII, Japanese, an emoji, and halfwidth katakana with borders that actually line up, on the interpreter and the C backend alike.

← Back to Mere: Building a Language