Typing Japanese

This probe cost less than any other in the series: a dozen lines that take a five-character Japanese greeting and run the string library over it. No server, no game, no benchmark — just typing Japanese at the language and writing down what comes back. The results were predictable in the way that makes them worth recording. str_len said 15. Indexing said \xe3. Reverse produced garbage that is not text in any language. Substring cut a character in half. None of this was a bug: the memory-model docs say plainly that str is bytes, and every operation did exactly what it promised. But “documented” and “usable for text” are different properties, and a language whose author writes in Japanese had, until this release, no way to count the characters in こんにちは.

Two bins

The probe’s useful move was refusing to treat “wrong” as one pile. Each surprising answer went into one of two bins. Bin one: bytes are the right answer. Concatenation, equality, hashing, splitting on ASCII delimiters, byte length for buffer sizing — UTF-8 is designed so that these work on raw bytes, and the probe verified each one against the Japanese string. No fix needed; the honest note is that they were never broken. Bin two: the program means characters. Length as “how many characters,” indexing as “the n-th character,” substring, reversal. These need a codepoint view, and faking them in user code means reimplementing UTF-8 lead-byte decoding in a language with no bitwise operators — the probe tried, produced a 40-line decoder built on div and mod, and filed its ugliness as the forcing measurement.

The smallest honest fix

The design question was scope: a full Unicode layer — grapheme clusters, normalization, locale-aware collation — or a codepoint view? The answer came from the bins: everything in bin two is solved by codepoints, and nothing in the probe needed graphemes. So the release added exactly two builtins: utf8_len, and utf8_chars to explode a string into a list of single-codepoint strings. Everything else — n-th character, codepoint-safe substring, reversal — is three short prelude functions composed on top, written in Mere itself. str stays bytes; the codepoint view is opt-in by spelling. The accidental milestone: because the prelude must run everywhere, both builtins had to land on all four backends — interpreter, C, wasm, LLVM — in the same release, the first feature in the project’s history born simultaneously everywhere. The rule that forced it (“the prelude may use only universal builtins”) was discovered the hard way, when the LLVM backend crashed on a prelude it could not supply. こんにちは now has length 5, its reverse is はちにんこ, and the remaining gap — graphemes, where は plus a combining mark is still two codepoints — is written down in the docs next to all the other gaps that wait for a program to need them.