The 54,000-Line File That Could Not Be Split

Mere is a small language I write for myself. mere-ruby is a Ruby interpreter written in it — about 1,100 of ruby/spec’s examples matching CRuby byte for byte, and a 207-program corpus that has to agree with real ruby on every run.

All of it lived in one file. main.mere, 54,063 lines.

Someone asked me the obvious question: is one file actually better, or have you just never split it? I assumed the answer was “never got round to it.” It was not. The language could not express the split, and finding that out took the day apart.

This is a measured piece. Every number came off this machine, and the times I was wrong are in the same tables as the times I was right.

1. Five spellings, all impossible

The interpreter is mutually recursive in the way interpreters are: eval_e calls call_method, which calls eval_body, which calls eval_e. So the first question is where a language draws the line that mutual recursion cannot cross.

I assumed it was the file, or the module. I measured instead:

How I tried to write it	What happened
Two let rec chains in one file	unbound variable: g
module A { } and module B { }	unknown constructor: B
import placed before the definitions	unbound variable: is_odd
import placed after them	the same — nothing before the splice point is visible
A file that begins with and	parse error: expected literal, identifier, or '('

The boundary is not the file and not the module. It is the let rec ... and ... chain. import is a splice performed by the parser — it drops the imported declarations in at the import point — so a chain closes wherever a splice happens, and no import can cross one.

An interpreter therefore has to be written as one chain. And a chain is one file, because a chain is a single syntactic construct.

2. And there was a second wall behind the first

Even the non-recursive 60% looked liftable: put each group in its own file and import them in topological order. I tried it at the line where I wanted the first cut, and got parse error: expected literal, identifier, or '('.

The cause is in the parser, not in the code:

| _ -> let main, toks = expr toks

parse_decls reads declarations until it meets a token that cannot start one, and that token begins the program’s main expression — everything after it belongs to that one expression. So import is legal only inside the declaration prefix.

main.mere’s declaration prefix ended at line 555. The remaining 53,500 lines were a single expression. There was nowhere to put an import.

Top-level let ... in was one of the constructs that ended the prefix, so I converted them to let ...;. The prefix moved from line 490 to 555, hit another construct, and I concluded that each one removed reveals another.

That was wrong, and it was my search that was wrong. My pattern was ^let .* in, which only matches a let ... in whose in is on the same line. There were exactly three more, all spanning multiple lines. Convert all 31 and the prefix runs to the end of the file: an import at line 54,000 parses.

The lesson is not subtle and I have written it down: a count produced by a pattern is a count of what the pattern matches. I stopped at 28 and drew a structural conclusion from it.

3. The verification rule, and the one time it broke

Moving code between files must not change the program. The compiler emits a single C file (58 MB, 36 seconds), so there is a strong check available: emit before, emit after, and diff.

Not byte-for-byte — the compiler numbers its temporaries and regions serially (_uq214, __rp7, __v2), so moving anything renumbers everything after it. The rule I used was: the generated C differs only in that numbering. Six of the seven extractions passed it exactly. The let ... in conversion of the previous section got the same treatment and came back with 183 global maps unchanged and 5,817 function names identical, the only difference being that two cross-chain duplicate names were mangled __v2 instead of _uq214.

The seventh did not, and I reported it wrong.

I said “four functions lose their __direct variant and four gain one” — __direct being the uncurried entry a saturated call goes to instead of building a closure environment. It sounded plausible: whether a call resolves statically depends on the callee’s position, and 840 definitions had moved.

It was my normalisation. I stripped _uq\d+ and __v\d+ from the emitted names and forgot the monomorphisation suffix, so

mu_bnd_snapshot__direct
mu_bnd_snapshot__Map_str_Val__Map_str_Val__direct

read as two different functions — one lost, one gained. They are the same function. Comparing source-level names: 2,549 before, 2,547 after, and every one has a __direct in both builds. What actually changed is that two dead functions stopped being emitted.

I committed the correction with the reasoning, because a wrong measurement in the record is worse than no measurement.

4. The split, with no language change at all

The chain had 1,451 members. 867 of them were in no cycle at all. They had been written inside let rec eval_e = ... and ... because that is where the code needing them was — not because anything required them to be there.

	main.mere	files
start of day	54,063	1
front end and driver out	46,168	3
four more modules	39,651	7
the 840 acyclic chain members out	30,610	8
dead code removed	30,535	8
the last five	30,392	8

⚠ The unit of splitting is the file, not the topic. My first attempt grouped the 840 by the section comments they sat under — Struct, Zlib, Marshal, sprintf — and produced thirteen cycles between those groups. No ordering of those files exists. As one let rec ... and ... group the internal order does not matter at all, and the 840 are mutually acyclic (every strongly-connected component among them is a singleton), so the file has no internal ordering constraint either.

One file, one mutual-recursion group. That is the granularity the language actually has.

5. Peeling runs to a fixpoint

After the 840 left, a gate I had just written found 30 top-level functions that nothing called. Removing them made five more chain members unreachable from inside the chain — arr_product2, errno_check, lp_seed, params_wo_defaults, register_builtin_consts — so they could leave too.

What remains is 596 members / 29,606 lines, and nothing further can move. The evaluator chain is now exactly its irreducible part.

6. Three gates, and what each one caught the day it was written

Splitting a file is a refactor, and a refactor needs something other than my attention watching it.

dup_defs_check.sh — no top-level name may be defined twice in one chain. Every call resolves to the later definition, so the earlier one is dead, silently, on a green build. I found this by adding a helper beside one that already existed under the same name: the types matched, the build stayed green, and my fix had no effect because every caller was reaching the old function. That cost most of an afternoon. Six had accumulated, and two of them differed in behaviour from the live version and had never run.

dead_defs_check.sh — no top-level function may be defined and never called. Thirty had accumulated, superseded by rewrites nobody finished.

⚠ Writing that one, I nearly deleted a live global. My rule for where a definition ends was “the next = fn”, and let gc_unsafe = map_new (); — a GC root table — sat between two functions and got swallowed. The compiler caught it (unbound variable: gc_unsafe). The correct rule is the next top-level item of any kind.

gen_structure_map.py — a map of which chain each definition is in and whether it is in a cycle. ⚠ It must read the program in splice order (the import expansion, not alphabetical). Reading the files alphabetically moved the evaluator’s cycle from 376 functions to 383: a different program.

7. The language change: let fn

To split the remaining 72% the language needed a way to say “these two call each other” without saying “these two are in one chain.” Four candidates:

1. Forward declarations — let fn f: A -> B; first, definition later.
2. rec { ... } grouping — an explicit mutual-recursion block spanning files.
3. Chain continuation — an import that declares it continues the chain.
4. Unit-scope order-independence — make the whole top level one recursive scope.

I judged 4 the strongest for a while. Then I measured the thing I had not measured:

let rec ident = fn x -> x
and useit = fn (n: int) -> ident n;
let b = ident "s";        → type error: expected `int`, got `str`

A let rec ... and group is monomorphic, inside and out. Candidate 4 makes the entire top level one such group — every top-level function monomorphic with respect to every other. A polymorphic helper could no longer be used at two types anywhere in the program. Candidates 2 and 3 have the same shape.

That leaves candidate 1, and the reason is the type system rather than taste: a type written in a declaration is quantified by the programmer, so the definition stays polymorphic. mere-ruby loses nothing at all — the evaluator chain is already one group, so its 376 functions are already monomorphic with respect to each other.

The syntax pairs with the extern fn <name>: <ty>; the language already has for names defined outside Mere; this one is defined inside it, later. It takes no new keyword — let fn cannot legally start a declaration today, because a pattern cannot be fn. (val would have broken 14 places across six files that use it as an ordinary name.)

let fn is_even: int -> bool;
let is_odd  = fn (n: int) -> if n == 0 then false else is_even (n - 1);
let is_even = fn (n: int) -> if n == 0 then true  else is_odd  (n - 1);

The implementation was small, as the investigation predicted. The C backend needed nothing — it already forward-declares every function. The interpreter binds the promise to a placeholder ref and the definition fills it, which is the same back-patching a let rec group already does.

Two things I could not have learned by reading. First, a written type variable is a promise of polymorphism, not a rigid name — and region parameters make that the common case, not the exotic one: 117 of the 161 declarations mere-ruby needs mention a type variable, because every function taking a Map or a Vec has one. A monomorphic version could not have written a single one. Second, a promise kept by a member of a let rec group has to count as kept — and that is the main path, because a chain is made of groups.

mere --decls <file> prints the declarations for a file’s own top-level functions. Writing 161 of them is transcription, not judgement.

Demonstrated in production: mere-ruby’s 38,856-line evaluator chain, cut in two by six generated declarations. Generated C identical but for numbering.

8. Then the feature turned out to have four defects

It shipped with seven unit tests, all green. All four defects were live underneath them.

What found them was not another unit test. It was requiring that mere --decls output, pasted back into the file it came from, produce byte-identical output — run over the compiler’s 178-program cross-backend corpus, of which 168 can be run standalone.

Defect	What it did
A promise on a name that shadows a builtin was never kept	let fn odd; + let odd = ... was refused as undefined
A definition more specific than its promise was accepted	a caller above it could pass a type the definition has no body for: the type checker passed the program and the C backend failed to compile it
Declaring a type removed polymorphism	a declared 'a -> 'a was fixed by its first call site — less general than the same definition with no declaration
--decls printed the prelude’s ~70 names, and ran the program	process_decls evaluates every top-level let, so asking for declarations executed the program and interleaved its stdout

The first is a pass that renames a top-level binding shadowing a builtin. It did not know about forward declarations, so the promise was registered as odd and kept as odd__v2. A declaration and its definition are one binding; the rename now happens at the declaration and the definition inherits it.

The middle two are one root. The declaration and the definition were related by instantiation where subsumption was meant. Unifying the definition against a fresh instance of the declared scheme lets the definition be narrower than the promise — and the instance’s variables, created at the outer level, drag the definition’s own variables down out of reach of generalisation. Unifying against the declaration as written fixes both at once: the parser makes 'a a rigid parameter that unifies only with itself or an unbound variable, which is exactly a skolem.

A fifth thing the round-trip found is not a defect. A declaration for a builtin-shadowing name moves the shadow up to the declaration, so a caller written above the definition stops seeing the builtin. That is the feature working — there is a corpus program that exists to hold exactly that ordering — but it is not what someone pasting a generated file expects. Those lines are now printed commented, with the reason.

167 of the 168 programs round-trip. The one that does not names a record type declared inside a module, which cannot be named in an annotation from outside it at all, with or without a declaration. It is exempted by name, and the gate fails if it ever starts passing, so the exemption cannot outlive its reason.

9. And then I decided not to use it

let fn makes the remaining 29,606 lines splittable. I measured what a split would cost before doing it:

Cut after member	File A	File B	Declarations needed
50	3,579	26,027	91
200	13,404	16,202	147
250	16,289	13,317	156
300	21,143	8,463	126
450	25,588	4,018	53

A balanced cut costs about 150 hand-maintained type signatures and buys two files of roughly 15,000 lines each. Fifteen thousand lines is not a file anyone navigates more easily than thirty thousand, and 150 signatures is 150 places for the declaration and the definition to drift.

So: no. The core stays whole. The feature exists, it is correct, it is documented and gated — and the right use of it here is none.

That is a real outcome and I want to state it plainly, because the tempting version of this story ends with the file split in half.

What it cost and what it bought

Everything below is unchanged, which is the point: 1,094 ruby/spec examples matching, 125 differing, 0 crashing; 207/207 corpus programs; 17 CLI tests. Identical before and after, verified by running the interpreter built by the new compiler against the one built by the old on every corpus program: zero differences.

	before	after
main.mere	54,063 lines	30,392
files	1	8
total	54,063	53,960
dead top-level functions	30	0, and gated
same-chain duplicate names	6	0, and gated

The 103 lines that vanished are the dead code. The split itself moves lines, it does not remove them — the honest summary is that the file people open first went from 54,063 lines to 30,392, and the other 23,568 are in seven files named after what they do.

Three things I would tell someone starting the same day:

Ask whether it is possible before asking whether it is wise. I spent the first hour weighing readability against churn for a refactor the language could not perform. The measurement that mattered took ten minutes and five three-line programs.

A refactor’s oracle is the artefact, not the tests. “Emit before, emit after, diff modulo numbering” caught things no test suite of mine covers — and when it did find a difference, my normalisation was wrong before the compiler was.

A generated description has one honest test: put it back in. Seven unit tests written by the person who wrote the feature see the three-line programs that person imagined. Requiring that --decls output be pasteable into the file it came from, over programs written for other reasons entirely, found four defects in one run.