Running Other Programs

The first tool was a task runner: read a small file of named tasks, each a command with optional dependencies, and run them in the right order. Call it mk — a make in miniature. It was chosen not because the world needs another make, but because a task runner cannot exist without the power to run other programs, and that was precisely the power the grep had reported missing. There was also a pleasing circularity to it: the language was, at that moment, being built by a shell script that ran the compiler over itself. A task runner written in the language would be running the very kind of commands that were building the language. The tool was pointed straight at its own foundations.

The predicted gap, and how small it was

The plan said the pain would be subprocess control, and the plan was right — for exactly one afternoon. Getting the first run "echo hi" to work required adding a single builtin, run, with the simplest honest type: a command string in, an exit code out. In the interpreter it became a call to the standard library’s command runner; in the C backend it became system with the exit status unpacked. That was the whole of it. The capability that the entire tool was built to force into existence turned out to be a few dozen lines, and once it was there the milestones fell quickly: capture the output, parse the task file, resolve dependencies in topological order, short-circuit on the first failure. Each of those was ordinary work in a language that was, by now, comfortable — records, mutual recursion between “run this task” and “run its dependencies”, tuples threaded through to carry an exit code and a done-flag. No pain at all. That absence is itself a finding: the core was solid, and the sharp edges were all out at the boundary with the operating system.

The gaps the plan didn’t name

The interesting failures were the ones the plan hadn’t predicted, and they only appeared when the tool was compiled to a native binary rather than run in the interpreter. The first time mk needed to report a failed command to standard error, the C backend simply did not have a way to print to stderr — the interpreter did, so the program had type-checked and run interpreted, and only the native build exposed the hole. It was a one-function fix, but it was a real gap that no amount of interpreter testing would have surfaced. The same story repeated when incremental builds needed to ask whether an output file already existed: the backend could read a file’s modification time but could not answer the plain question “does this exist?” Two small holes, both on the same seam — behaviour present in the interpreter and absent in the compiled backend — and both found the same way, by a real program walking across them. This is the argument for keeping backends in agreement stated in the negative: every place they disagree is a latent bug waiting for a program of the right shape.

The teeth were in the operating system

The last milestone was the one that bit hardest, and it bit in a place no one had marked on the map. Running independent tasks in parallel should have been free — the language already had threads and a parallel-map, and the tasks were independent by construction. But three tasks that each slept for a third of a second, run in parallel, took a full second instead of a third: they were running one after another, not together. The threads were real; the parallelism was not. Isolating it with a small C probe gave the verdict: on this operating system, the C library’s system call takes a global lock, so any number of threads all calling it queue up behind a single door. The parallelism had been strangled not by the language but by a serialization hidden inside the platform’s own runtime. The fix was to stop going through that door entirely — to start processes with the lower-level spawn call and wait on them individually — after which three third-second tasks finished in a third of a second, as they always should have. The bug the tool was built to expose was subprocess control; the bug it actually taught was that “run a command” and “run a command without secretly serializing everyone else” are two different capabilities, and only the second one is worth having.

By the end the runner did everything a small make should — parse, dependencies, topological order, incremental skipping, parallel groups, exit-code propagation — and it did it as a native binary. It drove five releases of the language, and every one of those releases was a capability that now belongs to every program, not just this one. The predicted gap was real but shallow; the valuable findings were the ones standing just behind it, which is the recurring lesson of building at the frontier: you aim at the gap you can see, and what you learn is whatever was hiding in its shadow. The next tool aimed somewhere entirely different — not at other programs, but at the terminal itself.