The Supervisor That Gives Up
Clean shutdown assumed every worker eventually finishes. The next probe made one worker deliberately hang — and the collector, blocked on a receive that will never arrive, hung the whole run with it. What a supervisor needs is the right to give up: a receive with a deadline, which waits up to N milliseconds and returns None instead of waiting forever. One stuck job out of six now costs one timeout entry in the ledger, not the program.
The last episode taught channels to end. That covers the case where workers finish their jobs and drain out — the cooperative ending. This episode is about the uncooperative one: the worker that neither finishes nor fails, but simply takes forever. Every distributed system and every thread pool eventually meets this worker, and what the meeting reveals is which side of the receive the language has been protecting.
One stuck job
The probe reused the worker pool and sabotaged it: six jobs, and job number three sleeps
two seconds while everyone else takes ten milliseconds — a stand-in for the network call
that never returns, the disk that went away, the loop that found an edge case. The
collector’s logic was the obvious channel_recv results six times. Five results arrive
promptly. The sixth never does, and the collector blocks on it — forever. Everything
about the previous episode’s machinery is working as designed: the jobs channel closed
cleanly, five workers drained and returned, joins would succeed. But the program as a
whole hangs on one blocking receive, held hostage by its slowest member. Closing channels
can’t fix this — nobody knows yet that the job is stuck, so nobody knows to close
anything. What the collector lacks is not information. It is the right to give up.
A receive with a deadline
channel_recv_timeout takes a channel and a millisecond budget, blocks up to that
long for a value, and returns None on timeout — or once the channel is closed and
drained, so it also subsumes the shutdown behavior of the untimed variant. The collector
becomes a loop that tries six times with a 300-millisecond budget each: five Some, one
None, and the run ends with a ledger — results=5 timeouts=1 — instead of a hang. On
the C backend the implementation is the textbook one, a pthread_cond_timedwait against
an absolute deadline; the reference interpreter polls at millisecond granularity, because
OCaml’s standard library has no timed condition wait, and the note in the source says so
plainly. One more piece of the ending problem turned out to already be solved: once a
supervisor notices a stuck worker, the natural next move is to cancel the rest of the
batch — and that is just channel_close on the jobs channel, from the first episode. No
new primitive needed. Which raised the obvious question: the classic concurrency toolkit
has one more famous primitive this language lacks — waiting on several channels at once.
The next episode goes looking for the program that needs it, and comes back with
something better than the primitive.