Skip to main content

Memory pressure

Admission keeps most trouble out, but jobs grow. turnstile responds in three steps.

Watch swap, not just free memory​

Free memory on macOS is kern.memorystatus_level, and the kernel holds it up by compressing and paging out. Once that starts, the level measures the stand-off rather than the room a new job has: a machine can sit at 30% free while it writes gigabytes to swap.

So turnstile also watches how fast swap is growing. If it grows by more than 256 MB in 20 seconds, the machine counts as swapping, and four times that, or a critical kern.memorystatus_vm_pressure_level, counts as critical. While either holds, nothing new starts, parallelism drops to its lowest setting, and jobs pause. Nothing is admitted until swap has been quiet for the rest of the window.

How much swap is in use is not a signal. A machine that has been busy sits on gigabytes of cold swap and reads "warn" all day with plenty of room; only growth means pages are going out right now.

An idle machine is the exception: the head of the queue always starts, since waiting can't free memory that nothing is holding.

Lower parallelism​

When memory runs low, turnstile lowers each tool's parallelism by injecting --jobs, --maxWorkers, CARGO_BUILD_JOBS, and Node's heap cap. Anything you pass yourself wins.

By default it leaves parallelism alone until free memory drops under 25%, then halves it, and quarters it under 15%. Node's heap is capped at 2 GB under 15% free. See throttle to change this.

Pause the newest agent job​

If free memory drops below pauseBelow (8% by default), or the machine is swapping whatever the level says, the newest agent compile job is paused with SIGSTOP. It resumes once free memory is back above resumeAbove (20%) and swap has been quiet for 30 seconds. Each time pressure pauses the same job, that wait doubles, up to 8 minutes, so a machine that stays over-committed lets the running job finish instead of stopping and starting the other one every few seconds. Your own jobs are never paused. Test and browser runners aren't paused by default either, since their timeouts keep counting through a pause. A project can change that with "throttle": {"pause": true}, or opt out entirely with "throttle": {"pause": false}.

One job always keeps running, so work still progresses and the pauses can't deadlock.

Stop runaways​

A job past its ceiling is a runaway candidate, not something to kill outright. The ceiling is three times the command's high-water peak in this project — the highest of its last twenty runs in the past month, not its recent average, because a cold compile can be tens of times an incremental one. It never falls below killFloor (25% of RAM), and with no history at all it's 75% of RAM.

What happens next depends on the machine, because memory a job isn't taking from anyone costs nothing:

  • Plenty free. The job keeps running. It's told once that it's using more than usual, and the daemon log records it.
  • Free memory under pauseBelow, or the machine swapping. The job is paused with SIGSTOP if it's one that can be paused, like any job paused for memory, and resumes once memory recovers.
  • Still under pressure 15 seconds later. Pausing didn't help, so the job is killed: SIGTERM, then SIGKILL after 5 seconds. The reason says it was turnstile's memory guard and that retrying won't help unless the run needs less memory.

maxMemory is the exception. It's an explicit limit someone asked for, so a job tree above it is killed straight away, whatever the machine is doing.