Build a pre-job health gate for cloud Mac CI runners

A cloud Mac that completed a build last night may still fail only after claiming its next job today because it is low on disk space, its selected Xcode path has changed, simulator state was left behind, or a child process has become stuck. By then, the pipeline has already occupied an execution slot, and the actual error may be buried under dependency installation, compiler output, and retry logs. A more reliable approach is to add an admission gate before the scheduler assigns a job: accept work only when all checks pass; otherwise, leave the queue and save a snapshot of the current state.

Put the health gate before job assignment

Admission checks should be fast, deterministic, and free of side effects. They are not a full system audit, nor should they attempt to repair the machine along the way. The gate only needs to answer five questions:

1. Does the developer tools path exist, and can xcodebuild start?
2. Does the working volume have enough free space?
3. Is memory pressure already high enough to affect a new job?
4. Can the simulator service return its device and runtime lists?
5. Did the previous job leave behind a managed process or job lock?

Runners on VMCache can be controlled by a self-hosted agent, a scheduled dispatcher, or a team’s own queue manager. Regardless of the implementation, the gate should run before the runner claims a job, not as the first step in the build script. At that point, the job already counts as a failure and pollutes success-rate and retry statistics.

The correct response to a failed health check is to stop accepting jobs temporarily and preserve evidence—not to immediately delete caches, kill every process, or restart the entire machine.

Build an auditable health-check script

The following script only reads system state and uses a nonzero exit code to reject new work. Adjust the thresholds to the size of each project rather than copying them as universal constants for every runner.

#!/bin/zsh
set -u

WORK_VOLUME="${WORK_VOLUME:-/}"
MIN_FREE_GB="${MIN_FREE_GB:-40}"
LOCK_FILE="${RUNNER_LOCK_FILE:-/tmp/vmcache-ci-job.lock}"
failures=0

fail() {
  print -u2 "FAIL: $1"
  failures=$((failures + 1))
}

developer_dir="$(xcode-select -p 2>/dev/null || true)"
[[ -n "$developer_dir" && -d "$developer_dir" ]] \
  || fail "developer directory is unavailable"

if ! xcodebuild -version >/dev/null 2>&1; then
  fail "xcodebuild cannot start"
fi

free_kb="$(df -Pk "$WORK_VOLUME" | awk 'NR==2 {print $4}')"
if [[ -z "$free_kb" ]]; then
  fail "free disk space cannot be read"
else
  free_gb=$((free_kb / 1024 / 1024))
  (( free_gb >= MIN_FREE_GB )) \
    || fail "free disk space is below ${MIN_FREE_GB} GB"
fi

if [[ -e "$LOCK_FILE" ]]; then
  fail "a previous job lock still exists"
fi

if ! xcrun simctl list devices available >/dev/null 2>&1; then
  fail "simulator service is not responding"
fi

exit $((failures > 0 ? 20 : 0))

Do not put sudo rm -rf, killall, or automatic Xcode switching in this script. The admission stage is responsible for making a decision, not changing the system under inspection. This also limits the damage if the gate itself is misconfigured.

Define stable exit-code semantics

A useful contract is to define 0 as ready to accept jobs, 20 as an environment that does not meet requirements, and 30 as a failure of the checker itself. When the scheduler receives 20, it should mark the runner as paused. When it receives 30, it should report a health-check infrastructure failure. Avoid returning 1 for every error, because that makes it impossible to distinguish a node problem from a script problem.

Check disk and memory without cleaning as you go

The disk threshold should cover the peak space needed by a single job, including source code, dependencies, DerivedData, archives, and temporary files. Derive it from the peak usage of successful historical jobs, then add enough headroom for rollback. Do not inspect only the root volume. If the workspace, caches, or simulator data live on other APFS volumes, run df -Pk against each one.

Memory checks should not rely only on “free memory.” macOS actively uses memory for file caching, so sustained memory pressure and swap activity are more meaningful signals. A lightweight gate can save the output of memory_pressure and vm_stat, allowing the monitoring system to compare trends:

snapshot_dir="${RUNNER_STATE_DIR:-$HOME/ci-state}"
mkdir -p "$snapshot_dir"

{
  date -u "+%Y-%m-%dT%H:%M:%SZ"
  xcode-select -p
  xcodebuild -version
  df -h /
  memory_pressure
  vm_stat
} > "$snapshot_dir/preflight-latest.txt" 2>&1

Do not reject a job based on a single page-count reading. A more reliable rule is to record the state at the gate and block new work only for conditions already known to cause the current workload to fail. For MLX inference or large linking jobs, define unified-memory requirements in a separate policy for the corresponding queue instead of forcing every job to share one threshold.

Detect leftover jobs without killing system processes

Leftover-process detection must be scoped by process ownership. Globally terminating processes based only on names such as xcodebuild, swift, or Simulator can kill an interactive session or another executor’s job. A safer approach is for each job to create a lock file containing its PID, job ID, and start time, then remove it from an exit trap.

job_lock="${RUNNER_LOCK_FILE:-/tmp/vmcache-ci-job.lock}"

cleanup() {
  rm -f "$job_lock"
}

if ! ( set -o noclobber; print "$$ ${CI_JOB_ID:-unknown}" > "$job_lock" ) 2>/dev/null; then
  print -u2 "another managed job owns the runner"
  exit 20
fi

trap cleanup EXIT INT TERM

When an old lock is found, first verify whether its PID still exists, then check the process start time and command line. PIDs are reused, so the number alone is not enough to justify terminating a process. If the process no longer exists, move the lock file and the check timestamp into a diagnostics directory, then clear the blockage through a controlled recovery step.

Set boundaries for simulator checks

A successful simctl list call only proves that the service is reachable; it does not mean a specific test target is available. The gate should validate the service layer, while the project should look up the target device within the job using an explicit destination. Do not automatically create, delete, or erase devices in the admission script. Those operations are slow and alter the baseline for subsequent tests.

Integrate the gate with the scheduler and design a recovery path

The scheduler should first mark the runner as “checking,” run the script, and then atomically transition it to either “ready” or “quarantined.” If the agent cannot provide atomic state transitions, pause job claims before running the check and resume them afterward. This prevents a new job from arriving while the check is in progress.

Layer recovery actions by risk:

1. Old lock with no corresponding PID: archive the lock file, then clear it.
2. Rebuildable directories exceed the storage budget: verify that no job is active, then clean them according to directory ownership.
3. Simulator service is unresponsive: save diagnostics, then run the team-approved service recovery procedure.
4. Xcode path is incorrect or disk state is abnormal: keep the runner quarantined for manual review.
5. The same failure occurs repeatedly: stop automatic recovery and retain the latest snapshots for comparison.

Finally, run a failure drill for the gate itself. Temporarily raise the minimum disk threshold and confirm that the runner leaves the queue. Restore the threshold and verify that it becomes available again. Then create a stale lock file and confirm that the system quarantines only the node without killing unrelated processes. The value of a health check is not that it catches every possible failure, but that failures happen before job assignment and leave a clear path to the next action.

Frequently asked questions