branch-and-checkpoint

Checkpoints, restores, pauses and branches

Verified on smolvm v1.23.0 on macOS arm64, 2026-10-04, and on v1.18.2 on Linux aarch64, 2026-09-24. Done means a restore brings back both the disk and the memory you left, the schedule keeps exactly the checkpoints you asked for and no fewer, and every child of a branch starts from a state you still hold as a checkpoint after the children are gone. The Linux runs used the scripts of their date; this version's preflight and cleanup scripts ran on Linux aarch64 on v1.22.2 on 2026-10-03.

README.md says what each of these operations is. This file is how to run them, and the traps.

Three facts shape everything here.

• smolvm has no scheduler and no retention. scripts/checkpoint.sh is the one capture a timer runs, with the rules the engine asks a scheduler to follow built in; references/scheduling.md puts it under cron, systemd or launchd.
• A branch's captured state stays on this host and cannot be exported. So a branch point is paired with a checkpoint taken at the same moment, which is the copy you keep.
• --branchable at start decides what a machine can do later. A branch source needs it on every host. On macOS a stored capture, which is what the scripts here take, needs it too, and the error when it was not, deferred durable save requires file-backed guest RAM on v1.22.2, names neither the flag nor the operation. Before v1.20.0 macOS also refused a checkpoint file and a pause without it, with guest RAM has no file-backed regions. Linux checkpoints and pauses without it. The scripts always start with it.

Procedure

- [ ] 1. preflight.sh               read result= and store_space_ok=
- [ ] 2. create-source.sh           a machine with state in RAM and on disk
- [ ] 3. checkpoint.sh / schedule.sh   captures, with retention
- [ ] 4. restore.sh                 any generation, into a new machine
- [ ] 5. pause and resume           the restored machine, or any branchable one
- [ ] 6. branch-point.sh            children plus a checkpoint of where they started
- [ ] 7. cleanup.sh                 machines, checkpoints, and smolvm's restore cache

1. Preflight. Read-only.

scripts/preflight.sh --store ./store

It checks that this binary has --store, --history, checkpoint-log, pause and restore --at, says whether this host needs --branchable, and reports scheduler=builtin_none and retention=builtin_none so nobody goes looking for them. store_space_ok=no blocks: a capture into a full disk publishes nothing, and a machine restored while the disk was full was left with guest I/O errors that stopped it being deleted normally.

2. A source with state you can check.

scripts/create-source.sh smolskill-src                 # for checkpoints and restores
scripts/create-source.sh smolskill-bp --branch-ready   # for a batch branch

The workload writes a counter into /tmp every second, which is tmpfs and so RAM, and a marker to /root/setup.txt on disk. A restore that brings the counter back proves memory came back, not only the disk. --branch-ready makes the workload park in smolvm-branch-ready after its setup, which a batch branch waits for.

Or the user's own machine. Name it with --name in the steps below. On macOS, if it was started without --branchable, the first capture fails with deferred durable save requires file-backed guest RAM on v1.22.2 or guest RAM has no file-backed regions before v1.20.0, and the only fix is a restart with the flag:

smolvm machine stop  --name worker
smolvm machine start --name worker --branchable

That restart is a normal stop and start: disk state survives, and anything in RAM or /tmp is lost, once. Say so to the user before doing it. Every start of that machine needs the flag from then on for it to stay checkpointable.

A capture also refuses a machine with resources bound to this host: a host mount (-v), a published socket, a remote volume, a host secret reference, custom DNS, a named network, a GPU, CUDA, Rosetta, --ssh-agent or a Docker socket. The error, the initial portable checkpoint profile does not support ...; stop or detach these resources before capture, names which.

3. Capture, once or on a schedule.

scripts/checkpoint.sh --name smolskill-src --store ./store --keep 6 --history 2
scripts/schedule.sh   --name smolskill-src --store ./store --every 10 --times 4 --keep 2 --history 2

Each capture goes to a new directory named <machine>-<label>-<UTC time>.smolcheckpoint beside the store, or in the directory --out <dir> names, which has to be on the same filesystem as the store. It is judged published only when checkpoint-log lists it as (this checkpoint), and only then are the oldest beyond --keep deleted and the store pruned. A capture started while another of the same machine runs exits 3 with result=skipped. schedule.sh is for a session you are watching; a schedule that outlives it is checkpoint.sh under a system timer, references/scheduling.md.

run=1 rc=0 took_s=2 ... kept=1 store_size= 59M result=captured
run=2 rc=0 took_s=5 ... kept=2 store_size= 76M result=captured
run=3 rc=0 took_s=2 ... kept=2 store_size= 95M result=captured
run=4 rc=0 took_s=1 ... kept=2 store_size=114M result=captured
captures_ok=4
captures_failed=0
longest_capture_s=5
result=schedule_ok

That is the v1.18.2 run. Ten seconds is for watching it work; a schedule uses minutes, references/scheduling.md.

Set the interval from longest_capture_s, not from the pause. The source pauses for a few hundredths of a second; the command takes seconds, because retained RAM is hashed and compressed after the source resumes. --keep alone does not bound the disk: every kept checkpoint also retains --history earlier generations, 32 by default, so the store above kept growing with two directories kept. Bound both.

4. Restore, any generation.

smolvm machine checkpoint-log ./smolskill-src-ckpt-<time>.smolcheckpoint
scripts/restore.sh --from <checkpoint> --name smolskill-old --at '~2'
scripts/restore.sh --from <checkpoint> --name smolskill-new

~0 is the checkpoint itself and ~N is N generations back along its history. restore.sh takes only a name starting with smolskill-, so cleanup can find it, and refuses any other with name must start with smolskill- so cleanup.sh will delete it; to restore under a name the user chooses, run smolvm machine create --name <name> --from <checkpoint> --at '~N' and then smolvm machine start --name <name> --branchable yourself. The restore path is machine create --from; there is no machine restore. A restored machine takes its own name as its hostname, and its disks are copy-on-write over what the checkpoint holds, so a restore costs about 29 MB of real disk on macOS for a 243 MiB checkpoint while du reports five times that.

5. Pause and resume.

smolvm machine pause  --name smolskill-new
smolvm machine resume --name smolskill-new

Pause saves RAM, disks and the running execution and stops the machine; resume brings the same execution back under the same name. Check a value the workload holds in memory: on both hosts the counter read 31 before the pause and 34 and 33 after the resume, so it continued where it stopped and did not run while paused. A paused machine refuses stop and start; resume it or delete it. A resume that fails keeps the saved state, and a later resume of the same machine succeeded.

6. Branch, and keep the branch point.

scripts/branch-point.sh --from smolskill-bp --store ./store --count 2 --name-prefix smolskill-w
scripts/branch-point.sh --from smolskill-src --store ./store --name smolskill-one

For a batch it waits until the source's PID 1 is smolvm-branch-ready, takes the checkpoint while the source is parked there, then branches. Every child and that checkpoint then hold the same state: a random value the source wrote to RAM before parking came back identical in both children and in a restore of the checkpoint.

source_parked_after_s=1
checkpoint=.../smolskill-bp-branchpoint-<time>.smolcheckpoint
child=smolskill-w-0
child=smolskill-w-1
source_state=running
result=branched

For one child with --name the checkpoint is taken first and the branch a second or two later, so the two points differ by whatever the source did in between.

A restored branch point is not a parked source. On start the helper releases and the workload runs the child program with an empty SMOLVM_BRANCH_NAME, so a batch branch from it waits for a branch point that never comes. Branch it with --name, one child at a time, or restore it once per worker; either way the child gets the state the original children started from.

7. Clean up.

scripts/cleanup.sh --purge --restore-base --checkpoints ./store ./*.smolcheckpoint

It deletes the machines the scripts recorded, children first by --cascade, removes the named checkpoint directories and stores, and with --restore-base removes smolvm's copies of restored checkpoints, which otherwise stay after every machine and checkpoint is gone, printing a removed= or absent line for each. It waits up to 20 seconds, polling the machine list, and says so, because an ephemeral entry retires after its run returns.

Traps

Full detail in references/traps.md. The ones that cost the most:

• --branchable is decided at start and cannot be added later. Branching needs it everywhere; on macOS a stored capture needs it as well.
• --output must end in .checkpoint or .smolcheckpoint, with --store too, where it names a directory. .checkpoint is the name from v1.19.1, and the scripts use .smolcheckpoint, which every release accepts.
• pgrep -f smolvm-branch-ready inside the guest always matches, because the sh -c running it contains the string. Read /proc/1/cmdline.
• smolvm machine list | grep -q under pipefail can report a machine missing that exists: grep -q closes the pipe early and the failed write fails the pipeline. Read the list into a variable first.
• machine stop --name on a name that does not exist leaves an empty directory under the VM cache on v1.18.2, and a second stop of it on v1.22.2 leaves the name inside, which reads as a leak. cleanup.sh stops only names still listed.
• smolvm keeps restored checkpoints in its VM cache, memory included: vms/_restore-base on macOS, 243 MB on v1.18.2, and from v1.22.0 vms/_restore-checkpoints, the last three restored, 258 MB after this procedure on macOS on v1.23.0. Deleting the checkpoint does not delete them.

Security defaults, and why they are the defaults

• A checkpoint is the machine's memory and disks. Anything the workload held, secrets included, is in it; keep stores on a disk only you can read, and remove _restore-base and _restore-checkpoints when you remove the checkpoints.
• Retention deletes only after the new checkpoint is published, so a failed capture never leaves you with fewer good checkpoints than before.
• A store is local. It protects against a bad change, not a lost host; export a checkpoint with machine checkpoint --export-from and copy it elsewhere for that.
• Cleanup deletes only machines the scripts recorded under the smolskill- prefix, and through --cascade any machine branched from one of them, whatever its name, and removes only the checkpoint paths you name.
• Nothing here escalates privilege or edits smolvm configuration.

Platform arms

references/platforms.md has each arm. In short: macOS arm64 ran every step here on v1.23.0, and Linux aarch64 on v1.18.2 and once on v1.22.2, with the one difference that Linux needs no --branchable. On v1.16.1 Linux aarch64 refused --store and froze a branch source; from v1.18.2 it does neither. Linux x86_64 was not re-run. Windows on v1.22.2 checkpoints to a file, pauses and branches with --freeze-source on every branch, and refuses --store, so checkpoint.sh cannot run there. From v1.23.0 a checkpoint from an arm64 Mac restores on arm64 Linux, run once from this Mac to Lima.

Eval prompts, and what they produced

Run 2026-09-24 PT against v1.18.2 from the published release, under an isolated HOME, on macOS 26.6.2 arm64 and Lima linux-kvm (Ubuntu 24.04 aarch64), every machine at 1024 MiB. Output is verbatim unless marked.

1. "Checkpoint this machine every few minutes, keep only the last few, and show me I can go back to any of them."

schedule.sh with --keep 2 --history 2, four captures ten seconds apart, both hosts: captures_ok=4, kept=2, and checkpoint-log on the newest listed ~0, ~1 and ~2. A marker written between captures came back as GEN1 from --at '~2' and GEN3 from the newest, with the RAM counter restored alongside. A second capture started while one ran:

result=skipped
note=a capture of smolskill-src is still running (pid 13935). Run one capture at a time per source: ...

2. "Fan this prepared machine out into workers, and keep a copy of exactly the state they started from."

branch-point.sh --count 2 on a --branch-ready source, macOS: the source wrote 8445 to RAM before parking; both children read SETUP_DONE, their own CHILD= name and 8445, and a restore of the branch-point checkpoint read 8445. Linux the same with 22642. source_state=running on both.

3. "Pause the machine I restored and bring it back later without losing what is running."

The restored machine's counter: ram before=31 after=34 on macOS and ram before=31 after=33 on Linux, across a 10 s pause. Separately, on macOS a restored machine paused while the host disk was full failed its first resume with extract paused checkpoint: failed to unpack ..., and the same machine resume succeeded once space was freed: counter 8 before the pause, 14 after, still counting.

Re-verified on v1.23.0

Run 2026-10-04 PT against v1.23.0 from the published release, checksum checked, under a fresh isolated HOME on macOS 27.0.1 arm64, once. On Lima linux-kvm (Ubuntu 24.04 aarch64) the checks named below ran once, so the Linux stamp stays on its earlier release.

macOS: every script. Three scheduled captures (captures_ok=3, longest_capture_s=1), checkpoint-log listing three generations, restores of ~2 and the newest, the counter at 20 before a pause and 24 after the resume, both children of a batch branch holding the source's token, and cleanup.sh --restore-base removing 258 MB of _restore-checkpoints. A capture now reports when the source resumed, Source resumed after 0.024s; packaging continued in the background (#1519).

A checkpoint taken on this Mac restored on Linux aarch64 (#1531). create-source.sh, then machine checkpoint --name smolskill-mac --output mac.checkpoint on the Mac with the RAM counter at 6, which printed 40 MiB written and 0.066s source pause. Copied to Lima unchanged, machine create --from and machine start there took a second each; the counter read 7 and five seconds later 12, the RAM token and the disk marker matched the Mac's, and the guest said Linux 6.12.95.

Re-verified on v1.22.2

Run 2026-10-03 PT against v1.22.2 from the published release, checksum checked, under an isolated HOME on macOS 27.0.1 arm64, twice, the second time from a fresh HOME. On Lima linux-kvm (Ubuntu 24.04 aarch64) on 2026-10-03 guests above 2048 MiB timed out, so the Linux lines below are a single run and the Linux stamp stays on its earlier release.

macOS: every script. Three scheduled captures (captures_ok=3, longest_capture_s=1), checkpoint-log listing three generations, restores of ~2 and the newest, the counter at 21 before a pause and 25 after the resume, both children of a batch branch holding the source's token, and cleanup.sh --restore-base removing 275 MB of _restore-checkpoints. The --branchable facts in "Branchability is decided at start" were measured on releases v1.18.2 to v1.22.2 on 2026-10-03.

Linux aarch64: every script, with checkpoint_needs_branchable=no, the counter 63 then 66 and both children holding the same token.

What was not run

• Linux x86_64, and the scripts on Windows, which are bash.
• A schedule over hours. Four captures ten seconds apart on each host, and single captures within each run; the growth of a store under --history 32 over a day was not measured.
• Restores on another host, beyond Mac to Linux. Through v1.22.2 a checkpoint restores only on a host of the same platform, and another is refused with checkpoint host platform '...' does not match '...'. From v1.23.0 one taken on an arm64 Mac restores on arm64 Linux, run once above; Moving a running machine from a Mac to Linux has the requirements. Linux to Mac, and a stored capture rather than a file, were not run.
• GPU and CUDA machines, which a capture refuses, and --share-weights.
• Branch pools (--hold, branch-release), and --freeze-source outside Windows.
• The API's restore route, the from field on create; the local-api packet covers pause and resume over HTTP.

Related packets

• dev-env for state that only has to survive a stop and start.
• pack for moving a machine to another host as a file, including a restored one.
• local-api for pause and resume over HTTP.
• teardown for the wider cleanup, and for what a leak check must exclude.