CPU transformer checkpoint recovery after real process kills. Four pinned public-data drills match an independent control; GPU and serving remain planned.
Deterministic replay of linked tests—not live telemetry.
Expected test outcome
After a real step-two SIGKILL, recovery from durable step one reaches the same six-step state as an independent control, with all 16 exact checks passing.
Replay of a pinned local CPU report, not live training. The killed worker pauses at a known boundary; this does not prove arbitrary kill timing, power-loss durability, GPU recovery, or distributed training.
01Run the control pathAn independently spawned control trains the tiny configuration for six deterministic CPU steps on pinned PEP data: 384 input tokens across 12 sampled windows.input
02Kill an uncommitted stepA second worker durably saves step one, pauses after the step-two optimizer update, and is killed by the parent with SIGKILL; the report requires exit code -9.failure
03Restore and replay the failed batchA fresh replay process loads durable generation one, restores model/optimizer/RNG/cursor state, consumes the identical failed batch and sample IDs, and durably commits step two.recovery
04Finish in another fresh processA fourth spawned worker loads the replayed checkpoint and completes the remaining four steps without inheriting in-memory training state.processing
05Require exact final-state equalityAll 16 report checks must pass, including every model/optimizer tensor, RNG state, histories, cursor, counters, logits, checkpoint state, fingerprints, and staging cleanup.verified
Animation is paused. The Previous and Next controls keep every step available.
Checkpoint recovery: step 1 of 5. Run the control path. An independently spawned control trains the tiny configuration for six deterministic CPU steps on pinned PEP data: 384 input tokens across 12 sampled windows.
4 / 4
SIGKILL boundaries recovered on public data
Kill the worker at each declared step-two boundary, restore durable step one, replay the failed batch, and finish the six-step pinned PEP run. Each report records exit code -9, zero durable committed steps lost, and all 16 exact verification checks passing.
for point in before-forward after-backward after-optimizer during-checkpoint-write; do uv run --frozen fttl-verify-process-recovery --config configs/peps-cpu.json --dataset-manifest .cache/fttl/peps-v1/manifest.json --failure-point "$point" --output "artifacts/local-process-$point"; done
Local deterministic CPU · pinned environment in each report; CI uses ubuntu-latest · Python 3.12 · CC0 fixture
Snapshot
Limit
Known paused boundaries in a tiny six-step CPU run; not arbitrary kill timing, power loss, GPU scale, model quality, or production readiness. CI does not rerun the public corpus.
5.5%
trainable parameters in the controlled LoRA run
The same initialized model, synthetic batches, seed, and step budget were used for full tuning and LoRA; 1,536 LoRA parameters were trainable versus 28,032 in full tuning. This is a parameter-efficiency result, not a quality claim.
Local CPU artifact · Python 3.12.14 · PyTorch 2.14.0; command also runs in CI
Snapshot
Limit
Parameter-efficiency observation on a randomly initialized tiny model and synthetic tokens; not a quality comparison.
rtol 0 / atol 0
resumed model-state equality
For each public-data kill drill, compare the uninterrupted and recovered six-step model tensors exactly with torch.equal. The same reports also require exact optimizer/RNG state, loss and sample histories, cursor, counters, logits, checkpoint state, and fingerprint equality.
Local deterministic CPU · pinned public PEP corpus · six steps; CI independently checks CC0 fixture recovery
Snapshot
Limit
Exact-state equality for this tiny deterministic CPU configuration and known failure boundary; not a model-quality, GPU, distributed restart, or recovery SLO claim.
Problem
The engineering question
Small model demos often stop at a successful training curve. This project treats restart safety, experimental controls, and negative results as first-class engineering outputs.
Constraints
What the design must respect
The public path must run deterministically on CPU for review and CI.
GPU and multi-GPU results must name the actual hardware and may never be simulated as evidence.
All datasets, implementations, and result artifacts must be public-safe and independently produced.
Architecture decisions
Deliberate choices and why they exist
01
Evidence before scale
The first release proves gradient correctness, deterministic restarts, and artifact lineage before adding distributed execution.
02
One variable per ablation
Experiment manifests hold the dataset, seed, model, and environment fixed while changing one declared variable.
03
Independent recovery workers
A parent pauses a worker at a known uncommitted step-two boundary and sends SIGKILL. Separate spawned control, replay, and completion processes prevent accidental inheritance of model, optimizer, or RNG state.
04
Public-data proof, network-free CI
Local evidence uses the pinned public PEP corpus. CI exercises the same verifier with an independently written CC0 fixture, so it checks recovery semantics without downloading or claiming to rerun the corpus proof.
Failure modes
Designed to fail visibly
SIGKILL at before-forward, after-backward, after-optimizer, and during-checkpoint-write boundaries, with durable step one restored after attempted step two.
An abandoned checkpoint staging file after mid-write termination; the next successful save removes it without treating it as a committed generation.
Checkpoint integrity or configuration, dataset, tokenizer, and run-contract mismatch during restart.
Worker startup/completion deadlines and error cleanup, without leaving a child process running.
Causal-mask or gradient regressions caught by focused tests.
Verification
How claims are checked
Finite-difference and framework gradient checks.
Repeatable CPU smoke training in CI.
Four actual parent-controlled SIGKILL drills on pinned public PEP data: six steps, 384 input tokens, and 12 sampled windows per control/recovered run.
Sixteen verification checks require exact model and optimizer tensors, RNG state, loss/batch/sample histories, cursor, counters, logits, checkpoint state, and fingerprints, plus staging cleanup.
Network-free CI uses the CC0 fixture for all four kill boundaries, deadline cleanup, dataset drift rejection, and CLI behavior; it is separate from the local public-corpus reports.
Limitations
What remains unproven
No public multi-GPU result is claimed yet.
The tiny CPU proof consumes 12 windows from 12 documents, not every document in the 536-document training split or 656-document manifest.
SIGKILL is sent while the worker is paused at a known boundary; arbitrary asynchronous kill timing remains unverified.
The OS and filesystem stay alive. This is not a power-loss durability proof, a model-quality result, distributed training, or production-readiness evidence.
Local startup, restore, replay, and IPC timings are not throughput benchmarks or recovery SLOs.
The public demo is an artifact explorer, not an always-on inference service.
Inspect the work
Source and evidence
The repository is the source of truth for implementation status. The case study never upgrades a planned feature into a shipped claim.