Deterministic replay of linked tests—not live telemetry.
Expected test outcome
A helper exits after commit but before acknowledgement. Redelivery leaves one accepted, completed job, one result, and one accounted duplicate.
Real PostgreSQL/RabbitMQ test-helper exit at a known boundary—not arbitrary SIGKILL timing or live telemetry.
01Accept durable workThe fixture commits one job and its outbox event, then publishes its stable identity through RabbitMQ.input
02Commit result and completed statusThe helper locks the job, validates it, and commits the result and terminal status in one PostgreSQL transaction.processing
03Exit before acknowledgementThe AfterCommit hook exits the real helper process with code 86, leaving the committed effect but an unsettled delivery.failure
04Absorb redeliveryThe next processor sees the completed job under its row lock, records the duplicate, and acknowledges without inserting a second result.recovery
05Assert durable accountingThe test requires one accepted job, one completed job, one result, and exactly one duplicate.verified
Animation is paused. The Previous and Next controls keep every step available.
Commit before ack: step 1 of 5. Accept durable work. The fixture commits one job and its outbox event, then publishes its stable identity through RabbitMQ.
12 / 12
external-path trials fully accounted
At clean measured source 6b01125, each of 1/2/4/8 worker-count groups runs 100 warmups followed by three 1,000-job trials. All 12,000 measured jobs complete; each isolated schema retains 3,100 accepted/completed jobs and results, no pending outbox, and no normal-load retries or duplicates. The reproduction command includes the full source pin; use the pinned README's service environment.
Local Darwin/arm64 · Go 1.26.4 · 11 logical CPUs · isolated Docker Desktop PostgreSQL/RabbitMQ · separate Go processes
Snapshot
Limit
Single-host synthetic accounting, not production capacity. Original runs did not capture executable hashes; later matching-build metadata cannot prove historical runtime bytes. CI validates recovery contracts separately.
262.73 – 323.21
jobs/s across all twelve local trials
Range over every measured 1,000-job trial with 1, 2, 4, or 8 independent workers. Throughput includes HTTP submission and polling through committed-result observation. The raw samples retain all three trials per group, without choosing only the fastest.
Local Darwin/arm64 · Go 1.26.4 · 11 logical CPUs · RabbitMQ/PostgreSQL in isolated Docker Desktop services
Snapshot
Limit
Lightweight synthetic validation on one host. The sequential HTTP producer and confirmed dispatcher can bottleneck load; these results do not demonstrate linear worker scaling or cluster capacity. Resource sampling excludes broker/database/VM costs.
95.43 – 151.08
ms observed p95 across trials
Each trial's p95 uses the floor of (n−1)×0.95 over sorted client-monotonic latencies. Timing begins immediately before HTTP submission and ends when a query first observes the committed completed-job/result join. The displayed range is over all twelve trials.
Local Darwin/arm64 · Go 1.26.4 · client observation polling approximately every 50 ms plus query/resource sampling and scheduling
Snapshot
Limit
Includes acceptance, delivery, processing, result commit, and observation overhead. Not isolated SQL commit duration; no fixed upper polling delay under load. Original pre-commit SQL timestamps are not used for this claim.
3.402 s
broker-restart observation · one sample
The corrected receipt records 3.401676166 seconds from Compose restart invocation through health wait and the outage job's committed-result observation. Final durable accounting is two accepted/completed jobs and two results, with no pending/retrying/failed work and one deliberately replayed duplicate.
Local Darwin/arm64 · Go 1.26.4 · separate dispatcher/worker binaries · actual isolated Compose broker stop/restart
Snapshot
Limit
One local sample including Compose startup/health, not a recovery SLO, database failover, or production result. The demo's later 5.119233-second restart is a different trial and is not substituted for this receipt.
Problem
The engineering question
Durable acceptance and message settlement are separate failure boundaries. The lab makes missing publication, duplicate delivery, process exit, broker interruption, retries, and database errors inspectable using public synthetic inputs.
Constraints
What the design must respect
The default test path must require no external services.
Local adapter verification must never be described as a production deployment or production benchmark.
No employer code, data, schemas, identifiers, or benchmark results may enter the repository.
Architecture decisions
Deliberate choices and why they exist
01
Transport-neutral contracts
Queue and store interfaces allow deterministic in-memory tests before RabbitMQ and PostgreSQL integration tests are enabled.
02
Stable identity
Every synthetic message receives a reproducible identifier so retries can prove idempotent behavior.
03
Failure as input
Validation and persistence failures are injected explicitly rather than inferred from flaky environments.
04
Atomic job and publication intent
POST /api/v1/jobs commits the canonical payload, stable idempotency key, and outbox event together. The dispatcher locks one due row with SKIP LOCKED and marks it sent only after broker confirmation and transaction commit.
05
Commit before acknowledgement
Workers commit result and terminal status together before acknowledging. Validation failures commit retry/dead events; SQL failures roll back and requeue without consuming the validation attempt budget.
06
Bounded work and visible accounting
Admission returns 429 at its unfinished-job capacity; worker prefetch is one and database pools are bounded. PostgreSQL-backed gauges expose accepted, pending, retrying, completed, failed, outbox, retry, and duplicate state.
Failure modes
Designed to fail visibly
Redelivery after a worker acknowledges too late.
Duplicate writes after a persistence retry.
Poison messages exhausting bounded retries.
Actual fault-helper subprocess exits before commit, after commit-before-ack, and after dispatcher publication; known injected boundaries, not arbitrary SIGKILL timing.
A live broker outage leaves accepted jobs and outbox state pending until independent dispatcher and worker processes reconnect.
Database/context failures roll back processing; AMQP startup obeys cancellation and timeout even when the peer stalls before handshake.
Verification
How claims are checked
Go unit and concurrency tests.
Deterministic workload generation from a fixed seed.
Equal generated and stored counts for successful runs.
Restricted Kubernetes workload security context.
Real-service integration covers idempotency/conflict/capacity, retry exhaustion, dead queues, SQL rollback, subprocess exits, broker restart, and graceful independent-process shutdown.
Corrected observed-result measurements retain every worker-count trial, all raw latencies and queue/resource samples, exact dependency image references, and the reported clean source commit.
Limitations
What remains unproven
External-path measurements are a single-host synthetic lab with RabbitMQ/PostgreSQL in Docker Desktop, not a distributed cluster or production capacity claim.
Observed latency spans client submission through the first observed committed result and includes approximately 50 ms polling, SQL/resource sampling, and scheduler overhead; it is not an isolated database commit duration.
RSS/CPU cover API, dispatcher, and workers only. Broker, database, and Docker VM resources are excluded; samples can miss peaks and ps CPU percentages are lifetime averages.
Recovery timing includes Compose startup and health checks. Known helper exit boundaries and one broker restart do not prove arbitrary crash timing, power loss, database failover, or a recovery SLO.
The unauthenticated localhost API is a single-tenant lab. Authorization, retention/redrive policy, and HA deployment are not implemented.
Historical 10,000-job evidence remains a single-process memory-adapter functional run. The original SQL-timestamp benchmark is superseded by corrected observed-result receipts.
The v2 measurement runs recorded source and clean state but did not capture runtime binary hashes. A later matching-build provenance check cannot retroactively prove the executed binary bytes.
The 90-second video replays actual agent-executed terminal output, with no learner narration. Its 5.119233-second broker restart is a separate trial from the v2 receipt's 3.401676166 seconds; both include Compose health wait.
Inspect the work
Source and evidence
The repository is the source of truth for implementation status. The case study never upgrades a planned feature into a shipped claim.