Deterministic Simulation Testing for Distributed Systems

Distributed systems bugs are notoriously hard to reproduce: a message arrives a millisecond late, a disk write is reordered, a node restarts mid-commit. Deterministic simulation testing (DST) attacks this by running the entire system inside a single-threaded simulator where every source of nondeterminism is controlled by one random seed.

The core idea

• One thread, one scheduler: all "nodes" run as tasks in a single process; the simulator decides which runs next.
• Fake time: clocks only advance when the simulator says so, so timeouts can fire in any order instantly.
• Fake network and disk: the simulator delivers, delays, duplicates, reorders or drops messages and can fail or corrupt writes.
• Seeded randomness: every choice comes from a PRNG seeded per run. Same seed, same execution, every time.

When a run violates an invariant, you rerun the same seed under a debugger and step through the exact failing history.

Why it finds bugs other tests miss

Because simulated time is free, you can run thousands of years of cluster time overnight, injecting faults far more often than production would. Rare interleavings that a staging cluster might hit once a year show up in minutes.

Who uses it

FoundationDB built its engine around simulation from the start; TigerBeetle, and others, have since popularised the approach, and libraries exist for Rust (madsim, turmoil) and other ecosystems.

What it requires from your code

1. All I/O, time and randomness go through interfaces the simulator can replace.
2. No hidden threads or unmocked system calls.
3. Strong invariants and assertions: the simulator can only flag bugs your checks can see (for example, linearizability checking of client histories).

Trade-offs to mention

• Large upfront design cost; retrofitting an existing codebase is hard.
• The simulator's fault model must match reality, or you'll miss real-world failures (like partial disk writes) that it never produces.
• It complements, rather than replaces, Jepsen-style black-box testing on real clusters.

Interview framing: "I'd make the system deterministic under a seeded simulator so every concurrency bug becomes a reproducible test case."

Disclosure: I work on Linkjob AI, an AI interview assistant. These notes are general study material.