Skip to the content
Diaspar

Decision D016 · accepted

Tests and benchmarks are part of the language from the start

A record of the language's design, as it stood on 2026-11-18: written for the people who build Diaspar, and kept as it was written. Its code is a sketch, not a program of the site, and an older record may show syntax that a later one changed. It names files of the language's source, which is not public yet.

Status: accepted (2026-11-15)

Amended 2026-11-18: the bootstrap interpreter is deleted (stage C, tag last-rust-interpreter). Tests and benchmarks are compiled, with the runners std.test.run and std.bench.run; what the interpreter gave (each case's expect form, the name-resolution snapshots, its runs of the language's tests) stays as recorded data, the reference compiled code and the evaluator are checked against.

Decision

Tests and benchmarks are written in Diaspar, next to the code, and run by the toolchain. They come with standard modules, std.test and std.bench, and commands, diaspar test and diaspar bench.

(deftest parses-ints
  (is (= 42 (parse-int "42")))
  (is (match? (Err failure: _) (attempt (parse-int "x")))))

(defbench sort-10k
  (sort (random-vector 10000)))
  • is reports which part of a pattern or comparison failed, not just "false" (D015, D014). A failed is stops its test, as in pytest and Rust; the other tests go on.
  • Tests are deterministic through effects (D005): time, randomness, and the network are replaced by handlers; tasks run under the sim scheduler (D035).
  • Benchmarks report the median and spread, compare with the previous run, and keep the compiler from discarding the measured computation.
  • In the REPL, the tests of the current module can be rerun after a redefinition.
  • The bootstrap interpreter (D012) gets minimal deftest and is, so Diaspar code is tested in Diaspar from the start (done; since stage C, compiled code has them, and the interpreter is gone).

Where tests live

Place What it is for Sees
deftest inside a module Unit tests next to the code, as in Rust, Zig, and Ferron The module's private items
tests/ Integration tests, each file its own module, as Rust's tests/ and Go's package foo_test Only the public API
benches/ Benchmarks of the public API Only the public API

There are no sibling test files such as Go's foo_test.go: a module's own deftests cover its private code, and tests/ covers its public API.

deftest and defbench forms are compiled only for diaspar test and diaspar bench, never into release builds. Dependencies needed only by tests are declared separately, with the package system.

Commands

As convenient as pytest and cargo test. A test's path is its module path plus its name (D011): geom.shapes.area-of-circle.

diaspar test                          # every test of the package
diaspar test geom                     # tests whose path contains "geom"
diaspar test geom.shapes.             # one module and its submodules
diaspar test 'geom.*.parse-*'         # a glob
diaspar test --exact geom.shapes.area-of-circle
diaspar test circle rect              # several filters: a test matching any of them

diaspar test --tag slow               # only tests tagged :slow
diaspar test --skip-tag slow          # all but those
diaspar test --unit / --integration   # only deftests in modules / only tests/

diaspar test -x                       # stop at the first failure
diaspar test --failed                 # rerun what failed last time, as pytest --lf
diaspar test --watch                  # rerun affected tests when a file is saved
diaspar test --list                   # list matching tests without running them
diaspar test -j 8                     # parallelism; all cores by default
diaspar test --seed 1234              # reproduce a run's randomness (D005); without it, each run gets a fresh seed, printed with a rerun command
diaspar test --shuffle                # run tests in a random order taken from the seed; the order is fixed otherwise
diaspar test --no-capture             # show the tests' output
diaspar test --durations 10           # the ten slowest tests
diaspar test --debug                  # the debugger at the first failure (D017)
diaspar test --json                   # machine-readable results, for agents and editors (D014)
diaspar test -p server                # one package of a workspace
  • A filter without * matches any test whose path contains it, as cargo test does; with * it is a glob; --exact matches one path.
  • Output is one line per failing test plus a progress counter, then the failures with the diagnostic format and a diff of expected and actual values (D014), then a summary line. The exit status is non-zero when a test fails.
  • The same filters work in the REPL: (run-tests) for the current module, (run-tests "parse") with a filter.

Tests take tags and cases as options (sketch):

(deftest parses-big-files :tags #[:slow]
  …)

(deftest parses-ints
  :cases #[#["1" 1] #["-7" -7] #["x" nil]]
  [input expected]
  (is (= expected (parse-int-or-nil input))))   ; each case is reported separately

Benchmarks use the same filters, plus baselines:

diaspar bench                         # all benchmarks
diaspar bench sort                    # filtered
diaspar bench --save-baseline before
diaspar bench --baseline before       # compare with a saved run: faster, slower, or the same within noise

The test report (accepted, 2026-11-16)

The report is part of the language's face (D019): structured, colored where the stream allows, and useful at a glance, as cargo-nextest, pytest, vitest and bun test are.

  • A diff of expected and actual values, not two whole values to compare by eye: for records and maps only the fields that differ, with the expected value in red and the actual in green (-/+ when there is no color); for vectors the differing positions; for long strings the differing lines, as pytest and Rust's pretty_assertions show.

    ✗ demo.user-defaults
    error: assertion failed: (= expected u)
       ╭─▶ tests/users.dias:10:5
     10 │     (is (= expected u))
        = diff (User):
            :email  - "ann@example.org"
                    + "ann@example.com"
            :age    - 31
                    + 30
  • A rich summary: passed in green, failed in red, skipped (by tag or filter) in gray, the time; the seed is in the summary, not at the top; one rerun command at the end (diaspar test --failed) instead of one per failure, with the exact command available per failure through --verbose.

    ✓ geom (12)   ✓ parse (40)   ✗ users (3 of 5)
    FAILED  52 passed · 3 failed · 1 skipped (slow) · 56 tests in 0.42 s · seed 2162542074
    rerun the failures: diaspar test --failed
  • A summary by module, as vitest: a line per module with its count and status; on a terminal, a progress line updated in place while tests run; in a pipe or CI, plain stable lines.

  • Written in Diaspar, in std.test: collecting results, diffs, and the whole report. The bootstrap interpreter only finds and runs the tests and hands results to std.test, as its REPL uses std.repl, so the work carries over to the compiled toolchain.

Property tests and fuzzing (accepted, 2026-11-16)

Part of std.test, for every program, as Go's fuzzing is part of go test and Ferron's property tests were part of its std.test.

(deftest reverse-twice
  for-all: [xs (gen.vector (gen.int))]
  (is (= xs (reverse (reverse xs)))))

(deftest json-round-trip
  for-all: [c (gen.of Config)]                 ; a generator derived from the deftype
  (is (= c (json.decode Config (json.encode c)))))
  • deftest with the label for-all: is a property: the bindings draw values from generators. Plain diaspar test checks each property on 100 values and on every saved failure, deterministically from the run's seed; diaspar test --fuzz [NAME] --time 60s searches longer.
  • Generators in std.test.gen: integers, strings (graphemes, emoji, several scripts), vectors, maps, sets, one-of, such-that, composition; gen.of derives a generator from a deftype or defenum and its field metadata, as Rust's Arbitrary and Hypothesis's from_type.
  • Shrinking as Hypothesis does it: generators draw from a stream of random choices, and shrinking works on the stream, so every generator, however composed, shrinks without a shrinker of its own. A failure is reported shrunk, with the structured diff and a rerun command.
  • Failures are saved under tests/fuzz/NAME/, as Go saves under testdata/fuzz/, and become regression tests run every time.
  • Coverage-guided fuzzing comes when the compiler can insert coverage counters.
  • The compiler's differential fuzzer is a user of std.test.gen: a program generator is a composed generator, and failing programs shrink through the same stream.

Concurrent tests (proposed, 2026-11-17)

Tests that start tasks run under the sim scheduler of D035 (accepted there): every fiber on one thread, every decision taken from the test's seed, time virtual. The details are in design/concurrency.md §6.

(deftest money-is-never-lost schedules: 1000
  (let [a (atom.new 100)
        b (atom.new 0)]
    (scope
      (spawn (transfer! a b 30))
      (spawn (is (= 100 (+ @a @b)))))))
  • One schedule per run by default, taken from the run's seed; schedules: N runs the test under N schedules, --schedules N overrides it for a run, and --fuzz searches longer, as for properties.
  • A failure is saved as a trace of the schedule under tests/fuzz/NAME/, next to the saved failures of properties, and is replayed first on every run; diaspar test --failed replays it. The trace records the code's hash, and a replay that no longer fits the code says so.
  • With for-all:, the schedule is part of the generated input and shrinks with the data.
  • scheduler: :prod runs a test on real threads (stress tests of the runtime, benchmarks of parallelism).
  • A deadlock is reported with what each fiber waits for and where.

Language test cases (accepted, 2026-11-16)

A test case of the language (tests/cases) is one file: the program and its expectation, as Diaspar data in an expect form that expands to nothing when the program runs. expect is defined in std.core, not std.test, so a case needs no use: loading std.test and its report would cost the compiler and the evaluator about 2 s per case (measured 2026-11-16), several minutes per column of the harness. The format is described in compiler/src/diaspar/tools/expect.dias.

(defn main [args]
  (print (/ 10 0)))

(expect
  exit: 1
  stdout: ""
  error: #{:message "division by zero" :at #[2 10]})
  • Diagnostics are compared as data, not as rendered text: both tools report what they print as data (the machine-readable form of D014) — severity, message, position, notes, help — and the harness compares those fields. With DIASPAR_DIAGNOSTICS_DATA=FILE, the interpreter, compiled programs, the compiler and the evaluator write every diagnostic to FILE as a line of data instead of drawing it on standard error, so the program's own standard error stays comparable. How diagnostics are drawn is covered by a small set of the renderer's own golden tests, so a change of layout does not rewrite every case: tests/render/, cases whose diagnostics stay drawn, so their stderr: pins the drawing (framed spans with labels, a warning, the unhandled report, a span over several lines, wide characters, a usage error), in every tool.
  • Expectations are recorded in place, as inline snapshots (Jest, insta): diaspar diff --record rewrites the expect form in the file, formatted, and the change shows in the diff next to the program.
  • Known differences live in the same form: known: #{:differs :interpreter :why "…"} (or :compiled, :evaluator), strict: a known difference that stops differing fails.
  • Name resolutions (the expander checked against the interpreter) are machine snapshots, not expectations of the program; they live apart, under tests/snapshots/, and go away with the interpreter.

Benchmarks (accepted, 2026-11-16)

Written in Diaspar, in std.bench, run by diaspar bench; the best of Rust's criterion and divan and Go's testing.B and benchstat.

(defbench sort-10k
  (sort (random-vector 10000)))

(defbench parse-config                          ; setup is not measured, as criterion's iter_batched
  setup: (fs.read-text "big.json")
  [text]
  (json.parse text))

(defbench sort-n cases: #[100 1000 10000]      ; one benchmark over several sizes, as divan's args
  [n]
  (sort (random-vector n)))
  • The number of iterations is chosen automatically, as Go's b.N: a warm-up, then samples until the estimate is stable; no hand-written loop.

  • Statistics: median, spread as the median absolute deviation, outliers; throughput when given (throughput: (bytes n)).

  • The measured result is kept alive: the runner consumes the body's value, and bench.keep keeps intermediate values, as Rust's black_box.

  • Allocations per iteration, as divan counts them: the runtime knows every allocation, since it counts references (D004).

  • Baselines: each run is compared with the previous one by default; --save-baseline NAME and --baseline NAME name them. A change is reported as faster or slower only when it is significant by a Mann-Whitney U test, as benchstat does, and its size is at least the noise threshold, 2% by default (--noise), as criterion's; otherwise it is the same within noise.

  • For CI: --fail-on-regression 5% and --json.

  • The report is a table:

    bench            median     ± MAD    allocs   vs before
    sort-10k         1.82 ms    ±1.1%    3        ▼ 12.4%  faster
    parse-config     4.10 ms    ±0.8%    1.2k     ~ 0.3%   same
    sort-n[10000]    1.79 ms    ±1.4%    3        ▲ 8.1%   slower

    faster in green, slower in red, the same in gray.

  • On the interpreter, numbers are only comparable with each other; real measurements are of compiled code, so diaspar bench is also the compiler's command.

Examples in documentation are tests (accepted, 2026-11-18)

Amended by D041: the example is a string after the name without string.dedent, runs in user with the module used, in a temporary directory, with … as a wildcard and --record filling in missing output. The sketch below is the original.

An example in a doc: string is written as a REPL session, and diaspar test checks it by default, as Elixir's doctests and Rust's documentation tests do:

(defn slice
  doc: (string.dedent "
    Part of a vector, list or string.

    user=> (slice #[1 2 3 4] 1 3)
    #[2 3]
    user=> (slice \"hello\" 1)
    \"ello\"
    ")
  …)
  • Each user=> line (and its ..> continuation lines) is evaluated in order, in a fresh session of the module's namespace, and what the REPL would print must equal the lines that follow, up to the next prompt or the end of the example. An expected error is written as the REPL prints it (error: …, its first line).
  • They run on the evaluator, not through cc, so they are fast; they are part of the module's tests, with the module's name and the line of the example, and diaspar test --doc-only or --no-doc selects them.
  • Examples are optional; the public API of std has them. An example that should be shown but not run is marked user=> ;; not run on its first line.
  • The same examples are what diaspar doc, the editor's hover (LSP) and the MCP adapter show (D037, D020), so documentation cannot drift from the code.

Why

  • Tests written in the language from the start keep the language and its standard library honest, and coding agents work best with a fast test loop.
  • Inline unit tests can test private code without exporting it; separate integration tests check what users of a module actually see.