Conformance and Differential Testing

The suite is the language

Osmol's truth lives in three places, in constitutional priority order: the conformance suite, the specification, and only then any implementation, including the reference one. GOVERNANCE.md states the order plainly, and the engineering dissertation calls the first clause what it is: constitutional, amendable by nothing. The suite, not the reference implementation, is the arbiter.

The suite itself is a simple thing: a repository of golden .osmol programs, each paired with a required outcome. Two kinds of pair exist:

  • Must-fail programs paired with expected diagnostics: code, span, and message class. spam.osmol must die with O-000 at line 2, in every implementation, forever.
  • Must-run programs paired with expected deterministic settlement traces. Same declarations, same injected human judgments, same trace, byte for byte, on every platform.

Any implementation claiming the name Osmol must pass it. This is how the language stays bigger than its first codebase, and how Law VII (federated, or failed) grows teeth: a protocol whose behavior is defined by whichever binary you happen to run is not a protocol, it is a product with documentation.

The seeds

Honesty first: the conformance repository does not exist yet. What exists is its first two cases, which are already canon because they ship with the reference interpreter.

Seed 1, the dinner mesh (must-run). Input: dinner.osmol plus the canonical resolve command. Required output: this exact trace.

$ python3 -m osmol dinner.osmol --resolve "raj:venue(dinner)=verde"
OSMOL v0.1 - reference interpreter
parsed dinner.osmol: 2 twins, 0 sends (no such construct exists)
initial gapcount: 3   (Theorem 1 bound: equilibrium in <= 3 flows)
  [human judgment] raj resolves venue(dinner) = verde
  human-lane  maya -> raj  (verbatim, provenance=human): "Saw a fox on the way home and thought of you."
t1  raj.venue(dinner) --exact--> maya   P=1.30 (R1.0*U1.3*T1.0*M1.0) > theta[family]=0.3   [ledger]
t2  maya.eta(dinner) --exact--> raj   P=1.30 (R1.0*U1.3*T1.0*M1.0) > theta[family]=0.3   [silent]
t3  maya.status(atlas.review) --category--> raj   P=0.65 (R1.0*U1.3*T1.0*M0.5) > theta[family]=0.3   [silent]

EQUILIBRIUM after 3 flows (Theorem 1 bound 3: respected)
messages composed by humans: 0 (the grammar has no verb for it)
interruptions: 0
ledger[maya]: venue(dinner) = verde
open gaps: 0 — silence is a fixpoint

Seed 2, the spammer (must-fail). Input: spam.osmol. Required outcome: O-000 at line 2.

$ python3 -m osmol spam.osmol
OSMOL v0.1 - reference interpreter
osmol: line 2: error[O-000]: 'send' — there is no send. Declare what you hold; declare what you seek; the mesh does the rest.

Two cases sounds thin. But look at what they already pin down between them: the tombstone lexer, one full parse of every major construct, the resolve pathway, the human lane's verbatim-first delivery, membrane casts at two granularities, the pressure arithmetic and threshold classes, placement (ledger vs silent), and the Theorem 1 bound. The dissertation's build order makes every subsequent step's definition of done a conformance addition, so the suite grows as the ladder climbs.

Differential testing

The plan for the Rust engine (see One Engine, Four Habitats) does not retire the Python interpreter, it re-employs it. During the transition both engines run identical corpora and their traces are diffed byte-for-byte. The Python interpreter's retirement gift is becoming the oracle that certifies its successor: Step 1 of the build order is done when osmol check matches Python's judgments on the corpus, and Step 2 is done when the settlement traces are byte-identical.

The precondition for any of this is determinism, which is why the tie-break rule is spec-level: among fireable flows of equal pressure, order lexicographically by (sender name, receiver name, fact). Without a fixed tie-break, two correct engines could settle the same mesh along different orders and both be "right", and then golden-file testing, membrane audits, and cross-implementation trust would all dissolve. Determinism is what makes a trace a fact about a program instead of an anecdote about a run.

One honest wrinkle, already documented in the pressure chapter: Seed 1's golden trace comes from the Python engine, whose tie-break is scan order, and on the dinner mesh that order differs from the mandated lexicographic rule (t1 and t2 tie at 1.30 and fire in the "wrong" order). Before Step 2's byte-identical goal can be met, the project must either re-derive the golden trace under the lexicographic rule or supersede the rule by RFC. The suite cannot canonize both.

The adversarial layer

Beneath the golden files sits the layer that goes looking for trouble:

  • Property tests (proptest) generate random meshes and assert the machine-checked invariants dynamically: flows never exceed the initial gapcount (Theorem 1's runtime shadow), denied membranes leak nothing, and granularity never refines along any path. The theorems say these cannot fail; the property tests are how an implementation proves it read the theorems.
  • Fuzzing (cargo-fuzz) hammers the lexer and parser, because parsers are where languages get owned.
  • The CI matrix builds what the doctrine demands: native Linux, macOS, and Windows; wasm32-unknown-unknown; and the Android FFI target, all against the same suite. Divergence between habitats is, by definition, a bug in whichever habitat disagrees with the suite.

Contributing a case

A conformance case should eventually be a small, self-describing bundle. One suggested shape:

cases/
  o-000-spam/
    input.osmol          -- the program
    cmd.txt              -- exact invocation, including any --resolve flags
    expected.txt         -- golden trace, or expected diagnostic (code, span, class)

New cases that pin down existing spec behavior are ordinary pull requests. Cases that would change what the language means are language changes, and those go through the RFC process. Remember the authority order: a merged suite case outranks everything, so merging one is the most consequential act in the project.

One last honesty clause, from the dissertation's risk chapter: the gravest failure mode for a young language is the one-implementation trap, where the reference quietly becomes the spec and every accident of its code hardens into law. The suite's constitutional supremacy is half of the counter; the other half is keeping the Python engine alive as a second implementation long after it stops being the fast one. Two engines, one suite, byte-identical traces: two vessels, one level.