Diagnostics and testing

tulam diagnostics identify a stable code, compiler stage, severity, source provenance, and structured payload. Tests should exercise source programs through the same strict path users depend on.

Diagnostic controls

tulam check --diagnostic-detail detailed
tulam build --verbosity trace
tulam check --message-format jsonl
tulam check --message-format sarif
tulam explain TLC-PROJECT-0018
tulam stages

Human, short, JSON Lines, and SARIF output share the same registered diagnostic model. Trace mode exposes bounded pass events and timings without dumping whole IR bodies.

Positive and negative programs

A positive test must pass parsing, checking, elaboration, lowering, and LLVM native execution when it claims runtime support. A negative test should fail at the intended stage for the intended reason—not simply “fail somehow.”

Known implementation gaps are XFAIL, with an owning phase and pinned observed failure. An unexpected pass requires promoting the expectation rather than being ignored.

Main suites

stack test
./tests/run_conformance.sh
./tests/run_diagnostics.sh
./tests/run_projects.sh
./tests/run_native.sh
./tests/run_classes.sh

The conformance catalog inventories every Language Reference feature with independent Tulam programs. Haskell unit tests complement but do not replace source-language acceptance.

Program assertions

Native source suites commonly print explicit pass/fail assertions whose runners validate counts and output. The current native wrapper prints scalar entry results but does not forward an Int result as the process exit status, so a bare returned 1 is not by itself an assertion-sensitive process failure.

Testing a feature

Cover ordinary use, composition, edge cases, and dependent or higher-order use where relevant. Include rejection cases for invalid programs and execute positive programs on the reference backend.

Common mistakes

Do not accept “some error occurred” as a negative test, replace source-language coverage with Haskell-only tests, or mark a normatively valid implementation gap as an invalid program. Pin the stage and stable diagnostic you intend.

Recap

Diagnostics are structured compiler output, and conformance programs are the executable inventory of the surface contract. Native execution completes the positive acceptance path.

Authoritative details: Diagnostic Design and the testing policy in Implementation Plan.