Package design, the three-lane evidence model, and the retained-evidence and signing contracts of a project-aware testing harness.
Calamum Test is a standalone, project-aware testing harness that consolidates pytest, sandbox_test, and empirical_test lanes behind one retained-evidence CLI and Python facade. Its job is to keep test definitions, project context, and evidence packets aligned so that a run can be reviewed without relying on terminal history or hidden local state.
The published package is calamum-test version 0.3.1, with Python >=3.8, a single runtime dependency on cryptography>=42.0.0, an MIT license, and a Beta classifier. The catalog is tracked in catalog/test_definitions.json, and the retained outputs include JSON, Markdown, manifests, checksums, and optional signatures.
The command families are calamum test, calamum project, and calamum monitor; the compatibility alias calamum test project ... remains available; and only presentation refinements remain pending on a small set of human-facing branches, with the --json path clean.
Tooling disclosure. This report and project were developed using a repository governance harness and AI-assisted coding/curation workflow engineered by Joe Waller and enabled through OpenAI and Anthropic tooling. These tools support development and documentation, including summarization and report drafting, and do not replace the author's judgment. Calamum appears here as the project-aware retained-evidence harness and operator-facing validation layer.
Calamum Test packages a reusable testing substrate for projects that need one consistent place to define tests, execute them across multiple validation lanes, and retain evidence worth reviewing later. The package exists as calamum-test on PyPI, is imported and run as calamum, and exposes the command families calamum test, calamum project, and calamum monitor.
The published version is 0.3.1. The project metadata fixes the runtime floor at Python >=3.8, keeps the runtime dependency set to a single package, cryptography>=42.0.0, and classifies the distribution as Beta under an MIT license. The report documents the current package and its retained-evidence surfaces as they stand now.
The system is designed around project context. Definitions, runs, and reports resolve against a project root and a local descriptor, and the compatibility alias calamum test project ... remains available for existing operators. That makes the package useful both as a direct CLI and as a project-local automation layer.
The package metadata is explicit: name = "calamum-test", author ORACL-Prime, repository https://github.com/joediggidyyy/calamum, and entry point calamum = calamum.cli:main. The README makes the naming split equally explicit: install the distribution as calamum-test, run the CLI as calamum, and import the library through the calamum package.
Scope is deliberately narrow. Calamum consolidates the three canonical lane classes (pytest, sandbox_test, and empirical_test) behind one definition model and one evidence contract. It also exposes project registration, current-project readback, and retained-run / aggregate-report inspection. These surfaces constitute a packaging and governance layer rather than a general workflow-automation engine.
The public test library is the tracked catalog at catalog/test_definitions.json. One definition is one named test, and a definition may use one, two, or all three canonical lane classes. In contract terms, a definition carries identity, classification, tags, policy rules, evidence requirements, and lane arrays; the lanes are execution media for the same test, not separate catalog entries.
| Definition id | Status | Category |
|---|---|---|
seed-cli-smoke | active | bootstrap |
seed-adversarial-smoke | active | adversarial |
monitor-install-handoff-sandbox | active | regression |
pcap-vendor-readiness-focused | active | regression |
pcap-vendor-readiness-adjacent | active | regression |
gifmoji-phase0-architecture-proof | active | regression |
gifmoji-phase0-architecture-adversarial | active | adversarial |
gifmoji-phase1-payload-proof | active | regression |
gifmoji-phase1-desktop-shell-proof | active | regression |
gifmoji-phase1-compatibility-selector-proof | active | regression |
gifmoji-phase1-adversarial-drift | active | adversarial |
The controlled v1 vocabulary is small on purpose. It keeps the catalog searchable without letting the definition model drift into ad hoc labels or hidden meaning. Status values describe lifecycle; category values describe the definition's function; profiles identify when or why a definition is run; policy flags encode governance or execution constraints; and evidence requirements identify the retained outputs that must exist after a run.
| Surface | Values | Count |
|---|---|---|
| Status | seed, active, experimental, deprecated, disabled | 5 |
| Category | adversarial, general, bootstrap, regression, security, performance, integration, compliance | 8 |
| Profile | default, smoke, fast, release, nightly | 5 |
| Policy flags | json-first, project-aware, containment, local-only, signed-output, privileged-operation, release-gate, deterministic-output | 8 |
| Evidence requirements | report_json, report_md, manifest_json, checksums_json, stdout_capture, stderr_capture, receipt_json, report_signature, manifest_signature | 9 |
The package contract can be stated in four simple checks — formal restatements of the documented behavior already present in the package sources and README.
The first equation captures the definition model: a definition has identity, classification, tags, policy rules, evidence requirements, and one or more lane classes drawn from the canonical trio. The second says a run only counts when the retained artifact set contains the required evidence. The third restates the content-addressing rule. The fourth captures the detached-signature posture for publishable or privileged flows under Ed25519-backed signing.
The filesystem split is intentionally small. Tracked inputs are the shared project descriptor at .calamum/project.json and the catalog at catalog/test_definitions.json; retained outputs live under .calamum/generated/runs/ and .calamum/generated/reports/generated/; and the local-only guard file keeps generated output out of version control by default.
The code makes that split explicit. projects.py resolves the project root, descriptor, overlay, state, catalog, runs, reports, and working directory into a single resolved project object. evidence.py persists run evidence, session evidence, and index rows. reports.py loads retained runs, materializes aggregate report outputs, and copies signatures and checksum sidecars when signing is requested.
| Artifact | Scope | Role |
|---|---|---|
report.json | Per run | Canonical machine-readable run packet. |
report.md | Per run | Human-readable retained summary. |
manifest.json | Per run | Artifact inventory for the run packet. |
checksums.json | Per run | Content-addressing ledger for retained files. |
stdout_capture | Per step | Captured standard output for step-level replay. |
stderr_capture | Per step | Captured standard error and heartbeat lines. |
run_index.jsonl | Per run | Append-only run index for discovery and review. |
receipt.json | Aggregate | Receipt for generated aggregate-report surfaces. |
report_signature | Aggregate | Optional detached signature over aggregate JSON. |
manifest_signature | Aggregate | Optional detached signature over aggregate manifest. |
report_index.jsonl | Aggregate | Index of generated aggregate report records. |
The README's default tree makes the same point from the operator's perspective: tracked inputs stay small, generated outputs stay local, and the aggregate directory keeps report state together under a predictable root. The signing section adds the relevant environment variables by name: CALAMUM_ED25519_PRIVATE_KEY, CALAMUM_ED25519_PUBLIC_KEY, CALAMUM_POLICY_SIGNING_KEY, and CALAMUM_CONFIG_ROOT. The repository also keeps placeholder-only local setup guidance for signing and configuration.
Across the CLI surfaces, content and structure are correct and the --json path is deterministic; the only outstanding work is presentation refinement in a handful of still-in-development human-facing emit branches. The pre-publish audit dated 2026‑05‑03 records this assessment.
| Surface group | Status | Audit note |
|---|---|---|
| Renderer-backed human surfaces | conformant | Route through named helpers; title, decision, section, and KV formatting are on-spec. |
| Live operational no-go surfaces | in development | monitor pnp snapshot, monitor pnp timeline, monitor pcap capture, monitor pcap interfaces, monitor session run, and project clear emit their human-facing output directly rather than through the shared render helpers; the content and structure conform to the contract. |
| Scaffold-only project evidence routes | in development | project evidence * routes carry the same presentation gap and are lower priority because they are not yet operationally central. |
--json path | clean | to_json_text() is deterministic with indent=2 and sort_keys=True. |
The package contract reads most clearly as a flow rather than a list of file names. The diagram below traces the path from a catalog definition to a run, then to retained evidence, then to aggregate report surfaces and public publication.
The diagram is intentionally compact. It shows the catalog-driven run path, the three canonical lane classes, the evidence pack, and the publication surface, scoped to what the source set supports. Optional signing sits on the aggregate side of the contract.
The public artifact lane is simple to inspect because the repository keeps the same surfaces that the report cites: the README, the changelog, the package metadata, the catalog, the validation audit, and the retained run / report indexes. That makes the report traceable back to the source tree without requiring any hidden workspace assumptions.
The current test inventory is also visible from the repository tree. The tests/ directory contains conftest.py plus fifteen test_*.py modules spanning API, CLI, catalog, project, report, signing, and monitor coverage. The inventory documents the package's testing posture; it is not part of the published contract.
The public-facing integration points are the PyPI page, the GitHub repository, the site-hosted report page, and the generated run / report indexes stored under the Calamum project root. These are the surfaces relevant to reproduction and review.
The report's scope is bounded to the documented source set. It presents no performance charts, benchmark claims, or release statistics absent from that set, and it ties the signing description to the documented environment variables and the package's actual fallback and Ed25519-backed paths.
The practical next steps are straightforward: preserve the retained-evidence contract as the package grows, and continue reducing presentation drift in any human-facing emit branch that still bypasses the shared render helpers. The current source tree is already rich enough to support that work without changing the package's core identity.
The appendix lists the exact files used as authority for this report. The paths are kept explicit so the page can be audited against the repository tree without guessing which file supplied which claim.
projects/calamum/README.md — package scope, command surface, retained evidence contract, project resolution order, and signing posture.projects/calamum/pyproject.toml — package identity, version plumbing, runtime dependency, license, and entry point.projects/calamum/src/calamum/__init__.py — version authority for 0.3.1.projects/calamum/CHANGELOG.md — release history and the 0.3.1 publication boundary.projects/calamum/catalog/test_definitions.json — the tracked public catalog used for Table 1.projects/calamum/docs/PRE_PUBLISH_VALIDATION_AUDIT.md — the pre-publish audit findings summarized in Table 4.projects/calamum/src/calamum/projects.py — project resolution, descriptor lookup, and resolved-project layout.projects/calamum/src/calamum/evidence.py — session evidence persistence and index behavior.projects/calamum/src/calamum/reports.py — aggregate report loading, generation, and signing paths.projects/calamum/src/calamum/signing.py — Ed25519 and fallback signing / verification environment names and artifact sidecars.