Design, construction mathematics, and operational utilization of the semantic memory layer of the ORACL system, as codified in the CodeSentinel repository.
Tooling disclosure. This report and project were developed using a repository governance harness and AI-assisted coding/curation workflow engineered by Joe Waller and enabled through OpenAI and Anthropic tooling. These tools support development and documentation, including summarization and report drafting, and do not replace the author's judgment. Names such as CodeSentinel, ORACL, Calamum, ORACode, and ORACall appear as internal identifiers for governance, agent assistance, validation, semantic indexing, and retrieval tooling.
Large software workspaces defeat naive retrieval. A repository that mixes production packages, operational runbooks, planning artifacts, deployment scripts, and archival material presents an agent — human or LLM — with a discovery problem that filesystem search does not solve: relevance in such a corpus is a function of semantics, governance state, and connectivity, not merely lexical match. The ORACL system addresses this with a dedicated semantic memory layer, described internally by the hippocampus metaphor: ORACode is the consolidation structure that encodes the workspace into durable, indexed memory, and ORACall is the recall interface that retrieves from it without rescanning the filesystem ("blind retrieval").
This report is a technical account of that layer as it exists in the CodeSentinel‑1 codebase. It has three goals: (i) to specify the construction pipeline and its mathematics precisely enough for independent reimplementation; (ii) to document the operational envelope — integrity guarantees, complexity bounds, telemetry — that make the system usable under the repository's SEAM governance regime (Security, Efficiency, Awareness, Minimalism); and (iii) to make the data-science content of the design explicit (Appendix A), since the system is, in substance, an applied exercise in feature engineering, graph analytics, and information retrieval.
Every mechanism described in Sections 3–7 is grounded in code shipped in the repository; parameters and population statistics are read from the live production artifacts. Where the report discusses capabilities that exist only in planning documents (the MCP tool surface, sensor-domain ingestion, Brain ORACL constraint seeding), the text is marked planned. Implemented components are marked implemented at first mention where the distinction matters. Internal source references are collected in the References section.
The system separates a build plane (ORACode: offline graph construction, run via codesentinel oracode graph rebuild) from a query plane (ORACall: online retrieval, run via codesentinel oracall search|trace|stats and, programmatically, through SessionMemory.query_knowledge_graph()). Both planes converge on a single artifact store, the semantics vault (semantics_vault/oracl_index/), which holds the append-only JSONL index, edge partitions, the weighted graph, cryptographic sidecars, signed snapshots, and a SQLite mirror bundle. The graph is the shared substrate: the CIDS sensor domain's local intelligence (SentryCortex) reads the same weighted_graph_index.json that ORACall serves, confirming the design intent of one unified semantic layer spanning code intelligence and system state.
Table 1 reports the production population at the most recent weighted build, read directly from the artifact's schema header, alongside the current (grown-since) Phase 1 index.
| Quantity | Value | Source |
|---|---|---|
| Phase 1 index records (current) | 47,373 | index.jsonl (32 MB) |
| Weighted-graph base nodes (artifacts) | 42,359 | wtls-1.0 header |
| Virtual domain hub nodes | 38 | wtls-1.0 header |
| Total nodes | 42,397 | wtls-1.0 header |
| Extracted edge rows (merged, deduplicated) | 46,658 | all_edges.jsonl |
| · of which Phase 2 repository edges | 4,516 | graph_edges.jsonl |
| Hub membership edges (one per base node) | 42,359 | derived, §3.5 |
| Total weighted edges | 89,017 | wtls-1.0 header |
| Association mean μ / std. σ | 0.7724 / 0.0759 | wtls-1.0 header |
| Learned (feedback) edges | 0 | learned_edges.json (§4.5) |
| Weighted artifact size | 65.8 MiB | weighted_graph_index.json |
Construction is orchestrated by tools/oracode/rebuild_graph.py as a phased pipeline with per-phase skip flags and tunable weighting parameters (--alpha, --beta). Phase naming is historical: Phase 2b (weighted assembly) executes after Phase 3, as the terminal stage.
Discovery enumerates the workspace under a twelve-suffix inclusion set (.py .md .txt .json .jsonl .yaml .yml .ini .cfg .toml .ps1 .sh) with a governance-driven exclusion list: VCS internals, virtual environments (matched by .venv* prefix), caches, build outputs, the vault itself (preventing index recursion), embedded external repositories, and derived-output containment paths (report_tmp/, deployment/staging/, docs/reports/_tmp/, client-data payloads) — scratch outputs are never authoritative graph inputs. Each surviving file yields an ArtifactRecord with two orthogonal identifiers:
The artifact ID ida keys the node and changes when the file moves or is rewritten (a versioning identity); the checksum ca is pure content addressing, enabling drift detection independent of location. This dual-identity scheme is what lets downstream integrity gates distinguish "file changed" from "file relocated."
Baseline tags are pure functions of the artifact path — the same input always yields the same features, a property the two-layer merge (§3.4) depends on. Seven fields are inferred: domain (first path segment through a curated map, e.g. codesentinel/ → core), component (second segment), lifecycle (active unless an ancestor segment matches an archival hint — archive, legacy, quarantine → archived — or the staging hint drafts → staging), classification (public only under docs/public|external, else internal), content_type (report/issue/instruction heuristics, else generic), placeholder (boolean), and format. Python artifacts additionally pass through a deep-AST analyzer contributing structural tags, and a learned-tag overlay (§4.5) merges feedback-derived tags last, with metadata keys (confidence, source) stripped so learning cannot corrupt the schema.
Each node receives a scalar priority expressing its retrieval and governance importance. The model is a twelve-bucket weighted composite — an expert-weighted linear index in data-science terms — modulated by two interaction multipliers:
where each contribution cb(a) ∈ [0, 1] is a normalized bucket signal (risk score scaled by 1/100, usage volume by 1/100, cost savings by 1/1000, link density clamped to 1, lifecycle = 1 for active artifacts and 0.5 otherwise). The bucket weights encode the SEAM ordering — security dominates at 0.30, triple the next tier:
compute_priority_score(). Weights sum to 1.00; the security bucket alone carries 30% of the composite. Hover any bar for the exact contribution definition.The two multipliers are interaction terms: classification mcls ∈ {1.0, 1.15, 1.35, 1.65} (public → sensitive) and sensitivity msens ∈ {1.0, 1.4, 1.8} (low → high). Their product spans [1.0, 2.97], but Eq. 2's hard cap at 2.0 truncates the extreme cell — a deliberate ceiling that prevents any single artifact from dominating ranked retrieval regardless of governance state:
Edges arrive from two independent producers. Phase 2 (graph_builder.py) extracts cross-references over the Phase 1 index — imports, markdown-link references, and discounted references_placeholder edges. Phase 3 (static_edge_generator.py) runs six pluggable edge providers — Python imports via an AST ImportVisitor, Markdown links, config references, test targets, path mentions, and comment references — emitting imports, references, configures, tests, and mentions edges, each with an extractor confidence κ. Every edge carries a relation label against which a prior weight ρ(r) encodes the strength of the coupling it evidences:
DEFAULT_RELATION_WEIGHTS). Compile-time coupling (imports) anchors the scale at 1.0; placeholder references are deliberately discounted to 0.4. Two taxonomy entries (depends_on, provides) are reserved — no current extractor emits them — while relations outside the table (tests, mentions) take the 0.5 fallback.Producer outputs are merged into all_edges.jsonl by canonical-key deduplication, Phase 2 edges taking precedence over Phase 3 on collision. An opt-in two-layer mode (--two-layer, or CODESENTINEL_ORACODE_TWO_LAYER=1) interposes an event-sourced stage: producers append to evidence partitions — producer-scoped JSONL streams (phase2_repo_edges, phase3_static_edges, overlay) — from which a committed view is materialized deterministically (the contract in two_layer_graph.py) and copied out as the merged artifact. In both modes, merge identity is the canonical edge key
key(e) = ( source_id, source_fragment, target_id, target_fragment, relation )with fragments (line-range locators such as L10-20) included so that two references anchored at different sites in the same file do not collapse into one edge. The two-layer mode is event-sourcing applied to graph construction — evidence is append-only and attributable per producer, the committed view is a reproducible function of it, and any producer can be re-run or audited in isolation — at the cost of an extra materialization pass, which is why it ships flag-gated rather than default.
The weighted builder joins the index and committed edges into the final graph. Each edge's effective weight combines its relation prior with the extractor's confidence:
Virtual hubs. For every distinct domain tag, a virtual hub node hub:domain:<d> is injected and every member artifact linked to it by a belongs_to edge (ρ = 0.8). In production this adds 38 hubs and 42,359 membership edges — the difference between the 46,658 extracted rows and the 89,017 total edges in Table 1. Hubs give the tag taxonomy a graph address: "the security domain" becomes a first-class node whose neighborhood is the domain listing, retrievable through the same top-k machinery as any file (§4.4).
Association statistics. Each node's local connectivity quality is the mean weight of its incident edges (both directions), with a 250-neighbor cap retaining the heaviest edges so that mega-hubs cannot stall the build:
Population statistics over all nodes (production: μ = 0.7724, σ = 0.0759) standardize this into a z-score, so a node is rewarded not for its absolute connectivity but for how its connection quality deviates from the corpus norm:
The α parameter balances governance-driven priority (Eq. 2) against emergent graph structure; at the production σ of 0.076, a node one standard deviation above the mean gains 0.35 priority — material, but never enough to outrank the security multipliers. Nodes also record a normalized degree centrality, link_density(v) = deg(v)/(n−1), which feeds back into Eq. 2's connectivity bucket.
Node priority orders artifacts; retrieval traversal needs an ordering over edges. The DuoRanker assigns every edge an association rank from two evidence sources. The direct term scales the edge weight by tag-profile similarity — the Jaccard coefficient over the two endpoints' tag-value sets — so that structurally linked artifacts that also share semantic context rank above incidental couplings:
The γ floor guarantees that a structural edge retains rank even with zero tag overlap. The inferred term adds triadic evidence: every length-2 directed path u→x→v that parallels the edge contributes the product of its constituent weights, attenuated by decay λ — a truncated Katz-style index [2, 13] restricted to distance 2 and computed only where a direct edge already exists (confirmation, not link prediction):
The vault is treated as evidence, so every terminal artifact carries a verifiable provenance chain. The weighted index is canonicalized (sorted keys, compact separators) before digesting, making the hash independent of serialization order; the digest is written as a .sha256 sidecar; and an Ed25519 signature (key via ORACALL_ED25519_PRIVATE_KEY) or, when unavailable, an HMAC fallback is written as an algorithm-prefixed .sig sidecar. Timestamped signed snapshots of the index rotate under snapshots/; a dedicated health checker (graph_health_check.py) appends population and health metrics to graph_health_history.jsonl — the longitudinal trail consumed by the stats audit; and the schema header embedded in the artifact itself (version, generation time, source artifacts, μ/σ, α/β) makes each build self-describing and reproducible.
The SQLite mirror is the query plane's deterministic materialized view: node and edge tables built from the weighted artifact, bundled with a manifest and addressed through a LATEST.json pointer. It exists so that exact-name lookups (§4.2) can be served without loading the 65.8 MiB graph into memory, and so that external consumers get a stable, verifiable query surface with standard tooling.
ORACall is the recall path: it accepts free text from an operator or agent, compiles it into a structured graph query, executes against an in-memory RetrievalIndex (or the SQLite mirror), and logs every interaction for the feedback loop. Figure 8 traces the full path of a query.
The SearchQueryBuilder compiles text to structure in six deterministic passes (Fig. 8). Two design choices merit emphasis. First, negation is first-class: both operator syntax (-archive) and natural phrasing ("but not archives") produce exclusion terms that survive to a path-level post-filter, applied even on the mirror fast path. Second, tokens are assigned to three weight classes — entity terms (ω = 5.0), type terms matching the tag-alias table (ω = 2.0), and stop words (ω = 0.1, retained rather than dropped so they can still tip ties) — a coarse, hand-set analogue of inverse-document-frequency term weighting [3, 4]. When type extraction consumes every token ("recent reports" → empty free text + filters), the query degrades gracefully to a filtered wildcard.
Queries matching file:, filename:, path:, or prefix: directives bypass the graph entirely: the mirror's LATEST.json pointer resolves the current SQLite bundle, and an indexed lookup returns results at score 1.0 with provenance tagged source: mirror. Any failure — missing bundle, invalid pointer, import error — falls open to the graph path: the mirror is an optimization, never a dependency. This is the inverse of the security posture (which fails closed, §7); availability-critical read paths degrade, integrity-critical write paths halt.
At session start the weighted artifact is parsed once into an in-memory RetrievalIndex: a node dictionary, an O(1) adjacency map (each edge registered on both endpoints), and a key-value edge store addressed by tag-derived bidirectional keys (a forward and reverse key per edge), which is what the "O(1) semantic retrieval" claim refers to — hash-path addressing for known entities. Free-text search over node metadata is a filtered linear scan scored per token per field:
where bf are the per-field boosts (Fig. 9b), m is the count of distinct query tokens matched, 𝟙[q ⊆ τa] is a unit bonus when the whole query appears as a substring of the artifact's tag string (applied additively, after amplification), and φext is the persona's multiplier for the artifact's file extension. The structure is BM25F-adjacent [4] — field-weighted lexical evidence with query-term weighting — minus corpus statistics, which is an acknowledged trade (§8).
Beyond text search, ORACall exposes the graph's associative structure directly. trace walks the adjacency map from a resolved node with direction and depth control (the CLI's dependency tracer). top_k_associations(v, k) returns v's neighborhood ordered by DuoRanker rank R̂, using heap selection — O(Nv log k) rather than a full sort — under a 5,000-neighbor latency guard. Because hubs are ordinary nodes, query_hub("security") is just top-k on hub:domain:security: domain browsing, ranked file listings, and dependency tracing are all the same primitive. Line-range fragments on edges resolve to source text via resolve_fragment (L{start}-{end}), which is how agent context assembly quotes evidence rather than whole files.
Every search appends a JSONL record — timestamp, raw query, parsed context, result count, latency, session ID — to logs/behavioral/search_history.jsonl, and every retrieval operation logs typed metrics (query class, latency, result count, an estimated compute cost in USD) through the agent-metrics channel. These are the observational inputs to the Phase 4 feedback loop: learned_edges.json (co-retrieval associations injected into the next weighted build as learned_association edges, confidence-weighted per Eq. 3) and learned_tags.json (tag corrections merged in Phase 1). The loop is fully plumbed end-to-end — builder merge points, schema, confidence handling — but both stores are empty in the current production vault: the system is instrumented for online learning while its behavior remains, today, fully deterministic. Telemetry failures never fail a search.
The graph's operational role is to be the memory the agent consults before it acts. Three consumption patterns are live today implemented:
codesentinel oracall search|trace|stats for interactive retrieval; codesentinel oracode graph rebuild (with oracall build as a deprecated alias, sunset 2026-Q3) and codesentinel oracode mirror query|verify for build-plane operations.SessionMemory.query_knowledge_graph() is the programmatic dispatch — concept queries, node queries (auto-selected when the query carries a : qualifier), and neighbor traversals — through which agent context assembly retrieves top-K relevant paths and fragments.The planned MCP surface planned promotes these same primitives to remote tools: oracl_agent_context (read-only graph query, no gate required) and oracl_agent_run, whose ten-step loop hydrates session memory, queries the graph for context, and emits mutations in the structured @@FILE/@@REPLACE/@@WITH/@@END contract under gate evaluation — with sensor-domain GraduatedEvent ingestion (QF-08) and a sixth pipeline phase (sensor_edge_ingestion) extending the substrate to system-state nodes. These are specified in the Surface Expansion Plan v1.2 and Full Synthesis documents and are not yet code.
The engine's cost model separates a paid-once load from cheap steady-state queries. Loading parses the 65.8 MiB weighted artifact into memory; the RetrievalAPI instruments this with before/after RSS deltas (psutil) so the graph's true memory footprint (graph_in_ram_mb) is a reported operational metric, not an estimate, and load latency is logged per artifact. Steady-state costs:
| Operation | Structure | Complexity | Bound |
|---|---|---|---|
| Node fetch by ID | hash map | O(1) | — |
| Keyed edge fetch (fwd/rev key) | hash map | O(1) | — |
| Neighborhood expansion | adjacency list | O(Nv) | — |
| Top-k associations | heap selection | O(Nv log k) | Nv ≤ 5,000 guard |
| Concept search (free text) | linear scan | O(V · |q|) | filters prune first |
| Prefix key search | key scan | O(E) | — |
| Mirror exact/prefix lookup | SQLite index | O(log n) | out-of-process |
| DuoRanker distance-2 (build) | hash join | O(E · c) | c = 250 cap |
Latency is measured, not assumed: the CLI reports per-query wall time, and the telemetry stream accumulates the latency distribution across sessions — the dataset from which regression in retrieval performance would be detected via the health-history trail.
| Principle | Mechanism in ORACode/ORACall |
|---|---|
| Security | Signing keys via environment only; Ed25519 with explicit-fallback HMAC; classification/sensitivity multipliers bias priority toward protected material; names-only outputs (paths and hashes, not contents) in evidence artifacts. |
| Efficiency | Selective hashing and mtime-seeded IDs; neighborhood caps; heap selection; mirror fast path for exact-name queries; lazy imports keeping CLI startup fast. |
| Awareness | Every build self-describes (schema header); every query is logged; health history is longitudinal; RSS and cost estimation make resource use observable. |
| Minimalism | Atomic temp-file-replace writes; append-only evidence partitions; snapshots instead of deletions; derived-output containment keeps scratch out of the graph. |
search_history.jsonl plus governance review of learned associations before they enter the signed graph.GraduatedEvent ingestion, Phase 6 sensor_edge_ingestion, and inter-node session transport (SessionMemory.load_remote over CaSTaP) remain design-stage, with schemas frozen in the plan documents. plannedORACode and ORACall demonstrate that a governed, reproducible semantic memory for a large working repository can be built from deterministic parts: pure-function feature extraction, an event-sourced edge store, a transparent two-term ranking model with six published parameters (α, β, γ, λ, and the cap pair), and a provenance chain strong enough to treat the graph itself as evidence. The design consistently prefers auditable simplicity over statistical power — every score in the system can be recomputed by hand from this report — while leaving explicit seams (learned edges, sensor ingestion, MCP tooling) through which adaptivity can be introduced without breaking the reproducibility contract. At production scale (4.2 × 10⁴ nodes, 8.9 × 10⁴ edges) the paid-once load / hash-path query model holds comfortably; the documented path forward is indexing, not architecture.
The system is an applied data-science artifact; this table names the method behind each mechanism.
| System element | Data-science method | Where |
|---|---|---|
| Deterministic tag inference from paths | Feature engineering (pure-function features) | §3.2 |
| SHA-256 dual identity (ID + checksum) | Content addressing; data versioning & drift detection | Eq. 1 |
| Twelve-bucket priority composite | Expert-weighted linear index model | Eq. 2, Fig. 3 |
| Classification × sensitivity multipliers | Interaction terms with response capping | Fig. 4 |
| Relation prior weights ρ(r) | Ordinal prior encoding over a closed taxonomy | Fig. 5 |
| Association delta δv | Z-score standardization against population statistics | Eq. 5 |
| link_density = deg/(n−1) | Normalized degree centrality [5] | §3.5 |
| Virtual domain hubs | Bipartite projection / category-node construction | §3.5 |
| Tag-set Jaccard in Rdir | Set-similarity measure [1] | Eq. 7 |
| Distance-2 decayed path evidence | Truncated Katz index; path-based link scoring [2, 13] | Eq. 9 |
| Max-normalization of edge ranks | Min-max feature scaling | Eq. 10 |
| Token weight classes (5.0 / 2.0 / 0.1) | Hand-set term-importance prior (IDF surrogate) [3] | §4.1 |
| Field boosts + multi-token amplification | Field-weighted lexical ranking (BM25F-adjacent) [4] | Eq. 11 |
| Synonym & temporal expansion, negation rules | Rule-based query understanding / query expansion | §4.1 |
| Persona file-type weights | Contextual re-ranking (personalization) | §4.3 |
| Top-k heap selection | Partial-order statistics, O(N log k) selection | §4.4 |
| Evidence partitions → committed view (opt-in) | Event sourcing; deterministic materialization | §3.4 |
| SQLite mirror bundle | Materialized view with integrity manifest | §3.7 |
| search_history.jsonl + agent metrics | Implicit-feedback logging; observational telemetry | §4.5 |
| learned_edges / learned_tags merge | Offline incremental (batch-online) learning loop | §4.5 |
| RSS-delta and latency instrumentation | Resource profiling; measurement-based cost models | §6 |
{ "artifact_id": SHA256(path|mtime_ns|size),
"path": "<repo-relative posix path>",
"checksum": SHA256(content),
"file_type": "py|md|json|...", "size_bytes": int,
"created_at": ISO8601Z, "modified_at": ISO8601Z,
"author": "system",
"tags": { "domain", "component", "lifecycle", "classification",
"content_type", "placeholder", "format", ...ast, ...learned },
"link_count": 0, "backrefs": 0, "usage": 0,
"estimated_savings": 0.0, "review_status": "draft" }{ "source": id, "target": id, "relation": r,
"metadata": { "confidence": κ, ... },
"relation_weight": ρ(r)·κ, // Eq. 3
"association_rank": R̂ ∈ [0,1] } // Eq. 10@dataclass(frozen=True)
class GraduatedEvent:
schema_version: str # "1.0"
timestamp_utc: str # ISO8601 Z
sensor_source: str # sentry | edge_graph | oracode | operator
event_type: str # heartbeat | drift | anomaly | operator
decision: str # go | no-go | pass | fail
summary: str # names-only
reason_codes: tuple[str,...]
motivator_hint: str | None # Threat | Awareness | Reward
cortex_routing_hint: str | None
source_id_hash: str | None # SHA256[:16] — never raw identifiers
artifacts: dict[str, str]| Symbol | Meaning | Value |
|---|---|---|
| α | Association-delta blend into node priority (Eq. 6) | 0.35 |
| β | Inferred distance-2 blend into edge rank (Eq. 10) | 0.5 |
| γ | Similarity floor in the direct term (Eq. 8) | 0.2 |
| λ | Distance-2 path decay (Eq. 9) | 0.6 |
| μ, σ | Population association mean / std. (Eq. 5) | 0.7724, 0.0759 |
| ρ(r) | Relation prior weight | Fig. 5 |
| κ | Extractor confidence per edge | (0, 1], default 1 |
| ωt | Query token-class weight | 5.0 / 2.0 / 0.1 |
| bf | Retrieval field boost | Fig. 9b |
| — | Build neighborhood cap / query neighbor guard | 250 / 5,000 |
| — | Priority cap / result limit (CLI · parser) | 2.0 / 10 · 20 |
The implementation references below originate in the sibling CodeSentinel-1 repository. For a self-contained email package, use the included CodeSentinel-1 import manifest to populate the local source cache.