Memory

Status: active Scope: src/nexrur/memory/ Issue: #156

0. One sentence

nexrur.memory provides generic embedding, vector persistence, indexing, and semantic recall. The app owns every source declaration and every physical runtime artifact; Memory never owns app business categories, diagnosis, or campaign decisions.

1. Ownership boundary

Memory owns:

• the Doubao embedding adapter and namespace-bound cache;
• FAISS and explicit in-memory vectorstore implementations;
• checksum-bound save/load and mechanical health checks;
• generic RAG indexing and retrieval;
• app-contract loading, source-glob expansion, limits, and build receipts;
• an explicit historical outcome.json backfill utility.

Memory does not own:

• App, product, subagent, phase, forum, paper, alpha, or audit semantics;
• which app files should be indexed;
• diagnosis reason selection or campaign route/halt/restart policy;
• schema or current-run evidence truth;
• automatic App-root discovery through CWD, workspace, asset_root, or a retired RunContext;
• fallback success when a contract, cache, or persisted store is damaged.

Memory recall is historical auxiliary context. It cannot replace current-run evidence, authorize a transition, or prove a business claim.

2. App-owned contracts

An app that uses the two shared stores declares:

<app_root>/.gemini/skills/_shared/memory/vectorstore.yml
<app_root>/.gemini/skills/_shared/memory/contentstore.yml

The contracts declare owner, the store kind, an explicit embedding dimension, an App-root-relative persistence path, source groups, exclusions, and mechanical limits. Source group names and source_type values are opaque app data.

The substrate may enforce a declared separation flag mechanically. For example, when forbid_raw_material: true, a group whose declared source_type is exactly raw_material is invalid. The substrate does not infer a type from a file name, directory name, product name, or payload field.

Rules:

• app_root is mandatory and must already exist;
• config paths, source globs, exclusion globs, persistence paths, cache paths, CLI paths, and resolved symlinks must stay below app_root;
• absolute or parent-traversing source globs are invalid;
• unknown contract fields and invalid types fail closed;
• required groups missing files fail; optional groups may produce an alarm;
• App contracts choose sources; Python does not contain App-specific globs.

There is no substrate-workspace or current-directory fallback. A caller that cannot name the app root does not have authority to read or write app memory.

3. Store split

The standard app layout is:

<app_root>/docs/ai-runs/.memory/
├── vectorstore/
└── contentstore/

vectorstore and contentstore are independent stores. An app may use them for different declared source classes, embedding models, dimensions, retention, and consumers. The substrate does not merge them or copy records between them.

The persistence paths above are conventional contract values, not hardcoded App discovery rules. The physical files always belong to the app.

4. Persistence and integrity

Production FAISS persistence uses:

index.faiss
docs.pkl
doc_id_map.pkl
config.json

The in-memory test backend persists docs.pkl plus config.json.

config.json is the generation manifest and is committed last. It records the format version, backend, embedding dimension, document count, and SHA-256 hash of every data file. Loading stages and validates the complete generation before mutating live state.

Hard rules:

• missing store: return unavailable;
• partial, legacy, malformed, checksum-conflicting, backend-conflicting, or dimension-conflicting store: raise an integrity error;
• persisted pickle data is loaded by a restricted unpickler that only admits the local Document type;
• batch add validates the full batch before mutation;
• replaying the same add/save/load is idempotent;
• no load failure may leave a half-mutated in-memory store;
• an old pre-manifest store must be explicitly rebuilt; it is not silently upgraded or treated as empty.

The store is a derived index. Its source documents remain the source of truth.

5. Embedding cache

The active backend is Doubao/Ark embedding. Its model endpoint credentials stay in the LLM/provider environment; Memory never prints or persists credentials.

Cache identity contains:

backend + model + embedding dimension

The cache format also has an explicit version. A legacy cache, malformed JSON, namespace conflict, wrong vector dimension, or non-finite vector fails closed. Writes use a same-directory temporary file, flush/fsync, and atomic replacement. Calling save() first loads any existing valid entries, so a no-op save cannot erase the cache.

6. Public execution surfaces

Core construction is explicit:

rag = create_rag_pipeline(
    app_root=app_root,
    config_path=config_path,
    store_path=store_path,
)

create_rag_pipeline requires the requested store path to equal the persistence path declared by the app contract. The embedding client and vectorstore must match the contract dimension before use.

App-wide builds use:

build_vectorstore_from_config(rag, app_root=app_root)
build_content_store_from_config(content_store, app_root=app_root)

Builds validate the caller-provided runtime object against the contract. They do not mutate the object's path or dimension to make a mismatch appear valid. Required-source, read, embedding, indexing, and persistence failures are reported as failures and raised; there is no strict=False compatibility path.

The low-level historical utility only scans explicitly supplied completed outcome.json files. Malformed outcomes fail the rebuild instead of being silently counted as zero errors. The app-wide source policy remains vectorstore.yml; the historical utility is not an alternative policy engine.

7. CLI

The operations entry point is:

nexrur-memory health
nexrur-memory rebuild
nexrur-memory query
nexrur-memory build-vectorstore
nexrur-memory build-contentstore

Every command requires --app-root. Contract-based commands accept an optional App-root-confined --config-path. Rebuild/query paths must equal the contract's vectorstore persistence path. health may inspect another explicit path below the same app root without changing it.

Backends are explicit:

• faiss: production;
• memory: tests and deliberate lightweight runs.

There is no auto backend and no silent FAISS-to-memory fallback.

8. Consumer boundary

Evidence/Diagnosis may read Memory through an explicitly declared memory source. They must provide the current app root and the relevant contract/store paths. Missing or unavailable optional recall may become a warning, but corruption or contract conflict must remain visible and may not be presented as a healthy empty store.

The app or a generic engine decides when a build is authorized. Memory only executes the requested action. Campaign-cycle counts, terminal rules, product promotion, and archive policy do not belong in this package.

9. Migration rule

This contract intentionally removes these retired forms:

• workspace= / asset_root= aliases;
• implicit CWD or substrate-root persistence;
• RunContext-owned memory construction/finalization;
• App-specific ContentStore source enums and directory backfill;
• diagnosis/terminal/lesson compatibility helpers in Memory;
• legacy vectorstore/cache formats without manifests and namespaces;
• silent configuration, backend, and load fallback.

Existing legacy index directories are data, not code migrations. Do not move or delete them implicitly. Rebuild them from the current app-owned source contract.

10. Submission gate

Changes to Memory must verify at least:

• Python compilation;
• focused Memory tests;
• bound App contract parsing where an app is in scope;
• path escape rejection;
• corrupt store/cache fail-closed behavior;
• backend/model/dimension conflict rejection;
• replay idempotence and no partial mutation;
• git diff --check and credential scan.

Implementation details and public symbol ownership are summarized in src/nexrur/memory/_DESIGN.md and src/nexrur/memory/contracts/.