CortexDB 0.9.0 — new storage engine, benchmark-grade recall, vector-only defaults

⚠️ BREAKING RELEASE. 0.9.0 ships a new on-disk format and new engine internals. Data directories created by 0.8.x are not supported — start 0.9.0 on a fresh data directory (export what you need from 0.8.x and re-ingest). The store now carries a format-version gate that refuses mismatched generations fail-closed instead of corrupting silently. The fresh-install configuration default also changes: see "Vector-only by default" below.

This is the largest CortexDB release to date: 372 commits merged from three long-running engineering campaigns — stress (storage engine + scale), tq2 (quantization + capacity), and openai (recall quality) — plus a release-day soak-validation pass that fixed four production defects at 20M+ vector scale.

From the stress campaign — the unified storage engine

• One WAL. The dual write-ahead-log stack is gone. Captured events and derived artifacts now live in a single unified, append-only log — one source of truth, one replay path, deterministic rebuilds of every index. This is the core of the breaking on-disk format change.
• Durable indexer tail. Indexing is a durable log consumer with a persisted floor: boot means "continue from the floor", crash recovery re-indexes at most one in-flight window, and a poison event is durably excluded instead of freezing ingestion. Bulk accepts measured at ~18,000 events/s aggregate on a 32-core box (~4× the dual-WAL stack).
• Layer-store snapshots (A1). Episodes/beliefs/concepts snapshot to disk and resume from sequence watermarks; combined with tail-only vector reconcile and a bi-temporal boot snapshot, cold boot at 72M vectors went from ~80 minutes to under 2 minutes.
• jemalloc everywhere. The Docker image and production guidance ship whole-process jemalloc with tuned decay — −55% RSS at scale versus glibc (63.5 GB vs ~140 GB at 72M vectors). (A per-column-family RocksDB cache-tiering experiment was measured net-negative and removed.)
• Bounded background maintenance. The memory-state (methylation) sweep is index-driven and budget-bounded, and compaction's content-store scan is budget-capped with resume — write-stall spikes from background scans are gone (compaction pass ~40s → ~5s).
• Validated at scale. The engine was soak-proven on a 72M-vector corpus (spot i4i.8xlarge): sustained mixed read/write load, graceful spot-reclaim handling, restart replay parity, and a measured capacity envelope (~2,400 indexed writes/s clean; read ceiling scales with corpus size).

From the tq2 campaign — quantization and capacity

• TQ2 quantized hot tier (new default). In-memory vectors are stored ternary-quantized with full-precision rescoring on candidates. Quality was gate-checked against real embeddings at recall@10 = 1.000 vs the fp32 baseline.
• fp16 vector rows on disk (new default). Halves vector row storage; all read paths decode through a single seam. Part of the breaking format change.
• Cold tier + residency cap. CORTEX_VECTOR_RESIDENT_MAX bounds the hot tier; overflow demotes to an fp16 cold tier that still participates in search via rescoring. RAM stays flat as the corpus grows.
• Single-write coalescing (default on) and TQ2-with-enrichment compatibility — the quantized stack is the production path, not a benchmark trick. Opt-outs exist for every default (CORTEX_HNSW_QUANTIZATION, CORTEX_VECTOR_FP16_ROWS, CORTEX_UNIFIED_WAL).

From the openai campaign — benchmark-grade recall

• Recall correctness at bulk scale. The inline mirror is now the primary applier for bulk writes with the tail as an idempotent catch-up; a whole class of "frozen watermark" defects was eliminated with structural guards (registration obligations that cannot leak, in-flight age reclaim, and IndexerGauges — watermark/head/lag/inflight — on /v1/admin/metrics). Drain-complete now has an exact machine-checkable gate (lag==0 && inflight==0).
• Read-path latency work. The WAL supplement is skipped entirely on a settled store, walks in small pages bounded by the indexed watermark otherwise, and query embeddings are front-cached — hundreds of milliseconds off every recall on quiescent stores.
• Deterministic query-shape classifier routes question types without oracle input, and namespace isolation (ADR-MEM-007) guarantees derived artifacts can never displace raw evidence in ranked retrieval.
• Provider reliability surfaced. Runtime provider health checks, freshness-gap records for provider outages (recoverable instead of silent enrichment loss), and transient embedding-transport retries.
• LiteLLM gateway support for routing embeddings/enrichment through a local gateway with multi-key pools — cheap, fast, provider-agnostic.
• Measured result: the targeted-recall gate hit 30/30 twice under a blind protocol, and the campaign's read path is what produced the published LongMemEval-S and LoCoMo numbers.

Release-day hardening (found by the 0.9.0 soak, fixed before release)

• Parallel indexer drain. CORTEX_INDEXER_CONCURRENCY now drives concurrent page application in the tail (default: available cores). Measured 13–16× drain throughput at 21M vectors (~365 → ~6,000 events/s).
• Recall memory blow-up fixed. A recovery path materialized the entire captured log per recall on large stores (OOM under concurrent load). It is now a hard-capped bounded walk; recalls at 21M vectors complete in ~45 ms.
• Snapshot persists no longer stall traffic. HNSW shard snapshots serialize from a lock-free clone; the former ~60s rolling write/read stall sweep on every persist cycle is gone (worst write p95 during a full snapshot: 3.5 s → 62 ms).
• 48-hour sustained soak on the release binary at 200 writes/s + 60 recalls/s over a 20M+ vector corpus: zero write errors, zero indexing failures, flat memory.

Vector-only by default (breaking configuration change)

A fresh 0.9.0 install (binary or Docker) runs exactly the published benchmark configuration: hybrid vector+BM25 retrieval — and nothing else. Every subsystem beyond retrieval is now an explicit opt-in:

If you relied on the 0.8.x default-on layer scheduler or the Docker image's baked-in enrichment config, set the flags above explicitly. With nothing set, CortexDB makes zero background LLM calls.

Also in this release

• /v1/admin/health and telemetry report the real release version (goodbye hardcoded 0.1.0).
• GET /v1/admin/flush-views is retired — graceful shutdown (SIGTERM) runs the final snapshot persist.
• Deep index repair re-chunks from surviving WAL content, closing a mid-crash recovery gap; POST /v1/admin/index-audit gained vector auditing ({"vectors": true}).
• Store format-version gate: a newer-generation data directory is refused fail-closed instead of half-opening.
• SDKs (cortexdbai on PyPI/npm), the MCP server, connectors, and the CLI are unchanged and fully compatible — the v1 API surface did not change in this release.

Upgrade notes

1. Do not point 0.9.0 at a 0.8.x data directory. Start fresh and re-ingest (the WAL format, vector row format, and quantization defaults all changed).
2. Review the vector-only defaults table above and explicitly enable the features your deployment uses.
3. docker pull cortexdb/cortexdb:0.9.0 (or :latest); binaries for linux-amd64/arm64 are on the releases page with sha256 checksums.