memshelf — Roadmap
Milestones are deliberately thin. M0 validates the pattern with zero new code; every later milestone must justify itself against what M0 already achieves.
M0 — Pattern validation, no code — complete (2026-07-13 → 2026-07-22)
Epic: #164 (closed).
Exit criteria met: recall test 5/5, ledger + recall-cost numbers written down
(demo.md), and the annoyance log became the M1 backlog verbatim.
Case B closed 2026-07-22 (33 episodes, zero loss; verdict episode on the shelf).
Prove the loop works with docshelf as-is plus conventions. Protocol, kit,
and measurement methodology: docs/M0.md; the prompt-only skill and
recall-rule snippet live in adapters/claude-code/.
- One real memory shelf (private repo — cloud sessions need a remote).
- Case A, retro-import: the author’s long sqst homework-review dialog, segmented into depersonalized episodes + a session digest (import mode of the skill; raw transcript never committed anywhere).
- Case B, live shelving: recall rule +
/shelveduring ~a week of normal work on a real project. - Measurement via the token ledger (
ledger.tsv, see ARCHITECTURE → Accounting): standing cost vs shelved mass vs recall cost per question, docshelf-benchmark methodology.
Exit criteria: 5 known-answer recall questions answered correctly from a fresh session via INDEX navigation; ledger numbers written down; the annoyance log filled. That log is the M1 backlog.
M1 — memshelf-mcp thin server — complete (closed 2026-09-27)
Epic: #165 (closed). Closed by the owner on 2026-09-27: every scope item shipped (0.1.0, 2026-07-25, plugin with it), and the exit clause holds on the record. In the 14 days to 2026-09-27 all 37 derived-layer renders on main-memshelf were made by the shelf bot and none by hand, doctor on its main is clean, and five project shelves ran shelve → doctor → rebuild on 2026-09-26. The one manual fix in CHANGELOG 0.2.0 (#56, 2026-07-27) predates that window.
Only what M0 proved annoying, expected:
memshelf_shelvewith the digest contract validation + redaction pass + auto-commit (the three things a prompt-only skill can’t guarantee).memshelf_recall/memshelf_search/memshelf_indexas thin wrappers.- Episode frontmatter schema +
memshelf_doctorchecks. - Claude Code adapter (plugin):
SessionStarthook injects INDEX; aSessionEnd/PreCompacthook pushes the shelf for durability (MEMSHELF_AUTOPUSH). Shelving-before-compaction and session digests need the LLM, so they stay agent-driven (the/shelveskill + recall rule) — a hook is a shell command, not the model (DECISIONS 2026-07-22). Adapter code only; core stays host-agnostic (ARCHITECTURE → Portability model). - CLI mirroring the MCP tools (
memshelf shelve|recall|search|index) — the portability surface for hosts without MCP. - Repo bootstrap:
memshelf init→ docshelfinit_shelfwith memory conventions (provider: none, fixed categories,storage: git-local— auto-commit, no remote;plainvia flag). memshelf_statsover the ledger (standing cost / shelved mass / compression ratio) — the transparent-savings feature.memshelf_import— tool-assisted whole-dialog backfill (M1 candidate; confirm need from M0 Case A experience).
Exit criteria: dogfooded on two real projects for two weeks; a full
shelve→compact→recall cycle survives without manual repair; doctor clean.
M2 — Policy, hygiene & the context advisor
Epic: #166 — the current period since 2026-09-27.
Token-budget monitor that proposes shelving (never forces).— done (#14): the advisor takes the budget and reports headroom; the proposal, not the action, is the output.Context advisor(the “where did my window go?” feature, MANIFEST hero scenario 2) — done (#14):memshelf advise/memshelf_advisereports static overhead vs memshelf’s own cost vs live topics vs reclaimable, and ranks shelve/drop/rollup proposals net of what each one costs. Host-agnostic as decided in ARCHITECTURE open question 7: the window breakdown is a caller input, and the tool contributes what a self-assessment cannot — measured overhead, verification of “already shelved” claims against the episodes, and a deterministic ranking. Called with no occupants it is the first-run view of the shelf, and says so rather than reporting a clean window. The exit criterion below (proposals accepted, not overridden) stays open — it is a dogfood measurement, not a code deliverable.Retention:— done (#15): opt-inretain_until, purge tool, reindex after purge.retain_until,memshelf purgedry-run by default, sweepsdocs/andarchive/, states the git-history caveat instead of implying erasure.Rollups: consolidate old episodes into digest-of-digests, archive category.— done (#15):memshelf rollupmoves a period’s episodes into thearchive/sub-shelf behind one digest-of-digests. Navigation shrinks; recall, search, ledger and stats are untouched.Configurable PII/secret pattern packs per shelf.— done (#16): a flatPOLICY.patternsfile (<kind> <regex>,#comments) at the shelf root, read by the shelve redaction pass, bydoctor(policy-pattern-at-rest,policy-pattern-invalid) and by the pre-commit guard alike — one pack, three consumers;initscaffolds the template.Derived files rendered by a bot, not by(#58, decided 2026-07-31) — done:shelvedate/notes/display_title/descriptionmoved into the episode frontmatter,shelvewrites and stages only the episode, andmemshelf rebuildrendersledger.tsv/INDEX.md/stats.svg/.meta.jsonfromdocs/. Two parallel shelves now merge cleanly by construction;memshelf resolvestays as the fallback for a real same-slug collision or a shelf without the bot. Bot + PR-guard workflows:adapters/shelf-repo/. Rollups (below) build on the same regeneration path.
Exit criteria: a shelf with 100+ episodes keeps INDEX within its budget
— INDEX_BASE_TOKENS + INDEX_TOKENS_PER_ENTRY × listed episodes, i.e. the
price of a line stays flat as the shelf grows — and recall precision doesn’t
degrade (re-run the M0 question set); the advisor’s shelve proposals are
accepted (not overridden) most of the time in dogfood use.
Revised 2026-08-21. This used to read “100+ episodes and INDEX under ~10 KB”, and the two clauses contradicted each other. INDEX lists episodes, so its size is O(episodes) by construction; a fixed ceiling is therefore unreachable past some shelf size, and the only mechanism that lowers the number afterwards is
rollup— which buys compliance by archiving live memory. Measured on the author’s 113-episode shelf: the structural floor, with every description deleted and the link de-duplicated, is ~3800 tokens (~15 KB), so “100+ episodes under 10 KB” was not merely tight but arithmetically impossible. The derived constantINDEX_BUDGET_TOKENS = 2500inherited the contradiction and added a unit error — “~10 KB at chars/4” holds only if one character is one byte, and this shelf’s Cyrillic runs ~1.42 bytes per character, making the two clauses two different budgets (10 KB ≈ 1800 tokens; 2500 tokens ≈ 14 KB). The budget is now linear in shelf size, which puts the check on the quantity formatting can actually control. Rollups stay in M2, triggered by INDEX’s share of the context window (INDEX_CONTEXT_SHARE) rather than by a threshold that growth alone would breach.
M3 — Retrieval upgrades, reuse layer & second surface
Epic: #167.
Embeddings sidecar behind the same— done (#17):searchsignature (docshelf’s documented extension point).memshelf semantic buildwrites a model2vec index under the state directory (never in the shelf);searchfuses grep and nearest-chunk rankings by reciprocal rank when the index is usable and is unchanged otherwise;MEMSHELF_SEMANTIC=offis the kill-switch;memshelf search-benchmeasures. Optional extramemshelf-mcp[semantic]; no new MCP tool (DECISIONS 2026-09-10).- Chat-project surface documented end-to-end (Desktop/web, manual triggers).
- Cross-shelf meta-INDEX experiment (federation open question).
Archive-as-raw-material (MANIFEST hero scenario 3): tag/graph views over episodes (frontmatter tags + cross-episode links in Decisions), quarter retrospectives, fork-a-thread (bootstrap a fresh session from INDEX + selected episodes — continue an old discussion branch as a sub-thread without its full history).— done (#18):memshelf tags,graph(JSON / Mermaid, edges carry the section they sit in),retro --quarter,fork --episode ID [--section S]; CLI only, by decision (DECISIONS 2026-09-10). One real fork on the dogfood shelf ran end-to-end.Artifact mirror experiment: INDEX (± episodes) as private claude.ai artifacts for phone-side reading (open question 8).— done (#18):memshelf mirrorrenders one static page; publishing it is a host action, no adapter (ARCHITECTURE open question 8, resolved).
Exit criteria: search-miss rate measurably better than grep baseline on
the dogfood shelves (met, #17: 30 hand-written queries on the dogfood shelf —
10 Russian paraphrases, 10 English queries against Russian episodes, 10 short
keyword queries — grep misses 29/30, the hybrid 14/30 at k=5 and 9/30 at k=10;
MRR 0.03 → 0.43; no query where the hybrid lost a grep hit); one non-author
user runs the chat-project flow from docs alone; one real “fork from
episode” session succeeds end-to-end (met, #18).
Explicitly deferred
- Automatic topic segmentation (needs M0/M1 experience to judge feasibility).
- Any UI.
- Multi-user / shared shelves.