memshelf — Decision log
Newest last. Earlier entries were made while the design lived as RFC-0001 in docshelf-mcp.
| Date | Decision | By |
|---|---|---|
| 2026-07-13 | RFC draft created. Form = companion project over docshelf (not a docshelf subpackage, not a prompt-only pattern); primary surface v1 = Claude Code / Cowork. | draft, author-merged (ignatenkofi/docshelf-mcp#42) |
| 2026-07-13 | Portability requirement: core/episode format/on-disk shelf contain nothing host-specific; three-ring model (core → MCP+CLI surfaces → host adapters); triggers are adapter territory; prompts are core assets. | author note, merged (ignatenkofi/docshelf-mcp#43) |
| 2026-07-13 | Prior-art survey (LANDSCAPE.md): niche confirmed open — no project combines episodes+digests+git+INDEX navigation. Design amendments: mechanical eviction, injection budgets/KV-cache discipline, prompt-injection-on-recall defense, platform-collision defense, subagent-deposit trigger (v2), memory-tool adapter ring. Direction kept, scope sharpened. |
draft, author-merged (ignatenkofi/docshelf-mcp#44) |
| 2026-07-13 | Name + placement resolved: memshelf, repo ignatenkofi/memshelf-mcp (PyPI memshelf / memshelf-mcp checked free). Repo created by author; seeded from the RFC. |
author |
| 2026-07-13 | Storage defaults reworked per author’s git concerns: modes plain / git-local (default — auto-commit, no remote configured) / git-remote (opt-in, private-only, doctor-enforced, autopush: false). Accidental-exfiltration threat handled by absence of a remote, not absence of git. “Store inside Claude” (artifacts/attachments) assessed: unsuitable as canonical store (breaks portability), adopted as optional read-mirror idea (M3 experiment). |
author + draft |
| 2026-07-13 | Positioning: hero scenarios added to MANIFEST — (1) the weeks-long dialog that must not die (primary wedge), (2) context advisor: surface where the window goes, flag shelvable episodes (promoted into M2; also the onboarding moment), (3) archive as raw material: tags/graph/retrospectives/fork-a-thread (M3). Many-repos overload noted as advisor diagnosis territory, not a direct memshelf fix. | author + draft |
| 2026-07-13 | Import mode adopted (author: “выгрузить весь диалог на полку”): whole-dialog retro-shelving = segmentation into episodes + one session digest, mode: import in the ledger. Distinct from the “transcript archiver” non-goal — segmentation and digests are mandatory, raw excerpts optional, the raw transcript itself is input-only and never committed. M0 does it manually via the skill; memshelf_import is an M1 candidate. |
author + draft |
| 2026-07-13 | Transparent token accounting adopted (author request, mirroring docshelf’s measured-savings ethos): ledger.tsv per shelf, one row per shelve; memshelf_stats in M1; derived metrics (standing cost / shelved mass / compression / recall cost per question) defined in docs/M0.md. |
author + draft |
| 2026-07-13 | M0 launched. Kit: prompt-only /shelve skill (live + import modes) and recall-rule snippet in adapters/claude-code/; protocol in docs/M0.md. Case A material: author’s long sqst homework-review dialog (Opus 4.7 era) as retro-import; episodes depersonalized per sqst PII policy (roles/codes, no student identifiers), raw transcript stays outside all repos. M0 shelf uses a private remote (cloud sessions are ephemeral) — a documented deviation from the git-local default. |
author + draft |
| 2026-07-22 | Claude Code plugin hooks scoped to what shell hooks can actually do (#11): a hook is a shell command, not the LLM, and per the current hook contracts SessionStart can inject context while PreCompact cannot and SessionEnd runs after the agent stops. So the plugin ships SessionStart → inject INDEX.md (recall bootstrap) and SessionEnd/PreCompact → mechanical shelf push (MEMSHELF_AUTOPUSH, off by default). The ROADMAP’s “PreCompact shelve / SessionEnd session digest” is revised: both need the LLM, so they stay agent-driven (the /shelve skill + recall rule), not hooks. |
author + draft |
| 2026-07-22 | Server topology: separate MCP process (closes open question 3 / #28). memshelf-mcp ships its own FastMCP stdio server (server.py, mirroring docshelf’s), not an in-process registration alongside docshelf tools — shipped this way in the M1 slices and confirmed as the decision: the core imports docshelf as a library (one dependency), while the server stays independently versioned, installable, and testable; users who want both attach two config entries. Revisit only if the two-entry config proves a real adoption burden. |
draft, per shipped M1 |
| 2026-07-22 | memshelf_stats reports claimed AND realized economy (#6; closes the Case B verdict’s measurement gap). Claimed = ledger-derived (standing cost = tokens(INDEX) + Σ digest tokens; shelved mass = Σ approx_tokens_in; compression = mass / standing). Realized = an opt-in recall log (recall --log → recall-log.tsv), where each fetch’s saving = the episode’s original mass − tokens actually fetched. The true fetch-hit rate (share of past-work questions closed by a shelf fetch vs re-reading the repo / owner re-explaining) needs a denominator that can’t be captured mechanically, so stats reports the measurable side (recall count, episodes recalled, realized tokens saved) and stays honest about the rest. chars/4 estimator, tokenizer-independent. |
author + draft |
| 2026-07-31 | Derived shelf files become bot-rendered; notes moves into the episode (#58; adopts project-atlas ADR 0007). Parallel shelve runs collide in four files at once — INDEX.md, ledger.tsv, docs/*/.meta.json, stats.svg — because every writer regenerates all of them. Target shape (M2): shelve writes only the episode file; ledger.tsv becomes derivable once its one non-derivable column (notes) lives in the episode frontmatter; INDEX/ledger/.meta/stats are rendered by a bot on main behind a PR guard on the derived paths. The conflict class then cannot occur by construction rather than being resolved after the fact. memshelf resolve (shipped, live run 2026-07-30) stays as the fallback for hand-merges and for shelves that keep the current layout. Rejected: .gitattributes merge=union — GitHub’s server-side merge does not run custom drivers, the same reason the atlas turned it down; and push-straight-to-main on shelve, which breaks the draft-PR discipline of night shifts without closing the two-simultaneous-push race. |
author + draft |
| 2026-08-01 | Context advisor: heuristics only; the window breakdown is an input, not something the tool hunts for (#14, closes open question 7). A library cannot see the window it is asked about, and a parser for a host’s /context output would work on exactly one host and rot with its next release. So the advisor reuses the split shelve and rollup already have: the model reports what only the model knows (labels, rough sizes, which topics are closed), and the tool contributes the four things a self-assessment cannot — memshelf’s own standing cost, measured; verification of every “already shelved” claim against the actual episodes (a false claim is refused loudly, since acting on it destroys work); arithmetic net of what shelving adds forever (~200 tokens per episode, so nothing under 2000 is proposed at all); and a deterministic ranking, without which the M2 exit criterion “proposals accepted, not overridden” cannot be measured. Proposals only — the tool writes nothing. Rejected: harness-parsing in core (host-bound, and portability rule 2 puts it in an adapter filling the same input); inferring “already shelved” from a title match (it becomes a hint, never an auto-drop). |
author-planned, draft |
| 2026-08-14 | A packaged host gets one global shelf through $MEMSHELF_SHELF_PATH; an explicit shelf_path still wins. Claude Desktop configures an extension only through the environment, so a bundle cannot ask the model to remember a path — but a shelf per project is the whole point of the portability rules, and one global setting would flatten that. Precedence resolves it: a call that names shelf_path uses it (a project pins its own shelf by saying so in its instructions), a call that names none falls back to the variable (the global shelf, and the case for a chat outside any project), and neither present is an error naming the variable rather than a write to "". Resolution lives in the input models, so the MCP surface and any future caller of tools.py inherit it; the CLI keeps --shelf required, which is an asymmetry left open deliberately rather than settled by default. Rejected: a user_config value the host injects into every call (hosts do not rewrite tool arguments), and reading the setting once at import (a host may set it after the process starts). |
author + draft |
| 2026-08-15 | The CLI honours $MEMSHELF_SHELF_PATH too, and announces when it did (#86, settles the asymmetry the row above left open). The variable working for the MCP tools while --shelf stayed required reads as a bug to whoever sets it, and the CLI is the documented portability surface — “anything that can run a shell command can drive the shelf” — so it is the one most likely to be scripted around, where an environment variable is the natural way to say which shelf. The objection to honouring it (an implicit shelf in a script is a footgun) is answered by removing the silence rather than the fallback: a resolved default prints memshelf: shelf from $MEMSHELF_SHELF_PATH: … on stderr, where it cannot corrupt piped JSON. One resolver, shared with the input models, so the two surfaces cannot drift apart. Rejected: documenting the split as deliberate (it makes the CLI the odd surface out for no benefit a script can use), and an argparse default= (frozen at import; a host may set the variable after the process starts). |
night shift, draft |
| 2026-09-10 | Archive reuse (#18) ships as CLI verbs, not MCP tools; cross-episode links are bare ids, not a link syntax. tags / graph / retro / fork / mirror are pure functions of the files on disk, and each is either a report a person reads outside a session or (fork) the first message of a different session — nothing a tool inside the session would do better, while every published tool costs every turn its description (#111, 320-char cap and the descriptions * 4 < total guard). The graph reads references the way the dogfood shelf already writes them: an episode names another by its id inside Decisions / Open threads (110 live episodes, 0 [[wiki]] links, ids mentioned up to 9× each), matched as whole tokens so …-auth does not fire inside …-auth-followup. fork wraps INDEX and every episode in the recall data envelope so the receiving session treats them as material, and refuses unknown ids or missing sections rather than emitting a document with a hole. mirror answers open question 8: the mirror is a generated static page; publishing it (claude.ai artifact, gist, a phone’s Files app) is a host action outside the tool, so no adapter and no vendor binding. Rejected: a memshelf_fork MCP tool (prefix cost for a document the session itself never consumes; recall already covers the in-session case); [[wikilinks]] as the link syntax (would have to be taught and back-filled — the ids are already there); an artifact-publishing adapter (vendor-bound, breaks portability principle 9; the page is enough). |
night shift, draft |
| 2026-09-10 | The embedding sidecar (#17) lives outside the shelf, fuses by rank, and is an optional extra built from the CLI — search keeps its signature and the tool surface does not grow. A shelf is a git repository whose every file is meant to be read and diffed; a derived vector blob inside it would ride into every commit and every clone, so the index is a JSON file under the state directory keyed by the resolved shelf path, rebuilt from the episodes (incrementally, by file (mtime, size)) and droppable at any time. Fusion is reciprocal rank, not a score mix: grep scores are occurrence counts and cosines live in [-1, 1], and no scale calibrates one against the other while ranks do; the fused score stays an integer so the hit shape is unchanged. The model is model2vec static embeddings (no GPU, no service, one process, ~3 s to load once) behind memshelf-mcp[semantic], because the base install must stay light and a shelf without the extra must behave byte-for-byte as before — which is also why $MEMSHELF_SEMANTIC=off exists and why search only goes hybrid when all three of extra, index, and switch are present. Building is a CLI verb, not a tool: it is a one-off per shelf that a session never needs to run, and every published tool is charged to every turn (#111). Measured, dogfood shelf, 30 paraphrase / cross-language / keyword queries: grep 1/30 in the top 5, hybrid 16/30 (21/30 in the top 10). Rejected: the index inside the shelf (see above); a vector database or an embedding service (a dependency to run and a place for the shelf’s text to leave the machine — portability principle 9); auto-building on first search (a multi-second surprise inside a tool call, and a write the caller did not ask for); a memshelf_semantic_build tool (prefix cost for a one-off). |
night shift, draft |
| 2026-09-25 | Served-code freshness is said in the tool’s own response, not only by doctor (#125, #158; dogfood shelf thread 2026-09-10). served-code-differs has been a doctor finding since #125, and both later incidents were still found sideways, because nobody runs doctor when a tool merely looks buggy. So the server hashes its own package once per process and the reference checkout (the resolution doctor uses: $MEMSHELF_CHECKOUT, else memshelf-mcp next to the shelf) once per path until HEAD moves or five minutes pass, and prepends a warning — same code, both hashes, both paths — as the first key of every envelope, success and error alike, at the one place responses are composed. Three outcomes stay distinct: differs → the warning; same → nothing; unknown → nothing per call, one line in the initialize instructions naming MEMSHELF_CHECKOUT. MEMSHELF_FRESHNESS_WARNING=0 is the opt-out for a host where the copy is meant to differ, and the instructions then say it is off. A new key rather than warnings, which already means three shapes across the tools. Rejected: a per-call freshness-unknown warning (noise with no fix attached on every machine without sources); re-hashing per call (~1.2 ms × 2, avoidable); an initialize-only line (read once at the handshake, while the stale answer is read per call); prepending into the tools’ own warnings lists (three shapes, no uniform slot). |
night shift, draft |
| 2026-09-28 | A rollup carries the absorbed episodes’ keywords forward — into its own keywords frontmatter, its digest and its description (#172, owner comment on the PR; closes the M2 measurement (b) gap, docs/demo.md). Two rollup generations deep, plain “INDEX → point fetch” navigation resolved 0/5: the rollup’s description — the field docshelf’s INDEX line actually renders, not digest — was a fixed “N episodes folded in”, carrying no trace of what. core/keywords.py derives a short, deterministic, deduplicated keyword list per rollup (tags minus the structural rollup marker, display_title words, ## Digest words, minus a generic RU+EN stopword list) and writes it into all three places a reader or the next rollup could look. Transitivity: a source tagged rollup contributes only its own stored keywords field, verbatim — never terms re-derived from its generic digest prose — so a rollup-of-a-rollup does not lose the first generation’s words the way #172 reported; a pre-#172 rollup with no such field contributes nothing, honestly, rather than mining its boilerplate. Overflow (both the stored list’s own cap and the digest’s 120-word contract when the caller’s own prose leaves little room) is resolved by one rule: frequency (how many absorbed units raised the term — an inherited rollup counts as one unit) first, then freshness (the newest contributing unit’s date), then alphabetical for full determinism. Rejected: mining a nested rollup’s own digest for terms (it is mechanical prose — “N episodes folded in” — not a topic, and would rank recordkeeping words over what the first generation actually carried); a manual --keywords override (the owner asked for the tool to carry them, not for another field to remember to fill in); backfilling already-published rollups from this change (a separate, owner-run step on the shelf itself, since rollup cannot safely re-open and re-absorb episodes a prior rollup already moved to archive/). |
night shift, draft |
| 2026-10-07 | rebuild --check also compares INDEX.md and stats.svg, and a derived file that cannot be rendered fails the run (#186, part 1; revises the #58 choice to leave INDEX.md and stats.svg to their writers and outside the check). --check is the bot’s idempotency proof and a plain shelf’s PR guard; skipping INDEX.md and stats.svg let it answer ok over a tampered INDEX, and outside --check a failed INDEX write was one warning among the description-clamp warnings while the CLI exited 0. INDEX is compared through docshelf’s public Shelf.doctor() stale-index: docshelf owns the format, and that check compares against the render rebuild_index writes. The chart is drawn in memory from the ledger. A render whose input drifted in the same run (.meta.json for INDEX, ledger.tsv for the chart) is not compared, and the warnings say so — the verdict is “drifted” already, and either answer would be about a file the rebuild is about to re-render. A comparison that cannot be made is an error, not a pass: a render that raises, or docshelf skipping its comparison while split directories are uncommitted. Errors go to a new errors field and make ok false (CLI exit 1); each message also stays in warnings. rollup, purge and resolve run the same rebuild and carry its errors into their own results, so a failed render fails them too (ok false, status: attention for resolve, exit 1). Still not compared: each split document’s SUBINDEX.md, which rebuild_index writes along with INDEX — stale-index covers INDEX.md only — and INDEX on a shelf without docs/, where docshelf compares nothing. The latter is a warning, not an error: git keeps no empty directory, so a clone of a shelf with no episodes yet has no docs/, and its rebuild passes. Rejected: rendering through docshelf’s private Shelf._index_text (exact, but the first private docshelf API this code would depend on, under a <1 pin); only saying in the output that --check does not cover INDEX/stats (the bot’s idempotency step would still prove nothing about the file every session reads); keeping a failed chart write a warning as “cosmetic” (--check now flags a stale chart, so rebuild exiting 0 before a --check that exits 1 would be the same contradiction). |
night shift, draft |
| 2026-10-07 | The recall log gets a fourth column, ts — the recall’s UTC time to the second — and a log that already exists keeps its three-column header (#193). Realized savings could only be summed over the whole file: rows pile up in the working tree and are committed in batches, sometimes with other files, so git history cannot date a recall. recall stamps the row itself. A header is written once, with the file, and never rewritten, so an existing log carries the three-column header with rows of both widths under it, and every reader takes both: stats reads the first three cells and skips the header by position (unchanged), and resolve drops either header — RECALL_LOG_HEADERS lists them — rather than counting the older one as a recall, and writes the newest header any side of the merge carries. The union stays a multiset: older rows have no time, and two recalls of one section within one second are still byte-identical. A resolve from memshelf ≤ 0.4.0 knows only the three-column header, so on a log created under the new one it would keep that header as a row (stats: one recall too many, with no tokens); existing logs never get the new header, which keeps that case to new logs. Open for the owner: the precision. The second is the issue’s example and leaves every later grouping (day, week) possible; the day is the alternative #193 offers for a shared shelf, where a recall time says when someone was working. Rejected: rewriting an existing log’s header on the first new recall (an older memshelf on another machine keeps appending three cells, so readers must take both widths anyway, and the append-only file would get a rewritten first line); a fourth cell under the unchanged three-column header (a column no reader can name); a stats --since filter in the same change (optional in #193, and it widens the stats tool — a follow-up once the column has data). |
night shift, draft |
| 2026-10-07 | shelve turns docshelf’s refusal of a foreign <slug>/ directory into a refusal of its own, instead of asking docshelf to relax the guard for split=False (#186, part 3; docshelf-mcp#115). docshelf after 0.5.0 refuses add_document beside a directory named like the document that is not a split it wrote, with split=False too, although its unsplit path deletes only split-shaped directories. The #115 PR left two options: relax that pre-flight for split=False, or catch the error in memshelf. docshelf kept the strict guard, so shelve catches it. It catches the base FileExistsError, because 0.5.0 has no SplitDirConflictError to import, and any FileExistsError from add_document means a path is in the write’s way. The error (EpisodePathBlocked, exit 1 in the CLI) says that nothing was written and that the directory has to be moved aside; --amend does not clear it. A kind-changing amend has moved the episode before the write (#90), so the refusal moves it back. The refusal needs a docshelf release with the guard and a floor raised to it: with docshelf 0.5.0, the release the >=0.5.0 floor installs, shelve exits 0 and docshelf deletes the directory and everything in it. That loss is the cost of the rejected pre-check below until such a release; reopening the choice is the owner’s. Rejected: moving or deleting the directory automatically, because it may hold the author’s own files; a memshelf-side pre-check, because it would need docshelf’s is_split_dir, absent from 0.5.0, and would drift from the guard it copies. |
night shift, draft |
| 2026-10-07 | The description cap never cuts inside a code span; within the cap, a single backtick left unpaired is closed at the end of the value, and a longer unpaired run that a code span follows is escaped (#190; the 120-character cap came with 0.3.0). The INDEX line prints the file name in backticks right after the description, so a single backtick the description leaves open pairs with the file name’s and the line loses its file name. A cut inside a span moves before it; when that keeps less than the floor (two thirds of the cap), the span is closed before the …. The value is read as CommonMark reads it: runs pair by equal length, the scan goes on past a run nothing closes, and a backtick inside an autolink or an HTML tag is no run. A longer run nothing closes is literal in CommonMark, but GitHub’s renderer (cmark-gfm) and markdown-it trust a cache once a closer search has failed, and a code span closed after that run can leave the cache saying the file name’s backtick has no closer: with GitHub’s POST /markdown, the line of “a `` b c d” loses its file name, which markdown-it still renders. Escaped, the run renders the same and no search fails. The repair lives in clamp_description, which the write path and the render path (rebuild, rollups) share, so descriptions already cut inside a span are rendered repaired by the next rebuild, and the frontmatter is left as it was. Not modelled: a link destination or title holding a backtick, which CommonMark reads as no run. Rejected: counting backticks for parity, which breaks double-backtick spans; escaping the single unpaired backtick, which renders a cut command as text instead of code; closing a longer run instead of escaping it, which turns the rest of a description that meant it literally into code; dropping a run, which deletes a backtick the author typed (a single one with nothing after it is dropped, as there is nothing to close); a fix in the write path only, which leaves every description on disk broken. Accepted cost: a lone backtick typed as a literal makes the rest of the description code, and an escaped run leaves a backslash in the raw INDEX.md that agents read, each with a warning that names it. |
night shift, draft |
| 2026-10-08 | An amend keeps a description and a section order the call passes back unchanged: the description is not capped again, and the sections keep the order the episode has (#205). The 120-character cap and the canonical section order are rules for what a shelve writes, and an amend applied both again to content already written: passing every field back cut a stored description past the cap to 119 characters, the … included, and moved Findings below Open threads. A description equal to the stored one once whitespace is collapsed on both sides bypasses the whole cap (since #190 the cap also closes a single backtick within it), and the stored value is what is written, its spaces included; rebuild caps the INDEX line it renders from it, as it already did for every description on disk, and the amend warns what that line gets. A changed description is capped as before. A section the episode does not have yet goes right after the last stored section the canonical order puts before it, so an episode in canonical order composes as before, and one that opens with a section the canonical order does not know keeps it on top. Reordering sections stays a hand-edit, which the next amend keeps. Rejected: refusing a stored description past the cap with an error that names the field (the issue’s other option), because the render path already caps the line every session pays for, and the refusal would leave such an episode no way to amend its digest without shortening a description the caller did not change; the order passed in the call, because an MCP sections value is a JSON object that a wrapper builds from whatever mapping it holds, so a call that changed nothing would still reorder; exact equality of the descriptions, because a wrapper that trims what it read (the issue’s own value ends in a space) would get the description cut again for a change it did not make; writing the value passed when only whitespace differs, because that rewrites the stored value just the same; inserting a new section before the first stored one ranked after it, because a section the canonical order does not know ranks last, and a new known section then jumped above it to the top. |
night shift, draft |
| 2026-10-09 | The SessionStart hook keeps its context within Claude Code’s 10,000-character cap on hook output, and an INDEX.md over the budget goes in as a short form ranked by recency (#208; refines “inject INDEX.md” in the 2026-07-22 row). The host replaces a longer hook output with a file path and a 2,000-character preview that the model is not asked to open (measured on Claude Code 2.1.237: 10,000 characters go in, 10,001 do not, and lengths are UTF-16 code units), so the old index[:20000] reached the model as 7 of 216 entries on a 56,301-character INDEX. Neither the linear budget in doctor nor the rollup policy keeps an INDEX under the cap: the budget grows with the shelf, and the rollup is a manual pass. The short form keeps the newest entries by the date in their id, lists them under their kinds newest first, gives titles to the newest while titles take a quarter of the budget (an id is a slug of its topic, so an id-only line still carries its words), and ends with the count and date range of what it leaves out and a reminder that a missing entry is not the shelf having nothing. On that INDEX: 155 of 216 entries, back to 2026-08-20, 17 with titles. Rejected: a prefix cut at the cap (docshelf sorts each kind by title, so a prefix is an arbitrary subset: 38 of the 216 entries, and none of the three episodes the incident needed); titles on every line (about 66 entries fit); several hook entries carrying 10,000-character slices (that multiplies the cap instead of keeping it, and the cost of every session start grows with the shelf, against the injection budget of the 2026-07-13 survey row); ranking by topic (that needs the prompt, so it belongs to a UserPromptSubmit hook, #208 item 2). |
owner request, draft |