A local memory server for coding agents: hybrid search over a project's notes, git history and code, served over MCP with path:line citations

Problem

A coding agent starts every session with no memory. It re-reads files, misses the decision made last week, and asks the same questions again. The usual fixes split two ways. Hosted memory APIs keep chat facts in someone else's cloud and know nothing about the code. A single CLAUDE.md or AGENTS.md works until it is longer than a few pages.

My own agentic workflow keeps its memory as plain markdown in git: briefs, decisions, review notes, work logs. That part works. Finding the right note did not. Agents grepped, read whole files, and still missed the note that answered the question in different words. recall is the search layer I built for that workspace, pulled out and made config-driven so it works on any project.

What it does

An agent asks "have we dealt with this before?" or "where is the code that does X?" and gets ranked path:line citations with the reason each one matched. It opens the one it wants with get and decides for itself. Six read-only tools:

• search: notes, decisions and commits by meaning, or code with scope: "code".
• get: one bounded section of a cited file or commit.
• why: for a line range or a symbol, the commits that introduced it (git blame), the issues and PRs they name, the notes linked to them, with notes that may have drifted from the code flagged.
• related, timeline, items: links both ways, the dated history of an issue ID, and work items filtered by frontmatter.

recall init looks at a project (git root, markdown folders, languages, issue IDs in commit messages), writes a commented config, builds the first index and prints the MCP config for Claude Code, Codex CLI, Cursor and VS Code.

Architecture

Indexing. Markdown splits at ## headings, with long sections cut into overlapping windows and each chunk embedded with its title and heading in front. Code splits at declarations found by per-language patterns, large classes at their members. A commit is one document: subject, body and changed paths. Every document carries a content hash, so only changed documents are re-embedded.

Ranking. The query runs against an FTS5 keyword index and every vector. The two rankings merge by reciprocal rank fusion, an exact issue ID outranks any similarity, and no kind may take more than half the results, so thousands of commits cannot crowd out one decision record.

Serving. The server answers the client at once and loads the models in the background; until they are warm, search ranks by keywords and says so. A file watcher hands changes to a child recall index process, so a query never waits for indexing. Several sessions share one index behind a lock file.

Key decisions

1. Memory stays plain files. recall only reads. Notes live in git where a person can diff, review and edit them; the index is a cache you can delete. That rules out the agent writing "memories" into a store nobody reviews, which is the point.

2. Local and private by default. Embeddings run on the CPU through ONNX, and after a one-time model download recall makes no network calls. Over stdio nothing listens on a port; over HTTP it binds to 127.0.0.1 and rejects requests whose Host or Origin is not localhost, which stops a web page reading the index through DNS rebinding. Files that look like they hold credentials are never indexed, and folders marked private keep vectors only.

3. Citations, not summaries. Every result is a file and line range with its match reason. No model writes an answer in between, so the agent cannot be handed a confident summary of a note that says something else.

4. Two embedding models. Prose and code embed differently, so each index has its own model. The cost is about 300 MB of downloads and roughly 2 GB of RAM while embedding.

5. Defaults chosen on held-out data. The ranking settings were tuned on half of a benchmark and scored once on the other half. A change that only helped the questions it was tuned on would not survive that.

Numbers

120 questions over three pinned public repositories recall was never tuned on (ripgrep, httpx and a content-publishing app), written to avoid the answer's own words. Recall at 5 is the share of questions with a relevant document in the top five:

On code questions alone, ripgrep finds the answer in the top five 14% of the time and recall 83%. On the 63 held-out questions the current defaults raised R@5 from 0.651 to 0.706, and no repository and question type got worse. Hybrid queries take about 30 ms at p50 and 71 ms at p95, against 55 and 215 ms for ripgrep. Indexing is the slow part: a mid-size repository takes two to six minutes the first time.

What broke

• Fusion hid good answers. At equal weight, the keyword ranking's top chunks were often weak matches on common words, and they pushed correct similarity hits out of the top five. On 17 of 41 hybrid misses one ranking alone had the answer. Counting the vector ranking twice fixed most of it.
• Tests outranked the code they test. Tests describe behaviour in plain words, so a question about behaviour found the test first. Code under test paths now scores half, unless the query asks about tests.
• My own eval was saturated. recall's eval over its own repository scored near perfect, so it could not tell whether a change helped. That is why the public benchmark exists.
• Promising changes that lost. Cleaning housekeeping lines out of commit bodies, adding changed declaration names to commits, finer markdown splits and a Ruby declaration splitter all looked right and all scored worse or no better on the tuning half. They were dropped and recorded.

Results

recall has served the memory of my own workspace every day since before it was extracted: every agent session there searches it before reading files. The standalone version adds recall init, stdio transport for every major agent client, a Claude Code plugin, the why tool and the benchmark.

What I'd change

• "Why" questions are still the weak spot. Hybrid R@5 is 0.50 on them, worst where commits are squash merges whose body is a list of work-in-progress subjects. Pulling PR descriptions in alongside commits is the next thing to try.
• Code chunking by regex. Per-language line patterns are cheap and good enough for six languages; a real parser would split more accurately and cover more.
• First index time. Minutes on CPU is fine once, but it is the first thing a new user meets.