pi-prefix-cache

Reuses Pi's stable harness prefix (system-prompt sections, tool schemas, chat_template_kwargs) in a llama-server KV-cache slot across sessions, and can persist that prefix to disk so a restarted server (or a new Pi install) resumes in seconds instead of re-ingesting the 5–20k-token prefix.

• Pi is never modified; the provider payload is never touched.
• Fail-open: any error is logged and the normal request proceeds.
• The extension only talks to llama-server's HTTP API; all server details live in warmup.ts.

See README.html for full documentation.

	Name	Last modified	Size	Description
	Parent Directory		-
	README.md	2026-10-09 14:03	15K
	old-sidecar-patch-version/	2026-10-09 14:02	-
	pi-prefix-cache.tgz	2026-10-09 14:03	19K
	pi-prefix-cache/	2026-10-09 14:03	-

pi-prefix-cache — Pi KV prefix checkpoint extension

How it works

Per provider request:

1. Extract the stable prefix from the developer message (everything before the earliest variable-section boundary). Three markers are recognised, whichever comes first:
   • \n<project_context> — present only when the project has an AGENTS.md/CLAUDE.md context file.
   • \n<cwd> — pi >= 0.87 sectioned prompt (the cwd section is rendered as <cwd>\n<path>\n</cwd>). Sections are joined with a blank line, so the leading \n always matches.
   • \nCurrent working directory: — pi < 0.87 plain-text cwd line, kept for backward compatibility.
   The cwd boundary matters: without it, a project that has no AGENTS.md has no variable marker at all on pi >= 0.87 and the extension silently no-ops.
2. Compute a fingerprint (SHA-256 over model id, model file identity (size+mtime), server id, pi version, extension version, tools[], chat_template_kwargs, stable prefix text).
3. If this process already warmed that fingerprint, check the slot is still alive (GET /slots, nonzero prompt counter). A server restart clears the slot — the extension detects it and falls back to restore/re-warm instead of a false in-process hit.
4. Otherwise (miss):
   • with persistence on: restore pi-prefix-<fp16>.bin into the slot, then verify in two stages:
     1. identical-prompt gate — re-send the prefix; a valid restore must serve ≥ 90% of it from cache. Invalid → erase + cold rebuild.
     2. divergent probe — same prefix with a non-empty user message, so the request must resume from a context checkpoint before the divergence point (a plain KV token match proves less on hybrid models). cache_n > 0 verifies the checkpoint chain. cache_n = 0 is logged loudly (usually a server-flags issue, e.g. prefix shorter than --checkpoint-min-step) and the restore is kept — a rebuild would produce the same chain.
   • no file / restore failed → cold warm-up (n_predict: 0 request carrying only the stable prefix + an empty user message).
   • with persistence on: save the freshly built prefix to disk.
5. The normal Pi request proceeds and reuses the slot's KV cache.

After each response, the extension polls the slot's last-prompt counters and logs the observed reuse of the normal request.

Requirements

• Pi (OpenAI-compatible provider path) pointing at a local llama-server.
• One slot for the shared prefix. With -np 1 there is only one — leave PI_PREFIX_CACHE_SLOT at 0.
• For hybrid/recurrent models (Qwen3.5/3.8 etc.), cross-prompt reuse resumes only from context checkpoints created at ubatch boundaries. --ctx-checkpoints N should be large enough to hold the prefix chain. --checkpoint-min-step is the minimum spacing between consecutive checkpoints: a larger value stores fewer checkpoints (less RAM) but a resume can only land on the last checkpoint before the divergence point, so up to min_step prefix tokens are re-prefilled per session start. The default 8192 is too coarse for typical 5–20k prefixes (no intermediate checkpoint → full re-prefill); 0 gives maximum reuse, an intermediate value (e.g. 4096) trades up to ~4k tokens of re-prefill for less checkpoint RAM. -ub sets the reuse granularity (last ubatch boundary ≤ divergence point) — a smaller -ub is finer at higher overhead. These are per-deployment tradeoffs (e.g. --ctx-checkpoints 32 -ub 384 on the 27B main server, --ctx-checkpoints 8 -ub 128 on the 9B test server).
• For PI_PREFIX_CACHE_PERSIST=1 on such models: a llama-server built from mainline ≥ c811cb8f (includes upstream #26004, which serializes the checkpoint chain into the slot save file). Older builds restore the KV state but lose the checkpoint chain → full re-prefill (see "Disk persistence — server requirement" below).

Install

Install from PiFFA Store

Download and unpack the extension, then copy it into Pi's extensions directory:

wget https://store.piffa.net/lm/picache/pi-prefix-cache.tgz
tar xzf pi-prefix-cache.tgz
mkdir -p ~/.pi/agent/extensions
cp -r pi-prefix-cache ~/.pi/agent/extensions/

Manual install

1. Copy the directory so Pi auto-loads it:

   cp -r pi-prefix-cache ~/.pi/agent/extensions/

   (or per-invocation: pi -e /path/to/pi-prefix-cache/index.ts)
2. Set the env vars in your shell profile (read once at Pi start). Minimal production set:

   export PI_PREFIX_CACHE_BASE_URL=http://localhost:8080/v1
   export PI_PREFIX_CACHE_PERSIST=1
   export PI_PREFIX_CACHE_SLOT_DIR=/home/eaman/llama/slot_caches

Environment variables

All numeric env vars fall back to their defaults when unset or invalid.

Provider gate (v1.2.0)

When PI_PREFIX_CACHE_BASE_URL is set, the extension acts only for models whose provider baseUrl in models.json (agent dir: $PI_CODING_AGENT_DIR or ~/.pi/agent) equals it — it can only warm its own server. Other models (remote providers, other local servers) are skipped with a log line: no warm-up, no restore/save, no slot polling. If models.json is missing or unreadable, all requests are skipped (safe direction: never warm a server the request cannot be attributed to).

Expected logs

Where logs go (v1.2.2+): all [prefix-cache] lines are written to the log file (PI_PREFIX_CACHE_LOG), not stdout — pi ≥ 1.0's fullscreen TUI renders raw console writes on top of the input area. In the TUI you get exactly one notification per session (the first request's outcome), rendered via ctx.ui.notify():

⚡ prefix-cache: WARM — restored 2335 tok from disk (1453 ms)   # checkpoint restore verified
⚡ prefix-cache: WARM — slot already hot                        # slot warm in this process
prefix-cache: COLD — warmed 2335 tok, checkpoint saved          # cold build (warning style)
prefix-cache: disabled this session (fail-open): <error>        # on failure (error style)

All lines below are prefixed [prefix-cache] (log file). First run for a fingerprint:

[prefix-cache] fingerprint=16ba1104…
[prefix-cache] no checkpoint (restore HTTP 400: …) → cold build
[prefix-cache] cache=MISS → warming slot=0 (cold)
[prefix-cache] warm-up done: HTTP 200 prompt_n=1228 cache_n=0
[prefix-cache] SAVED pi-prefix-16ba1104….bin: n_saved=…
[prefix-cache] normal request proceeding (source=warm-up)
[prefix-cache] observed normal request: prompt_n=1360 cache_n=1096 (reuse 81%)

Later sessions (same fingerprint):

[prefix-cache] RESTORE pi-prefix-16ba1104….bin: n_restored=…
[prefix-cache] RESTORE verified: identical 1224/1228, divergent probe 1096/1240
[prefix-cache] normal request proceeding (source=restore)

Within one Pi process, subsequent requests log cache=HIT (slot N already warmed for this fingerprint).

Verify server-side reuse with LLAMA_SERVER_SLOTS_DEBUG=1: restored context checkpoint (pos_min = …, … n_past = …).

Files

Disk persistence — server requirement

/slots save|restore on mainline ≥ c811cb8f is sufficient: upstream #26004 ("server: preserve context checkpoints across slot save/restore") appends the server-side slot.prompt.checkpoints chain to the slot save file itself (the SCKP appendix) and rebuilds it on restore, with size-bounded reads, draft-context mismatch handling, and a fallback to full re-prefill when a restored checkpoint fails to load. No llama.cpp patch is required — the extension is fully patch-free on such builds.

Migration notes

• Checkpoints saved by older sidecar-patched builds (.bin + .ckpt files) restore with a working KV state but no checkpoint chain on a #26004 server: the identical-prompt gate passes, the divergent probe logs cache_n = 0 loudly, and the next normal request re-prefills the conversation tail. Bump PI_PREFIX_CACHE_SERVER_ID after deploying a new server build to force one clean cold rebuild, which re-saves the checkpoint in the native format.
• Stale .ckpt sidecar files are ignored by #26004 servers; the extension's cleanup still removes them alongside their .bin.

Legacy patches (only relevant for servers that cannot upgrade to ≥ c811cb8f) are no longer shipped in this repo — they were removed from the tree and are recoverable from git history (commit 180ec5a: kv_prefix_checkpoint_sidecar_9adc7f42.patch, SLCK v2 sidecar, base 9adc7f42; prompt_cache_best_match_9adc7f42.patch). An older …_ddddf03d.patch (base ddddf03d) has an external copy at /home/eaman/llama/bug/patches/. Do not apply any of these to mainline.

Verified (2026-10-08, 9B MTP test server, port 8081)

Full cold-save/restore-verify cycle, non-interactive Pi sessions:

Server: --ctx-checkpoints 8 -ub 128 --checkpoint-min-step 1024 --slot-save-path ... + tools-first --chat-template-file. The divergent probe cache_n > 0 on the unpatched binary is the acceptance criterion: the checkpoint chain survives the disk round-trip via the native SCKP appendix alone.

Notes & limitations

• Boundary markers are Pi-core text; if Pi renames <project_context> or the cwd line, extraction fails open (logged) and the cache is simply unused.
• The rendered system-open marker plus newline (19 bytes before the payload content) is template/model-specific; it is implicitly pinned by the model id in the fingerprint.
• Reuse granularity = last ubatch boundary ≤ token-match point (1096 of 1221 with -ub 128). Tune -ub for tighter reuse vs prompt-processing speed; ~51.5 MiB VRAM per checkpoint.
• Slot counters reflect the last prompt, not a cumulative total; the post-request diagnostics rely on that. With two Pi instances on one server the line may attribute the other process's prompt.
• No locking between concurrent Pi instances (single active session is the assumed setup, -np 1).
• A corrupt index.json disables cleanup for that run (logged); checkpoint files are never deleted against an index that cannot be trusted. A missing index is fine (treated as empty).
• The model fingerprint includes the GGUF file size + mtime; replacing the file at the same path forces a clean rebuild.
• piVersion() resolves via the pi CLI entrypoint realpath; falls back to unknown (still deterministic within a machine).
• Changing the extension version changes the fingerprint → one cold rebuild after an upgrade.