oc-prefix-cache — opencode KV prefix cache

Keeps opencode's stable harness prefix (its 18 tool schemas + the stable head of the system prompt) resident in a llama-server KV-cache slot across sessions, and can persist that prefix to disk so a restarted server resumes in 0.25 s instead of 27 s on a slow GPU.

• Fail-open: any error is logged and the normal request proceeds.
• It only talks to llama-server's HTTP API; all server details live in warmup.ts.

Measured on opencode 1.18.34 + Qwen3.8-27B (ROCm): a real agent turn is 8,619 rendered tokens; 7,666 of them (89%) are restored from the cached prefix, warm reuse costs 0.8 s and a disk restore 0.25 s.

How it works

The plugin's config hook wraps the fetch of the provider whose baseURL matches OC_PREFIX_CACHE_BASE_URL, so it sees the exact bytes opencode sends (including the tools array) and can hold the request while the slot is prepared.

Per agent request:

1. Skip anything that is not the agent turn (no tools, no system message) — e.g. opencode's title generator.
2. Cut the system message at the first volatile marker: <env>, Instructions from:, <mcp_instructions> or the skills block. The <mcp_instructions> block mutates every turn, so it must never be part of the fingerprint.
3. Fingerprint: model id, GGUF identity, server build id, opencode version, extension version, tools[] verbatim, stable head.
4. Already warm in this process? Check the slot is still alive (a server restart zeroes its prompt counter), then forward.
5. Miss → with persistence on, restore oc-prefix-<fp16>.bin and verify it in two stages (≥ 90 % of the identical prompt from cache, then a divergent probe that must resume from a context checkpoint). Invalid → erase and rebuild. No checkpoint → cold warm-up (n_predict: 0, prefix + empty user turn), then save.
6. Forward the request untouched and report the observed reuse.

Requirements

• opencode ≥ 1.0 with a provider whose options.baseURL points at your local llama-server (the default @ai-sdk/openai-compatible path).
• One slot for the shared prefix; with -np 1 leave OC_PREFIX_CACHE_SLOT at 0.
• Server flags for hybrid/recurrent models (Qwen3.5/3.8 …): cross-prompt reuse resumes from context checkpoints at ubatch boundaries, so --ctx-checkpoints N must be large enough for the prefix chain and --checkpoint-min-step must be smaller than the prefix. The stock default (8192) is too coarse for a ~7.7k prefix — use e.g. 4096, or 0 for maximum reuse at the cost of more checkpoint RAM. -ub sets the reuse granularity. See Recommended llama-server flags below for a full baseline and the reasoning.
• For OC_PREFIX_CACHE_PERSIST=1 on such models: a llama-server built with the checkpoint sidecar patch (below). Without it a restored slot re-prefills the whole prompt.

Recommended llama-server flags

A working baseline for a Qwen3.x hybrid model served to a code harness (this is the shape of the launcher in ~/run/scripts/one_shot.sh_ctx_long):

--slot-save-path ~/llama/slot_caches/      # where save/restore writes .bin + .ckpt
--ctx-checkpoints 32                       # checkpoint chain depth per slot
--checkpoint-min-step 4096                 # spacing between checkpoints
--cache-ram 6000                           # parked contexts, MiB
-np 1                                      # one context = the whole ctx pool
-ub 128                                    # reuse granularity
-b 1024
--jinja --chat-template-file chat_template_3.8.jinja

How --checkpoint-min-step and -ub decide how much is cacheable

A resume does not need a perfect match; it needs a checkpoint at or before the point where the new request diverges from the cached one, plus the tokens from there to the divergence point. So:

reusable ≈ (position of the last checkpoint ≤ divergence point)
           rounded down to a ubatch boundary

• --checkpoint-min-step sets how far apart checkpoints may be. A resume can only land on the last checkpoint before the divergence point, so up to min_step tokens of the prefix are re-read on every start. Smaller ⇒ finer landings, less re-reading, more checkpoint VRAM. The stock 8192 is coarser than a typical harness prefix (this one is 7,666 tokens): there would be no checkpoint inside the prefix at all, and a restored slot would re-read all of it. 4096 puts one in the middle and lets later checkpoints track the conversation; 0 (no minimum) maximises reuse at the cost of VRAM. An intermediate value is the usual choice.
• -ub (ubatch size) is the finishing granularity: checkpoints land on ubatch boundaries, so the resume position is the last boundary at or before the divergence point. Smaller -ub = finer landing = fewer wasted tokens, slightly more overhead. With -ub 128 the real request resumed at token 7,534 of a 7,666-token prefix — 132 tokens re-read; with -ub 512 (the default) it would land on a coarser boundary.

Checkpoint memory is not free: each one holds the KV/recurrent state up to its own position. On the 27B IQ4_XS at q8_0/q5_0 the server reported size = 158.2 MiB for a checkpoint at 7,662 tokens, i.e. ≈21 KiB per token, so the chain's cost scales with the positions it spans, not with N. Budget for the largest prefixes you expect, and let the server's fit logic (--fit-target) keep it inside VRAM.

Pin one chat template across models

Serve every Qwen 3.x model through the same external jinja template (--jinja --chat-template-file …) rather than letting each GGUF select its built-in one. It is sound because the cache is a property of the rendered token stream, and the template is what produces it:

• the template decides where the tools block lands relative to the system content, the special tokens, and the tool-call syntax. This one renders tools before the system message, which is why the 18 tool schemas are 5,806 of the 8,619 tokens of a real request;
• if a model upgrade or a GGUF swap silently changed the renderer, every cached prefix would be built against a layout the server no longer produces — and nothing in the fingerprint would notice (the fingerprint pins the model id, the GGUF's size+mtime, the tools, and the plugin and opencode versions, but not the template or the server flags).

So: one template for the family, and bump OC_PREFIX_CACHE_SERVER_ID whenever you change the template, the flags, or the server build — that is the single knob that invalidates every checkpoint exactly once.

--cache-ram with -np 1: many sessions, one at a time

With -np 1 there is a single context, but the RAM prompt cache (needs the best-match/park part of the patch below) lets the server keep other prompts around: when a request would evict the resident context, the server parks it in RAM (up to --cache-ram MiB), and when a later request matches a parked prompt better than what is resident, it swaps that one back in. So a handful of non-concurrent sessions — one per project, say — can each keep a warm context without re-reading, as long as they are not in flight at the same moment. Watch for parked displaced context in prompt cache in the server log; the plugin's log will then show a fast warm-up done: … from_cache=… even though the slot itself changed contents.

Budget it against your prefixes: ~21 KiB/token means a 7.7k-token context is roughly 160 MiB of KV plus the recurrent state, so 6000 MiB holds comfortably more than a dozen; 0 disables the cache, -1 removes the limit (not advisable — it is host RAM, and it grows silently).

-np is a VRAM-versus-concurrency dial

The KV/recurrent pool is sized by the total context the server allocates; -np only decides how it is divided:

• -np 1: the whole pool is one context. Maximum context length per agent, minimal per-slot overhead, but only one agent can generate at a time — a second concurrent request queues behind the first.
• -np 2 or more: the same pool is split, so each agent gets roughly total / np context (this server reports n_ctx ≈ 95,744 on -np 1; halve it per slot on -np 2) in exchange for real parallelism — two agents, or an agent and a subagent, generating at once. Asking for the same per-slot context as -np 1 means asking for np × the VRAM, which is usually what actually breaks the fit.

For this plugin, more slots is strictly better if two harnesses share the server: point each at its own slot with OC_PREFIX_CACHE_SLOT (0 for pi, 1 for opencode) and neither evicts the other.

Install

1. Download and unpack (the tarball is the oc-prefix-cache/ directory exactly as it lives in the source repo — copy it as-is):

   wget https://store.piffa.net/lm/ocache/oc-prefix-cache.tgz
   tar xzf oc-prefix-cache.tgz
   mkdir -p ~/.config/opencode/plugin
   cp -r oc-prefix-cache ~/.config/opencode/
   

   That gives ~/.config/opencode/oc-prefix-cache/ with the five modules (index.ts, prefix.ts, fingerprint.ts, warmup.ts, cache-store.ts), this README.md, and the llama.cpp patch used in the build section below.

2. Write the shim. opencode auto-discovers any *.ts in plugin/, so no opencode.json edit is needed:

   printf 'export { default } from "../oc-prefix-cache/index.ts"\n' \
     > ~/.config/opencode/plugin/oc-prefix-cache.ts
   

3. Export the variables below in your shell profile — they are read once, at opencode start.

4. Restart opencode. opencode --pure disables plugins, so the cache is inert in that mode.

Configuration

All configuration is environment variables. Nothing in opencode.json has to change.

# required: enables the plugin and acts as the provider gate
export OC_PREFIX_CACHE_BASE_URL=http://127.0.0.1:8080/v1
# recommended
export OC_PREFIX_CACHE_PERSIST=1
export OC_PREFIX_CACHE_SLOT_DIR=/home/eaman/llama/slot_caches
# bump when you deploy a new llama-server build
export OC_PREFIX_CACHE_SERVER_ID=9adc7f42-slckv2-ckpt4096
# lets the fingerprint notice a replaced GGUF
export OC_PREFIX_CACHE_MODEL_PATH=/path/to/model.gguf

Numeric variables fall back to their defaults when unset or invalid.

Expected logs

[prefix-cache] plugin loaded ext=1.0.0 opencode=1.18.34 base=http://127.0.0.1:8080/v1 slot=0 persist=true
[prefix-cache] interception armed on provider "local" (baseURL http://127.0.0.1:8080/v1)
[prefix-cache] request: provider=local model=qwen boundary=env@8676 stable-chars=8676 tools=18 fingerprint=7e3272f8…
[prefix-cache] cache=MISS → warming slot=0 (cold)
[prefix-cache] warm-up done: HTTP 200 total=7666 new=7666 from_cache=0 (26.7s, 287 tok/s)
[prefix-cache] SAVED oc-prefix-7e3272f88e0dfe4f.bin: n_saved=7666 (151.1 ms)
[prefix-cache] observed real request: prompt_n=8493 cache_n=7534 (89% from cache)

After a server restart (the disk-restore path):

[prefix-cache] RESTORE oc-prefix-7e3272f88e0dfe4f.bin: n_restored=7666 (254.4 ms)
[prefix-cache] RESTORE verified: identical 7662/7666, divergent probe 7656/7672

Server-side confirmation to look for in the llama-server log:

slot 0 | restored context checkpoint (pos_min = 7661, n_past = 7666, size = 158.219 MiB)

Disk persistence — the llama.cpp patch

/slots save|restore alone is not enough for hybrid models: the state file carries the tokens and the KV/recurrent state, but the server-side slot.prompt.checkpoints chain is not serialized, so a restored slot re-prefills the whole prompt. One self-contained patch fixes that:

kv_prefix_checkpoint_sidecar_9adc7f42.patch — applies to stock llama.cpp commit 9adc7f42 (upstream latest) and builds on its own. A copy sits next to this README. It does two things:

• Sidecar (needed for OC_PREFIX_CACHE_PERSIST=1): persists the checkpoint chain next to the slot file as <file>.ckpt (format SLCK v2).
• RAM prompt cache (optional, only matters when several sessions share one slot): consults the cache when a cached prompt beats the resident slot, and parks the displaced context on a slot restore. Needs a server built with --cache-ram and --ctx-checkpoints.

With the sidecar patch, action=save writes <file>.ckpt next to the slot file and action=restore rebuilds the checkpoint chain from it, validating every entry (including a token-list hash that rejects stale sidecars). The sidecar is transparent to this plugin. Sidecars written by a v1 build are rejected on restore and fail open to a full re-prefill.

How to build

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 9adc7f42
wget https://store.piffa.net/lm/picache/kv_prefix_checkpoint_sidecar_9adc7f42.patch
patch -p1 < kv_prefix_checkpoint_sidecar_9adc7f42.patch
cmake -B build
cmake --build build --config Release -j$(nproc)

Notes:

• use patch -p1 (it tolerates the # comment header at the top of the file; older git apply versions reject it).
• build with the backend you actually run (ROCm: -DGGML_HIP=ON, Vulkan: -DGGML_VULKAN=ON) — the patch touches server code only and is backend-agnostic.
• deploy both llama-server and libllama-server-impl.so to your server's bin directory: the code lives in the shared object, so replacing only the executable is a silent no-op. Replace with an atomic mv while other servers run from that directory (a plain write gives ETXTBSY).
• then set OC_PREFIX_CACHE_SERVER_ID to something new so the extension invalidates checkpoints written by the previous build.

Uninstall

rm -rf ~/.config/opencode/oc-prefix-cache
rm -f    ~/.config/opencode/plugin/oc-prefix-cache.ts
rm -f    ~/.cache/opencode/llama-prefix/index.json
rm -f    "$OC_PREFIX_CACHE_SLOT_DIR"/oc-prefix-*.bin "$OC_PREFIX_CACHE_SLOT_DIR"/oc-prefix-*.bin.ckpt
# and unset the OC_PREFIX_CACHE_* variables

Notes & limitations

• The boundary markers are opencode's own text. If it renames <env> / <mcp_instructions> / Instructions from:, extraction fails open (logged) and the cache is simply unused — it never serves a wrong prefix.
• The tools array is part of the fingerprint: adding, removing or editing a tool (including MCP tool schemas) forces one cold rebuild.
• The tools are rendered before the system content by the Qwen3.8 chat template, and they are the bulk of the prefix (5,806 of 8,619 tokens). The warm request therefore carries the tools verbatim, in the same order — that is why the plugin intercepts the request instead of re-serializing schemas from hooks.
• Slot counters reflect the last prompt, not a running total, so the observed real request line is only meaningful for the request that follows it.
• No locking between concurrent opencode processes; within one process the warm/restore work is serialized. If another harness (e.g. pi) warms the same slot, the two evict each other — give each its own slot (-np 2) if that matters.
• Per cached prefix the server writes ~377 MB .bin + ~498 MB .ckpt; set OC_PREFIX_CACHE_MAX_GB if that matters.
• A corrupt index.json disables cleanup for that run (logged); checkpoints are never deleted against an index that cannot be trusted. A missing index is fine.
• Changing the extension version changes the fingerprint → one cold rebuild after an upgrade.