# Fresh Technical Investigation for the Eaman llama.cpp Patch

## Executive summary

The Eaman patch has moved substantially beyond the 2026-08-07 research baseline. The largest items from that earlier search are no longer open opportunities: **recurrent rollback depth reduction is now implemented as MTP Compact Rollback; recurrent checkpoint allocation is measured exactly and preallocated; adaptive MTP draft sizing is implemented; upstream’s newer joint target/draft fitter superseded the old Eaman MTP-estimation loop; fixed-layout per-device fitting remains; target pipeline policy and non-pipeline MTP remain; and the HIP quantized-KV VEC override is now an opt-in diagnostic/capacity control rather than a universal default.** fileciteturn0file0 The older `improvemets_list.md` does not reflect several of these additions and should be rewritten rather than incrementally edited. fileciteturn0file1

The most important genuinely new finding is a **long-prompt MTP ordering bug** represented by upstream PR **#26827**, commit **`2295535`**, and newly corroborated on September 2 by issue **#28252**. Multiple internally generated MTP ubatches can be queued asynchronously while sharing mutable MTP KV state. The proposed fix synchronizes only multi-ubatch MTP decode, leaving target decode and single-ubatch MTP asynchronous. The PR is still open and therefore is not in current llama.cpp mainline; its most recent code activity I could verify is August 10. This deserves the first Eaman experiment because the trigger—native MTP, very long prompt, multi-GPU operation—is unusually close to the Eaman workload, even though the reported hard-lock reproducer used CUDA tensor splitting rather than HIP layer splitting. citeturn29search0turn29search1turn24search14

The most attractive **VRAM** opportunity is upstream issue **#27282**: keep target and MTP backend schedulers independent but let them reuse one backing graph allocator/compute arena when their execution is guaranteed not to overlap. A CUDA Qwen-family proof of concept turned a failed 785–1040 MiB second MTP compute allocation into a working 196,608-token configuration. This is unmerged experimental evidence, not proof that the design is safe on HIP. On the Eaman Qwen3.6 dense configuration the likely reclaim is much smaller than that CUDA example—the existing measurements put the MTP ROCm compute allocation around the low hundreds of MiB—but it attacks one of the last genuinely duplicated MTP allocations rather than shrinking another data structure. citeturn26search0 The strongest design dependency is that this experiment should follow, or explicitly incorporate, correct MTP execution ordering: an allocator must not be reused while the previous context still has asynchronous GPU work outstanding. citeturn29search0

The third implementation experiment I recommend is not another context compressor. It is a **runtime-transient-aware fit guard**. Upstream issue **#26432** shows that a nominally successful HIP/MTP load can silently spill into GTT on first use and lose more than 60% throughput. citeturn24search2 More importantly, the authoritative Eaman history independently found the same class of gap after the newer joint fitter: a MoE configuration fitted to the 262,144-token ceiling with `ubatch=512` and then OOMed inside Flash Attention on the first request, while `ubatch=256` survived. fileciteturn0file0 A bounded post-fit execution canary or transient-workspace reservation would make “fitted context” mean **usable context under the configured HIP kernel and microbatch policy**, not merely context whose persistent contexts can be constructed.

A fourth candidate becomes important if the project moves beyond its current one-slot production mode: issue **#27572**, linked to open PR **#27705**, identifies an independent HIP-native MTP race in the asynchronous device-to-host copy of `t_h_nextn`. With several slots and multiple ubatches, the scheduler can reuse the extra buffer before the asynchronous copy completes; the reported result is NaN MTP input and acceptance collapsing exactly to zero. A full scheduler synchronize fixed correctness but caused roughly a 40% long-prompt generation penalty, so the upstream report explicitly favors event ordering or rotating/staged buffers instead. citeturn27search0 Eaman’s existing two-slot Compact Rollback tests do **not** supersede this finding: those tests establish per-slot checkpoint isolation, whereas #27572 concerns lifetime of the target→MTP hidden-state handoff under long, multi-ubatch continuous batching. fileciteturn0file0

I would **not** make lower-precision rollback checkpoints, ROCm VMM, TOP_K, TurboQuant, HIP AllReduce, more fitter refinement passes, or another generic adaptive-draft controller top recommendations. Compact Rollback has already captured most of the low-risk rollback-state saving; the TOP_K gap produced no measurable advantage when Eaman disabled draft backend sampling; current TurboQuant ROCm code is explicitly restricted to head dimension 128 while the fresh Qwen3.6-27B HIP report identifies head dimension 256; HIP AllReduce targets a different multi-GPU communication pattern from the fixed layer split; and VMM/pool work still has significant ROCm portability risk. fileciteturn0file0 citeturn22search0turn22search4turn22search7

One source limitation matters. `summary_eaman.md`, `improvemets_list.md`, and `web_search.md` were available to this investigation, but `status` and `AGENTS.md` were not exposed in the attachment set I could inspect. I therefore treated `summary_eaman.md` as authoritative and the August 7 document strictly as a historical baseline; any conflict with the unavailable `status` or `AGENTS.md` should be resolved in favor of those files as requested. fileciteturn0file0 fileciteturn0file2

Finally, upstream has already advanced beyond the Eaman patch’s August 31 base: a llama.cpp release attestation created on **September 3, 2026** identifies current master commit **`9cffdcc801582616250520966699cb5b25d28243`**, whereas the authoritative Eaman refresh is based on **`662a0b0121a53c23b825a71e64ab6eff59b7f4d8`**. citeturn29search3 fileciteturn0file0 I was able to verify the states of the important individual PRs/issues below, but not exhaustively diff every commit between those two tips; containment judgments therefore rely on explicit PR status plus the Eaman August 31 refresh record.

## Material changes since the 2026-08-07 search

The August 7 baseline correctly identified recurrent speculative state, HIP temporary allocation lifetime, MTP fitting, TOP_K, alternative KV formats, MTP placement, and long-session correctness as areas worth watching. fileciteturn0file2 Several of those have since crossed from “idea” to “implemented or superseded,” while a different class of asynchronous-state hazards has emerged.

| Change since the baseline | State as of 2026-09-03 | Consequence for the Eaman patch |
|---|---|---|
| **MTP Compact Rollback** | Implemented in Eaman; refreshed to upstream `662a0b01` | The August 7 “configurable rollback depth” proposal is complete enough that it should no longer be a top recommendation. Depth 1 measured 149,504 versus 136,704 tokens on dense Q6_K_L after the Aug23 rebase, +9.36%. fileciteturn0file0 |
| **Exact device-checkpoint fit reservation and startup preallocation** | Implemented in Eaman | Supersedes the earlier approximation based on recurrent-plane memory deltas and directly addresses the delayed checkpoint-allocation OOM found during testing. fileciteturn0file0 |
| **Adaptive MTP draft sizing** | Implemented in Eaman, opt-in | Generic acceptance-driven dynamic draft length is no longer a new candidate. Eaman testing showed workload-dependent gains and losses, which is why default-off remains appropriate. fileciteturn0file0 |
| **Upstream joint target/draft fitting** | Already incorporated by Eaman’s Aug23 refresh | Supersedes the original Eaman fixed-pass MTP estimator/refinement loop and the temporary MoE scratch-placement workaround. The Eaman residual fit code now concerns fixed-layout per-device behavior and matching MTP pipeline policy. fileciteturn0file0 |
| **HIP VEC policy** | Implemented; current cumulative patch defaults forcing **off** | The old “force VEC” proposal is no longer new. It is now a selectable context-capacity versus prefill-throughput policy. fileciteturn0file0 |
| **Fine target-tensor balancing** | Operationally implemented through the launcher/manual override, not generalized into automatic fixed-layout fitting | Moving the layer-39 FFN triplet to ROCm1 gave +9,728 tokens in the matched Q8 target-KV test while retaining three target graph splits. An automatic search is conceivable, but the raw capacity win is already realized in the selected production profile. fileciteturn0file0 |
| **MTP multi-ubatch shared-state ordering bug** | PR #26827 open; commit `2295535`; issue #28252 open | New, high relevance. It was not in the August 7 baseline and is not in mainline. citeturn29search0turn24search14 |
| **Independent MTP compute arena identified as shareable in a POC** | Issue #27282 open, no merged upstream solution | New VRAM opportunity after the August 7 search. citeturn26search0 |
| **Multi-slot `t_h_nextn` asynchronous-copy race** | Issue #27572 open; linked PR #27705 open | New correctness finding specifically involving HIP, MTP, multi-ubatch operation, and `-np > 1`. citeturn27search0 |
| **Fresh quantized-KV HIP performance evidence** | Issue #27796 open, Aug27 | Qwen3.6-27B/head-dim-256 on RDNA4 showed q8_0 decode about 4% below F16 and q4_0 about 21% below F16. This supports treating quantized KV as a capacity/performance trade rather than assuming compression improves decode. It is not direct RDNA2 evidence. citeturn22search4 |
| **MTP/NextN `n_gpu_layers` accounting remains disputed upstream** | PR #27207 open; latest indexed technical discussion Aug20; related #26605 was merged | Relevant to fitter maintenance, but not a compelling change for an all-layers-offloaded `-ngl 99` Eaman configuration. citeturn27search1 |
| **Current upstream advanced beyond Eaman base** | Master attested at `9cffdcc8…` on Sep3 | Before implementation work, rebase candidates against a fresh master or at least check the Sep1–3 diff. citeturn29search3 |

Two new correctness reports deserve special distinction. PR #26827 concerns **ordering between internal MTP ubatches sharing mutable MTP KV state**. citeturn29search0 Issue #27572 concerns **lifetime of an asynchronous copy of NextN hidden states into host-visible storage that can be overwritten by later graphs**. citeturn27search0 They are related in that both expose unsafe assumptions about asynchronous graph lifetime, but they are not the same bug and should not be “fixed” by one indiscriminate scheduler-wide synchronize.

There is also fresh corroboration for a broader long-session state concern. Issue **#26425** remains open and reports Qwen3.6 MTP results depending on earlier requests even with fresh contexts; its CPU and Vulkan tests suggested the symptom was not purely a GPU-backend issue. citeturn25view2 On August 18, issue **#27296** reported a long-prompt-then-short-prompt sequence that produced repeated leaked-looking content with MTP enabled but behaved normally with MTP disabled. citeturn24search9 Eaman has much stronger long-run validation than those reports had, and its Compact Rollback/adaptive state explicitly resets per sequence, so this is not enough evidence for a blind patch. It does justify adding a **cross-request state-scrub oracle** to the validation suite.

## Ranked candidate table

Scores below are for the stated **RX 6800 16 GiB + RX 6700 XT 12 GiB, fixed asymmetric layer placement, HIP first/Vulkan regression, Qwen3.6-27B native MTP, normally one slot**. “Benefit” is not a generic upstream score.

| Rank | Candidate and exact source | State / latest meaningful activity verified | In current mainline? | In or superseded by Eaman? | Benefit | Feasibility | Evidence | Risk | Priority |
|---:|---|---|---|---|---:|---:|---:|---|---|
| **1** | MTP multi-ubatch serialization: PR **#26827**, commit **`2295535`**; corroborating issue **#28252**. `https://github.com/ggml-org/llama.cpp/pull/26827` citeturn29search0turn29search1turn24search14 | **Open**; code last changed Aug10; new corroborating issue Sep2 | **No**; PR still open | **No** | **5** | **4** | **4** | Medium | **now** |
| **2** | Shared target/MTP backing compute arena while retaining independent schedulers: issue **#27282**. `https://github.com/ggml-org/llama.cpp/issues/27282` citeturn26search0 | **Open**; POC reported Aug17 | **No** | **No**; Eaman accounts the MTP arena but does not eliminate it | **4** | **3** | **3** | High | **now** |
| **3** | Runtime-transient-aware fit/canary and GTT/spill guard: issue **#26432**, plus post-joint-fitter Eaman runtime-OOM evidence. `https://github.com/ggml-org/llama.cpp/issues/26432` citeturn24search2 | Upstream issue **open** since Aug2; materially strengthened by Eaman Aug23 runtime findings | No complete guard identified | **No**; persistent fit is much improved, but runtime transient failure has occurred | **4** | **3** | **4** | Medium | **now** |
| **4** | Ordered/staged `t_h_nextn` handoff for multi-slot MTP: issue **#27572**, linked PR **#27705**. `https://github.com/ggml-org/llama.cpp/issues/27572` citeturn27search0 | Issue **open**, linked PR **open**; latest technical source verified Aug22 | No, based on open linked work | **No**; two-slot CR checkpoint validation is a different subsystem | **4** | **3** | **4** | Medium | **next** |
| **5** | Explicit inter-request MTP state-scrub/reset audit and regression test: issues **#26425** and **#27296**. `https://github.com/ggml-org/llama.cpp/issues/26425`; `https://github.com/ggml-org/llama.cpp/issues/27296` citeturn25view2turn24search9 | Both **open**; fresh corroboration Aug18 | No general fix identified | Partly mitigated/tested by Eaman sequence-state resets and long-run validation, but not explicitly superseded | **3** | **3** | **3** | Medium | **next** |
| **6** | MTP/NextN layer-position accounting in fitter: PR **#27207**, commits `4510cea`, `eba788f`, `52a7800`; related merged PR **#26605**. `https://github.com/ggml-org/llama.cpp/pull/27207` citeturn27search1 | **Open**; maintainer disagreement; latest verified discussion Aug20 | Proposed correction: **No** | Current `-ngl 99` all-offload usage reduces practical exposure | **2** | **2** | **3** | Medium | **watch** |
| **7** | Native/faster quantized HIP KV path for Qwen3.6 head-dim 256: issue **#27796**. `https://github.com/ggml-org/llama.cpp/issues/27796` citeturn22search4 | **Open**, Aug27 | No demonstrated fix | Eaman VEC already solves much of the *workspace* problem, not necessarily unpack throughput | **3** | **2** | **3** | High | **watch** |
| **8** | HIP internal AllReduce: PR **#27825**. `https://github.com/ggml-org/llama.cpp/pull/27825` citeturn22search7 | **Open**, Aug27 | No | Not implemented, but low relevance to fixed layer split | **1** | **3** | **3** | Medium | **reject** |
| **9** | HIP TOP_K work suggested by issue **#27021**; backend-sampling gap from prior Eaman logs | **Open** issue after Aug7; Eaman already benchmarked disabling backend sampling | No complete relevant implementation | Operationally superseded as a priority by measured near-zero benefit | **1** | **2** | **3** | Medium | **reject** |
| **10** | ROCm TurboQuant KV fork | Active third-party implementation; head_dim=128 requirement | No | **Incompatible with this Qwen3.6 head-dim-256 target as published** | **1** | **1** | **2** | High | **reject** |

The ranking intentionally gives correctness fixes high weight. A feature that adds two or three thousand nominal context tokens but can wedge a machine or poison MTP state at 100K+ tokens is not an improvement for the stated “long-session correctness” goal.

## Detailed analysis of each promising candidate

**MTP multi-ubatch ordering — PR #26827 / issue #28252**

PR #26827 changes the internal loop in `llama_context::decode()`: it detects whether an MTP decode has more than one internal ubatch and calls `synchronize()` before allowing the next MTP ubatch to proceed. The patch is only 13 changes in `src/llama-context.cpp` plus regression-test changes in `tests/CMakeLists.txt` and `tests/test-llama-archs.cpp`. It deliberately does **not** synchronize target contexts or one-ubatch MTP decodes. citeturn29search1

The source reproducer used long 100K–180K prefills on dual RTX 3090s. MTP-off was stable; MTP-on could hard-lock the host. Three high-prefill sweeps completed after serialization. The submitter specifically ruled out disabling CUDA graphs and several unrelated CUDA/multi-GPU changes as sufficient fixes. citeturn29search0 The September 2 issue #28252 independently describes the same fundamental problem: two draft graphs can be queued against the same mutable KV state, and the application-level serialization prevents overlap even though the exact backend dependency failure remains unidentified. citeturn24search14

For Eaman, the transferability is high but not proven. HIP uses the same high-level `llama_context::decode()` scheduling path; Qwen3.6 native MTP and very large contexts are exact matches. The topology differs: Eaman normally uses **layer split**, whereas the report used CUDA tensor split. The reasonable inference is therefore “high-priority HIP regression experiment,” not “known RX 6800/6700 XT bug.” citeturn29search0

Expected effects on the dual RDNA2 machine are approximately **context: 0; persistent VRAM: 0; prompt speed: neutral for one-ubatch operations and potentially somewhat lower during long MTP catch-up; ordinary decode: near-neutral unless it creates multiple internal ubatches; correctness: potentially large improvement**. Those performance expectations are architectural inference from the patch’s narrow activation condition, not measured RDNA2 results. citeturn29search1

Implementation difficulty is low-to-medium because the upstream patch is small, but Eaman needs additional CR-specific testing. A deep Compact Rollback replay performs more target/draft state transitions than stock MTP, and the test must demonstrate that the new synchronization point neither changes replay accounting nor creates a redundant host-side fence on every normal speculative step. Likely modified files are `src/llama-context.cpp` and the generated-model test suite; no fit changes should be necessary. citeturn29search1

**Shared target/MTP compute arena — issue #27282**

Issue #27282 is the strongest fresh memory idea because it does not sacrifice KV precision, rollback precision, or draft length. The reported stock configuration attempted a separate **1,040.28 MiB** MTP compute buffer at `n_ubatch=512`; limiting MTP to `n_ubatch=32` reduced that to **785.27 MiB** but still OOMed. The proof of concept kept target and MTP schedulers independent but shared their backing `ggml` graph allocator/compute arena, after which the 196,608-token model loaded and generated at 61.80 t/s. citeturn26search0

That result is on CUDA, Qwen3.8 rather than Qwen3.6, and a single RTX 4090, so its absolute memory numbers should **not** be projected onto RDNA2. On the Eaman dense measurements the MTP compute allocation has generally been closer to roughly 140–165 MiB on ROCm1 once the non-pipeline and VEC-era changes are in place. fileciteturn0file0 Therefore a realistic first hypothesis for this machine is “recover roughly the standalone MTP compute arena if the target arena on that device is already large enough,” not “recover a gigabyte.”

The subtlety is asynchronous lifetime. Sharing is safe only when every graph using the arena has completed before the other context reuses that storage. That makes #26827 conceptually important: the same recent upstream research demonstrates that “the API call returned” does not necessarily mean MTP GPU work completed. citeturn29search0 A HIP prototype should therefore begin with explicit target↔MTP handoff fences and optimize those fences only after correctness is established.

The fitter must also change. Today Eaman correctly **adds** the measured MTP compute cost because it is an independent allocation. fileciteturn0file0 If the allocator becomes shared, fit accounting should reserve the shared arena’s **per-device maximum compatible requirement**, not target compute plus MTP compute. A naive runtime-only allocator-sharing patch without matching fit semantics would make the fitter unnecessarily conservative.

Likely source areas are `src/llama-context.cpp` for scheduler/allocator ownership, GGML backend scheduler/gallocr ownership APIs, `common/speculative.cpp` for passing or negotiating the reusable arena, and the fit measurement path for the new maximum-versus-sum rule. Implementation difficulty is medium-high; regression risk is high because allocator lifetime bugs can produce silent corruption rather than clean OOM.

The likely Eaman benefit is **VRAM: moderate; context: low-to-moderate directly and potentially higher after placement rebalancing; prompt/decode speed: nominally neutral but dependent on synchronization; correctness: neutral if implemented correctly, severe downside if not**. This should remain opt-in until HIP and Vulkan have passed stress testing.

**Runtime-transient-aware fitting and delayed-OOM protection — issue #26432**

Issue #26432 is particularly instructive because it uses **Qwen3.6-27B, native MTP, HIP/ROCm, one slot**. It reports that context plus MTP can load successfully but push a large amount of allocation into GTT on first execution, dropping throughput by more than 60% with no useful startup error. citeturn24search2

The current Eaman patch has significantly better fitting than the affected upstream build: joint target/draft fitting, per-device fixed-layout handling, exact CR checkpoint reservation, and startup-pinned CR device allocations all remove important delayed-allocation surprises. fileciteturn0file0 Nevertheless, the Eaman MoE diagnostics after the joint-fitter integration showed a residual problem of exactly the right category: `ubatch=512` could fit the 262,144 model ceiling and then fail during `launch_fattn<256,2,1>` on the first real request, whereas `ubatch=256` survived at that ceiling. fileciteturn0file0

The proposed Eaman feature is therefore not “add another arbitrary fit margin.” It is a **bounded post-fit runtime canary** tied to the actual `n_batch`, `n_ubatch`, KV types, Flash Attention policy, MTP settings, and device placement. A practical first implementation can:

1. complete the existing joint fit and all startup-pinned allocations;
2. create a disposable/scratch context using the selected placement;
3. execute a representative maximum-microbatch attention workload at a sufficiently large KV position to engage the real HIP kernel/workspace path;
4. inspect allocation failure or unacceptable GTT/residency behavior;
5. if needed, reduce context using the existing fitter’s per-device slope and retry a bounded number of times.

This proposal is an inference from the two independent failure modes; no upstream PR currently supplies that design. citeturn24search2 fileciteturn0file0

Raw fitted context can go **down** under such a guard. Usable context should go **up** in the operational sense: the returned context becomes one expected to survive the configured first real request. Once the transient envelope is measured rather than guessed, it may also become possible to reduce overly conservative static `--fit-target` margins for known-safe profiles.

Files would likely include `common/fit.cpp`, `tools/server/server-context.cpp`, and backend memory-reporting helpers. Difficulty is medium. The principal risk is startup time: running too realistic a canary could make model load expensive. The prototype should therefore first establish the minimum graph execution needed to expose the HIP transient high-water condition.

**Multi-slot NextN staging/event ordering — issue #27572 / PR #27705**

Issue #27572 is much more specific than a generic “multi-slot MTP might be buggy” warning. It identifies `ggml_backend_tensor_get_async()` copying the `t_h_nextn` tensor while the scheduler’s extra buffer can be reused by another internal ubatch or another slot’s graph. In the reported HIP reproducer, device-side rows remained sane while the eventual host-side MTP input became NaN; speculative acceptance then collapsed to exactly zero. citeturn27search0

A full `ggml_backend_sched_synchronize()` immediately after the asynchronous copy fixed 16/16 concurrent reproductions, but the reporter measured roughly a **40%** generation-speed loss on long prompts because the broad synchronization interfered with the scheduler’s graph-reuse behavior. The issue therefore explicitly recommends a narrower solution such as event-ordering the copy or rotating the extra buffer. citeturn27search0

For Eaman, a small **ring of NextN staging buffers**, indexed by active ubatch/slot and recycled only after a backend event signals copy completion, is more attractive than a global sync. Memory cost should be small relative to KV/recurrent storage because the staging object is hidden-state rows rather than context-length KV; however, exact size must be measured from Qwen3.6’s actual `n_embd`, batch rows, and per-slot concurrency rather than assumed.

This is not a current one-slot performance enhancement. With `-np 1`, Eaman’s existing workload is less exposed, and its real two-slot CR test already proved checkpoint isolation. fileciteturn0file0 The candidate becomes **high priority before making multi-slot MTP a supported performance configuration**, especially with long prompts that force multiple ubatches.

Likely files: `src/llama-context.cpp` around NextN extraction, scheduler extra-buffer ownership, and MTP/generated-model concurrency tests. Correctness effect is high; context and persistent VRAM effects are negligible; prompt throughput should remain close to current behavior if event ordering avoids global fences.

**Inter-request MTP state sanitation**

Issue #26425 reports a Qwen3.6 MTP session in which identical fresh requests behaved differently depending on previous MTP requests, while non-MTP requests remained reproducible; the report reproduced aspects of the symptom on both Vulkan and CPU. citeturn25view2 Issue #27296 later reported a long-prompt→short-prompt ordering where the short MTP request repeatedly emitted text resembling content from the earlier request, while MTP-off did not. citeturn24search9

These reports do not establish the exact stale object. For that reason, I would first implement a **deterministic regression harness rather than a speculative fix**: server start → short A → long B → fresh short A; compare target-visible state, MTP context KV/recurrent sequence ranges, pending NextN embedding rows, speculative counters, and greedy output against a restarted-server A reference.

Eaman already resets adaptive state on sequence begin/slot reuse and has substantial CR replay and long-run coverage. fileciteturn0file0 That lowers the probability that the old report directly maps to current Eaman, but it does not prove every MTP-owned sequence and host-side handoff buffer is scrubbed. This is therefore **next**, not **now**.

**Quantized HIP KV execution for Qwen3.6**

Issue #27796 is useful new evidence, but not yet useful new code. On an RDNA4 GPU with the same Qwen3.6-27B family and head dimension 256, F16 decoded at 23.44 t/s at 16K context, q8_0 at 22.43 t/s and q4_0 at 18.56 t/s. The issue attributes the ordering to quantized unpacking cost. citeturn22search4

This is materially newer than the August 7 research and supports the Eaman project’s own conclusion that KV compression cannot be judged by persistent byte count alone. The Eaman data already demonstrated both sides: quantized MTP KV can save hundreds of MiB, while forcing VEC on large quantized-target-KV prefills can greatly reduce prompt throughput. fileciteturn0file0

I would therefore keep a direct/fused q8/q4 HIP kernel for head-dim 256 on the watch list, but **not start there**. There is no fresh upstream implementation with RDNA2 measurements to port. The ROCm TurboQuant fork is not a substitute: its published GPU path requires exactly head dimension 128 and bulk-dequantizes before standard Flash Attention, whereas Qwen3.6-27B in #27796 is head dimension 256. citeturn22search0turn22search4

## Ideas rejected or already implemented

**Compact rollback / configurable rollback depth — already implemented.** The August 7 report’s highest-value proposal has become `--spec-mtp-cr-depth`; the current code includes deep-rejection restore/replay, exact per-device checkpoint sizing, startup-pinned device allocation, host fallback, multi-slot tests, and integration with adaptive accounting. Dense Q6_K_L measured +12,800 context tokens at depth 1 after the Aug23 rebase. It must not reappear as a “new” recommendation. fileciteturn0file0

**Lower-precision rollback checkpoints — reject for now.** This was explicitly deferred in the project after Compact Rollback proved that reducing the *number* of stored recurrent states captured a large memory gain without altering numerical state representation. fileciteturn0file0 I found no material post-Aug7 upstream implementation demonstrating safe low-precision Qwen3.6 recurrent rollback. Because a restored checkpoint becomes the basis for subsequent recurrent computation after a deep rejection, even a small quantization error can become a correctness issue. A shared compute arena is a better next VRAM experiment because it removes duplication without approximating model state.

**Generic adaptive draft sizing — already implemented.** Eaman ported the relevant technique, fixed CR accounting and sequence reset behavior, and found that adaptive control helps some creative workloads while costing about 2% on long code in the recorded matrix. fileciteturn0file0 A future controller could become replay-cost-aware, but there is not enough fresh evidence to prioritize another controller over the concrete correctness and memory findings above.

**More MTP fit-refinement passes / old MTP estimator — superseded.** Current upstream joint fitting reevaluates the extra context as target context changes, and Eaman adopted it during the Aug23 refresh. fileciteturn0file0 The old proposal from `web_search.md` to eventually unify fitting has effectively happened. fileciteturn0file2

**Generic fixed-layout per-device fitting — already implemented.** This remains an Eaman patch feature and should be preserved, not proposed anew. fileciteturn0file0

**Automatic boundary-tensor rebalancing — useful, but not top-three because the capacity gain is already operationally realized.** Eaman’s manual FFN-triplet override on layer 39 has already converted asymmetric headroom into roughly a 7.7% context increase in the matched Q8 target-KV experiment, while preserving the baseline graph-split count. fileciteturn0file0 Automating that search would improve maintainability when cache types or CR depth change, but it would not give the current production profile the same +9,728 tokens a second time. Upstream PR #27207 also shows that layer/MTP accounting in fit remains contested, which argues against expanding fixed-placement mutation logic before that code is rebased and understood. citeturn27search1

**PR #27207 MTP/NextN layer accounting — watch, not cherry-pick.** Its proposed fix is explicitly disputed by a maintainer as potentially overfitting when MTP is not in use. citeturn27search1 The Eaman setup explicitly offloads all target layers with `-ngl 99`, further lowering immediate benefit. A regression test around layer indices and tensor overrides is worthwhile on future rebases; importing the disputed correction now is not.

**HIP TOP_K — reject as an Eaman priority.** The August 7 search identified ROCm TOP_K/backend-sampling fallback. fileciteturn0file2 Eaman subsequently ran a controlled 9B comparison: backend sampling enabled versus disabled had identical acceptance and changed median generation speed by only −0.33% when disabled, with mean difference within noise. fileciteturn0file0 The newer TOP_K complaints concern other uses such as very wide indexer operations; they do not overturn the exact Eaman measurement.

**HIP AllReduce PR #27825 — reject for this topology.** The PR is new and open and aims to enable llama.cpp’s internal AllReduce path on HIP. citeturn22search7 The Eaman configuration uses asymmetric **layer** placement, where the principal communication is crossing layer/device boundaries; this is not the tensor-parallel reduction workload that makes an internal AllReduce implementation compelling. It is worth revisiting only if the patch later experiments with row/tensor split.

**TurboQuant ROCm fork — incompatible as published.** The current fork reports 72–78% KV savings and a ROCm implementation, but its documented GPU path requires **head_dim=128** and performs bulk pre-dequantization to F16 before standard Flash Attention. citeturn22search0 Fresh llama.cpp issue #27796 identifies Qwen3.6-27B’s tested configuration as **head_dim=256**. citeturn22search4 It therefore cannot simply be ported to the present model, and its pre-dequantization design also reintroduces the kind of temporary F16 storage that motivated Eaman’s VEC work.

**Third-party shared-model/tensor claims — largely superseded for native Qwen MTP.** Native Eaman MTP already initializes its draft context against the target model rather than loading an independent full model. fileciteturn0file0 Fork claims about eliminating duplicate target model mappings should not be confused with issue #27282’s still-relevant duplication of the **compute allocator**.

**ROCm VMM as a general llama allocator — reject.** It remains too risky for this consumer-RDNA2 patch. Recent ROCm reports include VMM failures even on newer Radeon hardware, including a gfx1201 report where `hipMemSetAccess()` made VMM unusable. citeturn28search6 The Eaman objective is long-session reliability; replacing a working allocator with a VMM layer without direct gfx1030/gfx1031 validation would invert that priority.

**HIP memory-pool trimming/bypass — watch only.** It remains relevant if VEC-off or tile/MFMA paths demonstrate retained temporary memory or delayed GTT spill, but the old PR #21830/#22094 proposals were already covered in the August baseline and were closed/unmerged. fileciteturn0file2 Current Eaman has a more controlled VEC on/off mechanism and much stronger delayed-OOM validation. Reopen this area only on a reproducible present-day pool-retention trace.

## Recommended top three implementation experiments

**First: port PR #26827 as an isolated Eaman correctness branch.**

Port commit `2295535` onto a fresh post-September-3 upstream base plus the Eaman patch, preserving its narrow predicate: only `LLAMA_CONTEXT_TYPE_MTP` and only after discovering a second internal ubatch. Do not replace it with a blanket `llama_synchronize(ctx_dft)` around every speculative call. The exact upstream diff is small and has a purpose-built synthetic test. citeturn29search1

For Eaman specifically, extend the test to exercise Compact Rollback replay after a multi-ubatch MTP catch-up. Acceptance accounting, CR replay count, and final target recurrent state must match the current branch. The expected win is not throughput; it is removal of an asynchronous shared-state hazard before it becomes a 100K+ HIP failure.

**Second: prototype shared target/MTP gallocr ownership at `-np 1`, behind an opt-in flag.**

Begin with the narrow semantics proven in issue #27282: **independent schedulers, common backing graph allocator**, only when target and MTP use compatible backend buffer types and their execution is serialized. citeturn26search0 Do not share the entire scheduler.

First instrument, do not optimize: log for each ROCm device the current target compute arena, MTP compute arena, required shared high-water mark, actual backing allocation, and every target↔MTP ownership handoff. Make a backend synchronization mandatory at the first prototype’s handoff. Once the result is correct, profile whether the synchronization is redundant given the #26827 ordering changes.

The fitter must be changed in the same experiment so that compatible shared compute is accounted as the required maximum on each device, not as two simultaneous arenas. Otherwise the code may work but recover no fitted context.

For this particular Q6 dense configuration I would consider the experiment successful even if it yields only **100–200 MiB** of real ROCm headroom rather than the CUDA POC’s much larger number. That amount is comparable to the tensor-placement increments that have previously moved the fitted-context boundary by thousands of tokens. fileciteturn0file0

**Third: add a measured runtime-transient fit guard, initially diagnostic-only.**

The first version should not automatically reduce context. It should run after fitting/preallocation, exercise the configured HIP attention path with the real `n_ubatch`, and report:

`fitted persistent headroom → transient minimum free VRAM → post-sync retained VRAM → GTT growth → pass/fail`

Use it against `ubatch=256/384/512` and both VEC policies. Once it reliably detects known bad configurations, promote it to an optional bounded fit correction.

This experiment directly addresses the discrepancy between successful persistent fitting and first-request failure already observed by Eaman, and the silent GTT behavior independently reported for Qwen3.6/MTP on HIP. fileciteturn0file0 citeturn24search2 It should improve confidence in the **maximum usable** context more than shaving another few MiB from a data structure would.

I would implement the multi-slot `t_h_nextn` fix immediately after these three if `-np 2` or greater is intended for regular use. For the current one-slot production profile, its urgency is lower despite excellent evidence.

## Concrete validation matrix

The three experiments should be tested against one common reference build: freshly rebased upstream plus the unmodified current Eaman feature set. Because upstream master was at `9cffdcc801582616250520966699cb5b25d28243` on September 3 while Eaman’s validated refresh is from August 31, record both upstream and Eaman commit hashes in every result. citeturn29search3 fileciteturn0file0

| Test dimension | Reference | PR #26827 serialization | Shared compute arena | Runtime-transient fit guard | Pass condition |
|---|---|---|---|---|---|
| **Build** | HIP+Vulkan, `gfx1030;gfx1031`, all required quantized FA instances | Same | Same | Same | HIP, Vulkan, server, parser and recurrent/generated-model tests all link/pass |
| **ROCm runtime** | Current pinned 7.14 tree | Same | Same | Same | No change in device discovery or kernel selection |
| **Second ROCm tree** | Current pinned 7.15/core-10.0 tree used in Aug31 validation | Same | Same | Same | No version-specific allocator/order regression fileciteturn0file0 |
| **Model** | Qwen3.6-27B Q6_K_L production GGUF | Same | Same | Same | Exact model/layout |
| **Placement** | `0.6,0.4`, `-ngl 99`, layer-39 FFN triplet override | Same | Same | Same | No implicit placement movement |
| **Target KV** | Q8_0/Q8_0 | Same | Same | Same | Fixed |
| **MTP KV** | Q4_0/Q4_0 | Same | Same | Same | Fixed |
| **MTP** | n-max 3, CR depth 1 and full-depth control | Both | Both | Both | CR statistics internally consistent |
| **Pipeline** | target off; MTP off | Same | Same | Same | No scheduler-copy policy drift |
| **Microbatch** | 256, **384**, 512 | Critical: force multi-ubatch long prompt | All three | Primary sweep variable | No hidden dependence on only one ubatch size |
| **Context** | 32K fixed | yes | yes | yes | Basic correctness/performance |
|  | 100K fixed | yes | yes | yes | Long-prompt stability |
|  | 130K–140K fixed | yes | yes | yes | Near current production region |
|  | automatic fit | yes | primary capacity metric | primary safety metric | Listening + real request succeeds |
| **Prompt pattern** | One long request | ≥3 identical repetitions | ≥3 | ≥3 | No hang/OOM/GTT collapse |
|  | prompt-cache restore then extension | yes | yes | yes | No stale-state or allocator hazard |
|  | long request → fresh short request → same short request | yes | yes | yes | No cross-request MTP contamination |
| **Generation** | 192 tokens | deterministic smoke | deterministic smoke | deterministic smoke | Counters/output within existing oracle |
|  | 1,024 tokens | yes | yes | yes | CR replay exercised |
|  | ≥8K tokens | at least one long run | at least one long run | at least one long run | No delayed OOM/corruption |
| **Parallel slots** | `-np 1` | primary | **only supported initial shared-arena mode** | primary | Stable |
|  | `-np 2` | regression | shared arena must remain disabled unless explicitly made slot-safe | regression | No cross-talk; use #27572-oriented long prompts |
| **Vulkan** | same fixed context, `pipeline=auto` unless project policy says otherwise | regression | allocator-sharing disabled first, then explicit experiment | guard diagnostic only first | No correctness/performance regression |

For **PR #26827**, add two acceptance thresholds beyond simple “did not hang”: fixed 32K/single-ubatch decode should stay within roughly **2%** generation throughput of the reference, while long-prompt prefill at the production `ubatch=384` should remain within roughly **5%** unless the reference reproduces the unsafe overlap. Those are engineering acceptance targets, not upstream measured guarantees. The upstream patch itself claims to leave one-ubatch MTP asynchronous. citeturn29search1

For the **shared compute arena**, capture per-device numbers before judging context. A good result is either (a) a measurable fitted-context increase, or (b) at least roughly **100 MiB** less persistent allocation on the device hosting the MTP arena at the same context with no throughput loss beyond noise. If the saving lands on the non-limiting GPU, combine it with a *temporary test-only* placement adjustment to determine whether the recovered headroom is convertible into context; do not declare “no benefit” merely because a coarse fixed placement leaves the other GPU limiting. The project has already seen exactly this phenomenon with Q4 MTP KV savings. fileciteturn0file0

For the **runtime fit guard**, the critical result is predictive power. It must flag a deliberately recreated known-unsafe large-ubatch profile and pass a known-safe profile. Only after that should automatic context reduction be enabled. Track both dedicated VRAM and GTT before startup, after fit, at maximum prompt-processing pressure, after synchronization, after a long generation, and after server shutdown. Issue #26432 demonstrates why throughput must be checked alongside “allocation succeeded.” citeturn24search2

A separate **multi-slot correctness extension** should reproduce the structure of #27572: simultaneous long prompts large enough to create many internal ubatches, then verify that MTP acceptance does not collapse toward zero and that NextN host rows never contain NaNs. citeturn27search0 This is valuable even before implementing #27705 because it tells whether the current dual-RDNA2 stack is exposed.

## Suggested update to improvemets_list.md

The current file is materially stale. It lists fixed-layout fitting, pipeline control, non-pipeline MTP, and HIP VEC as if those were the complete Eaman delta. fileciteturn0file1 It omits Compact Rollback, exact checkpoint reservation/preallocation, adaptive MTP, the upstream joint-fitter supersession, the change of VEC forcing to default-off, and the August 31 dual-ROCm refresh. fileciteturn0file0

A replacement should describe only what remains true relative to the current upstream base and should explicitly separate **upstream functionality**, **Eaman-only changes**, and **experimental controls**. Suggested text:

```markdown
# Eaman Patch Improvements Relative to llama.cpp Mainline

This file describes the maintained Eaman patch. Historical Qwen/Sol names are
retained only for old artifacts.

The authoritative project state is `summary_eaman.md` plus `status` and
`AGENTS.md`. This file is a feature inventory, not the authoritative validation
record.

## Current upstream base

The latest fully validated Eaman refresh recorded in the project state is based
on llama.cpp commit:

    662a0b0121a53c23b825a71e64ab6eff59b7f4d8

Cumulative Eaman integration tip:

    9ee6674f

Standalone MTP Compact Rollback integration tip:

    cdc3d024

The refreshed cumulative build was validated with both HIP and Vulkan for
gfx1030/gfx1031 under the two pinned ROCm installations described in
`summary_eaman.md`.

Upstream moved to the joint target/draft fitting design before this base.
The obsolete Eaman fixed-pass MTP estimator/refinement and the temporary MoE
scratch-placement workaround are therefore not current Eaman features.

## Fixed-layout per-device fitting

When GPU layer placement is explicitly fixed, the Eaman patch fits the context
against each selected GPU independently and uses the limiting device instead of
assuming that model layers can be redistributed.

This remains relevant to asymmetric fixed placement such as the RX 6800 +
RX 6700 XT configuration.

Automatic placement retains upstream fitting behavior.

Primary file:

    common/fit.cpp

## Explicit target pipeline control

The target context supports:

    --pipeline-parallel auto|on|off

`auto` preserves normal upstream selection. `off` is useful when scheduler
copies and duplicated compute workspace cost more VRAM than pipeline execution
provides in throughput.

MTP uses a separate policy and remains non-pipeline.

Primary files include:

    common/arg.cpp
    common/common.cpp
    common/common.h
    include/llama.h
    src/llama-context.cpp
    src/llama.cpp

## Non-pipeline MTP scheduling

The native MTP context is fitted and initialized with pipeline scheduling
disabled. Fit and runtime therefore use the same MTP scheduler policy.

This avoids unnecessary scheduler copies/workspace on the validated
asymmetric dual-GPU topology.

Primary files include:

    common/speculative.cpp
    src/llama-context.cpp

## MTP Compact Rollback

The Eaman patch supports:

    --spec-mtp-cr-depth N

The default preserves full upstream rollback depth. A smaller opt-in depth keeps
fewer target recurrent rollback states resident and uses checkpoint
restore/replay when a rejection requires state older than the retained direct
rollback window.

The implementation includes:

- direct recurrent rollback for shallow rejection;
- pre-verification PARTIAL_ONLY checkpoint restore plus replay for deep
  rejection;
- on-device recurrent checkpoints where the layout supports them;
- host fallback for fragmented/unsupported device checkpoint layouts;
- exact per-device checkpoint allocation measurement;
- startup allocation/pinning so request-time checkpoint creation cannot consume
  unreserved fitted headroom;
- per-slot checkpoint allocation;
- parser, recurrent-state and generated-model regression coverage;
- replay statistics exposed to the server timing/accounting path.

Compact Rollback is an Eaman feature and must not be described as pending work.

## Adaptive MTP draft sizing

The Eaman patch includes the opt-in control:

    --spec-draft-adaptive

`--spec-draft-n-max` remains the upper bound.

The controller resets per sequence/slot, distinguishes an adaptive ceiling from
confidence-driven early termination, and is integrated with Compact Rollback so
a replay does not double-count acceptance feedback.

The feature remains default-off because measured benefit is workload-dependent.

## HIP quantized-KV VEC control

The Eaman HIP Flash Attention changes allow supported quantized-KV shapes to use
the VEC path, which dequantizes without the large context-sized F16 conversion
workspace of the alternative HIP path.

The current public diagnostic/capacity control is:

    --hip-fa-force-vec on|off

Current cumulative behavior defaults forcing to OFF. Normal backend dispatch is
therefore used unless VEC forcing is explicitly requested.

This is a trade-off control, not a universal performance optimization:
forced VEC can materially improve fitted context for memory-heavy quantized-KV
configurations, while upstream large-query dispatch can be much faster for
prompt processing.

Unsupported shapes continue through normal upstream dispatch.

Primary production file:

    ggml/src/ggml-cuda/fattn.cu

The option also touches the common argument/context plumbing needed to keep fit
measurement and runtime kernel policy identical.

## Operational asymmetric tensor placement

The current dense Qwen3.6-27B production profile uses a manually selected
boundary tensor override that moves the FFN triplet of block 39 to ROCm1 while
retaining the 0.60/0.40 layer boundary:

    -ot '^blk\.39\.ffn_(up|gate|down)\.weight$=ROCm1'

This is an operational launcher/profile optimization, not an automatic Eaman
fitter feature.

Do not claim that the patch automatically discovers this override.

## Features now supplied or superseded by upstream

Do not list the old Eaman MTP fixed-pass estimator/refinement or the old MoE
scratch-placement workaround as current patch features. Upstream's joint
target/draft fitting replaced those mechanisms and the Eaman branch was adapted
to it.

Likewise, upstream recurrent-state changes included by the current base should
not be claimed as Eaman-authored work.

## Current research candidates, not implemented features

The following are under investigation and must not be described as present in
the Eaman patch:

- llama.cpp PR #26827 / commit 2295535: serialization of internally split MTP
  ubatches to avoid overlapping mutation of MTP state;
- llama.cpp issue #27282: possible reuse of a backing target/MTP compute arena
  while retaining independent schedulers;
- llama.cpp issue #27572 / PR #27705: safe lifetime/event ordering for the
  asynchronous NextN hidden-state handoff under parallel slots;
- runtime-transient-aware fitting/canary logic for HIP kernels whose execution
  allocates or retains memory beyond the persistent startup estimate.

These remain experiments until validated on the RX 6800 + RX 6700 XT system
with both HIP and Vulkan regression coverage.

## Validation scope

Current positive results include long dense Qwen3.6 sessions, Compact Rollback
replay, concurrent checkpoint isolation, and successful combined HIP/Vulkan
builds on the pinned ROCm installations recorded in `summary_eaman.md`.

A startup fit or a short generation is not by itself evidence of near-context
or delayed-OOM safety. Performance claims should distinguish matched benchmarks
from smoke tests and historical cross-build comparisons.
```

That update removes the two most misleading historical implications in the present file: that the old four-feature inventory is complete, and that HIP VEC forcing is still simply the default Eaman behavior. fileciteturn0file1 It also avoids attributing upstream joint fitting or recurrent changes to the Eaman patch, while making Compact Rollback and adaptive MTP visible as maintained Eaman work. fileciteturn0file0