# ROCm MTP Memory Accounting and Context Optimization

Available from: https://store.piffa.net/lm/bug/
This document is supposed to be ingested by a LLM in order to re apply or customize this patch.

## Current Status

Native MTP speculative decoding creates an additional llama context alongside the target context. This patch improves accounting for this context, fits fixed multi-GPU layouts per device, makes target pipeline scheduling explicit.

The current complete patch is:

```text
/home/eaman/llama/bug/sol4.patch

or:

latest_rocm_improvement_*.patch

if downloaded online.
```


The latest patch contains the original MTP fitting and pipeline changes plus the merged HIP VEC flash-attention dispatch from `vec_sol`.

It provides:

1. Complete MTP context and compute accounting.
2. MTP measurement at the target's fitted context with iterative refinement.
3. Per-device fitting when GPU placement is explicitly fixed.
4. Non-pipeline scheduling for the MTP context.
5. Explicit `--pipeline-parallel auto|on|off` control for the target context.
6. HIP VEC dispatch for supported quantized-KV attention shapes.

The best previously validated ROCm configuration used F16 MTP KV and disabled target pipeline parallelism:

```text
--pipeline-parallel off
--cache-type-k-draft f16
--cache-type-v-draft f16
```

The merged VEC update initially reached **147,712 context tokens** with Q4/Q4 MTP KV on the tested Qwen3.6 27B ROCm system, compared with 121,088 for the previous F16/off build: +26,624 tokens (+21.99%). Startup and short generation succeeded; long-run stability and throughput comparisons remain pending. This is a historical cross-build result, not the current matched-cache comparison described below.

---

## Test System

| Component | Configuration |
|---|---|
| Operating system | Debian Sid |
| Target model | ThinkingCap Qwen3.6-27B Q6_K_L |
| Model layout | 64 target layers plus 1 NextN/MTP layer |
| GPU 0 | AMD Radeon RX 6800, 16 GiB |
| GPU 1 | AMD Radeon RX 6700 XT, 12 GiB |
| Backend | HIP/ROCm build with Vulkan support |
| Split mode | Layer |
| Tensor split | 0.6,0.4 |
| GPU layers | 99, explicitly fixed |
| Target KV cache | K=q8_0, V=q5_1 |
| MTP KV cache | K=F16, V=F16 |
| Maximum MTP draft length | 3 |
| Fit target | 50 MiB per device |
| Batch / microbatch | 1024 / 384 |
| Parallel slots | 1 |
| Pipeline mode | Off for target and MTP |
| Base source | build 909 (`7bd8282`) |
| Working branch | `eaman` |

Model:

```text
/home/eaman/lm/models/bottlecapai/ThinkingCap-Qwen3.6-27B-Q6_K_L.ggu.gguf

on-line: https://store.piffa.net/lm/bug/llama_scripts/
```

Current launcher:

```text
/home/eaman/launch/dense.sh
```

---

## Issues Addressed

### MTP owns a separate runtime context

Native MTP initializes another `llama_context` for the draft path:

```cpp
llama_context * ctx_dft = llama_init_from_model(model_tgt, cparams);
```

This context owns context-dependent memory in addition to the target context:

- MTP KV memory;
- a GPU compute buffer;
- a host compute buffer;
- backend-specific temporary workspace.

The complete MTP allocation must be included in the memory reservation used by the target fitter.

### MTP memory depends on the fitted context

The model is trained for 262,144 tokens, but available VRAM requires a smaller runtime context. Measuring MTP at the trained context overstates its runtime allocation. Measuring it only once at a preliminary fitted value also remains conservative after the target fit changes.

The dependency is:

```text
target context -> MTP memory -> target reservation -> fitted target context
```

The MTP allocation must be measured at the fitted target context and refined.

### Pipeline scheduling has a high memory cost on this topology

Pipeline parallelism created four scheduler copies for the target context and duplicated a large compute allocation on both GPUs. With one active server slot and these asymmetric GPUs, the extra workspace and synchronization did not provide a measured performance advantage.

The MTP graph also did not benefit from pipeline scheduling because its useful GPU work was placed on one device.

The solution therefore keeps pipeline policy explicit:

- `auto` preserves the normal llama.cpp decision;
- `on` requests pipeline scheduling and warns if unavailable;
- `off` disables it;
- MTP remains non-pipeline.

### Fixed GPU placement was fitted with a generic allowance

The tested command explicitly fixes placement:

```text
-ngl 99
--split-mode layer
--tensor-split 0.6,0.4
```

For this case, reserving memory for possible layer redistribution is unnecessary because the fitter cannot change `n_gpu_layers`. The useful constraint is the memory limit of each individual device.

The optimized fitter interpolates a context limit for every selected GPU and uses the lowest result. Automatic GPU-layer placement retains the existing conservative behavior.

---

## Solution Implemented by `sol4.patch`

### 1. Use the fitted target context for runtime MTP

File: `common/speculative.cpp`

```cpp
if (spec_mtp) {
    cparams.ctx_type = LLAMA_CONTEXT_TYPE_MTP;
    cparams.pipeline_parallel_type = LLAMA_PIPELINE_PARALLEL_TYPE_DISABLED;
    cparams.n_ctx = llama_n_ctx(ctx_tgt);
}
```

The runtime MTP context now uses the actual fitted target context instead of reserving context-dependent workspace for the full trained context.

### 2. Measure the complete MTP allocation

File: `tools/server/server-context.cpp`

```cpp
const size_t bytes =
    (measure_model_bytes ? dmd[j].model : 0) +
    dmd[j].context +
    dmd[j].compute;
```

Both MTP context storage and its independent compute workspace are added to the per-device target reservation.

### 3. Refine the MTP reservation

File: `tools/server/server-context.cpp`

The server performs two refinement passes:

```text
1. Fit the target with the current MTP reservation.
2. Measure MTP at the newly fitted context.
3. Replace the reservation with the refined per-device values.
4. Repeat once and perform the final target fit.
```

This resolves most of the fitting dependency without an open-ended loop.

### 4. Fit fixed multi-GPU layouts per device

File: `common/fit.cpp`

When `n_gpu_layers` is explicitly fixed, the fitter calculates the context supported by each GPU after its configured margin, rounds it to the required 256-token boundary, and selects the limiting device.

### 5. Keep MTP non-pipeline

Files:

```text
common/speculative.cpp
src/llama-context.cpp
tools/server/server-context.cpp
```

MTP fitting and runtime initialization both use the same non-pipeline scheduler policy. This prevents estimation and runtime allocation from selecting different compute-buffer layouts.

### 6. Add explicit target pipeline control

Files:

```text
include/llama.h
src/llama.cpp
common/common.h
common/common.cpp
common/arg.cpp
src/llama-context.cpp
```

The public context parameters and common command-line parser expose:

```text
--pipeline-parallel auto|on|off
```

The default is `auto`, so applying the patch does not silently change target scheduling. The optimized ROCm launcher explicitly selects `off`.

---

## ROCm MTP Cache Finding

For this model and HIP flash-attention path, quantized MTP KV did not minimize total GPU memory. Q4_0 K and V required a large F16 conversion workspace in the selected ROCm flash-attention implementation.

Measured MTP allocations before target pipeline optimization were:

| Draft KV | Context | ROCm1 KV | ROCm1 compute | MTP GPU total |
|---|---:|---:|---:|---:|
| Q4_0 | 99,072 | 108.84 MiB | 519.84 MiB | 628.68 MiB |
| F16 | 104,704 | 409.00 MiB | 127.70 MiB | 536.70 MiB |

F16 used more persistent KV memory but reduced the conversion workspace enough to save approximately 92 MiB of total MTP GPU memory.

F16 is therefore the best validated ROCm MTP cache for this model:

```text
--cache-type-k-draft f16
--cache-type-v-draft f16
```

This is a runtime recommendation, not hard-coded patch behavior. Vulkan and other models must be measured separately.

---

## Final Validation

### Fit sequence

With F16 MTP KV and target pipeline mode off:

```text
Target-only fit:         121,856
Initial MTP estimate:    616.26 MiB
First fitted context:    118,784
Refined MTP estimate:    602.01 MiB
Second fitted context:   119,808
Refined MTP estimate:    606.76 MiB
Final context:           119,552
```

The runtime target and MTP contexts both initialized at 119,552 tokens.

### Runtime allocation

Target context:

```text
ROCm0 KV buffer:         2116.09 MiB
ROCm1 KV buffer:         1269.66 MiB
ROCm0 compute buffer:     614.84 MiB
ROCm1 compute buffer:     614.84 MiB
Host compute buffer:      102.84 MiB
Scheduler copies:              1
```

MTP context:

```text
ROCm1 KV buffer:          467.00 MiB
ROCm1 compute buffer:     138.57 MiB
Host compute buffer:      102.58 MiB
Scheduler copies:              1
```

The validation log contained no `cudaMalloc` failure and no scheduler fallback.

### Pipeline auto versus off

| Metric | F16 / auto | F16 / off | Change |
|---|---:|---:|---:|
| Final context | 104,704 | **119,552** | **+14,848 (+14.18%)** |
| Target compute per GPU | 806.86 MiB | 614.84 MiB | -192.02 MiB |
| Target host compute | 322.87 MiB | 102.84 MiB | -220.03 MiB |
| Target scheduler copies | 4 | 1 | -3 |

### Initial performance sample

One old-prompt sample produced:

| Metric | Comparison run | `sol3`, F16/off | Change |
|---|---:|---:|---:|
| Prompt processing | 206.31 t/s | 251.09 t/s | +21.71% |
| Token generation | 32.49 t/s | 36.05 t/s | +10.96% |
| Draft acceptance | 90.640% | 88.349% | -2.291 points |

This is encouraging but not a controlled benchmark. The generated lengths, build configuration, and draft-cache configuration were not identical. Stability testing and repeated same-build measurements remain necessary.

---

## Source Files Modified

| Source file | Purpose |
|---|---|
| `common/arg.cpp` | Parse the target pipeline mode |
| `common/common.cpp` | Transfer pipeline mode to llama context parameters |
| `common/common.h` | Store the common pipeline mode parameter |
| `common/fit.cpp` | Fit fixed GPU placement against per-device limits |
| `common/speculative.cpp` | Use fitted `n_ctx` and non-pipeline mode for runtime MTP |
| `include/llama.h` | Define the public pipeline mode API |
| `src/llama-context.cpp` | Apply pipeline policy to scheduler creation |
| `src/llama.cpp` | Provide the pipeline mode name helper |
| `ggml/src/ggml-cuda/fattn.cu` | Dispatch supported HIP quantized-KV attention shapes to VEC |
| `tools/server/server-context.cpp` | Measure and refine complete MTP memory |

---

## Important Artifacts

| Artifact | Description |
|---|---|
| `/home/eaman/llama/bug/sol4.patch` | Current complete source patch |
| `/home/eaman/llama/bug/rocm_improvement.patch` | Duplicate copy of `sol4.patch` |
| `/home/eaman/llama/bug/logs/pipeline_off_f16.log` | Final F16/off validation and prompt sample, 119,552 context |
| `/home/eaman/llama/bug/logs/sol2_f16.log` | F16/auto comparison log, 104,704 context |
| `/home/eaman/llama/bug/logs/q6_current_full.log` | Q4_0/auto comparison log, 99,072 context |
| `/home/eaman/models/think.sh_sol` | Current optimized ROCm launcher |
| `/home/eaman/llama/llama.cpp/build/bin` | Source-tree build output |
| `/home/eaman/llama/bin_vulkan` | Deployed HIP and Vulkan build updated by the user |
| `/home/eaman/llama/bug/old/possible_improvements.md` | Archived, superseded improvement notes |
| `/home/eaman/llama/bug/dual_gpu_context_balancing_guide.md` | Reusable manual dual-GPU placement and tensor-override procedure |

Patch base and checksum:

```text
Historical base commit: 7bd8282
Historical solution commit: e37aae4
Historical complete patch SHA-256: afc4216b15e833712c9c9228b41a1134d6f703614f6aec334c73e53f2ef46839
Current eaman tip commit: a4e47992
Merged mainline commit: 8144f319
Recovery branch: eaman-pre-latest-8144f31
Current standalone patch SHA-256: dc266cba88936c9e6db596dca897e4b7470c51cc9f3597c93026d967512ed6f5
```

---

## Deployment Notes

The current build contains both HIP and Vulkan backends:

```text
GGML_HIP=ON
GGML_VULKAN=ON
```

The complete build output, including all llama and GGML shared libraries, must be deployed together. The new `a4e47992` build is in `/home/eaman/llama/llama.cpp/build/bin`; `/home/eaman/llama/bin_vulkan` was not updated by this refresh.

ROCm recommended settings for this model:

```text
--pipeline-parallel off
--cache-type-k-draft f16
--cache-type-v-draft f16
```

Vulkan should initially retain `--pipeline-parallel auto` and compare Q4_0 with F16 MTP KV because the ROCm conversion-workspace result does not automatically apply to Vulkan.

---

## Historical d2f8305 IQ4_XS Context Comparison

On 2026-08-10, `sol4.patch` was applied with `git apply --3way` to current
mainline commit `d2f8305`. The patched and unpatched builds used Release mode,
HIP and Vulkan backends, and HIP architectures `gfx1030;gfx1031`. Both builds
compiled and linked `llama-server` successfully.

The standalone patch rebased directly to this mainline commit is:

```text
/home/eaman/llama/bug/latest_rocm_improvement_d2f8305.patch
SHA-256: d215611605f0d1a3299613f8e34e4bf65ce7cdbd5e36a1383f2f6d4fdf61c319
```

It applies cleanly without three-way fallback to commit `d2f8305` and carries
header metadata for llama.cpp version 996 and ROCm 7.14/Vulkan validation.

The single-GPU IQ4_XS launcher settings were taken from
`/home/eaman/models/tester_iq4_single_gpu.sh`: target KV Q5_1/Q5_1, draft KV
Q4_0/Q4_0, draft maximum 2, batch 1024, microbatch 128, one slot, fixed 99 GPU
layers, and a 60 MiB fit target. Patched runs explicitly used
`--pipeline-parallel off`; unpatched mainline used its default scheduling because
that option is supplied by the patch.

| Backend | Previous stock | Current stock | Previous patched | Current patched | Current patch gain |
|---|---:|---:|---:|---:|---:|
| ROCm | 19,456 | 22,272 | 76,032 | **76,544** | **+54,272** |
| Vulkan | 68,352 | 69,120 | 78,592 | **79,616** | **+10,496** |

Every current run completed fitting, initialized the model, and reached the
server listening state. The patched results remain close to the previous build:
+512 tokens on ROCm and +1,024 on Vulkan. This confirms that context allocation
is consistent across the mainline update and that the patch remains necessary,
especially for ROCm.

### ROCm Q4_0 versus F16 MTP cache comparison

On 2026-08-10, the deployed patched binary and libraries from
`/home/eaman/llama/bin_vulkan` were tested on ROCm0 with the single-GPU IQ4_XS
launcher settings. The two runs differed only in the MTP draft K/V cache types.
Both completed fitting, loaded the model, and reached the server listening state.

| MTP draft KV | Fitted context |
|---|---:|
| Q4_0/Q4_0 | **76,544** |
| F16/F16 | 68,864 |

Q4_0/Q4_0 gained 7,680 context tokens, or 11.15% relative to F16/F16. On this
model and supported HIP VEC shape, quantized MTP KV therefore has a meaningful
context-capacity advantage. This startup test does not establish throughput,
draft acceptance, delayed-OOM behavior, or long-context stability.

### Current Q6_K_L dual-GPU cache comparison

Also on 2026-08-10, `/home/eaman/models/think.sh_test` was reproduced with the
deployed patched binary. The command used ROCm0 and ROCm1, fixed layer placement
with a 0.6/0.4 tensor split, a 20 MiB fit target, draft maximum 3, and pipeline
mode off. Only the two MTP draft-cache types changed.

| MTP draft KV | Fitted context | MTP KV on ROCm1 | MTP compute on ROCm1 | Refined MTP total |
|---|---:|---:|---:|---:|
| F16/F16 | **153,856** | 601.00 MiB | 163.70 MiB | 764.70 MiB |
| Q4_0/Q4_0 | **153,856** | 169.03 MiB | 163.96 MiB | 333.00 MiB |

Q4_0 reduced the MTP allocation on ROCm1 by 431.70 MiB but produced no fitted
context increase. The fixed layout is constrained by the per-device fit and the
MTP allocation resides on ROCm1; in this run, the saving became ROCm1 headroom
rather than changing the limiting 256-token-rounded context. Thus Q4_0 has no
context-capacity advantage over F16 for this Q6 dual-GPU configuration, although
it still materially reduces memory allocated on ROCm1.

#### Converting the ROCm1 headroom into context

Changing the whole-layer split from 0.60/0.40 to 0.59/0.41 moved approximately
300 MiB of model weights to ROCm1 but overshot the balance: fitted context fell
from 153,856 to 145,152. Layer placement is too coarse for this topology.

A target tensor-buffer override provided finer placement while retaining the
0.60/0.40 layer boundary. Moving the three FFN weights of boundary layer 39 to
ROCm1 used:

```text
-ot '^blk\.39\.ffn_(up|gate|down)\.weight$=ROCm1'
```

This moved 209.18 MiB of model storage from ROCm0 to ROCm1 and raised the fitted
context to **165,632**, a gain of 11,776 tokens (+7.65%) over Q4_0 without the
override. Startup completed and the server reached the listening state. This
configuration still requires controlled throughput, acceptance, delayed-OOM,
and long-context stability validation because fine-grained placement can add
cross-device traffic.

#### Applying the helper to `think.sh_small`

On 2026-08-10, the procedure in `dual_gpu_context_balancing_guide.md` was also
applied to `/home/eaman/models/think.sh_small`, using the Q6_K model and deployed
ROCm build 996 (`d2f8305`). With the original 0.60/0.40 layer split, F16/F16 and
Q4_0/Q4_0 both fitted **189,440** context. Q4_0 nevertheless reduced the refined
MTP allocation on ROCm1 from 929.76 MiB to 398.15 MiB, a 531.61 MiB saving that
the coarse placement could not convert into context.

The bounded placement search produced:

| Q4_0 placement | Context | Change | Target graph splits |
|---|---:|---:|---:|
| Original 0.60/0.40 | 189,440 | baseline | 35 |
| Whole-layer 0.59/0.41 | 187,392 | -2,048 | 35 |
| Boundary `ffn_up` | 193,280 | +3,840 | 37 |
| Boundary FFN triplet | 201,216 | +11,776 | 35 |
| FFN triplet plus attention Q/output | **205,312** | **+15,872 (+8.38%)** | 39 |

The max-context override is:

```text
-ot '^blk\.39\.(ffn_(up|gate|down)|attn_(q|output))\.weight$=ROCm1'
```

It moved 283.01 MiB from ROCm0 to ROCm1 and left the modeled device headroom
within approximately 23 MiB. The server reached the listening state, `/health`
returned OK, and a short real request completed at 27.09 generated tokens/s with
11/11 MTP draft tokens accepted. This single short request is only a functional
smoke test. The FFN-only override remains an attractive lower-complexity profile:
it retains the baseline 35 target graph splits while still gaining 11,776 tokens.
Comparative throughput, delayed-OOM, and long-context stability remain pending.

Logs:

```text
logs/iq4_backend_patch_comparison_20260810.log
logs/mtp_cache_q4_vs_f16_rocm_20260810.log
logs/q6_mtp_cache_q4_vs_f16_rocm_20260810.log
logs/q6_rocm_q4_tensor_rebalance_20260810.log
logs/q6_k_small_rocm_tensor_balance_20260810.log
```

#### Q8_0/Q8_0 target-KV calibration for `think.sh_test`

On 2026-08-11, the same manual placement procedure was repeated for
`/home/eaman/models/think.sh_test` after its target V cache changed from Q5_1 to
Q8_0. The matched trials used Q8_0/Q8_0 target KV, Q4_0/Q4_0 MTP KV, the
0.60/0.40 layer split, and a 20 MiB fit target. They began with 16,310 MiB free
on ROCm0 and 12,248 MiB free on ROCm1.

| Placement | Context | Model MiB ROCm0/ROCm1 | Target scheduler splits |
|---|---:|---:|---:|
| No override | 126,720 | 12,569.81 / 9,094.91 | 3 |
| Boundary `ffn_up` | 130,048 | 12,500.09 / 9,164.64 | 5 |
| Boundary FFN triplet | **136,448** | 12,360.63 / 9,304.09 | 3 |
| FFN triplet plus attention Q/output | 139,008 | 12,279.54 / 9,385.19 | 7 |

The selected Q8_0 override remains the boundary FFN triplet:

```text
-ot '^blk\.39\.ffn_(up|gate|down)\.weight$=ROCm1'
```

It moved 209.18 MiB from ROCm0 to ROCm1 and gained 9,728 tokens (+7.68%) over
the matched baseline while retaining 3 target scheduler graph splits. The
larger override gained another 2,560 tokens but raised those splits to 7, so it
was not selected without a controlled throughput comparison. A clean repeat of
the final launcher, with 32 MiB more initial ROCm0 headroom, fitted 137,984 and
reached the listening state. This is startup-capacity evidence only; throughput,
acceptance, near-context, delayed-OOM, and long-context testing remain pending.

Detailed measurements:
`logs/q6_kl_q8_target_tensor_balance_20260811.log`.

---

## TODO and Development Path

### Phase 0: Current stability and performance validation

- [x] Build HIP and Vulkan backends successfully.
- [x] Initialize the 27B Q6 model with F16 MTP KV and pipeline mode off.
- [x] Confirm 119,552 target and MTP context without allocation fallback.
- [x] Run an initial real prompt and confirm normal MTP generation.
- [x] Exercise the current build with the established old-prompt collection (user-confirmed for build 1180).
- [x] Test prompts that grow close to the fitted context limit.
- [x] Run several long generations and watch for delayed OOM, corruption, or scheduler errors.
- [x] Record repeated prompt-cache and generation runs for prompt speed, generation speed, and acceptance.
- [x] Perform a controlled same-build F16 `auto` versus `off` comparison at fixed context and seed.
- [x] Smoke-test representative models on Vulkan with pipeline mode `auto`.
- [x] Compare Q4_0 and F16 MTP KV on Vulkan before choosing a Vulkan default.

The selected production Q8_0/Q8_0 target-KV, Q4_0 MTP, VEC, and FFN-triplet
placement profile completed repeated 32K-token prompt runs and accumulated
approximately 129K slot tokens without truncation, allocation failure, or
logged corruption. Long generations reached approximately 8K output tokens.
The final runs used a stabilized GPU clock/voltage profile; an earlier apparent
decode failure coincided with excessive junction temperature and is not treated
as a patch or context-limit failure. These results establish functional
long-context stability for this profile, but not a formal throughput benchmark.
The matched baseline-versus-override performance comparison remains open.

#### Build 1180 controlled MoE pipeline and Vulkan comparison

On 2026-08-20, build 1180 (`5850e07c`) was tested with the dual-GPU
Qwen3.6-35B-A3B-UD-Q5_K_S MoE model at fixed 32,768 context, target KV
Q8_0/Q8_0, tensor split `0.572,0.428`, MTP draft maximum 4, a fixed 50-token
prompt and seed 424242, and 512 generated tokens. ROCm used
`GGML_CUDA_DISABLE_GRAPHS=1`; all runs used backend sampling disabled.

| Backend | MTP KV | Pipeline | Prompt t/s | Generation t/s | Draft acceptance |
|---|---|---|---:|---:|---:|
| ROCm | F16/F16 | off | 81.12 | 69.67 | 96.283% (259/269) |
| ROCm | F16/F16 | auto | 80.28 | 70.38 | 96.283% (259/269) |
| Vulkan | Q4_0/Q4_0 | auto | 159.40 | 41.45 | 97.133% (271/279) |
| Vulkan | F16/F16 | auto | 159.47 | 39.00 | 96.198% (253/263) |

Verbose startup evidence confirmed that ROCm `auto` enabled pipeline
parallelism for the target context while MTP remained disabled. Against `off`,
`auto` changed generation by +1.02% and prompt processing by -1.03%, too small
for a winner from one matched request. On Vulkan, Q4_0 improved generation by
6.27% over F16 with effectively identical prompt speed and 0.935 percentage
points higher acceptance in this sample. Q4_0 is the provisional Vulkan choice
for this model, pending repeated runs and a matched fitted-context comparison.
Raw logs are `logs/controlled_*_20260820.log`; the fixed request is
`logs/controlled_comparison_request_20260820.json`.

### Phase 1: Recurrent-state snapshot memory

The design investigation reached a validated boundary on 2026-08-20. An opt-in prototype was implemented and tested on 2026-08-21 in the isolated llama.cpp branch `eaman-rs-snapshot-depth`, with its linked worktree at `/home/eaman/llama/llama-rs-snapshot-depth`. Production branch `eaman` remains unchanged, and omitting the new option preserves the existing `n_rs_seq = draft.n_max` behavior.

The dense Q6 target context allocates 598.50 MiB of recurrent state for three rollback snapshots:

```text
n_rs_seq = 3
ROCm0 recurrent state = 374.06 MiB
ROCm1 recurrent state = 224.44 MiB
```

Planned work:

- [x] Compare the active MoE model with `--spec-draft-n-max 4` and 5 for context, acceptance, and tokens/second.
- [x] Determine the exact base-state and per-snapshot allocation.
- [x] Implement configurable GPU rollback depth with recomputation when rejection exceeds the stored depth on an isolated feature branch.
- [ ] Deferred: investigate lower-precision rollback snapshots only if the simpler rollback-depth design is insufficient.
- [x] Validate the existing rollback and rejection path while keeping the existing behavior as the default.
- [x] Fall back to host storage when an on-device recurrent checkpoint has a fragmented layout.
- [x] Add parser and fitter-input coverage for default devices, split-mode none, asymmetric targets, and reduced/full depth.
- [x] Validate two concurrent slots with independent on-device checkpoints on ROCm.
- [ ] Reduce redundant startup measurement work with a load-scoped scratch-fit cache; this is an efficiency follow-up, not a correctness requirement.
- [ ] Complete repeated dense-Q6 and MoE benchmarks plus long-prompt and delayed-OOM testing before considering integration into `eaman`.

The recorded full-fit contexts are 203,776 with MTP `n-max 4` and 195,584 with `n-max 5`, an 8,192-token capacity advantage for 4. In `/home/eaman/launch/bench/moe_large`, the initial results use `n-max 4`; later results switch to 5 only where that setting is explicitly labeled. The file records multiple model/configuration stages, so timings across those stage boundaries are not treated as one matched pair. The user concluded that 4 has the better overall tradeoff while 5 provides a small speculative improvement that may be more useful at long context; the active launcher therefore retains 5 for now.

#### Build 1180 recurrent-state allocation and rollback validation

On 2026-08-20, the single-GPU Qwopus3.5 9B Q6_K MTP model was tested on the RX 6800 with build 1180 (`5850e07c`). Controlled MTP-enabled startup runs changed only `--spec-draft-n-max`:

| `n_rs_seq` | Total recurrent state | R | S |
|---:|---:|---:|---:|
| 0 | 50.25 MiB | 2.25 MiB | 48.00 MiB |
| 1 | 100.50 MiB | 4.50 MiB | 96.00 MiB |
| 4 | 251.25 MiB | 11.25 MiB | 240.00 MiB |

The measured formula is `50.25 MiB * (1 + n_rs_seq)` with zero deviation in all three runs. Raw logs are `logs/qwopus9b_rs_n{0,1,4}_build1180_20260820.log`.

The dedicated `test-recurrent-state-rollback` binary reached its final dirty-context case, which means its clean full restore and `PARTIAL_ONLY` recurrent restore comparisons passed. The dirty full checkpoint restore then failed at replay position 6 with a token-0 logit mismatch (`7.30374` versus `6.80103`). This is separate from the server MTP path, which uses `PARTIAL_ONLY` target checkpoints. The failing test log is `logs/qwopus9b_recurrent_rollback_build1180_20260820.log`.

The operational server path was validated with a fixed 62-token prompt, greedy sampling, seed 1234, and 128 generated tokens. MTP `n-max 4` generated 181 draft tokens and accepted 81 (44.751%), exercising 100 rejected drafts. Its output was byte-identical to a target-only run with the same target settings. The logs are `logs/qwopus9b_mtp_rejection_n4_build1180_20260820.log` and `logs/qwopus9b_target_only_baseline_build1180_20260820.log`.

The minimum operational MTP setting was then tested with the same request. MTP `n-max 1` allocated 100.50 MiB of target recurrent state, generated 72 draft tokens, accepted 54 (75.000%), and exercised 18 rejected drafts. Its 128-token output was also byte-identical to the target-only result. The log is `logs/qwopus9b_mtp_rejection_n1_build1180_20260820.log`.

MTP `n-max 0` is valid for allocation-only startup but is not an operational server baseline. Its first request aborted at `GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max)` after both target and draft contexts selected `n_outputs_max = 1`; see `logs/qwopus9b_mtp_rejection_n0_build1180_20260820.log`. Use target-only mode as the zero-draft generation baseline.

The existing `n_rs_seq = draft.n_max` behavior is validated for real rejection rollback at both the minimum operational value 1 and the active value 4, and remains the default. A lower independent rollback limit cannot be introduced as a memory-only clamp: recurrent partial rollback currently fails when the requested rollback exceeds `n_rs_seq`. A safe design must either bound the verification batch to the available snapshots or provide a recomputation fallback.

#### Isolated Phase 1 prototype (2026-08-21)

The feature branch adds `--spec-mtp-rs-depth N` and the environment equivalent `LLAMA_ARG_SPEC_MTP_RS_DEPTH`. The accepted range is `1..--spec-draft-n-max`, independent of command-line argument order. When omitted, MTP retains the full draft maximum; EAGLE3, DFlash, and DSpark continue to request full depth even if combined with MTP. Deep rejection restores the existing pre-verification `PARTIAL_ONLY` checkpoint and replays the previously sampled token, accepted draft prefix, and replacement token. The existing direct recurrent rollback remains the fast path when the rejected suffix fits in the retained depth.

The server reports `draft_replay_count` and `draft_replay_n` in response timings and prints an MTP replay summary. Argument tests cover the default, both valid argument orders, zero, negative values, and depth above the draft maximum. The deterministic rollback matrix uses draft maximum 5, depth 1, and rejection after every accepted-prefix length from 0 through 4. On HIP with 20 target layers offloaded, all replacement and continuation logits were bit-identical to full-depth rollback. On Vulkan, every greedy replacement and continuation token matched; the largest logged absolute logit drift was 0.0189342. The executable subsequently reaches the pre-existing unrelated dirty full-checkpoint case; Vulkan reports its known dirty-context mismatch, while the two-context HIP run lacks VRAM for that final 2.7 GiB legacy context.

The combined build compiled and linked HIP, Vulkan, `llama-server`, `test-recurrent-state-rollback`, and `test-arg-parser`. The CPU/Vulkan argument test reaches all new cases, then fails only at the existing network download test in the restricted environment.

Matched RX 6800 ROCm measurements used the Qwopus3.5 9B Q6_K model, target Q8_0/Q8_0 KV, MTP Q4_0/Q4_0 KV, draft maximum 5, and backend draft sampling disabled:

| Run | RS buffer | Generation | Draft accepted | Deep replay |
|---|---:|---:|---:|---:|
| depth 5, 192 tokens | 301.50 MiB | 64.75 t/s | 140/246 | none |
| depth 1, 192 tokens | 100.50 MiB | 54.03 t/s | 129/202 | 20 events / 69 tokens |
| depth 5, 512 tokens | 301.50 MiB | 60.24 t/s | 366/718 | none |
| depth 1, 512 tokens | 100.50 MiB | 47.08 t/s | 311/616 | 75 events / 231 tokens |

The initial host-checkpoint depth-1 implementation saved 201.00 MiB, or 66.7% of the target recurrent-state buffer. The matched 192-token output was byte-identical. The 512-token depth-1 and depth-5 outputs diverged; the depth-5 result was byte-identical on repeat, but a target-only 512-token run also differed from both speculative results. Long-output byte identity is therefore not treated as a sufficient oracle for this existing MTP path. The feature remains opt-in and isolated pending repeated dense/MoE workloads and longer stability testing. Relevant logs use the `logs/phase1_rs_*_20260821` prefix.

#### Hardcoded on-device checkpoint proof of concept

Feature commit `cccc8c78` changes only recurrent speculative fallback checkpoints to `LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY | LLAMA_STATE_SEQ_FLAGS_ON_DEVICE`; other checkpoint types remain host-backed. This remains intentionally hardcoded for the experiment: there is no additional command-line flag. Feature commit `be535675` subsequently added automatic fit accounting for the lazy device allocation.

The deterministic depth-1 rejection matrix passed at every accepted-prefix length on both Vulkan and HIP. HIP replacement and continuation logits remained bit-identical to full-depth rollback. The later pre-existing dirty full-checkpoint case remains separate; Vulkan reaches its known mismatch, while the HIP two-context test lacks VRAM for the final 2.7 GiB legacy context.

On the same fixed 192-token RX 6800 9B request, device depth 1 generated identical content with the same 202 drafts, 129 accepted tokens, and 20 replay events / 69 replayed tokens. It improved generation from 54.03 to 60.39 t/s versus the host-checkpoint implementation (+11.76%) and remained 6.74% below the 64.75 t/s full-depth result. The device checkpoint allocated exactly one additional 50.25 MiB state group. Effective target recurrent storage was therefore 150.75 MiB versus 301.50 MiB at depth 5, retaining a 150.75 MiB (50%) saving for this draft-maximum-5 configuration. Logs are `logs/phase1_rs_depth1_device_rocm_9b_trace_20260821.{log,json}`.

The fixed-context dense draft-maximum-3 run completed at 136,448 tokens with `--fit off`. It generated 11,761 tokens at 26.50 t/s, accepted 8,009 of 8,733 draft tokens (91.710%), averaged 3.55 accepted tokens, and used 98 replay events / 288 replayed tokens. This is 10.60% faster than the earlier 23.96 t/s host-checkpoint depth-1 run and 5.12% below the 27.93 t/s normal depth-3 baseline. The device checkpoint therefore recovers most of the host-copy slowdown in this long dense workload as well as in the 9B test. The supplied timing record is `logs/phase1_rs_depth1_device_dense_fixed_20260821.log`.

For this dense model, one checkpoint group consumes 149.625 MiB across the two GPUs. Depth 1 plus that checkpoint uses 448.875 MiB versus 598.50 MiB at depth 3, retaining a net saving of 149.625 MiB (25% of recurrent allocation).

Feature commit `be535675` adds automatic fit accounting for the lazy on-device checkpoint. When MTP uses a recurrent depth below its draft maximum, server startup measures the target context at `n_rs_seq` and `n_rs_seq + 1`, takes only the per-device context-memory delta, and adds that one-group reservation to the fitter margin in the measured model-device order. Device-count/order changes, a missing fit target, decreasing memory, a zero total, or a measurement exception fail model loading instead of permitting a delayed OOM. Vulkan and combined HIP/Vulkan server builds both compiled and linked successfully with `-j 12`.

The successful dense `--fit-target 20` run measured and reserved 93.52 MiB on ROCm0 plus 56.11 MiB on ROCm1, 149.62 MiB total, and fitted 149,504 context tokens. Final allocation reached the listening state with 114/124 MiB reported free on ROCm0/ROCm1. A deterministic 192-token request triggered the lazy device checkpoint allocation at the same 93.516/56.109 MiB split and completed without OOM at 23.15 t/s. It accepted 103/112 draft tokens (91.964%) and exercised one deep-rejection replay event / three replayed tokens. The server then shut down cleanly and both GPUs released their allocations. Logs are `logs/phase1_rs_device_fit_dense_success_20260821.{log,json}`; the earlier occupied-port attempt remains `logs/phase1_rs_device_fit_dense_20260821.log` for historical context.

#### Fragmented-checkpoint and multi-slot hardening

Feature commit `732dd501` converts fragmented on-device recurrent checkpoint
saves from an abort into a catchable failure. The common checkpoint wrapper
retries with host storage and records the actual storage mode for restore. A
forced fragmented whole-cache fixture validates this fallback. Normal positive
per-slot recurrent allocation remains contiguous, so the fallback is defensive
for unusual, malformed, or future layouts rather than an expected `-np > 1`
path.

The same commit expands parser and fitter-input tests across fit disabled,
default margins, split-mode none, asymmetric targets, reduced/full depth,
combined EAGLE3, and dynamic two-device argument order. Vulkan rollback testing
still passes every accepted-prefix position from 0 through 4; only the later
known dirty full-checkpoint case differs. Clean build 1184 (`732dd501`) compiled
and linked `llama-server`, `test-arg-parser`, and the rollback test in both the
Vulkan-only and combined HIP/Vulkan configurations.

A real build-1184 ROCm `-np 2` run used two concurrent Qwopus3.5 9B requests.
Each slot allocated an independent 50.250 MiB device checkpoint. Both completed
192 generated tokens without OOM or host fallback. Slot 0 achieved 35.99 t/s,
accepted 118/222 drafts, and replayed 90 tokens in 27 events; slot 1 achieved
28.73 t/s, accepted 91/283 drafts, and replayed 123 tokens in 44 events. The raw
capture lacks the original startup fitter section, so 100.50 MiB is the measured
total lazy allocation and only an inferred fitter reservation, not a captured
reservation line. See
`logs/phase1_rs_hardening_np2_rocm_9b_summary_20260821.log` and its referenced
raw log/responses.

An audit of the extra startup measurements found no safe local pass deletion:
the D, D+1, MTP-context, and refinement measurements have distinct inputs.
Reducing repeated baseline/minimum-context work requires an explicit
load-scoped scratch-fit cache with a complete cache key; the final fit should
remain fresh. This optimization is deferred because it affects startup time,
not runtime correctness or memory safety. Rebasing is also deferred until the
production `eaman` base advances; the feature branch is already based on the
current `eaman` tip.

#### Exported `a3b1eff` test patches (2026-08-22)

Two experimental standalone exports now support controlled testing without changing either source branch:

- `latest_rs_cumulative_a3b1eff.patch` applies directly to `a3b1eff` and contains the complete Eaman ROCm/MTP/VEC/MoE work plus recurrent-state rollback through `732dd501`. Clean application produced tracked-tree hash `80b60d13ccc56bd3ea975c54ecb2461dc373bbc0`, exactly matching the feature branch tip. SHA-256: `870cae983d962cb250d9e9b347cd3d51b58bcc12a5b7d07063fee81f3e4876a4`.
- `mtp_rs_only_a3b1eff.patch` applies directly to pristine `a3b1eff` and contains only the four-commit recurrent rollback delta (`e04c6d9b`, `cccc8c78`, `be535675`, and `732dd501`). It excludes the cumulative Eaman fitter, pipeline-control, HIP VEC, and MoE scratch-placement changes. Clean application and `git diff --check` passed; CPU/default build 1171 linked `llama-server`, `test-arg-parser`, and `test-recurrent-state-rollback`, and `test-arg-parser` passed. The model-dependent rollback executable and isolated GPU runtime were not rerun. SHA-256: `2aa8149d703d730e57b0b8dd4acef62bbef80e5e0baac00b9d55c7bfeb19c3b8`.

Both files are pinned test artifacts for the exact `a3b1eff` base. Any upstream submission should first rebase the RS-only change onto a freshly fetched upstream `master`, then rebuild and repeat the focused GPU/model validation.

#### Original deferred-work report and resume point

The objective was to keep a useful MTP draft maximum while reducing the GPU memory occupied by target recurrent rollback snapshots. The allocation and correctness questions are resolved, but the implementation is not a small allocator change.

Measured facts:

- Recurrent memory contains one live state plus `n_rs_seq` rollback snapshots and scales exactly with `1 + n_rs_seq`.
- The dense Q6 configuration with `n_rs_seq = 3` uses 598.50 MiB: 374.06 MiB on ROCm0 and 224.44 MiB on ROCm1. One state group is 149.625 MiB total, split as approximately 93.515 MiB on ROCm0 and 56.110 MiB on ROCm1.
- Reducing stored depth from 3 to 1 while retaining draft maximum 3 would save approximately 299.25 MiB total. The measured dense-Q6 placement slope estimates approximately 8,704 more context tokens, moving the normal 136,448-137,984 fit to roughly 145,152-146,688.
- At draft maximum 5, reducing stored depth from 5 to 1 would save approximately 598.50 MiB total and an estimated 17,408 tokens compared with unmodified `n-max 5`. Relative to the usual unmodified `n-max 3` launcher, the expected final context gain remains approximately 8,704 because unmodified `n-max 5` first spends two additional snapshots.
- These are fit estimates, not measured final contexts. Actual fitting is rounded to 256 tokens and varies with initial free VRAM and per-device placement.

Correctness boundary:

- If five draft tokens are verified together and token 3 is rejected, the correct target state is the state after the two accepted tokens. A depth-1 snapshot cannot restore that state directly from the end of the verification batch.
- Clamping the rollback distance to the stored depth is incorrect and can corrupt subsequent output.
- Restricting verification batches to the stored depth is simple but imposes a regular speed penalty and removes much of the benefit of larger draft maxima.
- The preferred design is a recomputation fallback: retain the current fast path while rejection fits within the stored snapshots; otherwise restore the pre-verification `PARTIAL_ONLY` checkpoint and replay the accepted prefix. Accepted output tokens are preserved, but their recurrent state is recomputed.
- The expected average slowdown may be small at the observed 95%-99% production acceptance, but this must be measured. Deep rejection frequency, not only aggregate acceptance, determines the fallback cost.

Relevant source map:

- `common/common.h`: `need_n_rs_seq()` currently returns `draft.n_max` for MTP and related draft implementations.
- `src/llama-memory-recurrent.cpp`: tensor widening uses `1 + n_rs_seq`; `seq_rm()` rejects rollback distances greater than `n_rs_seq`.
- `tools/server/server-context.cpp`: the hardcoded proof of concept uses `PARTIAL_ONLY | ON_DEVICE` for recurrent speculative target checkpoints around verification and rollback.
- `tests/test-recurrent-state-rollback.cpp`: existing full, partial, and dirty-context checkpoint coverage.

Known separate edge cases:

- MTP `n-max 0` is not a useful operational setting. It starts and measures allocations but aborts on its first server request. Normal MTP use starts at 1; target-only mode is the zero-draft baseline.
- The dedicated rollback test passes clean full restore and the server-relevant `PARTIAL_ONLY` restore, then fails when a full checkpoint is loaded over a dirty hybrid context. Do not use that full dirty-restore path as the proposed fallback without resolving it. The current server MTP path does not use it.

Resume checklist:

1. Define an opt-in rollback-depth parameter with range `1..draft.n_max`; preserve `draft.n_max` as the default so existing behavior does not change.
2. Specify the recomputation state machine before editing code: checkpoint position, accepted-prefix replay, replacement token handling, target attention-cache rollback, draft-context synchronization, and statistics accounting.
3. Add deterministic tests for `n-max 5` with depth 1 and rejection at every draft position. Compare tokens and logits against full-depth behavior across repeated speculative rounds and prompt-checkpoint restoration.
4. Build HIP and Vulkan, then repeat the 9B single-GPU rejection test before testing the dense Q6 dual-GPU fit and throughput.
5. Measure context, tokens per second, deep-rejection frequency, replay count, and acceptance on matched dense Q6 and MoE workloads. Keep the feature opt-in unless the memory gain clearly outweighs the measured slowdown.

The prototype portion is complete. Focused dense/MoE performance validation and long-context confidence testing remain before integration.

### Phase 2: MTP placement and backend overhead

- [x] Measure fine-grained target placement for the boundary FFN tensors; the
  FFN-triplet override is the preferred lower-complexity candidate because it
  preserves the baseline graph-split count.
- [x] Retain the FFN-triplet override as the production choice. The attention-inclusive
  override adds only 2,560 context tokens while increasing target graph splits from
  3 to 7, so no further comparison is planned unless priorities change.
- [ ] Measure free VRAM and MTP tensor placement on both GPUs after long runs.
- [ ] Investigate whether selected NextN tensors or MTP workspace can use the less constrained GPU.
- [x] Test `--no-spec-draft-backend-sampling` to quantify the ROCm sampling fallback overhead.
- [ ] Separate context-capacity changes from performance-only backend changes.

#### Build 1180 ROCm draft-backend sampling comparison

On 2026-08-20, the single-GPU Qwopus3.5 9B Q6_K MTP model was tested on the RX 6800 with explicit `--spec-draft-backend-sampling` and `--no-spec-draft-backend-sampling`. Each mode received one 64-token warm-up followed by the same three 512-token requests with seeds 4101-4103, MTP `n-max 4`, Q4_0/Q4_0 draft KV, and identical temperature, top-k, top-p, and penalty settings.

| Draft backend sampling | Mean generation t/s | Median generation t/s | Draft acceptance |
|---|---:|---:|---:|
| enabled | 66.98 | 65.43 | 95.187% (890/935) |
| disabled | 66.13 | 65.22 | 95.187% (890/935) |

Disabling the option changed mean throughput by -1.27% and median throughput by -0.33%, which is below run-to-run noise. All three matched outputs were byte-identical. With backend sampling requested, ROCm reported that `ROCm0` does not support the `TOP_K` operation required by the draft top-k sampler; disabling the option removed that warning and the extra sampler graph reservation, but produced no measurable generation benefit in this workload. Keep the active launcher setting unchanged where it is already disabled, but do not claim a general ROCm speedup from this 9B result. Raw logs are `logs/qwopus9b_spec_backend_sampling_{on,off}_build1180_20260820.log`.

### Phase 3: Native quantized HIP flash attention

This remains the most valuable long-term backend improvement:

- [ ] Identify the exact HIP flash-attention kernels selected for target and MTP graphs.
- [ ] Measure F16 K/V conversion allocations independently.
- [ ] Prototype direct q8_0 K support.
- [ ] Add q5_1 V and q4_0 MTP support if the first kernel is successful.
- [ ] Validate long-context numerical accuracy and performance against F16.

Native quantized kernels could remove the main reason F16 MTP KV can use less
total ROCm memory than Q4_0. The cache choice is currently model-, backend-,
and topology-dependent: Q4_0 wins on the tested single-GPU IQ4_XS case, while
the dual-GPU Q6_K_L case shows equal fitted context with substantially lower
Q4_0 MTP allocation. Do not promote F16 or Q4_0 to a universal default until
the corresponding runtime and Vulkan comparisons are complete.

---

## 2026-08-15 Mainline Refresh

The dedicated `latest` branch was fast-forwarded to upstream `master` commit
`5f754ea`. The previously validated `77918ca` patch was committed and merged
into the durable integration branch first, then `latest` was merged as `d00e998`.
That integration branch was originally named `sol` and is now named `eaman`.
Recovery branch `eaman-pre-latest-5f754ea` preserves pre-merge tip `e5c647b`.
Older recovery branch names `sol-pre-latest-d2f8305` and
`sol-pre-latest-77918ca` were pruned after their tips were verified as ancestors
of the integration branch; their commits remain in `eaman` history.

The persistent Release build enabled HIP, Vulkan, all quantized flash-attention
instances, and HIP architectures `gfx1030;gfx1031`. HIP and Vulkan libraries
compiled, and the final `llama-server` linked successfully. The fresh binary
reports llama.cpp build 1093 at commit `d00e998`. A separate CPU/default
`llama-server` build also completed in a temporary detached worktree.

The standalone patch is:

```text
/home/eaman/llama/bug/latest_rocm_improvement_5f754ea.patch
SHA-256: 5b653a6aaa50fd30cfc05cb9cf50b7a75adc491395c285b587f7e5b237b847c9
```

After the 2026-08-18 MoE integration, it applies cleanly without three-way
fallback to `5f754ea`. Its applied tree is
`0f28cb047c374d20ac95e9912167eec13ad0caf1`, exactly matching committed `eaman`
tip `773364c`. The temporary validation worktree was removed. The earlier
mainline refresh itself performed no runtime benchmark; the later MoE runtime
results are recorded in their dedicated section below.

## 2026-08-20 Mainline Refresh

The dedicated `latest` branch was fast-forwarded to upstream `master` commit
`a3b1eff`. The previous validated Eaman state at `773364c` was preserved as
`eaman-pre-latest-a3b1eff`, then `latest` was merged into `eaman` without
conflicts as `5850e07`.

The persistent Release build used HIP architectures `gfx1030;gfx1031` with
`GGML_HIP=ON`, `GGML_VULKAN=ON`, `GGML_CUDA_FA_ALL_QUANTS=ON`, and
`LLAMA_BUILD_SERVER=ON`. The HIP and Vulkan targets compiled, the final
`llama-server` linked, and the fresh binary reports build 1180 at commit
`5850e07c` on ROCm `7.14.60850-0000000` / Vulkan.

The standalone patch is:

```text
/home/eaman/llama/bug/latest_rocm_improvement_a3b1eff.patch
SHA-256: 55812a942789e7e8e234cb01c87bc00db75208e3caad4feb408e6b4d70ac275f
```

It applies cleanly without three-way fallback to `a3b1eff`. Its applied tracked
tree is `1eeaabc8556036d82687956e582cefccfb890d63`, exactly matching committed
`eaman` tip `5850e07`. The temporary detached validation worktree was removed.
After verifying ancestry, the older recovery branch name
`eaman-pre-latest-5f754ea` was pruned; `eaman-pre-latest-a3b1eff` is retained.
No runtime benchmark or binary deployment was performed by this refresh.

## 2026-08-23 Mainline Refresh

The dedicated `latest` branch already matched fetched upstream `master` commit
`8144f319`. Production tip `5850e07` was preserved as
`eaman-pre-latest-8144f31`, then `latest` was merged into `eaman` as `a4e47992`.
The sole conflict was the old server-side MTP estimator/refinement block. It was
removed in favor of upstream's joint target/draft fitter, which measures the
extra context again whenever the fitted target context changes and does not
publish provisional placement into `params_base`. This makes the old MoE
scratch-copy workaround unnecessary. The residual Eaman fit adapts fixed-layout
per-device interpolation to upstream's total `n_streams` context accounting and
forces the fitted MTP extra context to use the same disabled pipeline mode as
runtime MTP.

The persistent Release build used HIP architectures `gfx1030;gfx1031` with
`GGML_HIP=ON`, `GGML_VULKAN=ON`, `GGML_CUDA_FA_ALL_QUANTS=ON`, and
`LLAMA_BUILD_SERVER=ON`. HIP, Vulkan, all quantized flash-attention instances,
and the final `llama-server` target built successfully. The fresh binary reports
build 1243 at `a4e47992` on ROCm `7.14.60850-0000000` / Vulkan, and
`test-arg-parser` passed.

Dense Q6_K_L joint fitting measured the MTP extra context at 262,144 and 4,096,
selected 136,704 context through the fixed-layout per-device path, and reached
listening without target-only fallback. This retained the production launcher's
manual layer-39 FFN tensor override to ROCm1. At the same reported starting free VRAM
(16,310/12,248 MiB), the historical matched placement trial fitted 136,448; the
separate 137,984 startup had 32 MiB more free on ROCm0, and its rendering/session
mode was not recorded. A 64-token request completed at 27.03 t/s with 41/44
drafts accepted (93.18%).

MoE Q5_K_S joint fitting measured the MTP extra context at 262,144, 4,096, and
the selected 233,216 context without falling back to target-only fitting. Final
automatic fine placement retained tensor-level overrides: ROCm0 received 25
layers with one partial `LAYER_FRACTION_UP` layer and ROCm1 received 17 full
layers. It reached listening and a 64-token request completed at 76.28 t/s with
43/47 drafts accepted (91.49%). Target and MTP pipeline modes were disabled in
both runs. Runtime `auto` and `on` were not repeated. These are startup and short
generation checks, not long-context stability or formal throughput benchmarks.
Logs are `logs/mainline_8144f31_dense_mtp_startup_20260823.log`,
`logs/mainline_8144f31_dense_mtp_request_20260823.log`, and
`logs/mainline_8144f31_moe_mtp_validation_20260823.log`, with adjacent response
JSON files.

The standalone patch is:

```text
/home/eaman/llama/bug/latest_rocm_improvement_8144f31.patch
SHA-256: dc266cba88936c9e6db596dca897e4b7470c51cc9f3597c93026d967512ed6f5
```

It applies cleanly without three-way fallback to `8144f319`. Its applied
tracked-tree ID is `673cdbe49ed18722a1bee022ac8d4a0cfedaf69a`, exactly matching
committed `eaman` tip `a4e47992`. The temporary validation worktree was removed.
The source `master` branch remains untouched at `77918caf`; no binary was
deployed to `/home/eaman/llama/bin_vulkan`.

### Rebased recurrent rollback validation

The isolated `eaman-rs-snapshot-depth` branch was rebased onto production
`a4e47992` without merging it into production. Recovery branch
`eaman-rs-snapshot-depth-pre-8144f31` preserves old tip `732dd501`. The four
rebased commits are `12938d2e`, `237d104a`, `b4764e61`, and tip
`2eb3b94f`; tracked-tree ID is
`53a2c3af96bdafe1d593760c1c446859f030d617`.

CPU/default and combined HIP/Vulkan Release builds passed at build 1247.
`ggml-hip`, `ggml-vulkan`, all quantized `fattn-vec` instances, and the
final server linked. `test-arg-parser` passed, and registered generated-model
tests passed for Qwen3.5 and Nemotron-H, including fragmented device-save host
fallback and depth-1 recomputation at every rejection position.

Matched real-model results:

| Model/configuration | Full depth | Depth 1 | Difference |
|---|---:|---:|---:|
| Dense Q6_K_L, n-max 3, target 20 | 136,704 | 149,504 | +12,800 (+9.36%) |
| MoE Q5_K_S, n-max 4, target 50 | 233,216 | 262,144 | +28,928 (+12.4%) |

The dense seeded 192-token responses were byte-identical, accepted 105/105
drafts, and ran at 24.79 versus 24.12 t/s for full/depth-1. Dense depth 1
directly reserved 93.52 MiB on ROCm0 plus 56.11 MiB on ROCm1. The MoE depth-1
run reserved 43.97/18.84 MiB, reached the model's 262,144-token ceiling, and
completed 192 tokens at 71.30 t/s with 111/119 accepted drafts plus 2 replay
events / 5 replayed tokens. At fixed 233,216 context, seeded MoE full/depth-1
and target-only outputs all diverged at the same late position; this isolates
the observed long-output non-identity from the rollback path and is consistent
with the existing real-GPU reproducibility limitation.

A directly captured one-GPU `-np 2` startup reserved the incremental 100.50
MiB checkpoint group before joint fitting. The final two-sequence target RS
buffer was 201.00 MiB because the ordinary 100.50 MiB recurrent group is
already part of the baseline. Both slots retained the full 262,144 context.
Two explicitly concurrent 192-token requests launched before either finished,
completed without OOM or cross-talk, and replayed 1 event / 2 tokens and 7
events / 21 tokens. A separate 1,024-token 9B depth-1 run completed at 62.94
t/s, accepted 523/612 drafts, and exercised 24 replay events / 69 tokens.

The new experimental artifacts are:

```text
mtp_rs_only_8144f31.patch
SHA-256: e8a5e296b2f77e5103159ca0f1b9c88795aa1ea7fb6bb69516448cdb14d2f611

latest_rs_cumulative_8144f31.patch
SHA-256: c1a8305d506f5f1cd7318eb82499c9e91cc5a2ced92d6e54b0db643b894184ae
```

Both apply cleanly without `--3way` to exact base `8144f319`. The RS-only
application produces Eaman-free synthetic tree
`ec6c607a78b20e68125ee3a6067736f305868b15`; the cumulative application
produces `53a2c3af96bdafe1d593760c1c446859f030d617`, exactly matching the RS
branch. Temporary validation worktrees were removed. The feature remains
experimental and opt-in; startup scratch-fit caching and near-limit
long-prompt/delayed-OOM validation remain follow-up work.

## Current Patch and Validation Status

- `latest` (`8144f319`) was merged into `eaman` as `a4e47992`.
- Upstream's joint fitter replaces the old server estimator/refinement and its
  MoE scratch-placement workaround; dense and MoE MTP joint fitting both passed.
- The current `eaman` tracked-tree hash is `673cdbe49ed18722a1bee022ac8d4a0cfedaf69a`,
  exactly matching the standalone patch applied to `8144f319`.
- Recovery branch `eaman-pre-latest-8144f31` preserves pre-merge tip `5850e07`.
- `sol4.patch` remains the historical complete patch based on `7bd8282`.
- The persistent HIP/Vulkan build compiled both backends and linked build 1243.
- The real source `master` branch was not modified; temporary validation worktrees were removed.
- Dense and MoE production short generation passed. The isolated RS branch also
  passed matched dense/MoE, concurrent two-slot, and 1,024-token replay tests.
  Near-limit long-prompt, delayed-OOM, controlled throughput, and production
  runtime pipeline `auto`/`on` testing remain pending.

## Scope and Limitations

- Production allocation is validated for the stated dense 27B and MoE 35B
  models and ROCm topology.
- Multiple short requests and one 1,024-token RS request completed, but
  near-context-limit stability testing is still in progress.
- The performance improvement is based on one non-controlled comparison and must not yet be treated as a formal benchmark.
- The patch preserves `auto` as the default target pipeline policy.
- F16 is the best validated MTP cache for this ROCm model; it is not assumed to be optimal on Vulkan or other architectures.
- The per-device fitting path applies when GPU layers are explicitly fixed. Automatic placement retains the existing conservative path.

---

## 2026-08-18 Qwen3.6-35B-A3B MoE MTP fitter investigation

The Qwen3.6-35B-A3B Q5_K_S model exposed an ordering bug in the server's MTP
pre-fit path. The preliminary target fit wrote its generated tensor split and
tensor buffer overrides directly into `params_base`. Refinement and final fit
passes then treated that provisional placement as user-supplied and failed with
`model_params::tensor_split already set by user`. Context reduction survived
the exception, but the final target loaded with the wrong MoE placement.

The fix committed as `773364c` (also available independently in
`mtp_fit_scratch_placement.patch`) runs every MTP
estimation fit against a scratch copy of `common_params`. Only the final target
fit publishes generated placement. This keeps the dense-model fitting logic
unchanged and lets the final MoE fit account for the device-local MTP load.

Test system and launch-relevant settings:

- ROCm0 RX 6800: 16,368 MiB total, 16,342 MiB initially free.
- ROCm1 RX 6700 XT: 12,272 MiB total, 12,248 MiB initially free.
- Target KV Q8_0/Q8_0, MTP KV Q4_0/Q4_0, batch 1024, one slot,
  target pipeline parallelism off, no warmup.
- Model: Qwen3.6-35B-A3B-UD-Q5_K_S.

Measured results:

| Configuration | Runtime context | Prompt/generation result |
|---|---:|---|
| Unpatched auto `--fit-target 50` | 235,008 | 12,309-token prompt OOM on ROCm1 |
| Explicit `-c 228000 --tensor-split 0.572,0.428` | 228,096 | 32,327 prompt tokens + 16 generated tokens passed |
| Explicit `-c 232000 --tensor-split 0.572,0.428` | 232,192 | 12,308-token prompt OOM on ROCm0 |
| Patched auto `--fit-target 150` | 220,672 | 32,327 prompt tokens + 16 generated tokens passed |
| Patched auto `--fit-target 100` | 227,840 | 32,327 prompt tokens + 16 generated tokens passed |
| Patched auto `--fit-target 50` | 235,008 | 32,327 prompt tokens + 16 generated tokens passed; separate 12,308 + 64 test also passed |

The patched target-50 fit completed all four fit passes successfully and loaded
13,797.79 MiB of model buffers on ROCm0 plus 10,031.37 MiB on ROCm1. The
unpatched run froze an earlier 13,620.73/10,208.43 MiB placement and OOMed.
This also shows why a simple tensor split is insufficient: `0.572,0.428`
crossed a discrete layer boundary and loaded 14,153.92/9,675.23 MiB; the fitter
uses tensor overrides to find an intermediate, MTP-aware placement.

The HIP/Vulkan `llama-server` compiled and linked successfully in the detached
temporary worktree. The patch is committed on source branch `eaman` as `773364c`,
is included in `latest_rocm_improvement_5f754ea.patch`, and the validated full
`build/bin` output is deployed to
`/home/eaman/llama/bin_vulkan`. The previous deployment is retained at
`/home/eaman/llama/bin_vulkan.pre-moe-fit-20260818`. The active
`/home/eaman/launch/moe_tester.sh` now uses no explicit context or placement,
`--fit-target 50`, and MTP n-max 4; its previous version is retained beside it
as `moe_tester.sh.pre-moe-fit-20260818`. Logs are under `logs/moe_*`.

### Delayed ROCm graph-update crash and MoE-only workaround

A production run later reached 96,484 cached tokens, restored/reused that prompt,
and generated at least 6,632 more tokens before receiving SIGSEGV. The core
identified LWP 44707 as the faulting thread. Its stack was
`hipGraphExecUpdate` -> `ggml_cuda_graph_evaluate_and_capture` ->
`ggml_backend_cuda_graph_compute` -> `llama_decode`. The signal was
`SEGV_MAPERR`; no OOM frame or prompt-checkpoint frame was present. This is a
separate delayed ROCm graph-executable update failure, not the original MTP
auto-fit placement bug.

The runtime environment variable `GGML_CUDA_DISABLE_GRAPHS=1` bypasses the
failing HIP graph capture/update path without changing the binary, fitter,
generated tensor split/overrides, MTP, or other launchers. It is exported only
by `/home/eaman/launch/moe_tester.sh`, so dense-model processes retain graphs.

With that variable and the unchanged patched `--fit-target 50` configuration,
the fitter again selected 235,008 context and the same 13,797.79/10,031.37 MiB
model placement. Validation ingested 32,318 tokens and generated 4,718 tokens
to EOS at 41.73 t/s. A second request restored the 81.180 MiB prompt checkpoint
and completed a forced 7,000-token generation at 47.31 t/s. The combined 11,718
generated tokens completed without a crash. This is a controlled regression
test of the observed failing path, not yet a full 235k-context stability claim.

A subsequent production run with the same MoE-only graph disable restored a
94,446-token checkpoint, ingested 49,401 new tokens into a 143,851-token prompt,
and generated 1,698 tokens at 30.26 t/s before EOS. Prompt ingestion averaged
102.94 t/s and the slot released cleanly at 145,550 tokens. The immediately
preceding request on the same process generated 10,385 tokens at 40.86 t/s and
released cleanly at 104,837 slot tokens. This exceeds the original delayed
failure's generation exposure and validates checkpoint restore plus long-prompt
ingestion in production, while still not claiming the full 235k allocation.

The final target-50 stress request went substantially further. It restored a
197,327-token checkpoint, processed a 203,372-token prompt (6,045 prompt-eval
tokens at 58.80 t/s), and generated 11,145 tokens at 25.78 t/s. Draft
acceptance was 99.252% (8,623/8,688), with mean accepted length 4.70. The slot
released cleanly at 214,518 tokens, 20,490 below the fitted 235,008 limit, with
no OOM or segfault. This is the strongest long-context production validation
of the MoE-only HIP-graph workaround so far.

Evidence:

- `logs/moe_hip_graph_exec_update_segv_coredump_info.log`
- `logs/moe_patched_auto_fit50_no_hip_graphs_success.log`
- `logs/moe_no_hip_graphs_production_145k_success.log`
- `logs/moe_no_hip_graphs_production_214k_success.log`

---

## Conclusion

`sol4.patch` aligns MTP estimation with runtime allocation, fits the fixed two-GPU layout against real per-device limits, and makes target pipeline scheduling controllable. Disabling target pipeline parallelism for the tested single-slot workload reduced scheduler copies and compute workspace while increasing fitted context from 104,704 to **119,552 tokens**.

The immediate priority is stability testing with existing prompts and a controlled performance A/B. If those tests remain clean, recurrent-state snapshot memory is the next practical code path to investigate, followed by MTP placement and native quantized HIP flash-attention kernels.