Current ROCm MTP status

Use for the validated F16/off configuration:

  --pipeline-parallel off
  --cache-type-k-draft f16
  --cache-type-v-draft f16

The current complete patch is:

  bug/sol4.patch

It includes the MTP fitting and pipeline changes plus the merged HIP VEC dispatch from vec_sol. rocm_improvement.patch is an identical duplicate of sol4.patch.

The VEC Q4/Q4 MTP run initially reached 147,712 context tokens and completed startup plus short generation. Long-prompt stability, delayed-OOM, throughput, and draft-acceptance testing remain pending.

Validation notes:

- latest mainline 8144f319 was merged into eaman as a4e47992 with one resolved
  server conflict. Upstream's joint target/MTP fitter replaces the obsolete
  server estimator/refinement and its MoE scratch-placement workaround.
- The standalone patch applies cleanly without three-way fallback and produces
  the exact eaman tree 673cdbe49ed18722a1bee022ac8d4a0cfedaf69a.
- Rollback branch: eaman-pre-latest-8144f31 at 5850e07.
- The persistent Release build compiled and linked HIP, Vulkan, and
  llama-server successfully for gfx1030 and gfx1031 as build 1243.
- `test-arg-parser` passed. Dense Q6_K_L joint fitting selected 136,704 context
  at the same reported starting VRAM as the historical 136,448 matched trial,
  then completed 64-token MTP generation at 27.03 t/s with 41/44 drafts
  accepted. The separate historical 137,984 startup had 32 MiB more free on
  ROCm0, and its rendering/session mode is not established.
- MoE Q5_K_S joint fitting measured the MTP extra context at every candidate
  without target-only fallback, selected 233,216 context, reached listening,
  and completed 64-token generation at 76.28 t/s with 43/47 drafts accepted.
  Automatic tensor-level placement remained active (25 ROCm0 layers with one
  partial UP layer, plus 17 full ROCm1 layers).
- Target and MTP pipeline modes were disabled in both runtime checks. Runtime
  `auto` and `on` were not repeated during this refresh.
- The source master branch was not modified; the validation worktree was removed.

Historical d2f8305 IQ4_XS single-16GB-GPU fit results:

  patched ROCm:      76,544
  mainline ROCm:     22,272
  patched Vulkan:    79,616
  mainline Vulkan:   69,120

Matched patched-ROCm MTP cache test on 2026-08-10:

  Q4_0/Q4_0 draft KV: 76,544
  F16/F16 draft KV:   68,864
  Q4_0 gain:          7,680 tokens (+11.15%)

Both reached the listening state. This confirms a meaningful context-capacity
advantage for Q4_0 on this supported HIP VEC model/shape, but generation,
throughput, acceptance, delayed-OOM, and long-context stability remain untested.

Matched current Q6_K_L dual-GPU ROCm cache test using think.sh_test:

  Q4_0/Q4_0 draft KV: 153,856 context, 333.00 MiB MTP on ROCm1
  F16/F16 draft KV:   153,856 context, 764.70 MiB MTP on ROCm1

Q4_0 saved 431.70 MiB on ROCm1 but gained no context because the fixed
per-device layout retained the same limiting 256-token-rounded fit. The cache
choice is therefore model/topology dependent: meaningful context gain on the
single-GPU IQ4_XS test, none on this dual-GPU Q6_K_L test.

The Q6 Q4 ROCm1 headroom can be converted into context with a fine-grained
target tensor override while retaining --tensor-split 0.6,0.4:

  -ot '^blk\.39\.ffn_(up|gate|down)\.weight$=ROCm1'

This reached 165,632 context, +11,776 (+7.65%) over the 153,856 Q4 baseline.
A whole-layer 0.59/0.41 split was worse at 145,152. The override has completed
startup only; throughput, acceptance, delayed-OOM, and long-context stability
remain pending.

The 2026-08-11 Q8_0/Q8_0 target-KV recalibration of think.sh_test selected the
same boundary FFN triplet after measuring it independently:

  -ot '^blk\.39\.ffn_(up|gate|down)\.weight$=ROCm1'

With matched free VRAM, no override fitted 126,720 and the selected override
fitted 136,448 (+9,728, +7.68%) while retaining 3 target scheduler graph splits.
Adding attention Q/output reached 139,008 but raised those splits to 7, so it
was not selected without throughput validation. A clean final-launcher repeat
fitted 137,984 and reached the listening state. Detailed measurements:
bug/logs/q6_kl_q8_target_tensor_balance_20260811.log

The helper was also applied to /home/eaman/models/think.sh_small with the Q6_K
model. Its original Q4 and F16 layouts both fitted 189,440 context even though
Q4 saved 531.61 MiB of MTP allocation on ROCm1. The max-context override is:

  -ot '^blk\.39\.(ffn_(up|gate|down)|attn_(q|output))\.weight$=ROCm1'

It fitted 205,312 context, +15,872 (+8.38%), and passed startup, health, and one
short MTP request. It increases target graph splits from 35 to 39. The FFN-only
alternative reaches 201,216 while retaining 35 graph splits:

  -ot '^blk\.39\.ffn_(up|gate|down)\.weight$=ROCm1'

Detailed measurements: bug/logs/q6_k_small_rocm_tensor_balance_20260810.log

All four reached the server listening state. Against the corrected historical
results, patched ROCm changed from 76,032 to 76,544 and patched Vulkan from
78,592 to 79,616. The patch remained necessary on that tested mainline.

Standalone current-mainline patch:

  bug/latest_rocm_improvement_8144f31.patch
  base 8144f31, eaman a4e47992, tested llama.cpp version 1243
  SHA-256 dc266cba88936c9e6db596dca897e4b7470c51cc9f3597c93026d967512ed6f5

For Vulkan, initially retain:

  --pipeline-parallel auto

Build 1180 controlled Qwen3.6-35B-A3B comparison at 32,768 fixed context:

- ROCm F16: pipeline auto 70.38 t/s versus off 69.67 t/s; identical 96.283% acceptance. The 1.02% generation difference is inconclusive from one matched request.
- Vulkan auto: Q4_0 41.45 t/s versus F16 39.00 t/s (+6.27%); acceptance was 97.133% versus 96.198%. Q4_0 is provisional for this model pending repeats and fitted-context comparison.
- Verbose startup confirmed target pipeline enabled under ROCm auto and disabled for MTP.
- Existing MoE n-max testing fitted 203,776 context with 4 versus 195,584 with 5. Four has the better capacity tradeoff, but the active launcher retains 5 for its small potential long-context speculative benefit.
- Raw logs: bug/logs/controlled_*_20260820.log; earlier n-max notes: /home/eaman/launch/bench/moe_large.
- Build 1180 Qwopus3.5 9B measurements confirmed recurrent state = 50.25 MiB * (1 + n_rs_seq): 50.25 MiB at 0, 100.50 MiB at 1, and 251.25 MiB at 4.
- The operational rollback path produced byte-identical 128-token greedy output to target-only generation at both n-max 1 and 4. The n-max 1 run rejected 18 of 72 drafts; n-max 4 rejected 100 of 181. The server `PARTIAL_ONLY` path is validated from the minimum useful depth through the active depth.
- The opt-in RS branch is rebased onto production `a4e47992` at `2eb3b94f`;
  production `eaman` remains unchanged. CPU and combined HIP/Vulkan build 1247,
  parser tests, and registered Qwen3.5/Nemotron-H rollback tests pass.
- Matched dense depth 3/1 fitted 136,704/149,504 context (+12,800, +9.36%);
  the seeded 192-token outputs were byte-identical at 24.79/24.12 t/s.
- Matched MoE depth 4/1 fitted 233,216/262,144 context (+28,928, +12.4%).
  Depth 1 generated 192 tokens at 71.30 t/s with 2 replay events / 5 tokens.
  Real-GPU MoE outputs diverged late even for target-only generation, so long
  output byte identity is not used as a rollback-correctness criterion.
- Direct `-np 2` startup captured a 100.50 MiB incremental fitter reservation
  and a 201.00 MiB final two-sequence RS buffer. Two concurrent 192-token
  requests completed without OOM/cross-talk and both exercised replay.
- A 1,024-token 9B depth-1 run completed at 62.94 t/s with 523/612 accepted
  drafts and 24 replay events / 69 replayed tokens.
- Current experimental exports: `latest_rs_cumulative_8144f31.patch`
  (SHA-256 `c1a8305d506f5f1cd7318eb82499c9e91cc5a2ced92d6e54b0db643b894184ae`)
  and `mtp_rs_only_8144f31.patch`
  (SHA-256 `e8a5e296b2f77e5103159ca0f1b9c88795aa1ea7fb6bb69516448cdb14d2f611`).
  Both cleanly apply to `8144f31`; the cumulative tree exactly matches the RS
  branch. Historical `a3b1eff` exports remain pinned historical artifacts.
- Remaining RS follow-up: cache repeated scratch-fit measurements and perform
  near-limit long-prompt/delayed-OOM testing before considering production merge.
- Separate findings remain: the dedicated test fails only when a full checkpoint is loaded over a dirty hybrid context, and MTP n-max 0 aborts on its first request. Neither affects normal n >= 1 MTP or the validated server `PARTIAL_ONLY` path.
- A matched RX 6800 Qwopus3.5 9B test found no benefit from `--no-spec-draft-backend-sampling`: disabled versus enabled changed mean generation from 66.98 to 66.13 t/s and median from 65.43 to 65.22 t/s, with byte-identical output and identical 95.187% draft acceptance. The enabled path reported unsupported ROCm TOP_K and fell back; the measured overhead was below run-to-run noise.

Continue comparing Q4_0 and F16 draft KV per model and monitor fitted context, pipeline mode, buffer sizes, allocation failures, throughput, draft acceptance, and long-prompt stability.

Author and maintainer: eaman. Historical Qwen and Sol labels identify
LLM-assisted development series, not patch authorship.

Historical notes and logs are under bug/logs/; archived patches are under bug/old/patches/.
Manual per-model dual-GPU placement procedure: bug/dual_gpu_context_balancing_guide.md

2026-08-18 MoE MTP auto-fit result:

- Unpatched `--fit-target 50` selected 235,008 context but OOMed on a
  12,309-token prompt because provisional MoE placement contaminated later fits.
- `mtp_fit_scratch_placement.patch` isolates estimation placement from the final
  fit. Patched target 50 selected the same 235,008 context and passed both a
  32,327-token prompt plus generation and a separate 12,308 + 64-token run.
- All four patched fit passes succeeded; HIP/Vulkan server compiled and linked.
- The fix is committed on source branch `eaman` as `773364c` and included in the
  current standalone patch; the validated HIP/Vulkan build is deployed to
  `/home/eaman/llama/bin_vulkan`.
- `moe_tester.sh` now uses no explicit context or placement, target 50, and
  MTP n-max 4. Binary and launcher rollback copies use the suffix
  `.pre-moe-fit-20260818`.
- A later production run reached 96,484 cached tokens plus at least 6,632 new
  tokens, then SIGSEGV occurred inside ROCm `hipGraphExecUpdate`, not in the
  fitter or checkpoint path. `moe_tester.sh` now exports
  `GGML_CUDA_DISABLE_GRAPHS=1` only for this MoE process; dense launchers are
  unaffected.
- The no-HIP-graphs regression test retained 235,008 fitted context, ingested
  32,318 tokens, generated 4,718 tokens, restored the prompt checkpoint, then
  generated a forced 7,000 tokens. It completed without a crash at 41.73 and
  47.31 t/s respectively. Full-context stability remains pending.
- Production validation then completed 10,385 generated tokens at a 104,837-token
  slot, followed by checkpoint restore, a 143,851-token prompt, and 1,698 tokens
  generated at 30.26 t/s. The server released cleanly at 145,550 slot tokens.
- Final target-50 stress validation restored a 197,327-token checkpoint,
  processed a 203,372-token prompt, generated 11,145 tokens at 25.78 t/s with
  99.252% draft acceptance, and released cleanly at 214,518 slot tokens. No OOM
  or segfault occurred; this is 20,490 tokens below the fitted 235,008 limit.