# Upstream Search Results for Further ROCm/MTP Improvements Search date: **2026-08-07** System in scope: RX 6800 16 GiB + RX 6700 XT 12 GiB, HIP/ROCm, Qwen3.6-27B with native MTP, fixed layer split, one server slot. The HIP VEC-for-quantized-KV patch is already being tested in the current `vec_sol` build and is intentionally not treated as future work here. ## Highest-value follow-up: recurrent-state memory The current Qwen3.6 configuration reports: ```text ROCm0 recurrent state: 374.06 MiB ROCm1 recurrent state: 224.44 MiB Total: 598.50 MiB n_rs_seq: 3 ``` The state contains one live recurrent state plus rollback snapshots for speculative decoding. This is independent of the quantized-KV flash-attention workspace and remains a meaningful VRAM/context target after VEC testing. Possible work: - Benchmark `--spec-draft-n-max 2` versus 3 with identical prompts and seeds. - Measure accepted draft length, generation speed, and context capacity. - Investigate configurable GPU rollback depth separately from draft length. - Investigate lower-precision rollback snapshots while retaining the live state at F32. Relevant upstream work: [PR #24785, recurrent-state shrink/expand for prompt cache](https://github.com/ggml-org/llama.cpp/pull/24785). This is mainly prompt-cache work, not a complete solution for live speculative rollback memory, but it identifies an active area for recurrent-state memory management. ## HIP quantized-KV allocation lifetime and VRAM growth Two upstream proposals address quantized-KV temporary-buffer lifetime rather than initial fitting: - [PR #21830, batch HIP flash-attention calls for quantized KV](https://github.com/ggml-org/llama.cpp/pull/21830) - [PR #22094, bypass the HIP memory pool for F16 flash-attention temporaries](https://github.com/ggml-org/llama.cpp/pull/22094) Both are closed and not merged. They target VRAM that grows or remains stranded during long prompts/generation because the HIP legacy pool retains increasingly large F16 dequantization buffers. The reported trade-off for the raw-allocation approach was approximately 5% prompt-throughput loss in one test, with little decode impact. These are worth revisiting only if long VEC runs still show steadily increasing VRAM or delayed OOM. They are less important if VEC keeps the quantized path free of the large temporary allocation. Related upstream issue: [ROCm VMM issue #2516](https://github.com/ROCm/rocm-systems/issues/2516), referenced by PR #22094. HIP VMM behavior on the relevant consumer GPUs affects whether unused allocations can be reclaimed. ## Fit integration and MTP accounting - [Issue #25408, speculative context over-allocation when context size is unspecified](https://github.com/ggml-org/llama.cpp/issues/25408) - [PR #25465, correct speculative-context fitting without a specified context](https://github.com/ggml-org/llama.cpp/pull/25465) - [Issue #26038, excessive ROCm MTP compute reservation](https://github.com/ggml-org/llama.cpp/issues/26038) These are directly related to the `sol` work. `sol3` already implements a more topology-specific two-pass/refinement approach, including the fitted target context and complete MTP compute accounting. The upstream PR remains useful for comparing future refactors, especially a unified fit method for the target, MTP, and other components. Potential future improvement: replace fixed refinement passes with a single fit coordinator that owns all context-dependent components. This could improve maintainability and reduce duplicated probing, but it should not be attempted until the current VEC result is stable. ## ROCm backend support and performance gaps ### RDNA2 native support [Issue #26702](https://github.com/ggml-org/llama.cpp/issues/26702) reports RX 6700 XT/gfx1031 support problems, including the need for `HSA_OVERRIDE_GFX_VERSION=10.3.0` and a flash-attention occupancy workaround. This overlaps the current hardware and is worth monitoring for a proper native gfx1031 fix. ### Backend sampling The current logs report: ```text device 'ROCm1' does not have support for op TOP_K needed for sampler 'top-k' ``` This causes backend-sampling fallback and is primarily a generation-performance issue, not a context-capacity issue. The same gap is discussed in [issue #26702](https://github.com/ggml-org/llama.cpp/issues/26702). A future TOP_K implementation or a controlled CPU-sampling comparison could improve decode latency and synchronization overhead. ### Quantized-KV format coverage [Issue #25007](https://github.com/ggml-org/llama.cpp/issues/25007) confirms that ROCm quantized-KV format coverage depends on compiling `GGML_CUDA_FA_ALL_QUANTS=ON`. The current build already has this enabled, so this is not an outstanding action item. It is useful as a regression check when changing the HIP flash-attention dispatch. ## Alternative KV quantization formats ### KVarN [Issue #24139](https://github.com/ggml-org/llama.cpp/issues/24139) describes KVarN research and a third-party BeeLlama.cpp implementation. Reported measurements claim better quality than ordinary Q4 at similar memory use, but the implementation is immature and its backend/performance support is not established for this ROCm system. Future action only if VEC plus standard Q4/Q5 formats remain insufficient: - Check whether KVarN can be ported to the current llama.cpp revision. - Verify HIP support and actual MTP compatibility. - Measure quality, prompt speed, decode speed, and memory independently. ### TurboQuant KV [PR #20995, TQ4_0 KV cache](https://github.com/ggml-org/llama.cpp/pull/20995) and [PR #21010, Vulkan TQ3_0 KV cache](https://github.com/ggml-org/llama.cpp/pull/21010) are closed/unmerged. They are not immediate options for the current ROCm build and should not distract from validating standard Q4/Q4 through VEC first. ## MTP correctness and long-session stability [Issue #23577](https://github.com/ggml-org/llama.cpp/issues/23577) reports repeated `/` output and MTP desynchronization after long sessions on Qwen3.6 across several backends. Reports are mixed and some appear environment-specific, but the issue is relevant to the current long-context stability testing. Future tests should specifically include: - Long reasoning sessions beyond the current prompt-cache checkpoint boundaries. - Repeated rollback and checkpoint restoration. - MTP acceptance monitoring before and after any delayed VRAM pressure. - Comparison with MTP disabled to separate backend corruption from speculative-state issues. ## Lower-priority or non-capacity options - MTP device placement could be investigated because ROCm1 is the limiting GPU, but moving NextN/MTP tensors may introduce cross-device transfers and reduce decode speed. - More MTP fit refinement passes are unlikely to recover more than a few hundred tokens. - Reducing `--fit-target` may recover a small amount of context but decreases safety margin and is not appropriate before delayed-OOM testing. - Smaller microbatch may reduce scheduler memory, but it can reduce prompt throughput and does not address the main context-sized HIP workspace problem. - Prompt-cache RAM and checkpoint settings mainly affect host memory, not the fitted GPU context. ## Recommended continuation order 1. Complete long-context, long-generation, rollback, and delayed-OOM testing of `vec_sol`. 2. Compare Q4/Q4 VEC against the previous F16/off build using identical prompts, seeds, context, and MTP settings. 3. If VRAM remains stable, benchmark `n_max=2` versus `n_max=3` to quantify recurrent-state savings. 4. If VRAM still grows, revisit PR #21830/#22094-style allocation lifetime fixes. 5. Investigate ROCm TOP_K support for decode performance. 6. Consider MTP placement balancing and alternative KV formats only after the standard path is stable.