# Possible Further Improvements for ROCm Context and Performance ## Overview `old/patches/sol2.patch` resolves the known MTP fitting errors and recovers nearly all of the context provided by the earlier aggressive patch: | Configuration | Context | Behavior | |---|---:|---| | `fix_all.patch`, Q4 MTP | 105,472 | Recoverable ROCm allocation failure; compute not included in fit | | `sol.patch`, Q4 MTP | 92,160 | Correct but conservative | | `old/patches/sol2.patch`, Q4 MTP | 99,072 | Correct per-device iterative fit | | `old/patches/sol2.patch`, F16 MTP | **104,704** | Correct fit, no allocation fallback | The remaining opportunities are mostly below the fit layer. Further substantial gains require reducing real scheduler, flash-attention, or recurrent-state allocations rather than making the fitter more optimistic. The recommendations below are ordered approximately by expected benefit and engineering value. --- ## 1. Native Quantized HIP Flash Attention ### Priority **Partially addressed in `sol4.patch`; further coverage remains a long-term priority.** ### Current behavior The HIP/CUDA flash-attention allocator can reserve F16 conversion workspace when the tile kernel consumes quantized K or V cache tensors. `sol4.patch` adds HIP VEC dispatch for supported quantized-KV attention shapes, so VEC dequantizes in-register instead of using the large TILE/MMA conversion workspace for those cases. The initial Q4/Q4 MTP run with the merged VEC change reached 147,712 context tokens, compared with 121,088 for the previous F16/off build. Startup and short generation succeeded, but long-context stability and controlled performance validation remain pending. ### Remaining improvement The current VEC dispatch is shape- and format-limited. Extend coverage to more target and MTP cache formats, or add native quantized tile kernels that read supported K/V formats directly without materializing full-context F16 copies. Relevant formats for this setup: - q8_0 target K - q5_1 target V - q4_0 MTP K and V ### Expected benefit - Retain small quantized KV caches. - Eliminate or greatly reduce context-sized F16 conversion buffers. - Potentially improve prompt-processing speed by avoiding repeated conversion. - Reduce the ROCm/Vulkan context-size difference. The measured VEC gain is **26,624 tokens (+21.99%)** in the initial Q4/Q4 MTP run. More formats and long-run tests are needed before treating this as a stable general result. ### Difficulty and risk - Requires HIP kernel and dispatch work. - Must test all supported head sizes, cache formats, attention masks, and prompt/token-generation paths. - Long-context correctness, delayed-OOM behavior, and performance testing are required. ### Relevant upstream references - [llama.cpp issue #19036 - quantized KV flash-attention memory](https://github.com/ggml-org/llama.cpp/issues/19036) - [llama.cpp issue #23873 - ROCm quantized KV VRAM behavior](https://github.com/ggml-org/llama.cpp/issues/23873) - [llama.cpp discussion #21526 - HIP/ROCm quantized KV and F16 workspace](https://github.com/ggml-org/llama.cpp/discussions/21526) - [llama.cpp issue #24485 - quantized flash-attention kernel availability](https://github.com/ggml-org/llama.cpp/issues/24485) --- ## 2. Configurable Target Pipeline Parallelism ### Priority **Best next practical experiment** ### Current behavior `old/patches/sol2.patch` disables pipeline parallelism only for the MTP context. The target context still enables it: ```text llama_context: pipeline parallelism enabled ROCm0 compute buffer size = 806.86 MiB ROCm1 compute buffer size = 806.86 MiB graph splits = 3 sched copies = 4 ``` Disabling pipeline parallelism for MTP substantially reduced its scheduler allocation and removed the failed allocation/retry sequence. The target graph may exhibit a similar, though not necessarily equal, memory reduction. ### Proposed improvement Add a context or command-line option that explicitly controls pipeline parallelism: ```text --pipeline-parallel auto|on|off ``` The default should remain `auto`. This setup could then test `off` without carrying a hard-coded local modification. An alternative automatic policy could consider: - `n_parallel == 1` - Available VRAM and requested context - Whether pipeline scheduling prevents the requested fit target - Prompt-processing versus token-generation workloads ### Expected benefit Estimated context gain: **approximately 4K-12K tokens** if a meaningful portion of the duplicated target scheduler memory is avoided. ### Performance tradeoff Pipeline parallelism can improve prompt and multi-GPU throughput, including some single-request workloads. Disabling it may: - Reduce prompt-processing speed. - Reduce multi-request scaling. - Have a smaller impact on single-token generation, where layer dependencies already serialize much of the work. The result must be benchmarked rather than assumed. ### Suggested benchmark Compare `auto` and `off` using the same model and context: - Selected `n_ctx` - Prompt-processing tokens/second - Token-generation tokens/second - MTP acceptance rate - ROCm0 and ROCm1 peak VRAM - Scheduler graph splits and copies ### Relevant upstream reference - [llama.cpp discussion #20252 - pipeline parallel behavior](https://github.com/ggml-org/llama.cpp/discussions/20252) --- ## 3. Recurrent-State Snapshot Memory ### Priority **Medium priority, model-specific** ### Current behavior Qwen3.6 uses recurrent layers and MTP requires rollback support. With: ```text --spec-draft-n-max 3 ``` the target context sets: ```text n_rs_seq = 3 RS buffer = 598.50 MiB ``` The recurrent memory allocates `1 + n_rs_seq` state groups: - One live recurrent state. - Three speculative rollback snapshots. Approximate breakdown: ```text Base live state: 149.6 MiB Three snapshots: 448.9 MiB Total: 598.5 MiB ``` The split observed at runtime was: ```text ROCm0 RS buffer: 374.06 MiB ROCm1 RS buffer: 224.44 MiB ``` ### Possible improvements #### A. F16 rollback snapshots Keep the active recurrent state in F32, but store rollback-only copies in F16 and restore them to F32 when required. Potential saving: roughly half of the 449 MiB snapshot portion. Risks: - Numerical drift after rollback. - Conversion overhead. - Additional kernel and state-management complexity. #### B. Configurable GPU rollback depth Separate the number of GPU-resident rollback snapshots from `--spec-draft-n-max`. Example design: ```text --spec-rs-depth N ``` Drafts deeper than the GPU rollback depth would use an existing full checkpoint path. This allows a memory/performance tradeoff without reducing the maximum draft length. #### C. Reduce maximum MTP draft length Changing: ```text --spec-draft-n-max 3 ``` to: ```text --spec-draft-n-max 2 ``` removes one recurrent-state snapshot. This requires no code change but can reduce speculative speed when the third draft token has a high acceptance rate. ### Expected benefit - One fewer F32 snapshot: approximately 150 MiB total VRAM. - F16 snapshot storage: approximately 224 MiB total VRAM saving. - Estimated context gain: **approximately 3K-5K tokens**, limited by how the savings are distributed across the two GPUs. --- ## 4. Better MTP Placement Across GPUs ### Priority **Medium priority, experimental** ### Current behavior The final F16 test ended with: ```text ROCm0 free: 514 MiB ROCm1 free: 276 MiB ``` ROCm1 is the limiting device because the last/NextN layer and the native MTP context execute there. ### Proposed improvement Investigate explicit placement for native MTP tensors and scheduler work rather than inheriting the target model's last-layer placement. Potential approaches: - Add a native-MTP device option distinct from draft-model placement. - Allow tensor-buffer overrides for NextN tensors. - Move MTP tensors to ROCm0 while moving an appropriate subset of normal tensors toward ROCm1. - Explore partial tensor placement instead of moving an entire dense layer. The goal is to balance remaining VRAM without introducing excessive cross-device copies. ### Expected benefit Estimated context gain: **approximately 3K-8K tokens** if the 238 MiB free-memory imbalance can be reduced. ### Risks - Hidden device-to-device transfers for target embeddings or MTP activations. - Reduced generation speed. - Whole-layer placement is too coarse and may make the balance worse. - The MTP scheduler normally follows the backend of its weights, so compute placement cannot be changed independently without deeper work. --- ## 5. ROCm Backend Sampling Support ### Priority **Performance improvement, negligible context benefit** ### Current behavior The test log reports: ```text device 'ROCm1' does not have support for op TOP_K needed for sampler 'top-k' ``` Native MTP requests backend sampling, but the ROCm backend cannot execute the required TOP_K operation and falls back to CPU sampling. ### Possible improvements - Implement or enable the required ROCm TOP_K backend operation. - Detect unsupported backend sampling earlier and avoid constructing the failed backend chain. - Use `--no-spec-draft-backend-sampling` when CPU sampling is known to be required. ### Expected benefit - Potentially lower per-token CPU synchronization overhead. - Cleaner startup behavior. - Possibly better generation performance at high token rates. - Little or no meaningful VRAM/context improvement. --- ## 6. Minor Fit and Configuration Gains ### Additional MTP refinement passes `old/patches/sol2.patch` uses two MTP refinement passes. The tested sequence is already close to convergence: ```text 103,680 -> 105,216 -> 104,704 ``` More passes are expected to recover at most a few hundred tokens while adding more startup measurements. This is not a high-value change. ### Lower fit target Reducing: ```text --fit-target 50 ``` could recover roughly 1K-3K tokens, but it reduces explicit free-memory protection. The final F16 configuration already uses VRAM tightly, so zero margin is not recommended for normal use. ### Smaller microbatch Reducing `-ub 384` may reduce some scheduler buffers, but the largest quantized flash-attention conversion workspace scales mainly with context length. A smaller microbatch can also reduce prompt-processing performance. This should be benchmarked only after testing target pipeline-parallel control. ### Context checkpoints and prompt cache These settings mainly affect host memory: ```text --ctx-checkpoints 148 --cache-ram 8000 ``` They are not primary VRAM/context targets, though they can affect host RAM usage and checkpoint-management overhead. --- ## Suggested Development Order ### Phase 1: Low-risk measurement 1. Add a target pipeline-parallel `auto|on|off` option. 2. Test the current F16 MTP configuration with pipeline mode on and off. 3. Record context, prompt speed, generation speed, and peak VRAM. 4. Test `--no-spec-draft-backend-sampling` for generation overhead. ### Phase 2: Memory/performance tradeoffs 1. Test `--spec-draft-n-max 2` against 3. 2. Measure lost accepted draft tokens and tokens/second. 3. Prototype configurable recurrent GPU rollback depth if the memory saving is worthwhile. 4. Investigate fine-grained NextN tensor placement. ### Phase 3: Backend kernel work 1. Identify the exact HIP tile kernels selected for target and MTP prompt processing. 2. Measure the F16 K/V conversion components separately. 3. Prototype direct q8_0 K support first. 4. Add q5_1 V and q4_0 MTP support. 5. Validate long-context accuracy and compare against F16. --- ## Recommended Next Step The best immediate experiment is a configurable target pipeline-parallel switch. It is a small and reversible code change, and it will establish whether a meaningful portion of the approximately 807 MiB per-device target compute allocation can be exchanged for more context without an unacceptable speed loss. The most valuable long-term improvement is native quantized HIP flash attention that avoids full-context F16 conversion workspace. That change could improve both context capacity and prompt performance, and it directly addresses the main architectural difference observed between ROCm and Vulkan. Until such a kernel exists, the best tested configuration remains: ```text old/patches/sol2.patch --cache-type-k-draft f16 --cache-type-v-draft f16 ``` This provides 104,704 context tokens with measured MTP compute accounting and no ROCm allocation fallback.