# Manual Dual-GPU Context Balancing for ROCm MTP ## Purpose Use this guide when quantized MTP KV reduces memory on one GPU but llama.cpp's fitted context does not increase. The procedure converts stranded per-device headroom into target-context capacity by refining model placement. Apply it separately to every new combination of: - model and model quantization; - GPU pair and device order; - target and draft KV formats; - batch, microbatch, parallel-slot, and draft settings; - llama.cpp build and backend. Do not copy a tensor override to another model without repeating the procedure. ## Contents - [Problem](#problem) - [Safety and test discipline](#safety-and-test-discipline) - [Required inputs](#required-inputs) - [Procedure](#procedure) - [Worked Q6_K_L example](#worked-q6_k_l-example) - [Launcher pattern](#launcher-pattern) - [Common failure modes](#common-failure-modes) - [Result record template](#result-record-template) - [Future automation target](#future-automation-target) ## Problem `--split-mode layer` places complete model layers on devices. A setting such as: ```text --device rocm0,rocm1 --split-mode layer --tensor-split 0.6,0.4 ``` defines a coarse layer boundary. Context-dependent allocations are then added per device: - target KV; - recurrent state; - target compute buffers; - MTP KV and compute buffers; - backend-specific workspace. The fitted context is the lowest context supported by any selected device. A quantized MTP cache can therefore save hundreds of MiB on ROCm1 without raising context when ROCm0 remains limiting. Total free memory is not interchangeable across GPUs. Changing `--tensor-split` may not solve this because its layer-mode placement is discrete. Moving one complete layer can transfer too much model memory and make the other GPU limiting. Use target tensor-buffer overrides for finer placement. ## Safety and test discipline 1. Confirm the notes repository is on `master` and the source repository is on `sol` before changing investigation artifacts. 2. Do not rebuild or modify the source repository merely to tune placement. 3. Use the intended binary and its matching shared libraries. 4. Ensure no earlier server owns the selected port before each trial. 5. Change one placement variable per trial. 6. Stop each diagnostic server after it reaches the listening state. 7. Treat startup fitting as capacity evidence only, not stability or performance validation. 8. Preserve at least the configured `--fit-target`; do not manufacture gains by silently reducing the safety margin. ## Required inputs Start from a known launcher and record: ```text binary and LD_LIBRARY_PATH model path and quantization device order split mode and tensor split n_gpu_layers target K/V cache types draft K/V cache types draft maximum batch and microbatch parallel slots pipeline mode fit target ``` Use `-lv 4` during fitting trials. Capture these output fields for every device: ```text fitted n_ctx_slot model buffer size target KV buffer size recurrent-state buffer size target compute buffer size MTP KV buffer size MTP compute buffer size refined MTP memory estimate graph splits and scheduler copies allocation failures or fallbacks ``` Verify the effective cache types in the speculative-context line. Do not assume that editing a launcher changed an already-running process. ## Procedure ### 1. Establish a matched cache baseline Run F16/F16 and the intended quantized draft cache with otherwise identical arguments. Require both runs to complete fitting, initialize the model, and reach the listening state. Calculate the per-device MTP saving: ```text MTP saving(device) = F16 MTP total(device) - quantized MTP total(device) ``` If the quantized MTP total does not fall materially, first verify kernel dispatch, supported attention shape, binary deployment, and effective cache arguments. Placement tuning cannot recover memory that the backend did not actually save. If MTP memory falls but context remains unchanged, continue. This is the stranded-headroom case. ### 2. Locate the saving and the limiting device Identify the device that owns the MTP allocation. Compare per-device model, target KV, recurrent, compute, and MTP allocations. Infer the limiting device from the per-device fit: - A cache saving on a non-limiting device becomes headroom, not context. - Moving target storage away from the limiting device can convert that headroom into context. - Moving storage in the other direction makes the imbalance worse. Do not add the two devices' free memory together. The fit is constrained by the minimum per-device capacity. ### 3. Calculate the adjacent whole-layer split Use this only as a diagnostic and retain it only if it improves the fit. For two devices, normalize the first split: ```text p0 = split0 / (split0 + split1) N = min(n_gpu_layers, n_layer_all + 1) i_gpu_start = max(n_layer_all + 1 - n_gpu_layers, 0) ``` Layer-mode placement uses cumulative split thresholds. The approximate number of GPU positions assigned to device 0 is: ```text b = ceil(N * p0) ``` Here, `b` is the approximate number of offloaded positions assigned to device 0. The last device-0 model-layer index is approximately: ```text boundary_layer = i_gpu_start + b - 1 ``` Verify the boundary empirically because the input/output tensors and architecture-specific placement can affect buffer totals. To move the last device-0 position to device 1, choose a new normalized split at or slightly below: ```text p0_next = (b - 1) / N ``` Use a decimal safely below the boundary to avoid floating-point ambiguity. For 66 GPU positions and `0.6,0.4`: ```text b = ceil(66 * 0.6) = 40 boundary = 39 / 66 = 0.590909... trial = 0.59,0.41 ``` Run the trial and compare fitted context and per-device model buffers. If the destination becomes limiting or context falls, restore the original split. This means a whole layer is too large for the available headroom. ### 4. Select a fine-grained target tensor override Keep the selected coarse `--tensor-split` in the command. If the whole-layer trial regressed, restore the original coarse split first. Then select tensors from the boundary layer on the limiting GPU and place them on the GPU with stranded headroom. The `-ot` rule is an exception layered on top of the coarse placement; it does not replace `--tensor-split`. Use: ```text -ot '=ROCm1' ``` Use `-ot`, not `-otd`: - `-ot` moves target-model tensors and changes the target device balance. - `-otd` changes draft-model tensor placement and does not solve this case. Quote the regex so the shell does not interpret parentheses or backslashes. When splitting a command across lines, place `\` at the end of every continued line with no trailing characters after it. Prefer a compute-coherent tensor group from one boundary layer. Useful candidate groups often include: ```text one FFN projection all FFN projections from the boundary layer one attention projection group another architecture-specific tensor group ``` Do not assume these names exist. Derive names from the model architecture or tensor listing. Verify a match by confirming that model-buffer bytes moved from the source GPU to the destination GPU. Unchanged model-buffer sizes mean the regex did not match. ### 5. Size the first override Treat the measured quantized-MTP saving as an upper placement budget, not as the amount to move immediately. Reserve space on the destination for the extra KV and compute memory created by a larger context. Start by moving approximately 20%-50% of the stranded headroom. Prefer one natural tensor group rather than arbitrary fragments. After each trial record: ```text X = model bytes moved from limiting GPU to destination GPU C = fitted context G = target graph splits ``` Accept the candidate provisionally when: - fitted context increases; - both devices satisfy the fit target; - model-buffer movement matches the requested override; - the server reaches the listening state; - there is no allocation fallback; - graph splits do not increase unexpectedly. If context falls, the override moved too much or targeted the wrong device. Reduce the tensor group or revert it. If context is unchanged, move another small coherent group or check whether the result was hidden by the 256-token rounding boundary. ### 6. Search for the best manual placement Use this bounded search: ```text baseline -> adjacent whole-layer trial -> revert if worse -> one boundary tensor/group -> add a second coherent tensor/group if context improves -> stop at the first regression, allocation failure, or unacceptable graph change ``` Keep a table for all trials: | Trial | Layer split | Override | Model MiB dev0/dev1 | Context | Graph splits | Result | |---|---|---|---:|---:|---:|---| | baseline | ... | none | .../... | ... | ... | keep/reject | Do not optimize only for the largest startup context. Prefer the smallest override that produces near-maximum context while preserving throughput and stability. ### 7. Validate the selected override After selecting a startup candidate: 1. Repeat startup from a clean GPU state. 2. Run a fixed-context comparison against the no-override configuration. 3. Compare prompt-processing and generation throughput. 4. Compare draft acceptance using the same prompts and sampling settings. 5. Exercise prompts close to the fitted context. 6. Run long generations and monitor delayed OOM, corruption, and scheduler errors. 7. Record peak per-device VRAM and free memory. 8. Keep the override model-specific until all checks pass. ## Worked Q6_K_L example Configuration: ```text GPUs: RX 6800 16 GiB + RX 6700 XT 12 GiB model: ThinkingCap Qwen3.6-27B Q6_K_L target KV: Q8_0/Q5_1 MTP KV: Q4_0/Q4_0 draft maximum: 3 split: layer, 0.6/0.4 fit target: 20 MiB pipeline: off ``` Matched cache result: | MTP cache | Context | MTP total on ROCm1 | |---|---:|---:| | F16/F16 | 153,856 | 764.70 MiB | | Q4_0/Q4_0 | 153,856 | 333.00 MiB | VEC and Q4_0 saved 431.70 MiB on ROCm1, but the coarse split could not convert it into context. Whole-layer trial: ```text --tensor-split 0.59,0.41 context: 145,152 ``` This moved approximately 300 MiB of model weights plus the associated context-dependent layer allocations to ROCm1 and overshot the balance. Fine-grained trial retained `0.6,0.4` and added: ```text -ot '^blk\.39\.ffn_(up|gate|down)\.weight$=ROCm1' ``` Measured result: ```text ROCm0 model: 12,569.81 -> 12,360.63 MiB ROCm1 model: 9,094.91 -> 9,304.09 MiB model storage moved: 209.18 MiB context: 153,856 -> 165,632 gain: 11,776 tokens (+7.65%) target graph splits: 35 ``` The override used part of the ROCm1 headroom while leaving room for the larger target and MTP caches. ## Launcher pattern Place the cache types, coarse split, and override in the same continued command. Both placement options are required to reproduce a calibrated result: ```bash --pipeline-parallel off \ --cache-type-k-draft q4_0 \ --cache-type-v-draft q4_0 \ --device rocm0,rocm1 \ --split-mode layer \ --tensor-split 0.6,0.4 \ -ot '^blk\.39\.ffn_(up|gate|down)\.weight$=ROCm1' ``` After starting, verify the effective command or log contains: ```text cache_k=q4_0, cache_v=q4_0 the expected ROCm0-to-ROCm1 model-buffer movement the improved n_ctx_slot ``` ## Common failure modes | Symptom | Cause | Action | |---|---|---| | Override produces no gain | Draft cache is still F16 | Change both draft-cache flags and restart | | Override produces no model-buffer movement | Regex matched no tensor | Correct the model-specific tensor name | | `command not found` on `-ot` | Previous line ended without `\` | Fix command continuation | | New run cannot bind the port | Earlier server is still running | Stop the earlier server before testing | | Whole-layer split reduces context | Complete layer overshot destination headroom | Restore split and use `-ot` | | Context changes by only 0 or 256 | Fit result crossed no additional rounding boundary | Try one small coherent tensor group | | More context but slower generation | Override added costly cross-device transfers | Reduce/regroup overrides and benchmark | | Different model fails or ignores override | Tensor names and boundary differ | Repeat the complete procedure | | Correct binary but old behavior | Shared libraries do not match binary | Deploy and select the complete build together | ## Result record template Store the following with every model-specific override: ```text date: llama.cpp commit/build: binary and library directory: model path and quantization: GPU names, sizes, and order: common launcher arguments: baseline split and context: draft cache types: per-device MTP allocation: tested whole-layer split and result: selected tensor override: model bytes moved per device: fitted context: graph splits and scheduler copies: startup status: throughput result: acceptance result: long-context result: known risks or pending validation: ``` ## Future automation target Automate this procedure by extending the fitter to measure target and MTP memory per device, identify the limiting device, enumerate boundary-layer tensor groups, simulate their placement, and choose the smallest transfer that maximizes the minimum per-device context capacity. Penalize extra graph splits and require a final measured refit before accepting the generated overrides.