# Improvements Compared with llama.cpp Mainline This document lists the changes introduced by the [final patch](https://store.piffa.net/lm/bug/) on the mainline llama.cpp source. This project increases usable context length by optimizing VRAM allocation. ## Complete change list - MTP uses the target context size after fitting instead of the original requested context size. - The server performs a preliminary target fit before estimating MTP memory. - MTP fitting reserves the memory that the runtime actually allocates. - MTP memory is measured again after fitting and the target fit is repeated with the refined reservation. - Fixed multi-GPU layouts are fitted against each GPU's individual memory limit. - MTP fitting and runtime initialization use a disabled pipeline scheduler. - The target context supports explicit `--pipeline-parallel auto|on|off` control. - Supported HIP quantized-KV flash-attention shapes use VEC dispatch to avoid the larger F16 conversion workspace. ## Detailed changes ### Fitted context size is used by MTP Native MTP creates a separate draft `llama_context`. The final patch sets its context size to the target context's fitted value, so MTP does not reserve context-dependent KV or flash-attention memory for the full requested context when the target cannot fit that size. File changed: - `common/speculative.cpp` ### MTP is estimated using a realistic context size Before reserving memory for the target, the server obtains a preliminary fitted target context and uses that size when measuring the MTP context. This avoids using the model's maximum trained context as the MTP estimate when the runtime will use a smaller context. File changed: - `tools/server/server-context.cpp` ### Complete MTP memory is included in fitting The target reservation includes the MTP allocation that exists at runtime, including MTP context storage and MTP compute memory, measured per device. This prevents the fitter from treating a standalone measurement as if it were either entirely shared or entirely independent when the runtime allocation differs. File changed: - `tools/server/server-context.cpp` ### MTP reservation is refined after fitting MTP memory depends on the fitted target context, while the target fit depends on the MTP reservation. The server performs bounded refinement passes so the final target context is fitted using MTP memory measured at a context close to the one that will actually run. File changed: -tes `tools/server/server-context.cpp` ### Fixed multi-GPU placement uses per-device limits When GPU layers and tensor placement are explicitly fixed, the fitter evaluates the available context on every selected GPU and chooses the limiting device. The result is rounded to the required 256-token boundary. This matches the actual constraint of asymmetric GPUs where a generic aggregate allowance can hide the device that runs out of memory first. File changed: - `common/fit.cpp` ### MTP uses non-pipeline scheduling MTP fitting and runtime initialization disable pipeline scheduling for the draft context. This avoids unnecessary scheduler copies and duplicated compute workspace for the MTP workload, while keeping the fitting decision consistent with runtime initialization. Files changed: - `common/speculative.cpp` - `src/llama-context.cpp` - `tools/server/server-context.cpp` ### Target pipeline scheduling is configurable The target context exposes: ```text --pipeline-parallel auto|on|off ``` The default `auto` mode preserves normal llama.cpp behavior. `on` requests pipeline scheduling and `off` disables it. This makes the target scheduler policy selectable for memory-constrained or asymmetric multi-GPU systems. Files changed: - `include/llama.h` - `src/llama.cpp` - `src/llama-context.cpp` - `common/common.h` - `common/common.cpp` - `common/arg.cpp` ### HIP quantized-KV attention uses VEC where supported The HIP flash-attention dispatcher sends supported quantized K/V attention shapes to the VEC implementation. VEC performs dequantization in registers for these paths, avoiding the larger context-sized F16 conversion workspace used by the fallback path. Coverage remains dependent on the supported quantization formats, head dimensions, and attention shapes; unsupported combinations continue through the existing dispatch paths. File changed: - `ggml/src/ggml-cuda/fattn.cu` ## Final source-file inventory | File | Final change | |---|---| | `common/arg.cpp` | Parse the target pipeline mode. | | `common/common.cpp` | Transfer the pipeline mode to context parameters. | | `common/common.h` | Store the common pipeline mode. | | `common/fit.cpp` | Fit fixed multi-GPU layouts against per-device limits. | | `common/speculative.cpp` | Use fitted MTP context sizing and non-pipeline MTP scheduling. | | `ggml/src/ggml-cuda/fattn.cu` | Dispatch supported HIP quantized-KV attention to VEC. | | `include/llama.h` | Expose the public pipeline-parallel context parameter. | | `src/llama-context.cpp` | Apply pipeline policy during scheduler creation. | | `src/llama.cpp` | Provide pipeline-mode helpers. | | `tools/server/server-context.cpp` | Measure, refine, and reserve complete MTP memory. | ## Relevant upstream references These references describe related upstream problems or implementation areas. They are included for comparison; the Sol patch is not claimed to be a direct copy of any one upstream change. - [Issue #23903](https://github.com/ggml-org/llama.cpp/issues/23903) — MTP draft-path buffer allocation. - [Issue #26038](https://github.com/ggml-org/llama.cpp/issues/26038) — excessive ROCm MTP compute reservation. - [Issue #25408](https://github.com/ggml-org/llama.cpp/issues/25408) and [PR #25465](https://github.com/ggml-org/llama.cpp/pull/25465) — speculative-context fitting. - [PR #21830](https://github.com/ggml-org/llama.cpp/pull/21830) — HIP flash-attention work for quantized KV, related to the VEC change. - [PR #22094](https://github.com/ggml-org/llama.cpp/pull/22094) — HIP flash-attention temporary allocation behavior. - [Issue #19036](https://github.com/ggml-org/llama.cpp/issues/19036), [Issue #23873](https://github.com/ggml-org/llama.cpp/issues/23873), and [Discussion #21526](https://github.com/ggml-org/llama.cpp/discussions/21526) — quantized-KV flash-attention memory behavior. - [Discussion #20252](https://github.com/ggml-org/llama.cpp/discussions/20252) — pipeline-parallel behavior and tradeoffs. ## Scope and limitations - The VEC dispatch is shape- and format-limited. - The patch does not force target pipeline scheduling off by default; `auto` remains the default. - ROCm cache-format recommendations are runtime configuration findings, not hard-coded patch behavior. - Long-context stability, delayed-OOM behavior, throughput, and draft-acceptance results require separate validation.