AMD llama.cpp: reducing MTP buffer overhead Qwen 27B

Tested with llama.cpp version 909, master commit 7bd8282, ROCm 7.14.

Name	Last modified	Size

Parent Directory		-
web_search.md	2026-08-08 21:14	7.7K
summary_sol.md	2026-08-08 21:14	16K
status	2026-08-08 21:14	1.2K
rocm_improvement.patch	2026-08-08 21:14	18K
possible_improvements.md	2026-08-08 21:14	12K
old/	2026-08-08 21:14	-
logs/	2026-08-08 21:14	-
llama_scripts/	2026-08-08 21:43	-
AGENTS.md	2026-08-08 21:14	2.6K

Available context length with and without the patch:

Model: QWEN 27B	ROCm stock	patched	Vulkan stock	patched
IQ4_XS Pure, single 16GB GPU	19.456	76.032	68,352	78,592
Q6_K_L on 16GB + 12GB	64,256	149,248	68,864	151,296

The issue is that llama.cpp overestimates the memory needed for MTP compute-buffer/scheduler allocation during auto-fit, that leaves much less ctx available to the user than what actually needed by MTP. This patch stops the fitter from throwing away context based on an inflated MTP memory estimate.

Patch, launch scripts used for llama-server and raw logs: https://store.piffa.net/lm/bug/
Tested against: llama.cpp version: 909, based on master commit 7bd8282, ROCm 7.14

Especially for ROCm with double GPU (16GPU + 12GB here) the amount of ctx gain is substantial, with longer session ROCm allows almost double prefill performances vs Vulkan yet on the mainline code the price to pay in ctx reduction for the extra compute is taxing.

You can build for both Vulkan and ROCm backends at the same time, the idea is that Vulkan saves some more vRAM while ROCm gives better prefill performance.
With a single 16GB GPU and limited ctx size you may wanna use Vulkan while when using 2 GPUs with layer splitting ROCm is worth the expense with this patch as you have much more ctx length for long sessions.