---
name: dual-gpu-tuner
description: Calibrate the optimal -ot tensor placement for dual-GPU llama.cpp (ROCm) servers to maximize fitted context. Use when the user asks to balance GPU memory, tune --tensor-split/-ot, recover stranded headroom after KV/MTP quantization, or run placement probes on a launcher script.
---

# Dual-GPU -ot Placement Tuner

## Starting a new model/launcher

User only needs to say: *"Tune -ot placement for <launcher script> (model <gguf>)"* — then:

1. `suggest_ot.py manifest --gguf <model> --data-dir data` (skip if manifest exists)
2. **Batch the probes** so the user takes the server down once (they may be
   prompting the agent through it!): plan baseline + top 3–4 stacked candidates
   from the model, then hand the user:
   ```
   printf '%s\n' "" '<ot1>' '<ot1> <ot2>' '<ot1> <ot2> <ot3>' \
   | ~/skills/dual-gpu-tuner/scripts/run_trials.sh \
       --history data/history_<model>.jsonl <launcher> [--restart]
   ```
   (one trial per line, space-separated -ot overrides, empty line = baseline).
   Fresh history per model.
   **Emit batches as a file, not paste-able text:** write the batch to
   `/tmp/probe.sh` (chmod +x) using shell variables for each -ot override so
   no line is long enough to wrap (wrapped chat output embeds newlines inside
   the single quotes and corrupts the regexes). User runs `bash /tmp/probe.sh`.
   Never auto-restart production at the end — the user restarts manually.
3. User runs it in one down window, pastes the output back.
4. `suggest_ot.py evaluate --history data/history_<model>.jsonl --manifest <m> --baseline best`
   → verify model0/model1 deltas, calibrate, plan a 2nd small batch if needed.
5. Stop when `stop: true` or best delta < 256. User validates TG/long-context;
   only then patch the launcher with the winning -ot lines.

Model-specific facts to calibrate from your own probes (never carry over between models):
- Where KV and RS land relative to the layer split vs `-ot`.
- How much of the manifest MiB actually moves (verify via model0/model1 delta per probe).
- Whether the fitter uses internal per-device budgets or rocm-smi free memory.
- The limiting device (the one whose implied c/1k >= measured KV/1k).
- How many graph splits a partial layer move adds → watch TG after convergence.

## What it does

Recovers context capacity that a coarse `--tensor-split` strands on one GPU by
moving small coherent tensor groups with `-ot`.

**Division of labor (never violate):**
- **Scripts** do measurement, log parsing, candidate generation, and simulation.
- **The model (you)** only reads the compact records, detects anomalies, picks
  among the constrained candidates, and explains the choice.
- **Never** invent free-form `-ot` regexes. Only use candidates emitted by
  `suggest_ot.py candidates/evaluate`, or a whole-block/family regex it printed.

## Files

```
dual-gpu-tuner/
├── SKILL.md
├── scripts/
│   ├── run_probe.sh          # short-lived fit probe (copies launcher, never edits it)
│   ├── parse_llama_log.py    # -lv 4 log -> normalized JSON record
│   └── suggest_ot.py         # manifest / candidates / evaluate / summary / record
└── data/
    ├── placement_history.jsonl
    └── manifests/<model>.manifest.json
```

## The balance principle (why moves can go toward the limiting device)

The fit is limited by the minimum per-device **capacity** = free MiB ÷
per-1000-token cost (KV+RS growth). A device with fewer attention layers has
a lower per-token cost, so **1 MiB of free there is worth more context**.
Therefore the optimal move is toward the device with the better
free÷per-token ratio — which is often the *limiting* device, not the surplus
one. `suggest_ot.py evaluate` computes the balance point (MiB) and the
ranked candidates; trust it over "move bytes away from the full GPU".

## Hard rules

1. The production server is **stopped by the user** before a probe runs.
   `run_probe.sh` refuses if a server for the model is still up (exit 3).
2. One placement variable per trial (the `-ot` override). Everything else in
   the launcher stays identical.
3. A probe is killed as soon as it reaches the listening state. Startup fit
   is capacity evidence only — it is **not** stability or throughput validation.
4. Keep the launcher's `--fit-target`. Do not lower it to manufacture gains.
5. Verify each accepted trial moved model bytes: `model0`/`model1` in the
   record must change by ~the candidate's `mib`. Unchanged sizes = regex
   matched nothing; discard the trial.
6. Context changes of 0 or 256 tokens may be rounding artifacts (256-token
   fit granularity), not real regressions.
7. Do not add per-device free memory together. The fit is limited by the
   minimum per-device capacity.
8. The result is model-specific: model+quant, KV types, VEC on/off, MTP depth,
   batch, tensor split, and llama.cpp build all invalidate a previous `-ot`.
   The history key carries all of these — never apply a winning `-ot` to a
   different key without re-running the search.

## Workflow

### 0. Read the launcher

Read the launcher script the user names (e.g. `~/run/my_model.sh`).
Note: model, devices, tensor split, cache types, fit target, VEC flag,
checkpoints. The probe extracts all of this automatically into the history key.

### 1. Baseline probe (no `-ot`)

```bash
~/skills/dual-gpu-tuner/scripts/run_probe.sh <launcher.sh>
```

Requires the production server to be stopped (user action). Pass the probe's
summary line back into the conversation — do not re-read the raw log.

### 2. Evaluate + propose

```bash
~/skills/dual-gpu-tuner/scripts/suggest_ot.py evaluate \
  --history ~/skills/dual-gpu-tuner/data/placement_history.jsonl \
  --gguf <model.gguf>
```

Output (compact JSON): latest run's per-device free / per-1000-token cost /
capacity, the **limiting device**, the **balance point** in MiB, and a ranked
candidate table with predicted context. If `stop: true`, the placement is
balanced — go to phase 3.

Pick the top feasible candidate whose move size brackets the balance point.
Prefer coherent families (whole FFN, attn group) over single tensors, and the
smallest move that reaches near-maximum predicted ctx.

### 3. Probe the candidate

```bash
~/skills/dual-gpu-tuner/scripts/run_probe.sh <launcher.sh> \
  '^blk\.36\.\w[\w.]*$=ROCm1' ['^blk\.37\.ffn_(gate|up)\.weight$=ROCm1']
```

Multiple overrides are allowed (one per line of the recommended move).

Then `evaluate` again. Repeat until `stop: true`, a trial regresses twice in a
row, or the best predicted gain is < 256 tokens.

**Phase discipline:**
- Coarse: whole blocks near the boundary (`kind: block` candidates) to find
  the imbalance sign and calibrate the MiB→ctx slope.
- Fine: tensor families on the boundary layer (`kind: family`) to land within
  ±~30 MiB of the balance point.
- If a whole-block move made the *destination* limiting, the block was too
  large for the headroom — fall back to families from the boundary layer.

### 4. Validate the winner (on the production launcher, user restarts)

1. Clean restart, confirm the fitted ctx matches the probe's.
2. Fixed-context PP/TG comparison against the no-override config, same prompts.
3. Draft-acceptance comparison (same sampling settings).
4. Long generation near the fitted context; watch for delayed OOM / scheduler
   errors.
5. Keep only if throughput regression is within the user's threshold
   (default: >5% TG loss rejects the candidate; try a smaller move).

### 5. Patch the launcher (only after validation)

Append the override to the launcher's continued command, quoted, with `\`
line continuations, plus a comment block:

```bash
# dual-gpu-tuner: calibrated <date>, ctx <ctx> (baseline <baseline_ctx>)
# key: <model> split <split> ctk/ctv <..> build <..>
-ot '^blk\.39\.attn_(output|k|v)\.weight$=ROCm1' \
```

Then record the result in `data/placement_history.jsonl` (already appended per
probe) and, if the user keeps notes, a short summary in their notes repo.

## History

`data/placement_history.jsonl` — one JSON object per line:
`{key: {model_file, tensor_split, ctk, ctv, ctk_draft, ctv_draft, vec,
fit_target, devices, batch, ctx_checkpoints}, run: {ctx_fitted, free_mib,
model_mib, kv_mib, rs_mib, compute_mib, graph_splits, load_ok, ot, ...}}`

`suggest_ot.py summary --history ...` prints the compact run table the
supervisor model should read instead of raw logs.

## Failure modes

| Symptom | Meaning | Action |
|---|---|---|
| probe exit 3 | production server still running | user stops it; never `--force` against a busy GPU |
| `load_ok: false` in record | fit failed / OOM | the candidate moved too much or wrong direction; revert |
| model0/model1 unchanged | regex matched no tensor | discard candidate; check manifest layer names |
| ctx delta 0 or 256 | 256-token rounding | try one smaller/larger coherent group |
| ctx up but TG down >5% | cross-device transfer cost | reduce or regroup the override |
| evaluate: no free memory data | record lacks rocm-smi snapshot | re-run probe (snapshot is taken automatically) |