Serve local GGUFs to opencode with llama-server router mode

Steps#

1. List every model in a download manifest with its exact size in bytes. gen-opencode.py reads <ROOT>/download_models.sh and treats a GGUF as complete only when its size matches. Each entry is one quoted line, "<hf-repo> <file>.gguf <bytes>". Two of my entries:

   MODELS=(
     "unsloth/LFM2.5-8B-A1B-GGUF LFM2.5-8B-A1B-UD-IQ4_NL.gguf 4358844000"
     "ornith-ai/Ornith-1.5-35B-A3B-GGUF Ornith-1.5-35B-Q4_K_M.gguf 21713463040"
   )
   

   Expected: one line per GGUF you plan to serve. A file with no manifest entry passes on the magic bytes alone.

   GotchaA partial download already starts with GGUF

   On 2026-10-21 Qwen-AgentWorld-35B-A3B-UD-IQ3_S.gguf was 236,871,911 bytes of 14,984,322,304 mid-download and its first four bytes were already GGUF. A magic-byte check alone would have offered it to opencode. The size match is what rejects it.

2. Write <ROOT>/serve/models.ini. One [*] section with what every model shares, then one section per model; the section name is the model id. Keys are llama-server options without the leading dashes (short forms allowed). Use absolute model paths. An excerpt of mine:

   ; llama-server router presets (b11292 Vulkan) for opencode. Section name = model id in the API.
   ; Only models whose file exists are exposed to opencode (gen-opencode.py checks).
   version = 1
   
   [*]
   ; whole context for one opencode session; fit picks -ngl / n-cpu-moe to fit the 680M pool
   c = 65536
   np = 1
   fit = on
   fitt = 1024
   ctk = q8_0
   ctv = q8_0
   fa = on
   jinja = true
   t = 8
   
   [lfm2.5-8b]
   model = /home/user/work/llm-testing/models/LFM2.5-8B-A1B-UD-IQ4_NL.gguf
   
   [ornith-35b]
   model = /home/user/work/llm-testing/models/Ornith-1.5-35B-Q4_K_M.gguf
   ; without it the CPU-side host buffer is too big for one RADV submission:
   ; "radv/amdgpu: Not enough memory for command submission" -> ErrorDeviceLost, even at -ngl 0
   no-host = true
   ; MTP head (blk.40.nextn): n-max 2 = 22.4 vs 21.0 t/s (+7%, ~67% accepted); n-max 3 slower. 2026-10-21
   spec-type = draft-mtp
   spec-draft-n-max = 2
   ; coding sampling, as lukesdevlab's [ornith-1.5-35b-coding]
   temp = 1.0
   top-p = 0.95
   min-p = 0.0
   presence-penalty = 0.0
   

   Expected: c in [*] is also the context limit opencode is told about (step 3). Write it as c: the script reads only that key, so ctx-size = … would give opencode a context of 0. np = 1 gives one slot, so the whole 64k context goes to the one opencode session.

3. Save <ROOT>/serve/gen-opencode.py. The listing below is my script as of 21:16 on 2026-10-21, the revision steps 4 and 5 ran; later revisions also write per-model thinking variants, which this manual does not cover. It writes active.ini (the presets whose GGUF is complete) and builds opencode's local provider from the same list. It locates everything relative to its own path; no edits needed.

   #!/usr/bin/env python3
   """Build serve/active.ini (presets whose GGUF exists) and the opencode `local` provider from it.
   
     gen-opencode.py            write active.ini, print diff of ~/.config/opencode/opencode.json
     gen-opencode.py --apply    also write opencode.json (backup next to it first)
     gen-opencode.py --ini-only write active.ini only (used by `oc`)
   """
   import difflib
   import json
   import re
   import shutil
   import sys
   import time
   from pathlib import Path
   
   SERVE = Path(__file__).resolve().parent
   SRC, ACTIVE = SERVE / "models.ini", SERVE / "active.ini"
   OPENCODE = Path.home() / ".config/opencode/opencode.json"
   BASE_URL = "http://127.0.0.1:8080/v1"
   OUTPUT_TOKENS = 8192
   
   
   def parse(path):
       """INI as ordered [(section or None, [lines])]; keeps comments, values as-is."""
       blocks, cur = [], (None, [])
       for line in path.read_text().splitlines():
           s = line.strip()
           if s.startswith("[") and s.endswith("]"):
               blocks.append(cur)
               cur = (s[1:-1], [])
           cur[1].append(line)
       blocks.append(cur)
       return blocks
   
   
   def kv(lines):
       out = {}
       for line in lines:
           s = line.split(";", 1)[0].strip()
           if "=" in s:
               k, v = (x.strip() for x in s.split("=", 1))
               out[k] = v
       return out
   
   
   def expected_sizes():
       """file name -> bytes, from download_models.sh (a partial download already starts with GGUF)."""
       script = SERVE.parent / "download_models.sh"
       out = {}
       for m in re.finditer(r'^\s*"\S+ (\S+\.gguf) (\d+)"', script.read_text(), re.M):
           out[m.group(1)] = int(m.group(2))
       return out
   
   
   def is_gguf(p, sizes):
       try:
           with open(p, "rb") as f:
               if f.read(4) != b"GGUF":
                   return False
           want = sizes.get(Path(p).name)
           return want is None or Path(p).stat().st_size == want
       except OSError:
           return False
   
   
   def main():
       blocks = parse(SRC)
       ctx = int(next(kv(l) for s, l in blocks if s == "*").get("c", 0))
       keep, models = [], {}
       sizes = expected_sizes()
       for sec, lines in blocks:
           if sec in (None, "*"):
               keep.append(lines)
               continue
           path = kv(lines).get("model", "")
           if not is_gguf(path, sizes):
               print(f"skip {sec}: no complete GGUF at {path}", file=sys.stderr)
               continue
           keep.append(lines)
           models[sec] = {
               "name": f"{sec} (local)",
               "tool_call": True,
               "limit": {"context": ctx, "output": OUTPUT_TOKENS},
           }
       if not models:
           sys.exit("no usable models in models.ini")
       ACTIVE.write_text("\n".join(l for lines in keep for l in lines) + "\n")
       print(f"active.ini: {', '.join(models)}", file=sys.stderr)
       if "--ini-only" in sys.argv:
           return
   
       old = OPENCODE.read_text()
       cfg = json.loads(old)
       cfg.setdefault("provider", {})["local"] = {
           "npm": "@ai-sdk/openai-compatible",
           "name": "llama.cpp local (680M)",
           "options": {"baseURL": BASE_URL},
           "models": models,
       }
       new = json.dumps(cfg, indent=2) + "\n"
       if new == old:
           print("opencode.json: unchanged", file=sys.stderr)
           return
       sys.stdout.writelines(difflib.unified_diff(
           old.splitlines(True), new.splitlines(True), str(OPENCODE), str(OPENCODE) + " (new)"))
       if "--apply" in sys.argv:
           bak = OPENCODE.with_name(f"opencode.json.bak-{time.strftime('%Y%m%d-%H%M%S')}")
           shutil.copy2(OPENCODE, bak)
           OPENCODE.write_text(new)
           print(f"written; backup {bak}", file=sys.stderr)
   
   
   if __name__ == "__main__":
       main()
   

4. Dry-run it and read the diff. Without flags it writes only active.ini and prints what it would change in opencode.json. This run used a copy of my serve/ directory and HOME pointed at a scratch directory holding that minimal three-line opencode.json, so the diff shows only the new provider:

   $ python3 -I serve/gen-opencode.py
   skip agentworld-35b-iq3: no complete GGUF at /home/user/work/llm-testing/models/Qwen-AgentWorld-35B-A3B-UD-IQ3_S.gguf
   skip nail-35b: no complete GGUF at /home/user/work/llm-testing/models/Nail-Qwen3.6-35B-A3B-MTP-UD-IQ4_XS.gguf
   skip nail-35b-iq3: no complete GGUF at /home/user/work/llm-testing/models/Nail-Qwen3.6-35B-A3B-MTP-UD-IQ3_S.gguf
   skip fablevibes-14b: no complete GGUF at /home/user/work/llm-testing/models/Qwen3.6-14B-A3B-FableVibes-Q4_K_M.gguf
   skip mellum-12b: no complete GGUF at /home/user/work/llm-testing/models/Mellum2.1-12B-A2.5B-Thinking-Q4_K_M.gguf
   skip swift-9b: no complete GGUF at /home/user/work/llm-testing/models/Swift-1.5-Ornith-9B-Q4_K_M.gguf
   active.ini: lfm2.5-8b, gemma4-26b, lfm2.5-24b, ornith-35b, agentworld-35b, swift-27b
   --- <SCRATCH_HOME>/.config/opencode/opencode.json
   +++ <SCRATCH_HOME>/.config/opencode/opencode.json (new)
   @@ -1,3 +1,62 @@
    {
   -  "$schema": "https://opencode.ai/config.json"
   +  "$schema": "https://opencode.ai/config.json",
   +  "provider": {
   +    "local": {
   +      "npm": "@ai-sdk/openai-compatible",
   +      "name": "llama.cpp local (680M)",
   +      "options": {
   +        "baseURL": "http://127.0.0.1:8080/v1"
   +      },
   +      "models": {
   +        "lfm2.5-8b": {
   +          "name": "lfm2.5-8b (local)",
   +          "tool_call": true,
   +          "limit": {
   +            "context": 65536,
   +            "output": 8192
   +          }
   +        },
   …
   +        "swift-27b": {
   +          "name": "swift-27b (local)",
   +          "tool_call": true,
   +          "limit": {
   +            "context": 65536,
   +            "output": 8192
   +          }
   +        }
   +      }
   +    }
   +  }
    }
   

   Expected: one skip line per preset whose file is missing or incomplete, an active.ini: line naming the rest, and a diff that adds or replaces only provider.local. Other keys in your opencode.json stay as they are.

5. Apply it.

   python3 -I <ROOT>/serve/gen-opencode.py --apply
   

   Expected: the same diff, then written; backup <path> on stderr. In the scratch copy from step 4:

   active.ini: lfm2.5-8b, gemma4-26b, lfm2.5-24b, ornith-35b, agentworld-35b, swift-27b
   written; backup <SCRATCH_HOME>/.config/opencode/opencode.json.bak-20261021-213941
   

   At about 21:40 on 2026-10-21, ~/.config/opencode/ on my real config held two such backups. The first is from my first --apply at 13:49:05; the second was written at 20:41:34 and at that time differed from the live file only by "lsp": true:

   $ ls -l ~/.config/opencode/opencode.json.bak-*
   -rw-------. 1 user user  515 Oct  5 16:08 /home/user/.config/opencode/opencode.json.bak-20261021-134905
   -rw-------. 1 user user 1684 Oct 21 13:49 /home/user/.config/opencode/opencode.json.bak-20261021-204134
   

   The name carries the time of the write; the listed date is the old file's, because the copy keeps its mtime. Re-run --apply whenever a download finishes: oc refreshes only active.ini, not opencode's model list.

6. Write the per-session config <ROOT>/serve/opencode-local.json. oc passes it as OPENCODE_CONFIG, so plain opencode keeps your global default model.

   {
     "$schema": "https://opencode.ai/config.json",
     "model": "local/ornith-35b",
     "small_model": "local/ornith-35b",
     "lsp": true
   }
   

   Expected: small_model is the same model on purpose. opencode uses small_model for side tasks such as session titles; with --models-max 1 a different model there would unload the main one on every such request. "lsp": true turns on opencode's built-in language servers, which are off when the key is absent (checked in Verify).

7. Save the wrapper <ROOT>/serve/oc, make it executable and put it on PATH. The ROOT= line is the one to change; on my box it is an absolute path. This is my serve/oc at commit 2a85e80 (file time 21:45 on 2026-10-21), verbatim except for the ROOT= line. The oc sessions in the field notes (20:26–20:56) ran earlier revisions of it; the health-check refusal and the SDKMAN block came later. The SDKMAN block has nothing to do with the router: it puts Java tools on PATH for opencode's non-interactive shell, and you can drop it. The "+32.5 s per session" in the compat comment is one n=1 A/B on one task; see the 20:30–20:36 field note before you rely on it.

   #!/usr/bin/env bash
   # opencode with the local llama.cpp router: start it if down, stop it when the last opencode exits.
   # Usage: oc [opencode args...]    Env: OC_KEEP=1 leaves the router running on exit,
   #        OC_CLAUDE=1 keeps Claude Code compat (CLAUDE.md, ~/.claude/skills) — see occ.
   set -uo pipefail
   ROOT="$HOME/work/llm-testing"
   UNIT=llm-router
   PORT=8080
   LOG="$ROOT/serve/router.log"
   
   if ! systemctl --user is-active -q "$UNIT"; then
     # a running bench would be skewed by the router, and the router by the bench
     if pgrep -x llama-bench >/dev/null && [ "${FORCE:-}" != 1 ]; then
       echo "oc: llama-bench is running (bench-all?); FORCE=1 to start the router anyway" >&2
       exit 1
     fi
     # opencode would otherwise talk to whatever else answers there (e.g. Fedora's b6153 llama-server)
     if ss -ltnH "sport = :$PORT" | command grep -q .; then
       echo "oc: port $PORT is taken by something other than $UNIT:" >&2
       ss -ltnpH "sport = :$PORT" >&2
       exit 1
     fi
     python3 -I "$ROOT/serve/gen-opencode.py" --ini-only || exit 1
     systemctl --user reset-failed "$UNIT" 2>/dev/null
     systemd-run --user --unit="$UNIT" -q \
       -p StandardOutput=append:"$LOG" -p StandardError=append:"$LOG" \
       "$ROOT/bin/vulkan/llama-server" --host 127.0.0.1 --port "$PORT" \
       --models-preset "$ROOT/serve/active.ini" --models-max 1 --sleep-idle-seconds 900 || exit 1
     for _ in $(seq 60); do
       curl -sf "http://127.0.0.1:$PORT/health" >/dev/null && break
       sleep 0.5
     done
     curl -sf "http://127.0.0.1:$PORT/health" >/dev/null || { echo "oc: router not healthy, see $LOG" >&2; exit 1; }
   elif ! curl -sf "http://127.0.0.1:$PORT/health" >/dev/null; then
     # the unit name is also used by benchmark/experiment jobs (so queued tests wait); do not touch them
     echo "oc: unit $UNIT is active but nothing answers on :$PORT (an experiment is using the name?)." >&2
     echo "    Wait for it to finish: systemctl --user status $UNIT" >&2
     exit 1
   fi
   
   # opencode's shell is non-interactive: the SDKMAN lazy-load functions from ~/.bashrc (mvn, mn, ...)
   # do not exist there, so put the current SDKMAN tools on PATH explicitly
   SDK="${SDKMAN_DIR:-$HOME/.sdkman}/candidates"
   for c in java maven mvnd gradle micronaut springboot jbang; do
     [ -d "$SDK/$c/current/bin" ] && PATH="$SDK/$c/current/bin:$PATH"
   done
   [ -d "$SDK/java/current" ] && export JAVA_HOME="$SDK/java/current"
   export PATH
   
   # default model only for oc sessions (Ornith); plain `opencode` keeps the global default.
   # small_model = same model: with --models-max 1 a different one would swap on every title request.
   # no Claude Code compat: measured +3.5k prompt tokens (17.2k vs 13.7k) = +32.5 s per session on Ornith;
   # `occ` (OC_CLAUDE=1) keeps it on for comparison; tui.json adds the local-LLM status (sidebar + home footer)
   claude_off=1
   [ "${OC_CLAUDE:-}" = 1 ] && claude_off=0
   OPENCODE_DISABLE_CLAUDE_CODE=$claude_off \
   OPENCODE_TUI_CONFIG="$ROOT/serve/opencode-tui/tui.json" \
   OPENCODE_CONFIG="$ROOT/serve/opencode-local.json" opencode "$@"
   rc=$?
   
   # process name: opencode.exe up to 1.18, opencode since 1.19
   if [ "${OC_KEEP:-}" != 1 ] && ! pgrep -x opencode >/dev/null && ! pgrep -x opencode.exe >/dev/null; then
     systemctl --user stop "$UNIT"
   fi
   exit "$rc"
   

   The OPENCODE_TUI_CONFIG line belongs to ocmon, which documents it. Without ocmon, delete that line [UNVERIFIED: oc without the OPENCODE_TUI_CONFIG line not run]. The optional occ keeps opencode's Claude Code compatibility on, for comparison:

   #!/usr/bin/env bash
   # oc with opencode's Claude Code compat ON (reads CLAUDE.md, ~/.claude/skills): for comparing against plain oc.
   OC_CLAUDE=1 exec "$(dirname "$(readlink -f "$0")")/oc" "$@"
   

   Install both:

   chmod +x <ROOT>/serve/oc <ROOT>/serve/occ
   ln -s <ROOT>/serve/oc ~/.local/bin/oc
   ln -s <ROOT>/serve/occ ~/.local/bin/occ
   

   Expected: occ finds oc next to its own target through readlink -f, so the symlinks work.

   NoteThe router listens on 127.0.0.1 only

   Every start logs security: no API key is set and CORS allows all origins and router mode enabled - do not expose to untrusted environments. oc binds --host 127.0.0.1; keep it that way.

8. Start a session with oc and send the first prompt. The router starts with no model loaded; the first request loads ornith-35b. From <ROOT>/serve/router.log of one such session:

   0.00.118.715 I srv    operator(): Available models (5):
   0.00.118.718 I srv    operator():   [    preset] gemma4-26b
   0.00.118.719 I srv    operator():   [    preset] lfm2.5-24b
   0.00.118.719 I srv    operator():   [    preset] lfm2.5-8b
   0.00.118.720 I srv    operator():   [    preset] ornith-35b
   0.00.118.720 I srv    operator():   [    preset] swift-27b
   0.00.118.789 W srv  llama_server: security: no API key is set and CORS allows all origins (see https://github.com/ggml-org/llama.cpp/pull/25655)
   0.00.118.793 W srv  llama_server: security: router mode enabled - do not expose to untrusted environments
   0.00.118.794 I srv  llama_server: starting server in router mode. models will be automatically loaded on-demand
   0.00.120.996 I srv  llama_server: listening on http://127.0.0.1:8080
   0.00.120.999 W srv  llama_server: notice: server default port will be changed to :9931 in a future release (ref: https://github.com/ggml-org/llama.cpp/pull/26508)
   0.00.486.331 I srv          join: request for name=ornith-35b queued at position 1
   0.00.486.340 I srv  ensure_model: waiting until model name=ornith-35b is fully loaded...
   0.00.486.345 I srv  ensure_model: slot available, loading queued model name=ornith-35b
   0.00.486.475 I srv          load: spawning server instance with name=ornith-35b on port 34017
   …
   [34017] 0.21.081.884 I srv  llama_server: model loaded
   [34017] 0.21.081.890 I srv  llama_server: listening on http://127.0.0.1:34017
   

   Expected: listening on http://127.0.0.1:8080 within the 30 s oc waits, then one spawning server instance per model switch. The child's own timestamps ([34017] 0.21…) count from the child's start: Ornith took 21 s to load here.

   The child is the router's own binary, not whatever llama-server is first on PATH. At commit 90c908d06, server-models.cpp resolves /proc/self/exe (line 461) and spawns that; the log shows it as the first argument:

   0.07.495.373 I srv          load: spawning server instance with args:
   0.07.495.382 I srv          load:   /home/user/work/llm-testing/bin/vulkan/llama-server
   

   So Fedora's llama-cpp-b6153 package never runs as a child.

9. Quit opencode. When no process named opencode (or opencode.exe) is left, oc runs systemctl --user stop llm-router. The router log of the same session ends with:

   3.36.594.919 I srv    operator(): operator(): cleaning up before exit...
   3.36.594.990 I srv    unload_all: stopping model instance name=ornith-35b
   [34017] 3.36.101.047 I srv    operator(): exit command received, exiting...
   [34017] 3.36.103.853 W ~llama_context:    Vulkan0 compute buffer size of 282.0000 MiB, does not match expectation of 275.5000 MiB
   3.37.349.512 I srv           run: instance name=ornith-35b exited with status 0
   

   Expected: exited with status 0. Start opencode with OC_KEEP=1 oc to leave the router running after exit.

   WarningThe unit name is global to your user session

   Other jobs can run under the transient unit name llm-router too. On 2026-10-21 at 21:15 another experiment on the same box started its benchmark script under that name: the unit was active, port 8080 had no listener, and curl -sf http://127.0.0.1:8080/health exited 7. An earlier oc checked only systemctl --user is-active, so in that state it would have skipped the start and, on exit, stopped the other job's unit. The oc above refuses instead. In a live run at about 21:45 that day, with an experiment holding the unit and nothing on :8080, it printed this and exited 1 before starting opencode, and the unit stayed active:

   oc: unit llm-router is active but nothing answers on :8080 (an experiment is using the name?).
       Wait for it to finish: systemctl --user status llm-router
   

   Use llm-router for the router only, or change UNIT=.