tsvinc on tsvinc guides

Add a log partition to a Fedora live USB stick

Thu, 12 Nov 2026 00:00:00 +0000

This adds a third partition, exFAT, label LOGS, in the free space a Fedora live image leaves on a USB stick; the live session mounts it read-write at /run/media/liveuser/LOGS, so scripts and their logs live on the stick, survive a hard power-off, and never touch the laptop's own disk.

Tested on: SanDisk Cruzer Blade 16 GB written with the Fedora 45 KDE live image, partitioned from Windows 11 Pro 10.0.26200 (DiskPart 10.0.26100.1150), used from the Fedora 45 live session (kernel 7.2.11-300.fc45) on an ASUS Zenbook 14 UX3405CA, BIOS 316 (2026-11-11).
Not verified: other Fedora releases or image writers; doing the same from Linux (fdisk, mkfs.exfat); removing the partition again.
Needs: a Fedora live stick larger than the image, a Windows PC with an elevated PowerShell or Command Prompt.
Time: 10 minutes.
Risk: diskpart on the wrong disk number destroys that disk's data. On the right disk the image partitions are left alone, but the boot menu's media check fails afterwards (Field notes).

Why I needed it: my first S3 test runs on 2026-11-11 were set to write their logs next to the script on the Windows C: volume. Windows had logged an unexpected shutdown (Kernel-Power 41) at its boot before those runs, and Fast Startup is on, so C: was most likely not clean and Linux mounted it read-only; the script then fell back to /tmp, and the hard power-offs after the two hung runs took /tmp and the logs with it. None of the three runs left a log.

Read S0ix residency and drain from powercfg /sleepstudy

Wed, 11 Nov 2026 00:00:00 +0000

This turns the HTML report from powercfg /sleepstudy into one line per sleep on battery: how much of it the SoC spent in its deepest hardware idle state (S0ix, which Windows calls hardware DRIPS), and how much battery it used, in mWh, mW and percent per hour.

Tested on: ASUS Zenbook 14 OLED UX3405CA, Intel Core Ultra 9 285H, BIOS UX3405CA.316, Windows 11 Pro 10.0.26200 (build 26200.9445), Python 3.12.10 (2026-11-10, on sleeps recorded 2026-11-06, 2026-11-07 to 09 and 2026-11-09).
Not verified: any other laptop or SoC; AMD machines (Microsoft says hardware DRIPS is reported on Intel and Qualcomm SoCs only); machines that sleep in S3 instead of Modern Standby; running powercfg /sleepstudy from a non-elevated shell; any Python version other than 3.12.10.
Needs: a Modern Standby laptop with a battery, an elevated PowerShell or Command Prompt, Python 3 (standard library only), the two scripts from the appendix.
Time: one night with the lid closed on battery, then about 10 minutes.
Risk: none: no step changes a setting. powercfg /sleepstudy writes one HTML file, and the scripts only read it.

Before you start#

Make a working folder and save the two scripts from the appendix in it as residency.py and blockers.py, side by side: residency.py imports its HTML parser from blockers.py. Run them with python -I (isolated mode: user site-packages and the current directory stay off the import path; the script adds its own folder itself).
Check Python:

PS> python --version
Python 3.12.10

Backups: none needed. Nothing is changed; keep each night's HTML under its own name (sleepstudy-<date>.html) so a later re-run reads the same data.
Placeholders: none. The commands below run inside the working folder; in the pasted output it appears as C:\Users\user\sleepstudy (my user name and folder path are replaced).

Steps#

Check that the laptop sleeps in Modern Standby.

PS> powercfg /a
The following sleep states are available on this system:
    Standby (S0 Low Power Idle) Network Connected
    Hibernate
    Fast Startup

The following sleep states are not available on this system:
    Standby (S1)
	The system firmware does not support this standby state.
	This standby state is disabled when S0 low power idle is supported.

    Standby (S2)
	The system firmware does not support this standby state.
	This standby state is disabled when S0 low power idle is supported.

    Standby (S3)
	This standby state is disabled when S0 low power idle is supported.

    Hybrid Sleep
	Standby (S3) is not available.
	The hypervisor does not support this standby state.

Expected: Standby (S0 Low Power Idle) under "available". Here S3 is listed as "disabled when S0 low power idle is supported", so every sleep on this laptop is a Modern Standby sleep, and SleepStudy records it. If your list shows Standby (S3) as available instead, this manual does not apply.

Check when the laptop will hibernate on battery.
```
PS> powercfg /q SCHEME_CURRENT SUB_SLEEP HIBERNATEIDLE
Power Scheme GUID: 381b4222-f694-41f0-9685-ff5bb260df2e  (Balanced)
  GUID Alias: SCHEME_BALANCED
  Subgroup GUID: 238c9fa8-0aad-41ed-83f4-97be242c8f20  (Sleep)
    GUID Alias: SUB_SLEEP
    Power Setting GUID: 9d7815a6-7ee4-497e-8888-515a05f02364  (Hibernate after)
      GUID Alias: HIBERNATEIDLE
      Minimum Possible Setting: 0x00000000
      Maximum Possible Setting: 0xffffffff
      Possible Settings increment: 0x00000001
      Possible Settings units: Seconds
    Current AC Power Setting Index: 0x00000000
    Current DC Power Setting Index: 0x00000000
```
Expected: Current DC Power Setting Index: 0x00000000 turns off the "Hibernate after" timer, and only the timer. Any other value is the number of seconds after which the sleep turns into hibernation, and the session ends there with the exit reason Hibernate, or Shutdown (Failure modes, row 4). With hibernation enabled (step 1 lists Hibernate), Windows can still hibernate from its own standby battery budget: my first night ended that way after 8.69 h with HibernateTimeoutInSeconds 0 in its session metadata (Field notes, 2026-11-06). The restricted-standby budget can also end an early session of the night, so the start of a night can be split into two or more sessions (step 5: the 1.48 h and 0.27 h lines). My 56.92 h single session ran with hibernation turned off on the laptop (Field notes, 2026-11-09); no step here changes that.
Sleep on battery with the lid closed. Unplug the charger, note the charge level, close the lid, and leave it closed for a whole night (6 h or more). With hibernation enabled, the sleep can end early in Hibernate, or Shutdown (step 2). Open the lid the next morning and log in. No command: SleepStudy records the session by itself.

NoteShort sessions

Microsoft's SleepStudy page says "overall" modern standby sessions under ten minutes are not tracked and that periods of more than one hour reflect real use. My reports listed a 23 s Sleep session anyway, with no residency data (step 5), and a 16 min one that read higher than the long sleeps. Use one long session per test.
Generate the report in an elevated PowerShell (Microsoft's SleepStudy page asks for an Administrator Command Prompt; I ran it elevated and did not try without). An elevated shell often starts in C:\Windows\System32: cd into the working folder first, so the report lands next to the scripts.
```
PS> powercfg /sleepstudy /duration 7 /output sleepstudy.html
Sleep Study report saved to file path C:\Users\user\sleepstudy\sleepstudy.html.
```
Expected: one line naming the saved file, about 3-5 MB on my laptop. /duration 7 covers the last seven days (default three, at most 28). The report is a web page; its data is the JSON object var LocalSprData inside it, which the scripts read.
Print residency and drain for each battery sleep.
```
PS> python -I residency.py sleepstudy.html
2026-11-06T00:43 1.48 h  HW 98.7%  SW 98.8%  drain 582 mWh = 392 mW = 0.58 %/h  austerity=0  exit: Restricted Standby Battery Drain Budget Exceeded
2026-11-06T02:12 8.69 h  HW 99.8%  SW 99.8%  drain 2710 mWh = 312 mW = 0.46 %/h  austerity=1  exit: Hibernate, or Shutdown
no residency data: 2026-11-07T03:47:52Z
2026-11-07T03:48 0.27 h  HW 97.9%  SW 98.0%  drain 112 mWh = 421 mW = 0.62 %/h  austerity=0  exit: Restricted Standby Battery Drain Budget Exceeded
2026-11-07T04:04 56.92 h  HW 99.3%  SW 99.9%  drain 19234 mWh = 338 mW = 0.50 %/h  austerity=1  exit: Lid
2026-11-09T21:52 2.00 h  HW 98.1%  SW 98.1%  drain 574 mWh = 287 mW = 0.42 %/h  austerity=0  exit: Hibernate, or Shutdown
```
Expected: one line per type-2 Sleep session on battery (residency.py keeps Type 2 with OnAc false), oldest first, and a no residency data: line for a session that carries no residency fields (Failure modes, row 3). Last night's session is the newest line. Here the 2026-11-07 04:04 line is a 56.92 h sleep: HW low-power 99.3 %, 19234 mWh used, 338 mW on average, 0.50 % of a full charge per hour.

How to read one line:
- Time: the session start, local time, to the minute.
- HW: Info.HwLowPowerStateTime as a share of the session. This is the hardware DRIPS figure, the SoC in S0ix; it is the number that answers "did it reach S0ix". SW is the software DRIPS share (Info.SwLowPowerStateTime); Microsoft's page calls it the DRIPS rate without defining it further. On my sessions SW was never below HW (largest gap 0.6 points). Both fields are in 100 ns units, Duration is in microseconds; the script stops with unit check failed if HW time comes out more than 1 % longer than the session. The report's own page computes its HW: percentage from the same two fields and units, rounded to a whole percent.
- drain: EntryRemainingCapacity minus ExitRemainingCapacity, in mWh, the battery's own reading at entry and exit. mW is that energy over the session hours, %/h is the same against the full-charge capacity at entry. Microsoft computes its "change rate" the same way: energy change divided by duration.
- austerity: Info.AusterityMode from the session metadata. On my laptop it was 1 for a session that started right after a Restricted Standby Battery Drain Budget Exceeded exit, 0 otherwise; Microsoft's pages I read do not define it.
- exit: why the session ended. Lid is the lid opening; Restricted Standby Battery Drain Budget Exceeded and Hibernate, or Shutdown are Windows' own exits (Microsoft lists the names, without explaining the budgets).
Grade the session against Microsoft's colour bands. Microsoft's SleepStudy page grades a Sleep session of 2 minutes or more as: red when "the DRIPS rate is <80% or the drain rate is >= 1% per hour", yellow when "the DRIPS rate is between 80-94% or the drain rate is between .33-1% per hour", green in all other cases. The same section also says a Sleep session "is colored Grey, since there is no information to determine if it's a good or bad session based on drain or time spent in DRIPS"; I grade by the thresholds. The page does not say whether the DRIPS rate here is the software or the hardware figure; on my sessions they differ by at most 0.6 points, so the band does not change. My 56.92 h session: DRIPS 99.3 % is in the green range, and 0.50 %/h is in the yellow drain band, so the session as a whole falls in yellow.

(If HW is low) find what kept the SoC awake.

PS> python -I blockers.py sleepstudy.html
type,onac -> count, hours
  (0, False): 25, 4.16 h
  (0, True): 29, 22.03 h
  (1, False): 25, 0.57 h
  (1, True): 14, 25.03 h
  (2, False): 6, 69.37 h
  (2, True): 1, 0.12 h
…
screen-off sessions >= 10 min:
…
- 2026-11-05T19:43 type=1 ac=True 3.58 h drain=0 mWh Video Idle Timeout -> Input Keyboard
     99%    214.5 min  [PDC Phases] No CS Phase
      0%      0.4 min  [Audio Activity] Audio Active
      0%      0.0 min  [Audio Activity] Audio Streams
…
- 2026-11-07T04:04 type=2 ac=False 56.92 h drain=19234 mWh Transition To Sleep -> Lid
      0%      3.6 min  [FX Devices] PCI Express Root Port (\_SB.PC00.RP11)
      0%      2.4 min  [SOC Subsystems] NoHwDrips.C10
      0%      0.5 min  [Processors] CPU C0 Time.Non-attributed Time
      0%      0.2 min  [FX Devices] Intel(R) Wi-Fi 7 BE201 320MHz (\_SB.PC00.CNVW)
      0%      0.1 min  [PDC Phases] Low Power Phase
      0%      0.1 min  [FX Devices] USB xHCI Compliant Host Controller (\_SB.PC00.XHCI)
- 2026-11-09T15:09 type=1 ac=True 6.71 h drain=0 mWh Video Idle Timeout -> Lid
     99%    402.7 min  [PDC Phases] No CS Phase
- 2026-11-09T21:52 type=2 ac=False 2.00 h drain=574 mWh Transition To Sleep -> Hibernate, or Shutdown
      1%      1.6 min  [FX Devices] Intel(R) Wi-Fi 7 BE201 320MHz (\_SB.PC00.CNVW)
      0%      0.9 min  [FX Devices] PCI Express Root Port (\_SB.PC00.RP11)
      0%      0.4 min  [Activators] TCPIP
      0%      0.1 min  [PDC Phases] Low Power Phase
      0%      0.1 min  [SOC Subsystems] NoHwDrips.C10

Expected: the first table counts sessions by (type, on AC); type 2 is the Sleep session that residency.py reads, and (2, False) the ones on battery. All six of mine started with the lid closed (EventLog.LidState Closed in the session metadata); the script does not check the lid. The list below it prints, for every session of 10 minutes or more, the top six blockers with the share and minutes they were active. Its header says "screen-off sessions", but the list holds type-2 sleeps too: read the type= field of each entry. On a healthy sleep every blocker is a fraction of a minute to a few minutes: in the 56.92 h session the largest was the PCIe root port RP11 at 3.6 min. A blocker near 100 % in a type-2 session is the thing to fix. The two type-1 lines are display-off time on AC, not sleep; see Failure modes, rows 1 and 2.

GotchaRead type 2, not type 1

Type-1 sessions are screen-off periods, not sleep. On battery they last seconds to minutes and carry no residency figure; on AC the long ones (10 of the 14 in my 2026-11-10 report, each over 30 minutes) are display-off time in which the laptop never entered standby, and show No CS Phase at 99 %. None of them has a hardware residency field. Counting them gave me "HW DRIPS = 0, never reaches S0ix" on the first night, which was wrong.

Serve local GGUFs to opencode with llama-server router mode

Wed, 11 Nov 2026 00:00:00 +0000

A wrapper named oc starts a llama.cpp llama-server in router mode as a transient systemd --user unit, runs opencode against it as a local provider that lists only fully downloaded GGUFs, and stops the router when the last opencode exits.

Tested on: Ryzen 7 PRO 6850U laptop (32 GiB LPDDR5, Radeon 680M with 4 GiB carve-out + 13.58 GiB GTT), Fedora 44, kernel 7.2.10, Mesa 26.2.4 RADV, llama.cpp b11292 ubuntu-vulkan-x64 prebuilt, opencode 1.19.3, Python 3.14.8 (2026-10-21). That session's logs and output files were not kept: the outputs, times and speeds pasted below come from it and cannot be re-checked. The llama.cpp, opencode and npm facts were re-checked against the sources at the end on 2026-11-05.
Not verified: opencode 2.x (@opencode/cli), other GPUs, --sleep-idle-seconds 900 end to end (the sleep test used 60).
Needs: the b11292 Vulkan build unpacked in <ROOT>/bin/vulkan/, GGUF files in <ROOT>/models/, opencode installed from npm (opencode-ai), a login session (the unit is a user unit).
Time: 30 min, plus model downloads.
Risk: low. gen-opencode.py --apply rewrites ~/.config/opencode/opencode.json after copying it to a timestamped backup. The router is a transient unit: nothing is enabled, and it does not survive a reboot, or a logout unless lingering is on for your user (loginctl enable-linger).

Before you start#

Placeholders used below:
- <ROOT> — the working directory. Mine is ~/work/llm-testing; the scripts set it in one line each.
- <MODEL_ID> — a section name from models.ini, for example ornith-35b. It becomes the model id in the API and in opencode.

The Vulkan build answers from its own directory:

$ ~/work/llm-testing/bin/vulkan/llama-server --version
version: 0.5.0-dev (build 11292, commit 90c908d06)
built with GNU 11.4.0 for Linux x86_64

opencode is the npm build:
```
$ opencode --version
1.19.3
```

RADV sees the iGPU:

$ vulkaninfo --summary 2>/dev/null | grep -E 'deviceName|driverInfo'
	deviceName         = AMD Radeon Graphics (RADV REMBRANDT)
	driverInfo         = Mesa 26.2.4
	deviceName         = llvmpipe (LLVM 22.1.8, 256 bits)
	driverInfo         = Mesa 26.2.4 (LLVM 22.1.8)

Port 8080 is free (no output = nothing listens). Fedora also packages an older llama-server that must not be the one answering there:
```
$ ss -ltnH 'sport = :8080'
$ rpm -qf /usr/bin/llama-server
llama-cpp-b6153-3.fc44.x86_64
```

opencode has a global config file. gen-opencode.py reads it unconditionally and fails without it. If it is missing, create the minimal one:

mkdir -p ~/.config/opencode && test -f ~/.config/opencode/opencode.json || printf '{\n  "$schema": "https://opencode.ai/config.json"\n}\n' > ~/.config/opencode/opencode.json

Backup: gen-opencode.py --apply copies ~/.config/opencode/opencode.json to opencode.json.bak-<YYYYmmdd-HHMMSS> before writing. To be safe before the first run, take your own copy too: cp ~/.config/opencode/opencode.json ~/opencode.json.before-router.

Steps#

List every model in a download manifest with its exact size in bytes. gen-opencode.py reads <ROOT>/download_models.sh and treats a GGUF as complete only when its size matches. Each entry is one quoted line, "<hf-repo> <file>.gguf <bytes>". Two of my entries:
```
MODELS=(
  "unsloth/LFM2.5-8B-A1B-GGUF LFM2.5-8B-A1B-UD-IQ4_NL.gguf 4358844000"
  "ornith-ai/Ornith-1.5-35B-A3B-GGUF Ornith-1.5-35B-Q4_K_M.gguf 21713463040"
)
```
Expected: one line per GGUF you plan to serve. A file with no manifest entry passes on the magic bytes alone.

GotchaA partial download already starts with GGUF

On 2026-10-21 Qwen-AgentWorld-35B-A3B-UD-IQ3_S.gguf was 236,871,911 bytes of 14,984,322,304 mid-download and its first four bytes were already GGUF. A magic-byte check alone would have offered it to opencode. The size match is what rejects it.

Write <ROOT>/serve/models.ini. One [*] section with what every model shares, then one section per model; the section name is the model id. Keys are llama-server options without the leading dashes (short forms allowed). Use absolute model paths. An excerpt of mine:

; llama-server router presets (b11292 Vulkan) for opencode. Section name = model id in the API.
; Only models whose file exists are exposed to opencode (gen-opencode.py checks).
version = 1

[*]
; whole context for one opencode session; fit picks -ngl / n-cpu-moe to fit the 680M pool
c = 65536
np = 1
fit = on
fitt = 1024
ctk = q8_0
ctv = q8_0
fa = on
jinja = true
t = 8

[lfm2.5-8b]
model = /home/user/work/llm-testing/models/LFM2.5-8B-A1B-UD-IQ4_NL.gguf

[ornith-35b]
model = /home/user/work/llm-testing/models/Ornith-1.5-35B-Q4_K_M.gguf
; without it the CPU-side host buffer is too big for one RADV submission:
; "radv/amdgpu: Not enough memory for command submission" -> ErrorDeviceLost, even at -ngl 0
no-host = true
; MTP head (blk.40.nextn): n-max 2 = 22.4 vs 21.0 t/s (+7%, ~67% accepted); n-max 3 slower. 2026-10-21
spec-type = draft-mtp
spec-draft-n-max = 2
; coding sampling, as lukesdevlab's [ornith-1.5-35b-coding]
temp = 1.0
top-p = 0.95
min-p = 0.0
presence-penalty = 0.0

Expected: c in [*] is also the context limit opencode is told about (step 3). Write it as c: the script reads only that key, so ctx-size = … would give opencode a context of 0. np = 1 gives one slot, so the whole 64k context goes to the one opencode session.

Save <ROOT>/serve/gen-opencode.py. The listing below is my script as of 21:16 on 2026-10-21, the revision steps 4 and 5 ran; later revisions also write per-model thinking variants, which this manual does not cover. It writes active.ini (the presets whose GGUF is complete) and builds opencode's local provider from the same list. It locates everything relative to its own path; no edits needed.

#!/usr/bin/env python3
"""Build serve/active.ini (presets whose GGUF exists) and the opencode `local` provider from it.

  gen-opencode.py            write active.ini, print diff of ~/.config/opencode/opencode.json
  gen-opencode.py --apply    also write opencode.json (backup next to it first)
  gen-opencode.py --ini-only write active.ini only (used by `oc`)
"""
import difflib
import json
import re
import shutil
import sys
import time
from pathlib import Path

SERVE = Path(__file__).resolve().parent
SRC, ACTIVE = SERVE / "models.ini", SERVE / "active.ini"
OPENCODE = Path.home() / ".config/opencode/opencode.json"
BASE_URL = "http://127.0.0.1:8080/v1"
OUTPUT_TOKENS = 8192


def parse(path):
    """INI as ordered [(section or None, [lines])]; keeps comments, values as-is."""
    blocks, cur = [], (None, [])
    for line in path.read_text().splitlines():
        s = line.strip()
        if s.startswith("[") and s.endswith("]"):
            blocks.append(cur)
            cur = (s[1:-1], [])
        cur[1].append(line)
    blocks.append(cur)
    return blocks


def kv(lines):
    out = {}
    for line in lines:
        s = line.split(";", 1)[0].strip()
        if "=" in s:
            k, v = (x.strip() for x in s.split("=", 1))
            out[k] = v
    return out


def expected_sizes():
    """file name -> bytes, from download_models.sh (a partial download already starts with GGUF)."""
    script = SERVE.parent / "download_models.sh"
    out = {}
    for m in re.finditer(r'^\s*"\S+ (\S+\.gguf) (\d+)"', script.read_text(), re.M):
        out[m.group(1)] = int(m.group(2))
    return out


def is_gguf(p, sizes):
    try:
        with open(p, "rb") as f:
            if f.read(4) != b"GGUF":
                return False
        want = sizes.get(Path(p).name)
        return want is None or Path(p).stat().st_size == want
    except OSError:
        return False


def main():
    blocks = parse(SRC)
    ctx = int(next(kv(l) for s, l in blocks if s == "*").get("c", 0))
    keep, models = [], {}
    sizes = expected_sizes()
    for sec, lines in blocks:
        if sec in (None, "*"):
            keep.append(lines)
            continue
        path = kv(lines).get("model", "")
        if not is_gguf(path, sizes):
            print(f"skip {sec}: no complete GGUF at {path}", file=sys.stderr)
            continue
        keep.append(lines)
        models[sec] = {
            "name": f"{sec} (local)",
            "tool_call": True,
            "limit": {"context": ctx, "output": OUTPUT_TOKENS},
        }
    if not models:
        sys.exit("no usable models in models.ini")
    ACTIVE.write_text("\n".join(l for lines in keep for l in lines) + "\n")
    print(f"active.ini: {', '.join(models)}", file=sys.stderr)
    if "--ini-only" in sys.argv:
        return

    old = OPENCODE.read_text()
    cfg = json.loads(old)
    cfg.setdefault("provider", {})["local"] = {
        "npm": "@ai-sdk/openai-compatible",
        "name": "llama.cpp local (680M)",
        "options": {"baseURL": BASE_URL},
        "models": models,
    }
    new = json.dumps(cfg, indent=2) + "\n"
    if new == old:
        print("opencode.json: unchanged", file=sys.stderr)
        return
    sys.stdout.writelines(difflib.unified_diff(
        old.splitlines(True), new.splitlines(True), str(OPENCODE), str(OPENCODE) + " (new)"))
    if "--apply" in sys.argv:
        bak = OPENCODE.with_name(f"opencode.json.bak-{time.strftime('%Y%m%d-%H%M%S')}")
        shutil.copy2(OPENCODE, bak)
        OPENCODE.write_text(new)
        print(f"written; backup {bak}", file=sys.stderr)


if __name__ == "__main__":
    main()

Dry-run it and read the diff. Without flags it writes only active.ini and prints what it would change in opencode.json. This run used a copy of my serve/ directory and HOME pointed at a scratch directory holding that minimal three-line opencode.json, so the diff shows only the new provider:

$ python3 -I serve/gen-opencode.py
skip agentworld-35b-iq3: no complete GGUF at /home/user/work/llm-testing/models/Qwen-AgentWorld-35B-A3B-UD-IQ3_S.gguf
skip nail-35b: no complete GGUF at /home/user/work/llm-testing/models/Nail-Qwen3.6-35B-A3B-MTP-UD-IQ4_XS.gguf
skip nail-35b-iq3: no complete GGUF at /home/user/work/llm-testing/models/Nail-Qwen3.6-35B-A3B-MTP-UD-IQ3_S.gguf
skip fablevibes-14b: no complete GGUF at /home/user/work/llm-testing/models/Qwen3.6-14B-A3B-FableVibes-Q4_K_M.gguf
skip mellum-12b: no complete GGUF at /home/user/work/llm-testing/models/Mellum2.1-12B-A2.5B-Thinking-Q4_K_M.gguf
skip swift-9b: no complete GGUF at /home/user/work/llm-testing/models/Swift-1.5-Ornith-9B-Q4_K_M.gguf
active.ini: lfm2.5-8b, gemma4-26b, lfm2.5-24b, ornith-35b, agentworld-35b, swift-27b
--- <SCRATCH_HOME>/.config/opencode/opencode.json
+++ <SCRATCH_HOME>/.config/opencode/opencode.json (new)
@@ -1,3 +1,62 @@
 {
-  "$schema": "https://opencode.ai/config.json"
+  "$schema": "https://opencode.ai/config.json",
+  "provider": {
+    "local": {
+      "npm": "@ai-sdk/openai-compatible",
+      "name": "llama.cpp local (680M)",
+      "options": {
+        "baseURL": "http://127.0.0.1:8080/v1"
+      },
+      "models": {
+        "lfm2.5-8b": {
+          "name": "lfm2.5-8b (local)",
+          "tool_call": true,
+          "limit": {
+            "context": 65536,
+            "output": 8192
+          }
+        },
…
+        "swift-27b": {
+          "name": "swift-27b (local)",
+          "tool_call": true,
+          "limit": {
+            "context": 65536,
+            "output": 8192
+          }
+        }
+      }
+    }
+  }
 }

Expected: one skip line per preset whose file is missing or incomplete, an active.ini: line naming the rest, and a diff that adds or replaces only provider.local. Other keys in your opencode.json stay as they are.

Apply it.
```
python3 -I <ROOT>/serve/gen-opencode.py --apply
```
Expected: the same diff, then written; backup <path> on stderr. In the scratch copy from step 4:
```
active.ini: lfm2.5-8b, gemma4-26b, lfm2.5-24b, ornith-35b, agentworld-35b, swift-27b
written; backup <SCRATCH_HOME>/.config/opencode/opencode.json.bak-20261021-213941
```
At about 21:40 on 2026-10-21, ~/.config/opencode/ on my real config held two such backups. The first is from my first --apply at 13:49:05; the second was written at 20:41:34 and at that time differed from the live file only by "lsp": true:
```
$ ls -l ~/.config/opencode/opencode.json.bak-*
-rw-------. 1 user user  515 Oct  5 16:08 /home/user/.config/opencode/opencode.json.bak-20261021-134905
-rw-------. 1 user user 1684 Oct 21 13:49 /home/user/.config/opencode/opencode.json.bak-20261021-204134
```
The name carries the time of the write; the listed date is the old file's, because the copy keeps its mtime. Re-run --apply whenever a download finishes: oc refreshes only active.ini, not opencode's model list.
Write the per-session config <ROOT>/serve/opencode-local.json. oc passes it as OPENCODE_CONFIG, so plain opencode keeps your global default model.
```
{
  "$schema": "https://opencode.ai/config.json",
  "model": "local/ornith-35b",
  "small_model": "local/ornith-35b",
  "lsp": true
}
```
Expected: small_model is the same model on purpose. opencode uses small_model for side tasks such as session titles; with --models-max 1 a different model there would unload the main one on every such request. "lsp": true turns on opencode's built-in language servers, which are off when the key is absent (checked in Verify).

Save the wrapper <ROOT>/serve/oc, make it executable and put it on PATH. The ROOT= line is the one to change; on my box it is an absolute path. This is my serve/oc at commit 2a85e80 (file time 21:45 on 2026-10-21), verbatim except for the ROOT= line. The oc sessions in the field notes (20:26–20:56) ran earlier revisions of it; the health-check refusal and the SDKMAN block came later. The SDKMAN block has nothing to do with the router: it puts Java tools on PATH for opencode's non-interactive shell, and you can drop it. The "+32.5 s per session" in the compat comment is one n=1 A/B on one task; see the 20:30–20:36 field note before you rely on it.

#!/usr/bin/env bash
# opencode with the local llama.cpp router: start it if down, stop it when the last opencode exits.
# Usage: oc [opencode args...]    Env: OC_KEEP=1 leaves the router running on exit,
#        OC_CLAUDE=1 keeps Claude Code compat (CLAUDE.md, ~/.claude/skills) — see occ.
set -uo pipefail
ROOT="$HOME/work/llm-testing"
UNIT=llm-router
PORT=8080
LOG="$ROOT/serve/router.log"

if ! systemctl --user is-active -q "$UNIT"; then
  # a running bench would be skewed by the router, and the router by the bench
  if pgrep -x llama-bench >/dev/null && [ "${FORCE:-}" != 1 ]; then
    echo "oc: llama-bench is running (bench-all?); FORCE=1 to start the router anyway" >&2
    exit 1
  fi
  # opencode would otherwise talk to whatever else answers there (e.g. Fedora's b6153 llama-server)
  if ss -ltnH "sport = :$PORT" | command grep -q .; then
    echo "oc: port $PORT is taken by something other than $UNIT:" >&2
    ss -ltnpH "sport = :$PORT" >&2
    exit 1
  fi
  python3 -I "$ROOT/serve/gen-opencode.py" --ini-only || exit 1
  systemctl --user reset-failed "$UNIT" 2>/dev/null
  systemd-run --user --unit="$UNIT" -q \
    -p StandardOutput=append:"$LOG" -p StandardError=append:"$LOG" \
    "$ROOT/bin/vulkan/llama-server" --host 127.0.0.1 --port "$PORT" \
    --models-preset "$ROOT/serve/active.ini" --models-max 1 --sleep-idle-seconds 900 || exit 1
  for _ in $(seq 60); do
    curl -sf "http://127.0.0.1:$PORT/health" >/dev/null && break
    sleep 0.5
  done
  curl -sf "http://127.0.0.1:$PORT/health" >/dev/null || { echo "oc: router not healthy, see $LOG" >&2; exit 1; }
elif ! curl -sf "http://127.0.0.1:$PORT/health" >/dev/null; then
  # the unit name is also used by benchmark/experiment jobs (so queued tests wait); do not touch them
  echo "oc: unit $UNIT is active but nothing answers on :$PORT (an experiment is using the name?)." >&2
  echo "    Wait for it to finish: systemctl --user status $UNIT" >&2
  exit 1
fi

# opencode's shell is non-interactive: the SDKMAN lazy-load functions from ~/.bashrc (mvn, mn, ...)
# do not exist there, so put the current SDKMAN tools on PATH explicitly
SDK="${SDKMAN_DIR:-$HOME/.sdkman}/candidates"
for c in java maven mvnd gradle micronaut springboot jbang; do
  [ -d "$SDK/$c/current/bin" ] && PATH="$SDK/$c/current/bin:$PATH"
done
[ -d "$SDK/java/current" ] && export JAVA_HOME="$SDK/java/current"
export PATH

# default model only for oc sessions (Ornith); plain `opencode` keeps the global default.
# small_model = same model: with --models-max 1 a different one would swap on every title request.
# no Claude Code compat: measured +3.5k prompt tokens (17.2k vs 13.7k) = +32.5 s per session on Ornith;
# `occ` (OC_CLAUDE=1) keeps it on for comparison; tui.json adds the local-LLM status (sidebar + home footer)
claude_off=1
[ "${OC_CLAUDE:-}" = 1 ] && claude_off=0
OPENCODE_DISABLE_CLAUDE_CODE=$claude_off \
OPENCODE_TUI_CONFIG="$ROOT/serve/opencode-tui/tui.json" \
OPENCODE_CONFIG="$ROOT/serve/opencode-local.json" opencode "$@"
rc=$?

# process name: opencode.exe up to 1.18, opencode since 1.19
if [ "${OC_KEEP:-}" != 1 ] && ! pgrep -x opencode >/dev/null && ! pgrep -x opencode.exe >/dev/null; then
  systemctl --user stop "$UNIT"
fi
exit "$rc"

The OPENCODE_TUI_CONFIG line belongs to ocmon, which documents it. Without ocmon, delete that line [UNVERIFIED: oc without the OPENCODE_TUI_CONFIG line not run]. The optional occ keeps opencode's Claude Code compatibility on, for comparison:

#!/usr/bin/env bash
# oc with opencode's Claude Code compat ON (reads CLAUDE.md, ~/.claude/skills): for comparing against plain oc.
OC_CLAUDE=1 exec "$(dirname "$(readlink -f "$0")")/oc" "$@"

Install both:

chmod +x <ROOT>/serve/oc <ROOT>/serve/occ
ln -s <ROOT>/serve/oc ~/.local/bin/oc
ln -s <ROOT>/serve/occ ~/.local/bin/occ

Expected: occ finds oc next to its own target through readlink -f, so the symlinks work.

NoteThe router listens on 127.0.0.1 only

Every start logs security: no API key is set and CORS allows all origins and router mode enabled - do not expose to untrusted environments. oc binds --host 127.0.0.1; keep it that way.

Start a session with oc and send the first prompt. The router starts with no model loaded; the first request loads ornith-35b. From <ROOT>/serve/router.log of one such session:

0.00.118.715 I srv    operator(): Available models (5):
0.00.118.718 I srv    operator():   [    preset] gemma4-26b
0.00.118.719 I srv    operator():   [    preset] lfm2.5-24b
0.00.118.719 I srv    operator():   [    preset] lfm2.5-8b
0.00.118.720 I srv    operator():   [    preset] ornith-35b
0.00.118.720 I srv    operator():   [    preset] swift-27b
0.00.118.789 W srv  llama_server: security: no API key is set and CORS allows all origins (see https://github.com/ggml-org/llama.cpp/pull/25655)
0.00.118.793 W srv  llama_server: security: router mode enabled - do not expose to untrusted environments
0.00.118.794 I srv  llama_server: starting server in router mode. models will be automatically loaded on-demand
0.00.120.996 I srv  llama_server: listening on http://127.0.0.1:8080
0.00.120.999 W srv  llama_server: notice: server default port will be changed to :9931 in a future release (ref: https://github.com/ggml-org/llama.cpp/pull/26508)
0.00.486.331 I srv          join: request for name=ornith-35b queued at position 1
0.00.486.340 I srv  ensure_model: waiting until model name=ornith-35b is fully loaded...
0.00.486.345 I srv  ensure_model: slot available, loading queued model name=ornith-35b
0.00.486.475 I srv          load: spawning server instance with name=ornith-35b on port 34017
…
[34017] 0.21.081.884 I srv  llama_server: model loaded
[34017] 0.21.081.890 I srv  llama_server: listening on http://127.0.0.1:34017

Expected: listening on http://127.0.0.1:8080 within the 30 s oc waits, then one spawning server instance per model switch. The child's own timestamps ([34017] 0.21…) count from the child's start: Ornith took 21 s to load here.

The child is the router's own binary, not whatever llama-server is first on PATH. At commit 90c908d06, server-models.cpp resolves /proc/self/exe (line 461) and spawns that; the log shows it as the first argument:

0.07.495.373 I srv          load: spawning server instance with args:
0.07.495.382 I srv          load:   /home/user/work/llm-testing/bin/vulkan/llama-server

So Fedora's llama-cpp-b6153 package never runs as a child.

Quit opencode. When no process named opencode (or opencode.exe) is left, oc runs systemctl --user stop llm-router. The router log of the same session ends with:
```
3.36.594.919 I srv    operator(): operator(): cleaning up before exit...
3.36.594.990 I srv    unload_all: stopping model instance name=ornith-35b
[34017] 3.36.101.047 I srv    operator(): exit command received, exiting...
[34017] 3.36.103.853 W ~llama_context:    Vulkan0 compute buffer size of 282.0000 MiB, does not match expectation of 275.5000 MiB
3.37.349.512 I srv           run: instance name=ornith-35b exited with status 0
```
Expected: exited with status 0. Start opencode with OC_KEEP=1 oc to leave the router running after exit.
WarningThe unit name is global to your user session

Other jobs can run under the transient unit name llm-router too. On 2026-10-21 at 21:15 another experiment on the same box started its benchmark script under that name: the unit was active, port 8080 had no listener, and curl -sf http://127.0.0.1:8080/health exited 7. An earlier oc checked only systemctl --user is-active, so in that state it would have skipped the start and, on exit, stopped the other job's unit. The oc above refuses instead. In a live run at about 21:45 that day, with an experiment holding the unit and nothing on :8080, it printed this and exited 1 before starting opencode, and the unit stayed active:
```
oc: unit llm-router is active but nothing answers on :8080 (an experiment is using the name?).
    Wait for it to finish: systemctl --user status llm-router
```
Use llm-router for the router only, or change UNIT=.

Show the local llama.cpp router's state in opencode with ocmon

Wed, 11 Nov 2026 00:00:00 +0000

Install ocmon, run it in a terminal, and wire it into opencode as a sidebar panel and home-screen footer line that show what the local llama.cpp router is doing: loading, ready, sleeping or failed, prompt progress with an ETA, generation speed, and the iGPU's load, memory, power and temperatures.

Tested on: Ryzen 7 PRO 6850U laptop (Radeon 680M), Fedora 44, Go 1.26.9, opencode 1.19.3, llama.cpp b11292 Vulkan build, ocmon commit f067f45 (2026-10-21). The logs of that evening were not kept: on 2026-11-05 every flag, default and output format below was checked against the source at f067f45, and the router-down outputs in steps 3, 4 and 9 were re-run on another AMD box (Ryzen 7 8845HS, Radeon 780M). Version: steps 1 and 5 pin f067f45. The repository's main moved on the next day (new -list, -load, -unload, -stop flags; an unreadable log no longer gives the error state; different load-failure text), so @latest installs a version this manual does not describe.
Not verified: the public plugin with its tui.json options (steps 6–8) was not run in opencode; what ran on this box was an earlier copy of the same plugin with the binary path written into the file. No request ran against the public build: the router was down, so its busy-state output comes from replaying my router log against a stub GET /models (Field notes). The waybar module (step 9) was not run. Toasts were not captured.
Needs: the router and the oc wrapper from Serve local models to opencode with the llama.cpp router, with the router's output going to a log file; Go 1.24 or newer; an AMD GPU on the amdgpu driver for the hardware line (everything else works without it).
Time: 20 min.
Risk: low. ocmon only reads the log, sysfs and the router's GET /models; it calls /slots only while a request is already running, with autoload=false. It writes one file, ~/.cache/ocmon/gen.json. The opencode change is one environment variable in your wrapper.

Before you start#

The router writes llama.cpp's slot timing lines to its log. ocmon reads them for prompt progress and speeds; check the log has them (after at least one prompt has run):
```
grep -m1 'prompt processing, n_tokens' <ROUTER_LOG>
```
Expected: one slot print_timing line with prompt processing, n_tokens = …, progress = …; the Verify section shows one from my log. The [<port>] prefix on it is the port of the model instance that wrote the line. Lines without it come from the router itself. Without a log, ocmon shows model states only.

Check that the amdgpu device exposes the files ocmon reads:

$ grep -l amdgpu /sys/class/drm/card*/device/uevent
/sys/class/drm/card1/device/uevent
$ ls /sys/class/drm/card1/device/gpu_busy_percent /sys/class/drm/card1/device/mem_info_gtt_used /sys/class/drm/card1/device/mem_info_vram_used
/sys/class/drm/card1/device/gpu_busy_percent
/sys/class/drm/card1/device/mem_info_gtt_used
/sys/class/drm/card1/device/mem_info_vram_used

ocmon picks the first /sys/class/drm/card*/device that has gpu_busy_percent; -dev overrides it. Power and temperatures come from the hwmon sensors named amdgpu and k10temp, found by name.

Check Go (the module's go.mod says go 1.24): go version must print go1.24 or newer.
Placeholders used below:
- <ROOT>: the prerequisite manual's working directory (its oc wrapper is <ROOT>/serve/oc).
- <ROUTER_LOG>: absolute path of the router's log file (in the prerequisite manual, oc appends the router's stdout and stderr to it).
- <OCMON>: absolute path of the installed binary, by default ~/go/bin/ocmon, written out in full: the plugin starts it with Node's execFile, which does not expand ~.
- <SRC>: absolute path of a directory you create for the plugin file (step 5).
- <TUI_JSON>: the tui.json used only by the wrapper. The prerequisite manual's oc already points OPENCODE_TUI_CONFIG at $ROOT/serve/opencode-tui/tui.json; use that path, or change the line in step 7.
Backups: copy your oc wrapper before step 7 (cp <ROOT>/serve/oc <ROOT>/serve/oc.bak, where <ROOT> is the prerequisite manual's working directory). Nothing else is changed. Your global ~/.config/opencode/tui.json is not touched.

Steps#

Install ocmon at the tested version. The source is public at gitlab.com/myrants-oss/ocmon (MIT), and go install fetches it from the Go module proxy, so there is nothing to clone. It is one main.go with no dependencies outside the standard library. The version string pins commit f067f45 (Scope, Version):
```
go install gitlab.com/myrants-oss/ocmon@v0.0.0-20261021182238-f067f458af51
```
Without GOBIN the binary goes to ~/go/bin. On 2026-10-21 @latest still resolved to this version; I set GOBIN to keep it out of my PATH for the test:
```
$ GOBIN="$PWD/bin" go install gitlab.com/myrants-oss/ocmon@latest
$ go version -m bin/ocmon | head -3
bin/ocmon: go1.26.9-X:nodwarf5
	path	gitlab.com/myrants-oss/ocmon
	mod	gitlab.com/myrants-oss/ocmon	v0.0.0-20261021182238-f067f458af51	h1:ruhYlOpc4JPqKYVhtGPKsxWarAhRMrV0c4OZqgbGh0A=
```
Expected: go install prints nothing, or one go: downloading … line on the first fetch, and go version -m names the module with the v0.0.0-20261021182238-f067f458af51 version.
Check the flags.
```
$ bin/ocmon -h
Usage of bin/ocmon:
  -dev string
    	amdgpu sysfs device dir (env OCMON_DEV; default: first card with gpu_busy_percent) (default "/sys/class/drm/card1/device")
  -i duration
    	refresh interval (default 1s)
  -json
    	print one JSON object (class, line, side, detail) and exit
  -log string
    	router log file (env OCMON_LOG); without it only model states are shown
  -url string
    	router base URL (env OCMON_URL) (default "http://127.0.0.1:8080")
  -waybar
    	print one JSON line for a waybar custom module and exit
```
Flags win over the OCMON_* variables. The README.md in the module directory (step 5) has the full table; the README on the repository page describes the later version. What each part reads:
- -url: GET <url>/models on every refresh, and GET <url>/slots?model=<name>&autoload=false only while the log shows a running request (for the live generated-token count).
- -log: the last 1 MiB of the log on every refresh.
- -dev: gpu_busy_percent, mem_info_gtt_used, mem_info_vram_used. A missing file prints ?. The busy figure is one reading per refresh, not an average.
- -waybar prints {"class","text","tooltip"} with an empty text when the router is down; -json prints {"class","line","side","detail"}, where side holds the short lines for opencode's sidebar.
Expected: the six flags above, with the env names in the help text.

Run it once in JSON mode.

<OCMON> -json -log <ROUTER_LOG>

With no router on port 8080 (re-run 2026-11-05 on the 780M box):

$ bin/ocmon -json
{"class":"down","detail":["GPU 27%  GTT 1772 MiB  VRAM 475 MiB  40.1 W  CPU 75°C  GPU 56°C","router: Get \"http://127.0.0.1:8080/models\": dial tcp 127.0.0.1:8080: connect: connection refused"],"line":"router down","side":null}

Expected: one JSON line whose class is one of the states below. error with router up, log unreadable means the -log path is wrong (Failure modes, row 2).

`class`	When	`line` printed
`down`	`GET /models` fails	`router down`
`error`	log unreadable, or the log shows a load error and no `model loaded` for the current instance	`router up, log unreadable` / `<model> FAILED to load (see router.log)`
`idle`	every model is `unloaded`	`idle, no model loaded`
`loading`	`/models` says `loading`	`ornith-35b loading…`
`sleeping`	`/models` says `sleeping`	`ornith-35b sleeping (memory freed)`
`loaded`	loaded, but no `-log`, or the newest instance in the log is another model	`ornith-35b loaded`
`prompt`	request running, prompt not finished	`ornith-35b reading prompt ██░░░░░░░░░░░░░░░░░░ 11% 2048/18618 tok 145 t/s · ~114 s left`
`gen`	request running, prompt done	`<model> generating <n> tok <rate> t/s` (`…` instead of the rate on the first sample)
`ready`	loaded, no request running	`ornith-35b ready last: prompt 21 tok @ 11 t/s, gen 263 tok @ 17.7 t/s, ctx 18353`

The line values with a model name come from a replay of my router log through this binary (Field notes, 21:24); that output was not kept, and each value matches the format string in the source. idle, gen and the load-failure text are from the source and were not observed. On a load failure the sidebar's second line is the fixed text see serve/router.log, whatever your log path is.

The prompt total is n_tokens / progress from the newest progress line, and the ETA is the remaining tokens divided by the running rate. llama.cpp prints progress with two decimals, so both are estimates (Field notes: 114 s shown, about 139 s real).

Run the full-screen view.
```
<OCMON> -log <ROUTER_LOG>
```
One frame with no router on port 8080 (re-run 2026-11-05 on the 780M box):
```
ocmon  11:05:12   (ctrl-c to quit)

  router down

  GPU 27%  GTT 1772 MiB  VRAM 475 MiB  46.9 W  CPU 75°C  GPU 56°C
  router: Get "http://127.0.0.1:8080/models": dial tcp 127.0.0.1:8080: connect: connection refused
```
Expected: the screen redraws every -i (1 s). The line under the clock is the same line as in JSON mode; below it are the hardware line and either the router error or the model list (models: <id>:<status> …).

GotchaCtrl-C leaves the cursor hidden

ocmon hides the cursor and restores it in a deferred call, which does not run when Ctrl-C or SIGTERM kills the process. Run tput cnorm afterwards to get the cursor back.
Get the plugin file. The opencode plugin is opencode-plugin/llm-status.tsx in the module. Commit f067f45 is no longer on the repository's branch, so take it from the same module version as the binary:
```
go mod download -json gitlab.com/myrants-oss/ocmon@v0.0.0-20261021182238-f067f458af51
```
It prints a JSON object whose "Dir" is the module's directory in Go's module cache (read-only). Copy the plugin out of it, with <DIR> = that Dir value:
```
mkdir -p <SRC>/opencode-plugin
cp <DIR>/opencode-plugin/llm-status.tsx <SRC>/opencode-plugin/
```
Expected: <SRC>/opencode-plugin/llm-status.tsx exists, from the same version as the binary.

What it does: runs <bin> -json <args…> every intervalMs (skipping a tick while the previous call still runs, 3 s timeout), fills two opencode slots, sidebar_content (session view) and home_footer (home screen), and hides both when ocmon fails or reports down. It raises a toast on three transitions: loading to ready, any state to sleeping, any state to error. opencode loads the .tsx file as it is; there is no build step.
Write <TUI_JSON> with the plugin's absolute path and its options. [UNVERIFIED: this options form was not run in opencode on this box.]
```
{
  "$schema": "https://opencode.ai/tui.json",
  "plugin": [
    ["<SRC>/opencode-plugin/llm-status.tsx",
     { "bin": "<OCMON>", "args": ["-log", "<ROUTER_LOG>"], "intervalMs": 1000 }]
  ]
}
```
bin defaults to ocmon from opencode's PATH; an absolute path avoids depending on it. args go after -json; add "-url", "…" if the router is not on 127.0.0.1:8080. intervalMs below 250 is ignored (1000 is used).
Pass that file to opencode from your wrapper only. The opencode call in oc must carry OPENCODE_TUI_CONFIG=<TUI_JSON>. The prerequisite manual's oc already has the line; if your wrapper does not, add it. The end of my oc:
```
OPENCODE_DISABLE_CLAUDE_CODE=$claude_off \
OPENCODE_TUI_CONFIG="$ROOT/serve/opencode-tui/tui.json" \
OPENCODE_CONFIG="$ROOT/serve/opencode-local.json" opencode "$@"
```
opencode's docs say OPENCODE_TUI_CONFIG points to a custom TUI config file. [UNVERIFIED: that opencode 1.19.3 merges it with the global tui.json, so global TUI settings still apply, was not checked against its source.] A plain opencode (cloud sessions) does not get the variable and shows no panel.
Start oc and send a prompt. [UNVERIFIED: with the public plugin and options form; see Scope.] Expected: the home screen shows local LLM: <line> at the bottom, and the session sidebar shows a Local LLM section (see Verify).
Optional: add a waybar module. [UNVERIFIED: not run on this box; there is no ocmon module in my waybar config.] Built from the waybar 0.15.0 waybar-custom(5) man page. Add "custom/ocmon" to one of the modules-* lists and define it:
```
"custom/ocmon": {
  "exec": "<OCMON> -waybar -log <ROUTER_LOG>",
  "return-type": "json",
  "interval": 1,
  "hide-empty-text": true
}
```
class becomes a CSS class, so #custom-ocmon.prompt, #custom-ocmon.error and the other class names from step 3 can be styled in style.css. -waybar output with no router on port 8080 (re-run 2026-11-05 on the 780M box):
```
$ bin/ocmon -waybar
{"class":"down","text":"","tooltip":"GPU 27%  GTT 1772 MiB  VRAM 475 MiB  46.4 W  CPU 75°C  GPU 56°C\nrouter: Get \"http://127.0.0.1:8080/models\": dial tcp 127.0.0.1:8080: connect: connection refused"}
```

Build a static native Micronaut image on FROM scratch

Sun, 08 Nov 2026 00:00:00 +0000

This builds a Micronaut 4.10.21 service into a statically linked (musl) GraalVM native executable inside Docker and ships it in a FROM scratch image with nothing else in it; the end state is a container that starts in 24 ms and answers /health.

Tested on: Fedora Linux 44, kernel 7.2.10-200.fc44.x86_64, Docker Engine 29.9.1 (rootful), builder ghcr.io/graalvm/native-image-community:21-muslib-ol9 (GraalVM CE 21.0.2), Micronaut 4.10.21 (2026-10-20).
Not verified: GraalVM 25 musl builders, TLS calls out of the scratch image (no CA bundle), a non-root USER, stripping the binary.
Needs: x86_64 Linux with Docker; about 5 GB of free RAM for native-image (peak RSS 4.22 to 4.55 GB here); network access to launch.micronaut.io, ghcr.io and Maven Central; curl, unzip, file, ss.
Time: 15 min. The build itself took 58 s with a cached Maven layer and 1m42s cold; the one-time pull of the builder image (531MB compressed) was not timed separately; the rest is reading and typing.
Risk: none to the host. The manual creates one image and one container bound to 127.0.0.1:8080; Undo removes both.

Before you start#

No placeholders. Every name below is literal: project demo-service, image demo-service:static, container demo-static. Run Step 1 in an empty directory; every later command runs inside demo-service/.

Docker is up and rootful. A rootless daemon lists name=rootless under security options.

$ docker info --format '{{.ServerVersion}} {{.OperatingSystem}} {{.Architecture}} {{.SecurityOptions}}'
29.9.1 Fedora Linux 44 (KDE Plasma Desktop Edition) x86_64 [name=seccomp,profile=builtin name=cgroupns]

This manual was not run on rootless Docker or Podman.

Enough memory. native-image used 4.22 to 4.55 GB peak RSS across two builds of this project.

$ free -g | head -2
               total        used        free      shared  buff/cache   available
Mem:              30           4           5           0          21          25

The builder image runs and is the version you expect. The first run also prints pull progress.

$ docker run --rm ghcr.io/graalvm/native-image-community:21-muslib-ol9 --version
native-image 21.0.2 2024-01-16
GraalVM Runtime Environment GraalVM CE 21.0.2+13.1 (build 21.0.2+13-jvmci-23.1-b30)
Substrate VM GraalVM CE 21.0.2+13.1 (build 21.0.2+13, serial gc)

Nothing to back up, but nothing to overwrite either. Check that the tag doesn't exist yet and port 8080 is free.

$ docker image ls demo-service; ss -ltn "sport = :8080"
IMAGE   ID             DISK USAGE   CONTENT SIZE   EXTRA
State Recv-Q Send-Q Local Address:Port Peer Address:Port

Expected: header lines only.

Steps#

Generate the project from Micronaut Launch, with the management feature for /health.

$ curl -s -o demo-service.zip 'https://launch.micronaut.io/create/default/com.example.demo-service?lang=JAVA&build=MAVEN&test=JUNIT&javaVersion=JDK_21&features=management' && unzip -q demo-service.zip && ls demo-service
aot-jar.properties
micronaut-cli.yml
mvnw
mvnw.bat
pom.xml
README.md
src

Expected: a Maven project with mvnw. Then cd demo-service; every later command runs there.

Pin the Micronaut platform to 4.10.21. Launch generated 5.2.2 on 2026-10-20; this manual is verified on 4.10.21. The version sits on two lines: the parent and micronaut.version.

$ grep -n '5\.2\.2' pom.xml
13:    <version>5.2.2</version>
20:    <micronaut.version>5.2.2</micronaut.version>
$ sed -i 's|5\.2\.2|4.10.21|g' pom.xml && grep -n '4\.10\.21' pom.xml
13:    <version>4.10.21</version>
20:    <micronaut.version>4.10.21</micronaut.version>

Expected: the same two lines, now 4.10.21. If Launch prints another version, replace that one instead.

Add a controller that reports the VM it runs on.

cat > src/main/java/com/example/HelloController.java <<'EOF'
package com.example;

import io.micronaut.http.annotation.Controller;
import io.micronaut.http.annotation.Get;

@Controller("/hello")
public class HelloController {
    @Get
    public String hello() {
        return "vm=" + System.getProperty("java.vm.name");
    }
}
EOF

$ ls src/main/java/com/example
Application.java
HelloController.java

Write the Dockerfile and .dockerignore. The builder is the muslib variant, which ships the musl toolchain. -Dpackaging=native-image makes Maven build the executable, and -DbuildArgs=--static,--libc=musl reaches native-image as two separate options. The runtime stage holds nothing but the binary.

cat > Dockerfile <<'EOF'
FROM ghcr.io/graalvm/native-image-community:21-muslib-ol9 AS builder
WORKDIR /app
COPY .mvn/ .mvn/
COPY mvnw pom.xml ./
RUN ./mvnw -q dependency:resolve
COPY src/ src/
RUN ./mvnw package -Dpackaging=native-image -DskipTests \
      -DbuildArgs=--static,--libc=musl

FROM scratch
COPY --from=builder /app/target/demo-service /demo-service
EXPOSE 8080
ENTRYPOINT ["/demo-service"]
EOF
printf 'target/\n' > .dockerignore

$ ls -A
aot-jar.properties
Dockerfile
.dockerignore
.gitignore
micronaut-cli.yml
.mvn
mvnw
mvnw.bat
pom.xml
README.md
src

Build the image.

$ time docker build --progress=plain -t demo-service:static .
…
#11 13.39 GraalVM Native Image: Generating 'demo-service' (static executable)...
#11 14.95  Java version: 21.0.2+13, vendor version: GraalVM CE 21.0.2+13.1
#11 14.95  C compiler: x86_64-linux-musl-gcc (linux, x86_64, 10.2.1)
#11 14.95  Garbage collector: Serial GC (max heap size: 80% of RAM)
…
#11 54.18 Produced artifacts:
#11 54.18  /app/target/demo-service (executable)
#11 54.18 Finished generating 'demo-service' in 41.5s.
#11 54.80 [INFO] BUILD SUCCESS
#11 DONE 54.9s
…
#13 naming to docker.io/library/demo-service:static done
…
real	0m57.948s

Expected: (static executable) and x86_64-linux-musl-gcc in the header. Plain (executable) with gcc (redhat, …) means a dynamic glibc binary that won't start on scratch (Failure modes, row 1). This run reused a cached Maven layer (#9 CACHED); a cold-cache build takes longer, see Field notes.

Check the image size.
```
$ docker image ls demo-service:static
IMAGE                 ID             DISK USAGE   CONTENT SIZE   EXTRA
demo-service:static   54933f63f033       97.3MB         24.4MB        
```
Expected: roughly these numbers. On Docker 29 DISK USAGE counts the binary plus CONTENT SIZE: docker image inspect --format '{{.Size}}' gave 97,325,596 bytes, the binary is 72,879,936 bytes (Verify), and the difference, 24,445,660 bytes, is the 24.4MB in CONTENT SIZE.

Start the container.

$ docker run -d --name demo-static -p 127.0.0.1:8080:8080 demo-service:static
891b261e529be8ce923b8df27d55cc5c2599fd7a97edc022a2945c177a8abf3c
$ sleep 2; docker logs demo-static 2>&1 | grep -o 'Startup completed in .*'
Startup completed in 24ms. Server Running: http://891b261e529b:8080

Expected: a startup line in tens of milliseconds. An exec /demo-service: no such file or directory instead is row 1 of Failure modes.

Install Window Maker as a startx session on Gentoo

Wed, 04 Nov 2026 00:00:00 +0000

Install Window Maker 0.96.0 and four dockapps on Gentoo, seed the per-user defaults, autostart the dockapps in the Dock, and give startx a client script that runs wmaker. End state: the packages are installed, ~/GNUstep is seeded, and startx ~/.xinitrc-windowmaker is ready to run. I logged into it on 2026-11-02 (Field notes).

Tested on: ThinkPad Yoga 370 (20JH, i5-7300U), Gentoo amd64 23.0 desktop profile (default/linux/amd64/23.0/desktop), OpenRC, x86-64-v3 binhost; x11-wm/windowmaker-0.96.0-r1, x11-base/xorg-server-21.1.24, x11-apps/xinit-1.4.4 (2026-11-02).
Not verified: the minimal client script (step 8) and the rollback. Everything else was checked structurally (packages, files, sh -n, wdread, the tty menu branch with a stub startx) and then in a real login on 2026-11-02 (Field notes).
Needs: root (sudo) for emerge and /etc/portage; a working Xorg with startx (x11-apps/xinit); a tty login. Optional: OpenRC user services and a tty session menu, which is how my box starts sessions (steps 9 to 11); step 10's script also sources my ~/.config/x11/common, which this manual does not create.
Time: 15 min: my first run took 13 min, from writing package.use (00:19) to the autostart lines (00:32). With my USE flags Window Maker builds from source (3 min 13 s here) and wmsystemtray always does.
Risk: low and reversible. Only new files under /etc/portage and ~, plus six new packages. wmaker.inst --batch overwrites ~/GNUstep/Defaults/WindowMaker without asking if it already exists: back it up first.

Before you start#

You are on a 23.0 profile and have Xorg and startx:

$ eselect profile show
Current /etc/portage/make.profile symlink:
  default/linux/amd64/23.0/desktop
$ qlist -Iv x11-base/xorg-server
x11-base/xorg-server-21.1.24
$ qlist -Iv x11-apps/xinit
x11-apps/xinit-1.4.4

Expected: an amd64 profile; both packages installed. If qlist prints nothing, install Xorg first; this manual does not cover it.

Back up any existing Window Maker user config (step 5 overwrites it):
```
[ -d ~/GNUstep ] && cp -a ~/GNUstep ~/GNUstep.bak-$(date +%F)
```
No wmaker is running. Steps 5 and 6 edit files a running Window Maker reads:
```
$ pgrep -x wmaker; echo rc=$?
rc=1
```
Expected: rc=1 (no process).

Steps#

Set Window Maker's USE flags.
```
echo 'x11-wm/windowmaker png jpeg webp xinerama xpm nls xrandr' | sudo tee /etc/portage/package.use/windowmaker
```
```
$ cat /etc/portage/package.use/windowmaker
x11-wm/windowmaker png jpeg webp xinerama xpm nls xrandr
```
Expected: the one line. xinerama and xrandr are what keep the binhost package from being used (step 2). The binhost build of 0.96.0-r1 has gif jpeg nls png tiff webp xpm and neither of them. Without them, the binary package matches. xinerama adds Xinerama multi-monitor geometry queries (use.desc); xrandr enables XRandR extension support (metadata.xml).

Emerge Window Maker.

$ sudo emerge --ask x11-wm/windowmaker
…
[ebuild  N     ] x11-wm/windowmaker-0.96.0-r1  USE="gif jpeg nls png tiff webp xinerama xpm xrandr -imagemagick -modelock"

!!! The following binary packages have been ignored due to non matching USE:

    =x11-wm/windowmaker-0.96.0-r1 -xinerama -xrandr
…
>>> Completed (1 of 1) x11-wm/windowmaker-0.96.0-r1::gentoo

Expected: Completed (1 of 1). [ebuild means a source build, [binary a binhost package. The "ignored due to non matching USE" lines are a notice, not an error (Failure modes, row 3).

Accept the ~amd64 keyword for wmsystemtray. It is the only one of the four dockapps without a stable amd64 keyword (KEYWORDS="~amd64 ~ppc64 ~x86"). Write it without a version, so later releases are accepted too:
```
$ echo 'x11-plugins/wmsystemtray ~amd64' | sudo tee /etc/portage/package.accept_keywords/wmsystemtray
x11-plugins/wmsystemtray ~amd64
```
Expected: the line echoed back. If you skip this step, step 4 fails with Failure modes, row 2.

Emerge the dockapps from x11-plugins. The category is x11-plugins, not x11-misc (Failure modes, row 1).

$ sudo emerge --ask x11-plugins/wmclock x11-plugins/wmifs x11-plugins/wmmemload x11-plugins/wmsystemtray
…
[binary  N g   ] x11-libs/libdockapp-0.7.3-1  USE="X -static-libs"
[binary  N g   ] x11-plugins/wmclock-1.0.16-r1-2
[binary  N g   ] x11-plugins/wmmemload-0.1.8-r1-1
[ebuild  N    ~] x11-plugins/wmsystemtray-1.4-r3
[binary  N g   ] x11-plugins/wmifs-1.9-1
…
>>> Jobs: 5 of 5 complete                                               Load avg: 1.02, 1.19, 1.13

Expected: 5 of 5 complete. libdockapp comes in as a dependency. Only wmsystemtray builds from source.

Seed ~/GNUstep with wmaker.inst --batch. It needs no X display. This copies the stock WindowMaker defaults, generates WMRootMenu with wmgenmenu, and creates Library/WindowMaker/ with an executable autostart:
```
$ wmaker.inst --batch
Installing WindowMaker 0.96.0 for current user...
Creating ~/GNUstep user directory
Copying defaults database...
Creating icon library directory ~/GNUstep/Library/Icons
Creating WindowMaker data library directory ~/GNUstep/Library/WindowMaker
Creating style library ~/GNUstep/Library/WindowMaker/Styles
Creating theme library ~/GNUstep/Library/WindowMaker/Themes
Creating bg image library ~/GNUstep/Library/WindowMaker/Backgrounds
Creating icon setup library ~/GNUstep/Library/WindowMaker/IconSets
Creating pixmap library ~/GNUstep/Library/WindowMaker/Pixmaps
Installation Finished
$ ls ~/GNUstep/Defaults
WMRootMenu
WindowMaker
```
Expected: Installation Finished and the two files. (This transcript is from a re-run on the same box with a scratch HOME; the first seed's output was not saved. Paths shortened to ~.)

WarningRun wmaker.inst --batch once

Batch mode never prompts. On a second run it replaces Defaults/WindowMaker with the stock file and regenerates WMRootMenu, so every edit made in steps 6 onward is lost. autostart survives because it is copied only if absent. Proof: Failure modes, row 4. To refresh only the menu, use wmgenmenu > ~/GNUstep/Defaults/WMRootMenu.
Stop Window Maker painting over your wallpaper (optional). The stock WorkspaceBack is a solid colour. If something draws the root window before wmaker starts (my x11/common does, step 10), set it to (none):
```
$ wdread WindowMaker WorkspaceBack
(solid, "rgb:50/50/75")
$ wdwrite WindowMaker WorkspaceBack '(none)'
$ wdread WindowMaker WorkspaceBack
(none)
```
Expected: (none) on read-back. (Transcript from the same scratch HOME as step 5; my real defaults read (none) too, see Verify.) Skip this step if Window Maker is the only thing that sets a background.

Start the dockapps from autostart. Window Maker runs this script before it starts handling events and waits for it to exit, so every command gets a trailing &. It also runs the script again on every Restart (root menu), while the first dockapps are still alive, so each line starts its program only if it is not already running. wmmemload -b -c leaves buffers and page cache out of "used" memory; without it the meter read 86% on a box with 13 GiB available. The file must stay executable. Append the four lines under the stock comment header:

cat >> ~/GNUstep/Library/WindowMaker/autostart <<'EOF'
pgrep -x wmclock >/dev/null || wmclock &
pgrep -x wmifs >/dev/null || wmifs &
pgrep -x wmmemload >/dev/null || wmmemload -b -c &
pgrep -x wmsystemtray >/dev/null || wmsystemtray &
EOF

(Transcript from a scratch HOME; my live file has one more dockapp, wmix, and a comment.)

$ tail -4 ~/GNUstep/Library/WindowMaker/autostart
pgrep -x wmclock >/dev/null || wmclock &
pgrep -x wmifs >/dev/null || wmifs &
pgrep -x wmmemload >/dev/null || wmmemload -b -c &
pgrep -x wmsystemtray >/dev/null || wmsystemtray &
$ sh -n ~/GNUstep/Library/WindowMaker/autostart; echo rc=$?
rc=0
$ stat -c "%A %n" ~/GNUstep/Library/WindowMaker/autostart
-rwxr-xr-x ~/GNUstep/Library/WindowMaker/autostart

Expected: the four lines, rc=0, x bits set.

Write a minimal startx client script. One file per window manager, so startx can be pointed at it:
```
cat > ~/.xinitrc-windowmaker <<'EOF'
#!/bin/sh
exec wmaker
EOF
chmod 755 ~/.xinitrc-windowmaker
```
```
$ cat ~/.xinitrc-windowmaker
#!/bin/sh
exec wmaker
$ sh -n ~/.xinitrc-windowmaker; echo rc=$?
rc=0
$ stat -c "%A %n" ~/.xinitrc-windowmaker
-rwxr-xr-x ~/.xinitrc-windowmaker
```
Expected: rc=0 and x bits set. exec wmaker is also what upstream's install page puts in ~/.xinitrc. (Transcript from a scratch HOME, paths shortened to ~; on my box this file is the longer script in step 10.) [UNVERIFIED: this minimal script was not started; my box runs the step 10 script]
(My box) Give the session its own OpenRC user runlevel. My session daemons are OpenRC user services; each window manager gets a runlevel with default stacked in, so entering it never stops dbus or PipeWire. Skip steps 9 to 11 if you don't run user services or a tty menu. The two commands are from my rebuild notes (the original run was not captured); the output below is from 2026-11-02 20:56:
```
mkdir -p ~/.config/rc/runlevels/windowmaker
rc-update --user -s add default windowmaker
```
```
$ rc-status --user windowmaker
Runlevel: windowmaker
Stacked Runlevel: default
 dbus                                                  [  started 20:41:24 (0) ]
 pipewire                                              [  started 20:41:24 (0) ]
 wireplumber                                           [  started 20:41:24 (0) ]
 pipewire-pulse                                        [  started 20:41:24 (0) ]
```
Expected: Stacked Runlevel: default. I add no services of my own to this runlevel: my terminal and notification services there need Wayland.

(My box) Replace the client script with the glue version. This is my ~/.xinitrc-windowmaker as it is on the box. It needs two things this manual does not create: ~/.config/x11/common (my shared X startup: keyboard layout, settings daemon, wallpaper, idle lock), and, for the openrc lines, the runlevel from step 9. /bin/sh stops at a failed ., so without that file wmaker never starts. In the comments, "the Path-B chooser" is the tty menu in step 11 and sway-session is my Wayland launcher. It runs wmaker without exec so the script can leave the runlevel after Window Maker exits:

#!/bin/sh
# X session: Window Maker. Launched by the Path-B chooser via
# `startx ~/.xinitrc-windowmaker -- vtN`.
#
# Shared startup (env, xkb layout, xsettingsd, cursor, root colour, .Xdefaults)
# lives in ~/.config/x11/common so the bare X WMs cannot drift apart.
. "$HOME/.config/x11/common"

export XDG_CURRENT_DESKTOP=WindowMaker

# Enter this WM's own OpenRC user runlevel. `default` (dbus/pipewire/wireplumber)
# is stacked into it, and -n (--no-stop) makes entry purely additive, so this can
# never stop the session bus out from under us. Done here rather than in each
# WM's own config so all the bare X WMs share one mechanism.
openrc -U -n windowmaker

# NOTE: deliberately not `exec` -- this script must outlive the WM to run the
# teardown below, exactly as /usr/local/bin/sway-session does.
#
# Window Maker provides its own Dock and Clip, so no external panel/bar is
# started here (unlike openbox/fvwm3, which run a standalone xfce4-panel).
# Dockapps go in ~/GNUstep/Library/WindowMaker/autostart if wanted.
#
# ~/GNUstep/Defaults/ was seeded with `wmaker.inst --batch`, so the user
# defaults database (WindowMaker, WMRootMenu) exists and edits go here instead
# of the read-only /etc/X11/WindowMaker fallback. Re-seed only if it is missing.
wmaker

# WM has exited. Leave its runlevel so any WM-bound service stops rather than
# lingering with a dead display. `default` is stacked, so dbus/pipewire survive;
# pam_openrc stops those at PAM session close.
openrc -U default >/dev/null 2>&1

$ sh -n ~/.xinitrc-windowmaker; echo rc=$?
rc=0

Expected: rc=0.

(My box) Add the session to the tty menu. My tty1 login runs a small menu script (~/.local/bin/session-chooser) whose startx_session function execs startx "$HOME/.xinitrc-$1" -- vtN, with N taken from the current tty. All ten digits were already used, so Window Maker got a letter. One line goes into the menu's list of printf lines, the other into the case before the *) fallback:
```
printf '   [w] windowmaker (X11)\n'
```
```
	w|W) startx_session windowmaker ;;
```
Checked with stubs for startx and sway-session on PATH, a scratch HOME holding a copy of the step 10 script, and the real menu script driven through script:
```
$ printf 'w\n' | script -qec 'bash /home/user/.local/bin/session-chooser' /dev/null | tail -3
   [w] windowmaker (X11)

  session [1]: STUB startx ~/.xinitrc-windowmaker -- vt0
```
Expected: the stub prints the startx line. vt0 is the pseudo-terminal's number under script; on tty1 it is vt1. An unmatched key still falls through to my default session (STUB sway-session).
Start the session. Log in on a tty and run:
```
startx ~/.xinitrc-windowmaker -- vt1
```
Use the number of the tty you are on in place of 1.

Alert on a stopped RouterOS container via ntfy and Telegram

Mon, 02 Nov 2026 00:00:00 +0000

This installs a RouterOS 7 scheduler script that checks named containers once a minute and sends one message to ntfy and Telegram each time the set of stopped containers changes. The endpoints and tokens are read from a container envlist at run time, so the script file holds no secrets.

Tested on: MikroTik hAP ax^3, RouterOS 7.24.4 (stable), watching containers observer and mosquitto (2026-10-16 install and test message; 2026-10-17 read-only check). The down path (a container actually stopping) has not been triggered yet.
Needs: RouterOS 7 with the container package and at least one container; SSH and SCP to the router (I used admin, group full; a narrower user needs at least ssh and ftp on top of read,write,test, untested); a disk on the router for the .rsc file; an ntfy topic URL and/or a Telegram bot token plus chat id.
Time: 15 min.
Risk: low. The script only reads container state and sends HTTP requests. Adding the sink settings to an envlist that a running container uses puts them into that container's environment on its next re-create. Rollback removes two scripts, one scheduler entry and the envlist keys.

Before you start#

Placeholders used below:

Run upsmon as a NUT secondary on Fedora and Fedora Atomic

Mon, 02 Nov 2026 00:00:00 +0000

This installs upsmon on a Fedora or Fedora Atomic host as a read-only secondary of a remote NUT server, checks the login before anything is armed, and enables it so it starts again after a reboot.

Tested on: Fedora 44 KDE Plasma Desktop and Fedora 45 Prerelease (installed, enabled, status read; reboot checked on Fedora 44); Fedora 44 Server and Bazzite 44.20261012.1 (installed, configured, login checked, service left disabled). nut-client / upsmon 2.8.5 (2026-10-16, re-checked 2026-10-17). The server side was my own NUT server (nut-observer), not upsd; the steps are the stock client side.
Not verified: the shutdown itself. No host has been powered down by upsmon yet; this manual proves login, status and start-at-boot only.
Needs: root on the host; a NUT server reachable on TCP 3493 with a UPS name and a user/password for this host; one reboot on Fedora Atomic if a deployment is already staged.
Time: 15 min (plus one reboot on Fedora Atomic).
Risk: once enabled, the host powers itself off when the server reports the UPS on battery and low battery, or when the server stays unreachable for DEADTIME while the UPS was last seen on battery. Reversible: Undo below restores the original configs.

Before you start#

Placeholders, used in every step below:
- <UPS>: the UPS name on the NUT server.
- <SERVER>: the NUT server's address.
- <NUT_USER>, <NUT_PASSWORD>: the login the server holds for this host, with secondary rights.
Open a root shell (sudo -i) and keep it for all steps: the variables set in step 3 are used later.
Check that the server answers and lists your UPS (nc is in the nmap-ncat package). Output from the Fedora 44 box, UPS names replaced, cut marked:
```
$ printf 'LIST UPS\nLOGOUT\n' | nc <SERVER> 3493
BEGIN LIST UPS
…
UPS <UPS> "BLUETTI AC180, tier t2, via nut-observer"
…
END LIST UPS
OK Goodbye
```
Expected: your <UPS> appears between BEGIN LIST UPS and END LIST UPS.

Fedora Atomic only: check free space with df -h /sysroot and look for a staged deployment with rpm-ostree status. Lines from both on the Bazzite box before the install, cut marked:

/dev/nvme0n1p3  930G  914G   15G  99% /sysroot
…
State: idle
Deployments:
  ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-nvidia-open:stable
…
                  Version: 44.20261012.1 (2026-10-13T00:06:38Z)
…
                     Diff: 406 upgraded, 13 removed, 54 added
…
● ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-nvidia-open:stable
…
                  Version: 44.20260921 (2026-09-21T05:59:02Z)

Expected: free space above rpm-ostree's 3 % floor (this box was at 99 % used, Failure modes row 1). A deployment listed above the booted one (●) is already staged; that is the state in which my live apply was refused (row 2).

Backups: step 2 copies the existing nut.conf and upsmon.conf to *.orig once. Undo restores from them.

Restart a Go server under systemd without dropping requests

Sun, 01 Nov 2026 00:00:00 +0000

A .socket unit keeps the port open across restarts, the Go server takes that socket from systemd and drains running requests on SIGTERM, and KillMode=mixed leaves the processes those requests wait on alone, so systemctl --user restart refuses no connection and cuts no stream that ends within the drain budget (2 min here).

Tested on: GEEKOM AE8 mini PC, Fedora Linux 44 (kernel 7.2.10), systemd 259 (259.9-1.fc44) user manager with linger on, Go 1.26.0, curl 8.18.0, iproute2 6.17.0 (2026-10-25). The demo service is tsvdemo-drain on 127.0.0.1:18490 (18491 after step 10).
Not verified: system (root) units; IPv6 and Unix-socket ListenStream= addresses (more than one ListenStream= the server refuses, Failure modes); Accept=yes; TLS and HTTP/2; WebSockets and other hijacked connections; a start at boot from the enabled units (I used the runtime unit directory and enable --runtime, no reboot); FreeBind= for a LAN address.
Needs: a working systemctl --user; Go (1.26.0 here); curl; ss; ports 18490 and 18491 free on 127.0.0.1.
Time: 20 min.
Risk: low for the demo: two units on 127.0.0.1 in a unit directory that a reboot empties. On your own service, a change of the socket unit refuses or resets new connections for as long as the drain runs (step 10), and a TimeoutStopSec= below the drain aborts the stop. Every change here is undone in Undo / rollback.

Before you start#

Versions and linger:

$ systemctl --version | head -1; go version; loginctl show-user "$USER" -p Linger
systemd 259 (259.9-1.fc44)
go version go1.26.0 linux/amd64
Linger=yes

The user manager's default stop timeout. It is 45 s here; the systemd 259 man page says 90 s. Fedora lowered it to 45 s in Fedora 38 (the "Shorter Shutdown Timer" change), and /usr/lib/systemd/user.conf lists #DefaultTimeoutStopSec=45s. A drain longer than this default needs its own TimeoutStopSec= (step 4):
```
$ systemctl --user show -p DefaultTimeoutStopUSec
DefaultTimeoutStopUSec=45s
```

Nothing listens on the demo ports, 18490 and 18491 (header line only):

$ ss -ltn '( sport = :18490 or sport = :18491 )'
State Recv-Q Send-Q Local Address:Port Peer Address:Port

Make a directory for the demo and run every command below from it. UNITDIR is where install-units.sh (step 5) puts the units. I used the runtime unit directory, which systemd reads like ~/.config/systemd/user and which lives on tmpfs (/run/user/<UID>), so a reboot removes the trial. For a permanent install leave UNITDIR unset: the script then writes to ~/.config/systemd/user. Export it again in every new shell: without it install-units.sh installs to ~/.config/systemd/user, and the drop-in and Undo commands below lose their directory.
```
mkdir tsvdemo-drain && cd tsvdemo-drain
export UNITDIR=$XDG_RUNTIME_DIR/systemd/user
```
To apply this to your own service later, copy its unit file first (cp ~/.config/systemd/user/<APP>.service ~/<APP>.service.orig, where <APP> is your unit's name). The demo itself replaces nothing.

Steps#

Write the server. Save it as main.go. listenerFrom takes the socket systemd passes (fd 3, LISTEN_PID = this process, LISTEN_FDS = 1) or binds -listen itself; main unsets the four LISTEN_* variables before any child process starts; the first SIGTERM or SIGINT starts the drain and a second one exits at once; serveUntil waits for Shutdown and calls Close when the drain budget runs out; /stream?n=N sends N events one second apart, and &wait=S first runs sleep S as a child process.

// tsvdemo-drain: an SSE demo that takes its listening socket from systemd and drains on stop.
package main

import (
	"context"
	"errors"
	"flag"
	"fmt"
	"log"
	"net"
	"net/http"
	"os"
	"os/exec"
	"os/signal"
	"strconv"
	"syscall"
	"time"
)

func main() {
	listen := flag.String("listen", "127.0.0.1:18490", "address to bind when not socket-activated")
	drain := flag.Duration("drain", 2*time.Minute, "on stop, how long running requests may finish")
	flag.Parse()

	// The listener first, and the LISTEN_* variables gone, before any child process starts.
	ln, inherited, err := listenerFrom(*listen)
	if err != nil {
		log.Fatal(err)
	}
	for _, k := range []string{"LISTEN_PID", "LISTEN_FDS", "LISTEN_FDNAMES", "LISTEN_PIDFDID"} {
		os.Unsetenv(k)
	}

	// First SIGTERM/SIGINT: drain. A second one: exit at once.
	ctx, cancel := context.WithCancel(context.Background())
	sigs := make(chan os.Signal, 1)
	signal.Notify(sigs, os.Interrupt, syscall.SIGTERM)
	go func() {
		<-sigs
		cancel()
		<-sigs
		log.Print("shutdown: second signal, exiting now")
		os.Exit(1)
	}()

	mux := http.NewServeMux()
	mux.HandleFunc("GET /stream", stream)
	srv := &http.Server{Handler: mux, ReadHeaderTimeout: 10 * time.Second}
	log.Printf("pid %d serving on %s (inherited=%t)", os.Getpid(), ln.Addr(), inherited)
	if err := serveUntil(ctx, srv, ln, *drain); err != nil {
		log.Fatal(err)
	}
	log.Print("exit")
}

// listenerFrom returns the socket systemd passed (fd 3, LISTEN_PID = this process,
// LISTEN_FDS = 1) or, without socket activation, binds addr itself.
func listenerFrom(addr string) (net.Listener, bool, error) {
	if os.Getenv("LISTEN_PID") != strconv.Itoa(os.Getpid()) {
		ln, err := net.Listen("tcp", addr)
		return ln, false, err
	}
	if n := os.Getenv("LISTEN_FDS"); n != "1" {
		return nil, false, fmt.Errorf("systemd passed %q sockets, want 1 (one ListenStream= line)", n)
	}
	f := os.NewFile(3, "systemd-socket")
	ln, err := net.FileListener(f) // a dup; close fd 3 itself so children don't inherit it
	f.Close()
	return ln, true, err
}

// serveUntil serves on ln until ctx ends, lets running requests finish for up to drain, then
// closes what is left. It returns only after that: Serve returns ErrServerClosed as soon as
// Shutdown starts, and a main that returned then would cut every running request.
func serveUntil(ctx context.Context, srv *http.Server, ln net.Listener, drain time.Duration) error {
	done := make(chan struct{})
	go func() {
		defer close(done)
		<-ctx.Done()
		log.Printf("shutdown: draining for up to %s", drain)
		sctx, stop := context.WithTimeout(context.Background(), drain)
		defer stop()
		if err := srv.Shutdown(sctx); err != nil {
			log.Printf("shutdown: drain time over (%v), closing", err)
			srv.Close()
			return
		}
		log.Print("shutdown: all requests finished")
	}()
	err := srv.Serve(ln)
	if errors.Is(err, http.ErrServerClosed) {
		<-done
		return nil
	}
	return err
}

// stream sends n events, one per second. With wait=S it first runs `sleep S` as a child
// process: a stand-in for an ssh call or a helper the request waits on.
func stream(w http.ResponseWriter, r *http.Request) {
	n, _ := strconv.Atoi(r.URL.Query().Get("n"))
	if s := r.URL.Query().Get("wait"); s != "" {
		if err := exec.CommandContext(r.Context(), "sleep", s).Run(); err != nil {
			log.Printf("stream: child failed: %v", err)
			http.Error(w, "child failed: "+err.Error(), http.StatusBadGateway)
			return
		}
	}
	w.Header().Set("Content-Type", "text/event-stream")
	rc := http.NewResponseController(w)
	for i := 1; i <= n; i++ {
		fmt.Fprintf(w, "data: pid=%d event=%d/%d\n\n", os.Getpid(), i, n)
		if err := rc.Flush(); err != nil {
			log.Printf("stream: client gone after %d/%d events", i, n)
			return
		}
		if i < n {
			select {
			case <-r.Context().Done():
				log.Printf("stream: cut after %d/%d events", i, n)
				return
			case <-time.After(time.Second):
			}
		}
	}
	log.Printf("stream: %d events sent", n)
}

Create the module and build it:

$ go mod init tsvdemo-drain
go: creating new go.mod: module tsvdemo-drain
go: to add module requirements and sums:
	go mod tidy
$ go vet ./... && go build -o tsvdemo-drain . && ./tsvdemo-drain -h
Usage of ./tsvdemo-drain:
  -drain duration
    	on stop, how long running requests may finish (default 2m0s)
  -listen string
    	address to bind when not socket-activated (default "127.0.0.1:18490")

Expected: no output from go vet, then the two flags.

Check the fallback without systemd. Started by hand, the server binds the address itself; SIGINT drains and exits 0:

$ ./tsvdemo-drain & sleep 0.5; curl -sN 'http://127.0.0.1:18490/stream?n=3'; kill -INT $!; wait $!; echo "exit status $?"
2026/10/25 14:20:56 pid 2187190 serving on 127.0.0.1:18490 (inherited=false)
data: pid=2187190 event=1/3

data: pid=2187190 event=2/3

data: pid=2187190 event=3/3

2026/10/25 14:20:59 stream: 3 events sent
2026/10/25 14:20:59 shutdown: draining for up to 2m0s
2026/10/25 14:20:59 shutdown: all requests finished
2026/10/25 14:20:59 exit
exit status 0

Expected: inherited=false, three events, shutdown: all requests finished, exit status 0. The log lines go to stderr, so they interleave with curl's output. Pasted into an interactive bash, the same line also prints the job lines [1] <PID> before the first log line and [1]+ Done … ./tsvdemo-drain before exit status 0.

Write the socket unit. Save it as tsvdemo-drain.socket:
```
[Unit]
Description=tsvdemo-drain socket: held across restarts, so connections wait while the old process drains

[Socket]
ListenStream=127.0.0.1:18490

[Install]
WantedBy=sockets.target
```
- One ListenStream= line: the server refuses any LISTEN_FDS other than 1 (Failure modes).
- An IP literal with a port: systemd-analyze refuses Go's :18490 and localhost:18490 forms (Failure modes).
- Accept= stays at its default, no: the listening socket itself goes to one service process.
Write the service unit. Save it as tsvdemo-drain.service; install-units.sh replaces @DEMO@ with the demo directory.
```
[Unit]
Description=tsvdemo-drain: SSE demo that drains running requests on stop
# Wants, not Requires: if the socket cannot bind, the service tries to bind the address
# itself and Restart= retries.
Wants=tsvdemo-drain.socket
After=tsvdemo-drain.socket

[Service]
ExecStart=@DEMO@/tsvdemo-drain -listen 127.0.0.1:18490 -drain 2m
Restart=on-failure
RestartSec=5
# SIGTERM to the main process only: children it waits on keep running during the drain.
# Whatever is left in the cgroup gets SIGKILL after the main process exits.
KillMode=mixed
# Above -drain: when the stop times out, systemd ends the drain.
TimeoutStopSec=150

[Install]
WantedBy=default.target
```
- Wants= and After= the socket: starting the service starts the socket first. With Requires=, a socket that cannot bind fails the service as a dependency and nothing retries; with Wants= the server tries to bind by itself, Restart=on-failure retries every 5 s, and the first retry after the port is free gets the socket again (Failure modes).
- KillMode=mixed: the default, control-group, sends SIGTERM to every process in the unit when the stop begins, so a child process a request waits on dies at once (step 8).
- TimeoutStopSec=150, above -drain 2m: when the stop timeout runs out, systemd ends the drain (Failure modes).
- -listen repeats the socket's address. The server uses it only without the socket; change both together.

Write the install script. Save it as install-units.sh. It checks both units with systemd-analyze before it stops anything. A new or changed socket file means stop of socket and service in one call, then start of both: with the service stopped alone, a client that connects during the drain starts it again through the socket, and the socket then refuses to restart while its service runs (Failure modes). An unchanged socket means stop of the service only, while new connections queue on the open socket, then start of both.

#!/bin/bash
# Install or update the tsvdemo-drain units and (re)start them without refusing connections.
set -euo pipefail
cd "$(dirname "$0")"
UNITDIR=${UNITDIR:-$HOME/.config/systemd/user}
mkdir -p "$UNITDIR" build
sed "s|@DEMO@|$PWD|" tsvdemo-drain.service > build/tsvdemo-drain.service
cp tsvdemo-drain.socket build/
systemd-analyze --user verify build/tsvdemo-drain.socket build/tsvdemo-drain.service
if cmp -s build/tsvdemo-drain.socket "$UNITDIR/tsvdemo-drain.socket"; then moved=; else moved=1; fi
install -m644 build/tsvdemo-drain.socket build/tsvdemo-drain.service "$UNITDIR/"
systemctl --user daemon-reload
if [ -n "$moved" ]; then
  # New or changed socket: stop both in one call, so no client re-activates the service in between.
  echo "socket file changed: stopping socket and service, then starting both"
  systemctl --user stop tsvdemo-drain.socket tsvdemo-drain.service
  systemctl --user start tsvdemo-drain.socket tsvdemo-drain.service
else
  # Same socket: stop the service behind it (new connections queue while it drains), start both.
  echo "socket file unchanged: restarting the service behind the open socket"
  systemctl --user stop tsvdemo-drain.service
  systemctl --user start tsvdemo-drain.socket tsvdemo-drain.service
fi

Make it executable:

$ chmod +x install-units.sh

Install, enable and start the units. For a permanent install, leave out --runtime.

$ echo "UNITDIR=$UNITDIR"; ./install-units.sh
UNITDIR=/run/user/1000/systemd/user
socket file changed: stopping socket and service, then starting both
$ systemctl --user enable --runtime tsvdemo-drain.socket tsvdemo-drain.service
Created symlink '/run/user/1000/systemd/user/sockets.target.wants/tsvdemo-drain.socket' → '/run/user/1000/systemd/user/tsvdemo-drain.socket'.
Created symlink '/run/user/1000/systemd/user/default.target.wants/tsvdemo-drain.service' → '/run/user/1000/systemd/user/tsvdemo-drain.service'.

Check that the service got the socket:

$ systemctl --user is-active tsvdemo-drain.socket tsvdemo-drain.service; ss -ltnp 'sport = :18490'
active
active
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess                                    
LISTEN 0      4096       127.0.0.1:18490      0.0.0.0:*    users:(("tsvdemo-drain",pid=2187490,fd=5))
$ journalctl --user -u tsvdemo-drain.service -g serving -n 1 -o cat; tr "\0" "\n" < /proc/$(systemctl --user show -p MainPID --value tsvdemo-drain.service)/environ | grep ^LISTEN_
2026/10/25 14:21:00 pid 2187490 serving on 127.0.0.1:18490 (inherited=true)
LISTEN_PID=2187490
LISTEN_PIDFDID=14783253
LISTEN_FDS=1
LISTEN_FDNAMES=tsvdemo-drain.socket

Expected: both units active; one LISTEN line on 127.0.0.1:18490; inherited=true in the log. /proc/<PID>/environ shows the environment systemd set at exec, including LISTEN_PIDFDID, which systemd 259 adds; the server's os.Unsetenv doesn't change that file, it changes what children get (step 8). If the log says inherited=false, see Failure modes.

Restart while a stream runs. Save restart-test.sh. Stream A asks for 15 events; the restart starts after 3.5 s; request B is made 2 s into the drain; ss shows the listening socket 1 s later. A second argument replaces the restart command (step 10).

#!/bin/bash
# Restart tsvdemo-drain while stream A (15 events) runs; request B is made during the drain.
# usage: restart-test.sh [PORT [COMMAND...]]; COMMAND defaults to systemctl --user restart.
P=${1:-18490}
U=http://127.0.0.1:$P
shift
[ $# -gt 0 ] || set -- systemctl --user restart tsvdemo-drain.service
ts() { date +%T.%3N; }
stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
{ curl -sSN "$U/stream?n=15" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
sleep 3.5
echo "$(ts) restart begins"
("$@"; echo "$(ts) restart returned rc=$?") &
sleep 2
echo "$(ts) B starts"
{ curl -sSN "$U/stream?n=1" -w 'http=%{http_code} connect=%{time_connect}s total=%{time_total}s\n' 2>&1; echo "curl rc=$?"; } | stamp B &
sleep 1
ss -Hltnp "sport = :$P" | stamp ss
wait

$ chmod +x restart-test.sh
$ ./restart-test.sh
14:21:00.774 A data: pid=2187490 event=1/15
14:21:01.775 A data: pid=2187490 event=2/15
14:21:02.776 A data: pid=2187490 event=3/15
14:21:03.777 A data: pid=2187490 event=4/15
14:21:04.271 restart begins
14:21:04.778 A data: pid=2187490 event=5/15
14:21:05.778 A data: pid=2187490 event=6/15
14:21:06.274 B starts
14:21:06.778 A data: pid=2187490 event=7/15
14:21:07.317 ss LISTEN 1      4096   127.0.0.1:18490 0.0.0.0:*
14:21:07.780 A data: pid=2187490 event=8/15
14:21:08.780 A data: pid=2187490 event=9/15
14:21:09.781 A data: pid=2187490 event=10/15
14:21:10.781 A data: pid=2187490 event=11/15
14:21:11.783 A data: pid=2187490 event=12/15
14:21:12.784 A data: pid=2187490 event=13/15
14:21:13.785 A data: pid=2187490 event=14/15
14:21:14.785 A data: pid=2187490 event=15/15
14:21:14.787 A http=200
14:21:14.789 A curl rc=0
14:21:15.293 restart returned rc=0
14:21:15.306 B data: pid=2188843 event=1/1
14:21:15.307 B http=200 connect=0.000108s total=9.024878s
14:21:15.307 B curl rc=0

Expected: A gets 15 of 15 events from the old PID and curl rc=0. B's connect completes in 0.11 ms, before the new process exists, and B gets its event from the new PID once the drain ends (9.0 s later here). During the drain the old process has already closed its listener (Shutdown closes the listeners first), yet ss shows the port listening, with Recv-Q at 1 while B waits and no process listed: systemd keeps its own copy of the socket. systemctl restart returns when the drain is over (11.0 s here). The journal for the same restart:

$ journalctl --user -u tsvdemo-drain.service --since -30s -o short-precise
Oct 25 14:21:00.455060 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:00.466518 host tsvdemo-drain[2187490]: 2026/10/25 14:21:00 pid 2187490 serving on 127.0.0.1:18490 (inherited=true)
Oct 25 14:21:04.276149 host tsvdemo-drain[2187490]: 2026/10/25 14:21:04 shutdown: draining for up to 2m0s
Oct 25 14:21:04.276225 host systemd[1397]: Stopping tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop...
Oct 25 14:21:14.784301 host tsvdemo-drain[2187490]: 2026/10/25 14:21:14 stream: 15 events sent
Oct 25 14:21:15.288520 host tsvdemo-drain[2187490]: 2026/10/25 14:21:15 shutdown: all requests finished
Oct 25 14:21:15.288520 host tsvdemo-drain[2187490]: 2026/10/25 14:21:15 exit
Oct 25 14:21:15.289782 host systemd[1397]: Stopped tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:15.291662 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:15.304665 host tsvdemo-drain[2188843]: 2026/10/25 14:21:15 pid 2188843 serving on 127.0.0.1:18490 (inherited=true)
Oct 25 14:21:15.305812 host tsvdemo-drain[2188843]: 2026/10/25 14:21:15 stream: 1 events sent

Restart while a request waits on a child process. Save child-test.sh. Request A runs sleep 10 as a child of the server, then streams 3 events; the restart starts 2 s in. The script also prints the child's file descriptors and how many LISTEN_* variables it inherited.

#!/bin/bash
# Restart tsvdemo-drain while request A waits on a child process (sleep 10) before it streams 3 events.
U=http://127.0.0.1:${1:-18490}
ts() { date +%T.%3N; }
stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
{ curl -sSN "$U/stream?n=3&wait=10" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
sleep 2
main=$(systemctl --user show -p MainPID --value tsvdemo-drain.service)
child=$(pgrep -P "$main" -x sleep)
echo "$(ts) child: pid $child, fds: $(ls /proc/$child/fd | tr '\n' ' ')LISTEN_ variables: $(tr '\0' '\n' < /proc/$child/environ | grep -c '^LISTEN_')"
echo "$(ts) restart begins"
systemctl --user restart tsvdemo-drain.service
echo "$(ts) restart returned rc=$?"
wait

$ chmod +x child-test.sh
$ ./child-test.sh
14:21:17.873 child: pid 2188987, fds: 0 1 2 LISTEN_ variables: 0
14:21:17.878 restart begins
14:21:25.841 A data: pid=2188843 event=1/3
14:21:26.842 A data: pid=2188843 event=2/3
14:21:27.843 A data: pid=2188843 event=3/3
14:21:27.845 A http=200
14:21:27.846 A curl rc=0
14:21:27.988 restart returned rc=0

Expected: the child has only fds 0, 1 and 2 and no LISTEN_* variable; it keeps running through the drain, A gets 3 of 3 events, and the restart returns after them (10.1 s here).

GotchaThe default KillMode kills the child the request waits on

With KillMode=control-group, the SIGTERM of the stop reaches the child too: the request fails with 502 within 6 ms of the restart, and the drain has nothing left to wait for. The same test with a drop-in, then the drop-in removed:

$ mkdir -p "$UNITDIR/tsvdemo-drain.service.d" && printf "[Service]\nKillMode=control-group\n" > "$UNITDIR/tsvdemo-drain.service.d/killmode.conf" && systemctl --user daemon-reload && systemctl --user show -p KillMode tsvdemo-drain.service
KillMode=control-group
$ ./child-test.sh
14:21:30.796 child: pid 2189923, fds: 0 1 2 LISTEN_ variables: 0
14:21:30.801 restart begins
14:21:30.807 A child failed: signal: terminated
14:21:30.808 A http=502
14:21:30.809 A curl rc=0
14:21:30.811 restart returned rc=0
$ rm "$UNITDIR/tsvdemo-drain.service.d/killmode.conf" && rmdir "$UNITDIR/tsvdemo-drain.service.d" && systemctl --user daemon-reload && systemctl --user show -p KillMode tsvdemo-drain.service
KillMode=mixed

End a drain early (optional). A second SIGTERM to the main process makes the server exit at once; a restart then goes on with the new process. The command is systemctl --user kill --kill-whom=main --signal=SIGTERM tsvdemo-drain.service. Save abort-test.sh to see it during a 60-event stream:

#!/bin/bash
# Restart tsvdemo-drain during a 60-event stream, then end the drain early with a second SIGTERM.
U=http://127.0.0.1:${1:-18490}
ts() { date +%T.%3N; }
stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
{ curl -sSN "$U/stream?n=60" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
sleep 2.5
echo "$(ts) restart begins"
(systemctl --user restart tsvdemo-drain.service; echo "$(ts) restart returned rc=$?") &
sleep 3
echo "$(ts) second SIGTERM to the main process"
systemctl --user kill --kill-whom=main --signal=SIGTERM tsvdemo-drain.service
wait

$ chmod +x abort-test.sh
$ ./abort-test.sh
14:21:31.564 A data: pid=2190293 event=1/60
14:21:32.565 A data: pid=2190293 event=2/60
14:21:33.566 A data: pid=2190293 event=3/60
14:21:34.059 restart begins
14:21:34.566 A data: pid=2190293 event=4/60
14:21:35.566 A data: pid=2190293 event=5/60
14:21:36.567 A data: pid=2190293 event=6/60
14:21:37.062 second SIGTERM to the main process
14:21:37.069 A curl: (18) transfer closed with outstanding read data remaining
14:21:37.070 A http=200
14:21:37.071 A curl rc=18
14:21:37.072 restart returned rc=0

Expected: A is cut (curl: (18)) 3.0 s into the drain, and the restart returns 10 ms after the second signal. The journal records exit status 1 as Failed with result 'exit-code', then starts the new process on the socket:

$ journalctl --user -u tsvdemo-drain.service --since -15s -o short-precise
[…]
Oct 25 14:21:34.065330 host systemd[1397]: Stopping tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop...
Oct 25 14:21:34.065848 host tsvdemo-drain[2190293]: 2026/10/25 14:21:34 shutdown: draining for up to 2m0s
Oct 25 14:21:37.067622 host systemd[1397]: tsvdemo-drain.service: Sent signal SIGTERM to main process 2190293 (tsvdemo-drain) on client request.
Oct 25 14:21:37.068169 host tsvdemo-drain[2190293]: 2026/10/25 14:21:37 shutdown: second signal, exiting now
Oct 25 14:21:37.068962 host systemd[1397]: tsvdemo-drain.service: Main process exited, code=exited, status=1/FAILURE
Oct 25 14:21:37.069055 host systemd[1397]: tsvdemo-drain.service: Failed with result 'exit-code'.
Oct 25 14:21:37.069645 host systemd[1397]: Stopped tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:37.071006 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
Oct 25 14:21:37.088652 host tsvdemo-drain[2190806]: 2026/10/25 14:21:37 pid 2190806 serving on 127.0.0.1:18490 (inherited=true)

Apply later changes with install-units.sh. After an edit to the service file only (here its Description=), the socket file is unchanged: the script stops the service behind the open socket and then starts both, so a request made meanwhile queues as in step 7:

$ sed -i 's/^Description=tsvdemo-drain:/Description=tsvdemo-drain (edited):/' tsvdemo-drain.service && ./restart-test.sh 18490 ./install-units.sh
14:21:37.604 A data: pid=2190806 event=1/15
14:21:38.605 A data: pid=2190806 event=2/15
14:21:39.605 A data: pid=2190806 event=3/15
14:21:40.606 A data: pid=2190806 event=4/15
14:21:41.098 restart begins
socket file unchanged: restarting the service behind the open socket
14:21:41.606 A data: pid=2190806 event=5/15
14:21:42.608 A data: pid=2190806 event=6/15
14:21:43.101 B starts
14:21:43.609 A data: pid=2190806 event=7/15
14:21:44.149 ss LISTEN 1      4096   127.0.0.1:18490 0.0.0.0:*
14:21:44.610 A data: pid=2190806 event=8/15
14:21:45.610 A data: pid=2190806 event=9/15
14:21:46.610 A data: pid=2190806 event=10/15
14:21:47.611 A data: pid=2190806 event=11/15
14:21:48.613 A data: pid=2190806 event=12/15
14:21:49.614 A data: pid=2190806 event=13/15
14:21:50.614 A data: pid=2190806 event=14/15
14:21:51.615 A data: pid=2190806 event=15/15
14:21:51.616 A http=200
14:21:51.617 A curl rc=0
Stopping 'tsvdemo-drain.service', but its triggering units are still active:
tsvdemo-drain.socket
14:21:51.727 restart returned rc=0
14:21:51.737 B data: pid=2192112 event=1/1
14:21:51.738 B http=200 connect=0.000108s total=8.627194s
14:21:51.739 B curl rc=0

Expected: socket file unchanged: …, systemd's warning that the socket is still active (that is the point here), B queued and answered by the new PID.

After an edit to the socket file (here the address, in both files), the script stops socket and service in one call before it starts both:

$ sed -i 's/127.0.0.1:18490/127.0.0.1:18491/' tsvdemo-drain.socket tsvdemo-drain.service && ./install-units.sh
socket file changed: stopping socket and service, then starting both
$ ss -ltnp '( sport = :18490 or sport = :18491 )'; journalctl --user -u tsvdemo-drain.service -g serving -n 1 -o cat
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess                                    
LISTEN 0      4096       127.0.0.1:18491      0.0.0.0:*    users:(("tsvdemo-drain",pid=2192375,fd=5))
2026/10/25 14:21:52 pid 2192375 serving on 127.0.0.1:18491 (inherited=true)

Expected: socket file changed: …, one LISTEN line on 18491, and inherited=true on 18491.

NoteA socket change refuses or resets new connections until the drain ends

In a separate run of the same move (18490 to 18491) with a stream running, two requests made during the drain, one to each port, both failed at once with curl: (7) … Could not connect to server, while the stream on the old port finished 8 of 8. With the address unchanged and another line of the socket file edited, a request made during the drain connected, waited, and got curl: (56) Recv failure: Connection reset by peer when the socket closed after the service. A plain restart (steps 7–9) has neither problem.

Update a RouterOS 7 container from a new image tar

Sun, 01 Nov 2026 00:00:00 +0000

This replaces the image of a running RouterOS 7 container with a new arm64 build. The container keeps its name, veth, mounts, envlist and NAT rule, and the router keeps the previous tar on disk for a rollback.

Tested on: MikroTik hAP ax^3, RouterOS 7.24.4 (stable), arm64, container files on a USB SSD (usb1-part1); image built on Fedora 44 with docker-ce 29.9.1 and docker-buildx-plugin 0.38.0 (2026-10-16, three updates).
Needs: a container that already runs from a file-imported tar with root-dir on an external disk; SSH and SCP access to the router as a user with write policy; Docker with buildx on the build machine.
Time: 10 min.
Risk: the container's service is down while it is re-created (about 18 s for the one NUT client I timed, on the first run). The config export holds the container's environment variables in plain text. Anything the container writes outside its mounts is lost: each update starts from a fresh root-dir, and the old one disappears. Rollback is a re-import of the previous version's script, which has not been tested.

Before you start#

Placeholders used below:
- <ROUTER> — SSH user and address of the router, such as admin@<ROUTER_IP>. Add your -i <KEY> option to every ssh and scp line.
- <NAME> — the container's name (mine: observer).
- <DISK> — the external disk (mine: usb1-part1). <DIR> — the container's directory on it, without a leading slash (mine: usb1-part1/observer).
- <IMAGE> — the image name and tar base name (mine: nut-observer).
- <VER> — the new version label. <PREV> — the version it replaces.
- <LIST> — the container's envlist (and mountlist) name. <KEY> / <VALUE> — an environment variable to set.
- <BUILD_CONTEXT> — the directory with your Dockerfile and binary.
The container exists, and exactly once:
```
$ ssh <ROUTER> ':put [:len [/container/find name=observer]]'
1
```
Expected: 1. 0 means there is nothing to update; create the container first.
Write down every parameter of the current container (/container/print detail where name=<NAME>, or the script you created it with). Step 5 has to repeat all of them; whatever you leave out is lost on re-create.
Backup: step 2 takes a config export before anything changes.

Steps#

Build an arm64 docker-save tar with uncompressed layers.

docker buildx build --platform linux/arm64 \
  --output "type=docker,dest=<IMAGE>.tar,name=<IMAGE>:<VER>,compression=uncompressed,force-compression=true" \
  <BUILD_CONTEXT>
tar xOf <IMAGE>.tar manifest.json; echo; ls -l <IMAGE>.tar

My build for eb59246 (one COPY of a static binary into FROM scratch):

$ docker buildx build --platform linux/arm64 \
    --output "type=docker,dest=router/observer/nut-observer.tar,name=nut-observer:eb59246,compression=uncompressed,force-compression=true" <BUILD_CONTEXT>
#0 building with "default" instance using docker driver
…
#4 [1/1] COPY nut-observer /nut-observer
#4 DONE 0.0s

#5 exporting to docker image format
#5 exporting layers
#5 exporting layers 0.2s done
#5 exporting manifest sha256:7218bea8a7911d4cb6875f74a93f350177f031a2a3df72618e2b250d66ab7b19 0.0s done
#5 exporting config sha256:7d351d5ee194596f0cf658da72d68e7de41ba71eb31343842a07ac01ad912491 0.0s done
#5 sending tarball 0.1s done
#5 DONE 0.3s
$ tar xOf router/observer/nut-observer.tar manifest.json; echo; ls -l router/observer/nut-observer.tar
[{"Config":"blobs/sha256/7d351d5ee194596f0cf658da72d68e7de41ba71eb31343842a07ac01ad912491","RepoTags":["nut-observer:eb59246"],"Layers":["blobs/sha256/610386223fecb3f4e631488ca51c5d440fa5384213b16ca9d802915e86cb98eb"]}]
-rw-------. 1 user user 6498816 Oct 16 14:32 router/observer/nut-observer.tar

Expected: RepoTags carries <IMAGE>:<VER> and Layers lists your layers. To confirm a layer is stored uncompressed, read its first two bytes. Here on the later 3f4f7db tar:

$ tar xOf router/observer/nut-observer.tar blobs/sha256/92a99037ad91a106e2190df5ebe01a9f9d25ed5dc133dec5af0d733712493314 | head -c 2 | od -An -tx1
 6e 75

6e 75 is nu, the start of the file name nut-observer in a plain tar header. A gzip stream starts with 1f 8b.

Export the router config and copy it off the router.

TAG=$(date +%F)-pre-<NAME>-<VER>
ssh <ROUTER> "/export file=<DISK>/backups/$TAG"
mkdir -p "backups/$TAG"
scp "<ROUTER>:/<DISK>/backups/$TAG.rsc" "backups/$TAG/"
chmod 700 "backups/$TAG"; chmod 600 "backups/$TAG/"*

The local copies from my three runs, after the chmod (my three runs predate it and had left them -rw-r--r--):

$ ls -l router/backups/2026-10-16-pre-observer-*/
router/backups/2026-10-16-pre-observer-27e8b5d/:
total 16
-rw-------. 1 user user 16074 Oct 16 13:23 2026-10-16-pre-observer-27e8b5d.rsc

router/backups/2026-10-16-pre-observer-3f4f7db/:
total 20
-rw-------. 1 user user 19132 Oct 16 23:55 2026-10-16-pre-observer-3f4f7db.rsc

router/backups/2026-10-16-pre-observer-eb59246/:
total 20
-rw-------. 1 user user 19132 Oct 16 14:32 2026-10-16-pre-observer-eb59246.rsc
$ head -1 router/backups/2026-10-16-pre-observer-eb59246/2026-10-16-pre-observer-eb59246.rsc
# 2026-10-16 14:32:46 by RouterOS 7.24.4

Expected: a non-empty .rsc whose first line names your RouterOS version. Store it like a password file: without show-sensitive the export still contains every /container envs entry with its value (on 7.24.4, all seven of mine, passwords and tokens included). Keep the directory out of git and off shared disks. The /export file= also leaves a plaintext copy on the router's disk: delete it after the scp, or make sure that directory isn't shared (RouterOS auto-shares new USB disks over SMB; on mine only the media folder is shared).

Set or refresh environment variables before the re-create. Put new or changed variables into the envlist before step 6, so the new container starts with them. [UNVERIFIED: the order comes from my script; whether a running container picks up envlist changes without a re-create was not tested]
```
ssh <ROUTER> '/container/envs/remove [find list=<LIST> key=<KEY>]; /container/envs/add list=<LIST> key=<KEY> value="<VALUE>"'
```
Check that the key exists without printing its value:
```
$ ssh <ROUTER> ':put [:len [/container/envs/find list=observer key=OBSERVER_PLUG_PASSWORD]]'
1
```
Expected: 1. Skip this step if no variable changes.

Upload the tar under a versioned name. The old tar stays on the disk as rollback material.

scp <IMAGE>.tar "<ROUTER>:/<DIR>/<IMAGE>-<VER>.tar"

Upload any files the container reads from its mounts in the same step. Check the disk:

$ ssh <ROUTER> '/file/print where name~"observer/nut-observer"'
Columns: NAME, TYPE, SIZE, LAST-MODIFIED
# NAME                                  TYPE       SIZE    LAST-MODIFIED      
0 usb1-part1/observer/nut-observer-eb5  .tar file  6.2MiB  2026-10-16 14:32:48
  9246.tar                                                                    
1 usb1-part1/observer/nut-observer-27e  .tar file  6.1MiB  2026-10-16 13:23:21
  8b5d.tar                                                                    
2 usb1-part1/observer/nut-observer-3f4  .tar file  6.2MiB  2026-10-16 23:55:46
  f7db.tar                                                                    
3 usb1-part1/observer/nut-observer.tar  .tar file  6.0MiB  2026-10-15 23:04:11

Expected: <IMAGE>-<VER>.tar is listed with the size of your local tar. This listing is from about ten minutes after the third update: every tar is still there.

Write the re-create script <NAME>-update-<VER>.rsc. Stop, wait, remove, then add the container again from the new tar into a new root-dir, repeating every other parameter of the original /container/add. Mine for eb59246 (generated by my update script, broker address replaced):
```
# Re-creates the observer container from nut-observer-eb59246.tar (generated by update-observer.sh).
/container/stop [find name=observer]
:delay 5s
/container/remove [find name=observer]
/container/add name=observer file="usb1-part1/observer/nut-observer-eb59246.tar" interface=veth-observer \
    root-dir="/usb1-part1/observer/root-eb59246" mountlists=observer envlists=observer \
    cmd="serve -listen 0.0.0.0:3493 -broker tcp://192.0.2.3:1883 -config /config/observer.json" \
    start-on-boot=yes restart-policy=always restart-interval=10s logging=yes memory-high=48MiB comment=observer
```
Change name, file, root-dir, interface, mountlists, envlists and cmd to yours. Keep restart-policy=always: on 7.24 the default is no, and a container that exits then stays down. The veth, the mountlist, the envlist and the NAT rule are separate objects, so removing the container leaves them in place and the add line picks them up by name. Keep a copy of this file next to your build: it is the next update's rollback.

Upload and import the script.

scp <NAME>-update-<VER>.rsc "<ROUTER>:/<DIR>/"
ssh <ROUTER> "/import file-name=<DIR>/<NAME>-update-<VER>.rsc"

$ ssh <ROUTER> '/import file-name=usb1-part1/observer/observer-update-eb59246.rsc'
Script file loaded and executed successfully

Expected: exactly that line. The service is down from here until step 7 ends.

Start the container once the image is extracted. /container/add only starts the extraction; the container has to be started by hand afterwards. MikroTik's page says to wait for status=stopped, but 7.24.4 has no status property, so retry start and read running:
```
for _ in $(seq 1 60); do
  run=$(ssh <ROUTER> ':do { /container/start [find name=<NAME>] } on-error={}; :delay 3s; :put [/container/get [find name=<NAME>] running]' | tr -d '\r')
  echo "running: $run"; [ "$run" = "true" ] && break; sleep 5
done
```
```
running: true
```
Expected: running: true. In all three runs it came on the first pass. If the loop prints running: false every time, read /log/print where topics~"container" before changing anything.

Find a model's real context window in the GGUF header (no llama.cpp)

Sat, 24 Oct 2026 00:00:00 +0000

A stdlib-only Python script reads a GGUF file's metadata header and prints the context length the model was trained for, plus its RoPE scaling keys (YaRN factor, original context), so you can set the server's context to match.

Tested on: GEEKOM A9 Mega, Fedora 45, Python 3.15.0rc3; GEEKOM AE8, Fedora 44, Python 3.14.8 (2026-10-16). GGUF versions 2 and 3, single-file and split models.
Needs: Python 3 on the machine that holds the .gguf file (or SSH access to it). No llama.cpp, no pip packages, no GPU.
Time: 5 min.
Risk: none. The script opens the file read-only and reads only the header; the model file is not modified (checked in Verify).

Before you start#

Python 3 is installed where the model lives:

$ ssh a9 'python3 --version; cat /etc/fedora-release'
Python 3.15.0rc3
Fedora release 45 (Forty Five)

Placeholders used below:
- <FILE.gguf> — path to the model file. For a split model, the shard named …-00001-of-<N>.gguf.
- <HOST> — SSH host that stores the model, if it is not the machine you type on.
- <SERVER_LOG> — your inference server's log file.
- <ORG>/<MODEL> — the upstream Hugging Face repo of the original (non-GGUF) model, for the optional cross-check.
Backups: none needed. Nothing is written.

Steps#

Save the script as ggufctx.py.

#!/usr/bin/env python3
"""Print a GGUF file's trained context length and RoPE scaling keys.

Reads only the metadata header (no tensor data), stdlib only, never writes.
Usage: python3 -I ggufctx.py <FILE.gguf>
"""
import struct
import sys

# GGUF metadata value types (ggml docs/gguf.md, enum gguf_metadata_value_type)
FIXED = {0: "<B", 1: "<b", 2: "<H", 3: "<h", 4: "<I", 5: "<i", 6: "<f",
         7: "<?", 10: "<Q", 11: "<q", 12: "<d"}
STRING, ARRAY = 8, 9


def wanted(key):
    return (key in ("general.architecture", "general.name")
            or key.endswith(".context_length")
            or ".rope.scaling." in key
            or key.startswith("split."))


def main(path):
    with open(path, "rb") as f:
        def read(n):
            b = f.read(n)
            if len(b) != n:
                sys.exit(f"{path}: header ends early (truncated file?)")
            return b

        def num(fmt):
            return struct.unpack(fmt, read(struct.calcsize(fmt)))[0]

        def string():
            return read(num("<Q")).decode("utf-8", "replace")

        def value(t):
            if t == STRING:
                return string()
            if t == ARRAY:
                et, n = num("<I"), num("<Q")
                if et in FIXED:
                    read(n * struct.calcsize(FIXED[et]))  # skip, works on pipes too
                else:
                    for _ in range(n):
                        value(et)
                return f"<array, {n} items>"
            if t not in FIXED:
                sys.exit(f"{path}: unknown value type {t}")
            return num(FIXED[t])

        if read(4) != b"GGUF":
            sys.exit(f"{path}: not a GGUF file (first 4 bytes are not 'GGUF')")
        version, n_tensors, n_kv = num("<I"), num("<Q"), num("<Q")
        print(f"gguf version {version}, {n_tensors} tensors, {n_kv} metadata keys")
        for _ in range(n_kv):
            key = string()
            v = value(num("<I"))
            if wanted(key):
                print(key, v)


if __name__ == "__main__":
    if len(sys.argv) != 2:
        sys.exit("usage: python3 -I ggufctx.py <FILE.gguf>")
    main(sys.argv[1])

Check that it parses:

$ python3 -I ggufctx.py
usage: python3 -I ggufctx.py <FILE.gguf>

Expected: the usage line (exit code 1). A SyntaxError means the paste lost indentation.

Run it on the model file. On the machine that has the file:

python3 -I ggufctx.py <FILE.gguf>

$ python3 -I ggufctx.py ~/models/llama-2-7b.Q4_0.gguf
gguf version 2, 291 tensors, 19 metadata keys
general.architecture llama
general.name LLaMA v2
llama.context_length 4096

If the file is on another box, pipe the script over SSH instead of copying it there (- makes Python read the script from stdin; the file path is the remote one):

ssh <HOST> 'python3 -I - <FILE.gguf>' < ggufctx.py

$ ssh a9 'python3 -I - ~/models/Laguna-XS-2.1-GGUF/Laguna-XS-2.1-Q8_0.gguf' < ggufctx.py
gguf version 3, 678 tensors, 61 metadata keys
general.architecture laguna
general.name Laguna XS 2.1
laguna.context_length 262144
laguna.rope.scaling.type yarn
laguna.rope.scaling.factor 32.0
laguna.rope.scaling.original_context_length 8192
laguna.rope.scaling.yarn_attn_factor 1.3465735912322998
laguna.rope.scaling.yarn_beta_fast 64.0
laguna.rope.scaling.yarn_beta_slow 1.0

Expected: a gguf version line, then general.architecture and one <arch>.context_length line. On the 35.6 GB Laguna file the run took 0.070 s wall time, because only the header is read. Errors: see Failure modes, rows 1-4.

For a split model, run it on shard 1. Only the first shard carries the model metadata:

$ ssh a9 'python3 -I - ~/models/Laguna-S-2.1-GGUF/Laguna-S-2.1-UD-Q5_K_XL-00001-of-00003.gguf' < ggufctx.py
gguf version 3, 0 tensors, 72 metadata keys
general.architecture laguna
general.name Laguna-S-2.1
laguna.context_length 262144
laguna.rope.scaling.type yarn
laguna.rope.scaling.factor 32.0
laguna.rope.scaling.original_context_length 8192
laguna.rope.scaling.yarn_attn_factor 1.0
laguna.rope.scaling.yarn_beta_fast 32.0
laguna.rope.scaling.yarn_beta_slow 1.0
split.no 0
split.tensors.count 814
split.count 3

Expected: split.no 0 (numbering starts at 0) and the context_length line. In this file shard 1 holds no tensors at all, only metadata. If you see only split.* lines, you ran a later shard: see Failure modes, row 5.

Read the numbers.
- <arch>.context_length is the model's context window. The GGUF spec defines it as the "length of the context (in tokens) that the model was trained on". In both files cross-checked here (Laguna-XS, Qwen3-30B-A3B) it equals max_position_embeddings in the original model's config.json. That is usually the ceiling to serve at, but a model card can state a smaller native window (Qwen3 example below), so read the card too (Verify).
- <arch>.rope.scaling.type yarn with factor and original_context_length means the window was reached by YaRN-extending a shorter base: Laguna-XS is 32.0 × 8192 = 262144. In all seven YaRN files on this box (factors 16 to 128: DeepSeek-V4-Flash, Mistral-Small-4, three gpt-oss, two Laguna), factor × original_context_length equals context_length; gpt-oss-120b is 32.0 × 4096 = 131072:
```
$ ssh a9 'python3 -I - ~/models/gpt-oss-120b-MXFP4.gguf' < ggufctx.py
gguf version 3, 687 tensors, 36 metadata keys
general.architecture gpt-oss
general.name gpt-oss-120b
gpt-oss.context_length 131072
gpt-oss.rope.scaling.type yarn
gpt-oss.rope.scaling.factor 32.0
gpt-oss.rope.scaling.original_context_length 4096
gpt-oss.rope.scaling.yarn_beta_fast 32.0
gpt-oss.rope.scaling.yarn_beta_slow 1.0
```
- No rope.scaling lines: the file stores no scaling. context_length is then whatever the config declares, which is not always the native window the card names:
```
$ ssh a9 'python3 -I - ~/models/Qwen3-30B-A3B-Q4_K_M.gguf' < ggufctx.py
gguf version 3, 579 tensors, 37 metadata keys
general.architecture qwen3moe
general.name Qwen3 30B A3B
qwen3moe.context_length 40960
```
  The Qwen3-30B-A3B card says "32,768 natively" and explains that max_position_embeddings = 40,960 reserves 32,768 tokens for output plus 8,192 for prompts; its config.json has max_position_embeddings 40960 and rope_scaling null.
- The yarn_* keys are inputs your inference engine has to read. A build that ignores them runs the model with its own defaults (see Notes, llama.cpp #30331).
Compare with the context your server actually uses. The configured value is not proof; read the number the server prints at load. With llama-server (-c / --ctx-size; 0 means "loaded from model"), it is the n_ctx_slot line in the server log:
```
$ ssh a9 'grep -H n_ctx_slot ~/bench-archive/serve/laguna-xs-q8-vulkan-new-2026101*.log'
/home/user/bench-archive/serve/laguna-xs-q8-vulkan-new-20261016T050201.log:0.09.628.876 I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 131072, kv_unified = 'false'
/home/user/bench-archive/serve/laguna-xs-q8-vulkan-new-20261016T143358.log:0.10.328.547 I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 262144, kv_unified = 'false'
```
For your own server: grep n_ctx_slot <SERVER_LOG>. Expected: the served value equals context_length from step 2. The first log above is a server started at 131072 for a model whose header says 262144: half the trained window. The second is the same model restarted at 262144. A lower served value is legitimate only when you chose it (memory budget); otherwise raise -c (or your engine's equivalent) to context_length.

Turn on OpenBeken web auth and manage scripts over HTTP

Sat, 17 Oct 2026 00:00:00 +0000

Two shell scripts switch on an OpenBeken plug's web password over HTTP without touching its Wi-Fi settings, prove that requests without the password get 401, then upload .bat scripts to the plug's filesystem and restart it, with the password kept out of every command line.

Tested on: RZTK C3680W plug (OEM FrankEver FK-PW802EC-F, Tuya WB2S module, BK7231T) running OpenBeken 1.18.318 (OpenBK7231T build); client Fedora 44, curl 8.18.0, bash 5.3.9 (2026-10-16).
Needs: an OpenBeken plug on your LAN with web auth still off; bash and curl on the client; the list of everything that polls the plug (Home Assistant, a monitor, a script).
Time: 15 min.
Risk: pollers without the password start getting 401 the moment auth goes on. The restart in the upload step drops the relay for a few seconds, so whatever is plugged into it loses power briefly. A lost password locks you out of the web UI and API. Undo is one request, but it was not exercised (see Undo).

What this protects and what it does not: OpenBeken's auth is HTTP Basic over plain HTTP, and the enable request carries the new password in a GET query string. It keeps a curious device on the LAN away from the relay, the firmware update endpoint and the flash reader. It does not stop anyone who can capture the plug's traffic.

Strip comments from a polyglot repo for a public mirror

Fri, 16 Oct 2026 00:00:00 +0000

This strips every comment from an exported code tree (Java, shell with embedded awk, Python, YAML, XML, Go, JS, properties, Dockerfiles), keeps only what a machine or a licence needs, and proves each changed file equivalent to its source with a per-language oracle, plus control tests that show each oracle can fail.

Tested on: Fedora Linux 44, kernel 7.2.10-200.fc44.x86_64; bash 5.3.9, Python 3.14.8, PyYAML 6.0.3, gawk 5.3.2, Go 1.26.0, Node v22.22.0, Temurin 25.0.4 with Maven 3.9.11 (2026-10-16), on 18 branches of one Java benchmark repo.
Needs: a git repo you will export from; the scripts attached to this page (oss-strip-comments.py, test-oss-strip-comments.py, blind-oracle.py, naive.py, residual.py, strip-all.sh); the tools above on PATH; Maven able to resolve the build offline (-o) or online.
Time: 30 min.
Risk: none to the source repo: every step works on a git archive copy. The stripper rewrites files in place, so run it only on that copy. Publishing (force-pushing to the mirror) is outside this manual.

Before you start#

Placeholders. <SRC_REPO>: path to the repo you export from. <BRANCH>: one branch to try first (mine was micronaut/imperative-synth-more-io). All commands run from one empty working directory that holds the six attached scripts.
Policy the script implements. Kept: the ASF Maven wrapper files untouched (mvnw, mvnw.bat, mvnw.cmd, maven-wrapper.properties; Apache-2.0 §4(c) requires keeping their attribution notices), a shebang on line 1, Dockerfile parser directives (# syntax=, # escape=, # check=) at the top, Go //go: and // +build lines. Removed: Java/JS/Go comments and Javadoc, XML comments, YAML and shell comments (whole-line and trailing), Python comments and docstrings, comments inside embedded awk programs (including programs spliced with '"$var"'), whole-line #, ! and // lines in .properties and .conf, whole-line # comments in .gitignore and .dockerignore.
Files the stripper never opens. .html, .txt, .json, .cmd, .md and any suffix not in its map. A vendored licence header in an HTML report (my reports inline Plotly with its MIT header and a /*! For license information please see plotly.min.js.LICENSE.txt */ pointer) survives for that reason, not because of an exception rule. If you ship a vendored .js bundle, its header would be stripped: add it to LICENCE_EXEMPT first.

Oracles per file type.

Type	Oracle	Strength
`.sh`, `.bash`	`bash --pretty-print` of source and result, blank lines dropped, must be equal	full parse
awk inside shell	`gawk -o-` (pretty-print) of each program before and after, comments dropped, must be equal	full parse, per program
`.py`	`ast.dump` of source with docstrings removed equals `ast.dump` of result	full AST
`.yml`, `.yaml`	`yaml.safe_load_all` of both equal	data equality
`.xml`	`ElementTree.canonicalize(with_comments=False, strip_text=True)` of both equal	canonical form
`.go`	token stream from `go/scanner` (comments skipped) of both equal	token stream
`.js`	`node --check` of the result	syntax only
`.java`, `.properties`, `.conf`, `.mod`, Dockerfile	none	covered by Step 8 for Java; the other four only by the unchanged-line net

Every stripper first blanks comments in place, so line numbers stay aligned for the safety checks; a post-pass then drops the lines that held only a comment, so the final file is shorter (Step 6's hunk header shows the shift). A second net aborts if any line the stripper did not touch is missing from the output.

Check the tools. bash --pretty-print is listed by bash --help but not documented in man bash; check that your build has it.

$ bash --version | head -1; python3 -c 'import sys, yaml; print("Python", sys.version.split()[0], "PyYAML", yaml.__version__)'; gawk --version | head -1; go version; node --version; bash --help | command grep -- --pretty-print
GNU bash, version 5.3.9(1)-release (x86_64-redhat-linux-gnu)
Python 3.14.8 PyYAML 6.0.3
GNU Awk 5.3.2, API 4.0, PMA Avon 8-g1, (GNU MPFR 4.2.2, GNU MP 6.3.0)
go version go1.26.0 linux/amd64
v22.22.0
	--pretty-print

Expected: all six lines. A missing go, node or gawk does not fail the run: the affected files are stripped unchecked and reported as WARN: no oracle could check … (see Failure modes).

Backup. Nothing in the source repo is written. Step 3 keeps a pristine/ copy of the export next to the stripped tree/.

Make Konsole follow Plasma's light/dark color scheme

Tue, 13 Oct 2026 00:00:00 +0000

Make Konsole switch its windows between a light and a dark profile whenever Plasma's color scheme changes, open windows included, using the profile sync that Konsole 26.08 ships switched off.

Tested on: Fedora 44 desktop, KDE Plasma 6.7.5 on X11, Konsole 26.08.1, kf6-kconfig 6.31.0, Qt 6.11.2 (2026-10-25).
Not verified: Plasma's own scheduled switch (AutomaticLookAndFeel) driving Konsole; I only watched scheme changes I triggered with step 5's commands. A new window opened while Plasma is light, and new tabs: the light-mode windows I tested that day all predate the light profile (Field notes). Starting the test in light mode. The settings-dialog route (Notes). A bundled light scheme such as SolarizedLight; I used my own. Whether a Konsole process that was already running picks up step 3. Wayland, other distributions, Konsole 26.08.0. The rollback on the live desktop (run on copies of the files only).
Needs: Konsole 26.08.0 or newer; a Plasma 6 session; kwriteconfig6 and kreadconfig6 (Fedora package kf6-kconfig); plasma-apply-colorscheme (Fedora package plasma-workspace) for the test; a dark Konsole profile whose file has a ColorScheme= line; a light Konsole color scheme. Optional: strings (binutils) to list Konsole's bundled schemes; xdotool and ImageMagick to screenshot the test window.
Time: 15 min.
Risk: low. The setup is per user: konsolerc gains one group of three keys and one profile file is added. With the sync on, Konsole puts the light or dark profile on every tab of that Konsole process whenever a window opens and at each switch (step 3). The test in step 5 changes Plasma's color scheme for the whole session and leaves an extra ColorScheme= line in ~/.config/kdeglobals; step 1 copies that file and step 6 puts it back. Plasma also rewrote its GTK and Qt theme files under ~/.config/ during my round trip (gtk-3.0/, gtk-4.0/, gtkrc, gtkrc-2.0, xsettingsd/, dconf/user, Trolltech.conf, all written at the second command); I didn't compare them before and after. Undo: delete the three keys and move the light profile out (Undo / rollback).

Before you start#

Konsole is 26.08.0 or newer. That is the first stable release with the setting.
```
$ rpm -q konsole
konsole-26.08.1-1.fc44.x86_64
```
Expected: 26.08.0 or later. On other distributions, ask the package manager.
The sync is still off, and you know which profile Konsole uses now:
```
$ kreadconfig6 --file konsolerc --group LightDarkTheme --key SyncProfileWithSystemTheme

$ kreadconfig6 --file konsolerc --group 'Desktop Entry' --key DefaultProfile
user-default.profile
$ kreadconfig6 --file kdeglobals --group KDE --key AutomaticLookAndFeel
true
```
Expected: an empty line for SyncProfileWithSystemTheme (unset, and off by default). DefaultProfile is the profile file new windows get today; mine is the dark one. AutomaticLookAndFeel is Plasma's own automatic light/dark switching. It was on here; Konsole reacts only to the color scheme, so the manual works without it by the source (not tried with it off).
Your dark profile has a ColorScheme= line, and you have a light scheme:
```
$ cd ~/.local/share/konsole
$ grep -H -e '^Name=' -e '^ColorScheme=' *.profile
user-default.profile:ColorScheme=islands-dark
user-default.profile:Name=user-default
$ ls -1 *.colorscheme
islands-dark.colorscheme
islands-light.colorscheme
```
Expected: the dark profile with its Name= and ColorScheme=. Step 2 assumes the file is named after the profile (<Name>.profile), as mine is. A scheme's name is its .colorscheme file name without the extension. A profile without a ColorScheme= line is not covered here.
If you have no light scheme of your own: Konsole 26.08.1 builds its bundled schemes into its library (in the Fedora package konsole-part), and this lists them:
```
$ strings -e b /usr/lib64/libkonsoleprivate.so.* | grep '\.colorscheme$'
*.colorscheme
RedOnBlack.colorscheme
BlueOnBlack.colorscheme
Breeze.colorscheme
GreenOnBlack.colorscheme
Solarized.colorscheme
SolarizedLight.colorscheme
BlackOnWhite.colorscheme
Campbell.colorscheme
DarkPastels.colorscheme
Linux.colorscheme
BlackOnLightYellow.colorscheme
WhiteOnBlack.colorscheme
BlackOnRandomLight.colorscheme
```
Expected: the bundled scheme names. The first line, *.colorscheme, is a file-name filter from Konsole's code, not a scheme. Going by their names, the light ones are SolarizedLight, BlackOnWhite, BlackOnLightYellow and BlackOnRandomLight. Konsole looks for <name>.colorscheme under ~/.local/share/konsole/ and the system data directories first, then in this bundle (source: ColorSchemeManager.cpp). Mine is islands-light, made from the light variant of my editor theme (Field notes); I haven't put a bundled one in a profile.
Variables instead of placeholders: step 2 sets DARK (your dark profile's Name=), LIGHT (a name for the new light profile) and LIGHT_SCHEME (the light scheme's name). Steps 2 and 3, Verify and the undo use them; in a new shell, set them again. Step 5's screenshot method names <PID>, <WINDOW_ID> and <FILE>.

Steps#

Back up konsolerc, kdeglobals and your profiles.

$ mkdir -p ~/konsole-backup
$ cp -p ~/.config/konsolerc ~/.config/kdeglobals ~/.local/share/konsole/*.profile ~/konsole-backup/
$ ls -1 ~/konsole-backup
kdeglobals
konsolerc
user-default.profile

Expected: the two config files and every .profile file. kdeglobals is in the list because step 5 writes to it.

Make a light twin of the dark profile. Put your names in the second line (mine are shown), then copy the dark profile under the new name with the light scheme:

$ cd ~/.local/share/konsole
$ DARK=user-default LIGHT=user-light LIGHT_SCHEME=islands-light
$ sed -e "s/^Name=.*/Name=$LIGHT/" -e "s/^ColorScheme=.*/ColorScheme=$LIGHT_SCHEME/" "$DARK.profile" > "$LIGHT.profile"
$ diff "$DARK.profile" "$LIGHT.profile"
2c2
< ColorScheme=islands-dark
---
> ColorScheme=islands-light
7c7
< Name=user-default
---
> Name=user-light

Expected: exactly two lines differ, ColorScheme= and Name=. If diff shows only the Name= line, both profiles use the same scheme: Failure modes, row 2.

Turn on the sync and name the two profiles.
```
$ kwriteconfig6 --file konsolerc --group LightDarkTheme --key SyncProfileWithSystemTheme true
$ kwriteconfig6 --file konsolerc --group LightDarkTheme --key LightThemeProfile "$LIGHT"
$ kwriteconfig6 --file konsolerc --group LightDarkTheme --key DarkThemeProfile "$DARK"
$ kreadconfig6 --file konsolerc --group LightDarkTheme --key SyncProfileWithSystemTheme
true
$ kreadconfig6 --file konsolerc --group LightDarkTheme --key LightThemeProfile
user-light
$ kreadconfig6 --file konsolerc --group LightDarkTheme --key DarkThemeProfile
user-default
```
Expected: true, then the light and the dark name, spelled exactly as the profiles' Name= lines. Konsole looks the profiles up by that name; a name that matches no profile gets Konsole's built-in profile instead, without a message (source: ProfileManager.cpp; not tried).

NoteEvery tab gets the pair

Whenever a window opens and on each switch, Konsole sets the light or the dark profile on every session of that Konsole process, so a tab you opened with another profile changes to the pair too (source: applyThemeProfile() in MainWindow.cpp; not tried).
Open a test window in a new Konsole process.
```
konsole --separate &
```
Don't pass --profile: the sync has to choose the profile. --separate starts a new process, which reads konsolerc when it starts, even where Konsole is set to run all windows in one process (source: main.cpp; that option, UseSingleInstance, is off by default). To see the ANSI palette as well as the background, you can run this loop in the test window:
```
for i in 0 1 2 3 4 5 6 7; do printf '\e[3%sm normal%s \e[9%sm bright%s \e[2;3%sm faint%s\e[0m\n' $i $i $i $i $i $i; done
```
It comes from the script I used for one of that afternoon's light-mode screenshots (13:49). The live-switch window in Verify printed a single color0–color7 row instead, with a command I didn't record.

Expected: the colors of the profile for Plasma's current mode. Plasma was dark here and the window came up dark, background pixel 24,25,27.
Switch Plasma to the other scheme and back while the test window is open. First, the scheme the session is on:
```
$ kreadconfig6 --file kdeglobals --group General --key ColorScheme
BreezeDark
```
Expected: the current scheme's name. Here ~/.config/kdeglobals itself has no ColorScheme= line; the value comes from ~/.config/kdedefaults/kdeglobals, which is first in XDG_CONFIG_DIRS in my session. Switch to the other scheme first: by its source, plasma-apply-colorscheme given the scheme that is already set only reports that and changes nothing (not tried). Mine was BreezeDark, so:
```
plasma-apply-colorscheme BreezeLight
```
Wait about 4 s and look at the test window, then switch back:
```
plasma-apply-colorscheme BreezeDark
```
By its source (plasma-workspace 6.7.5), each successful call prints Successfully applied the color scheme <name> to your current Plasma session. Expected: the open window turns light, then dark again.

Screenshots, for the screen half of Verify (optional): save the test window before the first command and about 4 s after each, as ~/konsole-test/1-dark-start.png, 2-after-light.png and 3-after-dark.png (names for a start in dark mode; mkdir -p ~/konsole-test first). I found the window with xdotool search --pid <PID> --onlyvisible, where <PID> is the konsole --separate process ($! right after step 4), and saved it with ImageMagick's import -window <WINDOW_ID> <FILE>. The exact command lines weren't recorded, and my files went to a different directory; I copied them into ~/konsole-test/ for the re-run in Field notes.

Run both directions. On my box a window that comes up dark proves little: DefaultProfile is my dark profile, so with the sync off a new window is dark in dark mode too. The switch to light shows that the sync works. The switch back shows that the dark profile is still dark; if both profiles carry the light scheme (Failure modes, row 2), a window in light mode looks right either way.

These two scheme names are the only ones I tried. My session was dark, on the Breeze Dark global theme (LookAndFeelPackage=org.kde.breezedark.desktop), so the round trip ended in dark mode, where it started. If you start in light mode or on another scheme, end on the one you started from [UNVERIFIED: not tried].

Gotchaplasma-apply-colorscheme writes to kdeglobals

After the round trip, ~/.config/kdeglobals had its own ColorScheme=BreezeDark line under [General] (step 6). A key in that file takes precedence over the kdedefaults value, so the scheme was now set in ~/.config/kdeglobals itself. I put the copy back so the file is as it was. Whether the extra line would block Plasma's automatic switching, I didn't check.
Put kdeglobals back.
```
$ diff ~/konsole-backup/kdeglobals ~/.config/kdeglobals
135a136
> ColorScheme=BreezeDark
$ cp -p ~/konsole-backup/kdeglobals ~/.config/kdeglobals
$ diff ~/konsole-backup/kdeglobals ~/.config/kdeglobals
```
Expected: the first diff shows what the test added (one line here). After the copy, diff prints nothing. The copy brings back the whole file as it was at step 1: if the first diff shows more than the ColorScheme= line, delete that one line in an editor instead (not tried). Don't remove it with kwriteconfig6 --delete: that leaves a ColorScheme[$d] marker behind (Failure modes, row 4). I didn't check what copying the file back does to a running session, so finish step 5 on the scheme you want to keep.
Close the test window and restart Konsole. By its source, Konsole applies the sync when a window is created and when its settings change from inside Konsole (MainWindow.cpp), and I didn't check whether a running Konsole notices an outside edit of konsolerc. A new Konsole process has it, as the test window did.

Prune a stale synced copy with an rsync checksum dry run

Wed, 23 Sep 2026 00:00:00 +0000

Compare an old one-way copy of another machine's tree with that machine by checksum, without moving any file contents, delete locally only the top-level entries whose files all exist on the source with the same checksum (git's .git/index and FETCH_HEAD aside), and revoke the temporary SSH key.

Tested on: Fedora 44 (kernel 7.2.10), rsync 3.5.1 protocol 33, GNU findutils 4.10.0, GNU Awk 5.3.2, against a loopback remote shell on synthetic trees (2026-10-25). Original run: a Fedora workstation against an older laptop over SSH (2026-10-23 or 2026-10-24, the day was not recorded); the rsync versions and the laptop's OS were not recorded.
Not verified: steps 1, 10 and 11 against a real SSH server, and step 2 over ssh (it ran through the loopback shell). No second machine was reachable for this manual; the key-removal command ran only against a fake authorized_keys through the loopback shell, and step 11's identity check only as ssh -G.
Needs: bash, rsync on both machines, an SSH login on the source, and the source owner's permission to add a temporary key and remove it again. GNU find and xargs, awk and coreutils on this machine.
Time: about 30 min of hands-on steps (an estimate; the original run was not timed), plus one checksum pass over both trees.
Risk: step 8 deletes local entries for good. Every file in them matched the source by checksum at the step 7 re-check, except .git/index and .git/FETCH_HEAD in git-noise entries: Undo brings back the source's versions of those two, so the copy's own index, and anything staged in it, cannot be restored. Undo needs access to the source again. On the source, step 1 adds one key and step 10 removes it; the dry runs write nothing there, but they make it read every file that has the same size as its local counterpart.

Before you start#

Placeholders used below:
- <ABSOLUTE_PATH_OF_COPY>: root of the stale copy on this machine.
- <USER>@<HOST>: the SSH login on the source machine.
- <PATH_ON_SOURCE>: the tree the copy was made from, relative to that login's home or absolute.
- <KEY>: file name for the temporary key, for example tmp-compare.
- <ENTRY>: one top-level entry name (Undo only).
- <ABSOLUTE_PATH_OF_SHIM>: where you saved the test shim (Notes only).
Set them once, in the bash shell you use for every step (the commands rely on bash splitting an unquoted $RSYNC_RSH into words):
```
LOCAL=<ABSOLUTE_PATH_OF_COPY>
SRC=<USER>@<HOST>
REMOTE=<PATH_ON_SOURCE>
```
Work from a directory outside $LOCAL, for example mkdir -p ~/copy-compare && cd ~/copy-compare. The steps write dry.txt, verdicts.tsv and similar files to the current directory; inside the copy they would turn up in the comparison.

rsync, GNU find, awk, and a UTF-8 character set (in the C locale rsync escapes non-ASCII names; see Failure modes):

$ rsync --version | head -1; find --version | head -1; awk --version | head -1; locale charmap
rsync  version 3.5.1-g315819c5  protocol version 33
find (GNU findutils) 4.10.0
GNU Awk 5.3.2, API 4.0, PMA Avon 8-g1, (GNU MPFR 4.2.2, GNU MP 6.3.0)
UTF-8

Expected: the last line is UTF-8.

Nothing under $LOCAL is unreadable to you:
```
$ find "$LOCAL" $ ! -type l ! -readable -o -type d ! -executable $ -prune -printf '%U %m %P\n'
```
Expected: no output. Each line it prints is an owner uid, a mode and a path that rsync cannot read or enter either. On a test tree with a mode-000 file and a mode-444 directory:
```
$ find "$LOCAL" $ ! -type l ! -readable -o -type d ! -executable $ -prune -printf '%U %m %P\n'
1000 0 both-locked/secret.bin
1000 444 noexec-dir
```
Both kinds can make an entry look identical. rsync prints no itemize line for the contents of a directory it cannot open, which shows up in step 3 as an error. A file it cannot read gets no error at all: when the source's copy of it is unreadable too and has the same size, rsync prints nothing for it (Failure modes, rows 2 and 3). Make the listed paths readable, or move them out of $LOCAL and decide on them separately, before step 3. ! -readable alone misses the mode-444 directory, hence the -executable test. ! -type l leaves symlinks out: rsync compares a symlink's target as text, so one that dangles on this machine is fine, and without that test the check listed a dangling symlink in the same test tree. The original copy held an unreadable directory, a Postgres data directory written by a container: owned by the container's uid, mode 700, and unreadable to du as well.
Backups: none. Nothing is deleted before step 8, and step 8 deletes only entries that step 7 has just re-checked against the source.

Steps#

Create a temporary key and install it on the source.
```
ssh-keygen -t ed25519 -N '' -C tmp-compare -f ~/.ssh/<KEY>
ssh-copy-id -i ~/.ssh/<KEY>.pub "$SRC"
export RSYNC_RSH="ssh -i $HOME/.ssh/<KEY> -o IdentitiesOnly=yes"
```
ssh-copy-id appends the public key to ~/.ssh/authorized_keys on the source, creating the file if needed; it logs in with the account's password (or a key that already works) to do so. RSYNC_RSH is the remote shell rsync uses when no -e is given, so every rsync command below goes through this ssh command. rsync splits the value into separate arguments: a logging shim set as RSYNC_RSH received -i, the key path, -o and IdentitiesOnly=yes as four arguments, followed by the host and rsync. IdentitiesOnly=yes keeps keys held by ssh-agent out of the login; IdentityFile lines from ~/.ssh/config still apply, which is why step 11 tests the key with -F /dev/null. Expected: ssh-copy-id reports one key added. [UNVERIFIED: steps 1 and 2 not run against an SSH server for this manual. The original run used a temporary key; its setup commands were not recorded.]
Check the path and rsync on the source without listing anything. The loopback run (its "source" is a second directory on the same machine, so it reports the local rsync):
```
$ $RSYNC_RSH "$SRC" "test -d $REMOTE && echo dir-ok; command -v rsync && rsync --version | head -1"; echo "rc=$?"
dir-ok
/usr/bin/rsync
rsync  version 3.5.1-g315819c5  protocol version 33
rc=0
```
Expected: three lines, dir-ok, the path of rsync on the source and its version. No dir-ok line: $REMOTE is not a directory there (in the loopback run, a path that does not exist printed only the two rsync lines). No path and version: rsync is not installed there, or not in the remote shell's PATH. The final rc comes from the last command and says nothing about the directory.
Run the checksum dry run, from the copy to the source. The command from the original run, with LOCAL, host and REMOTE as its placeholders: rsync --dry-run -rlc --itemize-changes --out-format='%i %n' LOCAL/ host:REMOTE/ The same command with the variables from Before you start, saving both output streams:
```
$ rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry.txt 2> dry.err; echo "rc=$?"
rc=0
```
Expected: rc=0 and an empty dry.err (step 4 shows its line count). Anything else: find the cause under Failure modes before going on; step 5 refuses to classify while dry.err has a line.
- --dry-run: rsync works out what it would send and reports it; no file contents are sent and nothing is written on the source. The copy's file list, with names, sizes and checksums, does go to the source (Notes). In the loopback runs, a listing of the source tree with sizes, mtimes and ctimes was the same before and after seven dry runs.
- -c: decide by checksum, not by size and modification time. The local side checksums its files while it builds the file list; the source checksums each of its files that has the same size as the local one, and a file whose size differs is reported without one. The algorithm is negotiated between the two ends (xxh128 between the two rsync 3.5.1 ends of the loopback run), so "identical" below means the same checksum. Without -c, the loopback run missed a same-size edit with an unchanged mtime and flagged a file whose bytes matched but whose mtime did not (Notes).
- -r -l: recurse, and compare symlinks as links instead of following them. Not -a: its -t -p -o -g add attribute-only lines, such as .f..t...... notes/touched.txt for a file with identical bytes, and those would turn an identical entry into differs.
- --itemize-changes --out-format='%i %n': one line per item that would change, made of an 11-character change code, a space and the path relative to $LOCAL. The explicit format leaves out the -> target that --itemize-changes alone appends to symlinks, so every line splits the same way.
- "$LOCAL"/ with the trailing slash: compare the contents of the copy with the contents of $REMOTE. The form without the slash is Failure modes, row 4.
- No --delete: it would itemize files that exist only on the source, and without --dry-run it would delete them there.

Read the itemized lines.

$ cat dry.txt; wc -l dry.txt dry.err
<fcsT...... edited/changed-size.txt
<fc.T...... edited/same-size.txt
cd+++++++++ gone-there/
<f+++++++++ gone-there/NOTES
cd+++++++++ gone-there/src/
<f+++++++++ gone-there/src/main.c
<f+++++++++ grown/new-here.txt
<fcsT...... repo-index/.git/FETCH_HEAD
<fc.T...... repo-index/.git/index
  9 dry.txt
  0 dry.err
  9 total

In this direction the local side is the sender, so every file line starts with <, sent to the remote host. On the loopback trees, notes/, repo-same/, top-file.txt and .dotfile print nothing: every file in them matched by checksum, including one whose mtime differs on the source, and the symlink in notes/ points at the same target on both sides.

Line	Meaning here
no line	the item exists on the source with the same checksum (a symlink: the same target)
`<f+++++++++ path`	file missing on the source
`cd+++++++++ dir/`	directory missing on the source; everything under it follows as new
`cL+++++++++ link`	symlink missing on the source
`<fcsT...... path`	content (`c`) and size (`s`) differ
`<fc.T...... path`	same size, different content

A leading c means the receiver would create the item, a directory or symlink here. The T only says the receiver would set the transfer time as mtime, because times are not preserved; it carries no information here. Line codes by kind:

$ cut -c1-11 dry.txt | sort | uniq -c
      2 cd+++++++++
      3 <f+++++++++
      2 <fcsT......
      2 <fc.T......

GotchaCount <f, not >f

> marks an item received by the local side, which happens in a pull. In the original run my first classifier looked for > and counted zero.

$ grep -c '^>f' dry.txt
0
$ grep -c '^<f' dry.txt
7

Give each top-level entry a verdict. Save this as classify.awk in the working directory:

# classify.awk: one verdict per top-level entry of LOCAL, from a push-direction
#   rsync --dry-run -rlc --itemize-changes --out-format='%i %n' LOCAL/ host:REMOTE/ > dry.txt 2> dry.err
# usage: awk -f classify.awk entries.txt dry.txt dry.err > verdicts.tsv
#   identical   no line: every file under it exists on the source with the same checksum
#   git-noise   only .git/index or .git/FETCH_HEAD differ
#   only-local  the entry itself is missing on the source
#   differs     anything else (new or changed files inside)
# Any line in dry.err, or any line of dry.txt it cannot map to an entry,
# aborts with exit 2 and no verdicts: an unreadable directory prints no line.
FILENAME == ARGV[1] { order[++n] = $0; v[$0] = "identical"; next }
FILENAME == ARGV[3] { print "rsync stderr: " $0 > "/dev/stderr"; bad = 1; next }
{
    code = substr($0, 1, 11); path = substr($0, 13); sub(/\/$/, "", path)
    top = path; sub(/\/.*/, "", top)
    if (substr($0, 12, 1) != " " || !(top in v)) {
        print "unmapped line: " $0 > "/dev/stderr"; bad = 1; next
    }
    new = (substr(code, 3) == "+++++++++")
    if (path == top && new) v[top] = "only-local"
    else if (v[top] == "only-local") next
    else if (!new && path ~ /\/\.git\/(index|FETCH_HEAD)$/) {
        if (v[top] == "identical") v[top] = "git-noise"
    }
    else v[top] = "differs"
}
END {
    if (bad) exit 2
    for (i = 1; i <= n; i++) printf "%s\t%s\n", v[order[i]], order[i]
}

List the entries, then classify:

$ find "$LOCAL" -mindepth 1 -maxdepth 1 -printf '%f\n' | sort > entries.txt; cat entries.txt
.dotfile
edited
gone-there
grown
notes
repo-index
repo-same
top-file.txt
$ awk -f classify.awk entries.txt dry.txt dry.err > verdicts.tsv; echo "rc=$?"; cat verdicts.tsv
rc=0
identical	.dotfile
differs	edited
only-local	gone-there
differs	grown
identical	notes
git-noise	repo-index
identical	repo-same
identical	top-file.txt

Expected: rc=0 and one verdict per entry. rc=2 with rsync stderr: or unmapped line: messages and an empty verdicts.tsv: see Failure modes.

identical: no line under the entry; every file in it exists on the source with the same checksum, and every symlink with the same target.
git-noise: the only differing files are .git/index and .git/FETCH_HEAD, and every object, ref and work-tree file matched by checksum. The index caches each tracked file's stat data, which differs between two machines even for identical files, and git status rewrites it when it refreshes that data; FETCH_HEAD records the last git fetch. On the loopback tree, one git status on the source side and a fetch there from a second URL produced exactly this. The index is also git's staging area, so whatever the copy had staged goes with it; the objects it names are on the source, because no file under .git/objects differed. Any other differing file under .git leaves the entry as differs.
only-local: the entry itself is missing on the source; the copy is the only one.
differs: new or changed files inside; either side may hold the newer version.

List the deletion candidates and look at what stays.

$ awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts.tsv | sort > planned.txt; cat planned.txt
.dotfile
notes
repo-index
repo-same
top-file.txt
$ awk -F'\t' '$1 == "only-local" || $1 == "differs"' verdicts.tsv
differs	edited
only-local	gone-there
differs	grown

Expected: planned.txt holds only identical and git-noise entries. Remove from it anything you want to keep anyway. The other entries stay: an only-local entry exists nowhere else, and a differs entry may hold the newer copy of a file.

Re-run the dry run right before deleting, and keep only entries that still qualify. The source can change between step 3 and the deletion:

rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry2.txt 2> dry2.err; echo "rc=$?"; cat dry2.err
awk -f classify.awk entries.txt dry2.txt dry2.err > verdicts2.tsv; echo "rc=$?"
awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts2.tsv | sort | comm -12 planned.txt - > delete.txt
comm -23 planned.txt delete.txt | sed 's/^/dropped: /'; cat delete.txt

In the loopback run, a line was appended to notes/b.txt on the source between the two dry runs (a harness step, to show what a change does):

$ rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry2.txt 2> dry2.err; echo "rc=$?"; cat dry2.err
rc=0
$ awk -f classify.awk entries.txt dry2.txt dry2.err > verdicts2.tsv; echo "rc=$?"
awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts2.tsv | sort | comm -12 planned.txt - > delete.txt
comm -23 planned.txt delete.txt | sed 's/^/dropped: /'; cat delete.txt
rc=0
dropped: notes
.dotfile
repo-index
repo-same
top-file.txt

Expected: rc=0 twice, and delete.txt lists the planned entries that are still identical or git-noise; dropped: lines name those that are not. When the classifier refuses (exit 2), verdicts2.tsv is left empty (0 bytes in the slashless test), so delete.txt comes out empty.

Delete the entries in delete.txt from the copy.

WarningThis step is irreversible on this machine

The entries are gone from the copy. Their files matched the source by checksum at step 7, apart from .git/index and FETCH_HEAD in git-noise entries, whose versions in the copy cannot be brought back. Undo pulls everything else back from the source while the source keeps it.

Size first:

$ (cd "$LOCAL" && xargs -d '\n' du -shc --) < delete.txt
4.0K	.dotfile
132K	repo-index
128K	repo-same
4.0K	top-file.txt
268K	total

Then delete:

while IFS= read -r e; do
  if [ -z "$e" ] || [ "$e" = . ] || [ "$e" = .. ] || [ "${e#*/}" != "$e" ]; then echo "skip: $e" >&2; continue; fi
  rm -rf -- "${LOCAL:?}/$e" && printf 'removed\t%s\n' "$e"
done < delete.txt | tee -a removed.log

$ while IFS= read -r e; do
  if [ -z "$e" ] || [ "$e" = . ] || [ "$e" = .. ] || [ "${e#*/}" != "$e" ]; then echo "skip: $e" >&2; continue; fi
  rm -rf -- "${LOCAL:?}/$e" && printf 'removed\t%s\n' "$e"
done < delete.txt | tee -a removed.log
removed	.dotfile
removed	repo-index
removed	repo-same
removed	top-file.txt

Expected: one removed line per entry in delete.txt and no skip: line. The loop refuses empty names, ., .. and anything containing /, and ${LOCAL:?} makes the shell refuse the rm when $LOCAL is empty or unset instead of running it against /<entry>.

Re-check the copy against the source while you still have access.

$ find "$LOCAL" -mindepth 1 -maxdepth 1 -printf '%f\n' | sort > entries-after.txt
rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry3.txt 2> dry3.err; echo "rc=$?"
awk -f classify.awk entries-after.txt dry3.txt dry3.err
rc=0
differs	edited
only-local	gone-there
differs	grown
differs	notes

Expected: rc=0, and only only-local and differs verdicts, plus any entry you took out of planned.txt on purpose. notes is differs here because of the source-side edit from step 7.

Remove the temporary key from the source. This is the one change on the source after step 1; ask its owner first.
```
BLOB=$(cut -d' ' -f2 ~/.ssh/<KEY>.pub)
$RSYNC_RSH "$SRC" "cd ~/.ssh && cp -p authorized_keys authorized_keys.tmpkey-bak && { grep -vF '$BLOB' authorized_keys.tmpkey-bak > authorized_keys; [ \$? -le 1 ]; } && rm authorized_keys.tmpkey-bak; grep -cF '$BLOB' authorized_keys"
```
Expected: 0, the number of lines still holding the key (the command exits 1 because that last grep matched nothing). This edits authorized_keys only, the file ssh-copy-id appended to; sshd also reads authorized_keys2 by default, which step 1 did not touch. > rewrites the file in place, so it keeps its mode; the backup is deleted only after the filter succeeded. If grep reports an error instead, authorized_keys may be cut short and the backup is still there: cp -p ~/.ssh/authorized_keys.tmpkey-bak ~/.ssh/authorized_keys on the source puts it back (not run). The same command, run through the loopback shell against a fake home, left a file that had held two other keys plus this one with exactly the two others, and a file that had held only this key empty (0 bytes); both kept mode 600, and the backup was gone both times. It needs a POSIX-style login shell on the source and was tested with /bin/sh only. [UNVERIFIED: not run against a real sshd.]
Close any shared connection and check that the key no longer logs in.
```
ssh -O exit "$SRC"
ssh -G -F /dev/null -i ~/.ssh/<KEY> -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true | grep -E '^(identityfile|identitiesonly|batchmode|controlpath) '
ssh -F /dev/null -i ~/.ssh/<KEY> -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true; echo "rc=$?"
```
ssh -O exit closes a master connection if one exists. Without one it fails harmlessly; with a control path that has no master behind it, a local test printed:
```
$ ssh -F /dev/null -o ControlPath=/nonexistent/master.sock -O exit "$SRC"; echo "rc=$?"
Control socket connect(/nonexistent/master.sock): No such file or directory
rc=255
```
ssh -G prints the configuration the test login will use and exits without connecting. With -F /dev/null, ssh reads no ~/.ssh/config and no system-wide config, so the temporary key is the only identity:
```
$ ssh -G -F /dev/null -i keytest/tmp-key -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true | grep -E '^(identityfile|identitiesonly|batchmode|controlpath) '
batchmode yes
identitiesonly yes
identityfile keytest/tmp-key
```
Without -F /dev/null, an IdentityFile line in a config still joins the list, even with IdentitiesOnly=yes:
```
$ cat keytest/ssh_config; ssh -G -F keytest/ssh_config -i keytest/tmp-key -o IdentitiesOnly=yes "$SRC" true | grep '^identityfile '
Host *
    IdentityFile keytest/other-key
identityfile keytest/tmp-key
identityfile keytest/other-key
```
Expected: the test login fails with a Permission denied message and rc=255. Any other error (Could not resolve hostname, Connection refused) means the test never reached authentication; if the source needs settings from ~/.ssh/config, such as Port or ProxyJump, add them as -o options and test again. Only after a Permission denied, delete the local key files: rm ~/.ssh/<KEY> ~/.ssh/<KEY>.pub; unset RSYNC_RSH.

GotchaAn open master connection logs in without the key

With ControlMaster and ControlPersist in ~/.ssh/config, a master connection stays open in the background after the rsync runs, and later sessions reuse its network connection instead of opening a new one. A test login over it reuses a connection that was authenticated before the key was removed, so it says nothing about the key. -O exit closes the master, ControlPath=none keeps the test off any master, and BatchMode=yes stops ssh from falling back to a password prompt.

[UNVERIFIED: the test login was not run against a real sshd. In the original run the multiplexed connection was closed and a fresh login was checked to fail.]