# Restart a Go server under systemd without dropping requests

> A systemd socket holds the port, the Go server drains on SIGTERM, KillMode=mixed spares its children: restarts refuse nothing, cut no stream within the drain.

Published 2026-11-01. HTML version: https://guides.tsvinc.dev/go-socket-activation-drain/


A `.socket` unit keeps the port open across restarts, the Go server takes that socket from systemd and drains running requests on SIGTERM, and `KillMode=mixed` leaves the processes those requests wait on alone, so `systemctl --user restart` refuses no connection and cuts no stream that ends within the drain budget (2 min here).

> **Scope** — Verified on: GEEKOM AE8 mini PC, Fedora Linux 44 (kernel 7.2.10), systemd 259 (259.9-1.fc44) user manager with linger on, Go 1.26.0, curl 8.18.0, iproute2 6.17.0 (2026-10-25). The demo service is `tsvdemo-drain` on 127.0.0.1:18490 (18491 after step 10).
> **Not verified:** system (root) units; IPv6 and Unix-socket `ListenStream=` addresses (more than one `ListenStream=` the server refuses, Failure modes); `Accept=yes`; TLS and HTTP/2; WebSockets and other hijacked connections; a start at boot from the enabled units (I used the runtime unit directory and `enable --runtime`, no reboot); `FreeBind=` for a LAN address.
> **Needs:** a working `systemctl --user`; Go (1.26.0 here); `curl`; `ss`; ports 18490 and 18491 free on 127.0.0.1.
> **Time:** 20 min. **Risk:** low for the demo: two units on 127.0.0.1 in a unit directory that a reboot empties. On your own service, a change of the socket unit refuses or resets new connections for as long as the drain runs (step 10), and a `TimeoutStopSec=` below the drain aborts the stop. Every change here is undone in Undo / rollback.

## Before you start

- Versions and linger:
  ```console
  $ systemctl --version | head -1; go version; loginctl show-user "$USER" -p Linger
  systemd 259 (259.9-1.fc44)
  go version go1.26.0 linux/amd64
  Linger=yes
  ```
- The user manager's default stop timeout. It is 45 s here; the systemd 259 man page says 90 s. Fedora lowered it to 45 s in Fedora 38 (the "Shorter Shutdown Timer" change), and `/usr/lib/systemd/user.conf` lists `#DefaultTimeoutStopSec=45s`. A drain longer than this default needs its own `TimeoutStopSec=` (step 4):
  ```console
  $ systemctl --user show -p DefaultTimeoutStopUSec
  DefaultTimeoutStopUSec=45s
  ```
- Nothing listens on the demo ports, 18490 and 18491 (header line only):
  ```console
  $ ss -ltn '( sport = :18490 or sport = :18491 )'
  State Recv-Q Send-Q Local Address:Port Peer Address:Port
  ```
- Make a directory for the demo and run every command below from it. `UNITDIR` is where `install-units.sh` (step 5) puts the units. I used the runtime unit directory, which systemd reads like `~/.config/systemd/user` and which lives on tmpfs (`/run/user/<UID>`), so a reboot removes the trial. For a permanent install leave `UNITDIR` unset: the script then writes to `~/.config/systemd/user`. Export it again in every new shell: without it `install-units.sh` installs to `~/.config/systemd/user`, and the drop-in and Undo commands below lose their directory.
  ```bash
  mkdir tsvdemo-drain && cd tsvdemo-drain
  export UNITDIR=$XDG_RUNTIME_DIR/systemd/user
  ```
- To apply this to your own service later, copy its unit file first (`cp ~/.config/systemd/user/<APP>.service ~/<APP>.service.orig`, where `<APP>` is your unit's name). The demo itself replaces nothing.

## Steps

1. **Write the server.** Save it as `main.go`. `listenerFrom` takes the socket systemd passes (fd 3, `LISTEN_PID` = this process, `LISTEN_FDS` = `1`) or binds `-listen` itself; `main` unsets the four `LISTEN_*` variables before any child process starts; the first SIGTERM or SIGINT starts the drain and a second one exits at once; `serveUntil` waits for `Shutdown` and calls `Close` when the drain budget runs out; `/stream?n=N` sends N events one second apart, and `&wait=S` first runs `sleep S` as a child process.
   ```go
   // tsvdemo-drain: an SSE demo that takes its listening socket from systemd and drains on stop.
   package main

   import (
   	"context"
   	"errors"
   	"flag"
   	"fmt"
   	"log"
   	"net"
   	"net/http"
   	"os"
   	"os/exec"
   	"os/signal"
   	"strconv"
   	"syscall"
   	"time"
   )

   func main() {
   	listen := flag.String("listen", "127.0.0.1:18490", "address to bind when not socket-activated")
   	drain := flag.Duration("drain", 2*time.Minute, "on stop, how long running requests may finish")
   	flag.Parse()

   	// The listener first, and the LISTEN_* variables gone, before any child process starts.
   	ln, inherited, err := listenerFrom(*listen)
   	if err != nil {
   		log.Fatal(err)
   	}
   	for _, k := range []string{"LISTEN_PID", "LISTEN_FDS", "LISTEN_FDNAMES", "LISTEN_PIDFDID"} {
   		os.Unsetenv(k)
   	}

   	// First SIGTERM/SIGINT: drain. A second one: exit at once.
   	ctx, cancel := context.WithCancel(context.Background())
   	sigs := make(chan os.Signal, 1)
   	signal.Notify(sigs, os.Interrupt, syscall.SIGTERM)
   	go func() {
   		<-sigs
   		cancel()
   		<-sigs
   		log.Print("shutdown: second signal, exiting now")
   		os.Exit(1)
   	}()

   	mux := http.NewServeMux()
   	mux.HandleFunc("GET /stream", stream)
   	srv := &http.Server{Handler: mux, ReadHeaderTimeout: 10 * time.Second}
   	log.Printf("pid %d serving on %s (inherited=%t)", os.Getpid(), ln.Addr(), inherited)
   	if err := serveUntil(ctx, srv, ln, *drain); err != nil {
   		log.Fatal(err)
   	}
   	log.Print("exit")
   }

   // listenerFrom returns the socket systemd passed (fd 3, LISTEN_PID = this process,
   // LISTEN_FDS = 1) or, without socket activation, binds addr itself.
   func listenerFrom(addr string) (net.Listener, bool, error) {
   	if os.Getenv("LISTEN_PID") != strconv.Itoa(os.Getpid()) {
   		ln, err := net.Listen("tcp", addr)
   		return ln, false, err
   	}
   	if n := os.Getenv("LISTEN_FDS"); n != "1" {
   		return nil, false, fmt.Errorf("systemd passed %q sockets, want 1 (one ListenStream= line)", n)
   	}
   	f := os.NewFile(3, "systemd-socket")
   	ln, err := net.FileListener(f) // a dup; close fd 3 itself so children don't inherit it
   	f.Close()
   	return ln, true, err
   }

   // serveUntil serves on ln until ctx ends, lets running requests finish for up to drain, then
   // closes what is left. It returns only after that: Serve returns ErrServerClosed as soon as
   // Shutdown starts, and a main that returned then would cut every running request.
   func serveUntil(ctx context.Context, srv *http.Server, ln net.Listener, drain time.Duration) error {
   	done := make(chan struct{})
   	go func() {
   		defer close(done)
   		<-ctx.Done()
   		log.Printf("shutdown: draining for up to %s", drain)
   		sctx, stop := context.WithTimeout(context.Background(), drain)
   		defer stop()
   		if err := srv.Shutdown(sctx); err != nil {
   			log.Printf("shutdown: drain time over (%v), closing", err)
   			srv.Close()
   			return
   		}
   		log.Print("shutdown: all requests finished")
   	}()
   	err := srv.Serve(ln)
   	if errors.Is(err, http.ErrServerClosed) {
   		<-done
   		return nil
   	}
   	return err
   }

   // stream sends n events, one per second. With wait=S it first runs `sleep S` as a child
   // process: a stand-in for an ssh call or a helper the request waits on.
   func stream(w http.ResponseWriter, r *http.Request) {
   	n, _ := strconv.Atoi(r.URL.Query().Get("n"))
   	if s := r.URL.Query().Get("wait"); s != "" {
   		if err := exec.CommandContext(r.Context(), "sleep", s).Run(); err != nil {
   			log.Printf("stream: child failed: %v", err)
   			http.Error(w, "child failed: "+err.Error(), http.StatusBadGateway)
   			return
   		}
   	}
   	w.Header().Set("Content-Type", "text/event-stream")
   	rc := http.NewResponseController(w)
   	for i := 1; i <= n; i++ {
   		fmt.Fprintf(w, "data: pid=%d event=%d/%d\n\n", os.Getpid(), i, n)
   		if err := rc.Flush(); err != nil {
   			log.Printf("stream: client gone after %d/%d events", i, n)
   			return
   		}
   		if i < n {
   			select {
   			case <-r.Context().Done():
   				log.Printf("stream: cut after %d/%d events", i, n)
   				return
   			case <-time.After(time.Second):
   			}
   		}
   	}
   	log.Printf("stream: %d events sent", n)
   }
   ```
   Create the module and build it:
   ```console
   $ go mod init tsvdemo-drain
   go: creating new go.mod: module tsvdemo-drain
   go: to add module requirements and sums:
   	go mod tidy
   $ go vet ./... && go build -o tsvdemo-drain . && ./tsvdemo-drain -h
   Usage of ./tsvdemo-drain:
     -drain duration
       	on stop, how long running requests may finish (default 2m0s)
     -listen string
       	address to bind when not socket-activated (default "127.0.0.1:18490")
   ```
   Expected: no output from `go vet`, then the two flags.

2. **Check the fallback without systemd.** Started by hand, the server binds the address itself; SIGINT drains and exits 0:
   ```console
   $ ./tsvdemo-drain & sleep 0.5; curl -sN 'http://127.0.0.1:18490/stream?n=3'; kill -INT $!; wait $!; echo "exit status $?"
   2026/10/25 14:20:56 pid 2187190 serving on 127.0.0.1:18490 (inherited=false)
   data: pid=2187190 event=1/3

   data: pid=2187190 event=2/3

   data: pid=2187190 event=3/3

   2026/10/25 14:20:59 stream: 3 events sent
   2026/10/25 14:20:59 shutdown: draining for up to 2m0s
   2026/10/25 14:20:59 shutdown: all requests finished
   2026/10/25 14:20:59 exit
   exit status 0
   ```
   Expected: `inherited=false`, three events, `shutdown: all requests finished`, `exit status 0`. The log lines go to stderr, so they interleave with curl's output. Pasted into an interactive bash, the same line also prints the job lines `[1] <PID>` before the first log line and `[1]+  Done` … `./tsvdemo-drain` before `exit status 0`.

3. **Write the socket unit.** Save it as `tsvdemo-drain.socket`:
   ```ini
   [Unit]
   Description=tsvdemo-drain socket: held across restarts, so connections wait while the old process drains

   [Socket]
   ListenStream=127.0.0.1:18490

   [Install]
   WantedBy=sockets.target
   ```
   - One `ListenStream=` line: the server refuses any `LISTEN_FDS` other than `1` (Failure modes).
   - An IP literal with a port: `systemd-analyze` refuses Go's `:18490` and `localhost:18490` forms (Failure modes).
   - `Accept=` stays at its default, `no`: the listening socket itself goes to one service process.

4. **Write the service unit.** Save it as `tsvdemo-drain.service`; `install-units.sh` replaces `@DEMO@` with the demo directory.
   ```ini
   [Unit]
   Description=tsvdemo-drain: SSE demo that drains running requests on stop
   # Wants, not Requires: if the socket cannot bind, the service tries to bind the address
   # itself and Restart= retries.
   Wants=tsvdemo-drain.socket
   After=tsvdemo-drain.socket

   [Service]
   ExecStart=@DEMO@/tsvdemo-drain -listen 127.0.0.1:18490 -drain 2m
   Restart=on-failure
   RestartSec=5
   # SIGTERM to the main process only: children it waits on keep running during the drain.
   # Whatever is left in the cgroup gets SIGKILL after the main process exits.
   KillMode=mixed
   # Above -drain: when the stop times out, systemd ends the drain.
   TimeoutStopSec=150

   [Install]
   WantedBy=default.target
   ```
   - `Wants=` and `After=` the socket: starting the service starts the socket first. With `Requires=`, a socket that cannot bind fails the service as a dependency and nothing retries; with `Wants=` the server tries to bind by itself, `Restart=on-failure` retries every 5 s, and the first retry after the port is free gets the socket again (Failure modes).
   - `KillMode=mixed`: the default, `control-group`, sends SIGTERM to every process in the unit when the stop begins, so a child process a request waits on dies at once (step 8).
   - `TimeoutStopSec=150`, above `-drain 2m`: when the stop timeout runs out, systemd ends the drain (Failure modes).
   - `-listen` repeats the socket's address. The server uses it only without the socket; change both together.

5. **Write the install script.** Save it as `install-units.sh`. It checks both units with `systemd-analyze` before it stops anything. A new or changed socket file means `stop` of socket and service in one call, then `start` of both: with the service stopped alone, a client that connects during the drain starts it again through the socket, and the socket then refuses to restart while its service runs (Failure modes). An unchanged socket means `stop` of the service only, while new connections queue on the open socket, then `start` of both.
   ```bash
   #!/bin/bash
   # Install or update the tsvdemo-drain units and (re)start them without refusing connections.
   set -euo pipefail
   cd "$(dirname "$0")"
   UNITDIR=${UNITDIR:-$HOME/.config/systemd/user}
   mkdir -p "$UNITDIR" build
   sed "s|@DEMO@|$PWD|" tsvdemo-drain.service > build/tsvdemo-drain.service
   cp tsvdemo-drain.socket build/
   systemd-analyze --user verify build/tsvdemo-drain.socket build/tsvdemo-drain.service
   if cmp -s build/tsvdemo-drain.socket "$UNITDIR/tsvdemo-drain.socket"; then moved=; else moved=1; fi
   install -m644 build/tsvdemo-drain.socket build/tsvdemo-drain.service "$UNITDIR/"
   systemctl --user daemon-reload
   if [ -n "$moved" ]; then
     # New or changed socket: stop both in one call, so no client re-activates the service in between.
     echo "socket file changed: stopping socket and service, then starting both"
     systemctl --user stop tsvdemo-drain.socket tsvdemo-drain.service
     systemctl --user start tsvdemo-drain.socket tsvdemo-drain.service
   else
     # Same socket: stop the service behind it (new connections queue while it drains), start both.
     echo "socket file unchanged: restarting the service behind the open socket"
     systemctl --user stop tsvdemo-drain.service
     systemctl --user start tsvdemo-drain.socket tsvdemo-drain.service
   fi
   ```
   Make it executable:
   ```console
   $ chmod +x install-units.sh
   ```

6. **Install, enable and start the units.** For a permanent install, leave out `--runtime`.
   ```console
   $ echo "UNITDIR=$UNITDIR"; ./install-units.sh
   UNITDIR=/run/user/1000/systemd/user
   socket file changed: stopping socket and service, then starting both
   $ systemctl --user enable --runtime tsvdemo-drain.socket tsvdemo-drain.service
   Created symlink '/run/user/1000/systemd/user/sockets.target.wants/tsvdemo-drain.socket' → '/run/user/1000/systemd/user/tsvdemo-drain.socket'.
   Created symlink '/run/user/1000/systemd/user/default.target.wants/tsvdemo-drain.service' → '/run/user/1000/systemd/user/tsvdemo-drain.service'.
   ```
   Check that the service got the socket:
   ```console
   $ systemctl --user is-active tsvdemo-drain.socket tsvdemo-drain.service; ss -ltnp 'sport = :18490'
   active
   active
   State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess                                    
   LISTEN 0      4096       127.0.0.1:18490      0.0.0.0:*    users:(("tsvdemo-drain",pid=2187490,fd=5))
   $ journalctl --user -u tsvdemo-drain.service -g serving -n 1 -o cat; tr "\0" "\n" < /proc/$(systemctl --user show -p MainPID --value tsvdemo-drain.service)/environ | grep ^LISTEN_
   2026/10/25 14:21:00 pid 2187490 serving on 127.0.0.1:18490 (inherited=true)
   LISTEN_PID=2187490
   LISTEN_PIDFDID=14783253
   LISTEN_FDS=1
   LISTEN_FDNAMES=tsvdemo-drain.socket
   ```
   Expected: both units `active`; one `LISTEN` line on 127.0.0.1:18490; `inherited=true` in the log. `/proc/<PID>/environ` shows the environment systemd set at exec, including `LISTEN_PIDFDID`, which systemd 259 adds; the server's `os.Unsetenv` doesn't change that file, it changes what children get (step 8). If the log says `inherited=false`, see Failure modes.

7. **Restart while a stream runs.** Save `restart-test.sh`. Stream A asks for 15 events; the restart starts after 3.5 s; request B is made 2 s into the drain; `ss` shows the listening socket 1 s later. A second argument replaces the restart command (step 10).
   ```bash
   #!/bin/bash
   # Restart tsvdemo-drain while stream A (15 events) runs; request B is made during the drain.
   # usage: restart-test.sh [PORT [COMMAND...]]; COMMAND defaults to systemctl --user restart.
   P=${1:-18490}
   U=http://127.0.0.1:$P
   shift
   [ $# -gt 0 ] || set -- systemctl --user restart tsvdemo-drain.service
   ts() { date +%T.%3N; }
   stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
   { curl -sSN "$U/stream?n=15" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
   sleep 3.5
   echo "$(ts) restart begins"
   ("$@"; echo "$(ts) restart returned rc=$?") &
   sleep 2
   echo "$(ts) B starts"
   { curl -sSN "$U/stream?n=1" -w 'http=%{http_code} connect=%{time_connect}s total=%{time_total}s\n' 2>&1; echo "curl rc=$?"; } | stamp B &
   sleep 1
   ss -Hltnp "sport = :$P" | stamp ss
   wait
   ```
   ```console
   $ chmod +x restart-test.sh
   $ ./restart-test.sh
   14:21:00.774 A data: pid=2187490 event=1/15
   14:21:01.775 A data: pid=2187490 event=2/15
   14:21:02.776 A data: pid=2187490 event=3/15
   14:21:03.777 A data: pid=2187490 event=4/15
   14:21:04.271 restart begins
   14:21:04.778 A data: pid=2187490 event=5/15
   14:21:05.778 A data: pid=2187490 event=6/15
   14:21:06.274 B starts
   14:21:06.778 A data: pid=2187490 event=7/15
   14:21:07.317 ss LISTEN 1      4096   127.0.0.1:18490 0.0.0.0:*
   14:21:07.780 A data: pid=2187490 event=8/15
   14:21:08.780 A data: pid=2187490 event=9/15
   14:21:09.781 A data: pid=2187490 event=10/15
   14:21:10.781 A data: pid=2187490 event=11/15
   14:21:11.783 A data: pid=2187490 event=12/15
   14:21:12.784 A data: pid=2187490 event=13/15
   14:21:13.785 A data: pid=2187490 event=14/15
   14:21:14.785 A data: pid=2187490 event=15/15
   14:21:14.787 A http=200
   14:21:14.789 A curl rc=0
   14:21:15.293 restart returned rc=0
   14:21:15.306 B data: pid=2188843 event=1/1
   14:21:15.307 B http=200 connect=0.000108s total=9.024878s
   14:21:15.307 B curl rc=0
   ```
   Expected: A gets 15 of 15 events from the old PID and `curl rc=0`. B's connect completes in 0.11 ms, before the new process exists, and B gets its event from the new PID once the drain ends (9.0 s later here). During the drain the old process has already closed its listener (`Shutdown` closes the listeners first), yet `ss` shows the port listening, with `Recv-Q` at 1 while B waits and no process listed: systemd keeps its own copy of the socket. `systemctl restart` returns when the drain is over (11.0 s here). The journal for the same restart:
   ```console
   $ journalctl --user -u tsvdemo-drain.service --since -30s -o short-precise
   Oct 25 14:21:00.455060 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
   Oct 25 14:21:00.466518 host tsvdemo-drain[2187490]: 2026/10/25 14:21:00 pid 2187490 serving on 127.0.0.1:18490 (inherited=true)
   Oct 25 14:21:04.276149 host tsvdemo-drain[2187490]: 2026/10/25 14:21:04 shutdown: draining for up to 2m0s
   Oct 25 14:21:04.276225 host systemd[1397]: Stopping tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop...
   Oct 25 14:21:14.784301 host tsvdemo-drain[2187490]: 2026/10/25 14:21:14 stream: 15 events sent
   Oct 25 14:21:15.288520 host tsvdemo-drain[2187490]: 2026/10/25 14:21:15 shutdown: all requests finished
   Oct 25 14:21:15.288520 host tsvdemo-drain[2187490]: 2026/10/25 14:21:15 exit
   Oct 25 14:21:15.289782 host systemd[1397]: Stopped tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
   Oct 25 14:21:15.291662 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
   Oct 25 14:21:15.304665 host tsvdemo-drain[2188843]: 2026/10/25 14:21:15 pid 2188843 serving on 127.0.0.1:18490 (inherited=true)
   Oct 25 14:21:15.305812 host tsvdemo-drain[2188843]: 2026/10/25 14:21:15 stream: 1 events sent
   ```

8. **Restart while a request waits on a child process.** Save `child-test.sh`. Request A runs `sleep 10` as a child of the server, then streams 3 events; the restart starts 2 s in. The script also prints the child's file descriptors and how many `LISTEN_*` variables it inherited.
   ```bash
   #!/bin/bash
   # Restart tsvdemo-drain while request A waits on a child process (sleep 10) before it streams 3 events.
   U=http://127.0.0.1:${1:-18490}
   ts() { date +%T.%3N; }
   stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
   { curl -sSN "$U/stream?n=3&wait=10" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
   sleep 2
   main=$(systemctl --user show -p MainPID --value tsvdemo-drain.service)
   child=$(pgrep -P "$main" -x sleep)
   echo "$(ts) child: pid $child, fds: $(ls /proc/$child/fd | tr '\n' ' ')LISTEN_ variables: $(tr '\0' '\n' < /proc/$child/environ | grep -c '^LISTEN_')"
   echo "$(ts) restart begins"
   systemctl --user restart tsvdemo-drain.service
   echo "$(ts) restart returned rc=$?"
   wait
   ```
   ```console
   $ chmod +x child-test.sh
   $ ./child-test.sh
   14:21:17.873 child: pid 2188987, fds: 0 1 2 LISTEN_ variables: 0
   14:21:17.878 restart begins
   14:21:25.841 A data: pid=2188843 event=1/3
   14:21:26.842 A data: pid=2188843 event=2/3
   14:21:27.843 A data: pid=2188843 event=3/3
   14:21:27.845 A http=200
   14:21:27.846 A curl rc=0
   14:21:27.988 restart returned rc=0
   ```
   Expected: the child has only fds 0, 1 and 2 and no `LISTEN_*` variable; it keeps running through the drain, A gets 3 of 3 events, and the restart returns after them (10.1 s here).

   > [!GOTCHA] The default KillMode kills the child the request waits on
   > With `KillMode=control-group`, the SIGTERM of the stop reaches the child too: the request fails with 502 within 6 ms of the restart, and the drain has nothing left to wait for. The same test with a drop-in, then the drop-in removed:

   ```console
   $ mkdir -p "$UNITDIR/tsvdemo-drain.service.d" && printf "[Service]\nKillMode=control-group\n" > "$UNITDIR/tsvdemo-drain.service.d/killmode.conf" && systemctl --user daemon-reload && systemctl --user show -p KillMode tsvdemo-drain.service
   KillMode=control-group
   $ ./child-test.sh
   14:21:30.796 child: pid 2189923, fds: 0 1 2 LISTEN_ variables: 0
   14:21:30.801 restart begins
   14:21:30.807 A child failed: signal: terminated
   14:21:30.808 A http=502
   14:21:30.809 A curl rc=0
   14:21:30.811 restart returned rc=0
   $ rm "$UNITDIR/tsvdemo-drain.service.d/killmode.conf" && rmdir "$UNITDIR/tsvdemo-drain.service.d" && systemctl --user daemon-reload && systemctl --user show -p KillMode tsvdemo-drain.service
   KillMode=mixed
   ```

9. **End a drain early (optional).** A second SIGTERM to the main process makes the server exit at once; a restart then goes on with the new process. The command is `systemctl --user kill --kill-whom=main --signal=SIGTERM tsvdemo-drain.service`. Save `abort-test.sh` to see it during a 60-event stream:
   ```bash
   #!/bin/bash
   # Restart tsvdemo-drain during a 60-event stream, then end the drain early with a second SIGTERM.
   U=http://127.0.0.1:${1:-18490}
   ts() { date +%T.%3N; }
   stamp() { while IFS= read -r l; do [ -n "$l" ] && echo "$(ts) $1 $l"; done; }
   { curl -sSN "$U/stream?n=60" -w 'http=%{http_code}\n' 2>&1; echo "curl rc=$?"; } | stamp A &
   sleep 2.5
   echo "$(ts) restart begins"
   (systemctl --user restart tsvdemo-drain.service; echo "$(ts) restart returned rc=$?") &
   sleep 3
   echo "$(ts) second SIGTERM to the main process"
   systemctl --user kill --kill-whom=main --signal=SIGTERM tsvdemo-drain.service
   wait
   ```
   ```console
   $ chmod +x abort-test.sh
   $ ./abort-test.sh
   14:21:31.564 A data: pid=2190293 event=1/60
   14:21:32.565 A data: pid=2190293 event=2/60
   14:21:33.566 A data: pid=2190293 event=3/60
   14:21:34.059 restart begins
   14:21:34.566 A data: pid=2190293 event=4/60
   14:21:35.566 A data: pid=2190293 event=5/60
   14:21:36.567 A data: pid=2190293 event=6/60
   14:21:37.062 second SIGTERM to the main process
   14:21:37.069 A curl: (18) transfer closed with outstanding read data remaining
   14:21:37.070 A http=200
   14:21:37.071 A curl rc=18
   14:21:37.072 restart returned rc=0
   ```
   Expected: A is cut (`curl: (18)`) 3.0 s into the drain, and the restart returns 10 ms after the second signal. The journal records exit status 1 as `Failed with result 'exit-code'`, then starts the new process on the socket:
   ```console
   $ journalctl --user -u tsvdemo-drain.service --since -15s -o short-precise
   […]
   Oct 25 14:21:34.065330 host systemd[1397]: Stopping tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop...
   Oct 25 14:21:34.065848 host tsvdemo-drain[2190293]: 2026/10/25 14:21:34 shutdown: draining for up to 2m0s
   Oct 25 14:21:37.067622 host systemd[1397]: tsvdemo-drain.service: Sent signal SIGTERM to main process 2190293 (tsvdemo-drain) on client request.
   Oct 25 14:21:37.068169 host tsvdemo-drain[2190293]: 2026/10/25 14:21:37 shutdown: second signal, exiting now
   Oct 25 14:21:37.068962 host systemd[1397]: tsvdemo-drain.service: Main process exited, code=exited, status=1/FAILURE
   Oct 25 14:21:37.069055 host systemd[1397]: tsvdemo-drain.service: Failed with result 'exit-code'.
   Oct 25 14:21:37.069645 host systemd[1397]: Stopped tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
   Oct 25 14:21:37.071006 host systemd[1397]: Started tsvdemo-drain.service - tsvdemo-drain: SSE demo that drains running requests on stop.
   Oct 25 14:21:37.088652 host tsvdemo-drain[2190806]: 2026/10/25 14:21:37 pid 2190806 serving on 127.0.0.1:18490 (inherited=true)
   ```

10. **Apply later changes with `install-units.sh`.** After an edit to the service file only (here its `Description=`), the socket file is unchanged: the script stops the service behind the open socket and then starts both, so a request made meanwhile queues as in step 7:
    ```console
    $ sed -i 's/^Description=tsvdemo-drain:/Description=tsvdemo-drain (edited):/' tsvdemo-drain.service && ./restart-test.sh 18490 ./install-units.sh
    14:21:37.604 A data: pid=2190806 event=1/15
    14:21:38.605 A data: pid=2190806 event=2/15
    14:21:39.605 A data: pid=2190806 event=3/15
    14:21:40.606 A data: pid=2190806 event=4/15
    14:21:41.098 restart begins
    socket file unchanged: restarting the service behind the open socket
    14:21:41.606 A data: pid=2190806 event=5/15
    14:21:42.608 A data: pid=2190806 event=6/15
    14:21:43.101 B starts
    14:21:43.609 A data: pid=2190806 event=7/15
    14:21:44.149 ss LISTEN 1      4096   127.0.0.1:18490 0.0.0.0:*
    14:21:44.610 A data: pid=2190806 event=8/15
    14:21:45.610 A data: pid=2190806 event=9/15
    14:21:46.610 A data: pid=2190806 event=10/15
    14:21:47.611 A data: pid=2190806 event=11/15
    14:21:48.613 A data: pid=2190806 event=12/15
    14:21:49.614 A data: pid=2190806 event=13/15
    14:21:50.614 A data: pid=2190806 event=14/15
    14:21:51.615 A data: pid=2190806 event=15/15
    14:21:51.616 A http=200
    14:21:51.617 A curl rc=0
    Stopping 'tsvdemo-drain.service', but its triggering units are still active:
    tsvdemo-drain.socket
    14:21:51.727 restart returned rc=0
    14:21:51.737 B data: pid=2192112 event=1/1
    14:21:51.738 B http=200 connect=0.000108s total=8.627194s
    14:21:51.739 B curl rc=0
    ```
    Expected: `socket file unchanged: …`, systemd's warning that the socket is still active (that is the point here), B queued and answered by the new PID.

    After an edit to the socket file (here the address, in both files), the script stops socket and service in one call before it starts both:
    ```console
    $ sed -i 's/127.0.0.1:18490/127.0.0.1:18491/' tsvdemo-drain.socket tsvdemo-drain.service && ./install-units.sh
    socket file changed: stopping socket and service, then starting both
    $ ss -ltnp '( sport = :18490 or sport = :18491 )'; journalctl --user -u tsvdemo-drain.service -g serving -n 1 -o cat
    State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess                                    
    LISTEN 0      4096       127.0.0.1:18491      0.0.0.0:*    users:(("tsvdemo-drain",pid=2192375,fd=5))
    2026/10/25 14:21:52 pid 2192375 serving on 127.0.0.1:18491 (inherited=true)
    ```
    Expected: `socket file changed: …`, one `LISTEN` line on 18491, and `inherited=true` on 18491.

    > [!NOTE] A socket change refuses or resets new connections until the drain ends
    > In a separate run of the same move (18490 to 18491) with a stream running, two requests made during the drain, one to each port, both failed at once with `curl: (7) … Could not connect to server`, while the stream on the old port finished 8 of 8. With the address unchanged and another line of the socket file edited, a request made during the drain connected, waited, and got `curl: (56) Recv failure: Connection reset by peer` when the socket closed after the service. A plain restart (steps 7–9) has neither problem.

## Verify

The restart test on the final address passes: A complete, B connected at once and served by the new PID, one connection queued during the drain.

```console
$ ./restart-test.sh 18491
14:21:52.937 A data: pid=2192375 event=1/15
14:21:53.937 A data: pid=2192375 event=2/15
14:21:54.939 A data: pid=2192375 event=3/15
14:21:55.940 A data: pid=2192375 event=4/15
14:21:56.432 restart begins
14:21:56.941 A data: pid=2192375 event=5/15
14:21:57.941 A data: pid=2192375 event=6/15
14:21:58.435 B starts
14:21:58.941 A data: pid=2192375 event=7/15
14:21:59.479 ss LISTEN 1      4096   127.0.0.1:18491 0.0.0.0:*
14:21:59.942 A data: pid=2192375 event=8/15
14:22:00.943 A data: pid=2192375 event=9/15
14:22:01.943 A data: pid=2192375 event=10/15
14:22:02.943 A data: pid=2192375 event=11/15
14:22:03.945 A data: pid=2192375 event=12/15
14:22:04.945 A data: pid=2192375 event=13/15
14:22:05.947 A data: pid=2192375 event=14/15
14:22:06.947 A data: pid=2192375 event=15/15
14:22:06.948 A http=200
14:22:06.949 A curl rc=0
14:22:07.041 restart returned rc=0
14:22:07.060 B data: pid=2193535 event=1/1
14:22:07.061 B http=200 connect=0.000109s total=8.615904s
14:22:07.062 B curl rc=0
```

## Undo / rollback

- **Drop socket activation and keep the service** (the path back for your own service): stop and remove the socket unit, remove the `Wants=`/`After=` lines, reload, restart. The server binds the address itself again:
  ```console
  $ systemctl --user disable --runtime --now tsvdemo-drain.socket && rm "$UNITDIR/tsvdemo-drain.socket" && sed -i "/tsvdemo-drain.socket/d" "$UNITDIR/tsvdemo-drain.service" && systemctl --user daemon-reload && systemctl --user restart tsvdemo-drain.service && sleep 1 && journalctl --user -u tsvdemo-drain.service -g serving -n 1 -o cat
  Removed '/run/user/1000/systemd/user/sockets.target.wants/tsvdemo-drain.socket'.
  2026/10/25 14:22:09 pid 2193892 serving on 127.0.0.1:18491 (inherited=false)
  ```
  What that gives back: a request during a restart is refused (`curl: (7)`), and `ss` prints nothing during the drain because nothing listens:
  ```console
  $ ./restart-test.sh 18491
  14:22:10.190 A data: pid=2193892 event=1/15
  14:22:11.190 A data: pid=2193892 event=2/15
  14:22:12.191 A data: pid=2193892 event=3/15
  14:22:13.193 A data: pid=2193892 event=4/15
  14:22:13.684 restart begins
  14:22:14.194 A data: pid=2193892 event=5/15
  14:22:15.194 A data: pid=2193892 event=6/15
  14:22:15.688 B starts
  14:22:15.697 B curl: (7) Failed to connect to 127.0.0.1 port 18491 after 0 ms: Could not connect to server
  14:22:15.699 B http=000 connect=0.000000s total=0.000153s
  14:22:15.700 B curl rc=7
  14:22:16.195 A data: pid=2193892 event=7/15
  14:22:17.196 A data: pid=2193892 event=8/15
  14:22:18.196 A data: pid=2193892 event=9/15
  14:22:19.197 A data: pid=2193892 event=10/15
  14:22:20.198 A data: pid=2193892 event=11/15
  14:22:21.199 A data: pid=2193892 event=12/15
  14:22:22.201 A data: pid=2193892 event=13/15
  14:22:23.201 A data: pid=2193892 event=14/15
  14:22:24.202 A data: pid=2193892 event=15/15
  14:22:24.203 A http=200
  14:22:24.204 A curl rc=0
  14:22:24.260 restart returned rc=0
  ```
- **Return your own service to its earlier unit file:** copy `~/<APP>.service.orig` back to `~/.config/systemd/user/<APP>.service`, remove its `.socket` unit as in the first bullet, run `systemctl --user daemon-reload` and restart the service. `[UNVERIFIED: not exercised; the demo had no earlier unit file]`
- **Remove the demo completely**, then delete the demo directory:
  ```console
  $ systemctl --user disable --runtime --now tsvdemo-drain.service && rm "$UNITDIR/tsvdemo-drain.service" && systemctl --user daemon-reload && systemctl --user list-units --all "tsvdemo-*"
  Removed '/run/user/1000/systemd/user/default.target.wants/tsvdemo-drain.service'.
    UNIT LOAD ACTIVE SUB DESCRIPTION

  0 loaded units listed.
  To show all installed unit files use 'systemctl list-unit-files'.
  ```
- For a permanent install, the same commands without `--runtime`, with `UNITDIR` = `~/.config/systemd/user`. `[UNVERIFIED: rollback not exercised outside the runtime unit directory]`

## Failure modes

| Symptom (exact message) | Cause | Fix |
|---|---|---|
| `curl: (18) transfer closed with outstanding read data remaining` as soon as a stop begins; the log shows `shutdown: draining for up to 2m0s` and `exit` in the same second | `main` returns when `Serve` returns `ErrServerClosed`, while `Shutdown` still drains in its goroutine | Wait for `Shutdown` before returning (`serveUntil`, step 1) |
| `child failed: signal: terminated`, HTTP 502, as soon as a restart begins | `KillMode=control-group` (the default) sends SIGTERM to every process in the unit | `KillMode=mixed` (step 4) |
| `State 'stop-sigterm' timed out. Aborting.`, `Killing process 1972107 (tsvdemo-drain) with signal SIGABRT.`, a Go `SIGABRT: abort` goroutine dump, `Failed with result 'timeout'` | `TimeoutStopSec=` below the drain (a 5 s drop-in in my run). Fedora 38 and later ship `/usr/lib/systemd/user/service.d/10-timeout-abort.conf` with `TimeoutStopFailureMode=abort`, so the timeout sends SIGABRT before any SIGKILL; systemd.service(5) gives `terminate` as the default | `TimeoutStopSec=` above `-drain` (step 4) |
| `shutdown: drain time over (context deadline exceeded), closing` and `stream: cut after 9/15 events`, then `curl: (18)` | The request ran longer than `-drain` (5 s in my run) | Working as designed; raise `-drain` and `TimeoutStopSec=` together if such requests must finish |
| `Stopping 'tsvdemo-drain.service', but its triggering units are still active:` `tsvdemo-drain.socket`; the next request makes the service `active` again | The socket still listens and starts the service on the next connection | Stop both: `systemctl --user stop tsvdemo-drain.socket tsvdemo-drain.service` |
| `tsvdemo-drain.socket: Unit configuration changed while unit was running, and no socket file descriptors are open. Unit not functional until restarted.`; after a service restart the log says `inherited=false` | `ListenStream=` changed and `daemon-reload` ran while the socket was active (the file was copied by hand, so `install-units.sh` then sees no change); `is-active` still says `active` | `systemctl --user stop tsvdemo-drain.socket tsvdemo-drain.service`, then `systemctl --user start tsvdemo-drain.socket tsvdemo-drain.service` |
| `Job failed. See "journalctl -xe" for details.` from `systemctl --user restart tsvdemo-drain.socket`; journal: `Socket service tsvdemo-drain.service already active, refusing.` | The socket unit cannot start while its service runs | Same as the row above |
| `systemd passed "2" sockets, want 1 (one ListenStream= line)`, the service exits 1 and `is-active` says `activating` (waiting for `Restart=`) | Two `ListenStream=` lines (in my run the second came from a drop-in) | One `ListenStream=` in the socket unit and its drop-ins |
| `Failed to parse address value in ':18490', ignoring: Invalid argument` and `Unit has no Listen setting (ListenStream=, ListenDatagram=, ListenFIFO=, ...). Refusing.` (the same for `localhost:18490`) | systemd reads an IP literal with a port, a bare port and other forms listed in systemd.socket(5), not Go's `:port` or a host name | `ListenStream=127.0.0.1:18490`; `install-units.sh` runs `systemd-analyze --user verify` before it stops anything |
| With `Requires=`: `A dependency job for tsvdemo-drain.service failed. See 'journalctl -xe' for details.`; journal: `Failed to create listening socket (127.0.0.1:18490): Address already in use`, `Dependency failed for tsvdemo-drain.service`; still down after the port is free | `Requires=` fails the service when the socket fails; no service process ran, so `Restart=` never fires | `Wants=` (step 4): the server's own bind fails (`listen tcp 127.0.0.1:18490: bind: address already in use`), `Restart=` retries every 5 s, and the first retry after the port is free gets the socket |
| `curl: (7) Failed to connect to 127.0.0.1 port 18491 after 0 ms: Could not connect to server` during a restart | No socket unit: the old process closes its listener when the drain starts, and the new one binds after the drain | The socket unit (steps 3–6) |

## Field notes

- **2026-10-25 13:38–14:22, GEEKOM AE8, Fedora Linux 44, kernel 7.2.10, systemd 259 (259.9-1.fc44), Go 1.26.0, curl 8.18.0, iproute2 6.17.0:** the demo above. Units in `$XDG_RUNTIME_DIR/systemd/user`, enabled with `--runtime`; the demo directory sat in my writing workspace, its files saved with mode 644; ports 127.0.0.1:18490–18493. Every block above comes from one run in step order (14:20:56–14:22:25), and so do the 502 and `curl: (7)` rows of Failure modes; the other rows and the note in step 10 come from two more runs (13:41–13:42 and 13:46–13:47).
  - I ran the step sequence four times. In the first pass `curl -s` hid curl's error texts, and the Undo test started its stream right after `systemctl restart` had returned: the self-binding server was not listening yet and the stream got `curl rc=7`. A server that binds by itself refuses connections right after its start too, not only during a drain. The second pass still had a unit comment that promised SIGKILL at the stop timeout (wrong on Fedora, see below), and the unchanged-socket branch of `install-units.sh` had never run; the third pass fixed both. The first three copied the files with their executable bits, so a `chmod` that named the test scripts before they existed went unnoticed; the fourth saved everything with mode 644, as from this page, and ran the `chmod` lines of steps 5 and 7–9.
  - My first port holder for the `Wants=`/`Requires=` cases, a Python socket without `SO_REUSEADDR`, failed to bind 18490 at all (`Address already in use`), most likely because connections from earlier tests were in TIME_WAIT on that port (I did not capture them). With `SO_REUSEADDR` set it held the port and the socket unit failed as intended.
  - The stop timeout case ended with SIGABRT, not the SIGKILL systemd.service(5) describes for the default `TimeoutStopFailureMode=terminate`. The `abort` comes from Fedora's drop-in `/usr/lib/systemd/user/service.d/10-timeout-abort.conf` (package systemd-259.9-1.fc44), part of the Fedora 38 "Shorter Shutdown Timer" change, which also set the 45 s default stop timeout. The Go runtime answered SIGABRT with a 221-line goroutine dump in the journal and exit status 2.
  - `ss -ltnp` never listed the systemd user manager as a holder of the socket, only the server process: as my user, reading the links in `/proc/<manager PID>/fd/` fails with `Permission denied` (154 of 154), so `ss` cannot attribute the manager's copy. During a drain the port showed as `LISTEN` with a queued connection and no process.
  - After a `ListenStream=` change copied in by hand plus `daemon-reload`, `systemctl is-active` still said `active` for a socket without file descriptors. Only the server's `inherited=false` log line showed it.
  - My first `install-units.sh` handled an unchanged socket file with `start` of the socket and then `restart` of the service. In the state from the previous point, `start` of the socket failed with "already active, refusing" and `set -e` ended the script. The published version stops the service first, as the Makefile of the service this comes from does.
  - `go mod init` with Go 1.26.0 wrote `go 1.25.0` into go.mod.
  - At 14:28 the same day a system upgrade installed systemd 259.10-1.fc44, after all of the runs above; nothing here was re-run on 259.10.
  - The job lines in step 2's Expected come from a separate paste of that line into an interactive bash on a pseudo-terminal driven by a script (14:12); everything else in step 2 is from the step-order run.
- **2026-10-25 00:45–02:03, same box, systemd 259 (259.9-1.fc44, installed 2026-09-21), a Go LLM routing proxy (the service this manual comes from; the Go version is not in my notes of these runs), throwaway units on 127.0.0.1:18471:** five rounds of tests of the change that introduced the socket unit and the drain. The default KillMode sent SIGTERM to the ssh call of an engine switch when a stop began, and the client got an error after 4 s; with `KillMode=mixed` the switch finished and the client got its full stream (62 s). A `ListenStream=` change with only a `daemon-reload` left the socket without file descriptors, and the proxy silently bound the port itself. A stopped service came back on the next client poll through the socket. `Requires=` turned a socket bind failure into a service that stayed down; `Wants=` let `Restart=` recover. The previous release's 5 s shutdown grace had never run, because `main` returned as soon as `ListenAndServe` returned `ErrServerClosed`.

## Notes

- The story of how the proxy got here: [Shutdown grace that never ran](https://tsvinc.dev/posts/shutdown-grace-never-ran/). For llama-server units, KillMode and a SIGTERM to the main process only: [Terminating immediately took 90 seconds](https://tsvinc.dev/posts/terminating-immediately-took-90-seconds/).
- Requests that arrive during a restart wait for the whole drain (B waited 9.0 s in step 7). A client whose timeout is shorter than the drain gives up. How many connections can wait is the socket's `Backlog=`; systemd.socket(5) gives its default as 4294967295, capped by `net.core.somaxconn` (4096 here).
- net/http's `Shutdown` "does not attempt to close nor wait for hijacked connections such as WebSockets", so `serveUntil` does not drain them. Not tested here.
- systemd 259 also sets `LISTEN_PIDFDID`, and sd_listen_fds(3) checks it against the process's pidfd; `listenerFrom` checks only `LISTEN_PID`.
- fd 3 arrives without close-on-exec: a probe under `systemd-socket-activate`, which passes sockets the same way, read `FD_CLOEXEC` unset. `net.FileListener` works on a close-on-exec duplicate (`poll.DupCloseOnExec` in net/file_unix.go), so closing fd 3 keeps the socket out of children (step 8: fds 0, 1, 2 only).
- `github.com/coreos/go-systemd/v22/activation` (v22.7.0) does the descriptor part; at v22.7.0 its `Files(true)` unsets `LISTEN_PID`, `LISTEN_FDS` and `LISTEN_FDNAMES` but not `LISTEN_PIDFDID`. I read its source and did not use it.
- For a socket on a specific LAN address, systemd.socket(5) recommends `FreeBind=true` ("for robustness reasons"), so the socket binds before the address is configured. Not tested here.

## Sources

- [net/http server.go at go1.26.0](https://raw.githubusercontent.com/golang/go/go1.26.0/src/net/http/server.go) — `Server.Shutdown` ("immediately return ErrServerClosed. Make sure the program doesn't exit and waits instead for Shutdown to return"), `Serve`, `Close`, hijacked connections. Same text on release-branch.go1.26.
- [net/file.go at go1.26.0](https://raw.githubusercontent.com/golang/go/go1.26.0/src/net/file.go) — `FileListener` returns a copy; closing one does not affect the other.
- [net/file_unix.go at go1.26.0](https://raw.githubusercontent.com/golang/go/go1.26.0/src/net/file_unix.go) — the copy is made with `poll.DupCloseOnExec`.
- [os/exec exec.go, release-branch.go1.26](https://raw.githubusercontent.com/golang/go/release-branch.go1.26/src/os/exec/exec.go) — `CommandContext` kills the process when the context is done.
- [systemd.kill(5), systemd 259](https://www.freedesktop.org/software/systemd/man/259/systemd.kill.html) — `KillMode=`: control-group (default) vs mixed.
- [systemd.socket(5), systemd 259](https://www.freedesktop.org/software/systemd/man/259/systemd.socket.html) — `ListenStream=` address forms, `Accept=no`, `Backlog=`, `FreeBind=`, no implicit dependency from socket to service.
- [sd_listen_fds(3), systemd 259](https://www.freedesktop.org/software/systemd/man/259/sd_listen_fds.html) — fd 3, `LISTEN_PID`/`LISTEN_FDS`, `LISTEN_PIDFDID` since 259, the variables to unset.
- [systemd.service(5), systemd 259](https://www.freedesktop.org/software/systemd/man/259/systemd.service.html) — `TimeoutStopSec=`, `TimeoutStopFailureMode=`, `Restart=`.
- [systemd.unit(5), systemd 259](https://www.freedesktop.org/software/systemd/man/259/systemd.unit.html) — `Wants=` vs `Requires=`.
- [systemd-system.conf(5), systemd 259](https://www.freedesktop.org/software/systemd/man/259/systemd-system.conf.html) — `DefaultTimeoutStopSec=` (90 s in the man page).
- [Fedora change: Shorter Shutdown Timer](https://fedoraproject.org/wiki/Changes/Shorter_Shutdown_Timer) — Fedora 38: unit stop timeout from 2 minutes to 45 seconds, SIGABRT for a core dump before SIGKILL; the source of `10-timeout-abort.conf`.
- [go-systemd activation/files_unix.go at v22.7.0](https://raw.githubusercontent.com/coreos/go-systemd/v22.7.0/activation/files_unix.go) — `Files(unsetEnv)`.