feat(deploy): add HAProxy blue/green stack for zero-downtime relay updates

.changeset/haproxy-blue-green.md

@@ -0,0 +1,7 @@
+---
+"nostream": minor
+---
+
+feat(deploy): add HAProxy blue/green compose stack
+
+Two relays behind HAProxy with `/readyz` health checks and `option redispatch`, plus a rolling recreate script that replaces one relay at a time for zero-downtime image updates. Optional Redis stream fan-out (`RELAY_BROADCAST_FANOUT`) lets both relays share live WebSocket broadcasts while workers keep using cluster `process.send`.

CONFIGURATION.md

@@ -54,6 +54,10 @@ The following environment variables can be set:
 | REDIS_PORT                       | Redis Port                       | 6379                   |
 | REDIS_USER                       | Redis User                       | default                |
 | REDIS_PASSWORD                   | Redis Password                   | nostr_ts_relay         |
+| RELAY_BROADCAST_FANOUT           | Publish accepted events to a Redis stream so multiple relay containers share live fan-out (`true`/`false`) | `false` |
+| RELAY_BROADCAST_STREAM_KEY       | Redis stream key used when `RELAY_BROADCAST_FANOUT` is enabled | `nostream:relay:broadcast` |
+| RELAY_BROADCAST_STREAM_MAXLEN    | Approximate max entries retained on the broadcast Redis stream (`XADD` trim). Dominant Redis memory use when fan-out is enabled. | `50000` |
+| RELAY_INSTANCE_ID                | Optional stable id for this relay instance (defaults to hostname and pid) | |
 | PROMETHEUS_URL                   | Prometheus base URL for admin metrics queries | http://127.0.0.1:9090 |
 | PROMETHEUS_QUERY_TIMEOUT_MS      | Timeout for each Prometheus admin metrics query (ms) | 5000 |
 | ADMIN_METRICS_SSE_INTERVAL_MS    | Interval between `/admin/metrics` SSE snapshots (ms) | 5000 |

deploy/README.md

@@ -156,3 +156,67 @@ the new checkout (or copy the updated files). Automated sync is planned separate
 ```
 
 Existing `.env` and `.nostr/settings.yaml` are preserved.
+
+## Zero-downtime updates (HAProxy blue/green)
+
+`deploy/docker-compose.haproxy.yml` replaces the single-relay stack with two
+relays (`nostream-blue`, `nostream-green`) behind HAProxy on `127.0.0.1:8008`.
+Postgres, Redis, and migrations are unchanged.
+
+The HAProxy compose file **always sets `RELAY_BROADCAST_FANOUT=true` on relay
+services** (even if bootstrap `.env` leaves it `false` for the single-relay
+stack). Both relays publish accepted events to a shared Redis stream; each
+cluster primary subscribes and fans out to its workers so live WebSocket clients
+stay in sync when HAProxy balances across blue and green.
+
+Do **not** run the single-relay `docker-compose.yml` stack and the HAProxy stack
+at the same time: both bind `127.0.0.1:8008`. Stop the old stack before starting
+blue/green:
+
+```bash
+cd /opt/nostream
+docker compose down   # single-relay stack, if it was running
+```
+
+Install alongside `.env` and `postgresql.conf`, then start:
+
+```bash
+cp deploy/docker-compose.haproxy.yml deploy/rolling-relay-recreate.sh /opt/nostream/
+cp -r deploy/haproxy /opt/nostream/
+chmod +x /opt/nostream/rolling-relay-recreate.sh
+cd /opt/nostream
+docker compose -f docker-compose.haproxy.yml up -d
+curl -s http://127.0.0.1:8008/readyz
+```
+
+HAProxy sets `X-Forwarded-For` (appended as the rightmost hop). In
+`.nostr/settings.yaml` (or your settings overrides), trust the HAProxy address
+from `docker-compose.haproxy.yml` (`172.28.0.2` on subnet `172.28.0.0/24`):
+
+```yaml
+network:
+  remoteIpHeader: x-forwarded-for
+  trustedProxies:
+    - "172.28.0.2"
+    - "127.0.0.1"
+    - "::ffff:127.0.0.1"
+    - "::1"
+```
+
+With a trusted proxy, the relay uses the **last** `X-Forwarded-For` hop (what
+HAProxy appended), not the leftmost value clients may supply.
+
+To update, load the new image, run migrations, then replace relays one at a time:
+
+```bash
+cd /opt/nostream
+docker compose -f docker-compose.haproxy.yml run --rm nostream-migrate
+./rolling-relay-recreate.sh
+```
+
+The script requires the peer relay to be running and `/readyz` healthy before it
+stops either backend. It waits for each replacement to become healthy before
+moving to the second relay. HAProxy health-checks `/readyz` every 2s and retries
+failed requests on the other backend (`option redispatch`). Set
+`STOP_GRACE_PERIOD` (default `45s`) above `WS_DRAIN_TIMEOUT_MS` (default 30s) so
+WebSocket drain finishes before Docker sends SIGKILL.

deploy/docker-compose.haproxy.yml

@@ -0,0 +1,146 @@
+# Blue/green alternative to docker-compose.prod.yml: two relays behind HAProxy.
+# Keep the shared services below in sync with docker-compose.prod.yml.
+#
+#   docker compose -f docker-compose.haproxy.yml up -d
+
+networks:
+  nostream_haproxy:
+    ipam:
+      config:
+        - subnet: 172.28.0.0/24
+
+x-nostream-relay: &nostream-relay
+  image: ghcr.io/cameri/nostream:main
+  pull_policy: never
+  env_file: .env
+  environment:
+    RELAY_PORT: 8008
+    NOSTR_CONFIG_DIR: /home/node/.nostr
+    DB_HOST: nostream-db
+    DB_PORT: 5432
+    DB_USER: ${DB_USER}
+    DB_PASSWORD: ${DB_PASSWORD}
+    DB_NAME: ${DB_NAME}
+    DB_MIN_POOL_SIZE: ${DB_MIN_POOL_SIZE:-16}
+    DB_MAX_POOL_SIZE: ${DB_MAX_POOL_SIZE:-64}
+    DB_ACQUIRE_CONNECTION_TIMEOUT: ${DB_ACQUIRE_CONNECTION_TIMEOUT:-60000}
+    REDIS_HOST: nostream-cache
+    REDIS_PORT: 6379
+    REDIS_USER: default
+    REDIS_PASSWORD: ${REDIS_PASSWORD}
+    READ_REPLICA_ENABLED: 'false'
+    # Required for blue/green; do not inherit RELAY_BROADCAST_FANOUT=false from bootstrap .env
+    RELAY_BROADCAST_FANOUT: 'true'
+    RELAY_BROADCAST_STREAM_KEY: ${RELAY_BROADCAST_STREAM_KEY:-nostream:relay:broadcast}
+    WORKER_COUNT: ${WORKER_COUNT:-2}
+    WS_DRAIN_TIMEOUT_MS: ${WS_DRAIN_TIMEOUT_MS:-30000}
+  user: node:node
+  volumes:
+    - ${PWD}/.nostr:/home/node/.nostr
+  depends_on:
+    nostream-cache:
+      condition: service_healthy
+    nostream-db:
+      condition: service_healthy
+    nostream-migrate:
+      condition: service_completed_successfully
+  restart: on-failure
+  stop_grace_period: ${STOP_GRACE_PERIOD:-45s}
+  networks:
+    - nostream_haproxy
+  # Gates `up --wait` during rolling recreate, so the next relay is only
+  # replaced once this one serves traffic.
+  healthcheck:
+    test:
+      [
+        'CMD-SHELL',
+        "node -e \"fetch('http://127.0.0.1:8008/readyz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))\"",
+      ]
+    interval: 5s
+    timeout: 5s
+    retries: 6
+    start_period: 60s
+
+services:
+  haproxy:
+    image: haproxy:3.0-alpine
+    container_name: nostream-haproxy
+    volumes:
+      - ./haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
+    ports:
+      - 127.0.0.1:8008:8008
+    depends_on:
+      - nostream-blue
+      - nostream-green
+    restart: on-failure
+    networks:
+      nostream_haproxy:
+        ipv4_address: 172.28.0.2
+
+  nostream-blue:
+    <<: *nostream-relay
+    container_name: nostream-blue
+
+  nostream-green:
+    <<: *nostream-relay
+    container_name: nostream-green
+
+  nostream-db:
+    image: postgres:15
+    container_name: nostream-db
+    networks:
+      - nostream_haproxy
+    environment:
+      POSTGRES_DB: ${DB_NAME}
+      POSTGRES_USER: ${DB_USER}
+      POSTGRES_PASSWORD: ${DB_PASSWORD}
+    volumes:
+      - ${PWD}/.nostr/data:/var/lib/postgresql/data
+      - ${PWD}/.nostr/db-logs:/var/log/postgresql
+      - ${PWD}/postgresql.conf:/postgresql.conf
+    command: postgres -c 'config_file=/postgresql.conf'
+    restart: always
+    healthcheck:
+      test: ['CMD-SHELL', 'pg_isready -U ${DB_USER}']
+      interval: 5s
+      timeout: 5s
+      retries: 5
+      start_period: 360s
+
+  nostream-cache:
+    image: redis:7.0.5-alpine3.16
+    container_name: nostream-cache
+    networks:
+      - nostream_haproxy
+    environment:
+      REDIS_PASSWORD: ${REDIS_PASSWORD}
+    volumes:
+      - cache:/data
+    command: sh -c 'redis-server --loglevel warning --requirepass "$$REDIS_PASSWORD"'
+    restart: always
+    healthcheck:
+      test: ['CMD-SHELL', 'redis-cli -a "$$REDIS_PASSWORD" ping | grep PONG']
+      interval: 2s
+      timeout: 5s
+      retries: 10
+
+  nostream-migrate:
+    image: ghcr.io/cameri/nostream:main
+    pull_policy: never
+    container_name: nostream-migrate
+    networks:
+      - nostream_haproxy
+    user: node:node
+    command: ['node_modules/.bin/knex', 'migrate:latest']
+    environment:
+      DB_HOST: nostream-db
+      DB_PORT: 5432
+      DB_USER: ${DB_USER}
+      DB_PASSWORD: ${DB_PASSWORD}
+      DB_NAME: ${DB_NAME}
+    depends_on:
+      nostream-db:
+        condition: service_healthy
+
+volumes:
+  cache:

deploy/env.example

@@ -21,3 +21,7 @@ DB_MIN_POOL_SIZE=16
 DB_MAX_POOL_SIZE=64
 DB_ACQUIRE_CONNECTION_TIMEOUT=60000
 WORKER_COUNT=2
+
+# Single-relay stack only. HAProxy compose (docker-compose.haproxy.yml) always enables fan-out on relay services.
+RELAY_BROADCAST_FANOUT=false
+RELAY_BROADCAST_STREAM_KEY=nostream:relay:broadcast

deploy/haproxy/haproxy.cfg

@@ -0,0 +1,40 @@
+global
+    log stdout format raw local0 info
+    maxconn 4096
+
+defaults
+    mode http
+    log global
+    option httplog
+    timeout connect 5s
+    timeout client 30s
+    timeout server 30s
+    timeout tunnel 1h
+    timeout check 5s
+    retries 3
+    option redispatch
+    retry-on conn-failure empty-response response-timeout 502 503 504
+
+# Docker's embedded DNS, so recreated relay containers are picked up by IP change.
+resolvers docker
+    nameserver dns 127.0.0.11:53
+    resolve_retries 3
+    timeout resolve 1s
+    timeout retry 1s
+    hold valid 2s
+
+frontend nostream_in
+    bind *:8008
+    option forwardfor
+    default_backend nostream_relays
+
+backend nostream_relays
+    balance roundrobin
+    option httpchk
+    http-check send meth GET uri /readyz
+    http-check expect status 200
+
+    default-server check inter 2s fall 2 rise 1 resolvers docker resolve-prefer ipv4 init-addr libc,none
+
+    server blue nostream-blue:8008
+    server green nostream-green:8008

deploy/rolling-relay-recreate.sh

@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Rolling relay recreate for the HAProxy blue/green stack. Replaces one relay
+# at a time so the other keeps serving traffic.
+#
+# Usage:
+#   ./rolling-relay-recreate.sh [/opt/nostream]
+#
+# Load the new image and run migrations before calling this.
+
+TARGET="${1:-/opt/nostream}"
+COMPOSE_FILE="${COMPOSE_FILE:-docker-compose.haproxy.yml}"
+RELAY_PORT="${RELAY_PORT:-8008}"
+
+cd "$TARGET"
+
+if [[ ! -f "$COMPOSE_FILE" ]]; then
+  echo "error: compose file not found: $TARGET/$COMPOSE_FILE" >&2
+  exit 1
+fi
+
+compose() {
+  docker compose -f "$COMPOSE_FILE" "$@"
+}
+
+relay_readyz_ok() {
+  local service=$1
+  local cid
+  cid="$(compose ps -q "$service" 2>/dev/null || true)"
+  if [[ -z "$cid" ]]; then
+    return 1
+  fi
+  docker exec "$cid" node -e \
+    "fetch('http://127.0.0.1:${RELAY_PORT}/readyz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"
+}
+
+peer_for() {
+  case "$1" in
+    nostream-blue) echo nostream-green ;;
+    nostream-green) echo nostream-blue ;;
+    *) echo "error: unknown service $1" >&2; exit 1 ;;
+  esac
+}
+
+require_healthy_peer() {
+  local peer=$1
+  if [[ -z "$(compose ps -q "$peer")" ]]; then
+    echo "error: peer $peer is not running; start it before replacing the other relay" >&2
+    exit 1
+  fi
+  if ! relay_readyz_ok "$peer"; then
+    echo "error: peer $peer /readyz is not healthy; fix it before continuing" >&2
+    exit 1
+  fi
+}
+
+for service in nostream-blue nostream-green; do
+  if [[ -z "$(compose ps -q "$service")" ]]; then
+    echo "Starting $service (not running)..."
+    compose rm -f "$service" >/dev/null 2>&1 || true
+    compose up -d --no-deps --wait "$service"
+    continue
+  fi
+
+  require_healthy_peer "$(peer_for "$service")"
+
+  echo "Replacing $service..."
+  compose stop "$service"
+  compose rm -f "$service"
+  compose up -d --no-deps --wait "$service"
+done
+
+echo "Rolling recreate complete"