add wal recovery guide

.gitignore

@@ -0,0 +1,15 @@
+.secrets
+.env
+
+
+
+webapp/nginx-test
+webapp/dad-domains
+webapp/qbittorrent
+webapp/openwebui
+webapp/wisemapping
+remote/rustdesk-server
+remote/ttyd-ubuntu
+mgmt/grafana
+mgmt/prometheus
+local/Y2JB-WebUI

AGENTS.md

@@ -1,5 +1,23 @@
 # Agent Guidelines for Docker Infrastructure Monorepo
 
+## CRITICAL: Forbidden Files
+
+**NEVER read, access, or attempt to open the following files under ANY circumstances:**
+
+- `.env` and ALL variants (`.env.local`, `.env.development`, `.env.production`, `.env.*`)
+- `.secrets/` directories at any level
+- Any file in a `secrets/` directory at any level
+- Credential files: `credentials.json`, `credentials.yml`, `credentials.yaml`
+- Key files: `*.pem`, `*.key`, `*.p12`, `*.pfx`, `id_rsa`, `id_ecdsa`, `id_ed25519`
+- Files named `token`, `tokens`, `.token`, `.tokens`, `password`, `passwords`
+
+**What to do instead:**
+- Use `.env.example` files to understand required environment variables
+- Ask the user to provide values explicitly if needed
+- Use placeholder values when demonstrating code
+
+This is a hard rule. No exceptions. No "just checking". If you need env info, read the `.env.example` file or ask the user.
+
 ## Repository Structure
 
 This is a Docker Compose monorepo for personal infrastructure hosting. The main focus is Docker services, with local tools in `local/`.
@@ -115,8 +133,7 @@ For full OpenClaw guidelines, see `local/openclaw/AGENTS.md`.
 
 ## Security Best Practices
 
-- **NEVER read, access, or attempt to open `.env` files or `.secrets/` directories** - These contain sensitive data
+- See **CRITICAL: Forbidden Files** section above — this is the highest priority rule in this repo
-- `.env` and `.secrets/` directories are gitignored everywhere
 - Never commit credentials, API keys, or certificates
 - Use Cloudflare Tunnel for external access (no port forwarding)
 - Placeholders in documentation (e.g., `user@example.com`)

backend/postgres/.gitignore

@@ -0,0 +1,3 @@
+.env
+.secrets/
+init-data.sh

backend/postgres/compose.yml

@@ -9,7 +9,7 @@ services:
       start_period: 20s
       test:
       - CMD-SHELL
-      - pg_isready -U $${POSTGRES_USER_FILE}
+      - pg_isready -U "$$(cat /run/secrets/POSTGRES_USER)"
       timeout: 5s
     env_file:
       - .env
@@ -19,16 +19,16 @@ services:
     volumes:
       - pgdata:/var/lib/postgresql
     networks:
-      - db
+      - backend
 volumes:
   pgdata:
     name: pgdata
 secrets:
   POSTGRES_USER:
-    file: ./.secrets/POSTGRES_USER
+    file: .secrets/POSTGRES_USER
   POSTGRES_PASS:
-    file: ./.secrets/POSTGRES_PASS
+    file: .secrets/POSTGRES_PASS
 networks:
-  db:
+  backend:
     external:
       true

backend/postgres/guides/WAL_RECOVERY.md

@@ -0,0 +1,107 @@
+# PostgreSQL WAL Corruption Recovery Guide
+
+## Symptoms
+
+PostgreSQL container crashes on startup with logs showing:
+
+```
+LOG:  unexpected pageaddr X/Y in WAL segment ...
+LOG:  invalid checkpoint record
+PANIC:  could not locate a valid checkpoint record at ...
+LOG:  startup process (PID N) was terminated by signal 6: Aborted
+LOG:  aborting startup due to startup process failure
+```
+
+The container restarts repeatedly, each time hitting the same error.
+
+## Cause
+
+The Write-Ahead Log (WAL) was corrupted by an unclean shutdown (power loss, host crash, force kill, etc.). PostgreSQL cannot find a valid checkpoint to resume from.
+
+## What You Risk Losing
+
+- **Committed data**: Safe. It is already written to the data files.
+- **Uncommitted transactions** from the moment of the crash: Lost. These were only in WAL.
+- **Recent changes** that were committed but not yet checkpointed: Usually safe, but there is a small risk of inconsistency.
+
+## Recovery Procedure
+
+### 1. Stop the Crashing Container
+
+```bash
+cd /path/to/postgres/service
+docker compose down
+```
+
+### 2. Run `pg_resetwal`
+
+This resets the WAL and forces a clean start.
+
+**If your data is in a named Docker volume (e.g., `pgdata`):**
+
+```bash
+docker run --rm \
+  -v pgdata:/var/lib/postgresql \
+  --user postgres \
+  postgres:18 \
+  pg_resetwal -f /var/lib/postgresql/18/docker
+```
+
+> Adjust the path `/var/lib/postgresql/18/docker` to match your `PGDATA` setting.
+
+**If your data is in a bind mount (e.g., `./data`):**
+
+```bash
+docker run --rm \
+  -v $(pwd)/data:/var/lib/postgresql/data \
+  --user postgres \
+  postgres:18 \
+  pg_resetwal -f /var/lib/postgresql/data
+```
+
+### 3. Start the Database
+
+```bash
+docker compose up -d
+```
+
+### 4. Verify
+
+```bash
+docker compose logs --tail=20
+docker inspect --format='{{.State.Health.Status}}' postgres
+```
+
+You should see:
+
+```
+LOG:  database system is ready to accept connections
+```
+
+And the health status should be `healthy`.
+
+## Prevention
+
+- Ensure graceful shutdowns: `docker compose down` instead of `docker kill`
+- Use a UPS if running on bare metal to avoid power-loss crashes
+- Keep backups of important data volumes
+- Consider setting `restart: unless-stopped` instead of `always` to prevent rapid crash loops
+
+## When NOT to Use This Fix
+
+Do **not** use `pg_resetwal` if:
+- You have a recent base backup and WAL archive — restore from backup instead
+- You suspect data file corruption (not just WAL corruption)
+- You can recover by other means (e.g., starting from a replication standby)
+
+If unsure, copy the data directory somewhere safe before running `pg_resetwal`.
+
+## One-Liner for Future Emergencies
+
+If you're sure it's WAL corruption and you know your setup:
+
+```bash
+docker compose down && \
+docker run --rm -v pgdata:/var/lib/postgresql --user postgres postgres:18 pg_resetwal -f /var/lib/postgresql/18/docker && \
+docker compose up -d
+```

backend/redis/compose.yml

@@ -15,12 +15,12 @@ services:
     env_file:
       - .env
     networks:
-      - db
+      - backend
     volumes:
     - redis_data:/data
 volumes:
   redis_data:
     name: redis_data
 networks:
-  db:
+  backend:
     external: true

backend/surrealdb/.env.example

@@ -0,0 +1,4 @@
+# SurrealDB Configuration
+# Root password is loaded from ./.secrets/SURREAL_ROOT_PASSWORD via Docker secrets
+# Timezone settings
+TZ=

backend/surrealdb/compose.yml

@@ -0,0 +1,28 @@
+services:
+  surrealdb:
+    image: surrealdb/surrealdb:v2
+    container_name: surrealdb
+    # entrypoint: ["/bin/sh"]
+    command: start --user root --pass $(cat /run/secrets/SURREAL_ROOT_PASSWORD) rocksdb:/mydata/mydatabase.db
+    user: root
+    restart: always
+    env_file:
+      - .env
+    secrets:
+      - SURREAL_ROOT_PASSWORD
+    volumes:
+      - surreal_data:/mydata
+    networks:
+      - db
+
+volumes:
+  surreal_data:
+    name: surreal_data
+
+secrets:
+  SURREAL_ROOT_PASSWORD:
+    file: ./.secrets/SURREAL_ROOT_PASSWORD
+
+networks:
+  db:
+    external: true

frontend/cloudflared/.gitignore

@@ -0,0 +1,3 @@
+.env
+tunnel.json
+config.yml

frontend/cloudflared/compose.yml

@@ -8,6 +8,8 @@ services:
       - traefik.enable=true
     env_file:
       - .env
+    expose:
+      - "7844"
     networks:
       - frontend
 networks:

frontend/traefik/.env.example

@@ -3,6 +3,8 @@ SUBDOMAIN=
 # TRAEFIK_USER=
 SSL_EMAIL_FILE=/run/secrets/CF_API_EMAIL
 CF_API_EMAIL_FILE=/run/secrets/CF_API_EMAIL
-CF_API_KEY_FILE=/run/secrets/CF_API_KEY
+ACME_EMAIL=
+CF_DNS_API_TOKEN_FILE=/run/secrets/CF_DNS_API_TOKEN
 SSH_PORT=
-TZ=
+TZ=
+TAG=

frontend/traefik/.gitignore

@@ -0,0 +1,2 @@
+.env
+traefik_data/

frontend/traefik/compose.yml

@@ -25,7 +25,7 @@ services:
       - "--entryPoints.websecure.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22"
       - "--certificatesresolvers.cloudflare.acme.dnschallenge=true"
       - "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare"
-      - "--certificatesresolvers.cloudflare.acme.email=${CF_API_EMAIL_FILE}"
+      - "--certificatesresolvers.cloudflare.acme.email=${ACME_EMAIL}"
       - "--certificatesresolvers.cloudflare.acme.storage=/letsencrypt/acme.json"
     labels:
       - traefik.enable=true
@@ -48,11 +48,12 @@ services:
       - traefik.http.middlewares.traefik_dashboard.headers.STSIncludeSubdomains=true
       - traefik.http.middlewares.traefik_dashboard.headers.STSPreload=true
       - traefik.http.middlewares.traefik_dashboard.headers.frameDeny=true
+      - traefik.docker.network=mgmt
     env_file:
       - .env
     secrets:
-      - CF_API_KEY
       - CF_API_EMAIL
+      - CF_DNS_API_TOKEN
     volumes:
       - ./traefik_data:/letsencrypt
       - /var/run/docker.sock:/var/run/docker.sock:ro
@@ -64,11 +65,12 @@ services:
       - webapp
       - mgmt
       - jump
+      - mcp
 secrets:
-  CF_API_KEY:
-    file: .secrets/CF_API_KEY
   CF_API_EMAIL:
     file: .secrets/CF_API_EMAIL
+  CF_DNS_API_TOKEN:
+    file: .secrets/CF_DNS_API_TOKEN
 networks:
   frontend:
     external:
@@ -82,3 +84,6 @@ networks:
   jump:
     external:
       true
+  mcp:
+    external:
+      true

local/adguard/.gitignore

@@ -0,0 +1,3 @@
+.env
+conf/
+work/

mgmt/adminer/.gitignore

@@ -0,0 +1 @@
+.env

mgmt/adminer/compose.yml

@@ -25,11 +25,11 @@ services:
       - .env
     networks:
       - mgmt
-      - db
+      - backend
 networks:
   mgmt:
     external:
       true
-  db:
+  backend:
     external:
       true

mgmt/authentik/.gitignore

@@ -0,0 +1,3 @@
+certs/
+media/
+custom-templates/

mgmt/authentik/compose.yml

@@ -13,7 +13,7 @@ services:
       - ./custom-templates:/templates
     networks:
       - mgmt
-      - db
+      - backend
     labels:
       - traefik.enable=true
       - traefik.http.routers.$SUBDOMAIN.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
@@ -55,7 +55,7 @@ services:
       - DB_PASS
     user: root
     networks:
-      - db
+      - backend
     volumes:
       # - /var/run/docker.sock:/var/run/docker.sock # Optional, only if using external outposts
       - ./media:/media
@@ -64,7 +64,7 @@ services:
 networks:
   mgmt:
     external: true
-  db:
+  backend:
     external: true
 secrets:
   SECRET_KEY:

mgmt/gitea/compose.yml

@@ -38,14 +38,14 @@ services:
       - /etc/localtime:/etc/localtime:ro
     networks:
       - mgmt
-      - db
+      - backend
 volumes:
   gitea-data:
     name: gitea-data
 networks:
   mgmt:
     external: true
-  db:
+  backend:
     external: true
 secrets:
   DB_PASS:

mgmt/portainer/.gitignore

@@ -0,0 +1,2 @@
+data/
+.env

mgmt/portainer/compose.yml

@@ -6,7 +6,7 @@ services:
     labels:
       - traefik.enable=true
       - traefik.http.routers.portainer.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
-      - traefik.docker.network=webapp
+      - traefik.docker.network=mgmt
       - traefik.http.services.portainer.loadbalancer.server.port=9000
       - traefik.http.routers.portainer.tls=true
       - traefik.http.routers.portainer.entrypoints=web,websecure
@@ -25,6 +25,8 @@ services:
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
       - ./data:/data
+    expose:
+      - 9000
     networks:
       - mgmt
 networks:

mgmt/vaultwarden/.gitignore

@@ -0,0 +1,2 @@
+.env
+.recovery_code

myprojects/kimi/.env.example

@@ -0,0 +1,2 @@
+# No env vars required for basic kimi CLI usage
+# Add API keys or config here as needed

myprojects/kimi/Dockerfile

@@ -0,0 +1,18 @@
+FROM alpine:3.23
+
+RUN apk add --no-cache bash curl
+
+RUN addgroup -g 1000 kimi && \
+    adduser -D -u 1000 -G kimi kimi
+
+USER kimi
+WORKDIR /home/kimi
+
+RUN curl -fsSL https://astral.sh/uv/install.sh | sh
+RUN /home/kimi/.local/bin/uv tool install --python 3.13 kimi-cli
+
+ENV PATH="/home/kimi/.local/bin:${PATH}"
+
+WORKDIR /workspace
+
+ENTRYPOINT ["kimi"]

myprojects/kimi/compose.yml

@@ -0,0 +1,12 @@
+services:
+  kimi:
+    build:
+      context: .
+      network: host
+    image: kimi:alpine
+    container_name: kimi
+    hostname: kimi
+    stdin_open: true
+    tty: true
+    volumes:
+      - ./workspace:/workspace

webapp/n8n/.gitignore

@@ -0,0 +1,2 @@
+.env
+local-files

webapp/n8n/compose.yml

@@ -30,7 +30,8 @@ services:
       - ./local-files:/files
     networks:
       - webapp
-      - db
+      - backend
+      - mcp
 secrets:
   DB_PASS:
     file: .secrets/DB_PASS
@@ -38,7 +39,10 @@ networks:
   webapp:
     external:
       true
-  db:
+  backend:
+    external:
+      true
+  mcp:
     external:
       true
 volumes:

webapp/navidrome/.gitignore

@@ -0,0 +1,3 @@
+data/
+music/
+music