Refresh streamhoard.md with current features, trim resolved historical incident narrative from index.md/servers.md

index.md

@@ -23,14 +23,14 @@ Myron Blair (`myronblair@outlook.com` / `myronblair@gmail.com`). Runs a home lab
 
 **No local codebase for most work** — it happens via SSH to remote servers/VMs and `git push` to GitHub (several repos have auto-deploy webhooks).
 
-### Infrastructure map (current as of 2026-07-24)
-- **DO (165.22.1.228)** — DigitalOcean VPS, CyberPanel/OpenLiteSpeed. Hosts 7 client/personal websites. NOT JARVIS, NOT FusionPBX (both moved on-prem).
+### Infrastructure map
+- **DO (165.22.1.228)** — DigitalOcean VPS, CyberPanel/OpenLiteSpeed. Hosts client/personal websites. NOT JARVIS, NOT FusionPBX (both on-prem).
 - **PVE1 (`orbisne.fortiddns.com` / 10.48.200.90)** — Proxmox primary hypervisor. Home of JARVIS, StreamHoard, NovaCPX, Ollama, FusionPBX, Home Assistant, Homebridge, and the NAS-mount passthrough for all of the above.
 - **PVE2 (10.48.200.91)** — Proxmox secondary hypervisor, 1 VM (NetworkBackup).
 - **JARVIS VM 211 (10.48.200.211)** — the JARVIS AI dashboard/agent-monitoring platform. nginx/PHP8.3/MariaDB/Redis/Arc Reactor.
-- **CT104 "StreamHoard" (10.48.200.104)** — consolidated media server: custom Netflix-style app + Sonarr/Radarr/Prowlarr/qBittorrent (all VPN'd via gluetun/NordVPN) + Portainer + MSP360 backup, all in Docker Compose. **Replaces the old Jellyfin VM112, MediaStack VM103, and WireGuard CT110 — all three were destroyed 2026-07-23.**
+- **CT104 "StreamHoard" (10.48.200.104)** — media server: custom Netflix-style app + Sonarr/Radarr/Prowlarr/qBittorrent (all VPN'd via gluetun/NordVPN) + Portainer + MSP360 backup, all in Docker Compose.
 - **NovaCPX VM120 (10.48.200.110)** — custom hosting control panel; also serves the `web.orbishosting.com` MSP dashboard.
-- **FusionPBX VM130 (10.48.200.130)** — FreeSWITCH PBX, moved on-prem from an old DO droplet (that droplet, 134.209.72.226, is destroyed — don't use it).
+- **FusionPBX VM130 (10.48.200.130)** — FreeSWITCH PBX.
 - **Synology NAS (10.48.200.249)** — primary media + MSP360 backup storage, own DSM reverse proxy, Gitea.
 - **NPM (10.48.200.200)** — Nginx Proxy Manager, terminates TLS for `jarvis.orbishosting.com` and a few others.
 - **FortiGate (10.48.200.1)** — router/firewall. Admin GUI is on **port 9443**, not 443 (redirects there). `admin` / `Joker1974!!!`.
@@ -41,5 +41,5 @@ Myron Blair (`myronblair@outlook.com` / `myronblair@gmail.com`). Runs a home lab
 - **web.orbishosting.com** — personal MSP links dashboard (repo `web-dashboard`), auto-deploys via webhook on VM110.
 - **Gitea** (`gitea.orbishosting.com`, on the NAS) mirrors every GitHub repo one-way (GitHub → Gitea). 5 repos exist Gitea-only by design (never pushed to GitHub — they hold real secrets): `fortigate-config`, `infra-private`, `jarvis-secrets`, `msp360-config`, `proxmox-secrets`.
 
-### Recent major change (2026-07-23/24)
-Jellyfin, MediaStack, and the WireGuard exit container were all decommissioned and consolidated into one host, StreamHoard (CT104). See [streamhoard.md](streamhoard.md) for the full detail — this was a large migration with several real infra bugs found and fixed along the way (a live duplicate-IP conflict, a NordVPN kill-switch regression, NFS/CIFS mount limitations in unprivileged LXCs, an MSP360 install gap). Don't be surprised if older docs/repos (`mediastack`, old `kino` repo) still reference the pre-migration state — they're stale now.
+### Copyright / attribution standard
+Every operational repo (kino-app, jarvis, web-dashboard, chucko, and the DO-server client sites) has a `NOTICE` file — "Property of TomTom Enterprises." Commits to these repos must not include AI-authorship trailers (no `Co-Authored-By: Claude` etc.). This does not apply to genuine product functionality that happens to call an AI API (e.g. JARVIS's own Claude integration is real functionality, not attribution, and stays as-is).

servers.md

@@ -36,8 +36,6 @@ sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no -o PreferredAuthentica
 
 **Password order to try if first fails:** `Joker1974!!!` → `Joker1974!` → `Gonewalk1974!@#` (DO only).
 
-**MediaStack-35's ad-hoc interactive SSH intermittently failed** (resolved as non-issue — the automated backup script's own key-based access always worked fine; root cause was never found, and it's moot now since that VM is destroyed). If a similarly-flaky interactive SSH pattern shows up on another host, don't assume it's a real access/security problem before checking whether an automated path using the same credentials still works.
-
 ## Server Map (current)
 
 | Host | IP | User | Password | Purpose |
@@ -57,14 +55,7 @@ sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no -o PreferredAuthentica
 | Synology NAS | 10.48.200.249 | nas | Joker1974!!! | Media + MSP360 backup storage, DSM 7.3.2, Gitea |
 | FortiGate | 10.48.200.1 | admin | Joker1974!!! | Router/firewall — admin GUI on **:9443**, not 443 |
 
-**Destroyed 2026-07-23 — do not use these IPs/VMIDs for anything:**
-| Was | IP | VMID |
-|---|---|---|
-| Jellyfin | 10.48.200.33 | VM112 |
-| MediaStack | 10.48.200.35 | VM103 (also briefly `.136` mid-migration) |
-| WireGuard exit container | 10.48.200.67 / .19 | CT110 |
-
-`10.48.200.35` is currently squatted by an **unidentified rogue device** (SSH banner `OpenSSH_8.4p1`, exposes NFS shares named `TimeMachineBackup`/`Public`/`iTunes` — looks like an old Buffalo-brand NAS). Found during the migration, never tracked down/removed. Don't be surprised if that IP answers but isn't what you expect.
+**Retired, do not use:** `10.48.200.33`, `.35`, `.67`, `.19`. `.35` is squatted by an unidentified device (SSH banner `OpenSSH_8.4p1`, NFS shares `TimeMachineBackup`/`Public`/`iTunes`) — don't assume that IP is safe or known if it answers.
 
 ## Proxmox API
 ```bash

streamhoard.md

@@ -1,60 +1,58 @@
 # StreamHoard Reference Card (CT104, 10.48.200.104)
 
-**Replaces the old Jellyfin (VM112), MediaStack (VM103), and WireGuard exit container (CT110) — all three destroyed 2026-07-23.** Everything media-related now lives on one Debian 12 LXC (CT104, 8 cores/12GB RAM/60GB disk) on PVE1.
-
 ## What it is
-A self-hosted Netflix-style media app (repo `myronblair/kino-app`, private) branded **"StreamHoard"** in the actual UI — internal names (container names, some file paths) still literally say `kino`/`Kino`, that's cosmetic leftover from the original scaffold, not a second app. FastAPI + React + MongoDB.
+Self-hosted Netflix-style media app — repo `myronblair/kino-app` (private), branded **"StreamHoard"** in the UI (internal names/container names still say `kino`, cosmetic only). FastAPI + React + MongoDB, Docker Compose in `/opt/kino` on CT104 (Debian 12 LXC on PVE1, 8 cores/12GB RAM/500GB disk).
 
-Deployed via Docker Compose in `/opt/kino` on CT104. Login: `admin@streamhoard.local` — password was rotated by the user via the app's own self-service "Change Password" feature (Settings page, requires current password) — not stored in this doc.
+Login: `admin@streamhoard.local` — password set via the app's own self-service "Change Password" (Settings page), not stored in this doc. App version shown in the navbar (`GET /api/version`, source of truth is `backend/VERSION` in the repo).
 
 ## Container stack
 | Container | Role | Port (host) |
 |---|---|---|
 | `streamhoard-frontend` | React UI, reverse-proxies `/api` to backend | 8080 |
-| `streamhoard-backend` | FastAPI — auth, catalog, streaming, transcode queue | internal 8001 |
-| `streamhoard-mongo` | MongoDB **pinned to 4.4** | internal |
-| `streamhoard-gluetun` | VPN gateway (NordVPN/WireGuard) for the whole *arr stack + qBittorrent | proxies 7878/8989/9696/8082 |
-| `streamhoard-radarr` / `-sonarr` / `-prowlarr` / `-qbittorrent` | Migrated **with existing config/library/history** from the old MediaStack VM — `network_mode: service:gluetun` so ALL their traffic is VPN'd, not just qBittorrent's | via gluetun |
-| `streamhoard-portainer` | Docker management GUI | 9443 (https) |
+| `streamhoard-backend` | FastAPI — auth, catalog, streaming, transcode queue, library scan/cleanup | internal 8001 |
+| `streamhoard-mongo` | MongoDB **pinned to 4.4** (PVE1's CPU lacks AVX, mongo:7+ SIGILL-crashes without this pin) | internal |
+| `streamhoard-gluetun` | VPN gateway (NordVPN/WireGuard), real kill switch — `iptables` default policy `DROP`, only `tun0`/LAN/VPN-handshake traffic allowed out | proxies 7878/8989/9696/8082 |
+| `streamhoard-radarr` / `-sonarr` / `-prowlarr` / `-qbittorrent` | `network_mode: service:gluetun` — share gluetun's netns, so kill switch covers all their traffic, not just torrent | via gluetun |
+| `streamhoard-portainer` | Docker management GUI (`https://10.48.200.104:9443`) | 9443 |
 
-**Why mongo:4.4:** PVE1's physical CPU lacks AVX, and mongo:7+ hard-requires it (SIGILL crash-loop without this pin).
+## Media/music mounts
+- NAS (`10.48.200.249`) mounts happen on **PVE1 itself**, then pass into CT104 as LXC mountpoints (`/etc/pve/lxc/104.conf`) — unprivileged LXCs can't do NFS/CIFS client mounts directly.
+- `mp0`: NFS `/volume1/video` → `/mnt/nas-video` (PVE1) → `/mnt/nas/video` (CT104). Radarr/Sonarr root folders `/mnt/nas/video/movies`, `/mnt/nas/video/tv`.
+- `mp1`: CIFS `Storage` share → `/mnt/nas-backups` — MSP360 backup destination.
+- `mp2`: CIFS `music` share (no NFS export exists for it) → `/mnt/nas-music` (PVE1, read-only) → `/mnt/nas/music` (CT104, read-only). Real content is under `Music/<Artist>/<Album>/<Track>`; sibling folders (`Apple Music`, `Automatically Add to Apple Music`, `Downloads-Music`) are Music.app-internal and deliberately not scanned — may contain DRM'd Apple Music subscription tracks, which no third-party player can decrypt/play.
+- Backend container bind-mounts `/mnt/nas/video` and `/mnt/nas/music` read-only at their identical absolute paths, so Radarr/Sonarr/Library-Scan-reported file paths resolve with zero translation.
 
-## Media path
-- NAS (`10.48.200.249:/volume1/video`) is mounted **on PVE1 itself** (`/mnt/nas-video`, `/etc/fstab`, `_netdev` — persists across PVE1 reboots), then passed into CT104 as an **LXC mountpoint** (`mp0`, config lives in `/etc/pve/lxc/104.conf` — persists across CT104 reboots too).
-- **Why not mount NFS/CIFS directly inside CT104:** unprivileged LXCs can't reliably do NFS *or* CIFS client mounts even with the `mount=nfs;cifs` feature flag enabled — confirmed both fail with "Operation not permitted." This is a hard Proxmox/kernel-capability limitation, don't waste time retrying it — always go through the PVE1-host-mount + LXC-mountpoint pattern instead.
-- Radarr/Sonarr root folders: `/mnt/nas/video/movies`, `/mnt/nas/video/tv` (unchanged from the old MediaStack setup). StreamHoard's backend container bind-mounts the same NAS path **read-only at the identical absolute path** (`/mnt/nas/video:/mnt/nas/video:ro`) so Radarr/Sonarr-reported `file_path` values resolve with zero translation.
-- StreamHoard's own Radarr/Sonarr integration settings point at `http://gluetun:7878` / `http://gluetun:8989` (internal Docker network — gluetun's container shares its netns with radarr/sonarr, so other containers reach them via gluetun's hostname).
-- NAS export ACL (`/etc/exports` on the NAS) now only allows **PVE1 (10.48.200.90)** — cleaned up the old MediaStack-only entry after that VM was destroyed.
+## Library Scan (admin, separate from Radarr/Sonarr)
+Walks the NAS mounts directly for movies/TV/music not already in the library — `storage_type="scan"` on the resulting Movie/Episode/Track record, fully integrated (streamable, transcodable, listed in Admin) same as Radarr/Sonarr imports. Select All / Deselect All per tab. Movie/TV scans are fast (filesystem stat only); music scan defers tag reads to import time (reading ID3/MP4 tags during the scan itself was the original bottleneck).
 
-## Transcoding
-Almost the entire imported library needed re-encoding — most source files are HEVC/H.265 video and/or DTS/EAC3 audio, neither browser-native. This is what causes playback "jitter"/silent-audio, **not** a network/NAS bottleneck (NAS read tested at 122 MB/s). Fix:
-- Queued ABR transcode (re-encode to H.264/AAC multi-bitrate HLS) for the whole existing library via the transcode-queue API.
-- Fixed a real code gap: `sonarr_import` was missing the auto-transcode-on-import hook that `radarr_import` already had (both now call `_enqueue_transcode` when the global `auto_transcode` setting is on). That setting is now globally `"abr"`, so future imports auto-queue transcoding with no manual step.
-- If a newly-imported title plays choppy or silent, check its codec first (`ffprobe` inside `streamhoard-backend`) before assuming a server problem.
+## Music library
+`Track` model (title/artist/album/track_number/duration/album_art_url), streamed via ranged `/api/stream/track/{id}`. Persistent bottom player (queue, seek, output-device picker via the Audio Output Devices API for already-paired Bluetooth speakers — Chrome/Edge only, browsers can't pair new BT devices). Album art: MusicBrainz + Cover Art Archive lookup (free, no key, 1 req/sec rate limit respected), triggered from Library Scan's Music tab; falls back to a branded placeholder (not a bare icon) until art is set. NAS mount is read-only, so Apple Music/iTunes can keep managing the same files independently.
 
-## MSP360 backup (weekly full / daily incremental to the NAS)
-- Installed via `dpkg-repack` from the working NovaCPX install (no public download URL for the Linux .deb worked — MSP360/CloudBerry doesn't publish a stable direct link).
-- **Real gap hit and fixed:** the package's config templates (`cloudBackup.conf`, `wt_config.xml`) are consumed/deleted from disk after a machine's first run, so `dpkg-repack`ing an already-initialized install captures a package that can never bootstrap its own runtime state (`/opt/local/Online Backup/<machine-guid>/config/...`) — the daemon just retries forever waiting for a file that will never appear. Fix: copy the *generated* versions of those two files from the source machine's live runtime directory (not the package) into the new install's equivalent path before first start.
-- Storage account: File System type, path `/mnt/nas-backups/MSPBackups` (same PVE1-mount + LXC-mountpoint pattern as the media NAS mount, separate mountpoint `mp1`).
-- Free license, self-registered via CLI: `cbbV2 license activate -f -e <email> -c`.
-- Plan **StreamHoard-Backup**: daily incremental 2:00 AM, weekly full every Monday 2:00 AM, 4-week retention. Excludes: the huge media NAS mount, the backup destination itself, ephemeral Docker image layers (`overlay2`/`containers`/`image`/`buildkit`), and the regenerable HLS transcode cache (`/opt/kino/media/hls`, several GB, not "configuration").
-- **A leftover worker process from a killed/deleted plan will silently block a new plan's worker** (near-zero CPU usage despite the job showing "Running" for many minutes is the tell) — `ps aux | grep cbbWorker` and kill any process referencing an old/deleted plan ID before assuming a new plan is broken.
-- CLI binary: `/opt/local/MSP360\ Backup/bin/cbb` (legacy, `plan -l`/`-r`/`-s -f`) and `cbbV2` (full plan create/edit/account management) — the space in the path needs quoting in every invocation.
+## Library Cleanup (admin)
+Background job (`POST .../scan/start`, `GET .../status`, `POST .../stop` — has a real Stop button, not just a spinner) checking for: missing files, orphaned episodes/HLS output, stuck transcode jobs (running 2h+ with no completion), duplicate imports, orphaned subtitles, dangling watchlist/progress rows. Nothing deletes until you click Fix. File-existence checks run in a small thread pool off the event loop — an earlier version did this synchronously and froze the *entire server* for as long as the scan took (a plain `GET /api/movies` would hang); confirmed fixed, don't reintroduce blocking filesystem calls in an async endpoint here.
 
-## Portainer
-- `https://10.48.200.104:9443` — first-run setup token appears in `docker logs streamhoard-portainer`, and expires after ~5 minutes; restart the container to get a fresh one if it lapses.
-- Endpoint renamed "StreamHoard" for clarity; local Docker socket auto-detected, no extra config needed.
+## Transcode queue
+One job at a time, `nice -n 10` (was 19), threads divided across however many ABR variants a job has so the total stays near a fixed budget instead of each variant claiming the full count independently. Auto-transcode setting (`off`/`quick`/`abr`, in Settings) auto-queues new imports.
+
+Three quality modes:
+- **quick** — stream-copy both streams (fast, no quality loss, but doesn't fix incompatible audio).
+- **abr** — full re-encode to multi-bitrate H.264/AAC HLS (slow — 4 variants, only way to fix HEVC/x265 video).
+- **audio** — video stream-copied untouched, only audio re-encoded to AAC (fast — no video re-encoding). Fixes the single most common playback problem: BDRip/AVI sources very often carry AC3/DTS/EAC3 audio, which no browser can decode natively — browsers respond by hiding the volume control entirely, not by erroring. `POST /transcode/queue/fix-audio-all` cancels the pending backlog and replaces it with the fast audio-only fix library-wide; run a real ABR pass later for actual multi-bitrate quality once there's no urgency.
+
+Retry (`/transcode/queue/{id}/retry`) and bulk retry-failed never delete the movie/episode — only the job history and `hls_status` flag change.
+
+## Requests → Radarr/Sonarr
+Pending user requests can be Approved (Movie or TV) by an admin: looks up the title via Radarr/Sonarr's own lookup API, adds the best match with search-on-add enabled. Atomically claims the request (`pending → approving → approved`) so a double-click or two admins can't both add the same content; releases the claim back to `pending` on failure so it can be retried. Live status line (searching/queued/downloading/downloaded) reads from Radarr/Sonarr's own queue, polled every 15s — real data relayed through Radarr's own refresh cycle, not a direct qBittorrent connection.
+
+## MSP360 backup
+Weekly full / daily incremental to the NAS (`/mnt/nas-backups/MSPBackups`, plan **StreamHoard-Backup**). Installed via `dpkg-repack` from the NovaCPX install (no public Linux .deb download exists). Excludes the media/music NAS mounts, the backup destination itself, Docker image layers, and the regenerable HLS transcode cache.
 
 ## JARVIS agent
-Installed and registered (hostname `streamhoard104`, `watch_services: ["docker"]`) — see jarvis.md Agent System section. Watch out for the container's actual OS hostname needing to match what you want reported (a typo here caused a duplicate stale agent registration once).
+Registered as `streamhoard104`, `watch_services: ["docker"]` — watches the Docker daemon only, not individual containers. A `/usr/local/bin/streamhoard-watchdog.sh` cron (every 5 min) separately restarts any StreamHoard container that isn't running, since Docker's own `restart: unless-stopped` policy doesn't always trigger reliably.
 
-## Incidents hit during the 2026-07-23 migration (useful context if déjà vu happens elsewhere)
-- **Live duplicate-IP conflict**: an unidentified device (SSH banner `OpenSSH_8.4p1`, NFS shares `TimeMachineBackup`/`Public`/`iTunes` — looks like an old Buffalo-brand NAS) was actively answering ARP for `10.48.200.35` alongside the real MediaStack VM's MAC. Not stale cache — a live second device on the wire, never identified/removed. Worked around by moving MediaStack to a different IP mid-migration (moot now, VM destroyed) — `.35` is still squatted, avoid it.
-- That IP-reassignment **silently broke MediaStack's NordVPN kill-switch** (the `fwmark → routing table` `ip rule` entry vanished after the netplan change) — traffic briefly went out unprotected. Lesson: any NIC/netplan change on a host running NordVPN's Linux CLI app should be followed by `ip rule show`, not just `nordvpn status` — status can say "Connected" while the actual policy routing is gone. Fix: `nordvpn disconnect && nordvpn connect` reasserts the rule.
-- The NAS's own NFS export root (`/volume1/video` itself) has mode `000` due to Synology's `hide` export option — real root (PVE1) bypasses this via DAC-override, an unprivileged LXC's "root" cannot, causing `Permission denied` even though subfolders were readable. Fixed with `chmod 755` on the mount root + `chmod 777` on `movies`/`tv`/`downloads` specifically (also sidesteps a UID mismatch between the NAS's real file ownership and whatever PUID/PGID the new containers use).
-- gluetun needs `/dev/net/tun` explicitly passed into the LXC (`pct set 104 -dev0 /dev/net/tun`) or it fails immediately trying to create the WireGuard interface.
+## Copyright
+Repo has a `NOTICE` file ("Property of TomTom Enterprises."). Commits to this repo omit AI-authorship trailers.
 
-## Not yet done / open items
-- FortiGate VIP `tunnel-51820 / Wireguard-CT110` — **already cleaned up** (removed 2026-07-24 along with its dead firewall policy, confirmed Ref:0 before deletion).
-- The rogue device on `10.48.200.35` was never identified or removed from the network.
-- No reverse proxy/TLS in front of StreamHoard yet, and no LAN-only firewall rule (the app itself has no built-in rate limiting).
+## Open items
+- No reverse proxy/TLS in front of StreamHoard, no LAN-only firewall rule.
+- Large transcode backlog from initial library import — being worked through via the fast audio-fix pass; a full ABR pass for real adaptive-bitrate quality is still pending for most of the library.