Full rewrite for current infra state — StreamHoard consolidation, VoIP split out, dead references purged

- Replace all Jellyfin/MediaStack/WireGuard-CT110 references (destroyed 2026-07-23) with StreamHoard/CT104 throughout; add streamhoard.md covering the full consolidated stack, MSP360 backup setup, and migration incidents - Split FusionPBX/SignalWire/landline content into its own voip.md (was scattered across jarvis.md and sites.md, and referenced the old destroyed DO droplet IP) - Fix stale facts: JARVIS webhook port (:1972 -> 80/https), rotated registration key/GitHub PAT, FortiGate admin port (9443 not 443), current WAN-IP-drift/VIP gotchas - Add reference.md for repo inventory notes, assistant hard limits, ZeroTraceGPT IQ, payments migration status - Update gotchas.md with unprivileged-LXC mount limitations and MSP360 dpkg-repack gotcha
2026-07-27 17:23:53 -05:00 · 2026-07-25 22:47:31 -05:00
parent f5446d3208
commit 4fad312f87
8 changed files with 398 additions and 231 deletions
@@ -1,63 +1,65 @@
 # Critical Gotchas — Read Before Running Commands

-## PHP / OLS
+## Unprivileged LXC containers (Proxmox) — mount limitations
+- **NFS and CIFS client mounts both fail** inside an unprivileged LXC even with the `mount=nfs;cifs` features flag set — "Operation not permitted." This is a hard kernel-capability limitation, not a config mistake; don't retry it.
+- **Fix:** mount the share on the PVE host itself (fstab, `_netdev`), then pass it into the container as an **LXC mountpoint** (`pct set <CTID> -mpN /host/path,mp=/container/path`). This persists across reboots on both the host and the container.
+- A network share's root export directory can have restrictive permissions (e.g. Synology's `hide` NFS option sets mode `000` on the share root) that real root on the PVE host silently bypasses via DAC-override, but an unprivileged container's "root" cannot — causes `Permission denied` even though subfolders are readable. Fix on the source NAS/host, not in the container.
+- Passing a TUN device (needed for WireGuard-based VPN containers like gluetun) needs explicit `pct set <CTID> -devN /dev/net/tun`.
+
+## MSP360 / CloudBerry Backup
+- CLI binary path has a space: `/opt/local/MSP360\ Backup/bin/cbb` (legacy CLI, `plan -l/-r/-s`) or `cbbV2` (full plan/account create/edit) — always quote it.
+- **`dpkg-repack`ing an already-initialized install captures a package missing its own bootstrap config templates** (`cloudBackup.conf`, `wt_config.xml` get consumed/deleted from the package tree after first run). A fresh install from that repacked .deb hangs forever waiting on a file that will never appear. Fix: copy the *generated* versions of those files from the source machine's live runtime directory into the new install before first start.
+- **A leftover worker process from a killed/deleted plan silently blocks a new plan's worker** — the tell is near-zero CPU usage despite the job showing "Running" for many minutes. `ps aux | grep cbbWorker`, kill anything referencing an old/deleted plan ID.
+- Free/Freeware license self-registers via `cbbV2 license activate -f -e <email> -c` — no manual key needed.
+
+## PHP / OpenLiteSpeed (DO server)
 - **NEVER** use `lsphp -l` for syntax check — it segfaults. Use `php8.3 -l file.php`
 - Run CLI scripts with: `/usr/local/lsws/lsphp85/bin/lsphp /path/script.php`
 - If endpoint uses `ob_start()` + header.php pattern → add `ob_end_clean()` before CSV/JSON output
 - MySQL charset: always `utf8mb4_unicode_ci` — mixing with `general_ci` breaks JOINs (error 1267)
+- Cron bug: `*/3 * * * *` restarts `lshttpd` if any `.htaccess` is newer than `/usr/local/lsws/cgid` — nothing re-touches that dir after restart, causing a restart loop until you `touch /usr/local/lsws/cgid` manually.

 ## SSH / Networking
- DO server (165.22.1.228) **cannot reach local network** (10.48.200.x) directly
- To reach local VMs from DO: use agent commands (shell type) or FortiGate DDNS
- PVE1 SSH via DDNS works: `root@orbisne.fortiddns.com` (Joker1974!!!)
- PVE2 has no external port forward — only reachable locally or via cluster API through PVE1
- Proxmox API port 8006 IS forwarded: `orbisne.fortiddns.com:8006` works from DO
+- DO server (165.22.1.228) **cannot reach the local network** (10.48.200.x) directly.
+- PVE1 SSH via FortiGate DDNS works: `root@orbisne.fortiddns.com` (survives WAN IP changes, which happen fairly often — check `curl ifconfig.me` before trusting a documented WAN IP).
+- PVE2 has no external port forward — only reachable locally or via cluster API through PVE1.
+- Proxmox API port 8006 IS forwarded: `orbisne.fortiddns.com:8006`.
+- FortiGate admin GUI is on **port 9443**, not 443 (443/80 both redirect there).
+- A host's SSH interactive access can be intermittently flaky (e.g. through nested nested plink→ssh→ssh hops) while its own automated scripts using the same key work fine every time — that's usually not a real security/access problem, just a quirk of the ad-hoc path.

 ## JARVIS Agents
 - Agent config: `/etc/jarvis-agent/config.json` | Runtime state: `/var/lib/jarvis-agent/state.json`
- **401 "Invalid agent key"** → state.json has stale key. Fix: overwrite state.json with correct agent_id + api_key from `registered_agents` table, then `systemctl restart jarvis-agent`
- Agent heartbeat uses `X-Agent-Key` header (NOT body field)
- `shell` command type requires `{"command":"...","allowed":true}` in command_data
- Metrics stored as JSON in `metric_data` column — use `JSON_EXTRACT(metric_data,'$.cpu_percent')` NOT direct columns
+- **401 "Invalid agent key"** → state.json has a stale key. Fix: overwrite with correct `agent_id`/`api_key` from `registered_agents` table, then `systemctl restart jarvis-agent`.
+- Metrics stored as JSON in `metric_data` column — use `JSON_EXTRACT(metric_data,'$.cpu_percent')`, no direct columns.
+- **Set the container/VM's actual OS hostname correctly before first agent registration** — a hostname typo becomes part of the generated `agent_id` and creates a stale duplicate registration that needs manual DB cleanup later.

 ## Groq AI
- Model name: `compound-beta-mini` — NOT `groq/compound-beta-mini` (that's OpenAI router syntax, 404s)
+- Model name: `compound-beta-mini` — NOT `groq/compound-beta-mini` (that's OpenAI router syntax, 404s).

 ## Proxmox
- stats_cache.php uses `orbisne.fortiddns.com:8006` NOT `PROXMOX_HOST` (local IP unreachable from DO)
- `--nameserver` in Proxmox must be space-separated: `"8.8.8.8 1.1.1.1"` (comma causes netplan bug)
- Run commands in VMs: `qm guest exec <VMID> -- bash -c 'cmd'` (requires guest agent installed)
+- `--nameserver` must be space-separated: `"8.8.8.8 1.1.1.1"` (comma causes a netplan bug).
+- Run commands in a VM even when SSH/network is down: `qm guest exec <VMID> -- bash -c 'cmd'` (needs guest agent; also needs `export HOME=/root` for many CLI tools since guest-exec has no HOME env by default).
+- Run commands in an LXC: `pct exec <CTID> -- bash -c 'cmd'`.

 ## Deploy
- Always `git add + commit + push` after editing files on server — webhook auto-deploys within 1 min
- PHP syntax validated before deploy — bad commits auto-reverted
- LSAPI session deadlock: `session_write_close()` must be called in api.php after auth check
+- Repos with auto-deploy webhooks (JARVIS, web-dashboard, StreamHoard's compose isn't webhook-deployed — it's manual `git pull` + `docker compose up -d`): always `git add + commit + push` — don't forget to also trigger the Gitea mirror-sync afterward (`curl -X POST -u myron:Joker1974!!! https://gitea.orbishosting.com/api/v1/repos/myron/<repo>/mirror-sync`).
+- The 7 DO-hosted sites have **no auto-deploy** — manual SSH + git pull only.

-## API Endpoint Auth
- Netscan endpoint (`/api/netscan`) bypasses main auth — uses `X-Registration-Key` header
- Admin portal uses separate PHP session name (`jarvis_admin`) — different from main JARVIS session
- Cloudflare real IP: use `$_SERVER['HTTP_CF_CONNECTING_IP']` not `REMOTE_ADDR`
+## API Endpoint Auth (JARVIS)
+- Netscan endpoint (`/api/netscan`) bypasses main auth — uses `X-Registration-Key` header.
+- Admin portal uses a separate PHP session name (`jarvis_admin`) — different from the main JARVIS session.
+- Cloudflare real IP: use `$_SERVER['HTTP_CF_CONNECTING_IP']` (or `HTTP_X_FORWARDED_FOR`), not `REMOTE_ADDR`.

 ## Network Scan
- The JARVIS "RUN NETWORK SCAN" button does NOT scan from DO (can't reach local network)
- It queues a shell command to PVE1 agent → PVE1 runs nmap → pushes results to /api/netscan
- Results appear ~40 seconds after clicking (10s for agent pickup + 30s nmap)
- Chat "scan network" intent returns real DB data — never hallucinated
+- The JARVIS "RUN NETWORK SCAN" button doesn't scan from wherever the browser is — it queues a shell command to the PVE1 agent, which runs nmap and pushes results to `/api/netscan`. Results appear ~40s after clicking.
+- Chat "scan network" intent returns real DB data — never hallucinated.

-## FusionPBX
- SIP config changes need cache delete before they take effect:
-  `rm /var/cache/fusionpbx/FusionPBX.configuration.sofia.conf`
- mod_presence is NOT installed on this server
+## FusionPBX / VoIP
+See [voip.md](voip.md) for the full reference — SignalWire has several non-obvious credential-type and API-endpoint gotchas that cost real debugging time in the past; don't re-derive them from scratch.
+- SIP config changes need the sofia.conf Lua cache cleared, not just a service restart: `rm /var/cache/fusionpbx/<hostname>.configuration.sofia.conf`

-## Backup Agent State Fix (Common Issue)
-```bash
-# If an agent shows "Invalid agent key" after reinstall:
-# 1. Get correct values from DB
-mysql -u jarvis_user -pJ4rv1s_Pr0t0c0l_2026! jarvis_db -e \
-  "SELECT agent_id, api_key FROM registered_agents WHERE hostname='<hostname>';"
-# 2. Overwrite state on the agent machine
-cat > /var/lib/jarvis-agent/state.json << EOF
-{"api_key": "<api_key_from_db>", "agent_id": "<agent_id_from_db>"}
-EOF
-systemctl restart jarvis-agent
-```
+## Git push confirmation
+Pushing to a repo's default branch (master/main) needs an explicit, branch-named confirmation from the user — a general "yes" to "want me to push this?" isn't enough and will get blocked. Ask "confirm push directly to master?" up front to save a round-trip. If a chained `git add && commit && push` gets denied, nothing in the chain ran — re-check `git status` before retrying individual steps.
+
+## Live log popup convention (JARVIS admin)
+Any server-side action triggered from JARVIS admin (setup, deploy, install, run backup) must show a live log popup — NOT a `confirm()`/`toast()`. Pattern: `openModal()` + scrollable dark log div + backend log-polling endpoint (`{lines:[], next_line:N}`) + color-coded lines (green=ok, red=error, yellow=warn) + auto-detect completion + CLOSE button that stops polling. Poll every 2s, snapshot log position before triggering the action to avoid replaying old output. `runIntentGenerator()` in the admin JS is the canonical reference implementation.
@@ -1,38 +1,45 @@
-# AI Context Index — Myron Blair Home Lab
+# AI Context Index — Myron Blair Home Lab / MSP

-**Load order for a new session:** Read this file first, then load the specific topic file for the task at hand.
+**Load order for a new session:** Read this file first, then load the specific topic file for the task at hand. This repo is meant to be handed to *any* AI assistant (not just Claude) to resume full context on this environment.

 ## Topic Files

 | File | Use When |
 |------|----------|
-| [servers.md](servers.md) | Connecting to any machine, SSH patterns, credentials |
+| [servers.md](servers.md) | Connecting to any machine, SSH patterns, credentials, VM map |
 | [jarvis.md](jarvis.md) | Working on JARVIS AI dashboard, API, chat, agents |
+| [streamhoard.md](streamhoard.md) | Working on the media server / *arr stack / VPN / Portainer / MSP360 backup on CT104 |
+| [voip.md](voip.md) | FusionPBX, SignalWire, landline/FXO, SIP registration, phones |
 | [sites.md](sites.md) | Working on any website, deploys, DBs, email |
 | [gotchas.md](gotchas.md) | Before running any command — critical quirks |
+| [reference.md](reference.md) | Repo inventory notes, assistant hard limits, ZeroTraceGPT IQ, payments migration status |
+
+## Who this is for
+Myron Blair (`myronblair@outlook.com` / `myronblair@gmail.com`). Runs a home lab + small MSP operation: Proxmox cluster, several VMs/LXCs, self-hosted AI (JARVIS + Ollama), VoIP (FusionPBX on-prem), a Docker-based media stack, and public client websites on DigitalOcean. Comfortable with Linux sysadmin, PHP, Python, bash, git, systemd — has deep familiarity with his own infra but won't remember every credential/IP off the top of his head, so don't assume he'll catch an error in a recalled detail.
+
+**Working style:** wants terse responses — state results, not intentions ("Let me..."), no trailing recaps of what was just done. For any server-side action triggered from an admin UI (JARVIS, etc.), use a live log popup pattern, not a toast/confirm dialog (see gotchas.md). Pushing to a repo's default branch (master/main) needs an explicit branch-named confirmation, not a generic "yes."

 ## 30-Second Overview

-**This is a home-lab + managed-hosting environment.**  
-No local codebase. Work happens via SSH to remote servers and git push to GitHub (auto-deploy active).
+**No local codebase for most work** — it happens via SSH to remote servers/VMs and `git push` to GitHub (several repos have auto-deploy webhooks).

-### Infrastructure
- **DO (165.22.1.228)** — DigitalOcean VPS. CyberPanel/OpenLiteSpeed. Hosts all websites (NOT JARVIS anymore).
- **JARVIS VM (10.48.200.211)** — PVE1 VM 211. nginx/PHP8.3/MariaDB/Redis/Arc Reactor. JARVIS AI lives here.
- **NPM (10.48.200.200)** — Nginx Proxy Manager. Routes jarvis.orbishosting.com + hoa.orbishosting.com → local VMs.
- **PVE1 (orbisne.fortiddns.com / 10.48.200.90)** — Proxmox primary hypervisor. 10+ VMs.
- **PVE2 (10.48.200.91)** — Proxmox secondary hypervisor. 2 VMs.
- **FusionPBX (134.209.72.226)** — FreeSWITCH PBX server.
- **Local VMs** — HA (10.48.200.97), Ollama (10.48.200.210), Homebridge (10.48.200.18), Jellyfin (10.48.200.33), NovaCPX (10.48.200.110), MediaStack (10.48.200.35).
+### Infrastructure map (current as of 2026-07-24)
+- **DO (165.22.1.228)** — DigitalOcean VPS, CyberPanel/OpenLiteSpeed. Hosts 7 client/personal websites. NOT JARVIS, NOT FusionPBX (both moved on-prem).
+- **PVE1 (`orbisne.fortiddns.com` / 10.48.200.90)** — Proxmox primary hypervisor. Home of JARVIS, StreamHoard, NovaCPX, Ollama, FusionPBX, Home Assistant, Homebridge, and the NAS-mount passthrough for all of the above.
+- **PVE2 (10.48.200.91)** — Proxmox secondary hypervisor, 1 VM (NetworkBackup).
+- **JARVIS VM 211 (10.48.200.211)** — the JARVIS AI dashboard/agent-monitoring platform. nginx/PHP8.3/MariaDB/Redis/Arc Reactor.
+- **CT104 "StreamHoard" (10.48.200.104)** — consolidated media server: custom Netflix-style app + Sonarr/Radarr/Prowlarr/qBittorrent (all VPN'd via gluetun/NordVPN) + Portainer + MSP360 backup, all in Docker Compose. **Replaces the old Jellyfin VM112, MediaStack VM103, and WireGuard CT110 — all three were destroyed 2026-07-23.**
+- **NovaCPX VM120 (10.48.200.110)** — custom hosting control panel; also serves the `web.orbishosting.com` MSP dashboard.
+- **FusionPBX VM130 (10.48.200.130)** — FreeSWITCH PBX, moved on-prem from an old DO droplet (that droplet, 134.209.72.226, is destroyed — don't use it).
+- **Synology NAS (10.48.200.249)** — primary media + MSP360 backup storage, own DSM reverse proxy, Gitea.
+- **NPM (10.48.200.200)** — Nginx Proxy Manager, terminates TLS for `jarvis.orbishosting.com` and a few others.
+- **FortiGate (10.48.200.1)** — router/firewall. Admin GUI is on **port 9443**, not 443 (redirects there). `admin` / `Joker1974!!!`.

-### Key Systems
- **JARVIS** — Iron Man AI dashboard at `https://jarvis.orbishosting.com` — agent monitoring, chat, network scan, admin portal
- **Admin portal** — `https://jarvis.orbishosting.com/admin/` — full CRUD for all JARVIS data
- **Auto-deploy** — push to GitHub `main` → webhook → server pulls within 1 minute
+### Key systems
+- **JARVIS** — Iron Man-style AI dashboard at `https://jarvis.orbishosting.com`, agent monitoring/chat/network-scan/admin. Auto-deploy: push to GitHub `master` → webhook → pulls into `/var/www/jarvis/` within ~1 min.
+- **StreamHoard** — self-hosted Netflix-style media app (repo `kino-app`, branded "StreamHoard" in the UI — internal names still say `kino` in places, cosmetic only) at `http://10.48.200.104:8080`.
+- **web.orbishosting.com** — personal MSP links dashboard (repo `web-dashboard`), auto-deploys via webhook on VM110.
+- **Gitea** (`gitea.orbishosting.com`, on the NAS) mirrors every GitHub repo one-way (GitHub → Gitea). 5 repos exist Gitea-only by design (never pushed to GitHub — they hold real secrets): `fortigate-config`, `infra-private`, `jarvis-secrets`, `msp360-config`, `proxmox-secrets`.

-### Current Status (as of 2026-06-26)
- JARVIS fully migrated to PVE1 VM 211 (10.48.200.211) — not on DO anymore
- JARVIS agents online: DO, PVE1, PVE2, NetworkBackup, HA, Homebridge, Jellyfin, MediaStack, NovaCPX, FortiGate, Yealink T48S
- Daily backups running at 7am UTC to `/var/backups/jarvis/`, downloadable from admin panel
- Network auto-scan: PVE1 cron every 3 min → pushes to `/api/netscan`
- Ollama at 10.48.200.210:11434 (listening on 0.0.0.0 — fixed 2026-06-24)
+### Recent major change (2026-07-23/24)
+Jellyfin, MediaStack, and the WireGuard exit container were all decommissioned and consolidated into one host, StreamHoard (CT104). See [streamhoard.md](streamhoard.md) for the full detail — this was a large migration with several real infra bugs found and fixed along the way (a live duplicate-IP conflict, a NordVPN kill-switch regression, NFS/CIFS mount limitations in unprivileged LXCs, an MSP360 install gap). Don't be surprised if older docs/repos (`mediastack`, old `kino` repo) still reference the pre-migration state — they're stale now.
@@ -4,22 +4,21 @@
 - **Dashboard:** https://jarvis.orbishosting.com (login: myron / Joker1974!!!)
 - **Admin portal:** https://jarvis.orbishosting.com/admin/ (same login)
 - **DB:** `jarvis_db` on VM 211 localhost — user: `jarvis_user` / `J4rv1s_Pr0t0c0l_2026!`
- **phpMyAdmin:** https://jarvis.orbishosting.com/phpmyadmin (myron / Joker1974!!!)
- **GitHub repo:** myronblair/jarvis (auto-deploy on push to main)
+- **GitHub repo:** `myronblair/jarvis` (private; auto-deploy on push to `master`)

 ## Server Location
-JARVIS runs on **PVE1 VM 211** at `10.48.200.211` (migrated from DO 2026-06-18).
- **SSH:** `sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.211`
+JARVIS runs on **PVE1 VM 211** at `10.48.200.211`.
 - **Stack:** nginx / PHP 8.3 / MariaDB / Redis / Arc Reactor
- **Web root:** `/var/www/jarvis/`
- Traffic routed via NPM (10.48.200.200) → VM 211
+- **Web root:** `/var/www/jarvis/` — this directory **is** the live git repo, push from it directly
+- **TLS:** terminated by NPM (VM105, 10.48.200.200), proxy_host → `10.48.200.211:80`, Let's Encrypt cert. **The old `:1972` port reference is wrong/dead** — use plain `http://jarvis.orbishosting.com` (port 80) or the HTTPS URL above. GitHub webhook is `https://jarvis.orbishosting.com/webhook.php`.

 ## File Structure (on VM 211 at /var/www/jarvis/)
 ```
 public_html/
  index.html          — main Iron Man HUD (all UI)
  api.php             — API router
-  admin/index.php     — admin portal (single PHP+JS file)
+  admin/index.php     — admin portal (single ~5000-line PHP+JS file, deliberately left unsplit — see gotchas)
+  admin/downloads/INFRASTRUCTURE-REFERENCE.md — master live infra doc, gitignored, edit here first
  agent/              — agent installers
 api/
  config.php          — all credentials/constants (gitignored)
@@ -27,109 +26,91 @@ api/
  lib/kb_engine.php    — KBEngine intent matching
  endpoints/
    agent.php          — agent registration/heartbeat/metrics/commands
-    chat.php          — 4-tier chat: KB→action intents→Ollama→Groq→Claude
+    chat.php           — tiered chat: KB → action intents → Ollama → Groq → Claude
    network.php        — network device list + scan endpoint
-    netscan.php       — push endpoint for PVE1 nmap results (no auth needed)
-    do_server.php     — reads /proc directly (no SSH loopback)
+    netscan.php        — push endpoint for PVE1 nmap results (uses X-Registration-Key, bypasses main auth)
    stats_cache.php    — every 5min cron: Proxmox cluster API, HA, weather, news
    facts_collector.php — every 3min cron: system stats, site health
-    system.php        — local system metrics
+    kb_intent_generator.php — every 6h cron, self-expanding knowledge base (see below)
    alerts.php         — alert CRUD + auto-generate
-    news.php          — serves api_cache['news'] + custom kb_facts(category='custom_news')
 deploy/
-  reactor.py          — Arc Reactor source (copy to /opt/jarvis-arc/reactor.py + restart)
+  reactor.py           — Arc Reactor source (copy to /opt/jarvis-arc/reactor.py + `systemctl restart jarvis-arc` to deploy)
 ```

-## Arc Reactor (AI Routing Daemon)
- **Daemon:** `/opt/jarvis-arc/reactor.py` (Python, port 7474), service: `jarvis-arc`
- **Deploy:** copy `deploy/reactor.py` → `/opt/jarvis-arc/reactor.py` + `systemctl restart jarvis-arc`
- Vision with actual screenshot → Claude `claude-opus-4-8-20251101`
- Guardian/SITREP/Vision text-only → Groq `llama-3.3-70b-versatile`
- Email drafting / research → Claude
+## Arc Reactor (AI routing daemon)
+- **Daemon:** `/opt/jarvis-arc/reactor.py` (Python venv), service: `jarvis-arc`, port 7474
+- **Logs:** `/var/log/jarvis/arc.log` (setup log: `arc-setup.log`)
+- **Vision cascade:** Claude API first; if Claude credits depleted, falls back to Ollama `llava:7b` (config via `/etc/systemd/system/jarvis-arc.service.d/vision.conf` → `OLLAMA_VISION_MODEL=llava:7b`)
+- Secrets (Claude/Groq keys, DB pass, Gmail/iCloud app passwords) live in `/etc/jarvis-arc/reactor.env` (root:www-data 0640, loaded via systemd EnvironmentFile drop-in) — NOT in the repo.
+
+## AI tiers (chat + reactor)
+```
+Tier 0.7: KB intents / planner
+Tier 1:   Knowledge Base (MySQL, self-expanding — see KB Generator below)
+Tier 1.5: Ollama llama3.1:8b (10.48.200.210:11434) — CPU-only, ~30s+/reply, can 504 on default 60s nginx timeout
+Vision:   Ollama llava:7b (via Arc Reactor vision cascade)
+Tier 2:   Groq (model name: compound-beta-mini — NOT groq/compound-beta-mini, that 404s)
+Tier 3:   Claude API (fallback; credits need periodic refill at console.anthropic.com)
+```
+Groq is effectively the fast/primary path when Claude credits are depleted — Ollama is too slow for interactive chat (504s).

 ## Agent System
- **Registration key:** `f846a9aaf7ce9a61742c63c87c4186052a71d2a580c65518`
- **Install one-liner (Linux):** `curl -sk https://jarvis.orbishosting.com/install-agent.sh | bash -s <hostname> <linux|proxmox>`
- **Agent config:** `/etc/jarvis-agent/config.json` (key: `jarvis_url`) — runtime state at `/var/lib/jarvis-agent/state.json`
- **If agent gets 401 "Invalid agent key":** state.json has stale key — overwrite with correct agent_id + api_key from DB
- **Heartbeat:** every 10s | **Metrics:** every 30s
-
-## Currently Online Agents
-| agent_id | hostname | IP | type |
-|----------|----------|----|------|
-| jarvis-do_orbis.or | jarvis-do | 165.22.1.228 | linux |
-| claude_pve | pve1 | 10.48.200.90 | proxmox |
-| pve2_pve2 | pve2 | 10.48.200.91 | proxmox |
-| networkbackup_NetworkB | networkbackup | 10.48.200.99 | linux |
-| homeassistant_ha | homeassistant | 10.48.200.97 | homeassistant |
-| homebridge_b57cbaea | homebridge | 10.48.200.18 | linux |
-| novacpx_e3b07264 | novacpx | 10.48.200.110 | linux |
-| jellyfin_7e386833 | jellyfin | 10.48.200.33 | linux |
-| MediaStack_2c00b1b8 | mediastack | 10.48.200.35 | linux |
-| fortigate_gw | fortigate | 10.48.200.1 | linux |
-| yealink_t48s | yealink | 10.48.200.43 | linux |
+- Agents register via `/api/agent` using a shared **registration key** (rotated multiple times historically — always read the current value from `/var/www/jarvis/api/config.php` `AGENT_REGISTRATION_KEY`, don't trust an old hardcoded value from any doc). Per-agent heartbeats then use their own `api_key` from the `registered_agents` table, not the shared reg key.
+- **Agent script:** `/opt/jarvis-agent/jarvis-agent.py` (Python3, no external deps), config at `/etc/jarvis-agent/config.json`:
+  ```json
+  {
+    "jarvis_url": "http://10.48.200.211",
+    "ssl_verify": false,
+    "registration_key": "<current key>",
+    "hostname": "<host>",
+    "agent_type": "linux",
+    "poll_interval": 30,
+    "heartbeat_every": 10,
+    "watch_services": ["<systemd unit names to health-check>"]
+  }
+  ```
+  Systemd unit `jarvis-agent.service` (Restart=always). State file `/var/lib/jarvis-agent/state.json`.
+- **401 "Invalid agent key"** → state.json has a stale key; delete it and restart the service to force re-registration, or manually overwrite with the correct `agent_id`/`api_key` from the DB.
+- Agent auto-detects capabilities: `docker` (if docker binary present), `proxmox`, `ollama`, `homeassistant`.
+- **Known registered agents (not exhaustive, check `registered_agents` table for current truth):** PVE1, PVE2, NetworkBackup, Home Assistant, Homebridge, NovaCPX, Ollama, StreamHoard/CT104 (added 2026-07-23, hostname `streamhoard104`). Jellyfin/MediaStack/WireGuard agents no longer exist (VMs destroyed).

 ## Network Scanning
- PVE1 cron: `*/3 * * * * /usr/local/bin/jarvis-netscan.sh`
- Script runs nmap, parses output, POSTs JSON to `https://jarvis.orbishosting.com/api/netscan` with `X-Registration-Key` header
- Scan Now button queues shell command to PVE1 agent (picks up within 10s)
+- PVE1 cron `*/3 * * * *` runs `jarvis-netscan.sh` → nmap → POSTs JSON to `/api/netscan` with `X-Registration-Key` header.
+- Chat "scan network" intent returns real DB data, never hallucinated.

-## Chat Architecture
-```
-Tier 0:   HA entity control (fuzzy match → call HA API)
-Tier 0.5: Network device management
-Tier 0.7: Planner intents (add task, my tasks, schedule, daily briefing)
-Tier 1:   KB intent engine (response type → instant reply)
-Tier 1b:  Action intents (network_scan → DB data + queue PVE1 scan)
-Tier 2:   Ollama llama3.2 at http://10.48.200.210:11434 (5s timeout)
-Tier 3:   Groq compound-beta-mini (cloud, fast)
-Tier 4:   Claude API fallback
-```
- Groq model name: `compound-beta-mini` (NOT `groq/compound-beta-mini`)
- HA_URL = `http://orbisne.fortiddns.com:8123`
-
-## DB Key Tables
+## Chat Architecture / DB
 ```sql
 registered_agents    — agent_id, hostname, agent_type, ip_address, api_key, status, last_seen, version
-agent_metrics        — agent_id, metric_type, metric_data(JSON), recorded_at
-  -- Extract: JSON_EXTRACT(metric_data,'$.cpu_percent'), JSON_EXTRACT(metric_data,'$.memory.percent')
-  -- NO cpu_pct/mem_pct columns — always use JSON_EXTRACT
+agent_metrics        — agent_id, metric_type, metric_data(JSON), recorded_at  -- use JSON_EXTRACT, no direct cpu/mem columns
 agent_commands       — agent_id, command_type, command_data(JSON), status(pending/delivered)
 network_devices      — ip, mac, hostname, alias, device_type, status, last_seen
 alerts               — alert_type, title, message, severity, resolved
-kb_facts             — category, fact_key, fact_value
+kb_facts             — category, fact_key, fact_value  -- fact_value is MEDIUMTEXT (was TEXT, HA entity-map JSON outgrew it and silently failed writes for days before this was caught)
 kb_intents           — intent_name, pattern(regex), response_template, action_type, priority, active
-api_cache            — cache_key(proxmox/news/weather/ha_entities), data(JSON), updated_at
+kb_generator_topics   — id, topic_id, category, topic_name, description, active, run_count, last_run_at
 tasks                — title, notes, category, priority, status, due_date, due_time, completed_at
 appointments         — title, description, category, start_at, end_at, location, all_day, reminder_min
-ha_entities          — real-time HA entity states (updated by HA custom component)
 ```

-## Proxmox API
- `stats_cache.php` uses cluster API at `orbisne.fortiddns.com:8006`
- Returns all VMs from both PVE1 and PVE2 via `/cluster/resources?type=vm`
- Token: `root@pam!jarvis=c45b5feb-f9a9-445d-a626-14fbb959f78b`
-
-## Cron Jobs (on VM 211)
-```
-*/3 * * * *  php /var/www/jarvis/api/endpoints/facts_collector.php
-*/5 * * * *  php /var/www/jarvis/api/endpoints/stats_cache.php
-0 7 * * *    /usr/local/bin/jarvis-backup.sh
-```
-**Cron on PVE1:**
-```
-*/3 * * * *  /usr/local/bin/jarvis-netscan.sh
-```
+## KB Intent Generator (self-expanding knowledge base)
+- Cron `0 */6 * * *`, processes 25 topics/run (BATCH_SIZE) cycling through `kb_generator_topics` table.
+- Topic count grew from 140 → 4530 across 170 categories (general knowledge, Fortune-500 companies, space/exploration, etc.) — at 25/run a full cycle now takes ~45 days; raise BATCH_SIZE if freshness matters.
+- Admin UI "MANAGE TOPICS" button — browse/add/edit/delete/toggle topics.
+- Groq model `llama-3.3-70b-versatile`, 20 intents/call, max_tokens=5000 (raising from 40/3500 fixed an 88% truncation failure rate).

 ## Backups
- Script: `/usr/local/bin/jarvis-backup.sh`
- Output: `/var/backups/jarvis/jarvis_backup_YYYY-MM-DD_HH-MM-SS.tar.gz`
- Daily at 7am UTC (2am CDT), 7-day retention
- Downloadable from admin portal → BACKUPS tab
+- Script: `/usr/local/bin/jarvis-backup.sh` → `/var/backups/jarvis/jarvis_backup_TIMESTAMP.tar.gz`, daily ~7am UTC, 7-day retention, downloadable from admin panel.
+- Also captures `/etc/jarvis-arc` (reactor.env), `/etc/jarvis` (db.env), service drop-ins, `/usr/local/bin/jarvis-*.sh`.
+- Health monitoring (`jarvis-health.sh`, PVE1 cron */5) checks stalled crons, stuck arc_jobs, disk >85%, watchdog restarts — auto-resolving `alerts` rows + email to `myronblair@gmail.com`.

 ## API Auth
 - Main JARVIS API: session token via `X-Session-Token` header (or PHP session)
- Agent endpoints: `X-Agent-Key` header (per-agent key from registered_agents.api_key)
+- Agent endpoints: per-agent key via header (see Agent System above)
 - Netscan endpoint: `X-Registration-Key` header (shared registration key)
- Admin portal: separate PHP session (`session_name('jarvis_admin')`)
- Cloudflare passes real client IP in `CF-Connecting-IP` header
+- Admin portal: separate PHP session name (`jarvis_admin`)
+- Cloudflare passes real client IP in `CF-Connecting-IP` header (or `HTTP_X_FORWARDED_FOR`)
+
+## Security posture (Phase 1 hardening, 2026-07-07 — mostly done, some user follow-up still open)
+- Secrets pulled out of git-tracked files into `/etc/jarvis-arc/reactor.env` and `/etc/jarvis/db.env`. **Still pending (user action): rotate Claude/Groq/Gmail-app-pw/iCloud-app-pw/DB-pass since they remain in GitHub history**, and do a `git filter-repo` + force-push history purge.
+- Login: Redis per-IP rate limit (10 fails/15min), session_regenerate_id on success.
+- `INFRASTRUCTURE-REFERENCE.md` is gitignored/untracked in the repo — lives only on-disk on VM211 (and its VM110 mirror copy), not in GitHub.
@@ -0,0 +1,20 @@
+# Misc Reference
+
+## Repo inventory notes
+- GitHub is source of truth; Gitea mirrors one-way (GitHub → Gitea only, never push to Gitea directly).
+- **Stale/legacy repos that predate current infra — don't treat as authoritative:** `mediastack` (MediaStack VM destroyed 2026-07-23, superseded by `kino-app`/StreamHoard), `kino` (an earlier/separate repo from the current `kino-app`), this `ai-context` repo itself was ~1 month stale before this rewrite.
+- `ai-context` (this repo) is the canonical "hand this to a new AI session" reference — keep it updated after any major infra change, the same way this file was updated 2026-07-24.
+
+## Assistant hard limits (what an AI assistant working this environment will/won't do)
+- Won't enter passwords/tokens into login forms or complete OAuth flows on the user's behalf.
+- Won't materialize a raw secret in a visible command/transcript even with permission — use env vars/files instead.
+- Destructive actions on shared/production infra need explicit, specific confirmation (a general "clean this up" isn't enough for an actual `rm`; pushing to a repo's default branch needs the branch named explicitly).
+
+## ZeroTraceGPT IQ (local AnythingLLM instance, not yet integrated into JARVIS)
+- Rebranded AnythingLLM instance on Myron's Windows workstation, base URL `http://127.0.0.1:7865`, API prefix `/api/v1/...`, Bearer token `9NPG3XG-32V47D6-PYET8BB-CDYYVRG`.
+- OpenAI-compatible endpoints (`/v1/openai/chat/completions`) once a model is loaded; RAG-backed workspace chat (`/v1/workspace/{slug}/chat`).
+- **Not yet wired into JARVIS's Arc Reactor AI tier cascade** — blocked because it's bound to `127.0.0.1` only (needs LAN or Tailscale exposure before VM211 can reach it). If asked to add it as a tier, follow the same pattern as the existing Ollama Tier 1.5 entry.
+- Separate from a lighter desktop app also called "ZeroTraceGPT" (GPT4All-based, different product, installed locally).
+
+## Payments consolidation (planned, not started)
+Move tomsjavajive.com off Stripe onto Square, to consolidate all payment-taking sites onto one processor (parkerslingshotrentals.com already uses Square). See sites.md for current per-site payment processor status.
@@ -3,59 +3,68 @@
 ## SSH Patterns

 ```bash
-# DigitalOcean (websites only — JARVIS no longer here)
+# DigitalOcean (websites only — JARVIS/FusionPBX no longer here)
 sshpass -p 'Gonewalk1974!@#' ssh -o StrictHostKeyChecking=no root@165.22.1.228 'cmd'

-# JARVIS VM (PVE1 VM 211)
-sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.211 'cmd'
-
-# PVE1 via FortiGate DDNS (survives IP changes)
+# PVE1 (hop node for everything else — password auth)
+sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.90 'cmd'
+# also reachable via FortiGate DDNS (survives WAN IP changes, which happen often):
 sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@orbisne.fortiddns.com 'cmd'

-# PVE1 direct (local network only)
-sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.90 'cmd'
+# Any VM behind PVE1 — nested hop through PVE1
+sshpass -p 'Joker1974!!!' ssh root@10.48.200.90 \
+  'ssh -o StrictHostKeyChecking=no root@<VM_IP> "cmd"'

-# PVE2 (local only)
+# PVE2 (local only, no external port-forward)
 sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.91 'cmd'

-# FusionPBX (must relay via DO — port 22 firewalled from internet)
-sshpass -p 'Gonewalk1974!@#' ssh -o StrictHostKeyChecking=no root@165.22.1.228 \
-  'sshpass -p "Joker1974!@#" ssh -o StrictHostKeyChecking=no root@134.209.72.226 "cmd"'
+# Run a command inside an LXC container (CT) via PVE1's pct
+sshpass -p 'Joker1974!!!' ssh root@10.48.200.90 'pct exec <CTID> -- bash -c "cmd"'

-# NovaCPX (direct — PVE1 hop broken)
-sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.110 'cmd'
+# Run a command inside a QEMU VM via guest agent (works even if network/SSH is down)
+sshpass -p 'Joker1974!!!' ssh root@10.48.200.90 'qm guest exec <VMID> -- bash -c "cmd"'
+# guest-exec has no HOME env by default — many CLI tools need: export HOME=/root; ...

-# MediaStack (SSH key auth)
-ssh -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa root@10.48.200.35 'cmd'
-
-# Local VMs (myron user + sudo, password Joker1974!)
+# Local VMs using myron user + sudo
 sshpass -p 'Joker1974!' ssh -o StrictHostKeyChecking=no myron@10.48.200.18 'sudo cmd'

-# Run command inside a Proxmox VM
-sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.90 \
-  'qm guest exec <VMID> -- bash -c "cmd"'
+# Synology NAS — needs password auth FORCED (pubkey silently preferred/fails otherwise)
+sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no -o PreferredAuthentications=password \
+  -o PubkeyAuthentication=no nas@10.48.200.249 'echo Joker1974!!! | sudo -S cmd'
+# scp to the NAS needs the legacy flag too: scp -O ...
 ```

-**Password order to try if first fails:** `Joker1974!@#` → `Joker1974!!!` → `Joker1974!`
+**Password order to try if first fails:** `Joker1974!!!` → `Joker1974!` → `Gonewalk1974!@#` (DO only).

-## Server Map
+**MediaStack-35's ad-hoc interactive SSH intermittently failed** (resolved as non-issue — the automated backup script's own key-based access always worked fine; root cause was never found, and it's moot now since that VM is destroyed). If a similarly-flaky interactive SSH pattern shows up on another host, don't assume it's a real access/security problem before checking whether an automated path using the same credentials still works.
+
+## Server Map (current)

 | Host | IP | User | Password | Purpose |
 |------|----|------|----------|---------|
-| DO | 165.22.1.228 | root | Gonewalk1974!@# | Websites (CyberPanel/OLS) |
-| JARVIS VM | 10.48.200.211 | root | Joker1974!!! | JARVIS AI — PVE1 VM 211 |
-| NPM | 10.48.200.200 | — | — | Nginx Proxy Manager (Docker) |
-| PVE1 | orbisne.fortiddns.com (10.48.200.90) | root | Joker1974!!! | Primary hypervisor |
+| DO | 165.22.1.228 | root | Gonewalk1974!@# | Websites (CyberPanel/OLS), 7 sites |
+| PVE1 | orbisne.fortiddns.com (10.48.200.90) | root | Joker1974!!! | Primary hypervisor, SSH hop for everything |
 | PVE2 | 10.48.200.91 | root | Joker1974!!! | Secondary hypervisor |
-| FusionPBX | 134.209.72.226 | root | Joker1974!@# | FreeSWITCH PBX |
-| NovaCPX | 10.48.200.110 | root | Joker1974!!! | Hosting panel — PVE1 VM 120 |
-| MediaStack | 10.48.200.35 | root | (SSH key) | Sonarr/Radarr/qBit — PVE1 VM 103 |
-| HomeAssistant | 10.48.200.97 | — | — | HA VM 109 (web terminal only) |
-| Homebridge | 10.48.200.18 | myron | Joker1974! | PVE1 VM 118 |
-| Jellyfin | 10.48.200.33 | myron | Joker1974! | PVE1 VM 112 |
-| Ollama | 10.48.200.210 | myron | Joker1974! | LLM VM 106, port 11434 |
+| JARVIS VM 211 | 10.48.200.211 | root | Joker1974!!! | JARVIS AI dashboard |
+| StreamHoard CT104 | 10.48.200.104 | root | Joker1974!!! (via `pct exec 104`) | Consolidated media server — see streamhoard.md |
+| NovaCPX VM120 | 10.48.200.110 | root | Joker1974!!! | Hosting panel + web.orbishosting.com dashboard |
+| FusionPBX VM130 | 10.48.200.130 | root | Joker1974!!! | FreeSWITCH PBX (on-prem now, not the old DO droplet) |
+| Ollama VM106 | 10.48.200.210 | root | Joker1974!!! | Local LLM (llama3.1:8b, llava:7b vision), port 11434 |
+| Home Assistant | 10.48.200.97 | — | — | HA VM 109/101 (web terminal only) |
+| Homebridge VM118 | 10.48.200.18 | myron | Joker1974! | HomeKit bridge |
 | NetworkBackup | 10.48.200.99 | myron | Joker1974! | PVE2 VM 302 |
-| WireGuard CT110 | 10.48.200.67 | — | — | VPN kill-switch for MediaStack |
+| NPM VM105 | 10.48.200.200 | — | — | Nginx Proxy Manager (Docker), admin `myronblair@outlook.com` |
+| Synology NAS | 10.48.200.249 | nas | Joker1974!!! | Media + MSP360 backup storage, DSM 7.3.2, Gitea |
+| FortiGate | 10.48.200.1 | admin | Joker1974!!! | Router/firewall — admin GUI on **:9443**, not 443 |
+
+**Destroyed 2026-07-23 — do not use these IPs/VMIDs for anything:**
+| Was | IP | VMID |
+|---|---|---|
+| Jellyfin | 10.48.200.33 | VM112 |
+| MediaStack | 10.48.200.35 | VM103 (also briefly `.136` mid-migration) |
+| WireGuard exit container | 10.48.200.67 / .19 | CT110 |
+
+`10.48.200.35` is currently squatted by an **unidentified rogue device** (SSH banner `OpenSSH_8.4p1`, exposes NFS shares named `TimeMachineBackup`/`Public`/`iTunes` — looks like an old Buffalo-brand NAS). Found during the migration, never tracked down/removed. Don't be surprised if that IP answers but isn't what you expect.

 ## Proxmox API
 ```bash
@@ -69,26 +78,45 @@ GET /api2/json/cluster/resources?type=vm

 ## Key Paths on JARVIS VM (10.48.200.211)
 ```
-/var/www/jarvis/              — JARVIS app root
+/var/www/jarvis/              — JARVIS app root, IS the live git repo (push from it directly)
 /var/www/jarvis/public_html/  — web root (index.html, api.php, admin/)
-/var/www/jarvis/api/          — backend (config.php, endpoints/, lib/)
+/var/www/jarvis/api/          — backend (config.php [gitignored], endpoints/, lib/)
 /var/www/jarvis/deploy/       — Arc Reactor source (reactor.py)
-/opt/jarvis-arc/reactor.py    — Arc Reactor daemon
+/opt/jarvis-arc/reactor.py    — Arc Reactor daemon (copy deploy/reactor.py here + restart to deploy)
 /var/backups/jarvis/          — daily backups (tar.gz, 7-day retention)
 /usr/local/bin/jarvis-backup.sh — backup script
+/var/www/jarvis/public_html/admin/downloads/INFRASTRUCTURE-REFERENCE.md — the master live infra doc (edit here first, gitignored, NOT in GitHub — relay-copy to VM110 through PVE1 after editing)
+```
+
+## Key paths on StreamHoard CT104 (10.48.200.104)
+See [streamhoard.md](streamhoard.md) for full detail.
+```
+/opt/kino/                 — docker-compose.yml, .env, app source (repo kino-app)
+/opt/streamhoard-config/   — migrated radarr/sonarr/prowlarr/qbittorrent config+library
+/mnt/nas/video/            — LXC mountpoint passthrough of the NAS media library (movies/tv)
+/mnt/nas-backups/          — LXC mountpoint passthrough for MSP360 destination
 ```

 ## Key Paths on DO (165.22.1.228)
 ```
-/home/tomsjavajive.com/public_html/       — Tom's Java Jive
-/home/epictravelexpeditions.com/public_html/ — Epic Travel
-/home/parkerslingshotrentals.com/public_html/ — Parker Slingshot
-/home/orbishosting.com/public_html/       — Orbis Hosting
-/home/orbis.orbishosting.com/public_html/ — Orbis Portal
-/home/tomtomgames.com/public_html/        — TomTom Games
+/home/tomsjavajive.com/public_html/
+/home/epictravelexpeditions.com/public_html/
+/home/parkerslingshotrentals.com/public_html/
+/home/orbishosting.com/public_html/       — NOT the real public apex site (see sites.md)
+/home/orbis.orbishosting.com/public_html/
+/home/tomtomgames.com/public_html/
+/home/worktracking.orbishosting.com/public_html/  — ChuckCo Time Keeper (repo: chucko)
 ```
+`.git` for all 7 sites lives OUTSIDE the web root at `/home/<site>/git-data` (a `gitdir:` pointer file sits in `public_html/.git`) — `git` commands from `public_html` work normally.

 ## PHP / OLS Notes (DO server)
 - Run scripts: `/usr/local/lsws/lsphp85/bin/lsphp /path/to/script.php`
 - Syntax check: `php8.3 -l file.php` (lsphp segfaults on -l)
 - Web server: OpenLiteSpeed (`lshttpd`), NOT apache/nginx
+- Cron bug: `*/3 * * * *` restarts `lshttpd` if any `.htaccess` is newer than `/usr/local/lsws/cgid` (a directory) — nothing re-touches that dir after restart, so editing `.htaccess` triggers a restart loop until you `touch /usr/local/lsws/cgid` manually.
+
+## GitHub / Gitea
+- GitHub is source of truth; **Gitea (`gitea.orbishosting.com`, on the NAS) is a one-way, read-only mirror** — never push directly to Gitea.
+- `gh` CLI at `C:\Program Files\GitHub CLI\gh.exe` (Windows workstation), auth via `GH_TOKEN` env var, PAT scopes `repo`+`workflow` (no `read:org`, so use `GH_TOKEN` directly or `gh api`, not `gh auth login --with-token`).
+- After any GitHub push, trigger the mirror manually: `curl -X POST -u myron:Joker1974!!! https://gitea.orbishosting.com/api/v1/repos/myron/<repo>/mirror-sync`
+- 5 repos are Gitea-only by design, never push to GitHub: `fortigate-config`, `infra-private`, `jarvis-secrets`, `msp360-config`, `proxmox-secrets` — these hold real credentials deliberately kept off a third-party cloud host.
@@ -1,5 +1,7 @@
 # Websites & Sites Reference Card

+FusionPBX has moved to its own file — see [voip.md](voip.md). It is no longer on the DO server.
+
 ## All Sites on DO (165.22.1.228)

 | Site | Path | GitHub Repo | DB |
@@ -8,35 +10,39 @@
 | epictravelexpeditions.com | /home/epictravelexpeditions.com/public_html/ | myronblair/epictravelexpeditions | epic_epic_db |
 | parkerslingshot (subdomain) | /home/epictravelexpeditions.com/parkerslingshot/ | myronblair/parkerslingshot | epic_parkersling |
 | parkerslingshotrentals.com | /home/parkerslingshotrentals.com/public_html/ | myronblair/parkerslingshotrentals | parker_db |
-| orbishosting.com | /home/orbishosting.com/public_html/ | myronblair/orbishosting | — |
-| orbis.orbishosting.com | /home/orbis.orbishosting.com/public_html/ | myronblair/orbis-hosting-portal | — |
 | tomtomgames.com | /home/tomtomgames.com/public_html/ | myronblair/tomtomgames | tomt_ttg_db |
+| worktracking.orbishosting.com | /home/worktracking.orbishosting.com/public_html/ | myronblair/chucko | workt_track_db |
+| orbishosting.com (apex) | /home/orbishosting.com/public_html/ | myronblair/orbishosting | — |
+| orbis.orbishosting.com | /home/orbis.orbishosting.com/public_html/ | myronblair/orbis-hosting-portal | — |
+
+**orbishosting.com (apex) and orbis.orbishosting.com are NOT the real live sites — don't audit/fix them expecting user traffic.** The apex's `/home/orbishosting.com` on this DO droplet is just the CyberPanel parent-domain container (its own public_html is SuiteCRM legacy + a static placeholder) — the *real* public apex orbishosting.com is a WHMCS billing portal hosted somewhere else entirely (location unknown). `orbis.orbishosting.com` is a confirmed-intentional pre-launch placeholder "sign in" page. The two subdomains that actually matter are **web.orbishosting.com** (MSP dashboard, on VM110 in the home-lab, not DO — see servers.md) and **jarvis.orbishosting.com** (JARVIS, VM211, not DO).
+
+## ChuckCo Time Keeper (worktracking.orbishosting.com)
+Work-tracking app, 5-day Fri–Thu week, flat day-rate pay, token-based access (no login for workers/payer, single shared admin password). DB `workt_track_db` / `workt_track_user` / `ZWCNMRP2N5NVPsghmve5aRS9`. Admin password `Joker1974!!!` at `/admin/login.php`.

 ## JARVIS (on PVE1 VM 211, NOT DO)
 | Site | Path | GitHub Repo | DB |
 |------|------|-------------|-----|
 | jarvis.orbishosting.com | /var/www/jarvis/ on 10.48.200.211 | myronblair/jarvis | jarvis_db |
+See [jarvis.md](jarvis.md) for full detail.

-## Deploy Workflow
+## web.orbishosting.com (MSP dashboard, NOT on DO)
+On VM110 (NovaCPX, 10.48.200.110), `/home/webacct/public_html/index.html`, repo `myronblair/web-dashboard`, branch `main`. `web-dashboard-deploy.sh` (cron */1min on VM110) pulls when a webhook queues the path in `/tmp/web-dashboard-deploy.txt`. Currently links out to StreamHoard/Sonarr/Radarr/Prowlarr/qBittorrent/Portainer (all on CT104, 10.48.200.104) — the old Jellyfin/MediaStack links were removed 2026-07-23 when those VMs were destroyed.
+
+## Deploy Workflow (DO sites)
 ```bash
 # 1. Edit files on server via SSH (for quick hotfixes)
 sshpass -p 'Gonewalk1974!@#' ssh -o StrictHostKeyChecking=no root@165.22.1.228 \
  'nano /home/site.com/public_html/file.php'

-# 2. ALWAYS commit + push to GitHub
+# 2. ALWAYS commit + push to GitHub — deploy for these 7 sites is MANUAL
 cd /home/site.com && git add -A && git commit -m "message" && git push
-
-# Auto-deploy triggers within 1 minute via webhook
-# Webhook handler: tomtomgames.com/webhook.php (DO sites)
-# JARVIS webhook: port 1972 on VM 211
-
-# For immediate deploy (skip 1-min wait):
-sshpass -p 'Gonewalk1974!@#' scp -o StrictHostKeyChecking=no \
-  /tmp/changed.php root@165.22.1.228:/home/site.com/public_html/changed.php
+# No webhook/cron auto-pull exists for these DO sites (unlike JARVIS/web-dashboard/StreamHoard,
+# which do have auto-deploy). .git lives outside the web root at /home/<site>/git-data.
 ```

 ## GitHub PAT
-`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` — expires ~2026-08-20
+Current: `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` (rotated 2026-07-05 after the previous one, `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`, was found exposed in `.git/config` on all 6 DO sites). This is also the Windows workstation's `GH_TOKEN` env var.

 ## Database Credentials

@@ -47,22 +53,19 @@ sshpass -p 'Gonewalk1974!@#' scp -o StrictHostKeyChecking=no \
 | epic_epic_db | epic_epic | (check config.php) |
 | epic_parkersling | epic_parkersling | Joker1974!!! |
 | parker_db | parker_db | (check config.php) |
+| workt_track_db | workt_track_user | ZWCNMRP2N5NVPsghmve5aRS9 |
 | MySQL root (DO) | root | b71e5c1a8c7457541b9c1db822de37adfa271926a38b6c20 |

-```bash
-# Quick DB access on DO
-sshpass -p 'Gonewalk1974!@#' ssh -o StrictHostKeyChecking=no root@165.22.1.228 \
-  'mysql -u toms_tjj_user -p"+60wlPc+55e@gFq4" toms_tjj_db -e "SELECT ..."'
-
-# Quick DB access on JARVIS VM
-sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.211 \
-  'mysql -u jarvis_user -pJ4rv1s_Pr0t0c0l_2026! jarvis_db -e "SELECT ..."'
-```
-
 ## Gitignored Credentials (never in GitHub)
 - `api/config.php` — JARVIS, epictravelexpeditions
 - `config/database.php` — tomsjavajive
 - `db.php`, `config.php` — parkerslingshot
+- `includes/config.php` — chucko/worktracking (DB creds, admin password hash, site-wide token)
+
+## Payments — status and pending migration
+- **tomsjavajive.com still uses Stripe** — a live Stripe secret/publishable/webhook key was found hardcoded in `config/config.php`, moved to `/home/tomsjavajive.com/config-secrets.php` (still in git history, needs rotation via Stripe dashboard — user action, not yet done).
+- **Planned future task (not started):** move tomsjavajive.com off Stripe onto **Square**, to consolidate all 3 payment-taking sites (tomsjavajive, parkerslingshotrentals, and presumably a third) onto one processor.
+- **parkerslingshotrentals.com uses Square** — access token rotated 2026-07-05: `EAAAl_Cj6NVggy_-8JAFzEPcTHQMv0cY27Sy_uwJt-2AiFVq5Jz-42QzyIIMcVhY`, live in `db-secrets.php`.

 ## Tom's Java Jive Quirks
 - No `slug` column on products — URLs use `?id=product_id`
@@ -76,14 +79,6 @@ sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.211 \
 - Login: `admin / Parker2026!`

 ## Email / SMTP
- All sites: CyberMail (CyberPersons) — API key `sk_live_7f9b0f9a29f6de31a0d229d4af75d56b094ad724fc58a57d`
+- All sites use CyberMail (CyberPersons) — current key `sk_live_df1965fc93751165d0bdd99a6eb0b46ca793ab9e44129aae` (rotated 2026-07-05, shared across tomsjavajive/parkerslingshotrentals/tomtomgames — they're all on the same CyberMail account).
 - Manage at: platform.cyberpersons.com
- DO blocks SMTP port 587 — use CyberMail API over HTTPS (port 443)
-
-## FusionPBX
- URL: https://fusion.orbishosting.com (admin / fY7XP5swgtpbzrYLhkeVYkA4744)
- Server: 134.209.72.226 (SSH: relay via DO → root / Joker1974!@#)
- Timezone: America/Chicago
- SIP profiles via Lua XML handler — config changes need cache delete:
-  `rm /var/cache/fusionpbx/FusionPBX.configuration.sofia.conf`
- Ext 1000 (Myron T48S), Ext 1001 (Tommy T48S), Ext 1002 (Myron AX86R), Ext 1003 (Kitchen T57W), Ext 1004 (Bedroom T57W)
+- DO blocks SMTP port 587 — use the CyberMail API over HTTPS (port 443) instead.
@@ -0,0 +1,60 @@
+# StreamHoard Reference Card (CT104, 10.48.200.104)
+
+**Replaces the old Jellyfin (VM112), MediaStack (VM103), and WireGuard exit container (CT110) — all three destroyed 2026-07-23.** Everything media-related now lives on one Debian 12 LXC (CT104, 8 cores/12GB RAM/60GB disk) on PVE1.
+
+## What it is
+A self-hosted Netflix-style media app (repo `myronblair/kino-app`, private) branded **"StreamHoard"** in the actual UI — internal names (container names, some file paths) still literally say `kino`/`Kino`, that's cosmetic leftover from the original scaffold, not a second app. FastAPI + React + MongoDB.
+
+Deployed via Docker Compose in `/opt/kino` on CT104. Login: `admin@streamhoard.local` — password was rotated by the user via the app's own self-service "Change Password" feature (Settings page, requires current password) — not stored in this doc.
+
+## Container stack
+| Container | Role | Port (host) |
+|---|---|---|
+| `streamhoard-frontend` | React UI, reverse-proxies `/api` to backend | 8080 |
+| `streamhoard-backend` | FastAPI — auth, catalog, streaming, transcode queue | internal 8001 |
+| `streamhoard-mongo` | MongoDB **pinned to 4.4** | internal |
+| `streamhoard-gluetun` | VPN gateway (NordVPN/WireGuard) for the whole *arr stack + qBittorrent | proxies 7878/8989/9696/8082 |
+| `streamhoard-radarr` / `-sonarr` / `-prowlarr` / `-qbittorrent` | Migrated **with existing config/library/history** from the old MediaStack VM — `network_mode: service:gluetun` so ALL their traffic is VPN'd, not just qBittorrent's | via gluetun |
+| `streamhoard-portainer` | Docker management GUI | 9443 (https) |
+
+**Why mongo:4.4:** PVE1's physical CPU lacks AVX, and mongo:7+ hard-requires it (SIGILL crash-loop without this pin).
+
+## Media path
+- NAS (`10.48.200.249:/volume1/video`) is mounted **on PVE1 itself** (`/mnt/nas-video`, `/etc/fstab`, `_netdev` — persists across PVE1 reboots), then passed into CT104 as an **LXC mountpoint** (`mp0`, config lives in `/etc/pve/lxc/104.conf` — persists across CT104 reboots too).
+- **Why not mount NFS/CIFS directly inside CT104:** unprivileged LXCs can't reliably do NFS *or* CIFS client mounts even with the `mount=nfs;cifs` feature flag enabled — confirmed both fail with "Operation not permitted." This is a hard Proxmox/kernel-capability limitation, don't waste time retrying it — always go through the PVE1-host-mount + LXC-mountpoint pattern instead.
+- Radarr/Sonarr root folders: `/mnt/nas/video/movies`, `/mnt/nas/video/tv` (unchanged from the old MediaStack setup). StreamHoard's backend container bind-mounts the same NAS path **read-only at the identical absolute path** (`/mnt/nas/video:/mnt/nas/video:ro`) so Radarr/Sonarr-reported `file_path` values resolve with zero translation.
+- StreamHoard's own Radarr/Sonarr integration settings point at `http://gluetun:7878` / `http://gluetun:8989` (internal Docker network — gluetun's container shares its netns with radarr/sonarr, so other containers reach them via gluetun's hostname).
+- NAS export ACL (`/etc/exports` on the NAS) now only allows **PVE1 (10.48.200.90)** — cleaned up the old MediaStack-only entry after that VM was destroyed.
+
+## Transcoding
+Almost the entire imported library needed re-encoding — most source files are HEVC/H.265 video and/or DTS/EAC3 audio, neither browser-native. This is what causes playback "jitter"/silent-audio, **not** a network/NAS bottleneck (NAS read tested at 122 MB/s). Fix:
+- Queued ABR transcode (re-encode to H.264/AAC multi-bitrate HLS) for the whole existing library via the transcode-queue API.
+- Fixed a real code gap: `sonarr_import` was missing the auto-transcode-on-import hook that `radarr_import` already had (both now call `_enqueue_transcode` when the global `auto_transcode` setting is on). That setting is now globally `"abr"`, so future imports auto-queue transcoding with no manual step.
+- If a newly-imported title plays choppy or silent, check its codec first (`ffprobe` inside `streamhoard-backend`) before assuming a server problem.
+
+## MSP360 backup (weekly full / daily incremental to the NAS)
+- Installed via `dpkg-repack` from the working NovaCPX install (no public download URL for the Linux .deb worked — MSP360/CloudBerry doesn't publish a stable direct link).
+- **Real gap hit and fixed:** the package's config templates (`cloudBackup.conf`, `wt_config.xml`) are consumed/deleted from disk after a machine's first run, so `dpkg-repack`ing an already-initialized install captures a package that can never bootstrap its own runtime state (`/opt/local/Online Backup/<machine-guid>/config/...`) — the daemon just retries forever waiting for a file that will never appear. Fix: copy the *generated* versions of those two files from the source machine's live runtime directory (not the package) into the new install's equivalent path before first start.
+- Storage account: File System type, path `/mnt/nas-backups/MSPBackups` (same PVE1-mount + LXC-mountpoint pattern as the media NAS mount, separate mountpoint `mp1`).
+- Free license, self-registered via CLI: `cbbV2 license activate -f -e <email> -c`.
+- Plan **StreamHoard-Backup**: daily incremental 2:00 AM, weekly full every Monday 2:00 AM, 4-week retention. Excludes: the huge media NAS mount, the backup destination itself, ephemeral Docker image layers (`overlay2`/`containers`/`image`/`buildkit`), and the regenerable HLS transcode cache (`/opt/kino/media/hls`, several GB, not "configuration").
+- **A leftover worker process from a killed/deleted plan will silently block a new plan's worker** (near-zero CPU usage despite the job showing "Running" for many minutes is the tell) — `ps aux | grep cbbWorker` and kill any process referencing an old/deleted plan ID before assuming a new plan is broken.
+- CLI binary: `/opt/local/MSP360\ Backup/bin/cbb` (legacy, `plan -l`/`-r`/`-s -f`) and `cbbV2` (full plan create/edit/account management) — the space in the path needs quoting in every invocation.
+
+## Portainer
+- `https://10.48.200.104:9443` — first-run setup token appears in `docker logs streamhoard-portainer`, and expires after ~5 minutes; restart the container to get a fresh one if it lapses.
+- Endpoint renamed "StreamHoard" for clarity; local Docker socket auto-detected, no extra config needed.
+
+## JARVIS agent
+Installed and registered (hostname `streamhoard104`, `watch_services: ["docker"]`) — see jarvis.md Agent System section. Watch out for the container's actual OS hostname needing to match what you want reported (a typo here caused a duplicate stale agent registration once).
+
+## Incidents hit during the 2026-07-23 migration (useful context if déjà vu happens elsewhere)
+- **Live duplicate-IP conflict**: an unidentified device (SSH banner `OpenSSH_8.4p1`, NFS shares `TimeMachineBackup`/`Public`/`iTunes` — looks like an old Buffalo-brand NAS) was actively answering ARP for `10.48.200.35` alongside the real MediaStack VM's MAC. Not stale cache — a live second device on the wire, never identified/removed. Worked around by moving MediaStack to a different IP mid-migration (moot now, VM destroyed) — `.35` is still squatted, avoid it.
+- That IP-reassignment **silently broke MediaStack's NordVPN kill-switch** (the `fwmark → routing table` `ip rule` entry vanished after the netplan change) — traffic briefly went out unprotected. Lesson: any NIC/netplan change on a host running NordVPN's Linux CLI app should be followed by `ip rule show`, not just `nordvpn status` — status can say "Connected" while the actual policy routing is gone. Fix: `nordvpn disconnect && nordvpn connect` reasserts the rule.
+- The NAS's own NFS export root (`/volume1/video` itself) has mode `000` due to Synology's `hide` export option — real root (PVE1) bypasses this via DAC-override, an unprivileged LXC's "root" cannot, causing `Permission denied` even though subfolders were readable. Fixed with `chmod 755` on the mount root + `chmod 777` on `movies`/`tv`/`downloads` specifically (also sidesteps a UID mismatch between the NAS's real file ownership and whatever PUID/PGID the new containers use).
+- gluetun needs `/dev/net/tun` explicitly passed into the LXC (`pct set 104 -dev0 /dev/net/tun`) or it fails immediately trying to create the WireGuard interface.
+
+## Not yet done / open items
+- FortiGate VIP `tunnel-51820 / Wireguard-CT110` — **already cleaned up** (removed 2026-07-24 along with its dead firewall policy, confirmed Ref:0 before deletion).
+- The rogue device on `10.48.200.35` was never identified or removed from the network.
+- No reverse proxy/TLS in front of StreamHoard yet, and no LAN-only firewall rule (the app itself has no built-in rate limiting).
@@ -0,0 +1,74 @@
+# VoIP Reference Card — FusionPBX / SignalWire / Landline
+
+## FusionPBX server
+- **Host:** VM130 on PVE1, `10.48.200.130` — moved on-prem from an old DO droplet (`134.209.72.226`, now **destroyed** — do not reference that IP anymore, any doc saying FusionPBX is at `134.209.72.226` is stale).
+- **Web UI:** `https://fusion.orbishosting.com` (NPM proxies to `10.48.200.130:80`)
+- **SSH:** `root@10.48.200.130` via PVE1 hop, password `Joker1974!!!` (no longer needs the old DO-relay hop)
+- **DB:** PostgreSQL (not MySQL), `fusionpbx` / `fusionpbx` / `RSfAjzefyYmufVqtnrYpz90yHaQ`, `psql -U fusionpbx -h 127.0.0.1 -d fusionpbx`
+- Timezone: America/Chicago
+
+## SIP registration settings (for a device/softphone registering against this PBX)
+| Setting | Value |
+|---|---|
+| SIP Server/Domain | `fusion.orbishosting.com` (or `10.48.200.130`) — **must be the domain name for some devices** (see FXO gotcha below), not always interchangeable with the IP |
+| Port (LAN) | `5060` UDP/TCP |
+| Port (WAN, per FortiGate forwards) | `5080` UDP or `5081` TCP/TLS |
+| Auth username | extension number, e.g. `1000` |
+| Auth password | per-extension, from `v_extensions.password` in the FusionPBX DB |
+
+Example working extensions (query `select extension, password, effective_caller_id_name from v_extensions;` for current full list):
+| Ext | Who |
+|---|---|
+| 1000 | Myron Blair (Desk) |
+| 1001 | Myron Blair |
+| 1002 | Tommy Ivy |
+| 1003 | Kitchen |
+| 1004 | Master Bedroom |
+| 1050 | Landline FXO gateway (Grandstream ATA, see below) |
+| 1091 | "Landline" ring group (rings all 8 phones) |
+
+**Extension key/BLF provisioning:** category differs by phone model — T57W/T48S use `category=line` in `v_device_keys`, the AX86R WiFi phone only has a `programmable` keys section (no `line` section). Guessing wrong silently produces no key at all — check the actual Smarty templates at `/var/www/fusionpbx/resources/templates/provision/yealink/<model>/*.cfg` before assuming a key type.
+
+## SignalWire outbound trunk (carrier side — separate from the extension/phone side above)
+FusionPBX registers **to** SignalWire as a SIP gateway (this is the outbound carrier trunk, not something a phone registers against):
+- Gateway UUID: `c0e12aab-2406-44ee-ac38-6602116eaf2f`, username `orbis`
+- Realm: `orbis-hosting-0364f5f67488.sip.signalwire.com`
+- Space: `orbis-hosting.signalwire.com`, Project ID `16fbbcfd-1c94-4827-869c-0364f5f67488`
+- Check status: `fs_cli -x "sofia status"` on VM130 — should show `REGED` for `external::c0e12aab-...`
+
+### SignalWire API gotchas (cost real debugging time — read before touching SignalWire)
+1. **Credential types are NOT interchangeable.** The dashboard shows a `PSK_`-prefixed Signing Key at the top of the API Credentials page that looks like "the" token but gives a real 401 on classic REST calls. A `pat_`-prefixed Personal Access Token also 401s. **Only a named Project Token (`PT`-prefix, created in a separate table further down the same page) works** for REST Basic Auth (`ProjectID:PT...`) and the Relay SDK.
+2. **Setting the SIP registration password requires the `relay` API, not the `Fabric` API.** `PUT /api/fabric/resources/sip_endpoints/{fabric_id}` returns HTTP 200 but does NOT affect registration auth. The correct call is `PUT https://{space}/api/relay/rest/endpoints/sip/{inner_sip_endpoint_id}` with `{"password":"..."}` — the inner sip_endpoint id is a *different* id than the Fabric resource id that wraps it. Symptom of using the wrong one: gateway stuck `FAIL_WAIT` with `[904] Operation has no matching challenge` even though the wire-level 401 challenge is valid.
+3. **Outbound INVITE `From` header user-part must equal the digest auth username** (`orbis`), not the caller's DID/Caller ID. In FusionPBX this is `v_gateways.caller_id_in_from` — must be `false`. If wrong: INVITE gets 407, retries with Proxy-Authorization, gets ANOTHER 407 with the same nonce (looks like a FreeSWITCH bug, isn't).
+4. **Destination must be full 11-digit E.164** (`+1` + 10 digits) — a dialplan bridge producing `+8172662022` (missing the `1`) gets `404`/`NO_ROUTE_DESTINATION`, easily misdiagnosed as "SignalWire doesn't recognize this number."
+5. **Trial-tier gotcha:** even with the right token, sending SMS to an unverified number returns `422 integration_test_verified_caller_required`.
+6. **FreeSWITCH `fs_cli chat` to a registered phone** needs the destination prefixed with the Sofia **profile name**: `chat sip|<from>@<domain>|internal/<ext>@<domain>|<body>` — just `user@domain` silently fails ("Invalid chat interface").
+7. **Diagnostic method that actually works for SIP auth issues:** independently reconstruct the digest (MD5 HA1/HA2/response) in a standalone script using the exact nonce/realm/uri from a live packet capture, compare byte-for-byte to what FreeSWITCH sent. Proves credentials/computation bug (mismatch) vs. carrier-side policy rejection (match, still rejected) before chasing further theories.
+
+## FortiGate SIP/RTP port forwards (VIPs, verify live via `show firewall vip` — the doc's static tables drift)
+```
+fusion-sip-5080 (udp), fusion-sip-5080-tcp, fusion-sip-5081-tcp, fusion-sip-5081-udp, fusion-rtp (udp 16384-32768)
+  → all forward to 10.48.200.130 (FusionPBX)
+```
+**WAN IP drifts silently and semi-regularly** — when it does, all VIP `extip` fields need updating, or calls will connect at the signaling level but have one-way/no audio (RTP arriving at the *real* current WAN IP with no matching forward rule). Always `curl ifconfig.me` from any on-prem host to check the *actual* live WAN IP before trusting a documented value.
+
+**FortiGate SIP ALG** (`default-voip-alg-mode`) must stay `proxy-based` (the default) — switching to `kernel-helper-based` breaks inbound call routing entirely (the inbound dialplan's `${sip_to_user}` handling is tuned around ALG being active).
+
+**FusionPBX's sofia.conf is cached independently of FreeSWITCH's process** at `/var/cache/fusionpbx/<hostname>.configuration.sofia.conf` (Lua file cache) — any sofia profile/gateway DB or vars.xml change needs this file cleared too, not just a service/process restart.
+
+## Landline/FXO integration (Grandstream ATA, working as of 2026-07-18)
+Old Flyingvoice FTA5111 (`10.48.200.119`) is **retired — do not reuse without a firmware fix from the vendor.** It has a 100%-reproducible crash-on-save bug (firmware V3.20) confirmed via extensive testing; not a config mistake.
+
+**Current device:** Grandstream HT8xx-family ATA at `10.48.200.45` (admin/Joker1974!!!), registered as **extension 1050**.
+
+FXO config (`/cgi-bin/config_a2`, Advanced Settings → FXO Port):
+- Primary SIP Server = `fusion.orbishosting.com` — **must be the domain, not the IP** (extension 1050 is only provisioned under that domain name in FreeSWITCH's directory, even though `10.48.200.130` also technically exists as a domain).
+- SIP User ID/Auth ID = `1050`, password = ext 1050's FusionPBX password.
+- `PSTN Ring Thru FXS` = **No** (device defaults to Yes — rings a nonexistent local FXS phone instead of routing to SIP).
+- Stage Method = default **2** (2-stage dialing) — gives the "landline button" real dial tone (SIP call seizes the line, ATA plays back real dial tone, DTMF passthrough).
+
+**Inbound PSTN→VoIP routing is on a DIFFERENT page** — Basic Settings (`/cgi-bin/config2`), NOT the FXO Port page:
+- `Unconditional Call Forward to VOIP`: User ID = `1091` (the "Landline" ring group, rings all 8 phones), Sip Server = `fusion.orbishosting.com`, port `5060`.
+- Without this field, incoming landline calls have nowhere to go once FXS-thru is disabled — easy to miss since the FXO page itself has no inbound-destination field.
+
+**Grandstream ATA web quirk:** login is `POST /cgi-bin/dologin`, but every subsequent settings-changing `POST /cgi-bin/update` needs a `session_token` hidden field (separate from login's `csrf_token`) scraped fresh from whatever config page you just loaded — omitting it silently bounces to a re-login page with the submission discarded, no error shown. Partial POSTs (only the fields you want changed) work fine.