From 4fad312f87eb1bac85247bb942de31848555c49a Mon Sep 17 00:00:00 2001 From: Myron Blair Date: Sat, 25 Jul 2026 22:47:31 -0500 Subject: [PATCH] =?UTF-8?q?Full=20rewrite=20for=20current=20infra=20state?= =?UTF-8?q?=20=E2=80=94=20StreamHoard=20consolidation,=20VoIP=20split=20ou?= =?UTF-8?q?t,=20dead=20references=20purged?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace all Jellyfin/MediaStack/WireGuard-CT110 references (destroyed 2026-07-23) with StreamHoard/CT104 throughout; add streamhoard.md covering the full consolidated stack, MSP360 backup setup, and migration incidents - Split FusionPBX/SignalWire/landline content into its own voip.md (was scattered across jarvis.md and sites.md, and referenced the old destroyed DO droplet IP) - Fix stale facts: JARVIS webhook port (:1972 -> 80/https), rotated registration key/GitHub PAT, FortiGate admin port (9443 not 443), current WAN-IP-drift/VIP gotchas - Add reference.md for repo inventory notes, assistant hard limits, ZeroTraceGPT IQ, payments migration status - Update gotchas.md with unprivileged-LXC mount limitations and MSP360 dpkg-repack gotcha --- gotchas.md | 84 ++++++++++++------------- index.md | 53 +++++++++------- jarvis.md | 163 ++++++++++++++++++++++--------------------------- reference.md | 20 ++++++ servers.md | 114 +++++++++++++++++++++------------- sites.md | 61 +++++++++--------- streamhoard.md | 60 ++++++++++++++++++ voip.md | 74 ++++++++++++++++++++++ 8 files changed, 398 insertions(+), 231 deletions(-) create mode 100644 reference.md create mode 100644 streamhoard.md create mode 100644 voip.md diff --git a/gotchas.md b/gotchas.md index e5f727d..fa39a1c 100644 --- a/gotchas.md +++ b/gotchas.md @@ -1,63 +1,65 @@ # Critical Gotchas — Read Before Running Commands -## PHP / OLS +## Unprivileged LXC containers (Proxmox) — mount limitations +- **NFS and CIFS client mounts both fail** inside an unprivileged LXC even with the `mount=nfs;cifs` features flag set — "Operation not permitted." This is a hard kernel-capability limitation, not a config mistake; don't retry it. +- **Fix:** mount the share on the PVE host itself (fstab, `_netdev`), then pass it into the container as an **LXC mountpoint** (`pct set -mpN /host/path,mp=/container/path`). This persists across reboots on both the host and the container. +- A network share's root export directory can have restrictive permissions (e.g. Synology's `hide` NFS option sets mode `000` on the share root) that real root on the PVE host silently bypasses via DAC-override, but an unprivileged container's "root" cannot — causes `Permission denied` even though subfolders are readable. Fix on the source NAS/host, not in the container. +- Passing a TUN device (needed for WireGuard-based VPN containers like gluetun) needs explicit `pct set -devN /dev/net/tun`. + +## MSP360 / CloudBerry Backup +- CLI binary path has a space: `/opt/local/MSP360\ Backup/bin/cbb` (legacy CLI, `plan -l/-r/-s`) or `cbbV2` (full plan/account create/edit) — always quote it. +- **`dpkg-repack`ing an already-initialized install captures a package missing its own bootstrap config templates** (`cloudBackup.conf`, `wt_config.xml` get consumed/deleted from the package tree after first run). A fresh install from that repacked .deb hangs forever waiting on a file that will never appear. Fix: copy the *generated* versions of those files from the source machine's live runtime directory into the new install before first start. +- **A leftover worker process from a killed/deleted plan silently blocks a new plan's worker** — the tell is near-zero CPU usage despite the job showing "Running" for many minutes. `ps aux | grep cbbWorker`, kill anything referencing an old/deleted plan ID. +- Free/Freeware license self-registers via `cbbV2 license activate -f -e -c` — no manual key needed. + +## PHP / OpenLiteSpeed (DO server) - **NEVER** use `lsphp -l` for syntax check — it segfaults. Use `php8.3 -l file.php` - Run CLI scripts with: `/usr/local/lsws/lsphp85/bin/lsphp /path/script.php` - If endpoint uses `ob_start()` + header.php pattern → add `ob_end_clean()` before CSV/JSON output - MySQL charset: always `utf8mb4_unicode_ci` — mixing with `general_ci` breaks JOINs (error 1267) +- Cron bug: `*/3 * * * *` restarts `lshttpd` if any `.htaccess` is newer than `/usr/local/lsws/cgid` — nothing re-touches that dir after restart, causing a restart loop until you `touch /usr/local/lsws/cgid` manually. ## SSH / Networking -- DO server (165.22.1.228) **cannot reach local network** (10.48.200.x) directly -- To reach local VMs from DO: use agent commands (shell type) or FortiGate DDNS -- PVE1 SSH via DDNS works: `root@orbisne.fortiddns.com` (Joker1974!!!) -- PVE2 has no external port forward — only reachable locally or via cluster API through PVE1 -- Proxmox API port 8006 IS forwarded: `orbisne.fortiddns.com:8006` works from DO +- DO server (165.22.1.228) **cannot reach the local network** (10.48.200.x) directly. +- PVE1 SSH via FortiGate DDNS works: `root@orbisne.fortiddns.com` (survives WAN IP changes, which happen fairly often — check `curl ifconfig.me` before trusting a documented WAN IP). +- PVE2 has no external port forward — only reachable locally or via cluster API through PVE1. +- Proxmox API port 8006 IS forwarded: `orbisne.fortiddns.com:8006`. +- FortiGate admin GUI is on **port 9443**, not 443 (443/80 both redirect there). +- A host's SSH interactive access can be intermittently flaky (e.g. through nested nested plink→ssh→ssh hops) while its own automated scripts using the same key work fine every time — that's usually not a real security/access problem, just a quirk of the ad-hoc path. ## JARVIS Agents - Agent config: `/etc/jarvis-agent/config.json` | Runtime state: `/var/lib/jarvis-agent/state.json` -- **401 "Invalid agent key"** → state.json has stale key. Fix: overwrite state.json with correct agent_id + api_key from `registered_agents` table, then `systemctl restart jarvis-agent` -- Agent heartbeat uses `X-Agent-Key` header (NOT body field) -- `shell` command type requires `{"command":"...","allowed":true}` in command_data -- Metrics stored as JSON in `metric_data` column — use `JSON_EXTRACT(metric_data,'$.cpu_percent')` NOT direct columns +- **401 "Invalid agent key"** → state.json has a stale key. Fix: overwrite with correct `agent_id`/`api_key` from `registered_agents` table, then `systemctl restart jarvis-agent`. +- Metrics stored as JSON in `metric_data` column — use `JSON_EXTRACT(metric_data,'$.cpu_percent')`, no direct columns. +- **Set the container/VM's actual OS hostname correctly before first agent registration** — a hostname typo becomes part of the generated `agent_id` and creates a stale duplicate registration that needs manual DB cleanup later. ## Groq AI -- Model name: `compound-beta-mini` — NOT `groq/compound-beta-mini` (that's OpenAI router syntax, 404s) +- Model name: `compound-beta-mini` — NOT `groq/compound-beta-mini` (that's OpenAI router syntax, 404s). ## Proxmox -- stats_cache.php uses `orbisne.fortiddns.com:8006` NOT `PROXMOX_HOST` (local IP unreachable from DO) -- `--nameserver` in Proxmox must be space-separated: `"8.8.8.8 1.1.1.1"` (comma causes netplan bug) -- Run commands in VMs: `qm guest exec -- bash -c 'cmd'` (requires guest agent installed) +- `--nameserver` must be space-separated: `"8.8.8.8 1.1.1.1"` (comma causes a netplan bug). +- Run commands in a VM even when SSH/network is down: `qm guest exec -- bash -c 'cmd'` (needs guest agent; also needs `export HOME=/root` for many CLI tools since guest-exec has no HOME env by default). +- Run commands in an LXC: `pct exec -- bash -c 'cmd'`. ## Deploy -- Always `git add + commit + push` after editing files on server — webhook auto-deploys within 1 min -- PHP syntax validated before deploy — bad commits auto-reverted -- LSAPI session deadlock: `session_write_close()` must be called in api.php after auth check +- Repos with auto-deploy webhooks (JARVIS, web-dashboard, StreamHoard's compose isn't webhook-deployed — it's manual `git pull` + `docker compose up -d`): always `git add + commit + push` — don't forget to also trigger the Gitea mirror-sync afterward (`curl -X POST -u myron:Joker1974!!! https://gitea.orbishosting.com/api/v1/repos/myron//mirror-sync`). +- The 7 DO-hosted sites have **no auto-deploy** — manual SSH + git pull only. -## API Endpoint Auth -- Netscan endpoint (`/api/netscan`) bypasses main auth — uses `X-Registration-Key` header -- Admin portal uses separate PHP session name (`jarvis_admin`) — different from main JARVIS session -- Cloudflare real IP: use `$_SERVER['HTTP_CF_CONNECTING_IP']` not `REMOTE_ADDR` +## API Endpoint Auth (JARVIS) +- Netscan endpoint (`/api/netscan`) bypasses main auth — uses `X-Registration-Key` header. +- Admin portal uses a separate PHP session name (`jarvis_admin`) — different from the main JARVIS session. +- Cloudflare real IP: use `$_SERVER['HTTP_CF_CONNECTING_IP']` (or `HTTP_X_FORWARDED_FOR`), not `REMOTE_ADDR`. ## Network Scan -- The JARVIS "RUN NETWORK SCAN" button does NOT scan from DO (can't reach local network) -- It queues a shell command to PVE1 agent → PVE1 runs nmap → pushes results to /api/netscan -- Results appear ~40 seconds after clicking (10s for agent pickup + 30s nmap) -- Chat "scan network" intent returns real DB data — never hallucinated +- The JARVIS "RUN NETWORK SCAN" button doesn't scan from wherever the browser is — it queues a shell command to the PVE1 agent, which runs nmap and pushes results to `/api/netscan`. Results appear ~40s after clicking. +- Chat "scan network" intent returns real DB data — never hallucinated. -## FusionPBX -- SIP config changes need cache delete before they take effect: - `rm /var/cache/fusionpbx/FusionPBX.configuration.sofia.conf` -- mod_presence is NOT installed on this server +## FusionPBX / VoIP +See [voip.md](voip.md) for the full reference — SignalWire has several non-obvious credential-type and API-endpoint gotchas that cost real debugging time in the past; don't re-derive them from scratch. +- SIP config changes need the sofia.conf Lua cache cleared, not just a service restart: `rm /var/cache/fusionpbx/.configuration.sofia.conf` -## Backup Agent State Fix (Common Issue) -```bash -# If an agent shows "Invalid agent key" after reinstall: -# 1. Get correct values from DB -mysql -u jarvis_user -pJ4rv1s_Pr0t0c0l_2026! jarvis_db -e \ - "SELECT agent_id, api_key FROM registered_agents WHERE hostname='';" -# 2. Overwrite state on the agent machine -cat > /var/lib/jarvis-agent/state.json << EOF -{"api_key": "", "agent_id": ""} -EOF -systemctl restart jarvis-agent -``` +## Git push confirmation +Pushing to a repo's default branch (master/main) needs an explicit, branch-named confirmation from the user — a general "yes" to "want me to push this?" isn't enough and will get blocked. Ask "confirm push directly to master?" up front to save a round-trip. If a chained `git add && commit && push` gets denied, nothing in the chain ran — re-check `git status` before retrying individual steps. + +## Live log popup convention (JARVIS admin) +Any server-side action triggered from JARVIS admin (setup, deploy, install, run backup) must show a live log popup — NOT a `confirm()`/`toast()`. Pattern: `openModal()` + scrollable dark log div + backend log-polling endpoint (`{lines:[], next_line:N}`) + color-coded lines (green=ok, red=error, yellow=warn) + auto-detect completion + CLOSE button that stops polling. Poll every 2s, snapshot log position before triggering the action to avoid replaying old output. `runIntentGenerator()` in the admin JS is the canonical reference implementation. diff --git a/index.md b/index.md index bd707e5..c611b82 100644 --- a/index.md +++ b/index.md @@ -1,38 +1,45 @@ -# AI Context Index — Myron Blair Home Lab +# AI Context Index — Myron Blair Home Lab / MSP -**Load order for a new session:** Read this file first, then load the specific topic file for the task at hand. +**Load order for a new session:** Read this file first, then load the specific topic file for the task at hand. This repo is meant to be handed to *any* AI assistant (not just Claude) to resume full context on this environment. ## Topic Files | File | Use When | |------|----------| -| [servers.md](servers.md) | Connecting to any machine, SSH patterns, credentials | +| [servers.md](servers.md) | Connecting to any machine, SSH patterns, credentials, VM map | | [jarvis.md](jarvis.md) | Working on JARVIS AI dashboard, API, chat, agents | +| [streamhoard.md](streamhoard.md) | Working on the media server / *arr stack / VPN / Portainer / MSP360 backup on CT104 | +| [voip.md](voip.md) | FusionPBX, SignalWire, landline/FXO, SIP registration, phones | | [sites.md](sites.md) | Working on any website, deploys, DBs, email | | [gotchas.md](gotchas.md) | Before running any command — critical quirks | +| [reference.md](reference.md) | Repo inventory notes, assistant hard limits, ZeroTraceGPT IQ, payments migration status | + +## Who this is for +Myron Blair (`myronblair@outlook.com` / `myronblair@gmail.com`). Runs a home lab + small MSP operation: Proxmox cluster, several VMs/LXCs, self-hosted AI (JARVIS + Ollama), VoIP (FusionPBX on-prem), a Docker-based media stack, and public client websites on DigitalOcean. Comfortable with Linux sysadmin, PHP, Python, bash, git, systemd — has deep familiarity with his own infra but won't remember every credential/IP off the top of his head, so don't assume he'll catch an error in a recalled detail. + +**Working style:** wants terse responses — state results, not intentions ("Let me..."), no trailing recaps of what was just done. For any server-side action triggered from an admin UI (JARVIS, etc.), use a live log popup pattern, not a toast/confirm dialog (see gotchas.md). Pushing to a repo's default branch (master/main) needs an explicit branch-named confirmation, not a generic "yes." ## 30-Second Overview -**This is a home-lab + managed-hosting environment.** -No local codebase. Work happens via SSH to remote servers and git push to GitHub (auto-deploy active). +**No local codebase for most work** — it happens via SSH to remote servers/VMs and `git push` to GitHub (several repos have auto-deploy webhooks). -### Infrastructure -- **DO (165.22.1.228)** — DigitalOcean VPS. CyberPanel/OpenLiteSpeed. Hosts all websites (NOT JARVIS anymore). -- **JARVIS VM (10.48.200.211)** — PVE1 VM 211. nginx/PHP8.3/MariaDB/Redis/Arc Reactor. JARVIS AI lives here. -- **NPM (10.48.200.200)** — Nginx Proxy Manager. Routes jarvis.orbishosting.com + hoa.orbishosting.com → local VMs. -- **PVE1 (orbisne.fortiddns.com / 10.48.200.90)** — Proxmox primary hypervisor. 10+ VMs. -- **PVE2 (10.48.200.91)** — Proxmox secondary hypervisor. 2 VMs. -- **FusionPBX (134.209.72.226)** — FreeSWITCH PBX server. -- **Local VMs** — HA (10.48.200.97), Ollama (10.48.200.210), Homebridge (10.48.200.18), Jellyfin (10.48.200.33), NovaCPX (10.48.200.110), MediaStack (10.48.200.35). +### Infrastructure map (current as of 2026-07-24) +- **DO (165.22.1.228)** — DigitalOcean VPS, CyberPanel/OpenLiteSpeed. Hosts 7 client/personal websites. NOT JARVIS, NOT FusionPBX (both moved on-prem). +- **PVE1 (`orbisne.fortiddns.com` / 10.48.200.90)** — Proxmox primary hypervisor. Home of JARVIS, StreamHoard, NovaCPX, Ollama, FusionPBX, Home Assistant, Homebridge, and the NAS-mount passthrough for all of the above. +- **PVE2 (10.48.200.91)** — Proxmox secondary hypervisor, 1 VM (NetworkBackup). +- **JARVIS VM 211 (10.48.200.211)** — the JARVIS AI dashboard/agent-monitoring platform. nginx/PHP8.3/MariaDB/Redis/Arc Reactor. +- **CT104 "StreamHoard" (10.48.200.104)** — consolidated media server: custom Netflix-style app + Sonarr/Radarr/Prowlarr/qBittorrent (all VPN'd via gluetun/NordVPN) + Portainer + MSP360 backup, all in Docker Compose. **Replaces the old Jellyfin VM112, MediaStack VM103, and WireGuard CT110 — all three were destroyed 2026-07-23.** +- **NovaCPX VM120 (10.48.200.110)** — custom hosting control panel; also serves the `web.orbishosting.com` MSP dashboard. +- **FusionPBX VM130 (10.48.200.130)** — FreeSWITCH PBX, moved on-prem from an old DO droplet (that droplet, 134.209.72.226, is destroyed — don't use it). +- **Synology NAS (10.48.200.249)** — primary media + MSP360 backup storage, own DSM reverse proxy, Gitea. +- **NPM (10.48.200.200)** — Nginx Proxy Manager, terminates TLS for `jarvis.orbishosting.com` and a few others. +- **FortiGate (10.48.200.1)** — router/firewall. Admin GUI is on **port 9443**, not 443 (redirects there). `admin` / `Joker1974!!!`. -### Key Systems -- **JARVIS** — Iron Man AI dashboard at `https://jarvis.orbishosting.com` — agent monitoring, chat, network scan, admin portal -- **Admin portal** — `https://jarvis.orbishosting.com/admin/` — full CRUD for all JARVIS data -- **Auto-deploy** — push to GitHub `main` → webhook → server pulls within 1 minute +### Key systems +- **JARVIS** — Iron Man-style AI dashboard at `https://jarvis.orbishosting.com`, agent monitoring/chat/network-scan/admin. Auto-deploy: push to GitHub `master` → webhook → pulls into `/var/www/jarvis/` within ~1 min. +- **StreamHoard** — self-hosted Netflix-style media app (repo `kino-app`, branded "StreamHoard" in the UI — internal names still say `kino` in places, cosmetic only) at `http://10.48.200.104:8080`. +- **web.orbishosting.com** — personal MSP links dashboard (repo `web-dashboard`), auto-deploys via webhook on VM110. +- **Gitea** (`gitea.orbishosting.com`, on the NAS) mirrors every GitHub repo one-way (GitHub → Gitea). 5 repos exist Gitea-only by design (never pushed to GitHub — they hold real secrets): `fortigate-config`, `infra-private`, `jarvis-secrets`, `msp360-config`, `proxmox-secrets`. -### Current Status (as of 2026-06-26) -- JARVIS fully migrated to PVE1 VM 211 (10.48.200.211) — not on DO anymore -- JARVIS agents online: DO, PVE1, PVE2, NetworkBackup, HA, Homebridge, Jellyfin, MediaStack, NovaCPX, FortiGate, Yealink T48S -- Daily backups running at 7am UTC to `/var/backups/jarvis/`, downloadable from admin panel -- Network auto-scan: PVE1 cron every 3 min → pushes to `/api/netscan` -- Ollama at 10.48.200.210:11434 (listening on 0.0.0.0 — fixed 2026-06-24) +### Recent major change (2026-07-23/24) +Jellyfin, MediaStack, and the WireGuard exit container were all decommissioned and consolidated into one host, StreamHoard (CT104). See [streamhoard.md](streamhoard.md) for the full detail — this was a large migration with several real infra bugs found and fixed along the way (a live duplicate-IP conflict, a NordVPN kill-switch regression, NFS/CIFS mount limitations in unprivileged LXCs, an MSP360 install gap). Don't be surprised if older docs/repos (`mediastack`, old `kino` repo) still reference the pre-migration state — they're stale now. diff --git a/jarvis.md b/jarvis.md index cb324f4..f1b3882 100644 --- a/jarvis.md +++ b/jarvis.md @@ -4,132 +4,113 @@ - **Dashboard:** https://jarvis.orbishosting.com (login: myron / Joker1974!!!) - **Admin portal:** https://jarvis.orbishosting.com/admin/ (same login) - **DB:** `jarvis_db` on VM 211 localhost — user: `jarvis_user` / `J4rv1s_Pr0t0c0l_2026!` -- **phpMyAdmin:** https://jarvis.orbishosting.com/phpmyadmin (myron / Joker1974!!!) -- **GitHub repo:** myronblair/jarvis (auto-deploy on push to main) +- **GitHub repo:** `myronblair/jarvis` (private; auto-deploy on push to `master`) ## Server Location -JARVIS runs on **PVE1 VM 211** at `10.48.200.211` (migrated from DO 2026-06-18). -- **SSH:** `sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.211` +JARVIS runs on **PVE1 VM 211** at `10.48.200.211`. - **Stack:** nginx / PHP 8.3 / MariaDB / Redis / Arc Reactor -- **Web root:** `/var/www/jarvis/` -- Traffic routed via NPM (10.48.200.200) → VM 211 +- **Web root:** `/var/www/jarvis/` — this directory **is** the live git repo, push from it directly +- **TLS:** terminated by NPM (VM105, 10.48.200.200), proxy_host → `10.48.200.211:80`, Let's Encrypt cert. **The old `:1972` port reference is wrong/dead** — use plain `http://jarvis.orbishosting.com` (port 80) or the HTTPS URL above. GitHub webhook is `https://jarvis.orbishosting.com/webhook.php`. ## File Structure (on VM 211 at /var/www/jarvis/) ``` public_html/ index.html — main Iron Man HUD (all UI) api.php — API router - admin/index.php — admin portal (single PHP+JS file) + admin/index.php — admin portal (single ~5000-line PHP+JS file, deliberately left unsplit — see gotchas) + admin/downloads/INFRASTRUCTURE-REFERENCE.md — master live infra doc, gitignored, edit here first agent/ — agent installers api/ config.php — all credentials/constants (gitignored) - lib/db.php — JarvisDB class (query/execute/single/insert) - lib/kb_engine.php — KBEngine intent matching + lib/db.php — JarvisDB class (query/execute/single/insert) + lib/kb_engine.php — KBEngine intent matching endpoints/ - agent.php — agent registration/heartbeat/metrics/commands - chat.php — 4-tier chat: KB→action intents→Ollama→Groq→Claude - network.php — network device list + scan endpoint - netscan.php — push endpoint for PVE1 nmap results (no auth needed) - do_server.php — reads /proc directly (no SSH loopback) - stats_cache.php — every 5min cron: Proxmox cluster API, HA, weather, news + agent.php — agent registration/heartbeat/metrics/commands + chat.php — tiered chat: KB → action intents → Ollama → Groq → Claude + network.php — network device list + scan endpoint + netscan.php — push endpoint for PVE1 nmap results (uses X-Registration-Key, bypasses main auth) + stats_cache.php — every 5min cron: Proxmox cluster API, HA, weather, news facts_collector.php — every 3min cron: system stats, site health - system.php — local system metrics - alerts.php — alert CRUD + auto-generate - news.php — serves api_cache['news'] + custom kb_facts(category='custom_news') + kb_intent_generator.php — every 6h cron, self-expanding knowledge base (see below) + alerts.php — alert CRUD + auto-generate deploy/ - reactor.py — Arc Reactor source (copy to /opt/jarvis-arc/reactor.py + restart) + reactor.py — Arc Reactor source (copy to /opt/jarvis-arc/reactor.py + `systemctl restart jarvis-arc` to deploy) ``` -## Arc Reactor (AI Routing Daemon) -- **Daemon:** `/opt/jarvis-arc/reactor.py` (Python, port 7474), service: `jarvis-arc` -- **Deploy:** copy `deploy/reactor.py` → `/opt/jarvis-arc/reactor.py` + `systemctl restart jarvis-arc` -- Vision with actual screenshot → Claude `claude-opus-4-8-20251101` -- Guardian/SITREP/Vision text-only → Groq `llama-3.3-70b-versatile` -- Email drafting / research → Claude +## Arc Reactor (AI routing daemon) +- **Daemon:** `/opt/jarvis-arc/reactor.py` (Python venv), service: `jarvis-arc`, port 7474 +- **Logs:** `/var/log/jarvis/arc.log` (setup log: `arc-setup.log`) +- **Vision cascade:** Claude API first; if Claude credits depleted, falls back to Ollama `llava:7b` (config via `/etc/systemd/system/jarvis-arc.service.d/vision.conf` → `OLLAMA_VISION_MODEL=llava:7b`) +- Secrets (Claude/Groq keys, DB pass, Gmail/iCloud app passwords) live in `/etc/jarvis-arc/reactor.env` (root:www-data 0640, loaded via systemd EnvironmentFile drop-in) — NOT in the repo. + +## AI tiers (chat + reactor) +``` +Tier 0.7: KB intents / planner +Tier 1: Knowledge Base (MySQL, self-expanding — see KB Generator below) +Tier 1.5: Ollama llama3.1:8b (10.48.200.210:11434) — CPU-only, ~30s+/reply, can 504 on default 60s nginx timeout +Vision: Ollama llava:7b (via Arc Reactor vision cascade) +Tier 2: Groq (model name: compound-beta-mini — NOT groq/compound-beta-mini, that 404s) +Tier 3: Claude API (fallback; credits need periodic refill at console.anthropic.com) +``` +Groq is effectively the fast/primary path when Claude credits are depleted — Ollama is too slow for interactive chat (504s). ## Agent System -- **Registration key:** `f846a9aaf7ce9a61742c63c87c4186052a71d2a580c65518` -- **Install one-liner (Linux):** `curl -sk https://jarvis.orbishosting.com/install-agent.sh | bash -s ` -- **Agent config:** `/etc/jarvis-agent/config.json` (key: `jarvis_url`) — runtime state at `/var/lib/jarvis-agent/state.json` -- **If agent gets 401 "Invalid agent key":** state.json has stale key — overwrite with correct agent_id + api_key from DB -- **Heartbeat:** every 10s | **Metrics:** every 30s - -## Currently Online Agents -| agent_id | hostname | IP | type | -|----------|----------|----|------| -| jarvis-do_orbis.or | jarvis-do | 165.22.1.228 | linux | -| claude_pve | pve1 | 10.48.200.90 | proxmox | -| pve2_pve2 | pve2 | 10.48.200.91 | proxmox | -| networkbackup_NetworkB | networkbackup | 10.48.200.99 | linux | -| homeassistant_ha | homeassistant | 10.48.200.97 | homeassistant | -| homebridge_b57cbaea | homebridge | 10.48.200.18 | linux | -| novacpx_e3b07264 | novacpx | 10.48.200.110 | linux | -| jellyfin_7e386833 | jellyfin | 10.48.200.33 | linux | -| MediaStack_2c00b1b8 | mediastack | 10.48.200.35 | linux | -| fortigate_gw | fortigate | 10.48.200.1 | linux | -| yealink_t48s | yealink | 10.48.200.43 | linux | +- Agents register via `/api/agent` using a shared **registration key** (rotated multiple times historically — always read the current value from `/var/www/jarvis/api/config.php` `AGENT_REGISTRATION_KEY`, don't trust an old hardcoded value from any doc). Per-agent heartbeats then use their own `api_key` from the `registered_agents` table, not the shared reg key. +- **Agent script:** `/opt/jarvis-agent/jarvis-agent.py` (Python3, no external deps), config at `/etc/jarvis-agent/config.json`: + ```json + { + "jarvis_url": "http://10.48.200.211", + "ssl_verify": false, + "registration_key": "", + "hostname": "", + "agent_type": "linux", + "poll_interval": 30, + "heartbeat_every": 10, + "watch_services": [""] + } + ``` + Systemd unit `jarvis-agent.service` (Restart=always). State file `/var/lib/jarvis-agent/state.json`. +- **401 "Invalid agent key"** → state.json has a stale key; delete it and restart the service to force re-registration, or manually overwrite with the correct `agent_id`/`api_key` from the DB. +- Agent auto-detects capabilities: `docker` (if docker binary present), `proxmox`, `ollama`, `homeassistant`. +- **Known registered agents (not exhaustive, check `registered_agents` table for current truth):** PVE1, PVE2, NetworkBackup, Home Assistant, Homebridge, NovaCPX, Ollama, StreamHoard/CT104 (added 2026-07-23, hostname `streamhoard104`). Jellyfin/MediaStack/WireGuard agents no longer exist (VMs destroyed). ## Network Scanning -- PVE1 cron: `*/3 * * * * /usr/local/bin/jarvis-netscan.sh` -- Script runs nmap, parses output, POSTs JSON to `https://jarvis.orbishosting.com/api/netscan` with `X-Registration-Key` header -- Scan Now button queues shell command to PVE1 agent (picks up within 10s) +- PVE1 cron `*/3 * * * *` runs `jarvis-netscan.sh` → nmap → POSTs JSON to `/api/netscan` with `X-Registration-Key` header. +- Chat "scan network" intent returns real DB data, never hallucinated. -## Chat Architecture -``` -Tier 0: HA entity control (fuzzy match → call HA API) -Tier 0.5: Network device management -Tier 0.7: Planner intents (add task, my tasks, schedule, daily briefing) -Tier 1: KB intent engine (response type → instant reply) -Tier 1b: Action intents (network_scan → DB data + queue PVE1 scan) -Tier 2: Ollama llama3.2 at http://10.48.200.210:11434 (5s timeout) -Tier 3: Groq compound-beta-mini (cloud, fast) -Tier 4: Claude API fallback -``` -- Groq model name: `compound-beta-mini` (NOT `groq/compound-beta-mini`) -- HA_URL = `http://orbisne.fortiddns.com:8123` - -## DB Key Tables +## Chat Architecture / DB ```sql registered_agents — agent_id, hostname, agent_type, ip_address, api_key, status, last_seen, version -agent_metrics — agent_id, metric_type, metric_data(JSON), recorded_at - -- Extract: JSON_EXTRACT(metric_data,'$.cpu_percent'), JSON_EXTRACT(metric_data,'$.memory.percent') - -- NO cpu_pct/mem_pct columns — always use JSON_EXTRACT +agent_metrics — agent_id, metric_type, metric_data(JSON), recorded_at -- use JSON_EXTRACT, no direct cpu/mem columns agent_commands — agent_id, command_type, command_data(JSON), status(pending/delivered) network_devices — ip, mac, hostname, alias, device_type, status, last_seen alerts — alert_type, title, message, severity, resolved -kb_facts — category, fact_key, fact_value +kb_facts — category, fact_key, fact_value -- fact_value is MEDIUMTEXT (was TEXT, HA entity-map JSON outgrew it and silently failed writes for days before this was caught) kb_intents — intent_name, pattern(regex), response_template, action_type, priority, active -api_cache — cache_key(proxmox/news/weather/ha_entities), data(JSON), updated_at +kb_generator_topics — id, topic_id, category, topic_name, description, active, run_count, last_run_at tasks — title, notes, category, priority, status, due_date, due_time, completed_at appointments — title, description, category, start_at, end_at, location, all_day, reminder_min -ha_entities — real-time HA entity states (updated by HA custom component) ``` -## Proxmox API -- `stats_cache.php` uses cluster API at `orbisne.fortiddns.com:8006` -- Returns all VMs from both PVE1 and PVE2 via `/cluster/resources?type=vm` -- Token: `root@pam!jarvis=c45b5feb-f9a9-445d-a626-14fbb959f78b` - -## Cron Jobs (on VM 211) -``` -*/3 * * * * php /var/www/jarvis/api/endpoints/facts_collector.php -*/5 * * * * php /var/www/jarvis/api/endpoints/stats_cache.php -0 7 * * * /usr/local/bin/jarvis-backup.sh -``` -**Cron on PVE1:** -``` -*/3 * * * * /usr/local/bin/jarvis-netscan.sh -``` +## KB Intent Generator (self-expanding knowledge base) +- Cron `0 */6 * * *`, processes 25 topics/run (BATCH_SIZE) cycling through `kb_generator_topics` table. +- Topic count grew from 140 → 4530 across 170 categories (general knowledge, Fortune-500 companies, space/exploration, etc.) — at 25/run a full cycle now takes ~45 days; raise BATCH_SIZE if freshness matters. +- Admin UI "MANAGE TOPICS" button — browse/add/edit/delete/toggle topics. +- Groq model `llama-3.3-70b-versatile`, 20 intents/call, max_tokens=5000 (raising from 40/3500 fixed an 88% truncation failure rate). ## Backups -- Script: `/usr/local/bin/jarvis-backup.sh` -- Output: `/var/backups/jarvis/jarvis_backup_YYYY-MM-DD_HH-MM-SS.tar.gz` -- Daily at 7am UTC (2am CDT), 7-day retention -- Downloadable from admin portal → BACKUPS tab +- Script: `/usr/local/bin/jarvis-backup.sh` → `/var/backups/jarvis/jarvis_backup_TIMESTAMP.tar.gz`, daily ~7am UTC, 7-day retention, downloadable from admin panel. +- Also captures `/etc/jarvis-arc` (reactor.env), `/etc/jarvis` (db.env), service drop-ins, `/usr/local/bin/jarvis-*.sh`. +- Health monitoring (`jarvis-health.sh`, PVE1 cron */5) checks stalled crons, stuck arc_jobs, disk >85%, watchdog restarts — auto-resolving `alerts` rows + email to `myronblair@gmail.com`. ## API Auth - Main JARVIS API: session token via `X-Session-Token` header (or PHP session) -- Agent endpoints: `X-Agent-Key` header (per-agent key from registered_agents.api_key) +- Agent endpoints: per-agent key via header (see Agent System above) - Netscan endpoint: `X-Registration-Key` header (shared registration key) -- Admin portal: separate PHP session (`session_name('jarvis_admin')`) -- Cloudflare passes real client IP in `CF-Connecting-IP` header +- Admin portal: separate PHP session name (`jarvis_admin`) +- Cloudflare passes real client IP in `CF-Connecting-IP` header (or `HTTP_X_FORWARDED_FOR`) + +## Security posture (Phase 1 hardening, 2026-07-07 — mostly done, some user follow-up still open) +- Secrets pulled out of git-tracked files into `/etc/jarvis-arc/reactor.env` and `/etc/jarvis/db.env`. **Still pending (user action): rotate Claude/Groq/Gmail-app-pw/iCloud-app-pw/DB-pass since they remain in GitHub history**, and do a `git filter-repo` + force-push history purge. +- Login: Redis per-IP rate limit (10 fails/15min), session_regenerate_id on success. +- `INFRASTRUCTURE-REFERENCE.md` is gitignored/untracked in the repo — lives only on-disk on VM211 (and its VM110 mirror copy), not in GitHub. diff --git a/reference.md b/reference.md new file mode 100644 index 0000000..ee5d71d --- /dev/null +++ b/reference.md @@ -0,0 +1,20 @@ +# Misc Reference + +## Repo inventory notes +- GitHub is source of truth; Gitea mirrors one-way (GitHub → Gitea only, never push to Gitea directly). +- **Stale/legacy repos that predate current infra — don't treat as authoritative:** `mediastack` (MediaStack VM destroyed 2026-07-23, superseded by `kino-app`/StreamHoard), `kino` (an earlier/separate repo from the current `kino-app`), this `ai-context` repo itself was ~1 month stale before this rewrite. +- `ai-context` (this repo) is the canonical "hand this to a new AI session" reference — keep it updated after any major infra change, the same way this file was updated 2026-07-24. + +## Assistant hard limits (what an AI assistant working this environment will/won't do) +- Won't enter passwords/tokens into login forms or complete OAuth flows on the user's behalf. +- Won't materialize a raw secret in a visible command/transcript even with permission — use env vars/files instead. +- Destructive actions on shared/production infra need explicit, specific confirmation (a general "clean this up" isn't enough for an actual `rm`; pushing to a repo's default branch needs the branch named explicitly). + +## ZeroTraceGPT IQ (local AnythingLLM instance, not yet integrated into JARVIS) +- Rebranded AnythingLLM instance on Myron's Windows workstation, base URL `http://127.0.0.1:7865`, API prefix `/api/v1/...`, Bearer token `9NPG3XG-32V47D6-PYET8BB-CDYYVRG`. +- OpenAI-compatible endpoints (`/v1/openai/chat/completions`) once a model is loaded; RAG-backed workspace chat (`/v1/workspace/{slug}/chat`). +- **Not yet wired into JARVIS's Arc Reactor AI tier cascade** — blocked because it's bound to `127.0.0.1` only (needs LAN or Tailscale exposure before VM211 can reach it). If asked to add it as a tier, follow the same pattern as the existing Ollama Tier 1.5 entry. +- Separate from a lighter desktop app also called "ZeroTraceGPT" (GPT4All-based, different product, installed locally). + +## Payments consolidation (planned, not started) +Move tomsjavajive.com off Stripe onto Square, to consolidate all payment-taking sites onto one processor (parkerslingshotrentals.com already uses Square). See sites.md for current per-site payment processor status. diff --git a/servers.md b/servers.md index ff2d983..f3883c5 100644 --- a/servers.md +++ b/servers.md @@ -3,59 +3,68 @@ ## SSH Patterns ```bash -# DigitalOcean (websites only — JARVIS no longer here) +# DigitalOcean (websites only — JARVIS/FusionPBX no longer here) sshpass -p 'Gonewalk1974!@#' ssh -o StrictHostKeyChecking=no root@165.22.1.228 'cmd' -# JARVIS VM (PVE1 VM 211) -sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.211 'cmd' - -# PVE1 via FortiGate DDNS (survives IP changes) +# PVE1 (hop node for everything else — password auth) +sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.90 'cmd' +# also reachable via FortiGate DDNS (survives WAN IP changes, which happen often): sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@orbisne.fortiddns.com 'cmd' -# PVE1 direct (local network only) -sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.90 'cmd' +# Any VM behind PVE1 — nested hop through PVE1 +sshpass -p 'Joker1974!!!' ssh root@10.48.200.90 \ + 'ssh -o StrictHostKeyChecking=no root@ "cmd"' -# PVE2 (local only) +# PVE2 (local only, no external port-forward) sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.91 'cmd' -# FusionPBX (must relay via DO — port 22 firewalled from internet) -sshpass -p 'Gonewalk1974!@#' ssh -o StrictHostKeyChecking=no root@165.22.1.228 \ - 'sshpass -p "Joker1974!@#" ssh -o StrictHostKeyChecking=no root@134.209.72.226 "cmd"' +# Run a command inside an LXC container (CT) via PVE1's pct +sshpass -p 'Joker1974!!!' ssh root@10.48.200.90 'pct exec -- bash -c "cmd"' -# NovaCPX (direct — PVE1 hop broken) -sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.110 'cmd' +# Run a command inside a QEMU VM via guest agent (works even if network/SSH is down) +sshpass -p 'Joker1974!!!' ssh root@10.48.200.90 'qm guest exec -- bash -c "cmd"' +# guest-exec has no HOME env by default — many CLI tools need: export HOME=/root; ... -# MediaStack (SSH key auth) -ssh -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa root@10.48.200.35 'cmd' - -# Local VMs (myron user + sudo, password Joker1974!) +# Local VMs using myron user + sudo sshpass -p 'Joker1974!' ssh -o StrictHostKeyChecking=no myron@10.48.200.18 'sudo cmd' -# Run command inside a Proxmox VM -sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.90 \ - 'qm guest exec -- bash -c "cmd"' +# Synology NAS — needs password auth FORCED (pubkey silently preferred/fails otherwise) +sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no -o PreferredAuthentications=password \ + -o PubkeyAuthentication=no nas@10.48.200.249 'echo Joker1974!!! | sudo -S cmd' +# scp to the NAS needs the legacy flag too: scp -O ... ``` -**Password order to try if first fails:** `Joker1974!@#` → `Joker1974!!!` → `Joker1974!` +**Password order to try if first fails:** `Joker1974!!!` → `Joker1974!` → `Gonewalk1974!@#` (DO only). -## Server Map +**MediaStack-35's ad-hoc interactive SSH intermittently failed** (resolved as non-issue — the automated backup script's own key-based access always worked fine; root cause was never found, and it's moot now since that VM is destroyed). If a similarly-flaky interactive SSH pattern shows up on another host, don't assume it's a real access/security problem before checking whether an automated path using the same credentials still works. + +## Server Map (current) | Host | IP | User | Password | Purpose | |------|----|------|----------|---------| -| DO | 165.22.1.228 | root | Gonewalk1974!@# | Websites (CyberPanel/OLS) | -| JARVIS VM | 10.48.200.211 | root | Joker1974!!! | JARVIS AI — PVE1 VM 211 | -| NPM | 10.48.200.200 | — | — | Nginx Proxy Manager (Docker) | -| PVE1 | orbisne.fortiddns.com (10.48.200.90) | root | Joker1974!!! | Primary hypervisor | +| DO | 165.22.1.228 | root | Gonewalk1974!@# | Websites (CyberPanel/OLS), 7 sites | +| PVE1 | orbisne.fortiddns.com (10.48.200.90) | root | Joker1974!!! | Primary hypervisor, SSH hop for everything | | PVE2 | 10.48.200.91 | root | Joker1974!!! | Secondary hypervisor | -| FusionPBX | 134.209.72.226 | root | Joker1974!@# | FreeSWITCH PBX | -| NovaCPX | 10.48.200.110 | root | Joker1974!!! | Hosting panel — PVE1 VM 120 | -| MediaStack | 10.48.200.35 | root | (SSH key) | Sonarr/Radarr/qBit — PVE1 VM 103 | -| HomeAssistant | 10.48.200.97 | — | — | HA VM 109 (web terminal only) | -| Homebridge | 10.48.200.18 | myron | Joker1974! | PVE1 VM 118 | -| Jellyfin | 10.48.200.33 | myron | Joker1974! | PVE1 VM 112 | -| Ollama | 10.48.200.210 | myron | Joker1974! | LLM VM 106, port 11434 | +| JARVIS VM 211 | 10.48.200.211 | root | Joker1974!!! | JARVIS AI dashboard | +| StreamHoard CT104 | 10.48.200.104 | root | Joker1974!!! (via `pct exec 104`) | Consolidated media server — see streamhoard.md | +| NovaCPX VM120 | 10.48.200.110 | root | Joker1974!!! | Hosting panel + web.orbishosting.com dashboard | +| FusionPBX VM130 | 10.48.200.130 | root | Joker1974!!! | FreeSWITCH PBX (on-prem now, not the old DO droplet) | +| Ollama VM106 | 10.48.200.210 | root | Joker1974!!! | Local LLM (llama3.1:8b, llava:7b vision), port 11434 | +| Home Assistant | 10.48.200.97 | — | — | HA VM 109/101 (web terminal only) | +| Homebridge VM118 | 10.48.200.18 | myron | Joker1974! | HomeKit bridge | | NetworkBackup | 10.48.200.99 | myron | Joker1974! | PVE2 VM 302 | -| WireGuard CT110 | 10.48.200.67 | — | — | VPN kill-switch for MediaStack | +| NPM VM105 | 10.48.200.200 | — | — | Nginx Proxy Manager (Docker), admin `myronblair@outlook.com` | +| Synology NAS | 10.48.200.249 | nas | Joker1974!!! | Media + MSP360 backup storage, DSM 7.3.2, Gitea | +| FortiGate | 10.48.200.1 | admin | Joker1974!!! | Router/firewall — admin GUI on **:9443**, not 443 | + +**Destroyed 2026-07-23 — do not use these IPs/VMIDs for anything:** +| Was | IP | VMID | +|---|---|---| +| Jellyfin | 10.48.200.33 | VM112 | +| MediaStack | 10.48.200.35 | VM103 (also briefly `.136` mid-migration) | +| WireGuard exit container | 10.48.200.67 / .19 | CT110 | + +`10.48.200.35` is currently squatted by an **unidentified rogue device** (SSH banner `OpenSSH_8.4p1`, exposes NFS shares named `TimeMachineBackup`/`Public`/`iTunes` — looks like an old Buffalo-brand NAS). Found during the migration, never tracked down/removed. Don't be surprised if that IP answers but isn't what you expect. ## Proxmox API ```bash @@ -69,26 +78,45 @@ GET /api2/json/cluster/resources?type=vm ## Key Paths on JARVIS VM (10.48.200.211) ``` -/var/www/jarvis/ — JARVIS app root +/var/www/jarvis/ — JARVIS app root, IS the live git repo (push from it directly) /var/www/jarvis/public_html/ — web root (index.html, api.php, admin/) -/var/www/jarvis/api/ — backend (config.php, endpoints/, lib/) +/var/www/jarvis/api/ — backend (config.php [gitignored], endpoints/, lib/) /var/www/jarvis/deploy/ — Arc Reactor source (reactor.py) -/opt/jarvis-arc/reactor.py — Arc Reactor daemon +/opt/jarvis-arc/reactor.py — Arc Reactor daemon (copy deploy/reactor.py here + restart to deploy) /var/backups/jarvis/ — daily backups (tar.gz, 7-day retention) /usr/local/bin/jarvis-backup.sh — backup script +/var/www/jarvis/public_html/admin/downloads/INFRASTRUCTURE-REFERENCE.md — the master live infra doc (edit here first, gitignored, NOT in GitHub — relay-copy to VM110 through PVE1 after editing) +``` + +## Key paths on StreamHoard CT104 (10.48.200.104) +See [streamhoard.md](streamhoard.md) for full detail. +``` +/opt/kino/ — docker-compose.yml, .env, app source (repo kino-app) +/opt/streamhoard-config/ — migrated radarr/sonarr/prowlarr/qbittorrent config+library +/mnt/nas/video/ — LXC mountpoint passthrough of the NAS media library (movies/tv) +/mnt/nas-backups/ — LXC mountpoint passthrough for MSP360 destination ``` ## Key Paths on DO (165.22.1.228) ``` -/home/tomsjavajive.com/public_html/ — Tom's Java Jive -/home/epictravelexpeditions.com/public_html/ — Epic Travel -/home/parkerslingshotrentals.com/public_html/ — Parker Slingshot -/home/orbishosting.com/public_html/ — Orbis Hosting -/home/orbis.orbishosting.com/public_html/ — Orbis Portal -/home/tomtomgames.com/public_html/ — TomTom Games +/home/tomsjavajive.com/public_html/ +/home/epictravelexpeditions.com/public_html/ +/home/parkerslingshotrentals.com/public_html/ +/home/orbishosting.com/public_html/ — NOT the real public apex site (see sites.md) +/home/orbis.orbishosting.com/public_html/ +/home/tomtomgames.com/public_html/ +/home/worktracking.orbishosting.com/public_html/ — ChuckCo Time Keeper (repo: chucko) ``` +`.git` for all 7 sites lives OUTSIDE the web root at `/home//git-data` (a `gitdir:` pointer file sits in `public_html/.git`) — `git` commands from `public_html` work normally. ## PHP / OLS Notes (DO server) - Run scripts: `/usr/local/lsws/lsphp85/bin/lsphp /path/to/script.php` - Syntax check: `php8.3 -l file.php` (lsphp segfaults on -l) - Web server: OpenLiteSpeed (`lshttpd`), NOT apache/nginx +- Cron bug: `*/3 * * * *` restarts `lshttpd` if any `.htaccess` is newer than `/usr/local/lsws/cgid` (a directory) — nothing re-touches that dir after restart, so editing `.htaccess` triggers a restart loop until you `touch /usr/local/lsws/cgid` manually. + +## GitHub / Gitea +- GitHub is source of truth; **Gitea (`gitea.orbishosting.com`, on the NAS) is a one-way, read-only mirror** — never push directly to Gitea. +- `gh` CLI at `C:\Program Files\GitHub CLI\gh.exe` (Windows workstation), auth via `GH_TOKEN` env var, PAT scopes `repo`+`workflow` (no `read:org`, so use `GH_TOKEN` directly or `gh api`, not `gh auth login --with-token`). +- After any GitHub push, trigger the mirror manually: `curl -X POST -u myron:Joker1974!!! https://gitea.orbishosting.com/api/v1/repos/myron//mirror-sync` +- 5 repos are Gitea-only by design, never push to GitHub: `fortigate-config`, `infra-private`, `jarvis-secrets`, `msp360-config`, `proxmox-secrets` — these hold real credentials deliberately kept off a third-party cloud host. diff --git a/sites.md b/sites.md index 7be9a6a..f1b27bb 100644 --- a/sites.md +++ b/sites.md @@ -1,5 +1,7 @@ # Websites & Sites Reference Card +FusionPBX has moved to its own file — see [voip.md](voip.md). It is no longer on the DO server. + ## All Sites on DO (165.22.1.228) | Site | Path | GitHub Repo | DB | @@ -8,35 +10,39 @@ | epictravelexpeditions.com | /home/epictravelexpeditions.com/public_html/ | myronblair/epictravelexpeditions | epic_epic_db | | parkerslingshot (subdomain) | /home/epictravelexpeditions.com/parkerslingshot/ | myronblair/parkerslingshot | epic_parkersling | | parkerslingshotrentals.com | /home/parkerslingshotrentals.com/public_html/ | myronblair/parkerslingshotrentals | parker_db | -| orbishosting.com | /home/orbishosting.com/public_html/ | myronblair/orbishosting | — | -| orbis.orbishosting.com | /home/orbis.orbishosting.com/public_html/ | myronblair/orbis-hosting-portal | — | | tomtomgames.com | /home/tomtomgames.com/public_html/ | myronblair/tomtomgames | tomt_ttg_db | +| worktracking.orbishosting.com | /home/worktracking.orbishosting.com/public_html/ | myronblair/chucko | workt_track_db | +| orbishosting.com (apex) | /home/orbishosting.com/public_html/ | myronblair/orbishosting | — | +| orbis.orbishosting.com | /home/orbis.orbishosting.com/public_html/ | myronblair/orbis-hosting-portal | — | + +**orbishosting.com (apex) and orbis.orbishosting.com are NOT the real live sites — don't audit/fix them expecting user traffic.** The apex's `/home/orbishosting.com` on this DO droplet is just the CyberPanel parent-domain container (its own public_html is SuiteCRM legacy + a static placeholder) — the *real* public apex orbishosting.com is a WHMCS billing portal hosted somewhere else entirely (location unknown). `orbis.orbishosting.com` is a confirmed-intentional pre-launch placeholder "sign in" page. The two subdomains that actually matter are **web.orbishosting.com** (MSP dashboard, on VM110 in the home-lab, not DO — see servers.md) and **jarvis.orbishosting.com** (JARVIS, VM211, not DO). + +## ChuckCo Time Keeper (worktracking.orbishosting.com) +Work-tracking app, 5-day Fri–Thu week, flat day-rate pay, token-based access (no login for workers/payer, single shared admin password). DB `workt_track_db` / `workt_track_user` / `ZWCNMRP2N5NVPsghmve5aRS9`. Admin password `Joker1974!!!` at `/admin/login.php`. ## JARVIS (on PVE1 VM 211, NOT DO) | Site | Path | GitHub Repo | DB | |------|------|-------------|-----| | jarvis.orbishosting.com | /var/www/jarvis/ on 10.48.200.211 | myronblair/jarvis | jarvis_db | +See [jarvis.md](jarvis.md) for full detail. -## Deploy Workflow +## web.orbishosting.com (MSP dashboard, NOT on DO) +On VM110 (NovaCPX, 10.48.200.110), `/home/webacct/public_html/index.html`, repo `myronblair/web-dashboard`, branch `main`. `web-dashboard-deploy.sh` (cron */1min on VM110) pulls when a webhook queues the path in `/tmp/web-dashboard-deploy.txt`. Currently links out to StreamHoard/Sonarr/Radarr/Prowlarr/qBittorrent/Portainer (all on CT104, 10.48.200.104) — the old Jellyfin/MediaStack links were removed 2026-07-23 when those VMs were destroyed. + +## Deploy Workflow (DO sites) ```bash # 1. Edit files on server via SSH (for quick hotfixes) sshpass -p 'Gonewalk1974!@#' ssh -o StrictHostKeyChecking=no root@165.22.1.228 \ 'nano /home/site.com/public_html/file.php' -# 2. ALWAYS commit + push to GitHub +# 2. ALWAYS commit + push to GitHub — deploy for these 7 sites is MANUAL cd /home/site.com && git add -A && git commit -m "message" && git push - -# Auto-deploy triggers within 1 minute via webhook -# Webhook handler: tomtomgames.com/webhook.php (DO sites) -# JARVIS webhook: port 1972 on VM 211 - -# For immediate deploy (skip 1-min wait): -sshpass -p 'Gonewalk1974!@#' scp -o StrictHostKeyChecking=no \ - /tmp/changed.php root@165.22.1.228:/home/site.com/public_html/changed.php +# No webhook/cron auto-pull exists for these DO sites (unlike JARVIS/web-dashboard/StreamHoard, +# which do have auto-deploy). .git lives outside the web root at /home//git-data. ``` ## GitHub PAT -`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` — expires ~2026-08-20 +Current: `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` (rotated 2026-07-05 after the previous one, `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`, was found exposed in `.git/config` on all 6 DO sites). This is also the Windows workstation's `GH_TOKEN` env var. ## Database Credentials @@ -47,22 +53,19 @@ sshpass -p 'Gonewalk1974!@#' scp -o StrictHostKeyChecking=no \ | epic_epic_db | epic_epic | (check config.php) | | epic_parkersling | epic_parkersling | Joker1974!!! | | parker_db | parker_db | (check config.php) | +| workt_track_db | workt_track_user | ZWCNMRP2N5NVPsghmve5aRS9 | | MySQL root (DO) | root | b71e5c1a8c7457541b9c1db822de37adfa271926a38b6c20 | -```bash -# Quick DB access on DO -sshpass -p 'Gonewalk1974!@#' ssh -o StrictHostKeyChecking=no root@165.22.1.228 \ - 'mysql -u toms_tjj_user -p"+60wlPc+55e@gFq4" toms_tjj_db -e "SELECT ..."' - -# Quick DB access on JARVIS VM -sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.211 \ - 'mysql -u jarvis_user -pJ4rv1s_Pr0t0c0l_2026! jarvis_db -e "SELECT ..."' -``` - ## Gitignored Credentials (never in GitHub) - `api/config.php` — JARVIS, epictravelexpeditions - `config/database.php` — tomsjavajive - `db.php`, `config.php` — parkerslingshot +- `includes/config.php` — chucko/worktracking (DB creds, admin password hash, site-wide token) + +## Payments — status and pending migration +- **tomsjavajive.com still uses Stripe** — a live Stripe secret/publishable/webhook key was found hardcoded in `config/config.php`, moved to `/home/tomsjavajive.com/config-secrets.php` (still in git history, needs rotation via Stripe dashboard — user action, not yet done). +- **Planned future task (not started):** move tomsjavajive.com off Stripe onto **Square**, to consolidate all 3 payment-taking sites (tomsjavajive, parkerslingshotrentals, and presumably a third) onto one processor. +- **parkerslingshotrentals.com uses Square** — access token rotated 2026-07-05: `EAAAl_Cj6NVggy_-8JAFzEPcTHQMv0cY27Sy_uwJt-2AiFVq5Jz-42QzyIIMcVhY`, live in `db-secrets.php`. ## Tom's Java Jive Quirks - No `slug` column on products — URLs use `?id=product_id` @@ -76,14 +79,6 @@ sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.211 \ - Login: `admin / Parker2026!` ## Email / SMTP -- All sites: CyberMail (CyberPersons) — API key `sk_live_7f9b0f9a29f6de31a0d229d4af75d56b094ad724fc58a57d` +- All sites use CyberMail (CyberPersons) — current key `sk_live_df1965fc93751165d0bdd99a6eb0b46ca793ab9e44129aae` (rotated 2026-07-05, shared across tomsjavajive/parkerslingshotrentals/tomtomgames — they're all on the same CyberMail account). - Manage at: platform.cyberpersons.com -- DO blocks SMTP port 587 — use CyberMail API over HTTPS (port 443) - -## FusionPBX -- URL: https://fusion.orbishosting.com (admin / fY7XP5swgtpbzrYLhkeVYkA4744) -- Server: 134.209.72.226 (SSH: relay via DO → root / Joker1974!@#) -- Timezone: America/Chicago -- SIP profiles via Lua XML handler — config changes need cache delete: - `rm /var/cache/fusionpbx/FusionPBX.configuration.sofia.conf` -- Ext 1000 (Myron T48S), Ext 1001 (Tommy T48S), Ext 1002 (Myron AX86R), Ext 1003 (Kitchen T57W), Ext 1004 (Bedroom T57W) +- DO blocks SMTP port 587 — use the CyberMail API over HTTPS (port 443) instead. diff --git a/streamhoard.md b/streamhoard.md new file mode 100644 index 0000000..9da4856 --- /dev/null +++ b/streamhoard.md @@ -0,0 +1,60 @@ +# StreamHoard Reference Card (CT104, 10.48.200.104) + +**Replaces the old Jellyfin (VM112), MediaStack (VM103), and WireGuard exit container (CT110) — all three destroyed 2026-07-23.** Everything media-related now lives on one Debian 12 LXC (CT104, 8 cores/12GB RAM/60GB disk) on PVE1. + +## What it is +A self-hosted Netflix-style media app (repo `myronblair/kino-app`, private) branded **"StreamHoard"** in the actual UI — internal names (container names, some file paths) still literally say `kino`/`Kino`, that's cosmetic leftover from the original scaffold, not a second app. FastAPI + React + MongoDB. + +Deployed via Docker Compose in `/opt/kino` on CT104. Login: `admin@streamhoard.local` — password was rotated by the user via the app's own self-service "Change Password" feature (Settings page, requires current password) — not stored in this doc. + +## Container stack +| Container | Role | Port (host) | +|---|---|---| +| `streamhoard-frontend` | React UI, reverse-proxies `/api` to backend | 8080 | +| `streamhoard-backend` | FastAPI — auth, catalog, streaming, transcode queue | internal 8001 | +| `streamhoard-mongo` | MongoDB **pinned to 4.4** | internal | +| `streamhoard-gluetun` | VPN gateway (NordVPN/WireGuard) for the whole *arr stack + qBittorrent | proxies 7878/8989/9696/8082 | +| `streamhoard-radarr` / `-sonarr` / `-prowlarr` / `-qbittorrent` | Migrated **with existing config/library/history** from the old MediaStack VM — `network_mode: service:gluetun` so ALL their traffic is VPN'd, not just qBittorrent's | via gluetun | +| `streamhoard-portainer` | Docker management GUI | 9443 (https) | + +**Why mongo:4.4:** PVE1's physical CPU lacks AVX, and mongo:7+ hard-requires it (SIGILL crash-loop without this pin). + +## Media path +- NAS (`10.48.200.249:/volume1/video`) is mounted **on PVE1 itself** (`/mnt/nas-video`, `/etc/fstab`, `_netdev` — persists across PVE1 reboots), then passed into CT104 as an **LXC mountpoint** (`mp0`, config lives in `/etc/pve/lxc/104.conf` — persists across CT104 reboots too). +- **Why not mount NFS/CIFS directly inside CT104:** unprivileged LXCs can't reliably do NFS *or* CIFS client mounts even with the `mount=nfs;cifs` feature flag enabled — confirmed both fail with "Operation not permitted." This is a hard Proxmox/kernel-capability limitation, don't waste time retrying it — always go through the PVE1-host-mount + LXC-mountpoint pattern instead. +- Radarr/Sonarr root folders: `/mnt/nas/video/movies`, `/mnt/nas/video/tv` (unchanged from the old MediaStack setup). StreamHoard's backend container bind-mounts the same NAS path **read-only at the identical absolute path** (`/mnt/nas/video:/mnt/nas/video:ro`) so Radarr/Sonarr-reported `file_path` values resolve with zero translation. +- StreamHoard's own Radarr/Sonarr integration settings point at `http://gluetun:7878` / `http://gluetun:8989` (internal Docker network — gluetun's container shares its netns with radarr/sonarr, so other containers reach them via gluetun's hostname). +- NAS export ACL (`/etc/exports` on the NAS) now only allows **PVE1 (10.48.200.90)** — cleaned up the old MediaStack-only entry after that VM was destroyed. + +## Transcoding +Almost the entire imported library needed re-encoding — most source files are HEVC/H.265 video and/or DTS/EAC3 audio, neither browser-native. This is what causes playback "jitter"/silent-audio, **not** a network/NAS bottleneck (NAS read tested at 122 MB/s). Fix: +- Queued ABR transcode (re-encode to H.264/AAC multi-bitrate HLS) for the whole existing library via the transcode-queue API. +- Fixed a real code gap: `sonarr_import` was missing the auto-transcode-on-import hook that `radarr_import` already had (both now call `_enqueue_transcode` when the global `auto_transcode` setting is on). That setting is now globally `"abr"`, so future imports auto-queue transcoding with no manual step. +- If a newly-imported title plays choppy or silent, check its codec first (`ffprobe` inside `streamhoard-backend`) before assuming a server problem. + +## MSP360 backup (weekly full / daily incremental to the NAS) +- Installed via `dpkg-repack` from the working NovaCPX install (no public download URL for the Linux .deb worked — MSP360/CloudBerry doesn't publish a stable direct link). +- **Real gap hit and fixed:** the package's config templates (`cloudBackup.conf`, `wt_config.xml`) are consumed/deleted from disk after a machine's first run, so `dpkg-repack`ing an already-initialized install captures a package that can never bootstrap its own runtime state (`/opt/local/Online Backup//config/...`) — the daemon just retries forever waiting for a file that will never appear. Fix: copy the *generated* versions of those two files from the source machine's live runtime directory (not the package) into the new install's equivalent path before first start. +- Storage account: File System type, path `/mnt/nas-backups/MSPBackups` (same PVE1-mount + LXC-mountpoint pattern as the media NAS mount, separate mountpoint `mp1`). +- Free license, self-registered via CLI: `cbbV2 license activate -f -e -c`. +- Plan **StreamHoard-Backup**: daily incremental 2:00 AM, weekly full every Monday 2:00 AM, 4-week retention. Excludes: the huge media NAS mount, the backup destination itself, ephemeral Docker image layers (`overlay2`/`containers`/`image`/`buildkit`), and the regenerable HLS transcode cache (`/opt/kino/media/hls`, several GB, not "configuration"). +- **A leftover worker process from a killed/deleted plan will silently block a new plan's worker** (near-zero CPU usage despite the job showing "Running" for many minutes is the tell) — `ps aux | grep cbbWorker` and kill any process referencing an old/deleted plan ID before assuming a new plan is broken. +- CLI binary: `/opt/local/MSP360\ Backup/bin/cbb` (legacy, `plan -l`/`-r`/`-s -f`) and `cbbV2` (full plan create/edit/account management) — the space in the path needs quoting in every invocation. + +## Portainer +- `https://10.48.200.104:9443` — first-run setup token appears in `docker logs streamhoard-portainer`, and expires after ~5 minutes; restart the container to get a fresh one if it lapses. +- Endpoint renamed "StreamHoard" for clarity; local Docker socket auto-detected, no extra config needed. + +## JARVIS agent +Installed and registered (hostname `streamhoard104`, `watch_services: ["docker"]`) — see jarvis.md Agent System section. Watch out for the container's actual OS hostname needing to match what you want reported (a typo here caused a duplicate stale agent registration once). + +## Incidents hit during the 2026-07-23 migration (useful context if déjà vu happens elsewhere) +- **Live duplicate-IP conflict**: an unidentified device (SSH banner `OpenSSH_8.4p1`, NFS shares `TimeMachineBackup`/`Public`/`iTunes` — looks like an old Buffalo-brand NAS) was actively answering ARP for `10.48.200.35` alongside the real MediaStack VM's MAC. Not stale cache — a live second device on the wire, never identified/removed. Worked around by moving MediaStack to a different IP mid-migration (moot now, VM destroyed) — `.35` is still squatted, avoid it. +- That IP-reassignment **silently broke MediaStack's NordVPN kill-switch** (the `fwmark → routing table` `ip rule` entry vanished after the netplan change) — traffic briefly went out unprotected. Lesson: any NIC/netplan change on a host running NordVPN's Linux CLI app should be followed by `ip rule show`, not just `nordvpn status` — status can say "Connected" while the actual policy routing is gone. Fix: `nordvpn disconnect && nordvpn connect` reasserts the rule. +- The NAS's own NFS export root (`/volume1/video` itself) has mode `000` due to Synology's `hide` export option — real root (PVE1) bypasses this via DAC-override, an unprivileged LXC's "root" cannot, causing `Permission denied` even though subfolders were readable. Fixed with `chmod 755` on the mount root + `chmod 777` on `movies`/`tv`/`downloads` specifically (also sidesteps a UID mismatch between the NAS's real file ownership and whatever PUID/PGID the new containers use). +- gluetun needs `/dev/net/tun` explicitly passed into the LXC (`pct set 104 -dev0 /dev/net/tun`) or it fails immediately trying to create the WireGuard interface. + +## Not yet done / open items +- FortiGate VIP `tunnel-51820 / Wireguard-CT110` — **already cleaned up** (removed 2026-07-24 along with its dead firewall policy, confirmed Ref:0 before deletion). +- The rogue device on `10.48.200.35` was never identified or removed from the network. +- No reverse proxy/TLS in front of StreamHoard yet, and no LAN-only firewall rule (the app itself has no built-in rate limiting). diff --git a/voip.md b/voip.md new file mode 100644 index 0000000..77af8a2 --- /dev/null +++ b/voip.md @@ -0,0 +1,74 @@ +# VoIP Reference Card — FusionPBX / SignalWire / Landline + +## FusionPBX server +- **Host:** VM130 on PVE1, `10.48.200.130` — moved on-prem from an old DO droplet (`134.209.72.226`, now **destroyed** — do not reference that IP anymore, any doc saying FusionPBX is at `134.209.72.226` is stale). +- **Web UI:** `https://fusion.orbishosting.com` (NPM proxies to `10.48.200.130:80`) +- **SSH:** `root@10.48.200.130` via PVE1 hop, password `Joker1974!!!` (no longer needs the old DO-relay hop) +- **DB:** PostgreSQL (not MySQL), `fusionpbx` / `fusionpbx` / `RSfAjzefyYmufVqtnrYpz90yHaQ`, `psql -U fusionpbx -h 127.0.0.1 -d fusionpbx` +- Timezone: America/Chicago + +## SIP registration settings (for a device/softphone registering against this PBX) +| Setting | Value | +|---|---| +| SIP Server/Domain | `fusion.orbishosting.com` (or `10.48.200.130`) — **must be the domain name for some devices** (see FXO gotcha below), not always interchangeable with the IP | +| Port (LAN) | `5060` UDP/TCP | +| Port (WAN, per FortiGate forwards) | `5080` UDP or `5081` TCP/TLS | +| Auth username | extension number, e.g. `1000` | +| Auth password | per-extension, from `v_extensions.password` in the FusionPBX DB | + +Example working extensions (query `select extension, password, effective_caller_id_name from v_extensions;` for current full list): +| Ext | Who | +|---|---| +| 1000 | Myron Blair (Desk) | +| 1001 | Myron Blair | +| 1002 | Tommy Ivy | +| 1003 | Kitchen | +| 1004 | Master Bedroom | +| 1050 | Landline FXO gateway (Grandstream ATA, see below) | +| 1091 | "Landline" ring group (rings all 8 phones) | + +**Extension key/BLF provisioning:** category differs by phone model — T57W/T48S use `category=line` in `v_device_keys`, the AX86R WiFi phone only has a `programmable` keys section (no `line` section). Guessing wrong silently produces no key at all — check the actual Smarty templates at `/var/www/fusionpbx/resources/templates/provision/yealink//*.cfg` before assuming a key type. + +## SignalWire outbound trunk (carrier side — separate from the extension/phone side above) +FusionPBX registers **to** SignalWire as a SIP gateway (this is the outbound carrier trunk, not something a phone registers against): +- Gateway UUID: `c0e12aab-2406-44ee-ac38-6602116eaf2f`, username `orbis` +- Realm: `orbis-hosting-0364f5f67488.sip.signalwire.com` +- Space: `orbis-hosting.signalwire.com`, Project ID `16fbbcfd-1c94-4827-869c-0364f5f67488` +- Check status: `fs_cli -x "sofia status"` on VM130 — should show `REGED` for `external::c0e12aab-...` + +### SignalWire API gotchas (cost real debugging time — read before touching SignalWire) +1. **Credential types are NOT interchangeable.** The dashboard shows a `PSK_`-prefixed Signing Key at the top of the API Credentials page that looks like "the" token but gives a real 401 on classic REST calls. A `pat_`-prefixed Personal Access Token also 401s. **Only a named Project Token (`PT`-prefix, created in a separate table further down the same page) works** for REST Basic Auth (`ProjectID:PT...`) and the Relay SDK. +2. **Setting the SIP registration password requires the `relay` API, not the `Fabric` API.** `PUT /api/fabric/resources/sip_endpoints/{fabric_id}` returns HTTP 200 but does NOT affect registration auth. The correct call is `PUT https://{space}/api/relay/rest/endpoints/sip/{inner_sip_endpoint_id}` with `{"password":"..."}` — the inner sip_endpoint id is a *different* id than the Fabric resource id that wraps it. Symptom of using the wrong one: gateway stuck `FAIL_WAIT` with `[904] Operation has no matching challenge` even though the wire-level 401 challenge is valid. +3. **Outbound INVITE `From` header user-part must equal the digest auth username** (`orbis`), not the caller's DID/Caller ID. In FusionPBX this is `v_gateways.caller_id_in_from` — must be `false`. If wrong: INVITE gets 407, retries with Proxy-Authorization, gets ANOTHER 407 with the same nonce (looks like a FreeSWITCH bug, isn't). +4. **Destination must be full 11-digit E.164** (`+1` + 10 digits) — a dialplan bridge producing `+8172662022` (missing the `1`) gets `404`/`NO_ROUTE_DESTINATION`, easily misdiagnosed as "SignalWire doesn't recognize this number." +5. **Trial-tier gotcha:** even with the right token, sending SMS to an unverified number returns `422 integration_test_verified_caller_required`. +6. **FreeSWITCH `fs_cli chat` to a registered phone** needs the destination prefixed with the Sofia **profile name**: `chat sip|@|internal/@|` — just `user@domain` silently fails ("Invalid chat interface"). +7. **Diagnostic method that actually works for SIP auth issues:** independently reconstruct the digest (MD5 HA1/HA2/response) in a standalone script using the exact nonce/realm/uri from a live packet capture, compare byte-for-byte to what FreeSWITCH sent. Proves credentials/computation bug (mismatch) vs. carrier-side policy rejection (match, still rejected) before chasing further theories. + +## FortiGate SIP/RTP port forwards (VIPs, verify live via `show firewall vip` — the doc's static tables drift) +``` +fusion-sip-5080 (udp), fusion-sip-5080-tcp, fusion-sip-5081-tcp, fusion-sip-5081-udp, fusion-rtp (udp 16384-32768) + → all forward to 10.48.200.130 (FusionPBX) +``` +**WAN IP drifts silently and semi-regularly** — when it does, all VIP `extip` fields need updating, or calls will connect at the signaling level but have one-way/no audio (RTP arriving at the *real* current WAN IP with no matching forward rule). Always `curl ifconfig.me` from any on-prem host to check the *actual* live WAN IP before trusting a documented value. + +**FortiGate SIP ALG** (`default-voip-alg-mode`) must stay `proxy-based` (the default) — switching to `kernel-helper-based` breaks inbound call routing entirely (the inbound dialplan's `${sip_to_user}` handling is tuned around ALG being active). + +**FusionPBX's sofia.conf is cached independently of FreeSWITCH's process** at `/var/cache/fusionpbx/.configuration.sofia.conf` (Lua file cache) — any sofia profile/gateway DB or vars.xml change needs this file cleared too, not just a service/process restart. + +## Landline/FXO integration (Grandstream ATA, working as of 2026-07-18) +Old Flyingvoice FTA5111 (`10.48.200.119`) is **retired — do not reuse without a firmware fix from the vendor.** It has a 100%-reproducible crash-on-save bug (firmware V3.20) confirmed via extensive testing; not a config mistake. + +**Current device:** Grandstream HT8xx-family ATA at `10.48.200.45` (admin/Joker1974!!!), registered as **extension 1050**. + +FXO config (`/cgi-bin/config_a2`, Advanced Settings → FXO Port): +- Primary SIP Server = `fusion.orbishosting.com` — **must be the domain, not the IP** (extension 1050 is only provisioned under that domain name in FreeSWITCH's directory, even though `10.48.200.130` also technically exists as a domain). +- SIP User ID/Auth ID = `1050`, password = ext 1050's FusionPBX password. +- `PSTN Ring Thru FXS` = **No** (device defaults to Yes — rings a nonexistent local FXS phone instead of routing to SIP). +- Stage Method = default **2** (2-stage dialing) — gives the "landline button" real dial tone (SIP call seizes the line, ATA plays back real dial tone, DTMF passthrough). + +**Inbound PSTN→VoIP routing is on a DIFFERENT page** — Basic Settings (`/cgi-bin/config2`), NOT the FXO Port page: +- `Unconditional Call Forward to VOIP`: User ID = `1091` (the "Landline" ring group, rings all 8 phones), Sip Server = `fusion.orbishosting.com`, port `5060`. +- Without this field, incoming landline calls have nowhere to go once FXS-thru is disabled — easy to miss since the FXO page itself has no inbound-destination field. + +**Grandstream ATA web quirk:** login is `POST /cgi-bin/dologin`, but every subsequent settings-changing `POST /cgi-bin/update` needs a `session_token` hidden field (separate from login's `csrf_token`) scraped fresh from whatever config page you just loaded — omitting it silently bounces to a re-login page with the submission discarded, no error shown. Partial POSTs (only the fields you want changed) work fine.