diff --git a/public_html/admin/downloads/INFRASTRUCTURE-REFERENCE.md b/public_html/admin/downloads/INFRASTRUCTURE-REFERENCE.md index 0bbd093..287f0b5 100644 --- a/public_html/admin/downloads/INFRASTRUCTURE-REFERENCE.md +++ b/public_html/admin/downloads/INFRASTRUCTURE-REFERENCE.md @@ -1,5 +1,5 @@ -ssh: connect to host 10.48.200.90 port 22: Connection timed out -:** 2026-07-06 +# INFRASTRUCTURE REFERENCE — COMPLETE SYSTEM MAP +**Last Updated:** 2026-07-06 **Owner:** Myron Blair — myronblair@outlook.com --- @@ -40,11 +40,11 @@ HOME NETWORK (FortiGate router at 10.48.200.1) ├─► PVE1 Proxmox 10.48.200.90 (primary hypervisor) │ ├── VM 101 10.48.200.97 Home Assistant │ ├── VM 112 10.48.200.33 Jellyfin - │ ├── VM 113 10.48.200.35 MediaStack (Sonarr/Radarr/qBT/Prowlarr) + │ ├── VM 103 10.48.200.35 MediaStack (Sonarr/Radarr/qBT/Prowlarr) │ ├── VM 118 10.48.200.18 Homebridge │ ├── VM 120 10.48.200.110 NovaCPX hosting panel - │ ├── VM 210 10.48.200.210 Ollama (local LLM + vision) — llama3.1:8b, llava:7b - │ └── CT110 10.48.200.19 WireGuard exit container + │ ├── VM 106 10.48.200.210 Ollama (local LLM + vision) — llama3.1:8b, llava:7b + │ └── CT110 10.48.200.19/.67 WireGuard exit container (disabled at boot 2026-07-06, legacy/unused) │ ├─► PVE2 Proxmox 10.48.200.91 (secondary hypervisor) │ └── VM 302 10.48.200.99 NetworkBackup @@ -189,11 +189,12 @@ qm guest exec -- bash -c "cmd" # run command inside VM (requires QEMU ag | Field | Value | |-------|-------| | **IP** | 10.48.200.33 | -| **OS** | Ubuntu 22.04 LTS | +| **OS** | Ubuntu 24.04.4 LTS (verified 2026-07-06; previously logged as 22.04 — reinstalled/upgraded at some point) | | **SSH** | `ssh root@10.48.200.33` — password: `Joker1974!!!` (enabled 2026-06-14) | -| **Web UI** | `http://10.48.200.33:8096` | +| **Web UI** | `http://10.48.200.33:8096` (Jellyfin 10.11.11) | | **Purpose** | Media streaming server — Movies and TV shows | -| **JARVIS Agent** | Not yet installed | +| **JARVIS Agent** | `jarvis-agent.service` running (verified 2026-07-06) | +| **Remote access** | Tailscale installed, node `jellyfin-112` at `100.81.145.48` — used for off-LAN streaming access | **Media Libraries:** - Movies: `/mnt/mediastack/movies` — NFS from MediaStack (10.48.200.35:/media/movies) @@ -217,14 +218,15 @@ mount /mnt/mediastack/movies && mount /mnt/mediastack/tv --- -### VM 113 — MediaStack (PVE1) +### VM 103 — MediaStack (PVE1) | Field | Value | |-------|-------| | **IP** | 10.48.200.35 | -| **OS** | Ubuntu 24.04 LTS | +| **OS** | Ubuntu 24.04.4 LTS | | **SSH** | Via PVE1: `ssh -i /root/.ssh/id_rsa root@10.48.200.35` (no direct access from DO) | | **Purpose** | Automated media download pipeline + NFS server to Jellyfin | | **JARVIS Agent** | ID: `MediaStack_2c00b1b8` | +| **Not Docker** | Despite the name, all services below run bare-metal via systemd, not docker-compose | **Services:** | Service | Port | Login | API Key | @@ -236,8 +238,7 @@ mount /mnt/mediastack/movies && mount /mnt/mediastack/tv **All services run as root** — required by Synology NFS ACL (only root can write). -**VPN:** NordVPN — `nordlynx` WireGuard interface — exit IP 181.214.226.188 (US Dallas) -All download traffic exits via NordVPN. If downloads stall, check: `ip rule show` for rules 32764/32765. +**VPN:** NordVPN — `nordlynx` WireGuard interface — exit IP rotates (US Dallas servers), not a fixed IP (previous "181.214.226.188" was just a snapshot, not stable). All download/general traffic exits via NordVPN with LAN traffic exempted (Kill Switch + Firewall + LAN Discovery all `enabled` as of 2026-07-06 — see Section 9 for the full incident/fix history). If downloads stall, check `nordvpn status` first, then `ip rule show` for rules 32764/32765/table 205. **Media Flow:** ``` @@ -306,12 +307,12 @@ sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.90 \ --- -### VM 210 — Ollama Local LLM + Vision (PVE1) +### VM 106 — Ollama Local LLM + Vision (PVE1) | Field | Value | |-------|-------| | **IP** | 10.48.200.210 | | **OS** | Ubuntu (cloud image) | -| **SSH** | `ssh root@10.48.200.210` via PVE1 hop — password: `Joker1974!!!` | +| **SSH** | `ssh root@10.48.200.210` via PVE1 hop — password: `Joker1974!!!` (also reachable as `ssh myron@10.48.200.210` — password `Joker1974!`, then `sudo`). VM's name is `Ollama-95` but its real IP is `.210`, not `.95` — a prior version of this doc had a stray SSH line pointed at `.95` (nothing listens there); confirmed 2026-07-06 that `.210` is correct. | | **Purpose** | Local AI inference — chat (llama3.1:8b) + vision (llava:7b) | | **API** | `http://10.48.200.210:11434` (Ollama REST API) | | **JARVIS Agent** | ID: `ollama-ai_ubuntu` | @@ -340,6 +341,7 @@ Vision is enabled via: `/etc/systemd/system/jarvis-arc.service.d/vision.conf` | **IP** | 10.48.200.19 / 10.48.200.67 | | **Purpose** | Legacy WireGuard exit tunnel to DO (10.200.0.4 via wg-exit) — currently NOT used by MediaStack/Jellyfin | | **Note** | MediaStack uses NordVPN directly; Jellyfin uses wg1 peer on MediaStack for NFS only | +| **2026-07-06 incident** | MediaStack's `wg0` client for this tunnel (`/etc/wireguard/wg0.conf`) had a `PostUp` hook installing its own iptables kill-switch (`REJECT` all output not via `wg0` or marked `51820`), and `wg-quick@wg0.service` was still **enabled at boot** on MediaStack despite this tunnel being unused — this caused a full LAN/SSH lockout to MediaStack the moment that service came up. **Disabled `wg-quick@wg0` at boot on MediaStack** to prevent recurrence; the tunnel itself and CT110 are untouched. See NordVPN section below for the related (separate) LAN Discovery bug found in the same incident. | --- @@ -353,7 +355,7 @@ Vision is enabled via: `/etc/systemd/system/jarvis-arc.service.d/vision.conf` | **DSM Web UI** | `http://10.48.200.249:5000` | | **Purpose** | Primary media and download storage | -**NFS Share:** `/volume1/video` (exported to MediaStack only) +**NFS Share:** `/volume1/video` and its subpaths `/volume1/video/movies`, `/volume1/video/tv` — all exported to MediaStack (10.48.200.35) only, per `/etc/exports` on the NAS (verified 2026-07-06) **Directory structure:** ``` @@ -686,10 +688,18 @@ fs_cli -x "reloadacl" # reload ACL (safe) ### NordVPN — MediaStack Internet Traffic - Interface: `nordlynx` on MediaStack -- Exit IP: 181.214.226.188 (US Dallas) -- Policy routing: table 205 (all traffic via nordlynx), managed by `nordvpn-routing.service` +- Exit: rotating US (Dallas) NordVPN servers, not a fixed IP +- Policy routing: table 205 (non-LAN traffic via nordlynx), managed partly by NordVPN itself and partly by a custom `nordvpn-routing.service` unit (`/etc/systemd/system/nordvpn-routing.service`) that adds the `ip rule` for fwmark `0xe1f1` → table 205 - Required for IPTorrents access (blocks non-VPN IPs) +**2026-07-06 incident (fixed):** `nordvpnd` had been crash-looping since ~2026-06-07 (`/var/lib/nordvpn/data/settings.dat` was corrupted/empty) — meaning NordVPN was **not actually protecting MediaStack's traffic for about a month**; downloads were exiting on the plain home IP. Fixed by clearing the corrupt file, restarting the daemon, and re-logging in. + +While fixing this, found NordVPN's own **"LAN Discovery" setting was `disabled`**. With `Routing: enabled` and LAN Discovery off, connecting NordVPN's full-tunnel routing suppresses the main routing table's resolution for anything that would exit via `eth0` — **including same-subnet LAN traffic** — so the entire VM became unreachable (SSH/ping) from the rest of the LAN the moment NordVPN connected. Fixed with `nordvpn set lan-discovery on`, which adds explicit priority rules for private ranges (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `169.254.0.0/16`) ahead of the tunnel catch-all. Verified LAN reachability holds with Kill Switch + Firewall both re-enabled after this fix. + +**If MediaStack becomes unreachable after a NordVPN reconnect again:** check `nordvpn settings` for `LAN Discovery: enabled` first. If SSH is already dead, the network path itself is unusable — recover via the Proxmox host instead: `qm guest exec 103 -- /bin/bash -c 'export HOME=/root; nordvpn ...'` (guest-exec needs `HOME` set manually or the `nordvpn` CLI fails). + +Current confirmed-good settings (2026-07-06): Firewall `enabled`, Kill Switch `enabled`, Auto-connect `enabled`, LAN Discovery `enabled`, Routing `enabled`. + --- ## 10. BACKUP SYSTEMS @@ -768,13 +778,14 @@ sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.33 sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.110 # Ollama / Homebridge / NetworkBackup (myron user, then sudo) -sshpass -p 'Joker1974!' ssh myron@10.48.200.95 # Ollama +sshpass -p 'Joker1974!' ssh myron@10.48.200.210 # Ollama (NOT .95 - that's a naming leftover, nothing listens there) sshpass -p 'Joker1974!' ssh myron@10.48.200.18 # Homebridge sshpass -p 'Joker1974!' ssh myron@10.48.200.99 # NetworkBackup -# Run command inside VM via Proxmox (requires QEMU agent installed) +# Run command inside VM via Proxmox (requires QEMU agent installed; use VMID not IP) sshpass -p 'Joker1974!!!' ssh root@10.48.200.90 \ - 'qm guest exec 210 -- bash -c "CMD"' + 'qm guest exec 106 -- bash -c "export HOME=/root; CMD"' # Ollama = VMID 106 +# Other VMIDs: Jellyfin=112, MediaStack=103, JARVIS=211, NovaCPX=120, HomeAssistant=101, Homebridge=118 ``` **Password fallback order:** `Joker1974!@#` → `Joker1974!!!` → `Joker1974!`