ai-context/gotchas.md

# Critical Gotchas — Read Before Running Commands

## Unprivileged LXC containers (Proxmox) — mount limitations
- **NFS and CIFS client mounts both fail** inside an unprivileged LXC even with the `mount=nfs;cifs` features flag set — "Operation not permitted." This is a hard kernel-capability limitation, not a config mistake; don't retry it.
- **Fix:** mount the share on the PVE host itself (fstab, `_netdev`), then pass it into the container as an **LXC mountpoint** (`pct set <CTID> -mpN /host/path,mp=/container/path`). This persists across reboots on both the host and the container.
- A network share's root export directory can have restrictive permissions (e.g. Synology's `hide` NFS option sets mode `000` on the share root) that real root on the PVE host silently bypasses via DAC-override, but an unprivileged container's "root" cannot — causes `Permission denied` even though subfolders are readable. Fix on the source NAS/host, not in the container.
- Passing a TUN device (needed for WireGuard-based VPN containers like gluetun) needs explicit `pct set <CTID> -devN /dev/net/tun`.

## MSP360 / CloudBerry Backup
- CLI binary path has a space: `/opt/local/MSP360\ Backup/bin/cbb` (legacy CLI, `plan -l/-r/-s`) or `cbbV2` (full plan/account create/edit) — always quote it.
- **`dpkg-repack`ing an already-initialized install captures a package missing its own bootstrap config templates** (`cloudBackup.conf`, `wt_config.xml` get consumed/deleted from the package tree after first run). A fresh install from that repacked .deb hangs forever waiting on a file that will never appear. Fix: copy the *generated* versions of those files from the source machine's live runtime directory into the new install before first start.
- **A leftover worker process from a killed/deleted plan silently blocks a new plan's worker** — the tell is near-zero CPU usage despite the job showing "Running" for many minutes. `ps aux | grep cbbWorker`, kill anything referencing an old/deleted plan ID.
- Free/Freeware license self-registers via `cbbV2 license activate -f -e <email> -c` — no manual key needed.

## PHP / OpenLiteSpeed (DO server)
- **NEVER** use `lsphp -l` for syntax check — it segfaults. Use `php8.3 -l file.php`
- Run CLI scripts with: `/usr/local/lsws/lsphp85/bin/lsphp /path/script.php`
- If endpoint uses `ob_start()` + header.php pattern → add `ob_end_clean()` before CSV/JSON output
- MySQL charset: always `utf8mb4_unicode_ci` — mixing with `general_ci` breaks JOINs (error 1267)
- Cron bug: `*/3 * * * *` restarts `lshttpd` if any `.htaccess` is newer than `/usr/local/lsws/cgid` — nothing re-touches that dir after restart, causing a restart loop until you `touch /usr/local/lsws/cgid` manually.

## SSH / Networking
- DO server (165.22.1.228) **cannot reach the local network** (10.48.200.x) directly.
- PVE1 SSH via FortiGate DDNS works: `root@orbisne.fortiddns.com` (survives WAN IP changes, which happen fairly often — check `curl ifconfig.me` before trusting a documented WAN IP).
- PVE2 has no external port forward — only reachable locally or via cluster API through PVE1.
- Proxmox API port 8006 IS forwarded: `orbisne.fortiddns.com:8006`.
- FortiGate admin GUI is on **port 9443**, not 443 (443/80 both redirect there).
- A host's SSH interactive access can be intermittently flaky (e.g. through nested nested plink→ssh→ssh hops) while its own automated scripts using the same key work fine every time — that's usually not a real security/access problem, just a quirk of the ad-hoc path.

## JARVIS Agents
- Agent config: `/etc/jarvis-agent/config.json` | Runtime state: `/var/lib/jarvis-agent/state.json`
- **401 "Invalid agent key"** → state.json has a stale key. Fix: overwrite with correct `agent_id`/`api_key` from `registered_agents` table, then `systemctl restart jarvis-agent`.
- Metrics stored as JSON in `metric_data` column — use `JSON_EXTRACT(metric_data,'$.cpu_percent')`, no direct columns.
- **Set the container/VM's actual OS hostname correctly before first agent registration** — a hostname typo becomes part of the generated `agent_id` and creates a stale duplicate registration that needs manual DB cleanup later.

## Groq AI
- Model name: `compound-beta-mini` — NOT `groq/compound-beta-mini` (that's OpenAI router syntax, 404s).

## Proxmox
- `--nameserver` must be space-separated: `"8.8.8.8 1.1.1.1"` (comma causes a netplan bug).
- Run commands in a VM even when SSH/network is down: `qm guest exec <VMID> -- bash -c 'cmd'` (needs guest agent; also needs `export HOME=/root` for many CLI tools since guest-exec has no HOME env by default).
- Run commands in an LXC: `pct exec <CTID> -- bash -c 'cmd'`.

## Deploy
- Repos with auto-deploy webhooks (JARVIS, web-dashboard, StreamHoard's compose isn't webhook-deployed — it's manual `git pull` + `docker compose up -d`): always `git add + commit + push` — don't forget to also trigger the Gitea mirror-sync afterward (`curl -X POST -u myron:Joker1974!!! https://gitea.orbishosting.com/api/v1/repos/myron/<repo>/mirror-sync`).
- The 7 DO-hosted sites have **no auto-deploy** — manual SSH + git pull only.

## API Endpoint Auth (JARVIS)
- Netscan endpoint (`/api/netscan`) bypasses main auth — uses `X-Registration-Key` header.
- Admin portal uses a separate PHP session name (`jarvis_admin`) — different from the main JARVIS session.
- Cloudflare real IP: use `$_SERVER['HTTP_CF_CONNECTING_IP']` (or `HTTP_X_FORWARDED_FOR`), not `REMOTE_ADDR`.

## Network Scan
- The JARVIS "RUN NETWORK SCAN" button doesn't scan from wherever the browser is — it queues a shell command to the PVE1 agent, which runs nmap and pushes results to `/api/netscan`. Results appear ~40s after clicking.
- Chat "scan network" intent returns real DB data — never hallucinated.

## FusionPBX / VoIP
See [voip.md](voip.md) for the full reference — SignalWire has several non-obvious credential-type and API-endpoint gotchas that cost real debugging time in the past; don't re-derive them from scratch.
- SIP config changes need the sofia.conf Lua cache cleared, not just a service restart: `rm /var/cache/fusionpbx/<hostname>.configuration.sofia.conf`

## Git push confirmation
Pushing to a repo's default branch (master/main) needs an explicit, branch-named confirmation from the user — a general "yes" to "want me to push this?" isn't enough and will get blocked. Ask "confirm push directly to master?" up front to save a round-trip. If a chained `git add && commit && push` gets denied, nothing in the chain ran — re-check `git status` before retrying individual steps.

## Live log popup convention (JARVIS admin)
Any server-side action triggered from JARVIS admin (setup, deploy, install, run backup) must show a live log popup — NOT a `confirm()`/`toast()`. Pattern: `openModal()` + scrollable dark log div + backend log-polling endpoint (`{lines:[], next_line:N}`) + color-coded lines (green=ok, red=error, yellow=warn) + auto-detect completion + CLOSE button that stops polling. Poll every 2s, snapshot log position before triggering the action to avoid replaying old output. `runIntentGenerator()` in the admin JS is the canonical reference implementation.