ai-context/gotchas.md

Critical Gotchas — Read Before Running Commands

Unprivileged LXC containers (Proxmox) — mount limitations

• NFS and CIFS client mounts both fail inside an unprivileged LXC even with the mount=nfs;cifs features flag set — "Operation not permitted." This is a hard kernel-capability limitation, not a config mistake; don't retry it.
• Fix: mount the share on the PVE host itself (fstab, _netdev), then pass it into the container as an LXC mountpoint (pct set <CTID> -mpN /host/path,mp=/container/path). This persists across reboots on both the host and the container.
• A network share's root export directory can have restrictive permissions (e.g. Synology's hide NFS option sets mode 000 on the share root) that real root on the PVE host silently bypasses via DAC-override, but an unprivileged container's "root" cannot — causes Permission denied even though subfolders are readable. Fix on the source NAS/host, not in the container.
• Passing a TUN device (needed for WireGuard-based VPN containers like gluetun) needs explicit pct set <CTID> -devN /dev/net/tun.

MSP360 / CloudBerry Backup

• CLI binary path has a space: /opt/local/MSP360\ Backup/bin/cbb (legacy CLI, plan -l/-r/-s) or cbbV2 (full plan/account create/edit) — always quote it.
• dpkg-repacking an already-initialized install captures a package missing its own bootstrap config templates (cloudBackup.conf, wt_config.xml get consumed/deleted from the package tree after first run). A fresh install from that repacked .deb hangs forever waiting on a file that will never appear. Fix: copy the generated versions of those files from the source machine's live runtime directory into the new install before first start.
• A leftover worker process from a killed/deleted plan silently blocks a new plan's worker — the tell is near-zero CPU usage despite the job showing "Running" for many minutes. ps aux | grep cbbWorker, kill anything referencing an old/deleted plan ID.
• Free/Freeware license self-registers via cbbV2 license activate -f -e <email> -c — no manual key needed.

PHP / OpenLiteSpeed (DO server)

• NEVER use lsphp -l for syntax check — it segfaults. Use php8.3 -l file.php
• Run CLI scripts with: /usr/local/lsws/lsphp85/bin/lsphp /path/script.php
• If endpoint uses ob_start() + header.php pattern → add ob_end_clean() before CSV/JSON output
• MySQL charset: always utf8mb4_unicode_ci — mixing with general_ci breaks JOINs (error 1267)
• Cron bug: */3 * * * * restarts lshttpd if any .htaccess is newer than /usr/local/lsws/cgid — nothing re-touches that dir after restart, causing a restart loop until you touch /usr/local/lsws/cgid manually.

SSH / Networking

• DO server (165.22.1.228) cannot reach the local network (10.48.200.x) directly.
• PVE1 SSH via FortiGate DDNS works: root@orbisne.fortiddns.com (survives WAN IP changes, which happen fairly often — check curl ifconfig.me before trusting a documented WAN IP).
• PVE2 has no external port forward — only reachable locally or via cluster API through PVE1.
• Proxmox API port 8006 IS forwarded: orbisne.fortiddns.com:8006.
• FortiGate admin GUI is on port 9443, not 443 (443/80 both redirect there).
• A host's SSH interactive access can be intermittently flaky (e.g. through nested nested plink→ssh→ssh hops) while its own automated scripts using the same key work fine every time — that's usually not a real security/access problem, just a quirk of the ad-hoc path.

JARVIS Agents

• Agent config: /etc/jarvis-agent/config.json | Runtime state: /var/lib/jarvis-agent/state.json
• 401 "Invalid agent key" → state.json has a stale key. Fix: overwrite with correct agent_id/api_key from registered_agents table, then systemctl restart jarvis-agent.
• Metrics stored as JSON in metric_data column — use JSON_EXTRACT(metric_data,'$.cpu_percent'), no direct columns.
• Set the container/VM's actual OS hostname correctly before first agent registration — a hostname typo becomes part of the generated agent_id and creates a stale duplicate registration that needs manual DB cleanup later.

Groq AI

• Model name: compound-beta-mini — NOT groq/compound-beta-mini (that's OpenAI router syntax, 404s).

Proxmox

• --nameserver must be space-separated: "8.8.8.8 1.1.1.1" (comma causes a netplan bug).
• Run commands in a VM even when SSH/network is down: qm guest exec <VMID> -- bash -c 'cmd' (needs guest agent; also needs export HOME=/root for many CLI tools since guest-exec has no HOME env by default).
• Run commands in an LXC: pct exec <CTID> -- bash -c 'cmd'.

Deploy

• Repos with auto-deploy webhooks (JARVIS, web-dashboard, StreamHoard's compose isn't webhook-deployed — it's manual git pull + docker compose up -d): always git add + commit + push — don't forget to also trigger the Gitea mirror-sync afterward (curl -X POST -u myron:Joker1974!!! https://gitea.orbishosting.com/api/v1/repos/myron/<repo>/mirror-sync).
• The 7 DO-hosted sites have no auto-deploy — manual SSH + git pull only.

API Endpoint Auth (JARVIS)

• Netscan endpoint (/api/netscan) bypasses main auth — uses X-Registration-Key header.
• Admin portal uses a separate PHP session name (jarvis_admin) — different from the main JARVIS session.
• Cloudflare real IP: use $_SERVER['HTTP_CF_CONNECTING_IP'] (or HTTP_X_FORWARDED_FOR), not REMOTE_ADDR.

Network Scan

• The JARVIS "RUN NETWORK SCAN" button doesn't scan from wherever the browser is — it queues a shell command to the PVE1 agent, which runs nmap and pushes results to /api/netscan. Results appear ~40s after clicking.
• Chat "scan network" intent returns real DB data — never hallucinated.

FusionPBX / VoIP

See voip.md for the full reference — SignalWire has several non-obvious credential-type and API-endpoint gotchas that cost real debugging time in the past; don't re-derive them from scratch.

• SIP config changes need the sofia.conf Lua cache cleared, not just a service restart: rm /var/cache/fusionpbx/<hostname>.configuration.sofia.conf

Git push confirmation

Pushing to a repo's default branch (master/main) needs an explicit, branch-named confirmation from the user — a general "yes" to "want me to push this?" isn't enough and will get blocked. Ask "confirm push directly to master?" up front to save a round-trip. If a chained git add && commit && push gets denied, nothing in the chain ran — re-check git status before retrying individual steps.

Live log popup convention (JARVIS admin)

Any server-side action triggered from JARVIS admin (setup, deploy, install, run backup) must show a live log popup — NOT a confirm()/toast(). Pattern: openModal() + scrollable dark log div + backend log-polling endpoint ({lines:[], next_line:N}) + color-coded lines (green=ok, red=error, yellow=warn) + auto-detect completion + CLOSE button that stops polling. Poll every 2s, snapshot log position before triggering the action to avoid replaying old output. runIntentGenerator() in the admin JS is the canonical reference implementation.