myron/ai-context
1
0
Fork 0
mirror of https://github.com/myronblair/ai-context synced 2026-07-28 01:32:38 -05:00
Code Issues Packages Projects Releases Wiki Activity

Full rewrite for current infra state — StreamHoard consolidation, VoIP split out, dead references purged

Browse Source 
- Replace all Jellyfin/MediaStack/WireGuard-CT110 references (destroyed 2026-07-23) with
  StreamHoard/CT104 throughout; add streamhoard.md covering the full consolidated stack,
  MSP360 backup setup, and migration incidents
- Split FusionPBX/SignalWire/landline content into its own voip.md (was scattered across
  jarvis.md and sites.md, and referenced the old destroyed DO droplet IP)
- Fix stale facts: JARVIS webhook port (:1972 -> 80/https), rotated registration key/GitHub
  PAT, FortiGate admin port (9443 not 443), current WAN-IP-drift/VIP gotchas
- Add reference.md for repo inventory notes, assistant hard limits, ZeroTraceGPT IQ, payments
  migration status
- Update gotchas.md with unprivileged-LXC mount limitations and MSP360 dpkg-repack gotcha
This commit is contained in:
Myron Blair
2026-07-25 22:47:31 -05:00
parent f5446d3208
commit 4fad312f87
8 changed files with 398 additions and 231 deletions
Download Patch File Download Diff File Expand all files Collapse all files
jarvis.md
+72 -91
View File
@@ -4,132 +4,113 @@
- **Dashboard:** https://jarvis.orbishosting.com (login: myron / Joker1974!!!)
- **Admin portal:** https://jarvis.orbishosting.com/admin/ (same login)
- **DB:** `jarvis_db` on VM 211 localhost — user: `jarvis_user` / `J4rv1s_Pr0t0c0l_2026!`
- **phpMyAdmin:** https://jarvis.orbishosting.com/phpmyadmin (myron / Joker1974!!!)
- **GitHub repo:** myronblair/jarvis (auto-deploy on push to main)
- **GitHub repo:** `myronblair/jarvis` (private; auto-deploy on push to `master`)
## Server Location
JARVIS runs on **PVE1 VM 211** at `10.48.200.211` (migrated from DO 2026-06-18).
- **SSH:** `sshpass -p 'Joker1974!!!' ssh -o StrictHostKeyChecking=no root@10.48.200.211`
JARVIS runs on **PVE1 VM 211** at `10.48.200.211`.
- **Stack:** nginx / PHP 8.3 / MariaDB / Redis / Arc Reactor
- **Web root:** `/var/www/jarvis/`
- Traffic routed via NPM (10.48.200.200) → VM 211
- **Web root:** `/var/www/jarvis/` — this directory **is** the live git repo, push from it directly
- **TLS:** terminated by NPM (VM105, 10.48.200.200), proxy_host → `10.48.200.211:80`, Let's Encrypt cert. **The old `:1972` port reference is wrong/dead** — use plain `http://jarvis.orbishosting.com` (port 80) or the HTTPS URL above. GitHub webhook is `https://jarvis.orbishosting.com/webhook.php`.
## File Structure (on VM 211 at /var/www/jarvis/)
```
public_html/
index.html — main Iron Man HUD (all UI)
api.php — API router
admin/index.php — admin portal (single PHP+JS file)
admin/index.php — admin portal (single ~5000-line PHP+JS file, deliberately left unsplit — see gotchas)
admin/downloads/INFRASTRUCTURE-REFERENCE.md — master live infra doc, gitignored, edit here first
agent/ — agent installers
api/
config.php — all credentials/constants (gitignored)
lib/db.php — JarvisDB class (query/execute/single/insert)
lib/kb_engine.php — KBEngine intent matching
lib/db.php — JarvisDB class (query/execute/single/insert)
lib/kb_engine.php — KBEngine intent matching
endpoints/
agent.php — agent registration/heartbeat/metrics/commands
chat.php — 4-tier chat: KB→action intents→Ollama→Groq→Claude
network.php — network device list + scan endpoint
netscan.php — push endpoint for PVE1 nmap results (no auth needed)
do_server.php — reads /proc directly (no SSH loopback)
stats_cache.php — every 5min cron: Proxmox cluster API, HA, weather, news
agent.php — agent registration/heartbeat/metrics/commands
chat.php — tiered chat: KB → action intents → Ollama → Groq → Claude
network.php — network device list + scan endpoint
netscan.php — push endpoint for PVE1 nmap results (uses X-Registration-Key, bypasses main auth)
stats_cache.php — every 5min cron: Proxmox cluster API, HA, weather, news
facts_collector.php — every 3min cron: system stats, site health
system.php — local system metrics
alerts.php — alert CRUD + auto-generate
news.php — serves api_cache['news'] + custom kb_facts(category='custom_news')
kb_intent_generator.php — every 6h cron, self-expanding knowledge base (see below)
alerts.php — alert CRUD + auto-generate
deploy/
reactor.py — Arc Reactor source (copy to /opt/jarvis-arc/reactor.py + restart)
reactor.py — Arc Reactor source (copy to /opt/jarvis-arc/reactor.py + `systemctl restart jarvis-arc` to deploy)
```
## Arc Reactor (AI Routing Daemon)
- **Daemon:** `/opt/jarvis-arc/reactor.py` (Python, port 7474), service: `jarvis-arc`
- **Deploy:** copy `deploy/reactor.py` → `/opt/jarvis-arc/reactor.py` + `systemctl restart jarvis-arc`
- Vision with actual screenshot → Claude `claude-opus-4-8-20251101`
- Guardian/SITREP/Vision text-only → Groq `llama-3.3-70b-versatile`
- Email drafting / research → Claude
## Arc Reactor (AI routing daemon)
- **Daemon:** `/opt/jarvis-arc/reactor.py` (Python venv), service: `jarvis-arc`, port 7474
- **Logs:** `/var/log/jarvis/arc.log` (setup log: `arc-setup.log`)
- **Vision cascade:** Claude API first; if Claude credits depleted, falls back to Ollama `llava:7b` (config via `/etc/systemd/system/jarvis-arc.service.d/vision.conf` → `OLLAMA_VISION_MODEL=llava:7b`)
- Secrets (Claude/Groq keys, DB pass, Gmail/iCloud app passwords) live in `/etc/jarvis-arc/reactor.env` (root:www-data 0640, loaded via systemd EnvironmentFile drop-in) — NOT in the repo.
## AI tiers (chat + reactor)
```
Tier 0.7: KB intents / planner
Tier 1: Knowledge Base (MySQL, self-expanding — see KB Generator below)
Tier 1.5: Ollama llama3.1:8b (10.48.200.210:11434) — CPU-only, ~30s+/reply, can 504 on default 60s nginx timeout
Vision: Ollama llava:7b (via Arc Reactor vision cascade)
Tier 2: Groq (model name: compound-beta-mini — NOT groq/compound-beta-mini, that 404s)
Tier 3: Claude API (fallback; credits need periodic refill at console.anthropic.com)
```
Groq is effectively the fast/primary path when Claude credits are depleted — Ollama is too slow for interactive chat (504s).
## Agent System
- **Registration key:** `f846a9aaf7ce9a61742c63c87c4186052a71d2a580c65518`
- **Install one-liner (Linux):** `curl -sk https://jarvis.orbishosting.com/install-agent.sh | bash -s <hostname> <linux|proxmox>`
- **Agent config:** `/etc/jarvis-agent/config.json` (key: `jarvis_url`) — runtime state at `/var/lib/jarvis-agent/state.json`
- **If agent gets 401 "Invalid agent key":** state.json has stale key — overwrite with correct agent_id + api_key from DB
- **Heartbeat:** every 10s | **Metrics:** every 30s
## Currently Online Agents
| agent_id | hostname | IP | type |
|----------|----------|----|------|
| jarvis-do_orbis.or | jarvis-do | 165.22.1.228 | linux |
| claude_pve | pve1 | 10.48.200.90 | proxmox |
| pve2_pve2 | pve2 | 10.48.200.91 | proxmox |
| networkbackup_NetworkB | networkbackup | 10.48.200.99 | linux |
| homeassistant_ha | homeassistant | 10.48.200.97 | homeassistant |
| homebridge_b57cbaea | homebridge | 10.48.200.18 | linux |
| novacpx_e3b07264 | novacpx | 10.48.200.110 | linux |
| jellyfin_7e386833 | jellyfin | 10.48.200.33 | linux |
| MediaStack_2c00b1b8 | mediastack | 10.48.200.35 | linux |
| fortigate_gw | fortigate | 10.48.200.1 | linux |
| yealink_t48s | yealink | 10.48.200.43 | linux |
- Agents register via `/api/agent` using a shared **registration key** (rotated multiple times historically — always read the current value from `/var/www/jarvis/api/config.php` `AGENT_REGISTRATION_KEY`, don't trust an old hardcoded value from any doc). Per-agent heartbeats then use their own `api_key` from the `registered_agents` table, not the shared reg key.
- **Agent script:** `/opt/jarvis-agent/jarvis-agent.py` (Python3, no external deps), config at `/etc/jarvis-agent/config.json`:
```json
{
"jarvis_url": "http://10.48.200.211",
"ssl_verify": false,
"registration_key": "<current key>",
"hostname": "<host>",
"agent_type": "linux",
"poll_interval": 30,
"heartbeat_every": 10,
"watch_services": ["<systemd unit names to health-check>"]
}
```
Systemd unit `jarvis-agent.service` (Restart=always). State file `/var/lib/jarvis-agent/state.json`.
- **401 "Invalid agent key"** → state.json has a stale key; delete it and restart the service to force re-registration, or manually overwrite with the correct `agent_id`/`api_key` from the DB.
- Agent auto-detects capabilities: `docker` (if docker binary present), `proxmox`, `ollama`, `homeassistant`.
- **Known registered agents (not exhaustive, check `registered_agents` table for current truth):** PVE1, PVE2, NetworkBackup, Home Assistant, Homebridge, NovaCPX, Ollama, StreamHoard/CT104 (added 2026-07-23, hostname `streamhoard104`). Jellyfin/MediaStack/WireGuard agents no longer exist (VMs destroyed).
## Network Scanning
- PVE1 cron: `*/3 * * * * /usr/local/bin/jarvis-netscan.sh`
- Script runs nmap, parses output, POSTs JSON to `https://jarvis.orbishosting.com/api/netscan` with `X-Registration-Key` header
- Scan Now button queues shell command to PVE1 agent (picks up within 10s)
- PVE1 cron `*/3 * * * *` runs `jarvis-netscan.sh` → nmap → POSTs JSON to `/api/netscan` with `X-Registration-Key` header.
- Chat "scan network" intent returns real DB data, never hallucinated.
## Chat Architecture
```
Tier 0: HA entity control (fuzzy match → call HA API)
Tier 0.5: Network device management
Tier 0.7: Planner intents (add task, my tasks, schedule, daily briefing)
Tier 1: KB intent engine (response type → instant reply)
Tier 1b: Action intents (network_scan → DB data + queue PVE1 scan)
Tier 2: Ollama llama3.2 at http://10.48.200.210:11434 (5s timeout)
Tier 3: Groq compound-beta-mini (cloud, fast)
Tier 4: Claude API fallback
```
- Groq model name: `compound-beta-mini` (NOT `groq/compound-beta-mini`)
- HA_URL = `http://orbisne.fortiddns.com:8123`
## DB Key Tables
## Chat Architecture / DB
```sql
registered_agents — agent_id, hostname, agent_type, ip_address, api_key, status, last_seen, version
agent_metrics — agent_id, metric_type, metric_data(JSON), recorded_at
-- Extract: JSON_EXTRACT(metric_data,'$.cpu_percent'), JSON_EXTRACT(metric_data,'$.memory.percent')
-- NO cpu_pct/mem_pct columns — always use JSON_EXTRACT
agent_metrics — agent_id, metric_type, metric_data(JSON), recorded_at -- use JSON_EXTRACT, no direct cpu/mem columns
agent_commands — agent_id, command_type, command_data(JSON), status(pending/delivered)
network_devices — ip, mac, hostname, alias, device_type, status, last_seen
alerts — alert_type, title, message, severity, resolved
kb_facts — category, fact_key, fact_value
kb_facts — category, fact_key, fact_value -- fact_value is MEDIUMTEXT (was TEXT, HA entity-map JSON outgrew it and silently failed writes for days before this was caught)
kb_intents — intent_name, pattern(regex), response_template, action_type, priority, active
api_cache — cache_key(proxmox/news/weather/ha_entities), data(JSON), updated_at
kb_generator_topics — id, topic_id, category, topic_name, description, active, run_count, last_run_at
tasks — title, notes, category, priority, status, due_date, due_time, completed_at
appointments — title, description, category, start_at, end_at, location, all_day, reminder_min
ha_entities — real-time HA entity states (updated by HA custom component)
```
## Proxmox API
- `stats_cache.php` uses cluster API at `orbisne.fortiddns.com:8006`
- Returns all VMs from both PVE1 and PVE2 via `/cluster/resources?type=vm`
- Token: `root@pam!jarvis=c45b5feb-f9a9-445d-a626-14fbb959f78b`
## Cron Jobs (on VM 211)
```
*/3 * * * * php /var/www/jarvis/api/endpoints/facts_collector.php
*/5 * * * * php /var/www/jarvis/api/endpoints/stats_cache.php
0 7 * * * /usr/local/bin/jarvis-backup.sh
```
**Cron on PVE1:**
```
*/3 * * * * /usr/local/bin/jarvis-netscan.sh
```
## KB Intent Generator (self-expanding knowledge base)
- Cron `0 */6 * * *`, processes 25 topics/run (BATCH_SIZE) cycling through `kb_generator_topics` table.
- Topic count grew from 140 → 4530 across 170 categories (general knowledge, Fortune-500 companies, space/exploration, etc.) — at 25/run a full cycle now takes ~45 days; raise BATCH_SIZE if freshness matters.
- Admin UI "MANAGE TOPICS" button — browse/add/edit/delete/toggle topics.
- Groq model `llama-3.3-70b-versatile`, 20 intents/call, max_tokens=5000 (raising from 40/3500 fixed an 88% truncation failure rate).
## Backups
- Script: `/usr/local/bin/jarvis-backup.sh`
- Output: `/var/backups/jarvis/jarvis_backup_YYYY-MM-DD_HH-MM-SS.tar.gz`
- Daily at 7am UTC (2am CDT), 7-day retention
- Downloadable from admin portal → BACKUPS tab
- Script: `/usr/local/bin/jarvis-backup.sh` → `/var/backups/jarvis/jarvis_backup_TIMESTAMP.tar.gz`, daily ~7am UTC, 7-day retention, downloadable from admin panel.
- Also captures `/etc/jarvis-arc` (reactor.env), `/etc/jarvis` (db.env), service drop-ins, `/usr/local/bin/jarvis-*.sh`.
- Health monitoring (`jarvis-health.sh`, PVE1 cron */5) checks stalled crons, stuck arc_jobs, disk >85%, watchdog restarts — auto-resolving `alerts` rows + email to `myronblair@gmail.com`.
## API Auth
- Main JARVIS API: session token via `X-Session-Token` header (or PHP session)
- Agent endpoints: `X-Agent-Key` header (per-agent key from registered_agents.api_key)
- Agent endpoints: per-agent key via header (see Agent System above)
- Netscan endpoint: `X-Registration-Key` header (shared registration key)
- Admin portal: separate PHP session (`session_name('jarvis_admin')`)
- Cloudflare passes real client IP in `CF-Connecting-IP` header
- Admin portal: separate PHP session name (`jarvis_admin`)
- Cloudflare passes real client IP in `CF-Connecting-IP` header (or `HTTP_X_FORWARDED_FOR`)
## Security posture (Phase 1 hardening, 2026-07-07 — mostly done, some user follow-up still open)
- Secrets pulled out of git-tracked files into `/etc/jarvis-arc/reactor.env` and `/etc/jarvis/db.env`. **Still pending (user action): rotate Claude/Groq/Gmail-app-pw/iCloud-app-pw/DB-pass since they remain in GitHub history**, and do a `git filter-repo` + force-push history purge.
- Login: Redis per-IP rate limit (10 fails/15min), session_regenerate_id on success.
- `INFRASTRUCTURE-REFERENCE.md` is gitignored/untracked in the repo — lives only on-disk on VM211 (and its VM110 mirror copy), not in GitHub.