source_key was minute-stamped (health:wd_restart:YYYYMMDDHHMM), so the same restart event seen across two 5-min cron runs within the logs 6-min lookback window got two different keys -> two alert rows + two emails, and the key never matched a clear_cond call so these rows stayed resolved=0 forever, accumulating in the active-alerts view.
Fixed to a stable key (health:wd_restart), matching the pattern used by the other three checks in this script: raise() now dedups via the existing COUNT..resolved=0 check, and clear_cond() runs when no recent restart line is found.
Verified live: injected a fake watchdog restart log line, ran the script twice -> exactly 1 alert row + 1 email (not 2). Removed the line -> alert auto-resolved (resolved=1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jarvis-backup.sh now also archives /etc/jarvis-arc (reactor.env), /etc/jarvis (db.env), the jarvis-arc.service.d systemd drop-ins, and /usr/local/bin/jarvis-*.sh. Previously a restore would have come back with no API keys or DB password since those moved outside /var/www/jarvis + /opt/jarvis-arc during the secrets sweep.
Verified: ran a real backup (45M) and confirmed all new paths are present in the archive.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
deploy/jarvis-health.sh (cron */5): detects silent failures the service watchdog cannot — stalled crons (cron.log >10min stale = 2+ missed runs), Arc jobs stuck running >30min, disk >85%, and watchdog service restarts. Writes auto-resolving rows to the alerts table (shown in admin panel) and emails myronblair@gmail.com on any NEW finding via the reactor Gmail SMTP creds.
Verified live end-to-end: injected a fake stuck job -> alert raised + email sent; cleared it -> alert auto-resolved. Installed in root crontab.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Removed live secret literals from git-tracked code (all were on GitHub):
- deploy/reactor.py: Claude/Groq API keys, DB pass, Gmail/iCloud app passwords now from os.environ (loaded via systemd EnvironmentFile=/etc/jarvis-arc/reactor.env, root:www-data 0640)
- public_html/login.php: used a private hardcoded PDO connection; now uses config.php DB_* constants
- deploy/jarvis-backup.sh (runs via cron), jarvis-deploy.sh, jarvis-watchdog.sh: DB pass now sourced from /etc/jarvis/db.env (root:root 0600)
- removed dead agent/jarvis-arc-reactor.py (unreferenced old duplicate leaking an old Groq key + stale Ollama IP)
- added deploy/reactor.env.example and deploy/db.env.example templates
Verified live: reactor restarted with all 21 handlers + DB poller (job round-trip OK), login works, mysqldump auth via env OK.
NOTE: these keys remain in GitHub history and should be rotated (Claude/Groq/Gmail/iCloud/DB). Separate decision needed on INFRASTRUCTURE-REFERENCE.md (full cred doc still tracked) + history purge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude