2 Commits

Author SHA1 Message Date
Claude 141191bcdd Fix jarvis-health.sh: watchdog-restart alerts never deduped or auto-resolved
source_key was minute-stamped (health:wd_restart:YYYYMMDDHHMM), so the same restart event seen across two 5-min cron runs within the logs 6-min lookback window got two different keys -> two alert rows + two emails, and the key never matched a clear_cond call so these rows stayed resolved=0 forever, accumulating in the active-alerts view.

Fixed to a stable key (health:wd_restart), matching the pattern used by the other three checks in this script: raise() now dedups via the existing COUNT..resolved=0 check, and clear_cond() runs when no recent restart line is found.

Verified live: injected a fake watchdog restart log line, ran the script twice -> exactly 1 alert row + 1 email (not 2). Removed the line -> alert auto-resolved (resolved=1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 07:43:20 -05:00
Claude 652e44f7b3 Add JARVIS health self-check (Phase 2 reliability monitoring)
deploy/jarvis-health.sh (cron */5): detects silent failures the service watchdog cannot — stalled crons (cron.log >10min stale = 2+ missed runs), Arc jobs stuck running >30min, disk >85%, and watchdog service restarts. Writes auto-resolving rows to the alerts table (shown in admin panel) and emails myronblair@gmail.com on any NEW finding via the reactor Gmail SMTP creds.

Verified live end-to-end: injected a fake stuck job -> alert raised + email sent; cleared it -> alert auto-resolved. Installed in root crontab.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 20:43:37 -05:00