myron/jarvis
1
0
Fork 0
mirror of https://github.com/myronblair/jarvis synced 2026-07-27 16:22:55 -05:00
Code Issues Packages Projects Releases Wiki Activity

Fix jarvis-health.sh: watchdog-restart alerts never deduped or auto-resolved

Browse Source 
source_key was minute-stamped (health:wd_restart:YYYYMMDDHHMM), so the same restart event seen across two 5-min cron runs within the logs 6-min lookback window got two different keys -> two alert rows + two emails, and the key never matched a clear_cond call so these rows stayed resolved=0 forever, accumulating in the active-alerts view.

Fixed to a stable key (health:wd_restart), matching the pattern used by the other three checks in this script: raise() now dedups via the existing COUNT..resolved=0 check, and clear_cond() runs when no recent restart line is found.

Verified live: injected a fake watchdog restart log line, ran the script twice -> exactly 1 alert row + 1 email (not 2). Removed the line -> alert auto-resolved (resolved=1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude
2026-07-08 07:43:20 -05:00
parent 8399048252
commit 141191bcdd
1 changed files with 3 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/jarvis-health.sh
+3 -1
View File
@@ -64,7 +64,9 @@ if [ -f "$WDLOG" ]; then
'match($0,/^\[([0-9-]+ [0-9:]+)\]/,m){ if(m[1]>=cutoff && /restarted successfully/) print }' "$WDLOG")
if [ -n "$RECENT" ]; then
SVC=$(printf '%s' "$RECENT" | grep -oE '(nginx|php8.3-fpm|mariadb|redis-server)' | sort -u | tr '\n' ' ')
raise "health:wd_restart:$(date +%Y%m%d%H%M)" "warning" "Service auto-restarted" "Watchdog restarted: ${SVC:-a service}. Investigate why it died."
raise "health:wd_restart" "warning" "Service auto-restarted" "Watchdog restarted: ${SVC:-a service}. Investigate why it died."
else
clear_cond "health:wd_restart"
fi
fi