Keeping Alert Noise Off My Phone: An AI Triage System for My VPS

My VPS runs several dockerized services — a Traefik reverse proxy, n8n, and this website itself — but until recently it had no active monitoring at all. I only found out something was wrong when someone told me, or when I happened to load the page myself. Installing a monitoring tool sounds like the obvious next step, but I already knew the trap it sets: the moment a tool notifies on every little wobble — a brief 502 from a container restart, a transient error during a certificate renewal — the alerts that actually matter get buried in noise, and eventually you start ignoring notifications without even noticing you’re doing it. That’s alert fatigue, and I’d already run into it firsthand in a previous ops role. I didn’t want to repeat it in my own project.

So this time, the plan was: from the moment I installed any monitoring at all, put an AI triage layer between it and me. Claude reads the relevant docker logs and decides whether an alert is likely self-healing or genuinely needs my attention — only the second kind gets to make my phone buzz.

Where this sits: Phase 1 of a three-tier spec

During planning I sketched out three tiers up front — leanest-possible, stable-long-term, and industry-grade — to figure out how small, how sturdy, and how big this system could go, and to lay out a staged build plan. What’s actually built right now is Phase 1, the leanest tier: the goal is to validate the core assumption — “does AI triage actually help?” — with the least resources before investing further. Phase 1 reuses the n8n instance already running on the VPS and adds nothing but one lightweight Uptime Kuma container — no new database, no new backend service. Infrastructure cost: $0. Claude API (Haiku) usage comes out to roughly $3–5/month.

Architecture

Tech stack

1. n8n (already running — the workflow automation engine, visual nodes + JavaScript Code nodes)
2. Uptime Kuma v2 (newly added, watching the website itself and n8n, backed by SQLite)
3. Claude API (claude-haiku-4-5, called directly via an HTTP Request node against the Anthropic Messages API — more on why below)
4. Telegram Bot API (the primary channel, direct message)
5. Slack Incoming Webhook (a live fallback for when Telegram itself fails)
6. n8n Data Tables (native table storage in place of a real database, holding the service map and two pending-record tables)
7. SSH (a scoped key with a forced command, letting n8n safely run docker logs)

The triage pipeline

This is the live path, triggered directly by Uptime Kuma’s webhook. A completely separate workflow wakes up on its own Cron schedule, batches every low-priority record from that day into one summary, and sends it — deleting the batch only after a successful send, so a failed send just gets retried the next day instead of silently losing data.

Design decisions worth calling out

A request that fails the check doesn’t even get an error back. The n8n webhook URL carries a shared-secret query parameter, set in Uptime Kuma’s notification config. If the token doesn’t match, the workflow stops immediately — no response, no trace left behind — so this public endpoint can’t be probed or guessed into forging events that burn Claude quota and manufacture fake alerts.

A missing service mapping doesn’t stall the pipeline. Uptime Kuma’s monitor names don’t always match the actual docker container names, so I maintain a lookup table between the two. When a monitor has no mapping, the system doesn’t get stuck — it just skips log retrieval and hands Claude the raw alert instead (monitor name, error message, response time). That fallback also naturally covers pure URL/ping/SSL-certificate monitors that were never going to have a matching container in the first place.

Logs pass through three filters, then exactly one masking function. Nothing gets dumped to the model wholesale: only the 5 minutes of log preceding the alert, only lines matching a case-insensitive error/warn keyword pattern, capped at 100 lines so a crash loop can’t blow up token cost or message length. After that, before the content reaches Claude or Telegram, it passes through exactly one masking function — IPs, API keys, database connection strings, JWTs, and AWS access keys all get replaced. Both destinations see the same masked output; there’s no second set of rules and no unmasked copy flowing downstream.

If log retrieval itself fails, that’s treated as high priority — no AI involved. When SSH or docker logs fails outright — the container might be mid crash-loop, unreachable, or already renamed or removed — that failure is itself often evidence something’s wrong, and it’s safer to act on it directly than to let the model guess from a bare alert with insufficient context. So that path skips classification entirely and sends immediately:

function routeOnSshResult(sshExecutionResult) {
  const { exitCode, stdout, stderr, error } = sshExecutionResult;
  if (exitCode === 0) {
    return { route: 'normal', logs: stdout };
  }
  const errorMessage = stderr || error || `exit code ${exitCode}`;
  return { route: 'highPriorityBypass', reason: `log retrieval failed: ${errorMessage}` };
}

Classification is deliberately biased conservative — better a false alarm than a missed one. The single most important line in Claude’s system prompt is: when the evidence isn’t enough to confirm something already self-healed, classify it high rather than risk downgrading a genuine emergency to low. Logs showing the service already recovered get low; a truncated log doesn’t change the verdict, only what’s actually visible does; and a monitor with no logs at all (pure certificate checks) gets judged on the raw alert text alone.

Getting the classification right doesn’t matter if the notification never arrives, so sending gets its own safety net. A high-priority alert goes through three layers: Telegram retries first (3 attempts, exponential backoff); if that still fails, it immediately falls back to a Slack webhook; if even that fails, it’s written to a Data Table as a last resort, and the next day’s digest checks that table and flags the failed-send count at the top of the summary — so nothing actually vanishes without a trace.

How this got validated: an accuracy number, not a checkbox

This project deliberately didn’t force TDD onto everything uniformly — it split validation by what kind of logic was actually being tested. Pure functions (validation, filtering, masking, message assembly) got strict Red/Green/Refactor: all nine modules were written and tested with Jest outside n8n first, then pasted into their Code nodes once green. The AI classification logic got eval-style validation instead — 14 test scenarios (7 known-benign, 7 genuinely needing a human), built mostly from real Traefik logs off the actual VPS plus a few textbook cases like expired certificates and truncated logs, each run through the Claude API and scored against expected labels. And anything with no testable logic at all — the visual node wiring itself, real webhook triggers, SSH key setup — got manually triggered against real events and checked against real output instead.

Two consecutive eval runs both landed at 13/14 (92.9%), zero missed high-priority cases. The one scenario that didn’t pass was an ACME certificate renewal failure classified as high when it should have been low — a conservative miss, not the dangerous direction of missing something that actually mattered.

Real-world, end-to-end testing

With Phase 1 built, the last step was pulling out every piece of fake data and running the full pipeline against real Uptime Kuma events — not curl commands simulating a payload — through all three main paths, each confirmed by an actual Telegram delivery.

I actually stopped the website’s container for a few seconds and let Uptime Kuma’s real webhook fire. This turned up something I hadn’t expected: this Astro site emits no shutdown-related log output at all when docker stop runs, so docker logs --since 5m came back empty, and Claude classified it low. Next I temporarily deleted the row from the service map to simulate the no-mapping path — this time Claude had no log to check at all, only the raw alert (an HTTP 404), and correctly classified it high under the conservative-by-default rule; Telegram got the notification immediately. Finally I pointed the mapping at a nonexistent container name to trigger an SSH failure — the system skipped Claude entirely and sent a “high priority (log retrieval failed)” message right away, also confirmed delivered. I triggered the daily digest workflow manually too, and it correctly batched the low-priority record from the first test into one summary, sent it, and cleared the corresponding row only after the send succeeded.

This round of real testing also honestly surfaced a limitation I haven’t fixed yet: the system’s conservatism isn’t actually consistent across the two ways “there’s no useful evidence” can happen. When SSH succeeds but the logs come back empty, the model leans low. When there’s no service mapping at all and no logs to check, the same conservative-by-default rule leans high instead. In other words, a quiet service that emits nothing when it goes down is exactly the case at risk of being misclassified as low priority during a real outage. That’s a genuine open engineering question for a future iteration, not something “the AI is pretty accurate” papers over.

A few real stories from building this

Integration testing turned up an actual shell injection vulnerability: the SSH forced command on the VPS was splicing $SSH_ORIGINAL_COMMAND straight into a shell command with no validation — if that scoped key ever leaked, an attacker could smuggle a semicolon or backtick in and run arbitrary commands. The fix was a wrapper script that validates the container name format before running docker logs on its behalf, verified with an actual injection payload as a test case.

n8n itself had a few undocumented behaviors that only surfaced from real use: the SSH node’s exit code field is actually called code, not exitCode as the docs imply; and the SSH node always prepends a working-directory cd to whatever command you send, with no way to turn that off from n8n’s side. The daily digest workflow originally read its two Data Tables through chained, sequential nodes — and it turned out that if the first table in the chain happened to be empty, n8n would skip every node after it entirely, silently dropping the whole run even when the other table genuinely had data waiting to be summarized. Switching to parallel reads with an always-output-data option fixed that, but introduced a new problem: an empty table’s output went from zero rows to one row containing an empty object — fixed with one extra filter step downstream.

None of this is “wire a few n8n nodes together and bolt on an AI API call” — every part of it got shaken loose and fixed through actual runtime behavior, not just design on paper.

Why not n8n’s built-in Anthropic node

During planning I checked n8n’s dedicated Anthropic node and found its docs didn’t list which model IDs it currently supports or how it keeps up with new model releases — meaning if I wanted to use the latest model, I might be stuck waiting on a package update before it showed up in the node’s dropdown. I went with an HTTP Request node hitting Anthropic’s Messages API directly instead, writing the model string into the JSON body myself. That sidesteps any lag from the node package, and lines up better with a Phase 2 idea — routing simple cases to Haiku and harder ones to Sonnet — that needs that kind of flexibility anyway.

What’s next

If this holds up under continued real use, the next tier adds Prometheus + Alertmanager for wider monitoring coverage, stores classification results in Postgres for history and a human feedback loop (a button on each Telegram message to flag a misclassification), a Grafana dashboard tracking classification accuracy over time, and a separate watchdog workflow that checks whether this alert system itself has gone down. None of that is built yet — it’s only worth the investment once Phase 1 has actually proven itself.