My VPS Was Already Watching Itself — I Just Wasn't Reading the Logs

Same alert-triage project as before, second track of Phase 2 — Track A (the one auditing my own chatbot) watches whether my AI is behaving; this one watches whether my VPS itself is under attack.

Starting point: verifying what’s actually running, not guessing

Planning started with three obvious candidates — auth.log, fail2ban.log, ufw.log. Then I actually checked the machine and found it’s also running AIDE (file-integrity monitoring), auditd (the kernel audit daemon), and Lynis (a security-hardening scanner) — and all three are genuinely scheduled and running (systemctl is-active/is-enabled auditd shows it resident; lynis.timer and dailyaidecheck.timer fire daily). This VPS’s actual security posture is meaningfully higher than a typical personal-project box, so the monitoring scope expanded to six sources, and the design work expanded with it.

The one decision that mattered most: the signals that rules can already decide with certainty never go through the AI at all. Not “sent to the prompt with instructions not to downgrade it” — the AI never gets the chance to downgrade, because five finding categories (a burst of fail2ban blocks, AIDE file changes, an auditd privilege escalation, an auditd config-file change, a new Lynis warning) are produced directly by plain rule-based JS in src/vps-anomaly-report-builder.js. That’s a stronger guarantee than “the AI shouldn’t downgrade” — instruction-following is probabilistic, zero AI involvement is structural. Claude only ever touches the one thing in the whole system that genuinely needs semantic reasoning: judging whether a successful auth.log login’s source or timing looks anomalous, where there’s no simple “known-good IP list” to check against.

Architecture

38 nodes in the actual workflow — the diagram above is the shape, not a 1:1 node count.

The SSH wrapper: a fixed alias list, no path ever assembled from input

Different design from Track A’s regex validation: Track B’s wrapper (docs/n8n-vps-security-log-wrapper.sh) maps a fixed set of aliases to absolute paths (auth_log/fail2ban_log/ufw_log/aide_log/lynis_log/lynis_report/auditd_events) — the string n8n sends in never gets spliced into a file path at all, which is a stronger guarantee than regex- validating an arbitrary path. Adversarial testing (non-whitelisted commands, ../ path traversal) was rejected correctly every time.

Six real pitfalls, each stranger than the last

Lynis log permissions. /var/log/lynis.log and /var/log/lynis-report.dat are owned by root:root (not the adm group I expected) — a regular account can’t read them at all. Fixed with a narrowly scoped sudoers.d rule granting only this one SSH key’s account passwordless cat on those two exact absolute paths.

AIDE reports that were 71% Docker noise. AIDE’s Debian packaging (aide-common) uses a split rule-file mechanism with no dedicated rule for Docker, so /var/lib/docker falls under the 99_aide_root catch-all (/ 0 Full) — one report came back with roughly a million changes, and 71% of them (712,512) were just normal container overlay2/volume churn, completely burying anything worth actually looking at. After adding exclusion rules and rebuilding with aideinit, the clean report came back as Total 919483 / Added 5 / Removed 3 / Changed 148.

A frozen AIDE baseline. /etc/default/aide was set to COMMAND=update + COPYNEWDB=no — aide.db.new regenerates every day, but it was never being promoted to aide.db, so the comparison baseline stayed frozen at the very first aideinit, meaning every daily report kept re-reporting the same old changes. Fixed at the system level, deliberately kept out of the n8n pipeline’s hands (a pipeline shouldn’t get to decide when a system’s own security baseline moves forward): a small aide-db-rotate.sh hooks into dailyaidecheck’s CRONEXITHOOK extension point (only fires once the whole check completes cleanly), promotes aide.db.new to the new aide.db, archives the old one, and keeps the last 7.

The systemd-timer / cron.daily trap. This VPS runs on systemd (dailyaidecheck.timer), and /etc/cron.daily/dailyaidecheck itself starts with if [ -d /run/systemd/system ]; then exit 0; fi — calling that entry point directly does nothing at all. You either wait for the timer or manually run sudo systemctl start dailyaidecheck.service.

ausearch silently returning nothing — the longest debugging session of the whole build (Step B6). Under an SSH session with no PTY, ausearch silently reports <no matches> — even when real matching data exists, with no error at all. I ruled out stdin, locale, timezone, HOME/PATH, and sudo permissions in that order before finally isolating it by running the identical query with and without a PTY side by side. The build guide had assumed n8n’s SSH node had a PTY-allocation toggle; it turns out the node’s Execute a Command operation uses the ssh2 library’s exec() under the hood, and the UI has no PTY option at all (other users on the n8n community forum have hit the same wall — not unique to this project). The real fix lives entirely on the VPS side, in the wrapper script itself:

script -qec "ausearch ... 2>/dev/null" /dev/null

That opens its own pseudo-terminal regardless of whether the outer SSH channel allocated one, so it doesn’t need to wait for a future n8n feature. After the fix, the same query window returned 12.13 million characters across 69,134 lines, and parseAuditdEvents correctly parsed all 8,664 events out of it in 88ms with zero parse errors.

A timezone trap in the timestamps themselves. ausearch’s time-> field was originally parsed with a bare new Date(...) — with no timezone info in the string, that silently falls back to whatever timezone the execution environment happens to be in, and drifts quietly on any host not set to Asia/Taipei. Found by accident while writing the Step B8 build docs, fixed by parsing with an explicit +08:00, with a new regression test that passes even under TZ=UTC npx jest so it can’t silently depend on the local machine’s clock again.

The cross-source AI prompt: two more pitfalls

Timezone ambiguity made Claude doubt itself and emit two contradicting JSON blocks. The first version of the prompt sent raw UTC ISO timestamps straight to Claude without telling it what timezone the VPS is actually in — and Claude’s replies genuinely contained self-doubting text along the lines of “the input is in UTC, I’d need the timezone to convert it,” followed by one wrong JSON block, a re-analysis, then a second, correct one. Fixed by removing the ambiguity at the source instead of asking the model to do timezone math: the JS side now converts every timestamp to VPS local time (YYYY-MM-DD HH:MM:SS (UTC+8), via Intl.DateTimeFormat) before it ever reaches the prompt.

The eval script’s JSON extraction broke on that dual-JSON output. The original extraction logic used a regex to find a JSON block, which choked on the two-block output above and silently fell back to {flagged: []} — making two genuinely anomalous logins look like the AI had missed them, when it had actually judged correctly in its own text. Fixed by switching from regex to brace-depth counting: scan for every balanced top-level {...} block, then walk backward from the last one until the first one that both parses successfully and has a flagged array.

Validation: 8 scenarios (normal login, source/time-anomalous login, a fail2ban burst, an AIDE change — each with at least 2 variants) scored 8/8 (100%), zero missed high-priority signals. A 30-cycle stress test (3 real anomalies scattered among 27 normal cycles) confirmed cycleId still matched correctly at scale — Track A’s list-index-mistaken-for-an-ID bug didn’t repeat here.

The real first launch (2026-08-14)

The last step was importing the full 38-node vps-anomaly-triage-main workflow into the real n8n instance and letting it run for real. Since vps_log_cursor started empty, the first run was a full backfill (execution 347490):

• all 38 nodes reported success, resultData.error was null
• fail2ban: 6,951 newly blocked IPs (cumulative), AIDE: 892 file changes (cumulative), auditd privilege escalations: 227 entries (cumulative, mostly routine apt-key/dpkg system maintenance)
• Claude Haiku 4.5 correctly flagged user joey logging in from 4 different source IPs within the same cycle — its stated reasoning matched the rule-side summary exactly
• routing: 4 high-priority alerts (all delivered via the three-layer Telegram/Slack send, zero entries in failed_notifications) + 1 low-priority (a new Lynis warning, MAIL-8818, correctly written to low_priority_alerts for the next digest)
• all 5 source cursors plus the Lynis snapshot wrote back successfully
• the Step B6 no-PTY silent-failure problem did not recur under the real schedule — the fix held

General lessons from working with n8n

Enough of these turned out to be non-obvious that they’re worth pulling into their own list:

1. Zero items upstream means the downstream node never fires at all — not “fires with zero output.” Any node where “no new data this cycle” is a legitimate outcome needs an explicit Always-Output-Data or sentinel-item handling.
2. Before wiring any loop exit or IF/Switch branch, ask what happens if you don’t wire it — especially whether an unwired branch means the loop just stops there forever.
3. Split In Batches’ internal state is bound to the node itself for the whole execution — nesting it inside another loop is a known limitation, not an occasional bug. If you actually need that, extract it into its own sub-workflow from the start.
4. A test plan needs a “make it fail on purpose” case — happy-path-only testing won’t surface wiring defects.
5. If an LLM’s output needs to map back to source data, don’t give it an easily-misused substitute label — a human-readable index is exactly the kind of thing a model will echo back instead of the real ID. Small eval sets won’t catch this; it only shows up at volume.
6. Scan for hardcoded credentials before exporting workflow JSON — a Code node has no credential- reference mechanism the way an HTTP Request node does, so some existing designs just hardcode a token directly inside one. Export that workflow to JSON and commit it, and the plaintext credential goes straight into git history.
7. Some commands depend on a PTY, and that failure mode never reports an error — it returns something that looks legitimate but is empty, which is more dangerous than a clear failure. If an SSH node runs anything beyond a plain cat/ls, compare its output with and without a PTY before trusting it in production.

What’s next

The evolution directions the Phase 1 post sketched out (Prometheus/Alertmanager, Postgres history, a Grafana accuracy dashboard, an independent watchdog workflow) still haven’t been started. What’s directly reusable for whenever that next tier does get built: the AIDE-baseline-freeze fix (aide-db-rotate.sh), the multi-source cursor pattern behind vps_log_cursor, and the n8n integration-stability checklist above — all three came out of this round already proven against real production data.