There are two failure modes in monitoring small infrastructure, and organisations tend to move directly from one to the other without pausing at the useful middle.
The first is no monitoring: you find out the server is down because someone walks over and says the thing is not working. Everyone recognises this one as bad.
The second is what happens after somebody fixes the first. A tool is installed, every default alert is enabled, and within a fortnight the channel is producing forty messages a day about disk usage on a volume nobody cares about. People mute it. You are now functionally back at no monitoring, but paying for it and believing otherwise.
The second state is worse, because it comes with the confidence of having solved the problem.
The rule that fixes most of it
Every alert must correspond to an action a human would take, right now.
If the answer to “what would I do about this at 2am?” is “nothing” or “look at it in the morning,” it is not an alert. It might be a dashboard row, a weekly report, or a ticket — all fine destinations — but sending it to the place reserved for “something needs a human” devalues everything else in that channel.
Corollary: silence must mean success. If your pipeline messages you on every successful deploy, the failure message arrives into a stream you have already learned to skim. Nothing should notify to say things are fine.
What to actually collect
For small infrastructure — a handful of servers, some containers, a network — the useful set is small and stable.
Availability, from outside. The single highest-value check, and the one most often missing. An external service that requests your site every minute from somewhere that is not your network. It catches the entire class of failures where the host is perfectly healthy and the service is unreachable anyway: expired certificate, DNS change, firewall rule, upstream provider. Internal monitoring cannot see any of those, because it is on the wrong side of them.
Host basics. CPU, memory, disk, load. Disk is the one that actually pages people — it fills predictably and takes everything down with it — so alert on trajectory where you can (“will be full in 48 hours”) rather than only on a threshold, which tends to fire when it is already an incident.
Certificate expiry. Thirty days out, then seven. Automatic renewal is standard now, which is precisely why nobody notices when it silently stops working. This costs nothing to check and prevents an entirely avoidable outage.
Backup success — and restore success. Whether the job ran, and separately, whether a restore works. The first is easy and mostly reassuring. The second is the one that tells you the truth.
Application-level checks. Not “is the process running” but “does the thing work” — an endpoint that touches the database and returns a real answer. Processes stay alive while being completely useless with impressive regularity.
Logs, centrally, searchable. Not alerted on. Collected, so that when something does break you can answer “what happened at 03:12” without SSHing into four machines and hoping the file has not rotated.
What to skip
Be equally deliberate about what you do not do:
- Alerting on CPU spikes. Servers are supposed to use their CPU. Alert on the user-facing symptom — slow responses — not the resource.
- Per-container process alerts when the orchestrator already restarts them. You will get paged for something that self-healed before you opened your laptop.
- Metrics you will never plot. Every collected metric costs storage and attention. If you cannot name the question it answers, do not collect it yet.
- A tracing stack for four services. Distributed tracing is genuinely excellent and genuinely solves a problem you probably do not have.
Alert routing, briefly
Three destinations, and the discipline is in keeping them separate:
| Urgency | Example | Where it goes |
|---|---|---|
| Wake someone | Site down, backup target unreachable, disk full in hours | Phone — push or SMS |
| Same-day | Certificate expiring in seven days, one node of three unhealthy | Chat channel |
| Whenever | Patch available, disk at 60%, weekly vulnerability report | Email or a report |
The mistake is routing everything to the top row because it feels responsible. The result is that the top row stops meaning anything, and the genuinely urgent alert arrives in a channel that has been muted since March.
The test
Once a quarter, look at the alerts that actually fired and ask, for each one, whether anybody did anything.
Alerts nobody acted on are not free — they are the reason the next one gets ignored. Delete them, raise their threshold, or move them down a tier. A monitoring system that produces six meaningful alerts a year is doing considerably better than one producing six a day, and it is the only kind anybody still reads.
We design and run monitoring remotely as part of infrastructure security and cloud & DevOps work.