Server monitoring: the four checks worth alerting on

An alert that fires often and means nothing is worse than no alert at all, because it teaches you to dismiss the whole category. After the third false page in a week you are not reading them any more, you are clearing them, and the one that mattered lands in the same folder as the noise.

So the goal is never more monitoring. It is a short list of checks that are true every time they fire, that you actually read, and that each have something you would do about them. For one server that list is four checks and about six thresholds. This post is what those four are, how to run them from outside the machine so they catch the failures that matter, what numbers to put in the thresholds, where the alert should land, and the single rule that stops the list growing back into wallpaper.

The four checks worth having#

Is the process running, asked from outside the machine. An agent on the box cannot tell you the box is gone. A check that runs somewhere else catches the whole class of failures that a local agent sleeps through: the container stopped, the node lost power, the network path broke, the firewall rule changed under you. This is the only check that catches a total outage, and it is the one most single-server setups skip because the panel already draws a green dot.

Does it answer. A TCP connection succeeding is not the same as the service working. On a panel-based host the port can be held open by the proxy in front of your container while the application behind it is in a crash loop. On a game server the game port can accept packets while the world is deadlocked and every player is timing out. Speak the actual protocol: request a URL and check the status code, or send the game's query packet and check that a player count comes back.

Is the disk filling. This is the failure with the longest warning and the least excuse. Logs, backups left on the same volume, a world that grows every week, a node_modules that gets reinstalled without the old one being cleared - disk creeps up over weeks and then everything stops at once, usually in the ugliest possible way. A database that cannot write is worse than a database that is down, because it can corrupt on the way out.

Did the last backup complete. Not "is the backup job scheduled" - whether a file of a plausible size appeared where it should, at the time it should. Backup jobs fail silently more often than any other job on a server, because nothing downstream depends on them until the day everything does. Backups that actually restore makes the wider case; the check here is the cheap half of it.

That is the list. Everything else - CPU graphs, memory graphs, tick rate, request latency, player counts - is diagnosis, not alerting. Keep the graphs, look at them the moment an alert fires, and do not page on them. The distinction between "I need to know now" and "I want to see this when I am looking" is the whole discipline.

Checking from outside the machine#

Two things make an external check worth having: it speaks the real protocol, and it does not alert on its own hiccups.

# HTTP: status code and total time, failing loudly on anything but 2xx$ curl -fsS --max-time 5 -o /dev/null \    -w '%{http_code} %{time_total}s\n' https://app.example.com/healthz200 0.184s

# Minecraft: the server list ping, which needs the game to be answering$ mcstatus play.example.com:25565 status# Any TCP port, when you have nothing else installed$ nc -z -w 3 203.0.113.10 25565 && echo open

mcstatus comes from the Python package of the same name and prints the version string and the player count. For Source engine games the equivalent is an A2S query on the query port, which most uptime services and every server list site already speak. If your monitor only offers "TCP port check", use it, but know what it is not telling you - see game server ports explained for why the query port and the game port fail independently.

Settings that decide whether the check is trustworthy:

• Interval of 60 seconds for most things. Thirty if the service is customer-facing and you will genuinely act faster. Anything under thirty seconds is load, not information.
• Three consecutive failures before alerting. That is three minutes to detection on a one-minute interval, which is fine, and it removes almost every false alarm caused by the monitor's own network.
• A timeout of 5 to 10 seconds, not the default 30. A server taking 25 seconds to answer is down as far as anyone using it is concerned.
• Two locations if the service offers it. One prober having a bad minute should not be an outage.
• Alert on recovery too. An alert with no matching "back up" line leaves you checking manually, which is the habit you were trying to remove.

Thresholds that are not guesses#

The numbers below are starting points that hold for a single server on a fixed plan. Adjust them once, after you have watched the graphs for a fortnight, and then leave them alone.

Two of those need comment. Memory is measured against your plan's limit, not the machine's, and what happens at the limit differs by host: here the container is stopped by the kernel and restarted clean rather than being allowed to swap, so a memory alert and a restart alert often arrive together and mean one thing. CPU is a hard throttle to the share you bought, so 100% is slow rather than broken - it is never grounds for suspension and it is not an emergency, but a server that lives there needs either tuning or a bigger plan. Reading a server load graph covers telling a healthy peak from a flat ceiling, and CPU vs RAM for game servers covers which one to buy more of.

The backup check is the one worth scripting even if you monitor nothing else:

# Alert if no backup larger than 1 MB has landed in the last 26 hours$ find /srv/backups -type f -mmin -1560 -size +1M | grep -q . || \    curl -fsS -H 'Content-Type: application/json' \      -d '{"content":"No backup in the last 26 hours"}' "$DISCORD_WEBHOOK"

The inverted version of that is better still and is what the hosted "cron monitoring" services sell: the backup job calls a URL when it finishes successfully, and the monitor alerts when the call stops arriving. A dead man's switch catches the case a find never will, which is the job not running at all because the scheduler itself is broken.

Reading the graphs before an alert fires#

Alerts tell you something is wrong now. Graphs tell you what normal looks like, which is the only way to know that something is wrong at all. The console here draws memory, CPU and disk against the limits of your plan rather than against the machine's capacity, which is the correct comparison and the one most dashboards get wrong.

Three shapes are worth recognising on sight:

• A sawtooth in memory that resets at each restart and reaches the ceiling slightly sooner each day. That is a leak, and a nightly restart is a bandage, not a fix. Restart schedules that help covers when the bandage is the right call.
• A step change that starts at a particular hour and never comes back down. Something was installed, enabled or scheduled at that time. Compare it against your own deploy or plugin history before you look at anything else.
• A flat ceiling in CPU during peak hours only. That is the plan being the limit rather than the code, and no amount of tuning moves it as far as one more core would.

Look at the graphs for five minutes after each change you make, not on a schedule. Monitoring you have to remember to look at is not monitoring.

Where the alert should land#

An alert that arrives somewhere nobody looks is a log entry with extra steps. For a small team the working setup is boring:

1. One channel, not five. A Discord channel or a single email address that only receives alerts. If it also receives deploy notifications, chat and a daily digest, it is a chat channel and you will mute it.
2. Human-readable subject lines. "app.example.com returned 502 for 3 minutes" beats "CHECK_HTTP CRITICAL". You will read this on a phone.
3. Recovery notices in the same place, so the channel is self-closing and you can tell at a glance whether anything is currently broken.
4. No escalation you have not agreed. Paging a person at night for a hobby Minecraft server is how people stop hosting Minecraft servers.

Webhooks are the cheapest route: most uptime services post to a Discord or Slack webhook directly, and anything that can run a shell command can curl one. Discord webhooks for server status has the setup and the payload format. Treat the webhook URL as a credential - anyone holding it can post as you - and keep it in an environment variable rather than in the script, as in environment variables and secrets.

What the panel already watches#

Some of this is done for you here, and knowing which parts saves you from building them twice.

A watcher polls every two minutes for a server that went offline or whose uptime went backwards, which is the signature of a crash and restart. Restarts you asked for are not counted. Three unplanned restarts in an hour puts a warning on the server page and opens a ticket automatically; six suspends the server so it stops thrashing the node and so somebody looks at it. That covers the "unplanned restarts" row in the table above without you configuring anything, and why your game server keeps restarting is the companion for working out the cause.

The console graphs cover memory, CPU and disk against your limits. The Schedules tab runs the backup job on a cron expression, with ordered tasks and delays, so "take a backup then restart" is one schedule rather than two things you hope happen in order - see /guides#schedules. The status page shows the same availability data we watch, sampled every five minutes, rather than a separate marketing version of it.

What is not done for you is the external check and the backup-age check. Nobody outside your account knows what your application is supposed to return, and nobody but you knows that last night's backup should have been 4 GB rather than 40 KB.

Logs, and the checks that read them#

The fifth check, when you want one, is a log check: alert when a particular line appears. It is useful and it is also where alert lists go to die, so it comes with rules.

Alert on lines that mean a specific thing you would act on. OutOfMemoryError, database is locked, Address already in use, permission denied on the data directory, a plugin's own "failed to load" line. Do not alert on the word ERROR, because half the software in the world logs recoverable conditions at error level and you will spend a fortnight tuning the filter before you turn it off.

# The last hour of a log, only the lines you decided matter$ grep -nE 'OutOfMemory|Address already in use|Can.t connect to' latest.log | tail -20

The live console shows unfiltered output, with web-server noise folded behind a count you can switch off, which makes it readable during an incident rather than a wall. Reading it properly is a skill of its own: reading the console covers what to look at during a start, and logs worth keeping covers what to retain afterwards and for how long. The rule that matters here is that a log line is evidence, not an alarm. Turn one into an alarm only when you have been caught by it twice.

The rule that keeps the list short#

If an alert fires and there is nothing to do about it, it should not have fired. Either make it actionable or delete it.

That rule is easy to state and uncomfortable to apply, because deleting an alert feels like accepting risk. It is not. An alert nobody acts on has already been deleted, informally, by everyone who filters it into a folder. Making that official is the only way to keep the remaining ones meaningful.

Apply it monthly, with three questions per alert:

1. Did it fire this month? If it never fires, check that it still works - a check pointed at a hostname you stopped using is a check that will never fire again. Test it by breaking something on purpose.
2. When it fired, was it true? Anything with a false-positive rate over about one in ten needs its threshold moved or its retries raised, today, before you learn to ignore it.
3. What did you do? If the answer is "looked at it and did nothing", the alert is a graph. Move it to the dashboard.

Four checks you read are worth more than forty you filter. The same principle decides what else earns a place on the server: scheduled tasks worth having applies it to cron jobs, and testing a restore before you need it applies it to the one job whose output you should be checking by hand anyway, because a backup nobody has restored is a hypothesis.

FAQ#

How often should an uptime check run?

Every 60 seconds for most services, with three consecutive failures required before it alerts. That gives roughly three minutes to detection and removes nearly all false alarms from the monitor's own network. Going down to 30 seconds is reasonable for something customer-facing where you would genuinely react faster; below that you are generating load rather than information.

What should a health endpoint actually check?

Two of them. A fast one that returns 200 with no work at all, proving the process is up and listening, checked every minute. A slower one that touches the database or queue and returns 503 when a dependency is broken, checked every five minutes. Keeping them separate means your monitor is not hammering the database, and a failure tells you which layer is at fault.

Do I need Prometheus and Grafana for one server?

No. For a single server the panel's own graphs plus one external uptime check cover the same ground with none of the maintenance, and a metrics stack on the same box competes for the memory you were trying to watch. Prometheus earns its keep once you have several servers, want history longer than the panel keeps, or need alert rules written against the data rather than against a status code.

Why did my monitor say the server was down when it was fine?

Usually the check itself: a timeout set too low, a single prober with a bad route, or an alert configured to fire on the first failure instead of the third. Check whether the outage appears from a second location and whether it lines up with anything in your own logs. If nothing in the logs moved, the network between the monitor and the server is the suspect, not the server.

Should I alert on high CPU?

Not on spikes. CPU on a plan with a fixed share is a hard throttle, so hitting 100% makes the server slow rather than broken, and short peaks during a world save or a restart are normal. Alert only on the ceiling being held for fifteen minutes or more, and treat that as "look at this today", not as an outage.

How do I know a backup actually worked?

Check for a new file of a plausible size, not for a job that reported success. Age and size together catch the two common failures: the job that did not run, and the job that ran against an empty directory and wrote a 40 KB archive. Once a quarter, restore one somewhere harmless and open it. That is the only check that proves the backup is a backup.