Every night at midnight, a program we wrote ourselves kicks off on our servers: the BNGH Security Scanner. It runs 67 checks, writes a report and emails it to us.
It is started by cron, the Linux service that runs commands at set times. The schedule looks like this:
0 0 * * *Read it left to right: minute 0, hour 0, any day of the month, any month, any day of the week. Midnight, every night.
Why our own scanner
Off-the-shelf antivirus tools and scanners exist, and they do their job. Their limitation is that they are generic: they have no idea what normal looks like on our servers.
They do not know which folders change every day and which should stay put. On a machine hosting many sites, a check with no context tends to flood the inbox with false alarms, or let through the one thing that matters.
A concrete example: in an uploads folder, a new file every day can be perfectly normal, but only if it is an image. Knowing that means knowing the site.
A scanner written by the people who run the server starts from a clear picture of how the machine should look, and flags whatever drifts from it.
What it checks
We are not listing the 67 checks one by one: a public list would only help anyone trying to get round them. The categories include, for example:
- Recently modified files in folders where they have no business being. A PHP file that turned up yesterday in the uploaded images folder is almost always bad news.
- New administrator accounts on the sites.
- Known attack signatures, the traces a previously seen attack leaves behind.
- Wrong permissions on files and folders, letting the wrong people write to them.
Critical means now
Most findings go into the report, which we read in the morning. If the scanner finds a CRITICAL issue, though, it switches off the dangerous thing on its own, without waiting for us.
That is deliberate. An attack that succeeds at two in the morning would have hours of head start before anyone reads an email. Stopping the dangerous thing straight away costs, at worst, a minor outage that we review calmly afterwards.
The signatures keep growing
The scanner knows the signatures of attacks we have already come across, and the list keeps getting longer. Every time we meet a new one, we extract its traces and add them.
So an attack seen once, on any site, gets looked for on every server from that night onwards.
The report always arrives
What it does not do
The scanner does not replace updates, backups or firewalls. It is one more layer, checking every night whether something has changed where it should not have.
If you run servers and wonder where to start, begin with one question: what does normal look like on my machine? Once that is written down, anything that departs from it can be checked.
