Bot Detection Methods

Over the past few years, I've come up with a few bot detection methods that don't seem to be in common use.  These won't do a job at stopping targeted attacks, but will block common scan tools and scripts.

Detect a bot by missing name

Most of the web has gone HTTPS.  Security certificates are usually for a single hostname or a wildcard within a domain.  The old practice of having a fallback to a default host can be turned on its head here.  Set up you virtualhosts as usual, but set the fallback virtualhost to a static page explaining the user has been blocked.  Use your server's log formatting to mimic a bot attack that would be caught by your normal bot detection systems.

Detect a bot by HTTP protocol version

This one leaves open the possibility of false-positives if you have things other than web browsers hitting your site.

Most libraries used by bots and various programming environments use HTTP 1.1.  Any connection not using IE, but using HTTP/1.1 for TLS is unlikely to be a real user.

Detect a bot by HTTP status code

I don't think I've ever seen a HTTP 400 result from a legitimate request.