If you run your own webserver you probably need to take a look at this daily auto-generated .htaccess file which will protect your server against numerous spider/cloud/hoster/tor/botnet/state actor attacks ;-)
URL: https://lightaffaire.com/download/htaccess-deny-bots-spiders
Background:
I only wanted humans or google/bing search accessing my own frontend webservers!
Features:
- Builds the "deny from" IP network rules from a curated list of Autonomous System Numbers (ASN) whose traffic I have carefully observed over time.
- Install in your webservers DocumentRoot and check your error_log file for "AH01797: client denied by server configuration:" and also check for HTTP 403 access denied requests in your access_log files.
- Using this .htaccess file will significantly reduce web attacks to your webserver.