How to prevent specific bot (bad bot) to crawl your web site or access you web site in NGINX

Prevent Bot

Web bot or www bot or Internet bot is a software application that runs automated programs (scripts) over the internet. Some of these bots are specifically from to do some malicious tasks like spamming large amount of content over the internet or do DDoS attack. Or do some scrapping tasks on a specific website or webpage.

Recently I face the similar issue on my web server by a specific bot. What happened is a specific bot continuously hit/access my websites page. The bot was so continuously and sententiously hit the server that my servers CPU utilization got full and my website become hanged (The same practice would in DDoS attack).

I’m trying to prevent this attack by some traditional method like preventing by robot.txt or by using some WP plugins but none of this working then I found a unique solution that solved my issue and cools down my server CPU.

As I’m using NGINX in my web server so this solution is for NGINX you can use the same method for Apache server where you have to configure the same as per Apache server.

In the web server all your website traffic handled by sites-available configuration file of Nginx and from here, you have to prevent the bot access because after that the request will process by PHP where it will start consuming CPU usage.

So go to server section of your websites sites-available file and enter this line at bottom of the server file

if ($http_user_agent ~* (BadBot|DirtyBot)) {
return 403;

In the above code, the “BadBot” and “DirtyBot” is the name of the Bot that causes the issue. You can get the bot name from server access log file where every request logged.

Also note that in my case, The Bot uses Mozilla Firefox as http_user_agent so that I was unable to filter the Bot’s request by user agent name and hence traditional methods are not working.

So what I did in the above code, I searched the bots name into the http_user_agent’s name and if any user agents name contains that specific bot name then it will return 403. You can return any other HTTP status code as you wish.

If you have any concern regarding this comment to this post and I will reply as soon as possible.

Tags: , , , ,

No Comments

Leave a reply

Post your comment
Enter your name
Your e-mail address

Story Page