How to block access to web bots?

Started by SIROTA, Aug 03, 2022, 11:52 AM

Previous topic - Next topic

SIROTATopic starter

Regularly, unidentified bots visit my site, and it's unclear why they're there. They may be collecting data or attempting to push my site down in search engine rankings. Perhaps a bot-tracking service is trying to pressure me into paying for their services.

Regardless, dealing with them was once easy, but after switching to a new hosting service, I no longer have access to an IP address tracking system. While I use the CIDRAM WordPress plugin to manage 80% of these bots, I still face some obstacles due to poor instructions. My goal is to ban these bots by name, as I suspect they come from a single source.
Do you have any suggestions for how to do so effectively?
  •  

Ali_Pro

Quote from: SIROTA on Aug 03, 2022, 11:52 AMHow to ban bots by name?

Just hang on Cloudflare and block the subnets you want in a couple of clicks.
It's free.
Ali.
  •  

SIROTATopic starter

Quote from: Ali_Pro on Aug 03, 2022, 12:10 PMJust hang on Cloudflare and block the subnets you want in a couple of clicks.
It's free.

Connected, now in the mail, which is on the domain, letters have stopped coming.

What needs to be screwed up where?
  •  

Ali_Pro

Quote from: SIROTA on Aug 03, 2022, 01:34 PMWhat needs to be screwed up where?
mx records in DNS settings in CF.
Add.

https://www.cloudflare.com/learning/dns/dns-records/dns-mx-record/

https://developers.cloudflare.com/dns/manage-dns-records/how-to/email-records/
Ali.
  •  

SIROTATopic starter

The fact of the matter is that the records have already been updated and Cloudflare has stopped sending letters through the connected domain. I turned off the proxy for email records in the DNS settings, and still the letters do not reach. Letters go to SMTP, but they do not come from IMAP.


Post Merge: Aug 03, 2022, 05:26 PM


In NX - MX is slightly confused)
Now everything is fine.

Great, why no one says that Cloudflare successfully keeps bots out of the site, and you don't need to pay anything.
I pressed only 2 buttons (check JavaScript and something else) and all the bots fell behind, 4 bots left, and to hell with them. I used to suffer, track down, set up SIDRAM, ban subnets along with real users. And it turns out everything is simple and even my participation is not required.

Thank you kind man for your advice!
  •  

Ali_Pro

This issue has been discussed on the forums for many years.

It is much more efficient to leave only the necessary countries, and start the rest on js check + for the necessary countries, add to js check the subnet of hosters that are googled.

Some bots are tested, then you can just hang a captcha, they will not break, it will be expensive for them.
Ali.
  •  

mariajones

To prevent bots, parsers, and spam from visiting your website, you can limit their access using .htaccess file rules. While using robots.txt for this purpose is recommended, it's often ignored. There are three methods to block unwanted bots: by IP address, User-Agent, or IP mask (by country). However, blocking bots by IP address can be tedious and ineffective since IP addresses change frequently.

A better solution is to ban them using their User-Agent name regardless of their IP address. You can also ban bots from specific countries where servers for SEO services, spam bots, and other parsers are located. There are various unwanted bots, the most popular of which include AhrefsBot, SemrushBot, MJ12bot, Riddler, and others. To block them via User-Agent, you can add certain code to .htaccess. Alternatively, you can also forbid access by IP address mask, which can be achieved by adding specific lines to .htaccess.
  •