Hello!

Theres an info site (built on WP) with around 20,000 visitors from search each day.

It's hosted on a VIP plan. Recently, the server load has spiked dramatically, especially in the last few days.

I set up AWStats.

The hosting support mentioned that there are many requests from different IPs hitting different pages, possibly even a DDoS attack. They recommended blocking through .htaccess. But that didn't help, since the IPs keep changing.

I also saw in the stats that the Yandex bot is very active. I have already limited the crawl speed to 0.7 requests per second in the webmaster tools.

Honestly, I'm at a loss now. The server load is massive and keeps breaking all the limits (I'm worried that they might kick me off the server sooner or later).

Maybe someone with knowledge can offer some advice or help (with compensation, of course).

P.S. Could cloudflare help here? Are there any other similar options?

It's definitely frustrating when things start to spiral out of control like this. I've had my fair share of headaches with server loads and bots, so I get where you're coming from.

Given your situation, it sounds like you're dealing with a mix of high bot activity and potentially some kind of DDoS-like behavior. While blocking IPs in .htaccess can be a quick fix, it's not going to cut it when the IPs keep changing. You need something more robust.

Cloudflare could definitely help. Here's why:

DDoS Protection: Cloudflare is pretty solid at mitigating DDoS attacks. They have a global network and can absorb a lot of traffic, filtering out the bad stuff before it even hits your server.

Bot Management: Cloudflare can help you manage bots more effectively. While you've already limited the Yandex bot, Cloudflare's bot management tools can give you more granular control. You can block or challenge traffic that looks suspicious.

Caching: Cloudflare's CDN (Content Delivery Network) can cache your content across their servers worldwide, which reduces the load on your origin server. This can be a game-changer, especially with 20,000 visitors daily.

Rate Limiting: You can set up rules to limit the number of requests an IP can make in a short period of time. This can be really effective in stopping abusive behavior.

Now, on the other hand, if you're not keen on going with Cloudflare, there are some other steps you can take:

WAF (Web Application Firewall): If you haven't already, consider using a WAF. Some hosting providers offer this, or you can get one separately. A WAF can block a lot of malicious traffic.

Review AWStats: Take a deep dive into AWStats and see if you can identify any patterns with the traffic. Sometimes, certain bots or scripts can be pinpointed and blocked more effectively once you know what you're looking for.

Server Configuration: Double-check your server configurations. Sometimes tweaking your Apache or Nginx settings can improve performance under load. Things like keep-alive settings, Gzip compression, and caching can make a difference.

Yandex Bot: Since you mentioned the Yandex bot, if it's still causing trouble even after limiting the crawl speed, you might want to temporarily block it completely to see if that helps. You can do this in your robots.txt file or directly in .htaccess.
Don't hesitate to get professional help if this is getting out of hand. There are folks who specialize in server optimization and security who could look into this in more depth. Given that your site is pretty big, it might be worth the investment to bring in someone who can dive into the logs, optimize your setup, and put long-term protections in place.

Set up a VPS and don't stress, bots are litteraly everywhere on the web.

Cloudflare is good, but only if you're on a paid plan, they have some starting at around 20 bucks.

If there was a DDOS attack, your hosting would've crashed right away, so it's probably not that, since your site isn't really a target for anyone.

In late June, i encountered similar issues on two of my sites, both with around 13k traffic each, tho they've been running smoothly for over 3 years now. Every 20 minuts or so, the CPU would spike to 100%, causing the sites to go offline for about 5 minuts. CPU usage per day reached a ridiculous 300+ minutes. I tried switching plans, bought extra limits and so on, but nothing worked. Finally, I figured out it was the caching plugin. Switched from wp super cache to LiteSpeed cache. Now, the CPU load hardly ever touches 3-7%, and daily CPU time is down to 5-10 minutes instead of 400+.

 We offer expert solutions to handle traffic spikes and ensure smooth performance under load.