I've got a couple of sites where users can snag games and apps, pulling in around 50k hits daily. Initially, I was using a VPS for hosting files, but they throttled my download speeds once I blew past the 1 Gbit/s bandwidth cap. Thinking it was due to the shared bandwidth, I switched to a dedicated server with a guaranteed 1 Gbit/s port elsewhere. Yet, on day one, I faced the same throttling issue due to high traffic.

Here's the breakdown:

Total file size: 400 GB (including some buffer)
Monthly data transfer: ~90 TB (around 2-4 TB daily)
Concurrent downloads: No clue
Budget: Not sure how much the ideal solution costs, but I'm open to options.

I'm reaching out to see if anyone can suggest viable solutions and provide some insight into the pricing. It's baffling how those big file servers manage traffic from 100k to a million users daily without hitting bandwidth limits. Any advice would be greatly appreciated!

I can tell you that your issue is not uncommon, especially when dealing with high-traffic file sharing websites. The key to avoiding bandwidth throttling lies in optimizing your infrastructure and leveraging content delivery networks (CDNs).

When you switched from a VPS to a dedicated server, you likely didn't address the underlying issue - your server is still the single point of failure and bottleneck for all your traffic. To mitigate this, consider using a CDN, which caches your files across multiple edge servers worldwide, reducing the load on your origin server and minimizing the distance between users and your content. This approach will not only help you avoid throttling but also improve download speeds and overall user experience.

CDNs like Cloudflare, MaxCDN, or KeyCDN can handle massive traffic and provide you with detailed analytics to help you better understand your concurrent downloads. Their pricing varies, but you can expect to pay around 0.05−0.15 per GB of data transferred, depending on the provider and your traffic volume.

Another option is to use a peer-to-peer (P2P) content delivery network, which allows users to share files with each other, reducing the load on your origin server. However, this approach may require significant development and infrastructure investments.

As a side note, it's surprising that you're not aware of your concurrent downloads, which is a crucial metric for capacity planning and optimization. I recommend implementing analytics tools to monitor your traffic and adjust your infrastructure accordingly.

I've been experiencing a peculiar issue with my infrastructure hosted on Selectel. For years, I've had an aggressive crawling setup in place, yet it hasn't triggered any alarms. However, on one of my Virtual Private Servers (VPS), I've noticed a persistent 100% CPU utilization for a couple of months now. Despite this, I've been procrastinating on investigating the root cause, and surprisingly, Selectel's monitoring system hasn't flagged this anomaly or alerted me to the high load.

Scaling a website horizontally by leveraging multiple servers is a no-brainer. You simply need to implement a load balancing strategy, where incoming traffic is distributed across4-6 servers to ensure optimal performance and minimize downtime. One way to achieve this is by utilizing round-robin DNS, where multiple IP addresses corresponding to each server are assigned to a single domain in the DNS zone.

This way, when a user requests access to the website, the DNS server will rotate through the list of IP addresses, directing the traffic to each server in a sequential manner. It's astonishing that some webmasters are still struggling to implement this basic high availability setup, when in fact, it's a relatively straightforward process that can be outsourced to a freelancer for a mere $100.