If you like DNray Forum, you can support it by - BTC: bc1qppjcl3c2cyjazy6lepmrv3fh6ke9mxs7zpfky0 , TRC20 and more...

 

Need for Reliable and High-Volume File Hosting

Started by Gamadvarp, Oct 25, 2023, 12:02 AM

Previous topic - Next topic

GamadvarpTopic starter

The issue at hand is the need for a photographer's website to have a platform that can accommodate a large quantity and size of files so that clients can easily access and download their photos later on.

Specifics:
The download channels don't necessarily require high-speed connections since the number of downloads/connections won't be enormous.
Each client should only have access to their own photo shoot. The main concern here is the significant file sizes, ranging from 2-10GB per client. In total, about 500-800GB of storage is needed. Google Drive is not a good fit as it can be slow or difficult to download from.

Solution: I have decided to build my own website, and I have some personal experience with this. Additionally, I can also create a gallery feature.

Now, the question is regarding hosting. The clientele is based in Ukraine, and the goal is to have file access speeds of at least 50-100MB/sec without interruptions. I have looked into AWS and their S3 section, but their pricing lists confuse me. I don't understand how much it could potentially cost.

Please advise on how to estimate the final hosting price on AWS or provide any other suggestions you may have.
  •  


kenaclekaf

Given your requirements, there are a couple of points to consider. First, AWS S3 is indeed a robust and versatile solution for storing large amounts of data. The cost structures for S3 are based on a number of different factors including:

Storage:  You're charged per GB stored per month.

GET requests: You are also charged for the number of download requests made.

Data transfer: Costs associated with moving data out of AWS.

However, to get a precise costing you'll need to input your data usage estimates (storage, requests, etc.) into the AWS Simple Monthly Calculator, which is available here: https://calculator.s3.amazonaws.com/index.html

Now, taking into account your requirement of data transfer speed of at least 50-100MB/sec, you will also need to look into AWS's Content Delivery Network (CDN) service, AWS CloudFront. While S3 is excellent at storing your data, CloudFront takes that data and distributes it to edge locations worldwide, ensuring that your customers have near-instant access to the data. This might be crucial for your requirements.

Similarly, you are charged for this service too. Main cost factors include:

Data Transfer Out: This is the cost of CloudFront delivering your files to your users.
HTTP/HTTPS Requests: This is the cost for each request a user makes to retrieve your files.
For complete details, you can visit the CloudFront pricing page (https://aws.amazon.com/cloudfront/pricing/).

The whole AWS ecosystem can seem a bit complex, but for your use case, you would probably need AWS S3 for storage and AWS CloudFront for delivering the data to your clients in Ukraine.

Also, consider that if your ICT infrastructure can afford to handle the task, especially in terms of bandwidth, you may want to use just AWS S3 storage and host the website and files delivery from your own servers. This will reduce the costs related to CloudFront.

As an alternative to AWS, you could also consider Microsoft Azure or Google Cloud storage solutions, which also offer scalable storage and CDN services. Their price models are comparable to AWS, and they might offer better cost-effectiveness depending on current prices and promotions.

If you're looking for a more detailed breakdown of AWS, here's how you could start estimating the costs based on the information provided so far:

Storage cost: If you're expecting to store 500-800GB of data, refer to the Amazon S3 storage pricing which is visible under "Amazon S3 Pricing"-> "Pricing for Amazon S3" -> "Storage pricing".

Example:  the cost of standard storage for the first 50TB per month is0.023perGB.Soifyouplantostore800GB,thestoragecostwouldbe800∗0.023 = $18.4 per month.

GET Request cost: You'll get charged for the number of download requests. If a user is downloading 2-10GB of photos, the number of GET requests cannot be estimated without knowing how you're dividing these photos. Is it one file download or multiple downloads? However, the cost per 1000 GET-requests  is $0.0004.

Data Transfer cost: Downloading (or transferring out) data from S3 is charged. In the US region, the first 1GB each month is free, beyond which up to 10 TB/month is charged at 0.09perGB.Soifyourclientsdownload800GBpermonth,thetransfercostwouldbe72.

Again, these costs are approximate, are based on US region pricing, and depend on your traffic pattern.

For AWS CloudFront:

Data Transfer Out cost: The cost is different based on the location of the viewers. For Europe region, the first 10TB is charged at $0.085 per GB.


Before making a decision, you might also want to consider some other factors such as:

Data sovereignty: Are there any rules about where the data for Ukrainian citizens should be stored?
Local internet infrastructure: Given the majority of your clientele are in Ukraine, how reliable is the internet infrastructure there? If it isn't highly reliable, you might want to have a mechanism to handle interrupted downloads.
Alternative solutions: Could services like Microsoft Azure, Google Cloud or a local provider in Ukraine serve your needs better?
Lastly, if you're handling many large files, you'll want to ensure that your application logic and the user interface for your clients is designed with that in mind. File compression and strategic scheduling of large transfers can also help manage traffic and costs.


Since the AWS pricing structures confuse you, let's explore some simpler options that might suit your needs:

Backblaze B2: This is a competitively priced cloud storage solution, which charges
0.005/GB/monthforstorage .Datadownloadis0.01/GB and there's no cost for upload. The available integrations might suit you if you're building your own platform.

Sync.com: This platform offers secure, end-to-end encrypted storage and has a business plan targeted at small firms where pricing is simple and straightforward.

pCloud: Another choice is pCloud that provides secure file storage and sharing. It features a straightforward plan structure with a Business plan allowing for multiple users.

Remember to keep in mind:

Data Security: This is crucial, especially when dealing with clients' private photographs. End-to-end encryption and stringent security measures need to be in place.
Integration with Your Platform: The storage provider should ideally have APIs or plugins available which lets you integrate with your website smoothly.
Traffic Management: You are dealing with very large files - the platform should be able to handle these without timeout errors or crashes.
Depending on the choice of your web development platform, there can be specific cloud storage solutions, too. For example, if you are using WordPress, plugins like WP Offload Media let you integrate directly with providers like Amazon S3, DigitalOcean Spaces, and Google Cloud Storage.

Remember to run cost calculations for these alternatives as well, including the costs of storage per month, download costs, and the number of transactions that will be performed. Each service provider should have pricing details available on their website.

In terms of improving the delivery speed to your users, you might want to then use a Content Delivery Network (CDN) to help improve the speed at which the content is delivered to your customers. The details on this would depend on the chosen hosting solution, as some might offer integrated CDN services, while for others, you might need to add an external CDN service.

Creating the best solution might require a mix of services, so don't be afraid to diversify based on your needs and what each platform offers. By spreading your needs across different services you might realize other operational benefits as well. Also, consider contacting the sales or support teams at these companies to discuss your specific needs and they may be able to offer tailored solutions.
  •  

pmyualxlg

You have the option to use a storage solution such as the Storage Box provided by Hetzner (https://www.hetzner.com/storage/storage-box). It offers a capacity of 10 terabytes and costs 40 euros per month. The service supports HTTPS, but it is important to note that there is a limit of 10 concurrent connections. Alternatively, you can also mount the storage via Samba.

Please take into consideration the specific needs of your project when choosing a storage solution. While Hetzner's Storage Box provides ample capacity and secure access through HTTPS, the connection limit may impact simultaneous usage. Meanwhile, mounting the storage via Samba can offer additional flexibility, but it's important to ensure compatibility with your system.
  •  

TechnoExponent

If you are still considering AWS, then S3 is a suitable option for storing photos, but with some reservations.
- Storing in S3 is cost-effective.
- Retrieving data frequently can be expensive. If you don't want to compete with services like Flickr and have low traffic, it's better to avoid storing many photos in S3 due to potential bots and high costs. It's advisable to show only thumbnails publicly and consider the following:
- To reduce the cost of S3 requests, it's highly recommended to implement a caching layer like Nginx on an EC2 instance which has a more predictable price.
- If I were in your position, I would store full-size photos in a section that requires user login, even if it's through Google. Another idea could be storing photos in Google Drive, granting access via email (even if the email address is not on Google), and using any hosting service (including AWS) to interact with the existing Google Drive storage and display generated thumbnails for quick browsing.
The pricing for Google Drive will be more predictable.

However, this is just an opinion, and I agree with the previous commentators. Pricing in cloud services, where payment is based on usage, can be surprisingly unpredictable. It's important to understand the nuances of each service before making a decision and hope that the cost will remain consistent.
  •  

Zinavopvtltd

Reliable and high-volume file hosting is crucial for businesses to securely store, manage, and share large files. It ensures data accessibility, seamless collaboration, and scalability, facilitating efficient workflows and robust data management solutions.
  •  


If you like DNray forum, you can support it by - BTC: bc1qppjcl3c2cyjazy6lepmrv3fh6ke9mxs7zpfky0 , TRC20 and more...