How to Efficiently Transfer 115GB of Data

Started by RobertMiller, Mar 20, 2023, 07:04 AM

Previous topic - Next topic

RobertMillerTopic starter

We recently needed to transfer a significant amount of data (115GB) between two machines with gray addresses and PPPoE. Unfortunately, our attempts at using torrents, Hamachi, NetBIOS, FTP in a virtual network, and Opera Unite all resulted in very slow speeds or no success altogether. We even considered physically carrying the disk since it seemed like the only remaining solution. The channels on both machines are acceptable, but we can't find any suitable free hosting services for such a large transfer and renting a service for a one-time transfer doesn't make sense.

Do you have any ideas on how to solve this problem efficiently? Perhaps there's a file transfer service that could handle such a large amount of data?
  •  

sahildd

To transfer 115GB of data more efficiently, one solution could be to archive the data quickly and divide it into approximately 1GB parts. Next, connect to a cheap VPN, or free test VPN for one day, to a computer where the data resides. On that computer, run a file-sharing service such as http or ftp, and map the port of the service to your real external VPN IP address. From the recipient computer, connect to the "address:port" using a corresponding client and download half of the files. Repeat steps 2-6 again until all files are transferred. Note that with a return of just 10 megabits and control traffic to account for, the transfer process will likely take around two days to complete.
  •  

tomfam

One possible solution to transfer data more securely and efficiently would be to use a modified version of torrents with added encryption. This method involves sending the file to a specific list of people, in this particular case just one person, and then immediately deleting it from RAM to reduce ROM consumption.

No information would be saved on the intermediate machines used to deliver the file. This approach allows for a secure transfer of data without burdening any of the involved parties with the need to save large encrypted copies.
  •  

Vanesill

For Windows users looking to transfer data, one solution is to try file synchronization through Windows Live Mesh. This approach involves creating multiple 20GB folders for synchronization, where the data itself is transferred directly between PCs without any intermediate servers.
This ensures a fast transfer of data without any additional delays.
  •  

David5

There are several methods for transferring files, one of which is using cloud storage services like Google Drive or OneDrive. These platforms offer unlimited opportunities for file transfer since they provide space for large file sizes, and the ability to upload and transfer files using online applications.

In situations where professional file transfers are necessary, Filemail is a viable option. The program offers a simple drag-and-drop interface for sending files of any size to email addresses or private URLs. Data transfer is carried out over UDP, which results in faster and more reliable delivery over the Internet.

For those who prefer more traditional methods, external drives like USB flash drives and hard drives remain popular options. However, it's important to remember that some drives may fail during file transfers, so it's important to use high-quality cables and seek assistance in replacing faulty parts.

Another method is configuring a home network. This way, devices connected to the same router can identify each other, and transfers can be made without establishing new connections every time.

Lastly, LAN cables provide faster data transfer speeds than Wi-Fi connections and eliminate the need to connect and disconnect Pen drives repeatedly.
  •  

NoleOwege

One option you could consider is using cloud storage services that offer large file transfers. Some popular services include Google Drive, Dropbox, and Microsoft OneDrive. These services allow you to upload your files to the cloud and then share them with the intended recipient, who can then download the files at their convenience. While there may be limitations on the maximum file size or bandwidth restrictions, it's worth exploring if any of these services meet your requirements.

Another alternative is using specialized file transfer services such as WeTransfer, SendGB, or pCloud Transfer. These services are designed specifically for sending large files over the internet and often offer more generous file size limits compared to traditional email attachments.

If none of these options work for you, physically carrying the disk might still be the most efficient solution, especially if the machines are in close proximity. It may also be worth reaching out to local computer repair shops or service providers to see if they offer data transfer services that could assist you.

If none of the previous solutions are viable or efficient for your situation, here are a few additional options you could explore:

1. Aspera: Aspera is a high-speed data transfer technology that provides accelerated file transfers over a wide area network (WAN). It is commonly used by media and entertainment companies for large data transfers. You can check if there are any service providers that offer Aspera transfers or consider using it directly if it fits your requirements.

2. Amazon S3: Amazon Simple Storage Service (S3) is a cloud storage service provided by Amazon Web Services (AWS). It offers scalable and reliable storage for large amounts of data. You can upload your files to an S3 bucket and then share access with the recipient so they can download the files at their convenience.

3. Rsync: Rsync is a powerful and efficient file synchronization and transfer tool. If both machines have SSH access, you can use rsync to transfer the files securely and with reduced bandwidth usage. This method is particularly useful for transferring only the changes or differences between the files, rather than the entire dataset.

4. USB drives or external hard drives: Although you mentioned physically carrying the disk didn't seem like an ideal solution, it may still be worth considering if the machines are in close proximity. USB drives or external hard drives can provide a fast and reliable way to transfer large amounts of data without relying on an internet connection. If security is a concern, you can encrypt the data on the drive before transferring it.
  •