How to Upload Big File in Parts?

Multipart Upload : Bucket Explorer now comes with feature of multipart upload. This facilitates you to upload a single object as a set of parts. In upload process, each part is a portion of the object's data, and each part can be uploaded independently. Multipart Upload offers many benefits over simple upload, specially when object size is very large.

Multipart upload facilitates user to :

1. Restart Uploading Objects : If a part fails to upload then you only need to resend the data for that part only.
2. Pause Upload Operation : You can pause a part which is being uploaded with multipart upload.
3. Parallel Upload : Multipart upload provides benefits of uploading object parts in parallel.
4. You do not need to worry about the size of object, you can upload parts without knowing the size of the final object.
5. Canceling multipart upload no longer be billed for any storage that was associated with the parts which have been stored.

View Multipart Preparing: To process a large file Bucket Explorer uses multipart operation, you can see the process for preparing the multipart operation at bottom before starting queue.

Bucket Explorer now supports multipart operation. This operation is designed in two ways :

• Upto 5 GB
• More than 5 GB ^{New Feature}
  

Multipart Upload upto 5 GB is checked by Etag matching with local file MD5 if it matches then it will move to original location on S3.

As Bucket Explorer do not get normal ETag for more than 5 GB file, like it gets in case of simple S3 file, So it is not possible to follow same process for more than 5 GB file. But Bucket Explorer increases the reliability by adding validation for parts before upload on local file because it may be changed till last merging operation is performed.

The upload may take several days/interruptions/needs to do resume upload. It is also possible for the file to get changed so it needs a way to validate file before calling the final merging on S3. So Bucket Explorer does prepare a config by the following way -

1. Calculate MD5 for the file, number of parts and MD5 for each part before starting the upload.
2. While uploading each part, compare its MD5 with the one that was saved in step 1. If the MD5 for any part is different, then stop the upload operation because file has been changed. Bucket Explorer allows you to resume upload operation if it takes time or operation gets interrupted before completion of upload process.

Set the size of a part :

• The size you set for a part divides entire object into equal parts. Let's say if you have set size of 50 MB for a part and your object is of size 2.5 GB (2560 MB), then there will be 52 parts, all parts except the last one will be of size 50 MB,  and last part will be of size 10 MB.
• The default size for multipart upload is 32 MB.
• For best performance, it is recommended that the size of a part lies between 5 MB and 100 MB.