download free 30 days trial version buy bucket explorer
Documentation  Download  Purchase  Support  FAQs   Forum   ScreenShots & Demos

Amazon S3 - Throttling of Maximum Parallel Data in Upload Queue

Introduction

Throttling is a process that is responsible for regulating the rate at which application processing is conducted.
We are using the throttling concept in Bucket Explorer’s upload process .

How throttling works with Bucket Explorer?

With the default setting of this feature, you can upload a maximum of 100 MB of file(s) or run 5 threads at a time. If the size of files being uploaded is more than 100 MB, only one thread will run. Suppose you are uploading 105MB of file, because its size is more than the maximum limit so only one thread will run.

Why You Need Throttling?

The following examples will help you understand Throttling:

If we have some files running in queue with different sizes, let us say we have first two files having the sum of their size more than in configuration xml, we run only a single file. Now we process the queue for the second and third file and if the sum of these files exceeds the maximum limit,  we process the second file only and the process will follow for all the remaining files.

Example # 1

We have three files to run at queue respectively ‘a of size 35 MB’, ‘b of size 70 MB’ and ‘c of size 65 MB’. In configuration xml, it is set to 100 MB. So when I process these files in queue, ‘a’ will process alone because the sum of the sizes of ‘a’ and ‘b’ exceeds 100 MB. Now file ‘b’ will also process alone in queue because the sum of the sizes of ‘b’ and ‘c’ exceeds 100 MB. Then ‘c will process after.’

Example # 2

Suppose you are uploading a file of 200MB and because its size is more than the maximum limit based on the default setting in Bucket Explorer, only one thread will run. Now if you want to run 5 threads of 200 MB each, then you can only do this by setting the maximum limit of parallel data in queue. If you set the maximum limit to 1000 MB, then you can run 5 threads of 200 MB each as per your requirement.

What is the advantage of using this feature?

The main advantage of setting the maximum limit of parallel queue is the reduced time spent in running big processes one by one in queue. Let’s talk in reference to the above example. On that example, you are uploading a file of 200 MB at a time and if you don’t increase the maximum limit, then you will have to wait for 5 processes which are supposed to be run one by one. If you increase the maximum limit to 1000; you can run 5 processes at once in parallel queue that will reduce the time spent in processing.

How to customize the maximum parallel data in queue?

Follow this simple step-by-step process to set the maximum parallel data in queue:

  1. Run Bucket Explorer.
  2. Go to Tool->Advance Preferences .
  3. Now change the value of the maximum allowed data size.
  4. Click on "ok" button.
  5. You’ve set the maximum parallel data in queue.


 

Related Topics: