download free 30 days trial version buy bucket explorer
Documentation  Download  Purchase  Support  FAQs   Forum   ScreenShots & Demos

Amazon S3 Bucket Access Logging Introduction

Amazon S3 Bucket Logging

Amazon S3 Bucket logging simply records all the activities and requests made on S3 Bucket. With that, Amazon S3 Bucket can be configured to create records of server access logs to be able to track the activities conducted in the system. This log records should have the details of the activities and requests. This includes the type of the request, the date and time the request was made, and the resource with which the request worked.

What is Logging?

Logging is a computational activity in which a program records all the activity in a simple text file in a custom format. This text file is generally stored with .log extension. Sometimes, these log files get larger and larger, containing too much data that only machines can process easily and accurately. Many log/data analysis programs are available on the internet to help you analyze your log data easily.

What is Bucket Logging?

Bucket logging is the process in which the operations performed on the Bucket will be logged by Amazon. You can choose your target bucket where logs will be delivered. The target bucket would be a different bucket from a logged bucket.

By using Bucket logging, users can upload , download , share , access, and publish contents on the internet. All these activities are available on a very low cost. Here, we need to pay for all these activities so we need to track all of these activities because these activities affect our billing.

Sometimes, bucket logging helps you in your content management and audit trials on objects.

Bucket Explorer helps the user to enable logging on required buckets. By logging on Bucket, user can track all the activities within a bucket(s). Suppose, a user creates bucket and make it publicly available to all internet users/outsiders, he/she can track the entire upload and download activities. Bucket logging also analyzed the traffic on his/her website, estimate data transfer, and control billing on the basis of data transfer analysis.

When you enable server access logging feature on an Amazon S3 bucket, there is no additional charge but you will still pay for the usual storage charges for log files delivered to your system. These log files can be deleted anytime. For log file delivery, there are no data transfer charges. You will only be charged for data transfer if you access these delivered log files.

Why use Bucket Logging?

Bucket logging is important because server access logs are the proofs of the activities conducted in a Bucket. With Bucket logging, we will:

  1. Know what operations have been performed on the bucket.
  2. Specify the time of the operations performed and the name of the user who perform the operation.
  3. Specify the file/object name on which the operation was performed as well as the size of the file/object.

Aside from that, we can also:

  1. Track suspicious activity on our public buckets.
  2. Find which bucket/object is most frequently used and affects our billing.
  3. Calculate data transfer to estimate our incoming bill.
  4. Audit user upload and download activity on our public bucket.
  5. Analyze the traffic on specific bucket and/or object.
  6. Track and analyze errors, find the root of the problem, and so on.
How can a user enable logging on Buckets?

To enable the logging on any specific bucket(s), you need to specify the targeted bucket in which the log files will be stored. You must specify any Bucket other than the source Bucket. Both Buckets should be from the same location. i.e. Both should be either from US or Europe; otherwise, Bucket Explorer will prompt you with the message "Source and target bucket must be from the same location. Cross S3 logging is not allowed."

When a log file is delivered to the target bucket, the format of the name of the log file is:

TargetPrefixYYYY-mm-DD-HH-MM-SS-UniqueString

Target Prefix may be the bucket name that has been logged or as per your choice. Here, YYYY, mm, DD, HH, MM, and SS are the digits of the year, month, day, hour, minute, and seconds (respectively) when the log file was delivered. When bucket logging is enabled for a particular bucket, automatically, WRITE and READ_ACP permissions are assigned for the log delivery user.

The log file consists of a sequence of log record. Log records appear in no particular order. Each log record represents one request and consists of the following space delimited fields.


S.No. Field Name Purpose
1. Bucket Owner The canonical user id of the owner of the source bucket.
2. Bucket The name of the bucket that the request was processed against.
3. Time The time when the request was processed.
4. Remote-IP This shows the Internet address of the requestor.
5. Requestor The canonical user Id of the request.
6. Request-Id The request Id is a unique string generated by Amazon S3.
7. Operation Either SOAP or REST.
8. Key '-' when no operation is performed on key; otherwise, name of the key.
9. Request-URI The request-URI of the HTTP request header.
10. HTTP Status The numeric status code of the response.
11. Error Code The Amazon S3 Error Code or '-' if no error occurred.
12. Bytes Sent The number of response bytes sent excluding HTTP protocol overhead or '-' if zero.
13. Object Size The total size of the object.
14. Total Time The total number of milliseconds from the time your request is received to the time the last byte of the response is sent.
15. Turn-Around Time The number of milliseconds that Amazon S3 spent for processing your request.
16. Referrer The value of the HTTP Referrer header if present. HTTP user-agents (e.g. browsers) typically set this header to the URL of the linking or embedding page when making a request.
17. User-Agent The value of the HTTP User-Agent header.

Error Code : Helps you to track which type of error occurs and how can it be resolved. It helps you find the root cause of the problem.

Let's suppose you are getting an error code like BucketNotEmpty , it means you are deleting a Bucket which contains some objects. To delete that Bucket, you need to empty the bucket first. Just delete all objects and you can then delete the bucket.

Object size: Helps you calculate which object is frequently accessed by the user and how many times it is being downloaded, and how much data is transferred.

Let's suppose you have 1MB object and is downloaded by 10 people which means that 10MB data has been transferred.

Referrer:

Helps you to know how many people are referring your buckets or objects. It also helps you in traffic analysis and will help you know which source gives you maximum traffic on your third party bucket. and which one is beneficial for you.

Set Grantee:

This option is used for adding details of the users with which you want to share bucket logging files.

How to perform Bucket logging Operations?

  • Using S3 REST API
  • You can perform Bucket logging operations by using Amazon Bucket logging supported REST API in your application code.
  • Using Bucket Explorer
  • You can perform the same Bucket logging operations using Bucket Explorer without the hassle of writing codes.

You can perform the following Bucket logging operations:

  • Set/Get/Remove Bucket logging: You can set/remove Bucket logging using PUT Bucket logging request. When setting Bucket logging, you can get the details of the logging status of a bucket using GET request. Check the link to get more details about Set/Get/Remove Bucket Logging on Amazon S3 Bucket using Amazon REST API and using Bucket Explorer.
  • Set Grantee: You can grant access on bucket logs to other people using PUT Bucket logging request and by using Grantee request element. Check the link to get more details about Bucket Logging Grantee