Amazon S3 Advanced: Lifecycle Rules, Events, Performance, Metadata, and Tags

Amazon S3 advanced topics are mainly about what happens after you already know the basic idea of buckets and objects. In simple words, S3 is not only a place where you upload files. It also gives you ways to move objects between storage classes, delete old data automatically, react when something happens in a bucket, improve upload and download performance, and attach extra information to objects using metadata and tags.

This section is important because in real applications, files do not stay in one place forever. Some files are used every day, some files are used rarely, and some files are only kept for archive. Some objects need to trigger a Lambda function after upload. Some large files need faster upload and download. So Amazon S3 Advanced is about managing objects properly after they are stored.

Prerequisites

Before reading this, you should already know the basic meaning of an S3 bucket, S3 object, object key, storage class, and S3 versioning. You do not need to be an expert, but you should understand that objects live inside buckets and that storage classes are different ways to store objects based on access pattern and cost.

Moving Objects Between S3 Storage Classes

In S3, objects can move between storage classes. This is useful because not every object needs the same access speed forever. For example, if an object is accessed often today but rarely after some time, keeping it in the same storage class forever may not be the best design.

For infrequently accessed objects, you can move them to Standard-IA. IA means Infrequent Access. The object is still available, but it is mainly for data that is not opened again and again. For archive objects where you do not need fast access, you can move them to Glacier or Glacier Deep Archive.

The common storage classes in this section are:

• S3 Standard
• S3 Standard-IA
• S3 Intelligent-Tiering
• S3 One Zone-IA
• S3 Glacier Instant Retrieval
• S3 Glacier Flexible Retrieval
• S3 Glacier Deep Archive

The main idea is simple. Keep active data in a storage class that is good for fast access. Move less active data to a storage class made for infrequent access or archive. Instead of doing this manually again and again, S3 can automate it using lifecycle rules.

S3 Lifecycle Rules

S3 lifecycle rules are rules that tell S3 what should happen to objects after some time. Lifecycle rules can transition objects to another storage class, or they can expire objects by deleting them.

Transition actions are used when you want to move objects from one storage class to another. For example, you can move objects to Standard-IA 60 days after they are created. You can also move objects to Glacier after 6 months for archiving.

Expiration actions are used when you want S3 to delete objects after some time. For example, access log files can be deleted after 365 days. If versioning is enabled, lifecycle rules can also delete old versions of files. Lifecycle rules can also delete incomplete multipart uploads, which is useful because incomplete uploads can remain in the bucket and take storage.

Lifecycle rules can be targeted. You do not always need to apply the same rule to the full bucket. You can create rules for a certain prefix:

s3://mybucket/mp3/*

You can also create lifecycle rules for objects with certain tags:

Department: Finance

This means lifecycle rules can be specific. One group of objects can move to archive, another group can expire after some days, and another group can stay where it is.

Lifecycle Rule Scenario: Profile Photos and Thumbnails

Imagine an application running on EC2. Users upload profile photos to S3, and the application creates image thumbnails after the upload. The thumbnails can be recreated easily, and they only need to be kept for 60 days. The original source images should be immediately available for the first 60 days, but after that, the user can wait up to 6 hours to retrieve them.

For the source images, you can keep them in S3 Standard first because they need immediate retrieval for the first 60 days. Then you can add a lifecycle configuration to transition them to Glacier after 60 days.

For the thumbnails, you can store them in One Zone-IA because they can be recreated and do not need the same durability design as the source image. Then you can add a lifecycle configuration to expire them after 60 days.

The design looks like this:

Source images:
S3 Standard -> after 60 days -> Glacier

Thumbnails:
One Zone-IA -> after 60 days -> expire/delete

This is a good example of thinking based on the data type. The original image matters more, so it gets a different lifecycle path. The thumbnail is temporary and can be created again, so it can be deleted after some time.

Lifecycle Rule Scenario: Recovering Deleted Objects

Now imagine a company rule says deleted S3 objects must be recoverable immediately for 30 days. After 30 days, and for up to 365 days, deleted objects should still be recoverable, but recovery can take up to 48 hours.

For this, first enable S3 versioning. With versioning enabled, when an object is deleted, the object is not really removed immediately in the normal simple way. A delete marker hides the object, and the older version can still be recovered.

Then you can use lifecycle rules on noncurrent versions. Noncurrent versions are older versions of an object. You can transition the noncurrent versions to Standard-IA, and after that transition them to Glacier Deep Archive.

The flow is:

Enable S3 Versioning
Deleted object becomes hidden by a delete marker
Noncurrent versions -> Standard-IA
Noncurrent versions later -> Glacier Deep Archive

This design matches the recovery requirement. For the first 30 days, recovery should be immediate. Later, the data can be kept in a deeper archive where recovery can take longer.

S3 Analytics Storage Class Analysis

S3 Analytics Storage Class Analysis helps you decide when to transition objects to the right storage class. This is helpful because sometimes you may not know the correct lifecycle rule from the beginning. You may need data first.

S3 Analytics can give recommendations for Standard and Standard-IA. It does not work for One Zone-IA or Glacier. The report is updated daily, and it can take 24 to 48 hours before you start seeing data analysis.

The output can be a CSV report with information like date, storage class, and object age. This report is useful as a first step before creating lifecycle rules or improving existing lifecycle rules.

Example report shape:

Date        StorageClass  ObjectAge
8/22/2022  STANDARD      000-014
8/25/2022  STANDARD      030-044
9/6/2022   STANDARD      120-149

So instead of guessing when objects should move, S3 Analytics helps you understand object access and age patterns.

S3 Event Notifications

S3 event notifications allow S3 to send events when something happens in a bucket. For example, when an object is created, removed, restored, or replicated, S3 can send an event.

Common event types include:

• S3:ObjectCreated
• S3:ObjectRemoved
• S3:ObjectRestore
• S3:Replication

These events can be sent to Lambda, SQS, or SNS. A very common use case is image thumbnail generation. When a user uploads an image to S3, S3 can send an event to a Lambda function, and the Lambda function can create the thumbnail.

You can also filter events by object name. For example, you may only want events for uploaded .jpg files.

Only trigger for:
*.jpg

You can create as many S3 events as needed. S3 event notifications usually deliver events in seconds, but sometimes delivery can take a minute or longer.

IAM Permissions for S3 Event Notifications

When S3 sends events to another AWS service, permissions matter. If S3 sends an event to Lambda, Lambda needs the correct resource policy. If S3 sends an event to SNS, SNS needs the correct resource access policy. If S3 sends an event to SQS, SQS needs the correct resource access policy.

The mapping is:

S3 -> Lambda: Lambda Resource Policy
S3 -> SNS:    SNS Resource Access Policy
S3 -> SQS:    SQS Resource Access Policy

This is one of those things that can cause confusion. The event configuration may look correct, but if the destination service does not allow S3 to send the event, the setup will not work properly.

S3 Event Notifications With Amazon EventBridge

S3 can also send events to Amazon EventBridge. With EventBridge, all events from the S3 bucket can go into EventBridge, and then EventBridge rules can send them to many possible destinations.

EventBridge gives more advanced filtering options using JSON rules. These rules can filter based on things like metadata, object size, or object name. It also supports multiple destinations, such as Step Functions, Kinesis Streams, and Kinesis Firehose.

EventBridge also has useful capabilities like archive, replay events, and reliable delivery. So when you need more advanced event routing and filtering, EventBridge gives more flexibility than a simple direct S3 event notification.

The flow is:

S3 bucket -> all events -> Amazon EventBridge -> rules -> destinations

S3 Baseline Performance

Amazon S3 automatically scales to high request rates. The normal latency range mentioned here is around 100 to 200 milliseconds.

For performance numbers, an application can achieve at least:

• 3,500 PUT, COPY, POST, or DELETE requests per second per prefix
• 5,500 GET or HEAD requests per second per prefix

There are no limits to the number of prefixes in a bucket. A prefix is the part of the object path before the object name.

Example prefixes:

bucket/folder1/sub1/file  -> /folder1/sub1/
bucket/folder1/sub2/file  -> /folder1/sub2/
bucket/1/file             -> /1/
bucket/2/file             -> /2/

If reads are spread evenly across these four prefixes, then the application can achieve 22,000 requests per second for GET and HEAD.

4 prefixes x 5,500 GET/HEAD requests per second = 22,000 requests per second

So the important point is that S3 performance is calculated per prefix. If your workload is spread across prefixes, the total request rate can increase.

Multipart Upload

Multipart upload is used for large files. It is recommended for files larger than 100 MB, and it must be used for files larger than 5 GB.

The idea is that a big file is divided into parts. Then those parts are uploaded in parallel to S3. Because the parts can upload at the same time, multipart upload can speed up transfers.

The flow is:

Big file -> divide into parts -> parallel uploads -> S3 bucket

This matters in real applications because one big upload as a single operation can be slow. Multipart upload makes large file upload more practical.

S3 Transfer Acceleration

S3 Transfer Acceleration is used to increase transfer speed. It works by transferring the file to an AWS edge location first. After that, AWS forwards the data to the S3 bucket in the target Region.

For example, if a file is uploaded from the USA to an S3 bucket in Australia, the file can first go to an edge location, then move through the AWS network to the bucket.

The idea is:

File in USA -> AWS edge location -> S3 bucket in Australia

S3 Transfer Acceleration is also compatible with multipart upload. So for large files, you can combine both ideas: divide the file into parts and use the accelerated transfer path.

S3 Byte-Range Fetches

S3 byte-range fetches are used when you request only a specific byte range from an object. Instead of downloading the whole object in one request, you can request specific parts.

This can be used to parallelize GET requests. For example, a file can be split into ranges, and each range can be requested in parallel. This can speed up downloads.

It also gives better resilience in case of failures. If one range request fails, you can retry that part instead of downloading the entire file again.

The flow is:

File in S3
Part 1 -> requested separately
Part 2 -> requested separately
Part N -> requested separately

Byte-range fetches can also be used when you only need partial data. For example, you may only need the header of a file, meaning the first few bytes, instead of downloading the full object.

Byte-range request:
first XX bytes -> header

So byte-range fetches are useful for both performance and partial reads.

S3 User-Defined Object Metadata

When uploading an object to S3, you can assign user-defined metadata. Metadata is stored as name-value pairs, or key-value pairs.

User-defined metadata names must begin with:

x-amz-meta-

Amazon S3 stores user-defined metadata keys in lowercase. Metadata can be retrieved when retrieving the object.

Example metadata:

Content-Length: 7.5 KB
Content-Type: html
x-amz-meta-origin: paris

Metadata is useful when you want extra information attached to an object. But there is one important limitation: you cannot search object metadata directly in S3.

S3 Object Tags

S3 object tags are also key-value pairs, but they are used differently from metadata. Tags are useful for fine-grained permissions, where access can depend on specific objects with specific tags. Tags are also useful for analytics purposes, because S3 Analytics can group by tags.

Example object tags:

Project: Blue
PHI: True

Just like metadata, you cannot search object tags directly in S3. If you need searchable metadata or tag information, you need to use an external database as a search index, such as DynamoDB.

The idea is:

S3 object metadata/tags -> index data in DynamoDB -> search using DynamoDB

This is important because many people think that because metadata and tags exist in S3, they can search objects by them directly. But S3 does not work like that. If search is needed, the searchable index must live somewhere else.

Common Pitfalls

One common mistake is keeping every object in the same storage class forever. Some objects need fast access, but some objects become old and rarely used. Lifecycle rules help move them automatically.

Another mistake is forgetting that lifecycle rules can target prefixes and tags. You do not need one rule for the whole bucket every time. You can make rules specific to object groups.

A third mistake is configuring S3 event notifications but forgetting destination permissions. Lambda, SNS, and SQS need the correct resource policy or access policy so S3 can send events.

For performance, another common mistake is not using multipart upload for large files. It is recommended above 100 MB and required above 5 GB. Also, for downloads, byte-range fetches can help when you need parallel downloads or only a part of the file.

Finally, do not expect S3 to search metadata or tags directly. Metadata and tags are attached to objects, but if you need search, use an external database like DynamoDB as the search index.

Summary

Amazon S3 Advanced is about managing objects properly after they are uploaded. Lifecycle rules help move objects between storage classes or delete them after some time. S3 Analytics helps you decide when objects should transition between Standard and Standard-IA. Event notifications allow S3 to react to object changes by sending events to Lambda, SQS, SNS, or EventBridge.

For performance, S3 scales automatically per prefix, multipart upload helps with large uploads, transfer acceleration can improve upload speed through AWS edge locations, and byte-range fetches can speed up downloads or retrieve only part of a file. Metadata and tags let you attach key-value information to objects, but they are not searchable directly in S3, so an external index like DynamoDB is needed when search is required.

In simple words, S3 Advanced means we are not only storing files. We are managing file lifecycle, reacting to file events, improving performance, and adding extra information to objects in a proper way.