S3 Data Lifecycle Policies for Cost Control

S3 lifecycle policies automate data transitions between storage classes, reducing storage costs while maintaining data accessibility based on access patterns.

Storage Class Overview

Cost Comparison

Storage Class	$/GB/Month	Retrieval	Use Case
Standard	$0.023	Instant	Frequent access
Intelligent-Tiering	$0.023-0.0125	Instant	Unknown patterns
Standard-IA	$0.0125	Instant	Infrequent access
One Zone-IA	$0.01	Instant	Reproducible data
Glacier IR	$0.004	Minutes	Archives with quick access
Glacier Flexible	$0.0036	Hours	Long-term archives
Glacier Deep	$0.00099	12+ hours	Rarely accessed

Lifecycle Policy Configuration

CloudFormation

DataBucket:
  Type: AWS::S3::Bucket
  Properties:
    BucketName: data-archive-bucket
    LifecycleConfiguration:
      Rules:
        - Id: TransitionToIA
          Status: Enabled
          Transitions:
            - StorageClass: STANDARD_IA
              TransitionInDays: 30
            - StorageClass: GLACIER_IR
              TransitionInDays: 90
            - StorageClass: GLACIER
              TransitionInDays: 180
            - StorageClass: DEEP_ARCHIVE
              TransitionInDays: 365
          NoncurrentVersionTransitions:
            - StorageClass: GLACIER
              TransitionInDays: 30
          NoncurrentVersionExpiration:
            NoncurrentDays: 90
            
        - Id: CleanupIncomplete
          Status: Enabled
          AbortIncompleteMultipartUpload:
            DaysAfterInitiation: 7
            
        - Id: ExpireTempData
          Status: Enabled
          Prefix: temp/
          ExpirationInDays: 7

Terraform Configuration

resource "aws_s3_bucket_lifecycle_configuration" "archive" {
  bucket = aws_s3_bucket.data.id

  rule {
    id     = "archive-old-data"
    status = "Enabled"

    filter {
      prefix = "logs/"
    }

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 90
      storage_class = "GLACIER_IR"
    }

    transition {
      days          = 365
      storage_class = "DEEP_ARCHIVE"
    }

    expiration {
      days = 2555  # 7 years
    }

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "GLACIER"
    }

    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }

  rule {
    id     = "intelligent-tiering"
    status = "Enabled"

    filter {
      prefix = "data/"
    }

    transition {
      days          = 0
      storage_class = "INTELLIGENT_TIERING"
    }
  }
}

Intelligent-Tiering Configuration

Archive Access Tiers

import boto3

s3 = boto3.client('s3')

def configure_intelligent_tiering(bucket):
    s3.put_bucket_intelligent_tiering_configuration(
        Bucket=bucket,
        Id='ArchiveConfiguration',
        IntelligentTieringConfiguration={
            'Id': 'ArchiveConfiguration',
            'Status': 'Enabled',
            'Tierings': [
                {
                    'Days': 90,
                    'AccessTier': 'ARCHIVE_ACCESS'
                },
                {
                    'Days': 180,
                    'AccessTier': 'DEEP_ARCHIVE_ACCESS'
                }
            ]
        }
    )

Cost Analysis

Storage Class Inventory

def analyze_storage_distribution(bucket):
    s3 = boto3.client('s3')
    
    storage_classes = {
        'STANDARD': {'count': 0, 'size': 0},
        'STANDARD_IA': {'count': 0, 'size': 0},
        'ONEZONE_IA': {'count': 0, 'size': 0},
        'GLACIER': {'count': 0, 'size': 0},
        'DEEP_ARCHIVE': {'count': 0, 'size': 0},
        'INTELLIGENT_TIERING': {'count': 0, 'size': 0}
    }
    
    paginator = s3.get_paginator('list_objects_v2')
    
    for page in paginator.paginate(Bucket=bucket):
        for obj in page.get('Contents', []):
            storage_class = obj.get('StorageClass', 'STANDARD')
            if storage_class in storage_classes:
                storage_classes[storage_class]['count'] += 1
                storage_classes[storage_class]['size'] += obj['Size']
    
    return storage_classes

def calculate_potential_savings(distribution):
    pricing = {
        'STANDARD': 0.023,
        'STANDARD_IA': 0.0125,
        'GLACIER': 0.004,
        'DEEP_ARCHIVE': 0.00099
    }
    
    current_cost = sum(
        (v['size'] / (1024**3)) * pricing.get(k, 0)
        for k, v in distribution.items()
    )
    
    # Calculate if all STANDARD moved to optimal
    potential_cost = sum(
        (v['size'] / (1024**3)) * pricing.get('GLACIER', 0)
        for k, v in distribution.items()
        if k == 'STANDARD'
    )
    
    return {
        'current_monthly_cost': current_cost,
        'potential_monthly_cost': potential_cost,
        'potential_savings': current_cost - potential_cost
    }

Monitoring Transitions

LifecycleMetricsAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: S3LifecycleTransitionErrors
    MetricName: NumberOfObjects
    Namespace: AWS/S3
    Statistic: Sum
    Period: 86400
    EvaluationPeriods: 1
    Threshold: 0
    ComparisonOperator: GreaterThanThreshold
    Dimensions:
      - Name: BucketName
        Value: !Ref DataBucket
      - Name: StorageType
        Value: StandardStorage

Working with Warqline

We are a cloud engineering consultancy and an official AWS and Google Cloud partner. If you are running this in production and want a second pair of eyes, we scope work in a free 45-minute technical call: you describe what you are running and what worries you, and we tell you what we would look at first.

Talk to an engineer

Conclusion

S3 lifecycle policies are essential for storage cost optimization. Implement tiered transitions based on access patterns and regularly analyze storage distribution to identify optimization opportunities.