AWS Macie for Data Security and Privacy

Discover how AWS Macie uses machine learning to automatically discover, classify, and protect sensitive data in S3.

AWS Macie is a fully managed data security service that uses machine learning to automatically discover, classify, and protect sensitive data in Amazon S3. This guide covers implementing Macie for comprehensive data protection.

Understanding AWS Macie

What Macie Does

Macie provides automated data protection:

  • Discovery: Finds sensitive data across S3 buckets
  • Classification: Categorizes data by type and sensitivity
  • Protection: Alerts on exposure risks
  • Compliance: Helps meet regulatory requirements

Sensitive Data Types

Macie detects multiple categories:

  • Personally Identifiable Information (PII)

    • Names, addresses, phone numbers
    • Email addresses
    • Social Security numbers
    • Driver's license numbers

  • Financial Information

    • Credit card numbers
    • Bank account details
    • Financial statements

  • Credentials

    • API keys and tokens
    • AWS access keys
    • Database credentials
    • Private keys

  • Healthcare Data

    • Medical records
    • Health insurance information
    • Prescription data

Enabling Macie

Initial Setup

Enable Macie in your account:

aws macie2 enable-macie \
  --finding-publishing-frequency FIFTEEN_MINUTES \
  --status ENABLED

CloudFormation Configuration

MacieSession:
  Type: AWS::Macie::Session
  Properties:
    Status: ENABLED
    FindingPublishingFrequency: FIFTEEN_MINUTES

Terraform Setup

resource "aws_macie2_account" "main" {
  finding_publishing_frequency = "FIFTEEN_MINUTES"
  status                       = "ENABLED"
}

resource "aws_macie2_classification_job" "pii_scan" {
  name = "pii-discovery-job"
  
  s3_job_definition {
    bucket_definitions {
      account_id = data.aws_caller_identity.current.account_id
      buckets    = ["customer-data-bucket", "application-logs"]
    }
    scoping {
      includes {
        and {
          simple_scope_term {
            key       = "OBJECT_EXTENSION"
            values    = ["csv", "json", "txt", "log"]
            comparator = "EQ"
          }
        }
      }
    }
  }
  
  job_type = "SCHEDULED"
  schedule_frequency {
    daily_schedule = true
  }
  
  sampling_percentage = 100
  
  tags = {
    Environment = "production"
    Compliance  = "GDPR"
  }
}

Classification Jobs

Job Types

Configure different scanning approaches:

One-Time Jobs

For initial discovery:

{
  "name": "initial-pii-discovery",
  "jobType": "ONE_TIME",
  "s3JobDefinition": {
    "bucketDefinitions": [
      {
        "accountId": "123456789012",
        "buckets": ["data-lake", "analytics-output"]
      }
    ]
  },
  "samplingPercentage": 100
}

Scheduled Jobs

For continuous monitoring:

{
  "name": "weekly-compliance-scan",
  "jobType": "SCHEDULED",
  "scheduleFrequency": {
    "weeklySchedule": {
      "dayOfWeek": "MONDAY"
    }
  },
  "s3JobDefinition": {
    "bucketDefinitions": [
      {
        "accountId": "123456789012",
        "buckets": ["production-data"]
      }
    ]
  }
}

Scoping Rules

Define what to scan:

ScopingRules:
  Includes:
    - SimpleScope:
        Key: "OBJECT_EXTENSION"
        Values: ["csv", "json", "parquet", "txt"]
    - SimpleScopeTerm:
        Key: "OBJECT_SIZE"
        Values: ["1048576"]  # > 1 MB
        Comparator: "GT"
  Excludes:
    - SimpleScope:
        Key: "OBJECT_KEY"
        Values: ["logs/debug/*", "temp/*"]

Custom Data Identifiers

Creating Custom Identifiers

Define organization-specific patterns:

CustomDataIdentifier:
  Type: AWS::Macie::CustomDataIdentifier
  Properties:
    Name: "EmployeeID"
    Description: "Internal employee identifier format"
    Regex: "EMP-[0-9]{6}"
    Keywords:
      - employee
      - staff
      - personnel
    MaximumMatchDistance: 50
    IgnoreWords:
      - example
      - test

Common Custom Patterns

{
  "customDataIdentifiers": [
    {
      "name": "InternalAccountNumber",
      "regex": "ACC-[A-Z]{2}[0-9]{8}",
      "keywords": ["account", "acct"]
    },
    {
      "name": "MedicalRecordNumber",
      "regex": "MRN[0-9]{10}",
      "keywords": ["patient", "medical", "record"]
    },
    {
      "name": "InternalProjectCode",
      "regex": "PRJ-[A-Z]{3}-[0-9]{4}",
      "keywords": ["project", "initiative"]
    }
  ]
}

Findings and Alerts

Finding Types

Macie generates findings for:

  • Sensitive Data Discovery: Found sensitive data types
  • Policy Findings: S3 bucket misconfigurations
  • Unusual Access: Anomalous data access patterns

EventBridge Integration

Automate responses to findings:

MacieFindingsRule:
  Type: AWS::Events::Rule
  Properties:
    Name: macie-high-severity-alerts
    EventPattern:
      source:
        - aws.macie
      detail-type:
        - Macie Finding
      detail:
        severity:
          description:
            - High
    Targets:
      - Arn: !Ref AlertSNSTopic
        Id: sns-alert
      - Arn: !GetAtt RemediationLambda.Arn
        Id: auto-remediate

Automated Remediation

Implement automatic responses:

def lambda_handler(event, context):
    finding = event['detail']
    bucket = finding['resourcesAffected']['s3Bucket']['name']
    
    if finding['severity']['description'] == 'High':
        # Block public access immediately
        s3_control.put_public_access_block(
            Bucket=bucket,
            PublicAccessBlockConfiguration={
                'BlockPublicAcls': True,
                'IgnorePublicAcls': True,
                'BlockPublicPolicy': True,
                'RestrictPublicBuckets': True
            }
        )
        
        # Enable encryption
        s3.put_bucket_encryption(
            Bucket=bucket,
            ServerSideEncryptionConfiguration={
                'Rules': [{
                    'ApplyServerSideEncryptionByDefault': {
                        'SSEAlgorithm': 'aws:kms',
                        'KMSMasterKeyID': KMS_KEY_ID
                    }
                }]
            }
        )
        
        # Notify security team
        sns.publish(
            TopicArn=SECURITY_TOPIC,
            Subject=f"Critical: Sensitive data exposed in {bucket}",
            Message=json.dumps(finding, indent=2)
        )
    
    return {'statusCode': 200}

Multi-Account Deployment

Organizations Integration

Deploy Macie across your organization:

MacieOrganizationAdmin:
  Type: AWS::Macie::Session
  Properties:
    Status: ENABLED
    
OrganizationConfiguration:
  # Delegated administrator setup
  DelegatedAdminAccountId: "111122223333"
  AutoEnable: true

Member Account Management

# Designate delegated admin
aws macie2 enable-organization-admin-account \
  --admin-account-id 111122223333

# Enable for member accounts
aws macie2 create-member \
  --account '{"accountId":"444455556666","email":"security@member.com"}'

Compliance Reporting

Regulatory Alignment

Map findings to compliance frameworks:

Regulation Macie Data Types
GDPR PII, Email, IP addresses
HIPAA PHI, Medical records
PCI-DSS Credit cards, Financial data
SOC 2 Credentials, Access logs

Generating Reports

Create compliance reports:

def generate_compliance_report():
    findings = macie.get_findings(
        sortCriteria={'attributeName': 'severity', 'orderBy': 'DESC'}
    )
    
    report = {
        'generatedAt': datetime.now().isoformat(),
        'summary': {
            'totalFindings': len(findings),
            'byType': {},
            'bySeverity': {}
        },
        'findings': findings
    }
    
    # Upload to compliance bucket
    s3.put_object(
        Bucket='compliance-reports',
        Key=f'macie/monthly/{datetime.now().strftime("%Y-%m")}.json',
        Body=json.dumps(report)
    )

Cost Optimization

Managing Macie Costs

Optimize spending on data discovery:

  1. Selective Scanning: Target specific buckets
  2. Sampling: Use percentage-based sampling for large datasets
  3. Scheduling: Run intensive scans during off-peak hours
  4. Exclusions: Skip known safe file types

Cost Monitoring

MacieCostAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: macie-monthly-cost-limit
    MetricName: EstimatedCharges
    Namespace: AWS/Billing
    Dimensions:
      - Name: ServiceName
        Value: AWSMacie
    Statistic: Maximum
    Period: 86400
    EvaluationPeriods: 1
    Threshold: 500
    ComparisonOperator: GreaterThanThreshold

Working with Warqline

We are a cloud engineering consultancy and an official AWS and Google Cloud partner. If you are running this in production and want a second pair of eyes, we scope work in a free 45-minute technical call: you describe what you are running and what worries you, and we tell you what we would look at first.

Talk to an engineer

Conclusion

AWS Macie provides essential data discovery and classification capabilities for modern cloud security. By implementing Macie alongside an independent review, organizations can maintain comprehensive visibility into their sensitive data and ensure ongoing compliance.