Amazon Simple Storage Service Developer Guide

Table of Contents What Is Amazon S3? .......................................................................................................................... 1 How Do I...? ............................................................................................................................... 1 Introduction ...................................................................................................................................... 2 Overview of Amazon S3 and This Guide ....................................................................................... 2 Advantages to Amazon S3 .......................................................................................................... 2 Amazon S3 Concepts .................................................................................................................. 3 Buckets ............................................................................................................................. 3 Objects ............................................................................................................................. 3 Keys ................................................................................................................................. 4 Regions ............................................................................................................................. 4 Amazon S3 Data Consistency Model ..................................................................................... 4 Features .................................................................................................................................... 6 Reduced Redundancy Storage .............................................................................................. 7 Bucket Policies ................................................................................................................... 7 AWS Identity and Access Management .................................................................................. 8 Access Control Lists ............................................................................................................ 8 Versioning ......................................................................................................................... 8 Operations ........................................................................................................................ 8 Amazon S3 Application Programming Interfaces (API) ..................................................................... 8 The REST Interface ............................................................................................................. 9 The SOAP Interface ............................................................................................................ 9 Paying for Amazon S3 ................................................................................................................ 9 Related Services ......................................................................................................................... 9 Making Requests .............................................................................................................................. 11 About Access Keys .................................................................................................................... 11 AWS Account Access Keys .................................................................................................. 11 IAM User Access Keys ........................................................................................................ 12 Temporary Security Credentials .......................................................................................... 12 Request Endpoints .................................................................................................................... 13 Making Requests over IPv6 ........................................................................................................ 13 Getting Started with IPv6 .................................................................................................. 13 Using IPv6 Addresses in IAM Policies .................................................................................. 14 Testing IP Address Compatibility ........................................................................................ 15 Using DualStack Endpoints ............................................................................................... 15 Making Requests Using the AWS SDKs ........................................................................................ 19 Using AWS Account or IAM User Credentials ........................................................................ 19 Using IAM User Temporary Credentials ............................................................................... 26 Using Federated User Temporary Credentials ....................................................................... 36 Making Requests Using the REST API .......................................................................................... 49 DualStack Endpoints (REST API) ........................................................................................ 50 Virtual Hosting of Buckets ................................................................................................. 50 Request Redirection and the REST API ................................................................................ 54 Buckets ........................................................................................................................................... 57 Creating a Bucket ..................................................................................................................... 58 About Permissions ............................................................................................................ 59 Accessing a Bucket ................................................................................................................... 59 Bucket Configuration Options .................................................................................................... 60 Restrictions and Limitations ....................................................................................................... 61 Rules for Naming ............................................................................................................. 62 Examples of Creating a Bucket ................................................................................................... 63 Using the Amazon S3 Console ........................................................................................... 63 Using the AWS SDK for Java .............................................................................................. 63 Using the AWS SDK for .NET .............................................................................................. 64 Using the AWS SDK for Ruby Version 2 ............................................................................... 66 API Version 20060301 iii Amazon Simple Storage Service Developer Guide Using Other AWS SDKs ..................................................................................................... 66 Deleting or Emptying a Bucket .................................................................................................. 66 Delete a Bucket ................................................................................................................ 66 Empty a Bucket ................................................................................................................ 69 Bucket Website Configuration .................................................................................................... 70 Using the AWS Management Console ................................................................................. 71 Using the SDK for Java ..................................................................................................... 71 Using the AWS SDK for .NET .............................................................................................. 73 Using the SDK for PHP ..................................................................................................... 75 Using the REST API .......................................................................................................... 77 Transfer Acceleration ................................................................................................................ 77 Why use Transfer Acceleration? .......................................................................................... 77 Getting Started ................................................................................................................ 78 Requirements for Using Amazon S3 Transfer Acceleration ...................................................... 79 Transfer Acceleration Examples .......................................................................................... 80 Requester Pays Buckets ............................................................................................................. 89 Configure with the Console ............................................................................................... 89 Configure with the REST API ............................................................................................. 90 DevPay and Requester Pays ............................................................................................... 92 Charge Details ................................................................................................................. 92 Access Control ......................................................................................................................... 92 Billing and Usage Reporting ...................................................................................................... 93 Billing Reports ................................................................................................................. 93 Usage Report ................................................................................................................... 94 Understanding Billing and Usage Reports ............................................................................ 96 Using Cost Allocation Tags .............................................................................................. 100 Objects .......................................................................................................................................... 102 Object Key and Metadata ........................................................................................................ 103 Object Keys ................................................................................................................... 103 Object Metadata ............................................................................................................. 105 Storage Classes ...................................................................................................................... 107 Subresources .......................................................................................................................... 110 Versioning ............................................................................................................................. 110 Object Tagging ....................................................................................................................... 112 API Operations Related to Object Tagging ......................................................................... 113 Object Tagging and Additional Information ....................................................................... 114 Managing Object Tags ..................................................................................................... 118 Lifecycle Management ............................................................................................................. 121 When Should I Use Lifecycle Configuration for Objects? ...................................................... 121 How Do I Configure a Lifecycle? ....................................................................................... 122 Additional Considerations ................................................................................................ 122 Lifecycle Configuration Elements ...................................................................................... 127 Examples of Lifecycle Configuration .................................................................................. 133 Setting Lifecycle Configuration ......................................................................................... 143 CrossOrigin Resource Sharing (CORS) ....................................................................................... 151 CrossOrigin Resource Sharing: Usecase Scenarios .............................................................. 152 How Do I Configure CORS on My Bucket? .......................................................................... 152 How Does Amazon S3 Evaluate the CORS Configuration On a Bucket? ................................... 154 Enabling CORS ............................................................................................................... 154 Troubleshooting CORS .................................................................................................... 163 Operations on Objects ............................................................................................................ 164 Getting Objects .............................................................................................................. 164 Uploading Objects .......................................................................................................... 176 Copying Objects ............................................................................................................. 225 Listing Object Keys ......................................................................................................... 240 Deleting Objects ............................................................................................................. 247 Restoring Archived Objects .............................................................................................. 274 API Version 20060301 iv Amazon Simple Storage Service Developer Guide Storage Class Analysis ..................................................................................................................... 280 How to Set Up Storage Class Analysis ....................................................................................... 280 Storage Class Analysis ............................................................................................................. 281 How Can I Export Storage Class Analysis Data? .......................................................................... 283 Storage Class Analysis Export File Layout .......................................................................... 284 Amazon S3 Analytics REST APIs ............................................................................................... 284 Inventory ....................................................................................................................................... 286 How to Set Up Amazon S3 Inventory ........................................................................................ 286 Amazon S3 Inventory Buckets .......................................................................................... 286 Setting Up Amazon S3 Inventory ...................................................................................... 287 Inventory Lists ....................................................................................................................... 287 Inventory Consistency ..................................................................................................... 288 Location of Inventory Lists ...................................................................................................... 288 What is an Inventory Manifest? ........................................................................................ 289 Notify When Inventory Complete ............................................................................................. 290 Amazon S3 Inventory REST APIs .............................................................................................. 290 Managing Access ............................................................................................................................ 291 Introduction ........................................................................................................................... 291 Overview ....................................................................................................................... 292 How Amazon S3 Authorizes a Request .............................................................................. 297 Guidelines for Using the Available Access Policy Options ..................................................... 302 Example Walkthroughs: Managing Access .......................................................................... 305 Using Bucket Policies and User Policies ..................................................................................... 331 Access Policy Language Overview ..................................................................................... 331 Bucket Policy Examples ................................................................................................... 359 User Policy Examples ...................................................................................................... 368 Managing Access with ACLs ..................................................................................................... 390 Access Control List (ACL) Overview ................................................................................... 390 Managing ACLs ............................................................................................................... 395 Protecting Data .............................................................................................................................. 405 Data Encryption ..................................................................................................................... 405 ServerSide Encryption .................................................................................................... 406 ClientSide Encryption ..................................................................................................... 432 Reduced Redundancy Storage .................................................................................................. 441 Setting the Storage Class of an Object You Upload ............................................................. 442 Changing the Storage Class of an Object in Amazon S3 ....................................................... 443 Versioning ............................................................................................................................. 444 How to Configure Versioning on a Bucket .......................................................................... 445 MFA Delete .................................................................................................................... 446 Related Topics ................................................................................................................ 447 Examples ....................................................................................................................... 447 Managing Objects in a VersioningEnabled Bucket .............................................................. 449 Managing Objects in a VersioningSuspended Bucket .......................................................... 463 Hosting a Static Website ................................................................................................................. 467 Website Endpoints .................................................................................................................. 468 Key Differences Between the Amazon Website and the REST API Endpoint ............................. 469 Configuring a Bucket for Website Hosting ................................................................................. 469 Enabling Website Hosting ................................................................................................ 470 Configuring Index Document Support ............................................................................... 470 Permissions Required for Website Access ........................................................................... 472 (Optional) Configuring Web Traffic Logging ....................................................................... 472 (Optional) Custom Error Document Support ....................................................................... 473 (Optional) Configuring a Redirect ..................................................................................... 474 Example Walkthroughs ............................................................................................................ 481 Example: Setting up a Static Website ................................................................................ 481 Example: Setting up a Static Website Using a Custom Domain .............................................. 483 Example: Speed Up Your Website with Amazon CloudFront .................................................. 491 API Version 20060301 v Amazon Simple Storage Service Developer Guide Clean Up Example Resources ........................................................................................... 493 Notifications .................................................................................................................................. 495 Overview ............................................................................................................................... 495 How to Enable Event Notifications ............................................................................................ 496 Event Notification Types and Destinations ................................................................................. 498 Supported Event Types ................................................................................................... 498 Supported Destinations ................................................................................................... 499 Configuring Notifications with Object Key Name Filtering ............................................................ 499 Examples of Valid Notification Configurations with Object Key Name Filtering ........................ 500 Examples of Notification Configurations with Invalid PrefixSuffix Overlapping ....................... 502 Granting Permissions to Publish Event Notification Messages to a Destination ................................ 504 Granting Permissions to Invoke an AWS Lambda Function ................................................... 504 Granting Permissions to Publish Messages to an SNS Topic or an SQS Queue .......................... 504 Example Walkthrough 1 .......................................................................................................... 505 Walkthrough Summary ................................................................................................... 506 Step 1: Create an Amazon SNS Topic ................................................................................ 506 Step 2: Create an Amazon SQS Queue .............................................................................. 507 Step 3: Add a Notification Configuration to Your Bucket ...................................................... 508 Step 4: Test the Setup .................................................................................................... 511 Example Walkthrough 2 .......................................................................................................... 511 Event Message Structure ......................................................................................................... 511 CrossRegion Replication ................................................................................................................. 514 Usecase Scenarios .................................................................................................................. 514 Requirements ......................................................................................................................... 514 Related Topics ........................................................................................................................ 515 What Is and Is Not Replicated .................................................................................................. 515 What Is Replicated .......................................................................................................... 515 What Is Not Replicated ................................................................................................... 516 Related Topics ................................................................................................................ 517 How to Set Up ....................................................................................................................... 517 Create an IAM Role ......................................................................................................... 517 Add Replication Configuration .......................................................................................... 519 Walkthrough 1: Same AWS Account .................................................................................. 522 Walkthrough 2: Different AWS Accounts ............................................................................ 523 Using the Console .......................................................................................................... 527 Using the AWS SDK for Java ............................................................................................ 527 Using the AWS SDK for .NET ............................................................................................ 528 Replication Status Information ................................................................................................. 530 Related Topics ................................................................................................................ 532 Troubleshooting ..................................................................................................................... 532 Related Topics ................................................................................................................ 533 Replication and Other Bucket Configurations ............................................................................. 533 Lifecycle Configuration and Object Replicas ....................................................................... 533 Versioning Configuration and Replication Configuration ...................................................... 533 Logging Configuration and Replication Configuration .......................................................... 533 Related Topics ................................................................................................................ 533 Request Routing ............................................................................................................................. 534 Request Redirection and the REST API ...................................................................................... 534 Overview ....................................................................................................................... 534 DNS Routing .................................................................................................................. 534 Temporary Request Redirection ........................................................................................ 535 Permanent Request Redirection ........................................................................................ 537 DNS Considerations ................................................................................................................ 537 Performance Optimization ............................................................................................................... 539 Request Rate and Performance Considerations ........................................................................... 539 Workloads with a Mix of Request Types ............................................................................. 540 GETIntensive Workloads ................................................................................................. 542 API Version 20060301 vi Amazon Simple Storage Service Developer Guide TCP Window Scaling ............................................................................................................... 542 TCP Selective Acknowledgement .............................................................................................. 543 Monitoring ..................................................................................................................................... 544 Monitoring Tools .................................................................................................................... 544 Automated Tools ............................................................................................................ 544 Manual Tools ................................................................................................................. 545 Monitoring Metrics with CloudWatch ......................................................................................... 545 Metrics and Dimensions ................................................................................................... 546 Amazon S3 CloudWatch Daily Storage Metrics for Buckets ................................................... 546 Amazon S3 CloudWatch Request metrics ........................................................................... 546 Amazon S3 CloudWatch Dimensions ................................................................................. 548 Accessing CloudWatch Metrics .......................................................................................... 549 Related Resources ........................................................................................................... 550 Metrics Configurations for Buckets ............................................................................................ 550 BestEffort CloudWatch Metrics Delivery ............................................................................ 550 Filtering Metrics Configurations ........................................................................................ 551 How to Add Metrics Configurations ................................................................................... 551 Logging API Calls with AWS CloudTrail ...................................................................................... 552 Amazon S3 Information in CloudTrail ................................................................................ 552 Using CloudTrail Logs with Amazon S3 Server Access Logs and CloudWatch Logs .................... 556 Understanding Amazon S3 Log File Entries ........................................................................ 556 Related Resources ........................................................................................................... 558 BitTorrent ...................................................................................................................................... 559 How You are Charged for BitTorrent Delivery ............................................................................. 559 Using BitTorrent to Retrieve Objects Stored in Amazon S3 ........................................................... 560 Publishing Content Using Amazon S3 and BitTorrent .................................................................. 561 Amazon DevPay ............................................................................................................................. 562 Amazon S3 Customer Data Isolation ......................................................................................... 562 Example ........................................................................................................................ 563 Amazon DevPay Token Mechanism ........................................................................................... 563 Amazon S3 and Amazon DevPay Authentication ......................................................................... 563 Amazon S3 Bucket Limitation .................................................................................................. 564 Amazon S3 and Amazon DevPay Process ................................................................................... 565 Additional Information ............................................................................................................ 565 Error Handling ............................................................................................................................... 566 The REST Error Response ........................................................................................................ 566 Response Headers .......................................................................................................... 567 Error Response ............................................................................................................... 567 The SOAP Error Response ........................................................................................................ 568 Amazon S3 Error Best Practices ................................................................................................ 568 Retry InternalErrors ........................................................................................................ 568 Tune Application for Repeated SlowDown errors ................................................................ 568 Isolate Errors ................................................................................................................. 569 Troubleshooting Amazon S3 ............................................................................................................ 570 Troubleshooting Amazon S3 by Symptom ................................................................................. 570 Significant Increases in HTTP 503 Responses to Requests to Buckets with Versioning Enabled .... 570 Unexpected Behavior When Accessing Buckets Set with CORS .............................................. 571 Getting Amazon S3 Request IDs for AWS Support ...................................................................... 571 Using HTTP to Obtain Request IDs ................................................................................... 571 Using a Web Browser to Obtain Request IDs ...................................................................... 571 Using AWS SDKs to Obtain Request IDs ............................................................................. 572 Using the AWS CLI to Obtain Request IDs .......................................................................... 573 Related Topics ........................................................................................................................ 573 Server Access Logging ..................................................................................................................... 574 Overview ............................................................................................................................... 574 Log Object Key Format ................................................................................................... 575 How are Logs Delivered? ................................................................................................. 575 API Version 20060301 vii Amazon Simple Storage Service Developer Guide Best Effort Server Log Delivery ........................................................................................ 575 Bucket Logging Status Changes Take Effect Over Time ........................................................ 576 Related Topics ........................................................................................................................ 576 Enabling Logging Using the Console ......................................................................................... 576 Enabling Logging Programmatically .......................................................................................... 576 Enabling logging ............................................................................................................ 577 Granting the Log Delivery Group WRITE and READ_ACP Permissions ..................................... 577 Example: AWS SDK for .NET ............................................................................................. 578 Log Format ............................................................................................................................ 579 Custom Access Log Information ........................................................................................ 583 Programming Considerations for Extensible Server Access Log Format ................................... 583 Additional Logging for Copy Operations ............................................................................ 583 Deleting Log Files ................................................................................................................... 586 AWS SDKs and Explorers ................................................................................................................. 587 Specifying Signature Version in Request Authentication ............................................................... 588 Set Up the AWS CLI ................................................................................................................ 589 Using the AWS SDK for Java .................................................................................................... 590 The Java API Organization ............................................................................................... 591 Testing the Java Code Examples ....................................................................................... 591 Using the AWS SDK for .NET .................................................................................................... 591 The .NET API Organization ............................................................................................... 592 Running the Amazon S3 .NET Code Examples .................................................................... 593 Using the AWS SDK for PHP and Running PHP Examples ............................................................. 593 AWS SDK for PHP Levels ................................................................................................ 593 Running PHP Examples ................................................................................................... 594 Related Resources ........................................................................................................... 594 Using the AWS SDK for Ruby Version 2 ................................................................................... 595 The Ruby API Organization .............................................................................................. 595 Testing the Ruby Script Examples ..................................................................................... 595 Using the AWS SDK for Python (Boto) ....................................................................................... 596 Appendices .................................................................................................................................... 597 Appendix A: Using the SOAP API .............................................................................................. 597 Common SOAP API Elements ........................................................................................... 597 Authenticating SOAP Requests ......................................................................................... 598 Setting Access Policy with SOAP ....................................................................................... 599 Appendix B: Authenticating Requests (AWS Signature Version 2) ................................................... 600 Authenticating Requests Using the REST API ...................................................................... 601 Signing and Authenticating REST Requests ........................................................................ 603 BrowserBased Uploads Using POST ................................................................................. 612 Resources ...................................................................................................................................... 627 Document History .......................................................................................................................... 629 AWS Glossary ................................................................................................................................. 641

Amazon Simple Storage Service Developer Guide API Version 2006-03-01 Amazon Simple Storage Service Developer Guide Amazon Simple Storage Service: Developer Guide Copyright © 2017 Amazon Web Services, Inc and/or its affiliates All rights reserved Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon Amazon Simple Storage Service Developer Guide Table of Contents What Is Amazon S3? How Do I ? Introduction Overview of Amazon S3 and This Guide Advantages to Amazon S3 Amazon S3 Concepts Buckets Objects Keys Regions Amazon S3 Data Consistency Model Features Reduced Redundancy Storage Bucket Policies AWS Identity and Access Management Access Control Lists Versioning Operations Amazon S3 Application Programming Interfaces (API) The REST Interface The SOAP Interface Paying for Amazon S3 Related Services Making Requests 11 About Access Keys 11 AWS Account Access Keys 11 IAM User Access Keys 12 Temporary Security Credentials 12 Request Endpoints 13 Making Requests over IPv6 13 Getting Started with IPv6 13 Using IPv6 Addresses in IAM Policies 14 Testing IP Address Compatibility 15 Using Dual-Stack Endpoints 15 Making Requests Using the AWS SDKs 19 Using AWS Account or IAM User Credentials 19 Using IAM User Temporary Credentials 26 Using Federated User Temporary Credentials 36 Making Requests Using the REST API 49 Dual-Stack Endpoints (REST API) 50 Virtual Hosting of Buckets 50 Request Redirection and the REST API 54 Buckets 57 Creating a Bucket 58 About Permissions 59 Accessing a Bucket 59 Bucket Configuration Options 60 Restrictions and Limitations 61 Rules for Naming 62 Examples of Creating a Bucket 63 Using the Amazon S3 Console 63 Using the AWS SDK for Java 63 Using the AWS SDK for NET 64 Using the AWS SDK for Ruby Version 66 API Version 2006-03-01 iii Amazon Simple Storage Service Developer Guide Using Other AWS SDKs 66 Deleting or Emptying a Bucket 66 Delete a Bucket 66 Empty a Bucket 69 Bucket Website Configuration 70 Using the AWS Management Console 71 Using the SDK for Java 71 Using the AWS SDK for NET 73 Using the SDK for PHP 75 Using the REST API 77 Transfer Acceleration 77 Why use Transfer Acceleration? 77 Getting Started 78 Requirements for Using Amazon S3 Transfer Acceleration 79 Transfer Acceleration Examples 80 Requester Pays Buckets 89 Configure with the Console 89 Configure with the REST API 90 DevPay and Requester Pays 92 Charge Details 92 Access Control 92 Billing and Usage Reporting 93 Billing Reports 93 Usage Report 94 Understanding Billing and Usage Reports 96 Using Cost Allocation Tags 100 Objects 102 Object Key and Metadata 103 Object Keys 103 Object Metadata 105 Storage Classes 107 Subresources 110 Versioning 110 Object Tagging 112 API Operations Related to Object Tagging 113 Object Tagging and Additional Information 114 Managing Object Tags 118 Lifecycle Management 121 When Should I Use Lifecycle Configuration for Objects? 121 How Do I Configure a Lifecycle? 122 Additional Considerations 122 Lifecycle Configuration Elements 127 Examples of Lifecycle Configuration 133 Setting Lifecycle Configuration 143 Cross-Origin Resource Sharing (CORS) 151 Cross-Origin Resource Sharing: Use-case Scenarios 152 How Do I Configure CORS on My Bucket? 152 How Does Amazon S3 Evaluate the CORS Configuration On a Bucket? 154 Enabling CORS 154 Troubleshooting CORS 163 Operations on Objects 164 Getting Objects 164 Uploading Objects 176 Copying Objects 225 Listing Object Keys 240 Deleting Objects 247 Restoring Archived Objects 274 API Version 2006-03-01 iv Amazon Simple Storage Service Developer Guide Storage Class Analysis How to Set Up Storage Class Analysis Storage Class Analysis How Can I Export Storage Class Analysis Data? Storage Class Analysis Export File Layout Amazon S3 Analytics REST APIs Inventory How to Set Up Amazon S3 Inventory Amazon S3 Inventory Buckets Setting Up Amazon S3 Inventory Inventory Lists Inventory Consistency Location of Inventory Lists What is an Inventory Manifest? Notify When Inventory Complete Amazon S3 Inventory REST APIs Managing Access Introduction Overview How Amazon S3 Authorizes a Request Guidelines for Using the Available Access Policy Options Example Walkthroughs: Managing Access Using Bucket Policies and User Policies Access Policy Language Overview Bucket Policy Examples User Policy Examples Managing Access with ACLs Access Control List (ACL) Overview Managing ACLs Protecting Data Data Encryption Server-Side Encryption Client-Side Encryption Reduced Redundancy Storage Setting the Storage Class of an Object You Upload Changing the Storage Class of an Object in Amazon S3 Versioning How to Configure Versioning on a Bucket MFA Delete Related Topics Examples Managing Objects in a Versioning-Enabled Bucket Managing Objects in a Versioning-Suspended Bucket Hosting a Static Website Website Endpoints Key Differences Between the Amazon Website and the REST API Endpoint Configuring a Bucket for Website Hosting Enabling Website Hosting Configuring Index Document Support Permissions Required for Website Access (Optional) Configuring Web Traffic Logging (Optional) Custom Error Document Support (Optional) Configuring a Redirect Example Walkthroughs Example: Setting up a Static Website Example: Setting up a Static Website Using a Custom Domain Example: Speed Up Your Website with Amazon CloudFront API Version 2006-03-01 v 280 280 281 283 284 284 286 286 286 287 287 288 288 289 290 290 291 291 292 297 302 305 331 331 359 368 390 390 395 405 405 406 432 441 442 443 444 445 446 447 447 449 463 467 468 469 469 470 470 472 472 473 474 481 481 483 491 Amazon Simple Storage Service Developer Guide Clean Up Example Resources Notifications Overview How to Enable Event Notifications Event Notification Types and Destinations Supported Event Types Supported Destinations Configuring Notifications with Object Key Name Filtering Examples of Valid Notification Configurations with Object Key Name Filtering Examples of Notification Configurations with Invalid Prefix/Suffix Overlapping Granting Permissions to Publish Event Notification Messages to a Destination Granting Permissions to Invoke an AWS Lambda Function Granting Permissions to Publish Messages to an SNS Topic or an SQS Queue Example Walkthrough Walkthrough Summary Step 1: Create an Amazon SNS Topic Step 2: Create an Amazon SQS Queue Step 3: Add a Notification Configuration to Your Bucket Step 4: Test the Setup Example Walkthrough Event Message Structure Cross-Region Replication Use-case Scenarios Requirements Related Topics What Is and Is Not Replicated What Is Replicated What Is Not Replicated Related Topics How to Set Up Create an IAM Role Add Replication Configuration Walkthrough 1: Same AWS Account Walkthrough 2: Different AWS Accounts Using the Console Using the AWS SDK for Java Using the AWS SDK for NET Replication Status Information Related Topics Troubleshooting Related Topics Replication and Other Bucket Configurations Lifecycle Configuration and Object Replicas Versioning Configuration and Replication Configuration Logging Configuration and Replication Configuration Related Topics Request Routing Request Redirection and the REST API Overview DNS Routing Temporary Request Redirection Permanent Request Redirection DNS Considerations Performance Optimization Request Rate and Performance Considerations Workloads with a Mix of Request Types GET-Intensive Workloads API Version 2006-03-01 vi 493 495 495 496 498 498 499 499 500 502 504 504 504 505 506 506 507 508 511 511 511 514 514 514 515 515 515 516 517 517 517 519 522 523 527 527 528 530 532 532 533 533 533 533 533 533 534 534 534 534 535 537 537 539 539 540 542 Amazon Simple Storage Service Developer Guide TCP Window Scaling TCP Selective Acknowledgement Monitoring Monitoring Tools Automated Tools Manual Tools Monitoring Metrics with CloudWatch Metrics and Dimensions Amazon S3 CloudWatch Daily Storage Metrics for Buckets Amazon S3 CloudWatch Request metrics Amazon S3 CloudWatch Dimensions Accessing CloudWatch Metrics Related Resources Metrics Configurations for Buckets Best-Effort CloudWatch Metrics Delivery Filtering Metrics Configurations How to Add Metrics Configurations Logging API Calls with AWS CloudTrail Amazon S3 Information in CloudTrail Using CloudTrail Logs with Amazon S3 Server Access Logs and CloudWatch Logs Understanding Amazon S3 Log File Entries Related Resources BitTorrent How You are Charged for BitTorrent Delivery Using BitTorrent to Retrieve Objects Stored in Amazon S3 Publishing Content Using Amazon S3 and BitTorrent Amazon DevPay Amazon S3 Customer Data Isolation Example Amazon DevPay Token Mechanism Amazon S3 and Amazon DevPay Authentication Amazon S3 Bucket Limitation Amazon S3 and Amazon DevPay Process Additional Information Error Handling The REST Error Response Response Headers Error Response The SOAP Error Response Amazon S3 Error Best Practices Retry InternalErrors Tune Application for Repeated SlowDown errors Isolate Errors Troubleshooting Amazon S3 Troubleshooting Amazon S3 by Symptom Significant Increases in HTTP 503 Responses to Requests to Buckets with Versioning Enabled Unexpected Behavior When Accessing Buckets Set with CORS Getting Amazon S3 Request IDs for AWS Support Using HTTP to Obtain Request IDs Using a Web Browser to Obtain Request IDs Using AWS SDKs to Obtain Request IDs Using the AWS CLI to Obtain Request IDs Related Topics Server Access Logging Overview Log Object Key Format How are Logs Delivered? API Version 2006-03-01 vii 542 543 544 544 544 545 545 546 546 546 548 549 550 550 550 551 551 552 552 556 556 558 559 559 560 561 562 562 563 563 563 564 565 565 566 566 567 567 568 568 568 568 569 570 570 570 571 571 571 571 572 573 573 574 574 575 575 Amazon Simple Storage Service Developer Guide Best Effort Server Log Delivery 575 Bucket Logging Status Changes Take Effect Over Time 576 Related Topics 576 Enabling Logging Using the Console 576 Enabling Logging Programmatically 576 Enabling logging 577 Granting the Log Delivery Group WRITE and READ_ACP Permissions 577 Example: AWS SDK for NET 578 Log Format 579 Custom Access Log Information 583 Programming Considerations for Extensible Server Access Log Format 583 Additional Logging for Copy Operations 583 Deleting Log Files 586 AWS SDKs and Explorers 587 Specifying Signature Version in Request Authentication 588 Set Up the AWS CLI 589 Using the AWS SDK for Java 590 The Java API Organization 591 Testing the Java Code Examples 591 Using the AWS SDK for NET 591 The NET API Organization 592 Running the Amazon S3 NET Code Examples 593 Using the AWS SDK for PHP and Running PHP Examples 593 AWS SDK for PHP Levels 593 Running PHP Examples 594 Related Resources 594 Using the AWS SDK for Ruby - Version 595 The Ruby API Organization 595 Testing the Ruby Script Examples 595 Using the AWS SDK for Python (Boto) 596 Appendices 597 Appendix A: Using the SOAP API 597 Common SOAP API Elements 597 Authenticating SOAP Requests 598 Setting Access Policy with SOAP 599 Appendix B: Authenticating Requests (AWS Signature Version 2) 600 Authenticating Requests Using the REST API 601 Signing and Authenticating REST Requests 603 Browser-Based Uploads Using POST 612 Resources 627 Document History 629 AWS Glossary 641 API Version 2006-03-01 viii Amazon Simple Storage Service Developer Guide How Do I ? What Is Amazon S3? Amazon Simple Storage Service is storage for the Internet It is designed to make web-scale computing easier for developers Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites The service aims to maximize benefits of scale and to pass those benefits on to developers This guide explains the core concepts of Amazon S3, such as buckets and objects, and how to work with these resources using the Amazon S3 application programming interface (API) How Do I ? Information Relevant Sections General product overview and pricing Amazon S3 Get a quick hands-on introduction to Amazon S3 Amazon Simple Storage Service Getting Started Guide Learn about Amazon S3 key terminology and concepts Introduction to Amazon S3 (p 2) How I work with buckets? Working with Amazon S3 Buckets (p 57) How I work with objects? Working with Amazon S3 Objects (p 102) How I make requests? Making Requests (p 11) How I manage access to my resources? Managing Access Permissions to Your Amazon S3 Resources (p 291) API Version 2006-03-01 Amazon Simple Storage Service Developer Guide Overview of Amazon S3 and This Guide Introduction to Amazon S3 This introduction to Amazon Simple Storage Service is intended to give you a detailed summary of this web service After reading this section, you should have a good idea of what it offers and how it can fit in with your business Topics • Overview of Amazon S3 and This Guide (p 2) • Advantages to Amazon S3 (p 2) • Amazon S3 Concepts (p 3) • Features (p 6) • Amazon S3 Application Programming Interfaces (API) (p 8) • Paying for Amazon S3 (p 9) • Related Services (p 9) Overview of Amazon S3 and This Guide Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web This guide describes how you send requests to create buckets, store and retrieve your objects, and manage permissions on your resources The guide also describes access control and the authentication process Access control defines who can access objects and buckets within Amazon S3, and the type of access (e.g., READ and WRITE) The authentication process verifies the identity of a user who is trying to access Amazon Web Services (AWS) Advantages to Amazon S3 Amazon S3 is intentionally built with a minimal feature set that focuses on simplicity and robustness Following are some of advantages of the Amazon S3 service: • Create Buckets – Create and name a bucket that stores data Buckets are the fundamental container in Amazon S3 for data storage API Version 2006-03-01 .. .Amazon Simple Storage Service Developer Guide Amazon Simple Storage Service: Developer Guide Copyright © 2017 Amazon Web Services, Inc and/or its affiliates All rights reserved Amazon'' s... viii Amazon Simple Storage Service Developer Guide How Do I ? What Is Amazon S3? Amazon Simple Storage Service is storage for the Internet It is designed to make web-scale computing easier for developers... Your Amazon S3 Resources (p 291) API Version 2006-03-01 Amazon Simple Storage Service Developer Guide Overview of Amazon S3 and This Guide Introduction to Amazon S3 This introduction to Amazon Simple