AWS Storage Services Overview
AWS Storage ServicesOverviewA Look at Storage Services Offered by AWSDecember 2016
2016, Amazon Web Services, Inc. or its affiliates. All rights reserved.NoticesThis document is provided for informational purposes only. It represents AWS’scurrent product offerings and practices as of the date of issue of this document,which are subject to change without notice. Customers are responsible formaking their own independent assessment of the information in this documentand any use of AWS’s products or services, each of which is provided “as is”without warranty of any kind, whether express or implied. This document doesnot create any warranties, representations, contractual commitments, conditionsor assurances from AWS, its affiliates, suppliers or licensors. The responsibilitiesand liabilities of AWS to its customers are controlled by AWS agreements, andthis document is not part of, nor does it modify, any agreement between AWSand its customers.
ContentsAbstract6Introduction1Amazon S31Usage Patterns2Performance3Durability and Availability4Scalability and Elasticity5Security5Interfaces6Cost Model7Amazon Glacier7Usage Patterns8Performance8Durability and Availability9Scalability and Elasticity9Security9Interfaces10Cost Model11Amazon EFS11Usage Patterns12Performance13Durability and Availability15Scalability and Elasticity15Security15Interfaces16Cost Model16
Amazon EBS17Usage Patterns17Performance18Durability and Availability21Scalability and Elasticity22Security23Interfaces23Cost Model24Amazon EC2 Instance Storage24Usage Patterns26Performance27Durability and Availability28Scalability and Elasticity28Security29Interfaces29Cost Model30AWS Storage Gateway30Usage Patterns31Performance32Durability and Availability32Scalability and Elasticity32Security33Interfaces33Cost Model34AWS Snowball34Usage Patterns34Performance35Durability and Availability36
Scalability and Elasticity36Security36Interfaces37Cost Model38Amazon CloudFront39Usage Patterns39Performance40Durability and Availability40Scalability and Elasticity40Security41Interfaces41Cost Model42Conclusion42Contributors43References and Further Reading44AWS Storage Services44Other Resources44
AbstractAmazon Web Services (AWS) is a flexible, cost-effective, easy-to-use cloudcomputing platform. This whitepaper is designed to help architects anddevelopers understand the different storage services and features available in theAWS Cloud. We provide an overview of each storage service or feature anddescribe usage patterns, performance, durability and availability, scalability andelasticity, security, interfaces, and the cost model.
Amazon Web Services – AWS Storage Services OverviewIntroductionAmazon Web Services (AWS) provides low-cost data storage with high durabilityand availability. AWS offers storage choices for backup, archiving, and disasterrecovery use cases and provides block, file, and object storage. In this whitepaper,we examine the following AWS Cloud storage services and features.Amazon Simple Storage ServiceA service that provides scalable and highly(Amazon S3)durable object storage in the cloud.Amazon GlacierA service that provides low-cost highlydurable archive storage in the cloud.Amazon Elastic File SystemA service that provides scalable network(Amazon EFS)file storage for Amazon EC2 instances.Amazon Elastic Block StoreA service that provides block storage(Amazon EBS)volumes for Amazon EC2 instances.Amazon EC2 Instance StorageTemporary block storage volumes forAmazon EC2 instances.AWS Storage GatewayAn on-premises storage appliance thatintegrates with cloud storage.AWS SnowballA service that transports large amounts ofdata to and from the cloud.Amazon CloudFrontA service that provides a global contentdelivery network (CDN).Amazon S3Amazon Simple Storage Service (Amazon S3) provides developers and IT teamssecure, durable, highly scalable object storage at a very low cost.1 You can storeand retrieve any amount of data, at any time, from anywhere on the web througha simple web service interface. You can write, read, and delete objects containingfrom zero to 5 TB of data. Amazon S3 is highly scalable, allowing concurrent reador write access to data by many separate clients or application threads.Page 1
Amazon Web Services – AWS Storage Services OverviewAmazon S3 offers a range of storage classes designed for different use casesincluding the following: Amazon S3 Standard, for general-purpose storage of frequently accessed data Amazon S3 Standard-Infrequent Access (Standard-IA), for long-lived, but lessfrequently accessed data Amazon Glacier, for low-cost archival dataUsage PatternsThere are four common usage patterns for Amazon S3.First, Amazon S3 is used to store and distribute static web content and media.This content can be delivered directly from Amazon S3 because each object inAmazon S3 has a unique HTTP URL. Alternatively, Amazon S3 can serve as anorigin store for a content delivery network (CDN), such as Amazon CloudFront.The elasticity of Amazon S3 makes it particularly well suited for hosting webcontent that requires bandwidth for addressing extreme demand spikes. Also,because no storage provisioning is required, Amazon S3 works well for fastgrowing websites hosting data-intensive, user-generated content, such as videoand photo-sharing sites.Second, Amazon S3 is used to host entire static websites. Amazon S3 provides alow-cost, highly available, and highly scalable solution, including storage forstatic HTML files, images, videos, and client-side scripts in formats such asJavaScript.Third, Amazon S3 is used as a data store for computation and large-scaleanalytics, such as financial transaction analysis, clickstream analytics, and mediatranscoding. Because of the horizontal scalability of Amazon S3, you can accessyour data from multiple computing nodes concurrently without beingconstrained by a single connection.Finally, Amazon S3 is often used as a highly durable, scalable, and securesolution for backup and archiving of critical data. You can easily move cold datato Amazon Glacier using lifecycle management rules on data stored in AmazonS3. You can also use Amazon S3 cross-region replication to automatically copyobjects across S3 buckets in different AWS Regions asynchronously, providingdisaster recovery solutions for business continuity.2Page 2
Amazon Web Services – AWS Storage Services OverviewAmazon S3 doesn’t suit all storage situations. The following table presents somestorage needs for which you should consider other AWS storage options.Storage NeedSolutionAWS ServicesFile systemAmazon S3 uses a flat namespace and isn’t meant toserve as a standalone, POSIX-compliant file system.Instead, consider using Amazon EFS as a file system.Amazon EFSStructured datawith queryAmazon S3 doesn’t offer query capabilities to retrievespecific objects. When you use Amazon S3 you need toknow the exact bucket name and key for the files youwant to retrieve from the service. Amazon S3 can’t beused as a database or search engine by itself. Instead,you can pair Amazon S3 with Amazon DynamoDB,Amazon CloudSearch, or Amazon Relational DatabaseService (Amazon RDS) to index and query metadataabout Amazon S3 buckets and objects.Amazon DynamoDBAmazon RDSAmazon CloudSearchRapidly changingdataData that must be updated very frequently might bebetter served by storage solutions that take into accountread and write latencies, such as Amazon EBS volumes,Amazon RDS, Amazon DynamoDB, Amazon EFS, orrelational databases running on Amazon EC2.Amazon EBSArchival dataData that requires encrypted archival storage withinfrequent read access with a long recovery timeobjective (RTO) can be stored in Amazon Glacier morecost-effectively.Amazon GlacierDynamic websitehostingAlthough Amazon S3 is ideal for static content websites,dynamic websites that depend on database interaction oruse server-side scripting should be hosted on AmazonEC2 or Amazon EFS.Amazon EC2Amazon EFSAmazon DynamoDBAmazon RDSAmazon EFSPerformanceIn scenarios where you use Amazon S3 from within Amazon EC2 in the sameRegion, access to Amazon S3 from Amazon EC2 is designed to be fast. Amazon S3is also designed so that server-side latencies are insignificant relative to Internetlatencies. In addition, Amazon S3 is built to scale storage, requests, and numbersof users to support an extremely large number of web-scale applications. If youaccess Amazon S3 using multiple threads, multiple applications, or multipleclients concurrently, total Amazon S3 aggregate throughput typically scales torates that far exceed what any single server can generate or consume.Page 3
Amazon Web Services – AWS Storage Services OverviewTo improve the upload performance of large objects (typically over 100 MB),Amazon S3 offers a multipart upload command to upload a single object as a setof parts.3 After all parts of your object are uploaded, Amazon S3 assembles theseparts and creates the object. Using multipart upload, you can get improvedthroughput and quick recovery from any network issues. Another benefit of usingmultipart upload is that you can upload multiple parts of a single object inparallel and restart the upload of smaller parts instead of restarting the upload ofthe entire large object.To speed up access to relevant data, many developers pair Amazon S3 with asearch engine such as Amazon CloudSearch or a database such as AmazonDynamoDB or Amazon RDS. In these scenarios, Amazon S3 stores the actualinformation, and the search engine or database serves as the repository forassociated metadata (for example, the object name, size, keywords, and so on).Metadata in the database can easily be indexed and queried, making it veryefficient to locate an object’s reference by using a search engine or a databasequery. This result can be used to pinpoint and retrieve the object itself fromAmazon S3.Amazon S3 Transfer Acceleration enables fast, easy, and secure transfer of filesover long distances between your client and your Amazon S3 bucket. It leveragesAmazon CloudFront globally distributed edge locations to route traffic to yourAmazon S3 bucket over an Amazon-optimized network path. To get started withAmazon S3 Transfer Acceleration you first must enable it on an Amazon S3bucket. Then modify your Amazon S3 PUT and GET requests to use the s3accelerate endpoint domain name ( bucketname .s3accelerate.amazonaws.com). The Amazon S3 bucket can still be accessed usingthe regular endpoint. Some customers have measured performanceimprovements in excess of 500 percent when performing intercontinentaluploads.Durability and AvailabilityAmazon S3 Standard storage and Standard-IA storage provide high levels of datadurability and availability by automatically and synchronously storing your dataacross both multiple devices and multiple facilities within your selectedgeographical region. Error correction is built-in, and there are no single points offailure. Amazon S3 is designed to sustain the concurrent loss of data in twofacilities, making it very well suited to serve as the primary data storage forPage 4
Amazon Web Services – AWS Storage Services Overviewmission-critical data. In fact, Amazon S3 is designed for 99.999999999 percent(11 nines) durability per object and 99.99 percent availability over a one-yearperiod.Additionally, you have a choice of enabling cross-region replication on eachAmazon S3 bucket. Once enabled, cross-region replication automatically copiesobjects across buckets in different AWS Regions asynchronously, providing 11nines of durability and 4 nines of availability on both the source and destinationAmazon S3 objects.Scalability and ElasticityAmazon S3 has been designed to offer a very high level of automatic scalabilityand elasticity. Unlike a typical file system that encounters issues when storing alarge number of files in a directory, Amazon S3 supports a virtually unlimitednumber of files in any bucket. Also, unlike a disk drive that has a limit on the totalamount of data that can be stored before you must partition the data acrossdrives and/or servers, an Amazon S3 bucket can store a virtually unlimitednumber of bytes. You can store any number of objects (files) in a single bucket,and Amazon S3 will automatically manage scaling and distributing redundantcopies of your information to other servers in other locations in the same Region,all using Amazon’s high-performance infrastructure.SecurityAmazon S3 is highly secure. It provides multiple mechanisms for fine-grainedcontrol of access to Amazon S3 resources, and it supports encryption.You can manage access to Amazon S3 by granting other AWS accounts and userspermission to perform the resource operations by writing an access policy.4You can protect Amazon S3 data at rest by using server-side encryption,5 in whichyou request Amazon S3 to encrypt your object before it’s written to disks in datacenters and decrypt it when you download the object or by using client-sideencryption,6 in which you encrypt your data on the client side and upload theencrypted data to Amazon S3. You can protect the data in transit by using SecureSockets Layer (SSL) or client-side encryption.Page 5
Amazon Web Services – AWS Storage Services OverviewYou can use versioning to preserve, retrieve, and restore every version of everyobject stored in your Amazon S3 bucket. With versioning, you can easily recoverfrom both unintended user actions and application failures. Additionally, you canadd an optional layer of security by enabling Multi-Factor Authentication (MFA)Delete for a bucket.7 With this option enabled for a bucket, two forms ofauthentication are required to change the versioning state of the bucket or topermanently delete an object version: valid AWS account credentials plus a sixdigit code (a single-use, time-based password) from a physical or virtual tokendevice.To track requests for access to your bucket, you can enable access logging.8 Eachaccess log record provides details about a single access request, such as therequester, bucket name, request time, request action, response status, and errorcode, if any. Access log information can be useful in security and access audits. Itcan also help you learn about your customer base and understand your AmazonS3 bill.InterfacesAmazon S3 provides standards-based REST web service application programinterfaces (APIs) for both management and data operations. These APIs allowAmazon S3 objects to be stored in uniquely named buckets (top-level folders).Each object must have a unique object key (file name) that serves as an identifierfor the object within that bucket. Although Amazon S3 is a web-based object storewith a flat naming structure rather than a traditional file system, you can easilyemulate a file system hierarchy (folder1/folder2/file) in Amazon S3 by creatingobject key names that correspond to the full path name of each file.Most developers building applications on Amazon S3 use a higher-level toolkit orsoftware development kit (SDK) that wraps the underlying REST API. AWS SDKsare available for Android, Browser, iOS, Java, .NET, Node.js, PHP, Python, Ruby,and Go. The integrated AWS Command Line Interface (AWS CLI) also provides aset of high-level, Linux-like Amazon S3 file commands for common operations,such as ls, cp, mv, sync, and so on. Using the AWS CLI for Amazon S3, you canperform recursive uploads and downloads using a single folder-level Amazon S3command and also perform parallel transfers. You can also use the AWS CLI forcommand-line access to the low-level Amazon S3 API. Using the AWSManagement Console, you can easily create and manage Amazon S3 buckets,Page 6
Amazon Web Services – AWS Storage Services Overviewupload and download objects, and browse the contents of your S3 buckets using asimple web-based user interface.Additionally, you can use the Amazon S3 notification feature to receivenotifications when certain events happen in your bucket. Currently, Amazon S3can publish events when an object is uploaded or when an object is deleted.Notifications can be issued to Amazon Simple Notification Service (SNS) topics,9Amazon Simple Queue Service (SQS) queues,10 and AWS Lambda functions.11Cost ModelWith Amazon S3, you pay only for the storage you actually use. There is nominimum fee and no setup cost. Amazon S3 Standard has three pricingcomponents: storage (per GB per month), data transfer in or out (per GB permonth), and requests (per thousand requests per month). For new customers,AWS provides the AWS Free Tier, which includes up to 5 GB of Amazon S3storage, 20,000 get requests, 2,000 put requests, and 15 GB of data transfer outeach month for one year, for free.12 You can find pricing information at theAmazon S3 pricing page.13 There are Data Transfer IN and OUT fees if you enableAmazon S3 Transfer Acceleration on a bucket and the transfer performance isfaster than regular Amazon S3 transfer. If we determine that TransferAcceleration is not likely to be faster than a regular Amazon S3 transfer of thesame object to the same destination, we will not charge for that use of TransferAcceleration for that transfer, and may bypass the Transfer Acceleration systemfor that upload.Amazon GlacierAmazon Glacier is an extremely low-cost storage service that provides highlysecure, durable, and flexible storage for data archiving and online backup.14 WithAmazon Glacier, you can reliably store your data for as little as 0.007 pergigabyte per month. Amazon Glacier enables you to offload the administrativeburdens of operating and scaling storage to AWS so that you don’t have to worryabout capacity planning, hardware provisioning, data replication, hardwarefailure detection and repair, or time-consuming hardware migrations.You store data in Amazon Glacier as archives. An archive can represent a singlefile, or you can combine several files to be uploaded as a single archive.Page 7
Amazon Web Services – AWS Storage Services OverviewRetrieving archives from Amazon Glacier requires the initiation of a job. Youorganize your archives in vaults.Amazon Glacier is designed for use with other Amazon web services. You canseamlessly move data between Amazon Glacier and Amazon S3 using S3 datalifecycle policies.Usage PatternsOrganizations are using Amazon Glacier to support a number of use cases. Theseuse cases include archiving offsite enterprise information, media assets, andresearch and scientific data, and also performing digital preservation andmagnetic tape replacement.Amazon Glacier doesn’t suit all storage situations. The following table presents afew storage needs for which you should consider other AWS storage options.Storage NeedSolutionAWS ServicesRapidlychanging dataData that must be updated very frequently might bebetter served by a storage solution with lower read/writelatencies, such as Amazon EBS, Amazon RDS, AmazonEFS, Amazon DynamoDB, or relational databasesrunning on Amazon EC2.Amazon EBSAmazon RDSData stored in Amazon Glacier is not availableimmediately. Retrieval jobs typically require 3–5 hours tocomplete, so if you need immediate access to yourobject data, Amazon S3 is a better choice.Amazon S3ImmediateaccessAmazon EFSAmazon DynamoDBAmazon EC2PerformanceAmazon Glacier is a low-cost storage service designed to store data that isinfrequently accessed and long-lived. Amazon Glacier retrieval jobs typicallycomplete in 3 to 5 hours.You can improve the upload experience for larger archives by using multipartupload for archives up to about 40 TB (the single archive limit).15 You can uploadseparate parts of a large archive independently, in any order and in parallel, toimprove the upload experience for larger archives. You can even perform rangeretrievals on archives stored in Amazon Glacier by specifying a range or portionPage 8
Amazon Web Services – AWS Storage Services Overviewof the archive.16 Specifying a range of bytes for a retrieval can help controlbandwidth costs, manage your data downloads, and retrieve a tar
In scenarios where you use Amazon S3 from within Amazon EC2 in the same Region, access to Amazon S3 from Amazon EC2 is designed to be fast. Amazon S3 is also designed so that server-side latencies are insignificant relative to Internet latencies. In addition, Amazon S3 is built to scale storage, requests, and numbers
4 AWS Training & Services AWS Essentials Training AWS Cloud Practitioner Essentials (CP-ESS) AWS Technical Essentials (AWSE) AWS Business Essentials (AWSBE) AWS Security Essentials (SEC-ESS) AWS System Architecture Training Architecting on AWS (AWSA) Advanced Architecting on AWS (AWSAA) Architecting on AWS - Accelerator (ARCH-AX) AWS Development Training
AWS SDK for JavaScript AWS SDK for JavaScript code examples AWS SDK for .NET AWS SDK for .NET code examples AWS SDK for PHP AWS SDK for PHP code examples AWS SDK for Python (Boto3) AWS SDK for Python (Boto3) code examples AWS SDK for Ruby AWS SDK for Ruby co
AWS Directory Amazon Aurora R5 instance Service AWS Server Migration Service AWS Snowball AWS Deep Amazon GameLift Learning AMIs AWS CodeBuild AWS CodeDeploy AWS Database Migration Service Amazon Polly 26 26 20 40 12 0 5 10 15 20 25 30 35 40 45 2018 Q1 2018 Q2 2018 Q3 2018 Q4 2019 Q1 New Services& Features on AWS
AWS instances with Nessus while in development and operations, before publishing to AWS users. Tenable Network Security offers two products on the AWS environment: Nessus for AWS is a Nessus Enterprise instance already available in the AWS Marketplace. Tenable Nessus for AWS provides pre-authorized scanning in the AWS cloud via AWS instance ID.
Amazon Web Services Cloud Platform The Cloud Computing Difference AWS Cloud Economics AWS Virtuous Cycle AWS Cloud Architecture Design Principles Why AWS for Big Data - Reasons Why AWS for Big Data - Challenges Databases in AWS Relational vs Non-Relational Databases Data Warehousing in AWS Services for Collecting, Processing, Storing, and .
the AWS Security Best Practices whitepaper and recommended reading on the AWS Security Learning webpage. 4. Amazon Web Services: Overview of Security Processes AWS Whitepaper AWS Compliance Program AWS Global Infrastructure Security AWS operates the global cloud infrastructure that you use to provision a variety of basic computing
Amazon Web Services How AWS Pricing Works Page 5 see which options fit your use case and budget best. For more information about AWS service pricing, see AWS Services Pricing. AWS offers couple of tools (free of cost) for you to use. If the workload details and services to be used are identified, AWS pricing calculator can help with calculating the
BSR/AWS B5.16-200x, Specification for the Qualification of Welding Engineers (revision of ANSI/AWS B5.16-2001) Obtain an electronic copy from: roneill@aws.org Order from: R. O’Neill, AWS; roneill@aws.org Send comments (with copy to BSR) to: Andrew Davis, AWS; adavis@aws.org; roneill@aws.org Single copy price: 25.00
