Amazon S3

Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services (AWS) that provides object storage through a web service interface.[1][2] Amazon S3 uses the same scalable storage infrastructure that Amazon.com uses to run its global e-commerce network.[3] Amazon S3 can be employed to store any type of object which allows for uses like storage for Internet applications, backup and recovery, disaster recovery, data archives, data lakes for analytics, and hybrid cloud storage. AWS launched Amazon S3 in the United States on March 14, 2006,[1][4] then in Europe in November 2007.[5]

Amazon S3
Type of site
Cloud storage
Available inEnglish
OwnerAmazon.com
URLaws.amazon.com/s3/
IPv6 supportYes
CommercialYes
RegistrationRequired (included in free tier layer)
LaunchedMarch 14, 2006 (2006-03-14)
Current statusActive

Design

Although Amazon Web Services (AWS) does not publicly provide the details of S3's technical design, Amazon S3 manages data with an object storage architecture[6] which aims to provide scalability, high availability, and low latency with 99.999999999% durability and between 99.95% to 99.99% availability (though there is no service-level agreement for durability).[3]

The basic storage units of Amazon S3 are objects which are organized into buckets. Each object is identified by a unique, user-assigned key.[7] Buckets can be managed using either the console provided by Amazon S3, programmatically using the AWS SDK, or with the Amazon S3 REST application programming interface (API). Objects can be managed using the AWS SDK or with the Amazon S3 REST API and can be up to five terabytes in size with two kilobytes of metadata.[8][9] Additionally, objects can be downloaded using the HTTP GET interface and the BitTorrent protocol.

Requests are authorized using an access control list associated with each object bucket and support versioning which is disabled by default.[10] Note that since buckets are typically the size of an entire file system mount in other systems, this access control scheme is very coarse-grained, i.e. you cannot have unique access controls associated with individual files. Bucket names and keys are chosen so that objects are addressable using HTTP URLs:

  • http://s3.amazonaws.com/bucket/key (for a bucket created in the US East (N. Virginia) region)
  • https://s3.amazonaws.com/bucket/key
  • http://s3-region.amazonaws.com/bucket/key
  • https://s3-region.amazonaws.com/bucket/key
  • http://s3.region.amazonaws.com/bucket/key
  • https://s3.region.amazonaws.com/bucket/key
  • http://s3.dualstack.region.amazonaws.com/bucket/key (for requests using IPv4 or IPv6)
  • https://s3.dualstack.region.amazonaws.com/bucket/key
  • http://bucket.s3.amazonaws.com/key
  • http://bucket.s3-region.amazonaws.com/key
  • http://bucket.s3.region.amazonaws.com/key
  • http://bucket.s3.dualstack.region.amazonaws.com/key (for requests using IPv4 or IPv6)
  • http://bucket.s3-website-region.amazonaws.com/key (if static website hosting is enabled on the bucket)
  • http://bucket.s3-website.region.amazonaws.com/key (if static website hosting is enabled on the bucket)
  • http://bucket.s3-accelerated.amazonaws.com/key (where the filetransfer exits Amazons network at the last possible moment so as to give the fastest possible transfer speed and lowest latency)
  • http://bucket.s3-accelerated.dualstack.amazonaws.com/key
  • http://bucket/key (where bucket is a DNS CNAME record pointing to bucket.s3.amazonaws.com)

Amazon S3 can be used to replace significant existing (static) web-hosting infrastructure with HTTP client accessible objects.[11] The Amazon AWS authentication mechanism allows the bucket owner to create an authenticated URL which is valid for a specified amount of time.

Every item in a bucket can also be served as a BitTorrent feed. The Amazon S3 store can act as a seed host for a torrent and any BitTorrent client can retrieve the file. This can drastically reduce the bandwidth cost for the download of popular objects. While the use of BitTorrent does reduce bandwidth, AWS does not provide native bandwidth limiting and, as such, users have no access to automated cost control. This can lead to users on the free-tier of Amazon S3, or small hobby users, amassing dramatic bills. AWS representatives have stated that a bandwidth limiting feature was on the design table from 2006 to 2010,[12] but in 2011 the feature is no longer in development.[13]

A bucket can be configured to save HTTP log information to a sibling bucket; this can be used in data mining operations.[14]

There are various User Mode File System (FUSE)-based file systems for Unix-like operating systems (Linux, etc.) that can be used to mount an S3 bucket as a file system such as S3QL. The semantics of the Amazon S3 file system are not that of a POSIX file system, so the file system may not behave entirely as expected.[15]

Hosting websites

Amazon S3 provides the option to host static HTML websites with index document support and error document support.[16] Websites hosted on S3 may designate a default page to display and another page to display in the event of a partially invalid URL, such as a 404 error, which provide useful content to visitors of a URL containing a CNAME record hostname rather than a direct Amazon S3 bucket reference when the URL does not contain a valid S3 object key, such as when a casual user initially visits a URL that is a bare non-Amazon hostname.

Amazon S3 logs

Amazon S3 allows users to enable or disable logging. If enabled, the logs are stored in Amazon S3 buckets which can then be analyzed. These logs contain useful information such as:

Logs can be analyzed and managed using third-party tools like S3Stat, Cloudlytics, Qloudstat, AWStats, and Splunk.

Amazon S3 tools

Amazon S3 provides an API for developers.[17] The AWS console provides tools for managing and uploading files but it is not capable of managing large buckets or editing files.[18] Third-party websites like S3edit.com or software like Cloudberry Explorer, ForkLift and WebDrive have the capability to edit files on Amazon S3.[19]

Amazon S3 storage classes

Amazon S3 offers four different storage classes that offer different levels of durability, availability, and performance requirements.[20]

  • Amazon S3 Standard is the default class.
  • Amazon S3 Standard Infrequent Access (IA) is designed for less frequently accessed data. Typical use cases are backup and disaster recovery solutions.
  • Amazon S3 One Zone-Infrequent Access is designed for data that is not often needed but when required, needs to be accessed rapidly. Data is stored in one zone and if that zone is destroyed, all data is lost.
  • Amazon Glacier is designed for long-term storage of data that is infrequently accessed and where retrieval latency of minutes or hours is acceptable. "Glacier Deep Archive" is an alternative with a retrieval time of at least 12 hours, but 1/4th the price. It is intended as an alternative to magnetic tape libraries, and is designed for long term retention of data for 7 to 10 years.

Notable users
  • Photo hosting service SmugMug has used Amazon S3 since April 2006. They experienced a number of initial outages and slowdowns, but after one year they described it as being "considerably more reliable than our own internal storage" and claimed to have saved almost $1 million in storage costs.[21]
  • Netflix uses Amazon S3 as their system of record. Netflix implemented a tool, S3mper,[22] to address the Amazon S3 limitations of eventual consistency.[23] S3mper stores the filesystem metadata: filenames, directory structure, and permissions in Amazon DynamoDB.[24]
  • reddit is hosted on Amazon S3.[25]
  • Bitcasa,[26] and Tahoe-LAFS-on-S3,[27] among others, use Amazon S3 for online backup and synchronization services. In 2016, Dropbox stopped using Amazon S3 services and developed its own cloud server.[28][29]
  • Mojang hosts Minecraft game updates and player skins on Amazon S3.[30]
  • Tumblr, Formspring, and Pinterest host images on Amazon S3.
  • Swiftype's CEO has mentioned that the company uses Amazon S3.[31]
  • Amazon S3 was used by some enterprises as a long term archiving solution until Amazon Glacier was released in August 2012.
  • The API has become a popular method to store objects.[32] As a result, many applications have been built to natively support the Amazon S3 API[33] which includes applications that write data to Amazon S3 and Amazon S3-compatible object stores:[34]
TypeCompany NameProduct
Client BackupHaystack Software LLCArq backup[35]
Client BackupCloudBerry LabCloudBerry Backup[36]
Client Backupopen-sourceDuplicati[37]
Client BackupNovosoft LLCHandy Backup[38]
File BrowserBinarynightsForkLift
File Browserodriveodrive[39]
MySQL BackupOracleMySQL Enterprise Backup
Oracle Database BackupOracleOracle Secure Backup Cloud Manager[40]
Server BackupActifioActifio[41]
Server Backup Atempo Digital Archive (ADA)[42]
Server BackupCommvaultCommvault[43]
Server Backup Veritas NetBackup[44]
Server Backup Asigra Asigra Cloud Backup[45]
Server Backup Rubrik Rubrik[46]
Cloud Storage Wasabi Wasabi Hot Storage
Cloud Storage Gateway CTERA Networks C00 Series[47]
Cloud Storage Gateway Avere FXT Series[48]
Cloud Storage Gateway EMC CloudArray[49]
Cloud Storage Gateway Microsoft StorSimple[50]
Cloud Storage Gateway Nasuni NF Series[51]
Cloud Storage Gateway NetApp Altavault[52]
Cloud Storage Gateway Panzura Global File System[53]
Sync & Share Storage Made Easy SME
Hybrid Storage Cloudian Cloudian HyperStore[54]
Hybrid Storage SeaweedFS Seaweed File System
Hybrid Storage NooBaa NooBaa Storage
Hybrid Storage Western Digital ActiveScale
Hybrid Storage SwiftStack SwiftStack 1space
On-Premises Storage MinIO MinIO[55]
On-Premises Storage OpenIO OpenIO SDS[56]
On-Premises Storage Pure Storage FlashBlade
On-Premises Storage Scality RING Storage[57]
Open Source Zenko.io Open Source S3 Server[58]

S3 API and competing services

The broad adoption of Amazon S3 and related tooling has given rise to competing services based on the S3 API. These services use the standard programming interface; however, they are differentiated by their underlying technologies and supporting business models.[59] A cloud storage standard (like electrical and networking standards) enables competing service providers to design their services and clients using different parts in different ways yet still communicate and provide the following benefits:[60]

  1. Increase competition by providing a set of rules and a level playing field, encouraging market entry by smaller companies which might otherwise be precluded.
  2. Encourage innovation by cloud storage & tool vendors, & developers because they can focus on improving their own products and services instead of focusing on compatibility.
  3. Allow economies of scale in implementation (i.e., if a service provider encounters an outage or as clients outgrow their tools and need faster operating systems or tools, they can easily swap out solutions).
  4. Provide timely solutions for delivering functionality in response to demands of the marketplace (i.e., as business growth in new locations increases demand, clients can easily change or add service providers simply by subscribing to the new service).

Examples of competing Amazon S3-compliant storage implementations:

History
At AWS Summit 2013 NYC, CTO Werner Vogels announces 2 trillion objects stored in S3.

Amazon Web Services introduced AmazonS3 in 2006.[66][67]

Amazon S3 is reported to store more than 2 trillion objects as of April 2013.[68] This is up from 10 billion objects as of October 2007,[69] 14 billion objects in January 2008, 29 billion objects in October 2008,[70] 52 billion objects in March 2009,[71] 64 billion objects in August 2009,[72] and 102 billion objects in March 2010.[73]. In November 2017 AWS added default encryption capabilities at bucket level. [74]

gollark: They're a great technology.
gollark: Incorrect. We've been through this.
gollark: <@160279332454006795>
gollark: None can escape I guess.
gollark: It's meant to be. I don't know what happened.

See also

References

Citations
  1. "Amazon Web Services Launches "Amazon S3"" (Press release). Amazon.com. 2006-03-14. Archived from the original on 2018-11-15. Retrieved 2018-11-14.
  2. Huang, Dijiang; Wu, Huijun (2017-09-08). Mobile Cloud Computing: Foundations and Service Models. Morgan Kaufmann. p. 67. ISBN 9780128096444. Archived from the original on 2018-11-15. Retrieved 2018-11-15.
  3. "Cloud Object Storage - Store & Retrieve Data Anywhere - Amazon Simple Storage Service". Amazon Web Services, Inc. Archived from the original on 2018-05-17. Retrieved 2018-05-17.
  4. "5 Key Events in the history of Cloud Computing - DZone Cloud". dzone.com. Archived from the original on 2018-09-29. Retrieved 2018-09-28.
  5. "Amazon Web Services Offers European Storage for Amazon S3" (Press release). Amazon.com. 2007-11-06. Archived from the original on 2018-11-15. Retrieved 2018-11-14.
  6. "What is Cloud Object Storage? – AWS". Amazon Web Services, Inc. 2019-10-16. Archived from the original on 2018-09-20. Retrieved 2018-07-09.
  7. "Tech Blog » Starting Websphere in Cloud and saving the data in S3". techblog.aasisvinayak.com. Archived from the original on 2010-03-12.
  8. "open-guides/og-aws". GitHub. Archived from the original on 2018-01-03. Retrieved 2018-05-17.
  9. "Error Responses - Amazon Simple Storage Service". docs.aws.amazon.com. Archived from the original on 2017-12-24. Retrieved 2018-05-21.
  10. "Introduction to Amazon S3 - Amazon Simple Storage Service". docs.aws.amazon.com. Archived from the original on 2018-05-12. Retrieved 2018-05-17.
  11. "How to use Amazon S3 for Web Hosting". bucketexplorer.com. Archived from the original on 2008-04-08. Retrieved 2008-05-06.
  12. AWS Developer Forums: Limit my own bandwidth? Archived 2012-01-18 at the Wayback Machine. Forums.aws.amazon.com. Retrieved on 2013-08-09.
  13. AWS Developer Forums: What is the status on the bill capping Archived 2012-01-18 at the Wayback Machine. Forums.aws.amazon.com. Retrieved on 2013-08-09.
  14. http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html Archived 2014-12-23 at the Wayback Machine Server Access Logging
  15. "Comparison of S3QL and other S3 file systems". Archived from the original on 2012-08-05. Retrieved 2012-06-29.
  16. Amazon Simple Storage Service Archived 2011-02-20 at the Wayback Machine. Docs.amazonwebservices.com. Retrieved on 2013-08-09.
  17. "Amazon Simple Storage Service (S3) Documentation". Amazon Web Services, Inc. Archived from the original on 2011-08-29. Retrieved 2015-03-23.
  18. "AWS Management Console". Amazon Web Services, Inc. Archived from the original on 2018-03-14. Retrieved 2018-03-06.
  19. "s3Edit - Online S3 File Editor, IDE on Browser". www.s3edit.com. Archived from the original on 2018-07-16. Retrieved 2018-11-22.
  20. "Cloud Storage Classes – Amazon Simple Storage Service (S3) – AWS". Amazon Web Services, Inc. Archived from the original on 2018-06-13. Retrieved 2018-05-17.
  21. "Amazon S3: Show Me the Money". SmugMug Blog. SmugMug. November 10, 2006. Archived from the original on 2017-03-03. Retrieved 2017-03-03.
  22. "S3mper: Consistency in the Cloud". Archived from the original on 2016-04-24. Retrieved 2016-05-01.
  23. "Introduction to Amazon S3". Amazon. Archived from the original on 2017-12-25. Retrieved 28 December 2017.
  24. Hern, Alex (2017-02-02). "Amazon Web Services: the secret to the online retailer's future success". the Guardian. Archived from the original on 2018-05-02. Retrieved 2018-04-23.
  25. "AWS Case Study: reddit". aws.amazon.com. 2015. Archived from the original on 2015-03-17. Retrieved March 18, 2015.
  26. "What is Tahoe-LAFS-on-S3?". August 21, 2012. Archived from the original on 2013-05-06. Retrieved 2012-08-21.
  27. "The Epic Story of Dropbox's Exodus From the Amazon Cloud Empire". WIRED. Archived from the original on 2018-01-25. Retrieved 2018-04-23.
  28. "Dropbox saved almost $75 million over two years by building its own tech infrastructure". GeekWire. 2018-02-23. Archived from the original on 2018-04-23. Retrieved 2018-04-23.
  29. "Minecraft Beta 1.2_02". January 21, 2010. Archived from the original on 2011-01-23. Retrieved 2011-01-22.
  30. "Swiftype Explains Their Cloud Stack". July 1, 2013. Archived from the original on 2014-12-08. Retrieved 2014-12-08.
  31. Lelii, Sonia (23 September 2013). "Amazon S3 API for cloud storage leads pack, for now". TechTarget.com. Archived from the original on 2016-08-03. Retrieved 31 May 2016.
  32. Evans, Chris (12 January 2016). "Has S3 Become the De-Facto API Standard?". Architecting.it. Archived from the original on 2016-06-11. Retrieved 31 May 2016.
  33. Leopold, George (July 11, 2017). "Scality Targets Multi-Cloud Data Storage". Datanami news portal. Archived from the original on 2017-10-03.
  34. Sadun, Erica (6 November 2012). "Arq cloud backup adds low-cost Amazon Glacier support". www.engadget.com. Archived from the original on 2016-03-31. Retrieved 31 May 2016.
  35. Moran, Joe (1 December 2015). "Data Backup Software Review: CloudBerry Lab Backup 4.5". www.smallbusinesscomputing.com. Archived from the original on 2016-06-08. Retrieved 31 May 2016.
  36. Sanders, James (4 August 2014). "Securely back up personal files with Duplicati: Q&A with the open source client's creators". www.TechRepublic.com. Archived from the original on 2016-06-04. Retrieved 31 May 2016.
  37. Handy, Backup (17 April 2017). "Amazon S3 Backup Software for Cloud Backup". www.handybackup.net. Archived from the original on 2017-09-12. Retrieved 12 September 2017.
  38. Lohnash, Mike (19 June 2015). "Odrive Review: One Folder for All Your Clouds". www.BackupReview.com. Archived from the original on 2016-05-02. Retrieved 31 May 2016.
  39. "Oracle Database Backup To Cloud: Amazon Simple Storage Service (S3)" (PDF). Oracle.com. Archived (PDF) from the original on 2016-03-01. Retrieved 31 May 2016.
  40. "Actifio on AWS Marketplace". Archived from the original on 2015-09-06. Retrieved 2018-06-27. "Backup & Replication Support". Actifio.com. Archived from the original on 2016-08-22. Retrieved 31 May 2016.
  41. "Atempo Digital Archive - for large scale NAS backup, migration and synchronization". Atempo Blog. Archived from the original on 2018-06-15. Retrieved 2018-03-27.
  42. "Cloud Storage Support". Commvault.com. Archived from the original on 18 August 2016. Retrieved 31 May 2016.
  43. "Veritas launches NetBackup 7.7 with emphasis on cloud backup". SearchDataBackup. Archived from the original on 2016-06-07. Retrieved 2016-05-31.
  44. "Asigra, Veeam remain top users' choice for backup applications". SearchDataBackup. Archived from the original on 2016-05-20. Retrieved 2016-05-31.
  45. "Startup Rubrik Aiming to Erase Backup, Recovery Software". www.eweek.com. Retrieved 2016-05-31.
  46. Ibm; Emc; Netapp; Seagate; Hp; Hill, Seagate rolls storage kit for manufacturers down Dot. "CTERA Networks offers up in-cloud server backup". Spectralogic CTO talks up hybrid flash-tape cartridge. Welcome tape robot overlords and backup, CTERA Networks offers up in-cloud server. Archived from the original on 2016-05-06. Retrieved 2016-05-31.
  47. Mellor, Chris (7 October 2015). "Like a wedding cake: Avere unveils three-tier AWS cloud NAS". TheRegister.com. Archived from the original on 2016-05-03. Retrieved 8 June 2016.
  48. Armstrong, Adam (28 January 2015). "EMC CloudArray 5.0 Launched". StorageReview.com. Archived from the original on 2016-06-02. Retrieved 8 June 2016.
  49. Mackie, Kurt (1 June 2015). "Microsoft StorSimple Extends Cloud Support to AWS, OpenStack". RedmondChannelPartner.com. Archived from the original on 2016-07-01. Retrieved 8 June 2016.
  50. Mellor, Chris (14 May 2015). "Azure gives AWS the blues again in Nasuni cloud storage poll". TheRegister.com. Archived from the original on 2016-06-01. Retrieved 8 June 2016.
  51. Ramel, David (28 May 2015). "NetApp Introduces AltaVault for Cloud Backup". AWSInsider.net. Archived from the original on 2016-08-07. Retrieved 8 June 2016.
  52. Knuth, Gabe (26 May 2015). "Panzura explains their Global File System and how they can help you deploy XenDesktop from AWS". Archived from the original on 2016-06-25. Retrieved 7 June 2016 – via BrianMadden.com.
  53. "Cloudian deploys Amazon S3-compatible on-premises object storage, sold and metered in AWS Marketplace | Cloudian". Cloudian. Archived from the original on 2016-11-22. Retrieved 2016-11-21.
  54. Coughlin, Tom (14 February 2017). "S3 Without Amazon And Intelligent Scale-Out NAS". Forbes.com. Forbes. Retrieved 12 June 2020.
  55. "OpenIO Data Access Features - S3 compatibility". OpenIO. Archived from the original on 2018-11-16. Retrieved 2018-11-16.
  56. "Scality RING S3 Connector - Scality". Scality. Archived from the original on 2017-09-02. Retrieved 2017-11-06.
  57. Armstrong, Adam (11 July 2017). "Scality Releases Zenko A Multi-Cloud Controller". Storage Review. Archived from the original on 2017-10-03.
  58. Watters, Audrey. "Cloud Community Debates, Is Amazon S3's API the Standard? (And Should It Be?)". SAY Media, Inc. Archived from the original on 2013-02-17. Retrieved 19 December 2012.
  59. Committee on Standards Workshop Planning, Board on Telecommunications and Computer Applications, Commission on Engineering and Technical Systems, National Research Council (1990). Crossroads of Information Technology Standards. Washington, DC: The National Academies Press, 1990. pp. 36–37. Archived from the original on 2014-03-25. Retrieved 2014-03-25.CS1 maint: uses authors parameter (link)
  60. Ceph Object Gateway S3 API — Ceph Documentation Archived 2012-10-15 at the Wayback Machine. Ceph.com. Retrieved on 2013-08-09.
  61. "Connectria Cloud Storage - Amazon S3® Compatible Cloud Storage Service". Connectria. Connectria. Retrieved 19 December 2012.
  62. "Connectria Hosting Launches Cloud Storage Solution". Connectria Hosting. Archived from the original on 2012-11-02. Retrieved 19 December 2012.
  63. DigitalOcean Spaces API - DigitalOcean Documentation Archived 2017-09-30 at the Wayback Machine. developers.DigitalOcean.com. Retrieved on 2017-10-30.
  64. Riak CS Archived 2013-01-28 at the Wayback Machine. Basho (2013-01-18). Retrieved on 2013-08-09.
  65. Overview of Amazon Web Services, 2018, https://d1.awsstatic.com/whitepapers/aws-overview.pdf Archived 2017-11-18 at the Wayback Machine
  66. Garfinkel, Simson L. 2007. An Evaluation of Amazon's Grid Computing Services: EC2, S3, and SQS. Harvard Computer Science Group Technical Report TR-08-07. https://dash.harvard.edu/bitstream/handle/1/24829568/tr-08-07.pdf?sequence=1 Archived 2018-07-29 at the Wayback Machine
  67. "Amazon S3 – Two Trillion Objects, 1.1 Million Requests / Second - Amazon Web Services". typepad.com. 18 April 2013.
  68. Vogels, Werner (2008-03-19). "Happy Birthday, Amazon S3!". All Things Distributed. Archived from the original on 2008-05-09. Retrieved 2008-05-23.
  69. "Amazon S3 - Busier Than Ever". Amazon.com. 2008-10-08. Archived from the original on 2008-10-11. Retrieved 2008-10-09.
  70. "Celebrating S3's Third Birthday With Special Anniversary Pricing - Amazon Web Services". typepad.com. 31 March 2009. Archived from the original on 2011-07-07. Retrieved 2009-04-01.
  71. "Amazon's Head Start in the Cloud Pays Off". eweek.com.
  72. "Amazon S3 Now Hosts 100 Billion Objects". datacenterknowledge.com. 9 March 2010. Archived from the original on 2010-03-12. Retrieved 2010-03-09.
  73. https://aws.amazon.com/blogs/aws/new-amazon-s3-encryption-security-features/

Sources
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.