0

I would like to create "infinite" directory in terms of storage in an EC2 instance. Is it possible to symlink directories to S3 bucket?

I did not find any solution in the Internet, maybe somebody knows?

user350413
  • 23
  • 1
  • 4

2 Answers2

4

Amazon S3 is not a file system. You cannot just mount it to your EC2 instance like a regular disk or EBS. It now follows that a symlink is not possible.

You can however use s3fs to mount your S3 bucket as a directory in your EC2 server.

I used this tutorial and I was able to successfully mount my S3 to my EC2 running on Ubuntu 14.04.

WARNING You are billed in S3 based on the number of request you make. I did not do any further research, but my S3 usage spiked when I did this. I believe that the connections maintained by the mounted S3 bucket was counted as a GET/PUT/LIST request.

Another Alternative is to use Amazon's Elastic File System. This is like S3 but a file system.

julio
  • 894
  • 1
  • 9
  • 13
  • s3fs had some performance issues for me, but it is only way how to achieve it. – Ondra Sniper Flidr Jun 01 '16 at 12:45
  • Some of the default settings of s3fs [need to be changed](http://serverfault.com/a/678908/153161), but when properly configured it does work pretty well... But it is important to remember that, as mentioned, S3 is not a filesystem. Also, you can get pretty close to "infinite" with [the new EBS Cold Storage volume class](https://aws.amazon.com/blogs/aws/amazon-ebs-update-new-cold-storage-and-throughput-options/). It has its own design limitations, but it it's a real block device where you can put a real filesystem, and $25/mo per provisioned terabyte isn't bad. – Michael - sqlbot Jun 02 '16 at 01:20
  • *I believe that the connections maintained by the mounted S3 bucket was counted as a GET/PUT/LIST request.* There are no connections maintained. S3 access is only/always via REST requests and s3fs uses libcurl. But for each `ls -l`, at least one `LIST` request is followed by a `HEAD` request *for each file* in the directory, because the S3 object metadata is where the unix owner, group, and file mode are stored. So, uses, this can add up. – Michael - sqlbot Jun 02 '16 at 01:25
1

What you're looking for is EFS(elastic file system), but that hasn't been released yet. What you could try is this: https://forums.aws.amazon.com/thread.jspa?messageID=475840 but I have never tried this myself.

MaartenS
  • 91
  • 4