0
I am using data on a s3 bucket for deep learning purposes. I mounted the s3 bucket onto ec2 instance using How to Mount S3 bucket on EC2 Linux Instance - Cloudkul.
When I cd into my local directory on the EC2 instance, I could see the folders that are on my S3 bucket as well. However, I could not see them when I open WinSCP and I'm not sure if it is supposed to happen or not. I input the correct directory for training on EC2 and opened up a terminal window using tmux. But nothing happens at all, not even error messages after I run the training program on python.
I ran that locally on my own computer and it works, so the only reason I can think of is that the S3 bucket was not mounted properly or the way I use the data is not allowed for that S3 bucket?
Ping
Posted 2019-07-15T17:08:02.363
Reputation: 1
Mounting an S3 bucket using S3FS isn't completely standard. I guess either the fuse file system isn't visible to WinSCP for some reason, or you have file permissions incorrect. What is the user id / group of the SSH user, and what are the permissions on the file? Please update your question to include that information. – Tim – 2019-07-15T22:07:35.850
I don't need ssh to access the files. I can edit any directory on my ec2 instance and the change shows up in s3 bucket as well. Same for the other way around. During my training, I'm mainly accessing the files from ec2 instance(mounted directory). I am on root user. Does this help? – Ping – 2019-07-16T02:26:35.733
My understanding is WinSCP connects to sshd, the SSD daemon. sshd then tries to access the files you want to access. You have to work out the user that sshd runs as, and the permissions of the files you want it to access. My knowledge of linux is only ok, so don't take this as fact. – Tim – 2019-07-16T02:53:19.440
@Tim I think I found out the issue. It was taking a long time to load the files stored in s3. But it was able to access s3 through ec2. Not sure why though. Wish there was a technical support for aws that's free of charge. – Ping – 2019-07-18T17:39:41.593
Please answer your question to help others in future. AWS support is excellent, but it takes a really highly qualified and experienced engineer at least an hour for any decent question, sometimes many hours, so even what they charge might not always cover the actual costs if you don't have many resources in your accounts. Since it's based on your AWS spend it can get quite expensive. – Tim – 2019-07-18T20:11:12.777