1

How performant is a shared mounted EFS if there are multiple ec2 instances with multiple containers all writing to separate files in separate directories? Can write performance on one container be detrimentally affected by another container writing (on the same or different node).

Writes are normally extremely small (~kbs) and can burst to ms frequency.

ajoseps
  • 113
  • 5
  • If the answer below helped you solve your problem please upvote and accept it. That's the StackExchange way to thank the users for taking their time answering your questions. Thanks :) – MLu Mar 27 '20 at 02:29
  • @Mlu I appreciate the help. The problem is still open however. We've also pulled in AWS support and not used EFS but still can't resolve it. – ajoseps Mar 27 '20 at 23:02

1 Answers1

1

It depends of course :)

Amazon recommends scaling your workload across multiple EC2 instances for higher aggregate throughput. On the other hand writing heaps of small files has higher a much higher overhead than writing the same amount of data in one big blob. Check out Amazon EFS Performance Tips for more details.

Also note that the actual throughput of your EFS volume depends on the amount of data stored. The more you store the higher the throughput. If you want high throughput even with little data your can pay for Provisioned throughput.

Lastly the performance can be improved by NFS caching - you can try to enable fsc or fs-cache or some other NFS caching mechanism that will pool writes locally and burst-write them in bigger chunks.

In the end you will have to benchmark how your application / k8s cluster performs under real load.

Do you really need to store the files on NFS by the way? Is there a better way to architect your application? Perhaps you could send the data from your K8S pods over Kinesis or SQS to a "consolidation microservice" that can collect a few and store them in bigger chunks? Maybe to S3 and not to EFS even? Or do whatever processing needs to be done without even storing them in the first place?

Storing zillions of tiny files individually can hardly lead to a high performance. I would seriously look at other options first.

Hope that helps :)

MLu
  • 23,798
  • 5
  • 54
  • 81
  • Do you by chance know what the per write overhead is? Not sure if this matters but I’m not writing many small files but overwriting the same files very frequently. There are probably better ways of architecting this system but this question was for determining if a specific bug is caused by EFS’ characteristics (specifically if one container can adversely impact another when writing) or something else. If it helps, the writing application calls fflush, which I think flushes the user buffer but I’m not sure if it forces the writes to be sync instead of Async. – ajoseps Feb 21 '20 at 02:37
  • What "specific bug" do you experience? Data corruption? Anyway, calling `fflush` will definitely slow down things. Overwriting the same file multiple times per second *from the same k8s node* will probably slow down things too, even between pods as they share the same kernel and same NFS mount. Pods on different nodes shouldn't affect each other's performance. – MLu Feb 21 '20 at 02:42
  • Apparently you're only interested in the last version of the file - I would store that in Memcache or Redis and save it to a file from there from time to time in an independent process. Much easier than trying to tune NFS for a workload that it wasn't designed to support. – MLu Feb 21 '20 at 02:45
  • We have a message layer which persists sequence numbers to EFS. I’m seeing certain messages being logged out on the receiver over a minute after they have been sent. We’ve tcpdumped the sender side and confirmed the issue is not on the sender, leaving the receiver end. The two variables seem to be network performance on the receiver and/or EFS which we have seen degraded performance with multiple writes from the same container. The receiving application writes the next expected sequence number to EFS by doing a fopen, rewind, and fflush. It writes after processing every message. – ajoseps Feb 21 '20 at 02:48
  • EFS is a *file storage service*, not a persistence layer for a high frequency counter. You've got lots of caches, network latencies, and consistency between nodes in play here. Simply DON'T do this. Or don't be surprised that you've got issues. This is NOT a workload suitable for EFS. – MLu Feb 21 '20 at 03:18
  • Yes we only care about the most recent version of the file and memcache or redis are probably better fit, but in terms of the bug in question I need to narrow it down to whether or not EFS would cause such a problem. I’m not certain if the containers that get backed up are on the same node or not. That’s something I need to double check. – ajoseps Feb 21 '20 at 03:19
  • What you describe is a job for a *queue* or *message bus* of some sort. Why don't you send the next expected number to a **SNS Topic** from Sender to Receiver? Or send the whole message from Sender to Receiver through **SQS Queue** or **Kinesis stream**? Much easier and much more reliable. – MLu Feb 21 '20 at 03:20
  • It definitely is a job for something like that. The application is quickfix if you are familiar with it. It’s a third party dependency that we’ve forked, so we can modify it but we try to avoid it. It has options to persist the seqnums to a file or DB out of the box. Like I said, there are better options for how to architect this but I just want to determine if the issue we are saying can be caused by EFS. Additionally our sender isn’t living on AWS. – ajoseps Feb 21 '20 at 03:25
  • Look, the "bug in question" is caused by using inappropriate tools for the job. Even if quickfix supports seqnum persistence in files they certainly didn't mean to use network filesystem for it. You say it supports DB - use DB then, it's made for concurrent access. **AWS RDS** is your friend. If you still insist on using EFS for this I'm afraid you'll have to open AWS support request to help you with your debugging. It's only $100/month and you can cancel after a month. Much cheaper than spending hours of someones time on it. – MLu Feb 21 '20 at 03:30
  • I agree it’s probably not the right tool for the job, but I need to prove that is indeed the issue before changing a fairly core part of the architecture. I just want to make sure I’m going in the right direction in terms of debugging this bug. Maybe this is a job for AWS debugging. Another option is to use EBS and have it track each respective container, e.g. mount and unmount EBS on the correct node as the respective container comes up and down. – ajoseps Feb 21 '20 at 03:33
  • 1
    Adding more complexity won't help you. Show this thread to whomever tasked you with debugging the problem. Hopefully they will realise that the solution is in **using the right tools for the job**, not in debugging and tweaking and working around the wrong tools. That's a waste of time for everyone involved. Good luck. – MLu Feb 21 '20 at 03:39