0

So I decided to use EFS as a shared filesystem for my EC2 instances. Our microservices, which are deployed on different instances, need to read and write to a shared space. One service basically puts the file to the known location, and the second service accesses it.

Now after reading the documentation and doing some tests myself I realized how latencies work and that files may already exist on disk but not be fully written yet. Are there any mechanisms to check when files are done replicating? My second service would need to know when the file isn't corrupt anymore.

codepushr
  • 111
  • 3
  • I can't find any documentation about this. Instead of reading a file as soon as it's available how about putting a message onto an SQS queue when the file is written, which the consumer can then access? – Tim Dec 04 '19 at 00:51
  • That's the issue. I know when I finished the write operation on machine A (through my code) but I don't know when the file is fully accessible and fully written on machine B. – codepushr Dec 04 '19 at 09:47
  • 1
    Are you actually experiencing a problem? EFS [claims to have close-to-open consistency](https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html#consistency) which (by my understanding) means that if `O_DIRECT` is used when opening for write, then the close operation should block until the data is consistent so that any open occurring after the close is complete will see the correct/intact file. – Michael - sqlbot Dec 04 '19 at 17:14
  • Yes, we work in pre-press and deal with very large PDFs (500mb is relatively small), so there's quite some delay. Would it be a feasible solution to create a checksum based on the file size when writing and then periodically checking for that checksum when reading? – codepushr Dec 05 '19 at 01:19

1 Answers1

0

I think you may want to call EFS API in order to retrieve creation status of the file.

The CreateFileSystem call returns while the file system's lifecycle state is still creating. You can check the file system creation status by calling the DescribeFileSystems operation, which among other things returns the file system state.

Be noted that calling API is throttled so you should handle it skillfully.

when files are done replicating

Also you only create a new file at a shared network storage and this should not understand as a replication.

Linh
  • 1