0

Is there any problem in doing this with lots of data?

mysqldump ... | gzip | s3cmd put - s3://bucket/file.sql.gz

The MySQL dump is about 100GB in size. What happens if gzip or s3cmd can't process the dump fast enough? Will it overflow the pipe buffer?

John W. Nesk
  • 3
  • 1

2 Answers2

1

The writer will block waiting for the reader to become ready.

What you really should worry about is what happens if s3cmd fails. Then you have to start all over.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • would this construct be signifficantly more efficient than doing in steps? – John W. Nesk Feb 08 '17 at 22:21
  • The MySQL server will kill the client connection if it blocks longer than the value of the `net_write_timeout` system variable on the server, which defaults (iirc) to 30 seconds. – Michael - sqlbot Feb 08 '17 at 22:37
  • @Michael-sqlbot You could make a whole answer out of that. Though I suspect it won't happen in practice since gzip will just read whenever it can. If it does happen it'll most likely because s3cmd went out to lunch. – Michael Hampton Feb 08 '17 at 22:40
  • You'd be surprised. Well... maybe. :) Gzip, bzip2, xz, and their multi-core friends pigz, pbzip2, and pixz will *all* block their input once they have a full set of data chunks to crunch and hit 100% CPU, and they'll continue to block their input until they can flush their output. It's least noticeable with gzip because it churns through the input faster as a result of its inferior compression ratio. If they really did read whenever they could, they'd potentially demand infinite memory if they couldn't compress and flush fast enough to keep up with the input data rate. – Michael - sqlbot Feb 08 '17 at 23:44
0

I'm doing physical backup and upload to s3 bucket in a two separate steps with no issues. Data is about 250 GB and .gz file is about 80 GB

rnix
  • 1