0

We're trying to upload 150TB into S3 using Commvault. However we're only able to upload about half a terrabyte a day with a 1.5Gbs dedicated line. We've also enabled the following recommendations:

Is there anything else we can do to increase the ingestion rate into S3?

Tim
  • 30,383
  • 6
  • 47
  • 77
Juraj Lišiak
  • 76
  • 7

2 Answers2

1

So after opening up a ticket with Commvault and several hours of troubleshooting we found out the issue was hot spots on the on-premise SAN and the slow SATA disks from where the backups are stored and send over to S3. Once that was fixed, the internet pipe became the bottleneck.

Bottom line, we were able to scale out from 500GB a day to 10TB+ a day. S3 was able to scale out fairly linearly as more streams were added from the Commvault end.

Juraj Lišiak
  • 76
  • 7
0

Have a read of this article. Latency can make a significant difference to S3 upload performance. What is your ping time to S3? What is your average object size? How many threads are you using?

S3 transfer acceleration uses CloudFront for uploads, which might make it a bit faster. I've read reports that it doesn't always give much benefit.

You could upload to an S3 region closer to you, then have the data replicated over. That will probably cost more.

You can also send a physical disk, using AWS import export.

Tim
  • 30,383
  • 6
  • 47
  • 77
  • So yes we tried CloudFront acceleration and it looks like it will go 7x faster - however the cost will be an extra $10k just for the acceleration to be turned on for that amount. The article doesn't really help much as everything is controlled by the Commvault client/server - so I'm wondering what setting to configure in Commvault. – Juraj Lišiak Nov 27 '19 at 22:13
  • 1
    Best ask Commvault customer support. – Tim Nov 27 '19 at 23:22
  • Yeah I've opened up a ticket with them but was hoping someone had some real world experience... – Juraj Lišiak Nov 27 '19 at 23:24