0

Our Hadoop cluster disk is getting full. So I want to scale up disk asap, and one way I can think is to increase ebs volume on all three nodes Hadoop 2.6.0-cdh5.5.1 (2 data-node, 1 name-node).

So my doubt is,

Is there any better way to scale up disk?

and if yes what is that way?

Info: We have one EBS volume mounted on /DATA mount point

piyushmandovra
  • 101
  • 2
  • "Better" in what sense? There's only really one way to make a disk bigger. (Make the disk bigger.) – Michael - sqlbot Feb 07 '17 at 13:27
  • ^ The comment above is pretty much the case. In addition, you're likely using a filesystem that supports online expansion. – Spooler Feb 07 '17 at 15:08
  • @Michael-sqlbot saying "Better" I mean either s3 or efs. I mean what are the best practices to choose disk size for growing hadoop hdfs. Hope I am making sense. – piyushmandovra Feb 07 '17 at 16:06
  • Have you read the documentation? What have you tried yourself? I suspect you may need to terminate your cluster and create it with larger disks, if you're using EBS (from the FAQ) http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-storage.html https://aws.amazon.com/emr/faqs/#ebs_volumes – Tim Feb 07 '17 at 18:31
  • @Tim yes I have read the documentation that says u have to reboot not terminate and that's only for root ebs, but mine is externally attached ebs, so I looking for the solution that provides increasing size without rebooting the machine. http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html – piyushmandovra Feb 08 '17 at 04:11
  • guys, For now, I just solved my problem by scaling EBS volume though still looking for some suggestion for long terms solutions to scale up Hadoop storage, Is s3 in any way useful. For your Info My data is in JSON format Links that helped me in doing these http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html http://stackoverflow.com/questions/11014584/ec2-cant-resize-volume-after-increasing-size – piyushmandovra Feb 12 '17 at 18:52
  • Why do you not add extra nodes (horizontal scaling)? – 030 Feb 21 '17 at 22:43
  • `2 data-node`? What replication factor has been configured? – 030 Feb 21 '17 at 22:46

0 Answers0