2

I had 3 nodes, all with ~550GB, and added 4 more. I hoped that data would migrate on its own from old servers to new ones. Unfortunately after one day, data seems to stay in old places.

In parallel, there is a constant stream of incoming data, and about 150GB more per node was added. So old nodes have currently 550+150 = ~700GB, new ones ~150GB.

Should I just wait? Is there a command or option that I should run to make them rebalance? Or it is a problem with replication 1 or inserting data while rebalancing?

--  Address   Load       Tokens  Owns   Host ID                               Rack
UN  x.x.x.x   702.48 GB  256     13.1%  3e9e7e79-c727-4a3f-86ad-0dbfb77a4540  rack1
UN  x.x.x.x   692.47 GB  256     15.5%  ac41b3df-301e-470a-b531-12f2aeba7328  rack1
UN  x.x.x.x   660.65 GB  256     14.2%  8b1bb70d-57fe-460a-b756-4ac82b24acb6  rack1
UN  x.x.x.x   156.7 GB   256     15.1%  d499acd3-8eae-4a4c-bba3-cd8e45bd1ce3  rack1
UN  x.x.x.x   140.99 GB  256     13.6%  f266de96-8547-4e68-8d31-50a13e6c6628  rack1
UN  x.x.x.x   138.44 GB  256     13.5%  cbf53e0d-398d-4fc7-befd-b83d173a4e67  rack1
UN  x.x.x.x   155.17 GB  256     15.0%  dddbcb68-9730-456a-8429-de06bcf2f2bb  rack1

I'm using Cassandra 2.09. Murmur 3 partitioner, 256 vnodes per node, replication 1 (no copy).

Jacek Kaniuk
  • 194
  • 1
  • 11

1 Answers1

2

You need to run a cleanup on the old nodes, and perhaps a repair -- it won't move the data around of its own accord. See the documentation here for adding new nodes to a cluster.

Mike Scott
  • 7,903
  • 29
  • 26
  • Thanks. `By default, the repair command takes a snapshot of each replica` - does that mean that I need double the space for repair? – Jacek Kaniuk Oct 03 '14 at 11:19
  • 1
    Repair will use more space, but not double the space, as it works on one replica at a time. So it will use additional space equal to the size of your largest replica. – Mike Scott Oct 03 '14 at 11:20
  • What about -par options for parallel repair? Where are intermediate data stored in such scenario? Documentation is very vague. – Jacek Kaniuk Oct 03 '14 at 11:22
  • Sorry, I've not used the -par option, so I can't comment on that. But having looked at the documentation, I wouldn't recommend using that option on a live database. – Mike Scott Oct 03 '14 at 11:39
  • I've stopped inserting and tried to run repair. But nodetool repair returns almost immediately on each node and do not change anything: `Starting repair command #6, repairing 256 ranges for keyspace xxx` – Jacek Kaniuk Oct 03 '14 at 14:47
  • Have you tried running cleanup on the old nodes? – Mike Scott Oct 03 '14 at 15:16
  • I had to add bigger disks to those first 3 nodes, but eventually I could run cleanup on all nodes. It reduced size on all nodes by 10-30% but apart from that, nothing improved. Data is still unbalanced. I undestand that as: all data previously on first 3 nodes that currently belongs to their token ranges are still there, but data that should be moved to new 4 nodes is now lost. – Jacek Kaniuk Oct 08 '14 at 08:34
  • so, it was lost? I've experienced the same issue, did you found a solution? Thanks in advance. – folex Sep 17 '15 at 12:33