We initially started our cluster using one datacenter, one rack, and the Simple replication strategy. We have keyspaces with several hundred gigabytes of data. When we changed the tokens in the cassandra.yaml on this cluster from 4 to 256, and restarted our hosts, the distribution changed to odd numbers, 5 on one server, 3 on another, etc...
We tried this again on a different cluster and we saw the distribution change to 256 on a couple of nodes, but then changed again to 18 on one, 58 on another, etc.. The system keyspace had a large number of transfers continuously, but never seemed to settle.
My question is: When changing your tokens from one virtual range distribution to another, what is the correct approach? The documentation does not clearly state how to do this. Do we need to launch a second datacenter and allow the region aware snitch to do it?
