Set CRUSHMAP for 3 way on a two node cluster

Question

I have an SSD storage system that contains two nodes with 6 SSD's drives. Not ideal, and so some point I will introduce another node. Right now I'm wanting 3 way replication though.

Under a default rule, this won't happen because we only have two nodes. So I thought I'd attempt to modify the crushmap and set the ruleset for the SSD storage nodes to place data across the two nodes and the third set can be on another OSD on the same node.

Being a novice and not entirely understanding how the choose firstn and chooseleaf firstn statements work I'm unsure if it'll be do what I intend.

Here is what I have so far:

rule ssd-all {
  ruleset 1
  type replicated
  min_size 1
  max_size 5
  step take ssd
  step choose firstn 0 type host
  step chooseleaf firstn 2 type osd
  step emit
}

Where ssd is a root type containing some hosts and those hosts containing multiple (6) OSD's.

Would that work? Somehow, I don't think it's right. I'd like a better understanding when to use choose and where to use chooseleaf. And a better understanding of the number after firstn. The documentation is lacking clear examples. Reading the CRUSH whitepaper made some sense, but psuedocode isn't that clear to me. Can someone help?

What is expected scenario if one node is down? What `min_size` are you going to use with pools under that rule? — poige, May 26 '19 at 18:07

score 0 · Accepted Answer · answered Jul 08 '15 at 21:13

It turns out it's OK.

rule ssd-all {
  ruleset 1
  type replicated

  # These lines mean ssd-all will be used when the replica 
  # count is between 1 & 5 inclusive
  min_size 1  
  max_size 5

  # Take the top level pool named 'ssd'
  step take ssd

  # Choose all host nodes.  In my case, there are only 2.
  step choose firstn 0 type host

  # Choose up to to 2 leaves of type osd.
  step chooseleaf firstn 2 type osd
  step emit
}

After runnning

   crushtool -t crushmap --test --show-statistics --show-mappings --rule 1 --min-x 1 --max-x 10 --num-rep 3

With various numbers of replicas for --num-rep, it seems that the conditions are met correctly. There are at least 3 replicas across the two hosts with up to 2 replicas on a single host on 2 different osd's.

Set CRUSHMAP for 3 way on a two node cluster

1 Answers1