I'm provisioning a Galera MySQL cluster under Vagrant using a multi machine VagrantFile.
I don't believe the issue lies with Vagrant
Vagrant Version
VagrantFile
Vagrant.configure(2) do |config|
config.vm.box = "ubuntu/trusty64"
config.vm.provider "virtualbox" do |vb|
vb.memory = "2048"
end
config.ssh.forward_agent = true
if Vagrant.has_plugin?("vagrant-cachier")
config.cache.scope = :box
config.cache.enable :apt
end
config.vm.define "core0" do |core0|
core0.vm.network "private_network", ip: "192.168.50.3"
core0.vm.hostname = "core0"
core0.vm.provision :hosts, :sync_hosts => true
core0.vm.provision "shell", inline: <<-SHELL
sudo python /vagrant/bootstrap.pex --core-nodes core0 core1 core2 --node-zero
SHELL
end
config.vm.define "core1" do |core1|
core1.vm.network "private_network", ip: "192.168.50.4"
core1.vm.hostname = "core1"
core1.vm.provision :hosts, :sync_hosts => true
core1.vm.provision "shell", inline: <<-SHELL
sudo python /vagrant/bootstrap.pex --master core0 --core
SHELL
end
config.vm.define "core2" do |core2|
core2.vm.network "private_network", ip: "192.168.50.5"
core2.vm.hostname = "core2"
core2.vm.provision :hosts, :sync_hosts => true
core2.vm.provision "shell", inline: <<-SHELL
sudo python /vagrant/bootstrap.pex --master core0 --core
SHELL
end
end
Vagrant Plugins
I'm using the vagrant-cachier and vagrant-hosts in vagrant.
Vagrant runs and creates each VM in turn and then I do a 2 stage provision to ensure the networking between the boxes is established before attempting clustering.
vagrant up --provision-with hosts && vagrant provision --provision-with shell
the shell provisioner uses salt to creates and installs mariadb and gluster
Mariadb versions
vagrant@core0:~$ sudo apt-cache policy mariadb-server-core-10.1
mariadb-server-core-10.1:
Installed: 10.1.18+maria-1~trusty
Candidate: 10.1.18+maria-1~trusty
vagrant@core0:~$ sudo apt-cache policy galera-3
galera-3:
Installed: 25.3.18-trusty
Candidate: 25.3.18-trusty
I configure the galera.cnf with the cluster addresses as
wsrep_cluster_address = gcomm://core2,core0,core1
when hosts core1 and core2 attempt to join with core0 they are unable to.
core1 joining cluster
core1 is able to find the core0 and retrieve the current cluster state.
Oct 12 15:15:02 core1 mysqld: 2016-10-12 15:15:02 140403237877696 [Note] WSREP: gcomm: connecting to group 'TestSystem', peer 'core2:,core0:,core1:'
Oct 12 15:15:02 core1 mysqld: 2016-10-12 15:15:02 140403237877696 [Note] WSREP: (a61950db, 'tcp://0.0.0.0:4567') connection established to a61950db tcp://127.0.0.1:4567
Oct 12 15:15:02 core1 mysqld: 2016-10-12 15:15:02 140403237877696 [Note] WSREP: (a61950db, 'tcp://0.0.0.0:4567') connection established to a61950db tcp://127.0.1.1:4567
Oct 12 15:15:02 core1 mysqld: 2016-10-12 15:15:02 140403237877696 [Warning] WSREP: (a61950db, 'tcp://0.0.0.0:4567') address 'tcp://127.0.1.1:4567' points to own listening address, blacklisting
Oct 12 15:15:02 core1 mysqld: 2016-10-12 15:15:02 140403237877696 [Note] WSREP: (a61950db, 'tcp://0.0.0.0:4567') connection established to a5301480 tcp://192.168.50.3:4567
Oct 12 15:15:02 core1 mysqld: 2016-10-12 15:15:02 140403237877696 [Note] WSREP: (a61950db, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
Oct 12 15:15:03 core1 mysqld: 2016-10-12 15:15:03 140403237877696 [Note] WSREP: declaring a5301480 at tcp://192.168.50.3:4567 stable
Oct 12 15:15:03 core1 mysqld: 2016-10-12 15:15:03 140403237877696 [Note] WSREP: Node a5301480 state prim
core2 not available
as expected at this time core2 is not available
Oct 12 15:15:03 core1 mysqld: 2016-10-12 15:15:03 140403237877696 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.50.5:4567
Oct 12 15:15:03 core1 mysqld: 2016-10-12 15:15:03 140403237877696 [Note] WSREP: gcomm: connected
SST failure
core1 attempts to connect to core0 using the address 10.0.2.15 which is the Vagrant NAT address
Oct 12 15:15:03 core1 mysqld: 2016-10-12 15:15:03 140403237563136 [Note] WSREP: State transfer required:
Oct 12 15:15:03 core1 mysqld: #011Group state: a530f9fd-908d-11e6-a72a-b2e3a6b91029:1113
Oct 12 15:15:03 core1 mysqld: #011Local state: 00000000-0000-0000-0000-000000000000:-1
Oct 12 15:15:03 core1 mysqld: 2016-10-12 15:15:03 140403237563136 [Note] WSREP: New cluster view: global state: a530f9fd-908d-11e6-a72a-b2e3a6b91029:1113, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 3
Oct 12 15:15:03 core1 mysqld: 2016-10-12 15:15:03 140403237563136 [Warning] WSREP: Gap in state sequence. Need state transfer.
Oct 12 15:15:03 core1 mysqld: 2016-10-12 15:15:03 140402002753280 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.0.2.15' --datadir '/var/lib/mysql/' --parent '9043' --binlog '/var/log/mariadb_bin/mariadb-bin' '
Oct 12 15:15:03 core1 mysqld: WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20161012 15:15:03.985)
Oct 12 15:15:03 core1 -wsrep-sst-joiner: Streaming with xbstream
Oct 12 15:15:03 core1 -wsrep-sst-joiner: Using socat as streamer
Oct 12 15:15:04 core1 -wsrep-sst-joiner: Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} )
Oct 12 15:15:04 core1 mysqld: 2016-10-12 15:15:04 140403237563136 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.0.2.15:4444/xtrabackup_sst//1
Oct 12 15:15:04 core1 mysqld: 2016-10-12 15:15:04 140403237563136 [Note] WSREP: REPL Protocols: 7 (3, 2)
Oct 12 15:15:04 core1 mysqld: 2016-10-12 15:15:04 140402075592448 [Note] WSREP: Service thread queue flushed.
Oct 12 15:15:04 core1 mysqld: 2016-10-12 15:15:04 140403237563136 [Note] WSREP: Assign initial position for certification: 1113, protocol version: 3
Oct 12 15:15:04 core1 mysqld: 2016-10-12 15:15:04 140402075592448 [Note] WSREP: Service thread queue flushed.
Oct 12 15:15:04 core1 mysqld: 2016-10-12 15:15:04 140403237563136 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (a530f9fd-908d-11e6-a72a-b2e3a6b91029): 1 (Operation not permitted)
Oct 12 15:15:04 core1 mysqld: #011 at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
Oct 12 15:15:04 core1 mysqld: 2016-10-12 15:15:04 140402019526400 [Note] WSREP: Member 1.0 (core1) requested state transfer from '*any*'. Selected 0.0 (core0)(SYNCED) as donor.
Oct 12 15:15:04 core1 mysqld: 2016-10-12 15:15:04 140402019526400 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1113)
Oct 12 15:15:04 core1 mysqld: 2016-10-12 15:15:04 140403237563136 [Note] WSREP: Requesting state transfer: success, donor: 0
Oct 12 15:15:04 core1 mysqld: 2016-10-12 15:15:04 140402019526400 [Warning] WSREP: 0.0 (core0): State transfer to 1.0 (core1) failed: -32 (Broken pipe)
Oct 12 15:15:04 core1 mysqld: 2016-10-12 15:15:04 140402019526400 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():736: Will never receive state. Need to abort.
wsrep status on core0
logging into mysql on core0 and running
SHOW GLOBAL STATUS LIKE 'wsrep_%'
+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
...
| wsrep_cluster_state_uuid | a530f9fd-908d-11e6-a72a-b2e3a6b91029 |
| wsrep_cluster_status | Primary |
| wsrep_gcomm_uuid | a5301480-908d-11e6-a84e-0b2444c3985f |
| wsrep_incoming_addresses | 10.0.2.15:3306 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_local_state_uuid | a530f9fd-908d-11e6-a72a-b2e3a6b91029 |
...
+------------------------------+--------------------------------------+
So it appears that core0 is advertising its wsrep incoming address as 10.0.2.15:3306, which isn't the address I'm expecting - 192.168.0.3:3306.
ifconfig on core0
this shows the NAT on eth0
vagrant@core0:~$ ifconfig
eth0 Link encap:Ethernet HWaddr 08:00:27:de:04:89
inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fede:489/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:218886 errors:0 dropped:0 overruns:0 frame:0
TX packets:81596 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:205966097 (205.9 MB) TX bytes:6015101 (6.0 MB)
eth1 Link encap:Ethernet HWaddr 08:00:27:bc:f7:ee
inet addr:192.168.50.3 Bcast:192.168.50.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:febc:f7ee/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:261637 errors:0 dropped:0 overruns:0 frame:0
TX packets:244284 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:59467905 (59.4 MB) TX bytes:114065906 (114.0 MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:246320 errors:0 dropped:0 overruns:0 frame:0
TX packets:246320 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:64552545 (64.5 MB) TX bytes:64552545 (64.5 MB)
How / Why does the address get set to this? Is there a way I can make the address the correct address?
updates
how wsrep_incoming_addresses is set
While wsrep_cluster_address has to be specified at the node start, wsrep_incoming_addresses is internally determined during the initialization. On linux operation systems, the command that is used to determine the IP address picks the first available global IP address from the list of interfaces.
ip addr show | grep '^\s*inet' | grep -m1 global | awk '
{print $2 }
' | sed 's/\/.*//'
https://mariadb.atlassian.net/browse/MDEV-5487
my output
vagrant@core0:~$ ip addr show | grep '^\s*inet' | grep -m1 global | awk '
> {print $2 }
> ' | sed 's/\/.*//'
10.0.2.15
