2

I have three Nginx servers + php7 + memcache + Cluster Galera + Mariadb + Monit.

Everything works and all three are connected.

But if I now want to edit the my.cnf file to play the utilized RAM or the number of connections.

I can not get that single server to connect to the other two servers that are still clustered.

When starting Mariadb:

sudo systemctl start mysql

Give this Error:

Job for mariadb.service failed because the control process 
exited with error code. See "systemctl status mariadb.service" 
and "journalctl -xe" for details.

my.cnf:

# MariaDB database server configuration file.
#
# You can copy this file to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port        = 3306
socket      = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket      = /var/run/mysqld/mysqld.sock
nice        = 0

[mysqld]
#
# * Basic Settings
#
user        = mysql
pid-file    = /var/run/mysqld/mysqld.pid
socket      = /var/run/mysqld/mysqld.sock
port        = 3306
basedir     = /usr
datadir     = /var/lib/mysql
tmpdir      = /tmp
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address        = 127.0.0.1
#
# * Fine Tuning
#
max_connections     = 100
connect_timeout     = 5
wait_timeout        = 600
max_allowed_packet  = 16M
thread_cache_size       = 128
sort_buffer_size    = 4M
bulk_insert_buffer_size = 16M
tmp_table_size      = 32M
max_heap_table_size = 32M
#
# * MyISAM
#
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched. On error, make copy and try a repair.
myisam_recover_options = BACKUP
key_buffer_size     = 128M
#open-files-limit   = 2000
table_open_cache    = 400
myisam_sort_buffer_size = 512M
concurrent_insert   = 2
read_buffer_size    = 2M
read_rnd_buffer_size    = 1M
#
# * Query Cache Configuration
#
# Cache only tiny result sets, so we can fit more in the query cache.
query_cache_limit       = 128K
query_cache_size        = 64M
# for more write intensive setups, set to DEMAND or OFF
#query_cache_type       = DEMAND
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# we do want to know about network errors and such
log_warnings        = 2
#
# Enable the slow query log to see queries with especially long duration
#slow_query_log[={0|1}]
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
#log_slow_rate_limit    = 1000
log_slow_verbosity  = query_plan

#log-queries-not-using-indexes
#log_slow_admin_statements
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id      = 1
#report_host        = master1
#auto_increment_increment = 2
#auto_increment_offset  = 1
log_bin         = /var/log/mysql/mariadb-bin
log_bin_index       = /var/log/mysql/mariadb-bin.index
# not fab for performance, but safer
#sync_binlog        = 1
expire_logs_days    = 10
max_binlog_size         = 100M
# slaves
#relay_log      = /var/log/mysql/relay-bin
#relay_log_index    = /var/log/mysql/relay-bin.index
#relay_log_info_file    = /var/log/mysql/relay-bin.info
#log_slave_updates
#read_only
#
# If applications support it, this stricter sql_mode prevents some
# mistakes like inserting invalid dates etc.
#sql_mode       = NO_ENGINE_SUBSTITUTION,TRADITIONAL
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
default_storage_engine  = InnoDB
# you can't just change log file size, requires special procedure
#innodb_log_file_size   = 50M
innodb_buffer_pool_size = 256M
innodb_log_buffer_size  = 8M
innodb_file_per_table   = 1
innodb_open_files   = 400
innodb_io_capacity  = 400
innodb_flush_method = O_DIRECT
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem

#
# * Galera-related settings
#
[galera]
# Mandatory settings
#wsrep_on=ON
#wsrep_provider=
#wsrep_cluster_address=
#binlog_format=row
#default_storage_engine=InnoDB
#innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
#bind-address=0.0.0.0
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0

[mysqldump]
quick
quote-names
max_allowed_packet  = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completion

[isamchk]
key_buffer      = 16M

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

/etc/mysql/conf.d/galera.cnf

[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0

# Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so

# Galera Cluster Configuration
wsrep_cluster_name="test_cluster"
wsrep_cluster_address="gcomm://111.111.111.11,111.111.1.12,111.111.1.13"

# Galera Synchronization Configuration
wsrep_sst_method=rsync

# Galera Node Configuration
wsrep_node_address="111.111.111.11"
wsrep_node_name="Cluster_Singapure_SGP1"

ERROR:

[....] Starting mysql (via systemctl): mysql.serviceJob for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details.
 failed!
root@ClusterLondonLON1:~# systemctl status mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: failed (Result: timeout) since Tue 2017-02-07 21:29:14 UTC; 30s ago
  Process: 2879 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 2875 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
  Process: 2134 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 5549 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || 
  Process: 5542 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 5538 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 2134 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 160.0K
      CPU: 456ms
   CGroup: /system.slice/mariadb.service

Feb 07 21:27:44 ClusterLondonLON1 systemd[1]: Starting MariaDB database server...
Feb 07 21:27:45 ClusterLondonLON1 mysqld[5549]: 2017-02-07 21:27:45 140543440165120 [Note] /usr/sbin/mysqld (mysqld 10.1.21-MariaDB-1~xenial) starting as process 5737 ...
Feb 07 21:29:14 ClusterLondonLON1 systemd[1]: mariadb.service: Start-pre operation timed out. Terminating.
Feb 07 21:29:14 ClusterLondonLON1 systemd[1]: Failed to start MariaDB database server.
Feb 07 21:29:14 ClusterLondonLON1 systemd[1]: mariadb.service: Unit entered failed state.
Feb 07 21:29:14 ClusterLondonLON1 systemd[1]: mariadb.service: Failed with result 'timeout'.

Error Mariadb:

● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: failed (Result: exit-code) since Thu 2017-02-09 23:54:56 UTC; 9s ago
  Process: 19840 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
  Process: 19537 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR ||
  Process: 19530 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 19526 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 19840 (code=exited, status=1/FAILURE)
   Status: "MariaDB server is down"

Feb 09 23:54:55 ClusterSingapureSGP1ApSoutheast1 mysqld[19840]: 2017-02-09 23:54:55 139691639642368 [Note] WSREP: Start replication
Feb 09 23:54:55 ClusterSingapureSGP1ApSoutheast1 mysqld[19840]: 2017-02-09 23:54:55 139691639642368 [Note] WSREP: 'wsrep-new-cluster' option used, bootstrapping the cluster
Feb 09 23:54:55 ClusterSingapureSGP1ApSoutheast1 mysqld[19840]: 2017-02-09 23:54:55 139691639642368 [Note] WSREP: Setting initial position to 195e675b-eb86-11e6-a890-4ba496b286e7:311911
Feb 09 23:54:55 ClusterSingapureSGP1ApSoutheast1 mysqld[19840]: 2017-02-09 23:54:55 139691639642368 [ERROR] WSREP: It may not be safe to bootstrap the cluster from this node. It was not the las
Feb 09 23:54:55 ClusterSingapureSGP1ApSoutheast1 mysqld[19840]: 2017-02-09 23:54:55 139691639642368 [ERROR] WSREP: wsrep::connect(gcomm://111.222.333.44,111.222.33.43,11.222.33.321) failed: 7
Feb 09 23:54:55 ClusterSingapureSGP1ApSoutheast1 mysqld[19840]: 2017-02-09 23:54:55 139691639642368 [ERROR] Aborting
Feb 09 23:54:56 ClusterSingapureSGP1ApSoutheast1 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Feb 09 23:54:56 ClusterSingapureSGP1ApSoutheast1 systemd[1]: Failed to start MariaDB database server.
Feb 09 23:54:56 ClusterSingapureSGP1ApSoutheast1 systemd[1]: mariadb.service: Unit entered failed state.
Feb 09 23:54:56 ClusterSingapureSGP1ApSoutheast1 systemd[1]: mariadb.service: Failed with result 'exit-code'.

Error Start cluster:

root@ClusterSingapureSGP1ApSoutheast1:~# mysqld --wsrep-new-cluster
2017-02-10  0:00:10 140210296817920 [Note] mysqld (mysqld 10.1.21-MariaDB-1~xenial) starting as process 21744 ...
2017-02-10  0:00:10 140210296817920 [Note] WSREP: Read nil XID from storage engines, skipping position init
2017-02-10  0:00:10 140210296817920 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2017-02-10  0:00:10 140210296817920 [Note] WSREP: wsrep_load(): Galera 25.3.19(r3667) by Codership Oy <info@codership.com> loaded successfully.
2017-02-10  0:00:10 140210296817920 [Note] WSREP: CRC-32C: using hardware acceleration.
2017-02-10  0:00:10 140210296817920 [Note] WSREP: Found saved state: 195e675b-eb86-11e6-a890-4ba496b286e7:-1, safe_to_bootsrap: 0
2017-02-10  0:00:10 140210296817920 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 188.166.206.22; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc
2017-02-10  0:00:10 140210296817920 [Note] WSREP: GCache history reset: old(195e675b-eb86-11e6-a890-4ba496b286e7:0) -> new(195e675b-eb86-11e6-a890-4ba496b286e7:-1)
2017-02-10  0:00:10 140210296817920 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2017-02-10  0:00:10 140210296817920 [Note] WSREP: wsrep_sst_grab()
2017-02-10  0:00:10 140210296817920 [Note] WSREP: Start replication
2017-02-10  0:00:10 140210296817920 [Note] WSREP: 'wsrep-new-cluster' option used, bootstrapping the cluster
2017-02-10  0:00:10 140210296817920 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2017-02-10  0:00:10 140210296817920 [ERROR] WSREP: It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates. To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1 .
2017-02-10  0:00:10 140210296817920 [ERROR] WSREP: wsrep::connect(gcomm://231.143.231.123,123.423.43.123,23.434.23.432) failed: 7
2017-02-10  0:00:10 140210296817920 [ERROR] Aborting
473183469
  • 1,350
  • 1
  • 12
  • 23
Tarrio
  • 41
  • 1
  • 1
  • 4

1 Answers1

0

Did you see "systemctl status mariadb.service" and "journalctl -xe" for details? If so, posting them would be prudent. Also, Galera should be managing the starting and stopping of your services, rather than you doing it manually. I find it likely that you may find a message related to that.

With that in mind, Galera will catch your service if it has gone down and start it for you. You can reasonably stop a service out of band of Galera, and Galera will start that service again as part of recovery. So in this case, issue only systemctl stop mariadb and Galera will handle the subsequent starting of that service, and initialize that node into the rest of the cluster during start (rather than sitting there looking stupid after you start a service out of band).

Spooler
  • 7,016
  • 16
  • 29
  • In my case if Mariadb falls by a peak of traffic. He falls down. Cluster Galera does not start the service for me. So I do not get any more reprized than starting it manually. For this function I have Monit that is responsible for notifying me of the problem if it is by CPU or by RAM. It also verifies that the database is working if it does not work, it only restarts it. The problem is that the Mariadb database can be stopped by the command: /etc/init.d/mysql stop But it can not be started by the command: /etc/init.d/mysql start – Tarrio Feb 07 '17 at 21:38
  • The mistake is that it takes a long time to start. But if not if I have no problem the database and cluster work perfectly. The only problem is if I have a punctual traffic peak or if I want to modify something in the database and I need to restart for the changes to be reflected. – Tarrio Feb 07 '17 at 21:40