2

We use MySQL 5.0.77 on CentOS 5.5 on VMWare:

Linux dev.ic.soschildrensvillages.org.uk 2.6.18-194.11.4.el5PAE #1 SMP Tue Sep 21 05:48:23 EDT 2010 i686 i686 i386 GNU/Linux

We have recently upgraded from 4GB RAM to 8GB. When we did this the time of our mysqldump overnight backup jumped from under 10 minutes to over 2 hours. It also caused unresponsiveness on our plone based web site due to database load. The dump is using the optimized mysqldump format and is spooled directly through a socket to another server.

Any ideas on what we could do to fix gratefully appreciated. Would a MySQL upgrade help? Anything we can do to MySQL config? Anything we can do to Linux config? Or do we have to add another server or go to 64-bit?

We ran a previous (non-virtual) server on 6GB PAE and didn't notice a similar issue. This was on same MySQL version, but Centos 4.4.

Server config file:

[mysqld]
port=3307
socket=/tmp/mysql_live.sock
wait_timeout=31536000
interactive_timeout=31536000
datadir=/var/mysql/live/data
user=mysql
max_connections = 200
max_allowed_packet = 64M
table_cache = 2048
binlog_cache_size = 128K
max_heap_table_size = 32M
sort_buffer_size = 2M
join_buffer_size = 2M
lower_case_table_names = 1

innodb_data_file_path = ibdata1:10M:autoextend
innodb_buffer_pool_size=1G
innodb_log_file_size=300M
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=1
innodb_file_per_table

[mysqldump]
# Do not buffer the whole result set in memory before writing it to
# file. Required for dumping very large tables
quick
max_allowed_packet = 64M

[mysqld_safe]
# Increase the amount of open files allowed per process. Warning: Make
# sure you have set the global system limit high enough! The high value
# is required for a large number of opened tables
open-files-limit = 8192

Server variables:

mysql> show variables;
+---------------------------------+------------------------------------------------------------------+
| Variable_name                   | Value                                                            |
+---------------------------------+------------------------------------------------------------------+
| auto_increment_increment        | 1                                                                |
| auto_increment_offset           | 1                                                                |
| automatic_sp_privileges         | ON                                                               |
| back_log                        | 50                                                               |
| basedir                         | /usr/local/mysql-5.0.77-linux-i686-glibc23/                      |
| binlog_cache_size               | 131072                                                           |
| bulk_insert_buffer_size         | 8388608                                                          |
| character_set_client            | latin1                                                           |
| character_set_connection        | latin1                                                           |
| character_set_database          | latin1                                                           |
| character_set_filesystem        | binary                                                           |
| character_set_results           | latin1                                                           |
| character_set_server            | latin1                                                           |
| character_set_system            | utf8                                                             |
| character_sets_dir              | /usr/local/mysql-5.0.77-linux-i686-glibc23/share/mysql/charsets/ |
| collation_connection            | latin1_swedish_ci                                                |
| collation_database              | latin1_swedish_ci                                                |
| collation_server                | latin1_swedish_ci                                                |
| completion_type                 | 0                                                                |
| concurrent_insert               | 1                                                                |
| connect_timeout                 | 10                                                               |
| datadir                         | /var/mysql/live/data/                                            |
| date_format                     | %Y-%m-%d                                                         |
| datetime_format                 | %Y-%m-%d %H:%i:%s                                                |
| default_week_format             | 0                                                                |
| delay_key_write                 | ON                                                               |
| delayed_insert_limit            | 100                                                              |
| delayed_insert_timeout          | 300                                                              |
| delayed_queue_size              | 1000                                                             |
| div_precision_increment         | 4                                                                |
| keep_files_on_create            | OFF                                                              |
| engine_condition_pushdown       | OFF                                                              |
| expire_logs_days                | 0                                                                |
| flush                           | OFF                                                              |
| flush_time                      | 0                                                                |
| ft_boolean_syntax               | + -><()~*:""&|                                                   |
| ft_max_word_len                 | 84                                                               |
| ft_min_word_len                 | 4                                                                |
| ft_query_expansion_limit        | 20                                                               |
| ft_stopword_file                | (built-in)                                                       |
| group_concat_max_len            | 1024                                                             |
| have_archive                    | YES                                                              |
| have_bdb                        | NO                                                               |
| have_blackhole_engine           | YES                                                              |
| have_compress                   | YES                                                              |
| have_crypt                      | YES                                                              |
| have_csv                        | YES                                                              |
| have_dynamic_loading            | YES                                                              |
| have_example_engine             | NO                                                               |
| have_federated_engine           | YES                                                              |
| have_geometry                   | YES                                                              |
| have_innodb                     | YES                                                              |
| have_isam                       | NO                                                               |
| have_merge_engine               | YES                                                              |
| have_ndbcluster                 | DISABLED                                                         |
| have_openssl                    | DISABLED                                                         |
| have_ssl                        | DISABLED                                                         |
| have_query_cache                | YES                                                              |
| have_raid                       | NO                                                               |
| have_rtree_keys                 | YES                                                              |
| have_symlink                    | YES                                                              |
| hostname                        | app.ic.soschildrensvillages.org.uk                               |
| init_connect                    |                                                                  |
| init_file                       |                                                                  |
| init_slave                      |                                                                  |
| innodb_additional_mem_pool_size | 1048576                                                          |
| innodb_autoextend_increment     | 8                                                                |
| innodb_buffer_pool_awe_mem_mb   | 0                                                                |
| innodb_buffer_pool_size         | 1073741824                                                       |
| innodb_checksums                | ON                                                               |
| innodb_commit_concurrency       | 0                                                                |
| innodb_concurrency_tickets      | 500                                                              |
| innodb_data_file_path           | ibdata1:10M:autoextend                                           |
| innodb_data_home_dir            |                                                                  |
| innodb_adaptive_hash_index      | ON                                                               |
| innodb_doublewrite              | ON                                                               |
| innodb_fast_shutdown            | 1                                                                |
| innodb_file_io_threads          | 4                                                                |
| innodb_file_per_table           | ON                                                               |
| innodb_flush_log_at_trx_commit  | 1                                                                |
| innodb_flush_method             |                                                                  |
| innodb_force_recovery           | 0                                                                |
| innodb_lock_wait_timeout        | 50                                                               |
| innodb_locks_unsafe_for_binlog  | OFF                                                              |
| innodb_log_arch_dir             |                                                                  |
| innodb_log_archive              | OFF                                                              |
| innodb_log_buffer_size          | 8388608                                                          |
| innodb_log_file_size            | 314572800                                                        |
| innodb_log_files_in_group       | 2                                                                |
| innodb_log_group_home_dir       | ./                                                               |
| innodb_max_dirty_pages_pct      | 90                                                               |
| innodb_max_purge_lag            | 0                                                                |
| innodb_mirrored_log_groups      | 1                                                                |
| innodb_open_files               | 300                                                              |
| innodb_rollback_on_timeout      | OFF                                                              |
| innodb_support_xa               | ON                                                               |
| innodb_sync_spin_loops          | 20                                                               |
| innodb_table_locks              | ON                                                               |
| innodb_thread_concurrency       | 8                                                                |
| innodb_thread_sleep_delay       | 10000                                                            |
| interactive_timeout             | 31536000                                                         |
| join_buffer_size                | 2097152                                                          |
| key_buffer_size                 | 8384512                                                          |
| key_cache_age_threshold         | 300                                                              |
| key_cache_block_size            | 1024                                                             |
| key_cache_division_limit        | 100                                                              |
| language                        | /usr/local/mysql-5.0.77-linux-i686-glibc23/share/mysql/english/  |
| large_files_support             | ON                                                               |
| large_page_size                 | 0                                                                |
| large_pages                     | OFF                                                              |
| lc_time_names                   | en_US                                                            |
| license                         | GPL                                                              |
| local_infile                    | ON                                                               |
| locked_in_memory                | OFF                                                              |
| log                             | OFF                                                              |
| log_bin                         | OFF                                                              |
| log_bin_trust_function_creators | OFF                                                              |
| log_error                       |                                                                  |
| log_queries_not_using_indexes   | OFF                                                              |
| log_slave_updates               | OFF                                                              |
| log_slow_queries                | OFF                                                              |
| log_warnings                    | 1                                                                |
| long_query_time                 | 10                                                               |
| low_priority_updates            | OFF                                                              |
| lower_case_file_system          | OFF                                                              |
| lower_case_table_names          | 1                                                                |
| max_allowed_packet              | 67108864                                                         |
| max_binlog_cache_size           | 4294963200                                                       |
| max_binlog_size                 | 1073741824                                                       |
| max_connect_errors              | 10                                                               |
| max_connections                 | 200                                                              |
| max_delayed_threads             | 20                                                               |
| max_error_count                 | 64                                                               |
| max_heap_table_size             | 33554432                                                         |
| max_insert_delayed_threads      | 20                                                               |
| max_join_size                   | 18446744073709551615                                             |
| max_length_for_sort_data        | 1024                                                             |
| max_prepared_stmt_count         | 16382                                                            |
| max_relay_log_size              | 0                                                                |
| max_seeks_for_key               | 4294967295                                                       |
| max_sort_length                 | 1024                                                             |
| max_sp_recursion_depth          | 0                                                                |
| max_tmp_tables                  | 32                                                               |
| max_user_connections            | 0                                                                |
| max_write_lock_count            | 4294967295                                                       |
| multi_range_count               | 256                                                              |
| myisam_data_pointer_size        | 6                                                                |
| myisam_max_sort_file_size       | 2146435072                                                       |
| myisam_recover_options          | OFF                                                              |
| myisam_repair_threads           | 1                                                                |
| myisam_sort_buffer_size         | 8388608                                                          |
| myisam_stats_method             | nulls_unequal                                                    |
| ndb_autoincrement_prefetch_sz   | 1                                                                |
| ndb_force_send                  | ON                                                               |
| ndb_use_exact_count             | ON                                                               |
| ndb_use_transactions            | ON                                                               |
| ndb_cache_check_time            | 0                                                                |
| ndb_connectstring               |                                                                  |
| net_buffer_length               | 16384                                                            |
| net_read_timeout                | 30                                                               |
| net_retry_count                 | 10                                                               |
| net_write_timeout               | 60                                                               |
| new                             | OFF                                                              |
| old_passwords                   | OFF                                                              |
| open_files_limit                | 8192                                                             |
| optimizer_prune_level           | 1                                                                |
| optimizer_search_depth          | 62                                                               |
| pid_file                        | /var/mysql/live/mysqld.pid                                       |
| plugin_dir                      |                                                                  |
| port                            | 3307                                                             |
| preload_buffer_size             | 32768                                                            |
| profiling                       | OFF                                                              |
| profiling_history_size          | 15                                                               |
| protocol_version                | 10                                                               |
| query_alloc_block_size          | 8192                                                             |
| query_cache_limit               | 1048576                                                          |
| query_cache_min_res_unit        | 4096                                                             |
| query_cache_size                | 0                                                                |
| query_cache_type                | ON                                                               |
| query_cache_wlock_invalidate    | OFF                                                              |
| query_prealloc_size             | 8192                                                             |
| range_alloc_block_size          | 4096                                                             |
| read_buffer_size                | 131072                                                           |
| read_only                       | OFF                                                              |
| read_rnd_buffer_size            | 262144                                                           |
| relay_log                       |                                                                  |
| relay_log_index                 |                                                                  |
| relay_log_info_file             | relay-log.info                                                   |
| relay_log_purge                 | ON                                                               |
| relay_log_space_limit           | 0                                                                |
| rpl_recovery_rank               | 0                                                                |
| secure_auth                     | OFF                                                              |
| secure_file_priv                |                                                                  |
| server_id                       | 0                                                                |
| skip_external_locking           | ON                                                               |
| skip_networking                 | OFF                                                              |
| skip_show_database              | OFF                                                              |
| slave_compressed_protocol       | OFF                                                              |
| slave_load_tmpdir               | /tmp/                                                            |
| slave_net_timeout               | 3600                                                             |
| slave_skip_errors               | OFF                                                              |
| slave_transaction_retries       | 10                                                               |
| slow_launch_time                | 2                                                                |
| socket                          | /tmp/mysql_live.sock                                             |
| sort_buffer_size                | 2097152                                                          |
| sql_big_selects                 | ON                                                               |
| sql_mode                        |                                                                  |
| sql_notes                       | ON                                                               |
| sql_warnings                    | OFF                                                              |
| ssl_ca                          |                                                                  |
| ssl_capath                      |                                                                  |
| ssl_cert                        |                                                                  |
| ssl_cipher                      |                                                                  |
| ssl_key                         |                                                                  |
| storage_engine                  | MyISAM                                                           |
| sync_binlog                     | 0                                                                |
| sync_frm                        | ON                                                               |
| system_time_zone                | GMT                                                              |
| table_cache                     | 2048                                                             |
| table_lock_wait_timeout         | 50                                                               |
| table_type                      | MyISAM                                                           |
| thread_cache_size               | 0                                                                |
| thread_stack                    | 196608                                                           |
| time_format                     | %H:%i:%s                                                         |
| time_zone                       | SYSTEM                                                           |
| timed_mutexes                   | OFF                                                              |
| tmp_table_size                  | 33554432                                                         |
| tmpdir                          | /tmp/                                                            |
| transaction_alloc_block_size    | 8192                                                             |
| transaction_prealloc_size       | 4096                                                             |
| tx_isolation                    | REPEATABLE-READ                                                  |
| updatable_views_with_limit      | YES                                                              |
| version                         | 5.0.77                                                           |
| version_comment                 | MySQL Community Server (GPL)                                     |
| version_compile_machine         | i686                                                             |
| version_compile_os              | pc-linux-gnu                                                     |
| wait_timeout                    | 31536000                                                         |
+---------------------------------+------------------------------------------------------------------+
237 rows in set (0.00 sec)
Duncan Harris
  • 216
  • 2
  • 6
  • 1
    Have you checked if the problem goes away when you run it with just 4Gb ram? (reboot with mem=4096M). You've got a PAE kernel - but have you got a PAE capable CPU (with nx bit? - cat /proc/cpuinfo) – symcbean Feb 21 '11 at 12:29
  • Yes, the problem goes away when running with 4GB. We have tried upgrade and downgrade cycle twice. CPU is PAE capable: # cat /proc/cpuinfo | fgrep pae flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss nx lm constant_tsc ida nonstop_tsc pni ssse3 cx16 lahf_lm [8] flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss nx lm constant_tsc ida nonstop_tsc pni ssse3 cx16 lahf_lm [8] – Duncan Harris Feb 22 '11 at 07:01

1 Answers1

1

This is pretty odd sounding. I'm not sure what might fix it, but there are a couple of things I'd recommend doing regardless of the problems you're having.

64bit OS

Mysql has a fair amount of internal 64bit math. All things being equal it always runs faster within a 64bit OS. Additionally if you're going to the trouble to give your server 8GB a 32bit Mysql will still only be able to address 3GB of that. You may still gain some performance by giving the OS more room to cache files, etc. but Mysql itself won't benefit much.

Don't use OS database packages

Especially if you're using a disto Mysql builds packages for. The latest from Mysql is 5.0.92. While I'm not aware of any fixes it's good practice to stay updated. Mysql moves fairly fast and there has been vast differences in speed and performance over the lifetime of major releases.

If I were in your place, I'd spin up a fresh 8GB VM and install Mysql 5.0.92. Don't forget to install the compat libs or CentOS will think you don't have any Mysql libs installed. I'd dump from the original server and import. Setup replication from the old to new and then test. Once it looks good switchover.

kashani
  • 3,922
  • 18
  • 18