We have been successfully using the load balancing system Piranha+Pulse for 15 months on our CentOS 6 system. We use it to load balance two web servers. Note that our web servers are also our load balancers, so we have two servers in the total setup.
Today, we upgraded from CentOS 6.4 to 6.5 and we updated all packages. We simply did both of this by executing yum update. Since the update, pulse won't start anymore, so the load balancer is down.
One of the packages that was updated was piranha, which was updated from piranha-0.8.5-19.el6.x86_64 to piranha-0.8.6-2.el6_4.1.x86_64. The issue might also have been caused by an updated dependency, who knows. A lot of updates were done.
At this moment, there are two options: either the issue should be fixed with the new package, or we should revert back to the old package. Any help on achieving this is greatly appreciated.
What I found out on Pulse
When pulse starts, it remains running for a few seconds, and then it crashes. service pulse status then shows pulse dead but pid file exists.
This is what /var/log/messages says when trying to start pulse:
Jan 8 13:12:25 XXX pulse[14028]: STARTING PULSE AS BACKUP
Jan 8 13:12:25 XXX pulse[14028]: Skipping death of unknown child 14029
Jan 8 13:12:26 XXX ntpd[9119]: Listen normally on 12 lo:1:0 xx.xx.xx.xx UDP 123
Jan 8 13:12:26 XXX ntpd[9119]: peers refreshed
Jan 8 13:12:31 XXX pulse[14028]: partner dead: activating lvs
Jan 8 13:12:31 XXX pulse[14028]: Error waiting for semaphore: Interrupted system call
Jan 8 13:12:32 XXX ntpd[9119]: Deleting interface #12 lo:1:0, xx.xx.xx.xx#123, interface stats: received=0, sent=0, dropped=0, active_time=6 secs
Jan 8 13:12:32 XXX ntpd[9119]: peers refreshed
I found out that someone has a similar issue: https://www.centos.org/forums/viewtopic.php?f=13&t=44198. I have a slightly different error message (Interrupted system call instead of Permission denied). The suggested answer does not work for me, unfortunately. I still get the same error.
What I tried with yum
yum history list piranha shows the following:
Loaded plugins: fastestmirror, security
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
19 | <yy> | 2014-01-08 11:39 | I, U | 184 EE
15 | <yy> | 2013-05-22 11:00 | I, O, U | 254 EE
11 | <yy> | 2012-10-18 16:37 | Install | 2
history list
I cannot post the full output of yum history info 19, because the content is too long for this post.
yum downgrade piranha fails:
Loaded plugins: fastestmirror, security
Setting up Downgrade Process
Loading mirror speeds from cached hostfile
* base: mirror.nl.leaseweb.net
* extras: centos.mirror.transip.nl
* updates: mirror.nl.leaseweb.net
Only Upgrade available on package: piranha-0.8.6-4.el6.x86_64
Nothing to do
I tried to revert the latest update with yum history undo 19 and yum history rollback 18. If I do this, I get the following errors:
Error: Depsolving loop limit reached.
Error: Package: yum-3.2.29-40.el6.centos.noarch (base)
Requires: python(abi) = 2.6
Removing: python-2.6.6-51.el6.x86_64 (@base)
python(abi) = 2.6
Error: Package: yum-3.2.29-40.el6.centos.noarch (base)
Requires: python-urlgrabber >= 3.9.0-8
Removing: python-urlgrabber-3.9.1-9.el6.noarch (@base)
python-urlgrabber = 3.9.1-9.el6
Error: Package: yum-3.2.29-40.el6.centos.noarch (base)
Requires: yum-metadata-parser >= 1.1.0
Removing: yum-metadata-parser-1.1.2-16.el6.x86_64 (@anaconda-CentOS-201111250358.x86_64/6.3)
yum-metadata-parser = 1.1.2-16.el6
Error: Package: yum-3.2.29-40.el6.centos.noarch (base)
Requires: python-iniparse
Removing: python-iniparse-0.3.1-2.1.el6.noarch (@anaconda-CentOS-201111250358.x86_64/6.3)
python-iniparse = 0.3.1-2.1.el6
Error: Package: yum-3.2.29-40.el6.centos.noarch (base)
Requires: pygpgme
Removing: pygpgme-0.1-18.20090824bzr68.el6.x86_64 (@anaconda-CentOS-201111250358.x86_64/6.3)
pygpgme = 0.1-18.20090824bzr68.el6
Error: Package: yum-3.2.29-40.el6.centos.noarch (base)
Requires: python >= 2.4
Removing: python-2.6.6-51.el6.x86_64 (@base)
python = 2.6.6-51.el6
Error: Package: yum-3.2.29-40.el6.centos.noarch (base)
Requires: rpm-python
Removing: rpm-python-4.8.0-37.el6.x86_64 (@base)
rpm-python = 4.8.0-37.el6
Error: Package: yum-3.2.29-40.el6.centos.noarch (base)
Requires: rpm >= 4.4.2
Removing: rpm-4.8.0-37.el6.x86_64 (@base)
rpm = 4.8.0-37.el6
Error: Package: yum-3.2.29-40.el6.centos.noarch (base)
Requires: /usr/bin/python
Removing: python-2.6.6-51.el6.x86_64 (@base)
Not found
Error: Package: yum-3.2.29-40.el6.centos.noarch (base)
Requires: python-sqlite
Removing: python-2.6.6-51.el6.x86_64 (@base)
python-sqlite = 2.3.2
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
About piranha, this script says (output pipe to grep piranha):
Updated piranha-0.8.6-2.el6_4.1.x86_64 @updates
--> Processing Dependency: initscripts for package: piranha-0.8.6-4.el6.x86_64
--> Processing Dependency: initscripts for package: piranha-0.8.6-4.el6.x86_64
---> Package piranha.x86_64 0:0.8.6-4.el6 will be erased
When I use --skip-broken as suggested, I still get an error, and piranha is still not downgraded. I get the following error:
Packages skipped because of dependency problems:
yum-3.2.29-40.el6.centos.noarch from base
Error: Trying to remove "yum", which is protected
You could try running: rpm -Va --nofiles --nodigest
Update
We've had contact with RedHat, and this is probably a bug. They have a possible bugfix, but RedHat cannot reproduce it so they are unable to test it at the moment. I have suggested to send us the possible fix, so we can try it. Haven't heard from them yet.
Currently, we are managing the loadbalancing ourselves through lvsd. This is not an optimal solution, but it will do for now.
