The problem
I'm manually inserting events into Zenoss through the zensendevent command line tool. The events are successfully inserted into Zenoss. The problem is that I can just insert about 8 events per second.
I have also experimented with inserting events through zensyslog. Although zensyslog receives the syslog events immediately (I can see them with tail -f $ZENHOME/log/origsyslog.log), the events are then chewed and inserted into Zenoss asynchronously at about the same rate (8 per second).
I haven't done a great deal of performance tuning for Zenoss or MySQL besides incrementing some cache settings for Zope and MySQL itself (as seen here).
Is the throughput I'm getting normal? What can I do to dramatically increase the number of inserted events per second (I need to be able to insert at least 100 per second). From what I've seen in the zensendevent code, it seems to use Zenoss' XML-RPC interface to insert the events.
Our machine and software specs are the following:
- Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz
- 4GB RAM
- Ubuntu Server 10.04.2 LTS
- Zenoss 3.1.0 compiled from source
- This is a dedicated physical installation, not a virtual machine
Some context if it helps:
We want to use Zenoss to monitor our machines and the software they run. Our machines are often not directly accessible from the central server Zenoss is supposed to run on. However, we can get the data we want form any machine to the central server. Right now we generate about 1.5 million events per day which gives an average of about 17 events per second which have to be inserted into Zenoss.
I'm using the following bash script to test the events insertion performance through zensendevent (160 events take about 20 seconds to be inserted):
#!/bin/bash
for a in {1..160}
do
zensendevent -d 'somemachinename' -y 'some_event_key' -p 'some_component' -s Error -c /App "Example Test message"
done
Thank you very much in advance.
