We have several servers running VMWare ESXi 4 on which are virtual machines running OpenSUSE and Windows 2008r2. We also have a server monitoring these vm's using OpenNMS. For several of the OpenSUSE vm's we get regular High Threshold messages for the percentage of discarded packets.
Inspecting the graphs showed that it are only incoming packages that are being discarded. Peaks are up to 10% discards. It doesn't happen on all Linux VM's. The ones it doesn't happen on are running a slightly older OpenSUSE 11.3 while the others are 11.4 or 12.1 however it is also true that the ones that suffer from this problem have very little load while the heaviest loaded VM has zero discarded packets for the last year.
The windows 2008 servers don't seem to have this issue at all.
So what is a likely cause for this? At what percentage should I start to worry?
Edit: I was wondering if snmp or opennms is showing incorrect data. When I look at the output of netstat -s I see: 0 incoming packets discarded
Here is the complete output (uptime is 137 days):
Ip:
89907150 total packets received
5047 with invalid addresses
0 forwarded
19748 with unknown protocol
0 incoming packets discarded
89882355 incoming packets delivered
93223508 requests sent out
133 dropped because of missing route Icmp:
4121 ICMP messages received
37 input ICMP message failed.
ICMP input histogram:
destination unreachable: 162
echo requests: 3950
timestamp request: 9
24159 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 20200
echo replies: 3950
timestamp replies: 9 IcmpMsg:
InType3: 162
InType8: 3950
InType13: 9
OutType0: 3950
OutType3: 20200
OutType14: 9 Tcp:
309309 active connections openings
175665 passive connection openings
582 failed connection attempts
19123 connection resets received
27 connections established
89397500 segments received
98178773 segments send out
39585 segments retransmited
7 bad segments received.
256696 resets sent Udp:
448836 packets received
524 packets to unknown port received.
0 packet receive errors
448854 packets sent
RcvbufErrors: 0
SndbufErrors: 0 UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0 TcpExt:
1897 invalid SYN cookies received
441 resets received for embryonic SYN_RECV sockets
ArpFilter: 0
125024 TCP sockets finished time wait in fast timer
123 packets rejects in established connections because of timestamp
2365891 delayed acks sent
5221 delayed acks further delayed because of locked socket
Quick ack mode was activated 1517 times
76762 packets directly queued to recvmsg prequeue.
32782 packets directly received from backlog
148216 packets directly received from prequeue
81827955 packets header predicted
11453 packets header predicted and directly queued to user
TCPPureAcks: 2002864
TCPHPAcks: 81030906
TCPRenoRecovery: 3
TCPSackRecovery: 5098
TCPSACKReneging: 46
TCPFACKReorder: 31
TCPSACKReorder: 17
TCPRenoReorder: 0
TCPTSReorder: 25
TCPFullUndo: 31
TCPPartialUndo: 599
TCPDSACKUndo: 1712
TCPLossUndo: 691
TCPLoss: 5407
TCPLostRetransmit: 201
TCPRenoFailures: 0
TCPSackFailures: 286
TCPLossFailures: 101
TCPFastRetrans: 11936
TCPForwardRetrans: 493
TCPSlowStartRetrans: 3237
TCPTimeouts: 15424
TCPRenoRecoveryFail: 0
TCPSackRecoveryFail: 165
TCPSchedulerFailed: 0
TCPRcvCollapsed: 0
TCPDSACKOldSent: 1569
TCPDSACKOfoSent: 5
TCPDSACKRecv: 7543
TCPDSACKOfoRecv: 6
TCPAbortOnSyn: 0
TCPAbortOnData: 249150
TCPAbortOnClose: 1391
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 160
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0
TCPSACKDiscard: 15
TCPDSACKIgnoredOld: 2257
TCPDSACKIgnoredNoUndo: 739
TCPSpuriousRTOs: 129
TCPMD5NotFound: 0
TCPMD5Unexpected: 0
TCPSackShifted: 38802
TCPSackMerged: 24929
TCPSackShiftFallback: 16027
TCPBacklogDrop: 0
TCPMinTTLDrop: 0
TCPDeferAcceptDrop: 87963
IPReversePathFilter: 0
TCPTimeWaitOverflow: 0 IpExt:
InNoRoutes: 0
InTruncatedPkts: 0
InMcastPkts: 2
OutMcastPkts: 0
InBcastPkts: 31370
OutBcastPkts: 0
InOctets: 17194554126
OutOctets: 21418612641
InMcastOctets: 72
OutMcastOctets: 0
InBcastOctets: 3492814
OutBcastOctets: 0
