I've tried both lighttpd and nginx as webservers. All to very same result: In the morning, when the load is low, files are lighting fast to download. But later, when the load is a BIT more, server starts to serve files extremly slow if any.
We're using Riak on server, some mono applications which are constantly sending udp packets to user's application and a web server to execute php scripts and to host some files: lots of images, which are constantly access and modified by users and some application-specific files with average 20-30mbs, which are also accessed constantly. The load is not more then 5-10k per day for now.
- The CPU is always more then 90% idling (top, dstat), I guess he is doing fine.
- Memory is more then enough (htop, free), it's not even half used.
- Network speed = 1 Gbit/s, duplex = full, autonegotiation = off
HDD
Timing cached reads: 28842 MB in 2.00 seconds = 14436.45 MB/sec Timing buffered disk reads: 766 MB in 3.01 seconds = 254.78 MB/secUbuntu 14+ unicorn, ulimit -n 65536, somaxconn = 40000
I'm kind of in despair now) I though it was something with my configuration of lighttpd, but after moving to nginx, situation didn't change at all. I've tried aio with nginx, but, unfortunately, to no breakthrough. Where should I look at?
Lighttpd conf: http://www.pastebin.ca/2962652 Nginx conf: http://pastebin.ca/2962656
upd 01:
netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 459293618 0 860 0 795794415 0 0 0 BMRU
lo 65536 0 38105807 0 0 0 38105807 0 0 0 LRU
upd 02:
-iostat
avg-cpu: %user %nice %system %iowait %steal %idle
4.80 0.00 1.09 0.01 0.00 94.11
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 7.50 6.91 371.47 2938271 157867024
sdb 7.35 2.88 371.47 1224325 157867024
sdc 7.32 2.56 371.47 1089356 157867024
md2 1.15 0.13 227.20 56169 96553984
md1 11.91 10.36 140.27 4402931 59610720
-atop
PRC | sys 0.79s | user 4.37s | #proc 182 | #tslpu 0 | #zombie 0 | #exit ? |
CPU | sys 7% | user 41% | irq 3% | idle 749% | wait 0% | curscal ?% |
CPL | avg1 0.39 | avg5 0.34 | avg15 0.39 | csw 245907 | intr 63997 | numcpu 8 |
MEM | tot 31.3G | free 22.6G | cache 4.0G | dirty 1.4M | buff 295.8M | slab 184.0M |
SWP | tot 1.5G | free 1.5G | | | vmcom 3.9G | vmlim 17.2G |
MDD | md2 | busy 0% | read 0 | write 14 | MBw/s 0.01 | avio 0.00 ms |
MDD | md1 | busy 0% | read 0 | write 93 | MBw/s 0.07 | avio 0.00 ms |
DSK | sda | busy 0% | read 0 | write 41 | MBw/s 0.09 | avio 0.10 ms |
DSK | sdb | busy 0% | read 0 | write 41 | MBw/s 0.09 | avio 0.10 ms |
DSK | sdc | busy 0% | read 0 | write 41 | MBw/s 0.09 | avio 0.10 ms |
NET | transport | tcpi 17140 | tcpo 35894 | udpi 5175 | udpo 4868 | tcpao 2 |
NET | network | ipi 22311 | ipo 24687 | ipfrw 0 | deliv 22310 | icmpo 0 |
NET | eth0 4% | pcki 20325 | pcko 39045 | si 2061 Kbps | so 41 Mbps | erro 0 |
NET | lo ---- | pcki 1987 | pcko 1987 | si 298 Kbps | so 298 Kbps | erro 0 |