4

I have a production server that is showing a very large number of forks when running vmstat -f. Any suggestions on steps that could be used to help find out what the origin of the forks are?

vmstat -f 1
      6650796 forks

EDIT:

[~]$ ./forks.sh 
Forks in last 2 seconds: 20 
Forks in last 2 seconds: 40 
Forks in last 2 seconds: 58 
Forks in last 2 seconds: 9 
Forks in last 2 seconds: 6 
Forks in last 2 seconds: 28 
Forks in last 2 seconds: 8 
Forks in last 2 seconds: 10 
Forks in last 2 seconds: 15 
Forks in last 2 seconds: 9
Taylor Leese
  • 179
  • 3
  • 10
  • What applications are you running? Apache in prefork mode? Oracle? Some applications are designed to use forking pretty heavily in normal utilization. – Tim Howland May 19 '09 at 19:24
  • The server is running a Tomcat insance which I wouldn't expect to fork. – Taylor Leese May 19 '09 at 19:31
  • My question is more along the lines of what steps can I take to investigate? Are there any other tools that would help? – Taylor Leese May 19 '09 at 19:33
  • 1
    Taylor, what has led you to believe that a system with a high fork count is indicative of a problem? I have never seen that as a metric of server health, and no monitoring tool or suite I've used uses it as a data point. I really think you're looking at something that's purely informational as if it's an error. – Luke May 19 '09 at 20:24
  • 1
    I agree with you. I'm trying to convince other folks on the team of the same. – Taylor Leese May 19 '09 at 22:34
  • Ah gotcha. That's never a fun situation :-) – Luke May 19 '09 at 23:22

5 Answers5

5

According to the man page, it includes all calls to fork, vfork or clone. The last one of these three (clone) is used by Java to implement its threads

So each time your Java server creates a new thread, that value increments.

Providing it doesn't go silly, it should be fine. How many per second do you see on average?

MarkR
  • 2,898
  • 16
  • 13
  • I added some stats to the question. – Taylor Leese May 19 '09 at 22:39
  • 1
    I'd just like to add to this answer - tomcat forks frequently under normal operation. To handle new requests, tomcat will fork a new process, handle the request, then close... some processes can last less than a couple seconds. You can see observe of this behavior with "pstree -pa" or by running top, viewing only the tomcat user, and enabling threads. – gharper May 20 '09 at 00:26
  • I would expect Tomcat to use a pool of threads, rather than creating one per connection. However, it may change the size of this pool from time to time, which will involve spawning new threads. – MarkR May 20 '09 at 10:49
  • Tomcat doesn't "fork" though. It maintains a pool of threads for various things under the one process. Forking is a specific os call resulting in a copy of the process in a new child process. – Matt Aug 17 '12 at 13:39
1

Any process that spawns another process without itself terminating is a fork - for example, every command that's executed at a shell will be counted as a fork. An very high number of fork calls since the system booted is totally normal.

Luke
  • 628
  • 1
  • 7
  • 14
1

The first thing to note is that running vmstat with out its two time arguments show the accumulated value since last reboot. You'd have to run it multiple times to get a "forks per second" number to see if it's really a big number or not. Something like this (which could obviously be made into a much friendlier script):

g3 0 /home/jj33 ># while true
> do
>   vmstat -f
>   sleep 15
> done
       278039 forks
       278044 forks
       278047 forks
       278051 forks

So, that system did 5, 3, and 4 forks in 3 15 second intervals, which, given that every process call on a *nix box involves a fork, doesn't seem like a big number.

jj33
  • 11,038
  • 1
  • 36
  • 50
0

A high fork count really isn't a problem - I've been running a Gentoo-based router for several months now, and my fork count is over twice yours, but the machine itself is rock-solid.

dijkstra ~ # vmstat -f 1
     14623947 forks
dijkstra ~ # uptime
 15:29:26 up 291 days, 14:02,  1 user,  load average: 0.02, 0.04, 0.07
Tim
  • 1,148
  • 1
  • 14
  • 23
0

If you suspect a certain process (like JVM) to be the reason for the high fork count (2/s is not high and not a problem) you can use strace/ltrace to see what it is doing.

Specifically fork should also be visible in process accounting (higher impact) with the accton command. But I dont think it covers clone() for starting threads.

If you get into the range of 100 clones/s then you should really have a look at the applicaton.

BTW: regarding the comment above (cant comment on it yet): no Tomcat does not fork, it only starts threads, but not for each request, it uses a pool.

eckes
  • 835
  • 9
  • 21