6

We recently set up an HP ProLiant DL360p server for some heavy grunt-work. For one reason or another we disabled hyperthreading in the system BIOS when setting it up. Being a dual-CPU 8-core system, that gave us 16 hardware threads. 32 GB RAM. We are running Ubuntu 12.04 64-bit.

The bulk of the work done is by a VHDL "synthesizer", or compiler. It is a QT application, but typically run in command-line mode (no GUI). This compiler has been running well for the last few weeks, kicked off by a continuous integration system (Jenkins).

Today we re-enabled hyperthreading to get access to all 32 hardware threads. However now this compiler hangs with the following error in every case I can contrive:

QThread::start: Thread creation error: Resource temporarily unavailable

The process appears to be stalled, not busy on any CPU, and ctrl-c interrupts it.

I did some searching online and it would appear that this may be related to OS maximum thread limits, but I'm not sure how to go about changing this. The default is supposedly around 800 threads anyway, which should be more than enough for this compiler that runs only a small number (maybe 2?).

In the meantime I will have to disable hyperthreading, but I was wondering if this is a known issue with high-end servers running 64-bit Linux? Is there a known workaround? Or is it more likely a problem with that specific application?

davidA
  • 353
  • 2
  • 11

1 Answers1

1

Could you describe how this process is started, and provide us with the output of ulimit -a from a context as close from that as possible?

As pthread_create(3) explains, you are running into:

ERRORS
       EAGAIN Insufficient  resources  to create another thread, or a system-
              imposed limit on the number of threads  was  encountered.   The
              latter  case  may  occur  in  two  ways:  the RLIMIT_NPROC soft
              resource limit (set via setrlimit(2)), which limits the  number
              of  process  for  a  real user ID, was reached; or the kernel's
              system-wide limit on  the  number  of  threads,  /proc/sys/ker‐
              nel/threads-max, was reached.
Pierre Carrier
  • 2,607
  • 17
  • 28
  • I can tell you (*with HT currently disabled*) that the binary is executed from a sh script (sets up environment) and a ulimit -a just before invoking the binary gives: time(seconds) unlimited file(blocks) unlimited data(kbytes) unlimited stack(kbytes) 131072 coredump(blocks) 0 memory(kbytes) unlimited locked memory(kbytes) 64 process 256966 nofiles 1024 vmemory(kbytes) unlimited locks unlimited . Command is /opt/diamond/1.4/synpbase/linux/mbin/synbatch -product synplify_pro -rxl isIWgMKg -batch build.tcl – davidA Jul 06 '12 at 02:20
  • OK, then it looks like a kernel bug. Can we have `uname -a`? – Pierre Carrier Jul 06 '12 at 08:51
  • Linux build01 3.2.0-26-generic #41-Ubuntu SMP Thu Jun 14 17:49:24 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux – davidA Jul 08 '12 at 21:26
  • Is this bug documented anywhere else? – davidA Aug 23 '12 at 01:02
  • Trying now, 3 months later, with kernel 3.2.0-33-generic, and the issue is exactly the same. I also tried with a newer version of the tool - Lattice Diamond 2.0 - and it has the same issue. So we'll be turning hyperthreading off again. – davidA Nov 20 '12 at 02:59
  • 1
    Have you tried upgrading to a newer Ubuntu kernel (i.e. 3.5.0 from 12.10) or building your own? It does sound like a kernel/RCU bug! – priestjim Nov 25 '12 at 17:33