2

I have a vServer (16GB RAM, 6 vCores) running Ubuntu Server (Linux 4.15.0) with a few docker containers.

A few days ago, I deployed a new container, and it would simply refuse to start with weird native crashdumps from docker-compose, most including these error message:

fork/exec /usr/bin/containerd-shim-runc-v2: resource temporarily unavailable: unknown
runtime/cgo: pthread_create failed: Resource temporarily unavailable SIGABRT:
failed to create new OS thread (have 5 already; errno=11) runtime: may need to increase max user processes (ulimit -u)

After that, this happened with many previously working containers as well. However, I restarted dockerd and all my containers, and the problem seemed to go away.

Today, I wanted to start up my Minecraft server (Forge modded 1.16 on Java 11 without docker) and the problem is back:

[136,281s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.

And the weirdest thing so far, it cannot open the TTY device?:

java.io.IOException: Cannot run program "stty": error=11, Resource temporarily unavailable

After these messages, the SSH session just locked up, and a second terminal showed me that I haven't reached a single limit of ulimit -a, and with Minecraft running I still have 9GB of RAM free. htop shows 90 tasks running. I've been working with Linux servers for years now, never seen errors like these - what could be the cause for this?

I know errors like these have been asked before on SF, but not a single one of the answers changed something for me.

Twometer
  • 91
  • 1
  • 8
  • Please post the output of `virt-what` (install it if it is not present). – Michael Hampton Aug 18 '21 at 19:46
  • @MichaelHampton It shows `openvz lxc` – Twometer Aug 18 '21 at 19:52
  • Well that's doubly bad news. A container inside a container and you're trying to use Docker? Just stop now and get a real virtual machine from another provider. – Michael Hampton Aug 18 '21 at 19:58
  • @MichaelHampton Does it mean that my hoster already uses two nested containers? I've actually never had problems up until now – Twometer Aug 18 '21 at 19:59
  • 1
    That's what it looks like. Or it could be the new OpenVZ 7, which is based on LXC. Either way you're nesting containers, Docker needs special help to do that, and OpenVZ is very weird with resource allocation anyway; among other things it has a lot of trouble with Java apps. OpenVZ is best avoided by virtually everyone. – Michael Hampton Aug 18 '21 at 20:01
  • @MichaelHampton okay, I didn't know that... I've been running docker containers on that system for months now and this never happened. So there's basically nothing I can do to fix this except serach for another provider? Another thing I noticed is that since the first docker crash, bash itself sometimes fails to fork() a new process - with the same error message... does that also come from the virtualizer? – Twometer Aug 18 '21 at 20:05
  • This also seems to just happen when there are already "a lot" of docker containers running (currently I have 18 to be exact) – Twometer Aug 18 '21 at 20:08

1 Answers1

1

Finally found the problem: My hoster is limiting the number of threads to 700 in their OpenVZ config.

I can see that in the /proc/user_beancounters. It seems that if numproc goes near 700, these errors occur. Nothing I can do except reduce the number of threads / docker containers.

Twometer
  • 91
  • 1
  • 8