I'm trying to run several lxc containers on an ubuntu 16.04 server. (libvirt version is 1.3.1). The nature of my team's use of these containers means that they are constantly booting and then being destroyed, and we manage this lifecycle using the python libvirt api bindings. Our tooling used to work fine on older hardware with older ubuntu (14.04) and an older libvirt version (1.2.4).
On the upgraded hardware/software setup, everything starts out working fine: containers behaving as they should. But as the server continues to run under the configured load, it seems like the containers aren't being fully destroyed. I say this because when I run virsh list, I see the number of running containers I'd expect -- however, when I list processes using ps and grep for the container names, I find that I have a whole bunch of libvirt processes still running for containers I've destroyed -- they seem to have been leaked by libvirt. I don't know why they aren't getting destroyed when we use the libvirt api to destroy them. Has anyone seen this kind of behavior before? If so, what did you do about it? I'm completely stumped.
