What is a container and how does it work?
A container is simply a way to run multiple processes on a system that expect to have full root privileges, but which should not actually be permitted to perform sensitive configuration changes or interfere with other processes outside of the container. When a process makes a system call, the kernel will traditionally check its privileges (such as UID and GID) and either grant or deny the request. When running in a container, the kernel will set up a "namespace", and any administrative actions will only take effect within that one namespaces. Certain actions, such as changing system-wide configurations, are not possible even with root in a container.
Containers can cause severe security issues when privileged processes run under them. If your application is not privileged and cannot influence or hijack another privileged process running under the same container, it won't be able to take advantage of a container's intrinsic weaknesses. This just means that the container won't necessarily reduce security, but it will also not improve it.
I give a long description of multiple sandboxing technologies in another answer, including information on containers that can be found under the heading "Containers".
Is a container the right solution here?
Is it any more secure to run an application inside a container which is also inside of a virtual machine?
There is no difference. The container does not add any extra security barrier or force the attacker to go through any extra steps. Regardless of which of the two setups you choose, the attacker will only have to compromise the application, compromise the kernel, then escape the VM. The only reason to use a container is if you want to run multiple processes that each believe they are root on the same system, without giving them sufficient privileged access to interfere with the other processes.
Furthermore, does using a container actually introduce security concerns in and of itself?
Yes, using a container can introduce security concerns if a compromise of the host kernel is a threat. This is because a compromised container kernel compromises the host as well. Instead, you should be sandboxing the untrusted application to minimize the attack surface area that is exposed to it.
The only obvious thing that comes to mind is that it's usually not in the interest of operational security to deploy a technology that you do not know how to properly configure, and I am not super experienced with containers
That is another reason not to use containers in this situation. If you don't know what extra privileges you may be granting containerized processes that wouldn't be granted otherwise, you shouldn't try to run one. Containers have a purpose and they can be configured fairly securely and used to enhance the security of a system, but simply using them is not enough and won't magically improve security.
Can container technology be used to improve security?
Containers on Linux are typically built on top of namespaces or analogous implementations (e.g. OpenVZ). While a full container may not improve security at all, namespace technology can be used to reduce attack surface area. For example, a network namespace will allow you to isolate network interfaces from the application, and a PID namespace will hide other processes from it so it cannot interact with them. Namespaces, along with other techniques such as seccomp and mandatory access controls, can be used to reduce the attack surface area (of both the kernel and other processes) that is exposed to the untrusted application.
