We have a simple 3-node Kubernetes cluster on Azure Kubernetes Service (AKS). A lot of the time that's more than sufficient infrastructure-wise, but occasionally we need to be able scale out for a couple hours to 50 instances of a service then scale back again.
Combining AKS with Azure Container Instances (ACI) via the Virtual Kubelet seems like an ideal solution for this scenario.
From a cost management perspective, we'd prefer stuff running on our VMs if there's available capacity. We're already paying for them, so there's no point paying for ACI instances too.
Question 1
If we scale out via the Virtual Kubelet using ACI, how does Kubernetes choose which pods to scale back later on? Is its approach consistent with our cost management requirement - i.e. "genuine" nodes are preferred - or could it be made to be consistent?
Question 2
Scenario: we run two apps - "App1" and "App2" - in Kubernetes. App1 is the one that causes the need to scale out to 50 instances via ACI; App2 is running on the "genuine" nodes.
Say we then use Helm to update App2. When it restarts, I'm guessing Kubernetes could place it on an ACI instance since we're at scale at that point.
Later, the need for the 50 instances of App1 goes away and we scale back again.
App2 would still be running on ACI instances when we'd prefer them to be back on genuine nodes at this point.
Would Kubernetes manage this according to our cost managemet requirements, or is some extra shepherding required?