2022 June - Update
If all you need is remote access to the machine, two newer approaches (if you're comfortable with AWS) would be to use one of:
That said, I would still opt for a pull mechanism for ensuring updates are deployed. You ideally want to use these direct shells only in case of an emergency. Otherwise, you will (inevitably) end up with a Frankenstein fleet of servers, each with their own funny configuration tweaks that were done manually by someone in a pinch, without documentation.
Pull updates, don't push
As you scale, it's going to become unfeasible to do push updates to all your products.
- You'll have to track every single customer, who might each have a different firewall configuration.
- You'll have to create incoming connections through the customer's firewall, which would require port-forwarding or some other similar mechanism. This is a security risk to your customers
Instead, have your products 'pull' their updates periodically, and then you can add extra capacity server-side as you grow.
How?
This problem has already been solved, as you suggested. Here's several approaches I can think of.
using apt: Use the built-in apt system with a custom PPA and sources list. How do I setup a PPA?
- Con: Unless you use a public hosting service like launchpad, Setting up your own apt PPA + packaging system is not for the faint of heart.
using ssh: Generate an SSH public key for each product, and then add that device's key to your update servers. Then, just have your software rsync / scp the files required.
- Con: Have to track (and backup!) all the public keys for each product you send out.
- Pro: More secure than a raw download, since the only devices that can access the updates would be those with the public key installed.
raw download + signature check:
- Post a signed update file somewhere (Amazon S3, FTP server, etc)
- Your product periodically checks for the update file to be changed, and then downloads / verifies the signature.
- Con: Depending on how you deploy this, the files may be publicly accessible (which may make your product easier to reverse engineer and hack)
ansible: Ansible is a great tool for managing system configurations. It's in the realm of puppet / chef, but is agentless (uses python) and designed to be idempotent. If deploying your software would require a complicated bash script, I'd use a tool like this to make it less complicated to perform your updates.
Of course, there are other ways to do this.. But it brings me to an important point.
Sign / validate your updates!
No matter what you do, it's imperative that you have a mechanism to ensure that your update hasn't been tampered with. A malicious user could impersonate your update server in any of the above configurations. If you don't validate your update, your box is much easier to hack and get into.
A good way to do this is to sign your update files. You'll have to maintain a certificate (or pay someone to do so), but you'll be able to install your fingerprint on each of your devices before you ship them out so that they can reject updates that have been tampered with.
Physical Security
Of course, if someone has physical access to the customer's deployment, they could easily take over the server. But at least they can't attack the other deployments! Physical security is likely the responsibiltiy of your customer.
If you would for a moment, imagine what would happen if you used a
large OpenVPN network for updates... They could then use the
compromised server to attack every instance on the VPN
Security
Whatever you do, security needs to be built in from the beginning. Don't cut corners here - You'll regret it in the end if you do.
Fully securing this update system is out of scope of this post, and I strongly recommend hiring a consultant if you or someone on your team isn't knowledgeable in this area. It's worth every penny.
