My team is trying to find a solution to remotely access and update multiple Linux devices behind a firewall at several locations with their own firewalls. Traditionally, we would visit the business' physical site and update the devices with Ansible running from within their network, or make changes via RDP and individual SSH access. Port forwarding is not an option, not even for a bastion.
||REMOTE SERVER|| ===> ||LOCATION 1 FIREWALL|| ===> [DEVICE 1, DEVICE 2, ..., DEVICE N]
(executing cmds) ===> ||LOCATION 2 FIREWALL|| ===> [DEVICE 1, DEVICE 2, ..., DEVICE N]
===> ||LOCATION 3 FIREWALL|| ===> [DEVICE 1, DEVICE 2, ..., DEVICE N]
...
===> ||LOCATION N FIREWALL|| ===> [DEVICE 1, DEVICE 2, ..., DEVICE N]
What is best practice with AutoSSH, or is there an alternative that would work better for maintaining these devices? We have two solutions so far.
Solution 1 - AutoSSH for every device:
-Establish AutoSSH connection from each and every individual device on each network (potential for hundreds per location). Configure SSH access for each of those devices.
Host loc_1_device_1
HostName localhost
Port 6000
User maintenance
IdentityFile /home/devops/.ssh/remote_maint
Host loc_1_device_2
HostName localhost
Port 6001
User maintenance
IdentityFile /home/devops/.ssh/remote_maint
Host loc_1_device_3
HostName localhost
Port 6002
User maintenance
IdentityFile /home/devops/.ssh/remote_maint
Host loc_2_device_1
HostName localhost
Port 7000
User maintenance
IdentityFile /home/devops/.ssh/remote_maint
Host loc_2_device_2
HostName localhost
Port 7001
User maintenance
IdentityFile /home/devops/.ssh/remote_maint
Solution 2-AutoSSH for proxy/jump host only:
- Establish AutoSSH connection from a single proxy (possibly a secondary as failover) on each of our client's networks. Configure SSH access for each of those devices to a port on the remote server.
Host location_1
HostName localhost
Port 6000
User maintenance
IdentityFile /home/devops/.ssh/remote_maint
Host location_2
HostName localhost
Port 6001
User maintenance
IdentityFile /home/devops/.ssh/remote_maint
From there, connection seems easy enough with ssh -t location_1 ssh user@192.168.0.2
, or some ansible config for proxy command.
I've done a proof of concept for solution 2, running Ansible commands from the remote server, and it seems to work well, but there is pushback from other members on the team to go with solution 1 to remain a 1-1 inventory of every device (Which should just be the entire point of Ansible?). What concerns me with solution 1 is that there could be thousands of devices trying to maintain reverse SSH simultaneously, which could result in needless network overhead. It also seems like a needless amount of work for adding and removing devices from that network.
Is there another way to approach this? Am I on the right track, or am I crazy for thinking the rest of the team is crazy?