After a dozen terminated instances and another dozen of tricks, I have figured it out. Apparently salt master is a somehow of a slacker:
aws-us-east-1-ubuntu-base: Cloning into '/tmp/docker-formula/.'...
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,161 [salt.cli.daemons ][INFO ][3762] Setting up the Salt Minion "ip-172-30-2-137.ec2.internal"
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,169 [salt.cli.daemons ][INFO ][3735] Setting up the Salt Master
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,321 [salt.daemons.masterapi][INFO ][3735] Preparing the root key for local communication
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,322 [salt.cli.daemons ][INFO ][3735] The salt master is starting up
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,329 [salt.master ][INFO ][3735] salt-master is starting as user 'root'
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,329 [salt.master ][INFO ][3735] Current values for max open files soft/hard setting: 100000/100000
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,329 [salt.master ][INFO ][3735] Creating master process manager
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,329 [salt.master ][INFO ][3735] Creating master maintenance process
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,331 [salt.master ][INFO ][3735] Creating master publisher process
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,332 [salt.master ][INFO ][3735] Creating master event publisher process
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,342 [salt.master ][INFO ][3735] Creating master request server process
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,347 [salt.master ][INFO ][3769] Starting the Salt Publisher on tcp://0.0.0.0:4505
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,349 [salt.master ][INFO ][3769] Starting the Salt Puller on ipc:///var/run/salt/master/publish_pull.ipc
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:08,412 [salt.master ][INFO ][3804] Setting up the master communication server
.....
aws-us-east-1-ubuntu-base: + sleep 10
.....
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:13,097 [salt.master ][INFO ][3784] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:13,097 [salt.master ][INFO ][3784] Clear payload received with command _auth
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:13,098 [salt.master ][INFO ][3784] Authentication request from ip-172-30-2-137.ec2.internal
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:13,098 [salt.master ][INFO ][3784] New public key for ip-172-30-2-137.ec2.internal placed in pending
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:13,100 [salt.crypt ][ERROR ][3762] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:13,100 [salt.crypt ][INFO ][3762] Waiting 10 seconds before retry.
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:13,199 [salt.master ][INFO ][3785] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:13,203 [salt.master ][INFO ][3795] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:13,203 [salt.master ][INFO ][3803] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:13,202 [salt.master ][INFO ][3793] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
aws-us-east-1-ubuntu-base: + sudo salt-key -A --yes
aws-us-east-1-ubuntu-base: The following keys are going to be accepted:
aws-us-east-1-ubuntu-base: + sleep 30
aws-us-east-1-ubuntu-base: Unaccepted Keys:
aws-us-east-1-ubuntu-base: ip-172-30-2-137.ec2.internal
aws-us-east-1-ubuntu-base: Key for minion ip-172-30-2-137.ec2.internal accepted.
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,112 [salt.master ][INFO ][3784] Clear payload received with command _auth
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,112 [salt.master ][INFO ][3784] Authentication request from ip-172-30-2-137.ec2.internal
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,112 [salt.master ][INFO ][3784] Authentication accepted from ip-172-30-2-137.ec2.internal
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,127 [salt.master ][INFO ][3795] Clear payload received with command _auth
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,127 [salt.master ][INFO ][3795] Authentication request from ip-172-30-2-137.ec2.internal
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,128 [salt.master ][INFO ][3795] Authentication accepted from ip-172-30-2-137.ec2.internal
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,265 [salt.loaded.int.module.cmdmod][INFO ][3762] Executing command 'date +%z' in directory '/root'
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,270 [salt.minion ][INFO ][3762] Added mine.update to scheduler
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,271 [salt.utils.schedule][INFO ][3762] Added new job __mine_interval to scheduler
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,272 [salt.cli.daemons ][INFO ][3762] The salt minion is starting up
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,272 [salt.minion ][INFO ][3762] Minion is starting as user 'root'
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,273 [salt.minion ][INFO ][3762] Starting pub socket on ipc:///var/run/salt/minion/minion_event_c0afd79315_pub.ipc
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,273 [salt.minion ][INFO ][3762] Starting pull socket on ipc:///var/run/salt/minion/minion_event_c0afd79315_pull.ipc
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,280 [salt.minion ][INFO ][3762] Minion is ready to receive requests!
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:23,281 [salt.utils.schedule][INFO ][3762] Running scheduled job: __mine_interval
aws-us-east-1-ubuntu-base: + sudo salt * -v -t 10 state.apply
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:51,306 [salt.master ][INFO ][3795] Clear payload received with command publish
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:51,308 [salt.master ][INFO ][3795] User sudo_root Published command state.apply with jid 20160913230251306897
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:51,309 [salt.minion ][INFO ][3762] User sudo_root Executing command state.apply with jid 20160913230251306897
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:51,314 [salt.minion ][INFO ][4361] Starting a new job with PID 4361
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:51,427 [salt.state ][INFO ][4361] Loading fresh modules for state activity
aws-us-east-1-ubuntu-base: 2016-09-13 23:02:51,438 [salt.fileclient ][INFO ][4361] Fetching file from saltenv 'base', ** done ** 'top.sls'
Notice the delays between my (intentional) sleeps. These delays were causing all sort of issues since most of the commands in my shell script are executed instantly.
Another point is the minion receiving the job __mine_interval
as soon as its key is accepted. I'm not sure why the minion could not just queue the job I asked it for, instead it kept responding with Minion did not return. [No response]
. Another sleep solved this bit.
Here is my working script: (uncomment the tails if you want to see salt logs)
#!/bin/bash
# show stuff being executed
set -x
# salt hostname for minions
echo 127.0.0.1 salt | sudo cat >> /etc/hosts
# salt
sudo add-apt-repository --yes ppa:saltstack/salt
sudo apt-get update
sudo apt-get install --yes salt-api salt-cloud salt-master salt-minion salt-ssh salt-syndic
# run on startup
sudo update-rc.d salt-master defaults
sudo update-rc.d salt-minion defaults
# increase log level
echo log_level: info | sudo cat >> /etc/salt/master
echo log_level: info | sudo cat >> /etc/salt/minion
# restart
sudo service salt-master restart
sudo service salt-minion restart
# show logs
# sudo tail -f /var/log/salt/master &
# sudo tail -f /var/log/salt/minion &
# get docker-formula and move it to /srv/salt
sudo mkdir /tmp/docker-formula
sudo git clone https://github.com/saltstack-formulas/docker-formula /tmp/docker-formula/.
sudo mkdir -p /srv/salt
sudo cp -vr /tmp/docker-formula/docker /srv/salt/docker/
# top.sls
sudo cp -v /ops/config/top.sls /srv/salt/
# let things .. settle
sleep 10
# accept all minions
sudo salt-key -A --yes
# let things .. settle
sleep 30
# apply to minions
sudo salt '*' -v -t 10 state.apply
# add user to docker group
sudo usermod -aG docker $USER