How to deal with race condition on Network Boot

Question

I have a remote container filled with many PXE booted computers connected to an on site tftp/dhcp server.

If there is a power outage, the server machine takes ages to boot and the PXE booted machines will time out and jump back into bios and never boot. This is bad because I don't have physical access to the machines.

I have a few ideas for solutions to this:

Find a way to make booting take longer on the client machines (haven't figured out how to do this)
Increase timeout on network boot on client machines (haven't figured out how to do this)
Get MAC addresses from DHCP lease file and send them wake on LAN multicast requests after server machine is ready (This seems like the least brittle but unnecessarily complex)
Replace server machine with one that boots faster (...)

I know there has to be a simple solution that I haven't considered.

What should I do?

@MichaelHampton I don't care if they die. I just want them to be able to come back up. WAL would be much more reasonable for me than thousands of dollars worth of UPS's. — mraaroncruz, Jan 16 '18 at 21:07
You could probably find ways to make your existing server boot faster. You didn't mention what hardware it is, though. — Michael Hampton, Jan 16 '18 at 21:23
I want to suggest a fifth option: * some bios/efi support the option of an eternal boot loop (or at least a very high counter) if none of the configured boot sources succeed — Phillip -Zyan K Lee- Stockmann, Jan 16 '18 at 21:57
@Phillip-ZyanKLee-Stockmann I was hoping for something like this but the ASUS BIOS is pretty packed. I will google this. — mraaroncruz, Jan 16 '18 at 22:20
@MichaelHampton Yeah. This sounds like an option, but it's still far from ideal. Thanks for the suggestions though! — mraaroncruz, Jan 16 '18 at 22:21
You don’t need thousands of dollars worth of UPSes. You only need ONE, on the server. Don’t let it shut down and you won’t have to wait for it to start up. — Appleoddity, Jan 17 '18 at 04:48
@Appleoddity lol. Of course. I don't know how I missed that. — mraaroncruz, Jan 17 '18 at 10:56

Zac67 · Answer 1 · 2018-02-01T20:21:45.527

1

Alternatively,

Get the server machine a UPS.
Use a PDU to boot the PXE clients with a delay.
Set PXE clients to use PXE exclusively and retry indefinitely (if possible).
Optimize the server so it boots faster (SSD, more RAM, ...).

Edit:

If 3. doesn't work put a USB stick into each computer which just a PXE client or a reboot setup on it.
Leave the computers off while the server boots and wake them by WoL when it's up. [...] Saw your script below doing the exact same. ;-)

edited Feb 01 '18 at 20:21

answered Jan 18 '18 at 12:16

Zac67

8,639
2
10
28

1. UPS could run out of power (reasonable priced ones only last an hour). 2. I like this but I still don't really trust it. 3. Looked at every setting and couldn't find this, I would love this. 4. Unpredictable. I wouldn't trust this. But more than deserving of an upvote! – mraaroncruz Feb 01 '18 at 15:42
1

Any UPS eventually runs out on battery. I usually size them for 30 minutes, longer outages are rare and if they do happen they're much longer (hereabouts). I guess that's a mobile setup? A power outage leads to various recovery procedures (resync RAID, check filesystem), so a UPS is definitely a plus. Option 4 has it perks as well. If budget and weight allow I'd go for these options. Opt 3 would be free but if it doesn't work it doesn't work. – Zac67 Feb 01 '18 at 20:17

score 0 · Accepted Answer · answered Feb 01 '18 at 15:42

0

I went with a quick and dirty Golang script, run in the crontab and set the PXE booted machines to not power on when given power.

Here is the script if you have a similar problem https://gist.github.com/mraaroncruz/f103b8af4d81f59a54a5f2af6dc238b6

answered Feb 01 '18 at 15:42

mraaroncruz

191
6

How to deal with race condition on Network Boot

2 Answers2