10

We use static DHCP leases for our Linux servers on our small business network. Recently something bad happened to the DHCP server and all the linux servers ended up losing their IP addresses since they couldn't find a DHCP server on the network.

Fast forward a day later, the DHCP server was back up and running, but the servers didn't seem to be looking for the DHCP server anymore, so therefore they didn't have an IP address and weren't trying to get one. I ended up having to manually and physically goto each server and restart the network, at which point they would immediately get their IP address back.

Is there anything I can do on the servers to prevent this problem in the future? I'm sure after the DHCP server went down, the servers maybe tried to get an IP address renewed for a while, but eventually they apparently just stopped trying.

Obviously a solution is to have a backup DHCP server, but let's just pretend that isn't an option. Is there some way to force servers to keep trying to renew their IP indefinitely?

The servers are a mix of CentOS and Fedora.

Skyhawk
  • 14,149
  • 3
  • 52
  • 95
Jake Wilson
  • 8,494
  • 29
  • 94
  • 121
  • 8
    *groan* - This is why I don't recommend DHCP for servers – Matt Simmons Nov 05 '10 at 17:10
  • Are you certain the machines where not trying to renew? How long did you wait. Please remind us non-CentOS people which dhcp client is CentOS using? There may be a client-specific options to set the retry interval. – Zoredache Nov 05 '10 at 17:46

4 Answers4

13

Did they revert to APIPA addressing? 169.254.0.0/16? You can disable that with the NOZEROCONF=Yes directive in /etc/sysconfig/network.

You should also extend your DHCP lease to be twice the longest reasonable outage interval of the DHCP server. Clients will generally wait half of the lease period before trying to renew; if they only check in once every couple of days, you have considerable leeway to fix the server or stand up a new one.

We run fully-reserved DHCP in production; it has not been a reliability problem. In fact, I contend that static IP addressing is one of the last manual operations that folks cling to, and it's largely an anachronism. I've not done a formal analysis, but in three years we've had one DHCP related issue, and that was an admin fat-finger which misfired a "release now" script. On the histogram of "Oops" causes, DHCP is way out on the long-tail.

I know everyone loves to slag on Windows, but IME when APIPA is disabled and DHCP enabled, Windows machines are downright tenacious about holding onto whatever their last DHCP assignment was. I've powered up machines after months of mothball, and wireshark shows them asking for the last DHCP-assigned address.

AndyN
  • 1,739
  • 12
  • 14
6

I use static IP-addresses for servers, DHCP for personal computers only.

Whilst I believe DHCP makes some effort to avoid changing a devices IP-address, this is not quaranteed - at the expiry of a lease the computer might be allocated a different IP-address by DHCP. I've always assumed this would be a bad thing to let happen to a server in the middle of a workday.

RedGrittyBrick
  • 3,792
  • 1
  • 16
  • 21
  • 4
    +1 Don't use DHCP where you want to assign static addresses. You've added extra complexity/methods of failure. Also, setup a backup DHCP server; redundancy of critical systems is basic stuff any professional should know. – Chris S Nov 05 '10 at 17:08
  • While your concerns are correct for dynamic DHCP leases, the poster is using static leases, which always assign the same IP based on the MAC-address and doesn't have this downside. – Martijn Heemels Nov 05 '10 at 17:19
  • @Martin - even when using DHCP reservations, the client systems still check in at the end of their lease, and if the DHCP server is unavailable, all bets are off. As Chris S stated, using DHCP for servers just adds an additional level of complexity that is unneeded. – EEAA Nov 05 '10 at 17:27
  • @Martin - ah yes I missed that, thanks for the correction. – RedGrittyBrick Nov 05 '10 at 17:36
  • 2
    There is chance of error from statically configuring hosts as well, from people duplicating addresses, trying to set the IP address to the gateway, setting a wrong mask, and so on. I have seen more errors in the last few years from statically configured addresses then from DHCP failures. Being able to actually dynamically update DNS servers, or deliever other options pushes me into using DHCP more. – Zoredache Nov 05 '10 at 17:51
6

You can set PERSISTENT_DHCLIENT=yes in /etc/sysconfig/network-scripts/ifcfg-ethX. This effectively removes the -1 flag that is sent to dhclient by default on RedHat style systems. From man dhclient:

The -1 flag cause dhclient to try once to get a lease. If it fails, dhclient exits with exit code two.

However, in the case that you have a power outage and your servers come back online before your switches do, this does not work due to https://bugzilla.redhat.com/show_bug.cgi?id=234075. Basically that bug says that ifup first check to see if a link is present on the interface before attempting to run dhclient. No link -> no dhclient.

My solution is to add this to root's crontab:

*/5 * * * * ifconfig eth0 | grep -q "inet addr:" || ifup eth0 2>&1 >/dev/null

Also add this to /etc/dhcp/dhclient.conf for speedy recovery:

retry 10
timeout 10
Andrew Schulman
  • 8,561
  • 21
  • 31
  • 47
Scott Duckworth
  • 806
  • 1
  • 8
  • 12
  • Could you explain the line going into the crontab? I gathered that you look for the ip address in the output of `ifconfig eth0`, but you lost me after that. – Shoan Oct 07 '11 at 16:56
  • 1
    grep has an exit code of 0 (true) if the search string is found and a non-zero exit code (false) if the search string is not found. The || (binary OR operator) uses short-circuit evaluation, so if the grep returns true then the ifup doesn't run. Otherwise, if the grep returns false, the ifup is run. The 2>&1 redirects standard-error to standard-output, and >/dev/null redirects standard-output to the bit-bucket (meaning there's no output and no annoying emails from cron). – Scott Duckworth Nov 09 '11 at 21:39
1

Lease times are set by the server, not by the client. You could try increasing the lease length for whatever scope your servers are assigned to, but that's really just a band-aid. You really need to either set up a redundant DHCP server or preferrably, moving your servers to statically-assigned addresses.

EEAA
  • 108,414
  • 18
  • 172
  • 242