Multiple same-net IPs on same interface bad idea? Why?

Question

We have different sets of IPs within the same subnet on the same interface for different purposes:

192.168.1.201..230 are for servers (all Linux)

192.168.1.1..199 are for services running on those servers

As a consequence, every server has multiple IPs - could be two or even more - on the same physical interface. The idea was to be able to move a service from one server to another easily.

There are some services - such as ssh - that are not bound to a specific IP, but all "productive" services (proxy, DNS, NTP, etc.) are bound to one specific IP (and DNS name) which is meant to be moved to another server if the service is moved.

Now, this setup may not be considered "best practice" but so far we haven't had any trouble whatsoever caused by this. However, a lot of people I talk to about this keep saying that "It's just a matter of time until very bad things happen in such a setup" - without being able to explicitly state what that might be. Right now we're discussing whether we should change the setup in the long run which would require a significant amount of effort. Basically I'm looking for arguments for and against changing this.

So this is the question: what "bad things" might be expected in such a setup? Is it something that can always be avoided by doing or refraining from doing certain things? Is there some problem that can be "triggered" in order to demonstrate the problem and show the need to change the setup?

Answer 1

As others have added to, this will work fine provided that the services all bind to OP addresses and you will not be originating requests from these servers which need to be seen from a specific address.

Depending on the nature of the services, you might also want to consider security/firewall implications of your network setup, particularly the increased complexity of the firewalls required and increased risk of attacks/compromised spreading laterally.

It won't solve your problem (but it could reduce it) - you have provided what seem to be arbitrary IP ranges - instead, why don't you arrange the ranges in different netblocks - if space is not an issue, maybe sequential class C's on a /23 boundary, or if space is an issue, 1-190 and 193-254 ?

Answer 2

As long as you can bind everything (or at least everything that matters) to a specific address, it will be fine.

The problems start with applications that can't bind to a specific address: They will choose one of the many IPs on the interface as source address, and that choice may not be particularly consistent. Which may break things on the other end.

This also applies to packets that are sent out by the kernel for some reason; they also will choose some of the many source addresses.

It will also matter if you start doing more complicated networking stuff with iptables or tc.

If none of these applies to your server, you'll be fine. If any of this starts to materialise, you can always add a network namespace (either for all your services, or for things that can't bind and should be guaranteed to use the single "main" address of the server).

Answer 3

There is nothing wrong with having multiple IP addresses on a server, and I occasionally do this myself. For more advanced setups I combine this with BGP so that I can run multiple instances of a service with redundancy.

There are only three things that have bitten me with setups like this, and one of them is not really relevant to this question as it's more BGP related.

1. Always bind on addresses, especially for UDP (like DNS). Listening on 0.0.0.0 or * can cause surprises. If you want to listen on multiple addresses, it's often better to use multiple Listen directives (a lot of software supports this)
2. When using BGP, beware not to accidentally create an anycast service if that's not what you were intending. As long as you make sure that every IP address is claimed by only one server, this cannot happen.
3. When switching addresses between servers, Duplicate Addresses Detection may stop a server from configuring an address if the previous owner of the address still answers on it. I asked a question about this on the Unix stack exchange.