How many VLANs are too few and too many?

Question

We're currently running a netwok of 800+ PCs and 20+ servers, the network infrastucture is along the lines of Core Switch 10Gb-> Area Switch 2GB-> Local Switch 1GB-> Desktop. All running 3Com equipment(1).

We have 3 Area switches for four areas (A, B ,C, D is merged with the core), each area switch will have between 10 and 20 local switches connected to these. There is also a backup core switch, lesser powered but connected as the main core switch is.

We also have an IP phone system. The Computers/servers and swicthes are on a 10.x ip range, the phones on a 192.168.x range. Computers generally don't have to talk to each other except in computer labs, but they do need to be able to talk to most of our servers (AD, DNS, Exchange, File storage etc.)

When we set up, it was decided that we were to have 3 VLANs, one for Switches & Computers, one for Phones and one for server replication (this was against the 3Com engineers advice). The network has been stable and working since this point(2), but we've now started to upgrade to SAN and Virtualisation environement. Now splitting this new infrastructure into seperate VLANs makes sense, and re visting how our VLANS are set up seems sensible.

It's now being proposed that VLANs should be set up in a room by room basis, ie a computer lab with 5+ PCs should be it's own VLAN, but if we follow this model we'll be looking at least at 25 "new" VLANS, plus the VLANS for SAN/Virtual servers. Which seems to me will add an excessive amount of administration, although I am quite happly to be proved wrong.

What would be best practice seem to suggest? Is there a certain number of PCs that are advisable not to go over/below in a VLAN.

(1) The 3Com switches (3870 & 8800) route between VLANs differently to how some others do it, it doesn't require a seperate router as they are layer3.

(2) We do sometimes get high discard rates, or STP changes, and at time 3Com Network director report that switches are underload and slow to respond to pings, or a failed switch managing to take down the network (all phone & computer VLANS!, once, no idea why)

Answer 1

It sounds like somebody in your organization wants to create VLANs w/o understanding the reasons why you'd do it and the pros/cons associated therewith. It sounds like you need to do some measurement and come up with some real reasons for doing this before moving forward, at least with the insane "VLAN for a room" silliness.

You shouldn't start breaking an Ethernet LAN into VLANs unless you have good reasons to do it. The best two reasons are:

• Mitigating performance problems. Ethernet LANs can't scale indefinitely. Excessive broadcasts or flooding of frames to unknown destinations will limit their scale. Either of these conditions can be caused by making a single broadcast domain in an Ethernet LAN too big. Broadcast traffic is easy to understand, but flooding of frames to unknown destinations is a bit more obscure (so much so that none of the other posters here even mention it!). If you get so many devices that your switch MAC tables are overflowing switches will be forced to flood non-broadcast frames out all ports if the destination of the frame doesn't match any entries in the MAC table. If you have a large enough single broadcast domain in an Ethernet LAN with a traffic profile that hosts talk infrequently (that is, infrequently enough that their entries have aged out of the MAC tables on your switches) then you can also get excessive flooding of frames.

• A desire to limit / control traffic moving between hosts at layer 3 or above. You can do some hackery examining traffic at layer 2 (ala Linux ebtables) but this is difficult to manage (because rules are tied to MAC addresses and changing out NICs necessitates rule changes) can cause what appear to be really, really strange behaviors (doing transparent proxying of HTTP at layer 2, for example, is freaky and fun, but is utterly un-natural and can be very non-intuitive to troubleshoot), and is generally difficult to do at lower layers (because layer 2 tools are like sticks and rocks at dealing with layer 3+ concerns). If you want to control IP (or TCP, or UDP, etc) traffic between hosts, rather than attacking the problem at layer 2, you should subnet and stick firewalls / routers with ACLs between the subnets.

Bandwidth exhaustion problems (unless they're being caused by broadcast packets or flooding of frames) are not solved with VLANs typically. They happen because of a lack of physical connectivity (too few NICs on a server, too few ports in an aggregation group, the need to move up to a faster port speed) and can't be solved by subnetting or deploying VLANs since that won't increase the amount of bandwidth available.

If you don't have even something simple like MRTG running graphing per-port traffic statistics on your switches that's really your first order of business before you start potentially introducing bottlenecks with well-intentioned but uninformed VLAN segmentation. Raw byte counts are a good start, but you should follow it up with targeted sniffing to get more details about the traffic profiles.

Once you know how traffic moves around on your LAN you can begin to think about segmenting the LAN for performance reasons.

If you're really going to try and button down packet and stream-level access between VLANs be prepared to do a lot of legwork with application software and learning / reverse-engineering how it talks over the wire. Limiting access by hosts to servers can often be accomplished with filtering functionality on the servers. Limiting access on the wire can provide a false sense of security and lull administrators into a complacency where they think "Well, I don't need to configure the app. securely because the hosts that can talk to the app. are limited by 'the network'." I'd encourage you to audit the security of your server configuration before I'd start limiting host-to-host communication on the wire.

Typically you create VLANs in Ethernet and map IP subnets 1-to-1 onto them. You're going to need a LOT of IP subnets for what you're describing, and potentially a lot of routing table entries. Better plan those subnets with VLSM to summarize your routing table entries, eh?

(Yes, yes-- there are ways not to use a separate subnet for every VLAN, but sticking in a strictly "plain vanilla" world you'd create a VLAN, think up an IP subnet to use in the VLAN, assign some router an IP address in that VLAN, attach that router to the VLAN, either with a physical interface or a virtual subinterface on the router, connect some hosts to the VLAN and assign them IP addresses in the subnet you defined, and route their traffic in and out of the VLAN.)

Answer 2

VLANs are only really useful for restricting broadcast traffic. If something is going to do a lot of broadcasting, then separate it into its own VLAN, otherwise I wouldn't bother. You might want to have a virtualised duplication of a live system on the same network and want to use the same address range, then again, that might be worth a separate VLAN.

Answer 3

VLANs are good as an additional security level. I don't know how 3Com handles it but usually you can segment different functional groups into different VLANs (e.g. Accounting, WLAN, etc.). You can then control who has access to a particular VLAN.

I don't believe there is any significant performance loss if there are many computers in the same VLAN. I do find it impractical to segment LAN in a room by room basis, but again, I don't know how 3Com handles it. Usually the guideline is not size, but rather security or operation.

In effect I don't see any reason to even segment LAN into different VLANs if there are no security or operational gains.

Answer 4

Unless you have 25 test and development groups that regularly kill the network with broadcast floods, 25 per-room VLAN's are 24 too many.

Obviously your SAN needs its own VLAN and not the same VLAN as the virtual systems LAN and Internet access! This can all be done through a single ethernet port on the host system, so no worries there about splitting those functions.

If you have performance crap-outs, consider putting your Phone and SAN on separate network hardware, not just VLAN's.

Answer 5

They're also useful for preventing DHCP broadcasts for reaching unwanted network devices.

Answer 6

There is always going to be broadcast traffic, whether it's name resolution broadcasts, ARP broadcasts, etc. The important thing is to monitor the amount of broadcast traffic. If it exceeds 3 - 5% of total traffic then it's a problem.

VLAN's are good for reducing the size of broadcast domains (as David stated) or for security, or for creating dedicated backup networks. They're not really meant as "management" domains. In addition, you'll add routing complexity and overhead to your network by implementing VLAN's.

Answer 7

Generally, you only want to consider using VLANs when you need to quarantine devices (such as an area where users can bring in their own laptops, or when you've got a critical server infrastructure that must be protected) or if your broadcast domain is too high.

Broadcast domains can typically be about 1000 devices large before you start to see issues on 100Mbit networks, though I'd bring that down to 250 devices if you're dealing with relatively noisy Windows areas.

For the most part, modern day networks don't need VLANs unless you're doing this quarantining (with appropriate firewalling using ACLs, of course) or broadcast limitation.