Practical (vs theoretical) max limit of TCP packet size

Question

I'm a web developer who is pretty new to the industry. I was presented with a coding challenge for a job interview where I need to design a messaging system and architect a system for how it deals with messages, malformed messages, different message types, status logging, et cetera...

My question is regarding packet size over TCP.

Incoming messages are at a rate of 10,000 messages/sec at a size of 2KB per message. I've been trying to find the max, max-safe, or max-practical packet size limitation. I've seen in several unverified places (i.e. not in technical documentation) that the max theoretical size is 64KB. Is that correct? In that case my example of sending 2KB messages would easily fit into a single packet and reduce the complexity of this system.

If 64KB is an incorrect number, what would the correct number be? Additionally, I'm not simply trying to understand the max theoretical size but the max practical size. I want to cover edge cases where messages may be slightly larger than the targeted 2KB as well as leave room for the various headers that TCP needs.

Answer 1

Ethernet has always been influential on TCP/IP packet sizes. Ethernet has a standard MTU of 1500 bytes, which, after typical IPv4 header overhead of 20 bytes and typical modern TCP header overhead of 32 bytes (used to be 20 bytes but we add a 12-bytes timestamp option nowadays), results in a TCP Maximum Segment Size of 1448 bytes.

PPPoE, which is popular among DSL-based ISPs, adds another 8 bytes of overhead, so TCP connections that traverse a PPPoE link end up with a TCP MSS of 1440 bytes. Other technologies may add more overhead.

Most modern TCP/IP stacks perform "Path MTU Discovery" (PMTUD) so they never have to rely on IP fragmentation. Unfortunately some sites block the ICMP messages necessary for PMTUD to work, accidentally creating "PMTUD black holes" where PMTUD doesn't work. To allow people behind PMTUD black holes to still connect to their services, Google sites choose to negotiate a very conservative TCP MSS of 1380 bytes (last time I checked).

So I'd say one could make a pretty good argument that if you want to tailor how your application does writes, to make sure they only fill a single packet most of the time, make your writes no larger than 1448 bytes, and perhaps no larger than 1380 bytes.

IPv4 datagrams can be as large as 64 KibiBytes, but exceedingly few paths across the Internet have a 64KiB MTU, so that number is irrelevant to most packet-size planning. Your theoretical messaging-protocol server is probably connected to an Ethernet-like network, which probably uses standard 1500 byte frames, so your own server's IP stack would have to fragment that 64KiB write into 46 or so separate packets before it could even begin transmitting them on the first hop. Even Ethernet networks that are set up to use nonstandard "jumbo frames" usually max out at 9000 byte MTUs. Off the top of my head, I can't even name a physical / data link (layer 1/2) networking technology that allows for 64KiB MTUs. Maybe IP over Thunderbolt.