52

What's the technical limitation preventing us, in the glorious year 2011, from emailing each other 1GB files?

Or is it just the main email platforms dragging their feet?

If I can set my inbox to grab headers only, and then full attachments if I want them, what is the problem?

I feel like email attachment sizes are stuck in 1992...

Drew
  • 661
  • 6
  • 9
  • 23
    Attachment sizes stuck in 1992? Puh-leeze. I'd like to see you *transfer* a 50 MB file in 1992, by *any* method available, let alone attach it to an e-mail (yes, I do have several such e-mails from this current month in 2011, no, I'm not very happy about it). Hint: the preferred method, in 1992, could have included copying to tape and driving to the destination, or perhaps an all-night dial-up and `uucp` session. – Piskvor left the building Aug 24 '11 at 07:56
  • 4
    @Piskvor: Alternatively, a grocery bag full of disks containing multi-volume-spanning arj archives. Kinda unrelated: http://cs-exhibitions.uni-klu.ac.at/index.php?id=187 – sum1stolemyname Aug 24 '11 at 12:00
  • Email was never intended to send files... but it became optional... Best way to share big files is to offer a download link or ftp login (http wasn't intended for file downloading either) And if you'd like to upload a 1gb email attachment for a quick email... be my guest – HTDutchy Aug 24 '11 at 12:03
  • 17
    Bandwidth has little or nothing to do with it; if what you have to communicate with someone else is bigger than 20 megabytes, *email is not the way to send it*. – Shadur Aug 24 '11 at 12:13
  • 2
    @Shadur: It does,in case of unwanted mail - a link in an e-mail (which the recipient *chooses* to click or not) takes thousands of bytes at the extreme end; an attached file in an e-mail is, in most cases, downloaded without prompting (I'm aware of IMAP's capabilities in this regard,but most users have this set to "download everything",which would *somewhat* affect the bandwidth - also used for other purposes besides mail:the usual complaint of the non-IT users before broadband:"Why is my web so slow? Yes,I did send a 10MB e-mail with dancing pigs with 100 people in BCC,how is that relevant?") – Piskvor left the building Aug 24 '11 at 12:22
  • 4
    @Piskvo "Never underestimate the bandwidth of a truck full of tapes"; as true today as ever: you can get >1TB on *one* tape.... – Richard Aug 24 '11 at 18:51
  • @Richard: "...hurtling down the highway" (as bandwidth also relates to speed). But indeed, tapes are as efficient now as ever for HUMONGOUS offline data transfers. – Piskvor left the building Aug 25 '11 at 08:45

6 Answers6

164

The problem is this: e-mail (SMTP/POP3/IMAP/what-have-you) is an ancient, simple protocol originally intended for sending plaintext messages in a trusted network. Using it for sending or receiving large amounts of binary data across today's Internet is a bolted-on hack, completely different from the original use case, and it performs rather miserably in this role.

When you attach a file to the e-mail, it gets base64-encoded, which increases its size by 1/3. Thus, your 1 GB file becomes another 300 MB larger; also, there is no built-in compression to the download protocol, thus no way to speed up the transfer (and in some cases (SMTP for sending,POP3 for receiving), even no way to resume a broken transfer - connection broke at 1.2 GB? Sorry, you need to re-transmit it all again). Moreover, SMTP is a store-and-forward protocol. Guess what? Yup, that 1.3 GB file needs to be copied across multiple servers; cue unbounded happiness from the mail server admins.

This was a problem in the 1990s, when there was no useful alternative (FTP? HTTP/1.0? Puh-leeze); but in the glorious year 2011, with various ways of seamlessly up/downloading data to/from the cloud (e.g. Dropbox, Ubuntu One, Amazon S3, to name the most known), the excuse of "there's no other useful way to do this" is not true any more.

Note also that not everyone is on a 100 Mbit link to the Internet - e.g. mobile and smartphone; not every mail client is capable of downloading only the headers (e.g. POP3 is still in much use), and not every user is willing to download the 20 inevitable "look at this funneh 1 GB video" e-mails per week that will appear (people will send as large files as the system will let them; and yes, there is something like FUP with most ISPs).

TL;DR: while it would be technically possible to do such things as e-mailing a 1GB file, it would also be technically possible to pound in a nail using a screwdriver - it's just not a good way to do it, as there are tools that are more suitable for such tasks.

Piskvor left the building
  • 2,093
  • 1
  • 17
  • 17
33

The same but from a slightly different view:

Email is electronic mail. You know mail as this ancient paper thingy in another little paper envelope. You could write a lot of text on it but not very more than 5 or 6 pages. And email is the same but electronic. It is designed for text (plain text like on a typewriter). Then there was a MIME extension where you could send these fancy red-blinking HTML mails.

No one in the world would complain and say "Oh mail stuck in the way it was at 1322 AD. Why can't I send an elephant in a paper envelope?" It is how it is. For this kind of stuff people invented something like a packet or a transport container. This is how to send goods and elephants. And the Internet guys invented FTP (file transfer protocol), got the name?

In the real world there are many alternatives to FTP because FTP is also an ancient protocol with big disadvantages (mostly in security and not in transferring files). But HTTP is not an alternative as it was developed for centralized document storage with meta data. That you can download and upload files is a brutal extension to it.

So use a letter to send text and a packet to send goods; use email to send information and file transport protocols to send files.

Edit:

To stay in the picture I have to add: Even if you convince your local post office to accept elephants in paper envelops (and pay the additional fee) there are more parties involved delivering the elephant. There is the mailman who has to carry it to the next post office and probably he has not the right bag for the elephant to fit in. But maybe he has and wants to deliver it to the next post office which in turn says: "No we don't accept elephants".

What to do then? The good mailman in the real world would carry the elephant back to the first post office -- back to sender afterwards. (In the electronic world this would be a bad mailman because he should have shot the elephant and only deliver the certificate of death back to the sender).

So even if you could convince all links in the chain of post delivery to accept elephants you are faced with the recipient. He could say that he wants the elephant but the letterbox is too small for an elephant to fit in. Leading to a return-to-sender elephant delivery. (Not to mention what happens if the elephant doesn't fit in the sender's letterbox...)

mailq
  • 16,882
  • 2
  • 36
  • 66
  • 18
    Come *on*! As long as there's the `Content-Type: application/x-pachyderm` header, HTTP is perfectly capable of transmitting elephants ;) Good points, although my protocol of choice would be `rsync` - reasonably well available, allows for compression, delta syncs, continued transfer, *plus* works well with SSH (for auth+encryption). – Piskvor left the building Aug 24 '11 at 13:01
  • 1
    Even a P2P approach is reasonable. It depends on the audience. Multicasting a file through email should ring everybody's alarm bell. And as you said in other words one shouldn't follow this approach: "If you only have a hammer then every problem looks like a nail". – mailq Aug 24 '11 at 14:46
  • Hmm, yes - for multiple intended recipients, e.g. torrents make a lot of sense; but in my experience, you still need a (FTP?HTTP?) fallback for those users who aren't savvy of all these newfangled transfer protocols. Depends on the audience, as you said. – Piskvor left the building Aug 24 '11 at 14:51
17

Having been in a situation with Exchange 2007 where management subscribed to the "no limit on email size" philosophy:

An internal user sent a message to their hotmail address with an .iso of a music CD. Jammed the queue on one transport server while processing the message, lit up back pressure, stopping message submission. The user's outlook then dutifully re-submitted the message to the other transport server that was functioning; back pressure, no message submission.

With both transport servers choking on the message, all outbound email halted for about 90 seconds. Hotmail, of course, rejected the message. There was a size limit in place very soon after.

Shane Madden
  • 112,982
  • 12
  • 174
  • 248
  • sometime in the 90s i received a 20MB email. the email was actually delivered into my mailbox, but the server sent back a 4.5.1 size error. at which point the sending server resent the message. creating a loop that kept repeating until my mailbox or our server was full and had to be manually fixed by the admin by removing the mail from the queue. – eMBee Aug 27 '19 at 04:02
5

Here is another view:

Since an email is stored in multiple instances along the way, sending a 1 GB file would use up several times that all the way.

It will usually be a copy on your client in "Sent items", and if using IMAP, a copy on the server might show up as well (on your account).

Then the receiving end will keep a copy (the server), as well at the local client on the receiving end. If using IMAP, then it will not be deleted on the server (once again).

That is a total of 4 GB for a single 1GB file. Of course, you can consider it a backup, but there are better options for that as well. Not to mention the slowness that might occur on the server because the users mailboxes grows indefinitely.

And I just realized that if the file is base64 encoded it will be even larger (approx 33% larger I guess).

jishi
  • 858
  • 1
  • 11
  • 25
4

To supplement Piskvor's answer.

Yes, the "main email platforms" are dragging their feet. They do this by using a protocol (SMTP) that is not up to today's standards (in many ways).

In today's world, we could easily design a protocol to replace SMTP that would solve the current attachment issue.

The problem would be getting the world to switch over to it.

user606723
  • 544
  • 1
  • 4
  • 10
  • Indeed. ["Email is the cockroach of communication mediums: you just can't kill it."](http://www.codinghorror.com/blog/2010/04/so-youd-like-to-send-some-email-through-code.html) – Piskvor left the building Aug 24 '11 at 15:31
-2

The problem mentioned are mostly logistic issues with storage and transmission of data - in the modern cloud abstraction, a file no-longer need to be physical - a file-handle abstraction may be used to wrap around various storage methods (e.g. local disk, ftp, http, torrent, youtube, cloud storage, darknet, attachment, mule, distributed fs, excerpts, revisions) - this isn't a new idea, it's just not fully here or in one piece yet. when or if it arrives, your mail attachment would simply be a file pointer that can be used directly (e.g. not a .torrent file nor a link) by video players or whatever software. the actual handling of content download or storage would happen transparently, content may be partially located from multiple sources defined in collaboratively-revisable manifest (e.g. like a .torrent file but universally accepted, and with revisable availability and locality constraints); actual download and storage/caching may often be partial, depending on which part was viewed and if you've even bothered to access the content - so your mother-in-law's huge attachment wouldn't eat up any of your to-house bandwidth if you are not a fan of her emails. For permanence or availability, maybe you'll have a mail client that can filter and revise the storage manifest that then result in relocation of certain unopened attachment from its sources to your cloud storage when its availability dwindles, for example

Alife Toy
  • 1
  • 1
  • 2
    As much as I hate "cloud" terminology, you're description is half true; but there are still transfer requirements (bandwidth), storage even if it's only intermediate, and significant delay could be induced by the lack of presence on a "local" server. Even if the file is never accessed by the recipient, the original sender still has to transmit the whole file "to the cloud", the "cloud" has to hold the whole file (perhaps indefinitely), and the structures to communicate all this would have to be put in place. If we're reinventing the wheel, it could be done better than this. – Chris S Aug 24 '11 at 20:38
  • 1
    "a file no-longer need to be physical" - hold on while I get rid of my disks, as they're no longer needed for those *spiritual* files ;) You do have a good point that the storage and transfer can be abstracted away, but they're merely hidden *somewhere* by the abstraction, not gone. You'd need a *really* fat pipe to alleviate the file access latency - e.g. start playing a HD video, seek to the middle, twiddle your thumbs for minutes while the requested data is streamed to you (as opposed to mere milliseconds for local storage). And there's also the one-foot-per-nanosecond pesky speed of light. – Piskvor left the building Aug 25 '11 at 08:56
  • true - this could all get pretty slow if locality or availability isn't defined or implemented well. but user gets to define them and take some responsibility for various trade-offs of speed, bandwidth, availability, so on, using prepackaged policies, filter rules, attributes, tags, inferencing rules. when I send an email with attachment, I don't need to 'upload' them, as the data is simply made available to the receiver, then the data move or replicate itself to disks and/or cloud storage based on behavior of two parties' storage managers' user-configured inference rules. – Alife Toy Aug 25 '11 at 09:37
  • "user gets to define them and take some responsibility" - You must be a manager. – Chris S Aug 25 '11 at 15:44
  • not manager but something much worse - a broken futurist – Alife Toy Aug 25 '11 at 17:58