This is a collection of the author's favourite technologies/gripes, not a useful proposal. A large number of errors can be found by considering that it'll have to talk SMTP on the back end.
Encoding everything as UTF-8 is nice, but breaks the first time some idiot mail client puts Shift-JIS encoded Japanese in the Subject: field. Without any charset marker, obviously. (This goes right back to the "where does the mail come from?" issue.)
IMAP has a THREAD extension (solving #4), a SEARCH extension (solving #9 and parts of others), and all sensible servers support IDLE (http://en.wikipedia.org/wiki/IMAP_IDLE). It also supports downloading MIME parts separately, solving the part of #7 that can actually be solved (again, you'll have to accept mail.)
Thanks for your suggestions Joachim! This is why I wrote that blog entry in the first place: To get some high-quality feedback.
I think IMAP could be replaced independently of SMTP. You could conceivable replace an IMAP server with a reMAP server and not affect the rest of the mail ecosystem. SMTP will likely not die for another hundred years.
1. I know about the OAuth problems. Do you think we'll see a more secure solution in the next few years?
2. Is there an easy way to validate UTF-8ness? Happy to hear suggestions.
3. I know about THREAD and SEARCH and IDLE but find them clunky. Under the assumption that we're replacing IMAP with something RESTful you might as well get those ones right.
Happy to take this offline as well - my email is on my HP
First off, thanks for taking this well. It was a bit of a rant, even if it was toned down from the first version. Sorry.
We agree SMTP will be around forever; but how do you plan on abolishing MIME then? Your system will have to handle it, no matter what. Do you want to push MIME handling to the server? IMAP already does that, there's nothing revolutionary about that.
Due to the special structure, it's usually possible to distinguish UTF-8 from other 8-bit text. The tricky part is what to do with unmarked non-UTF-8 8-bit text - it could be pretty much anything. Mutt, for instance, allows the user to configure this according to what the user expects to see. Quoting the manual,
For example, Japanese users might prefer this:
set assumed_charset="iso-2022-jp:euc-jp:shift_jis:utf-8"
There is a lot more information about such things on the Mutt site. Basically, Mutt tries to parse the headers/message according to a specific character set and falls back to the next one if that fails.
I probably wouldn't trust my e-mail to a protocol that was only a few years old. I don't anticipate any new security protocols with significant uptake either; "serious" stuff already uses SSL/TLS plus an authentication method (passwords/certificates/SASL), and the web crowd is unlikely to replace that (OAuth is about weakening security in a controlled way, not replacing SSL.)
And yeah, parts of IMAP are clunky. But that's not enough reason to rip-and-replace it.
It's easy to validate UTF-8. The danger is if other implementations crop up that are more permissive and your users are complaining that they can't get mail from those people. I think you could probably manage to avoid that, but I don't know if it would really be preferable. I want to claim that in asia and europe there are some widely used non-unicode encodings used.
With regards to search, I recently installed Cyrus Squatter for full text IMAP searching and it is blazing fast. To me it seems just as fast as any of the special case local searches I've seen people build (Spotlight/recent Thunderbirds).
Any new mail storage protocol should anticipate the need to store encrypted messages at rest. Sure, it can play a part by storing an encrypted index on the server to deliver to authorized clients, but all decryption and index updates should take place client-side with the user's private key. Stable UID's can help make this a reality. There is no reason to store messages in plaintext on the server, and no new protocol should depend on it.
Comments
This is a collection of the author's favourite technologies/gripes, not a useful proposal. A large number of errors can be found by considering that it'll have to talk SMTP on the back end.
I'll only point to some "highlights".
OAuth isn't nearly secure enough (http://hueniverse.com/2009/04/explaining-the-oauth-session-f...) for your e-mail (which, don't forget, gives access to everything via password resets). Let's just ignore the part about third-party access.
Encoding everything as UTF-8 is nice, but breaks the first time some idiot mail client puts Shift-JIS encoded Japanese in the Subject: field. Without any charset marker, obviously. (This goes right back to the "where does the mail come from?" issue.)
IMAP has a THREAD extension (solving #4), a SEARCH extension (solving #9 and parts of others), and all sensible servers support IDLE (http://en.wikipedia.org/wiki/IMAP_IDLE). It also supports downloading MIME parts separately, solving the part of #7 that can actually be solved (again, you'll have to accept mail.)
Thanks for your suggestions Joachim! This is why I wrote that blog entry in the first place: To get some high-quality feedback.
I think IMAP could be replaced independently of SMTP. You could conceivable replace an IMAP server with a reMAP server and not affect the rest of the mail ecosystem. SMTP will likely not die for another hundred years.
1. I know about the OAuth problems. Do you think we'll see a more secure solution in the next few years?
2. Is there an easy way to validate UTF-8ness? Happy to hear suggestions.
3. I know about THREAD and SEARCH and IDLE but find them clunky. Under the assumption that we're replacing IMAP with something RESTful you might as well get those ones right.
Happy to take this offline as well - my email is on my HP
First off, thanks for taking this well. It was a bit of a rant, even if it was toned down from the first version. Sorry.
We agree SMTP will be around forever; but how do you plan on abolishing MIME then? Your system will have to handle it, no matter what. Do you want to push MIME handling to the server? IMAP already does that, there's nothing revolutionary about that.
Due to the special structure, it's usually possible to distinguish UTF-8 from other 8-bit text. The tricky part is what to do with unmarked non-UTF-8 8-bit text - it could be pretty much anything. Mutt, for instance, allows the user to configure this according to what the user expects to see. Quoting the manual,
There is a lot more information about such things on the Mutt site. Basically, Mutt tries to parse the headers/message according to a specific character set and falls back to the next one if that fails.I probably wouldn't trust my e-mail to a protocol that was only a few years old. I don't anticipate any new security protocols with significant uptake either; "serious" stuff already uses SSL/TLS plus an authentication method (passwords/certificates/SASL), and the web crowd is unlikely to replace that (OAuth is about weakening security in a controlled way, not replacing SSL.)
And yeah, parts of IMAP are clunky. But that's not enough reason to rip-and-replace it.
It's easy to validate UTF-8. The danger is if other implementations crop up that are more permissive and your users are complaining that they can't get mail from those people. I think you could probably manage to avoid that, but I don't know if it would really be preferable. I want to claim that in asia and europe there are some widely used non-unicode encodings used.
With regards to search, I recently installed Cyrus Squatter for full text IMAP searching and it is blazing fast. To me it seems just as fast as any of the special case local searches I've seen people build (Spotlight/recent Thunderbirds).
Any new mail storage protocol should anticipate the need to store encrypted messages at rest. Sure, it can play a part by storing an encrypted index on the server to deliver to authorized clients, but all decryption and index updates should take place client-side with the user's private key. Stable UID's can help make this a reality. There is no reason to store messages in plaintext on the server, and no new protocol should depend on it.
Message-IDs are already stable, though.