That Courier FUD article does not list a single specific complaint with the specification (well, other than the version number ;P):
Um, that version number issue is pretty significant I think, and it really calls into question the competence of the IMAP committee at the most basic level:
In 2003, Appendix B of RFC 3501 identified over a hundred corrections to RFC 2060. Observe that RFC 3501 does not define a new version of the IMAP protocol. It's the same version. This is a “revised” document, which attempts to clarify and address numerous issues and inaccuracies in the original specification (some of which I noted publicly).
And, to make things even more interesting, as of this date there are already several reported errata to 3501. Furthermore, RFC 3501 actually changed the IMAP protocol, adding several new requirements, but it kept the IMAP4rev1 version. In other words: the protocol has changed, but it's still the same protocol, officially. Figure that one out.
Isn't updating version numbers when you change the protocol the most basic job of any standards organization? Is this really the sort of thing that we should be failing at?
If you still insist on using an inferior server, the result will be weird and will be, well, inferior ;P, but at least it will work.
All compliant client implementations have to be designed to deal with these "inferior servers" because they still exist. So client implementers cannot rely on usable persistent UIDs (if they want to be standards compliant, which is apparently very important to Crispin). And I'll note that you still haven't given me an explanation for why this feature exists in the spec at all: was the IMAP team really so shortsighted that they believed that no one would ever make a real IMAP server datastore?
that doesn't make it an accurate description of the state of affairs, nor certainly does it indicate anything about the design constraints of the protocol.
OK, what is inaccurate about it? And what are these mythical design constraints? Shouldn't they be written down somewhere?
the original IMAP protocol was highly asynchronous, and simply streamed protocol state to the client whenever things were updated so that the local client would have a live-updated display.
This seems insane to me. Live updates are all well and good, but why on Earth do we need live updates WHILE THE CLIENT HAS AN OUTSTANDING REQUEST?
Can you name other protocols that adopt this gloriously unstructured style? I mean, lots of protocols could in theory benefit from this 'ultra low latency' trick, right?
The result is that the protocol, to a very real extent, doesn't even have "command responses"
Then describing the protocol in terms of commands and command responses seems a bit absurd, right?
> Um, that version number issue is pretty significant I think, and it really calls into question the competence of the IMAP committee at the most basic level: ... Isn't updating version numbers when you change the protocol the most basic job of any standards organization? Is this really the sort of thing that we should be failing at?
Actually, no: these are all protocol drafts; the official RFC for HTTP/1.1 is RFC2616, but RFC2068 also documents a protocol that happens to be called HTTP/1.1. RFC2616 "is an update to RFC 2068". There were actually protocol-requirement changes (the addition of new status codes, changes to existing ones to fit what actually got implemented, modifications to compatibility modes for proxies, etc.) between RFC2068 and RFC2616: if you disagree with this practice, it is either the fault of using a "request for comment" as if it were standard or a bug in the IETF itself, not an issue specific to IMAP.
> All compliant client implementations have to be designed to deal with these "inferior servers" because they still exist.
Doing so is trivial, because if the UIDVALIDITY changes you simply delete all the data and redownload it as if it were a new folder; you would need this functionality anyway if the user deleted the folder and created a new one in its place: it just so happens that some servers are so inferior that their notion of the IMAP mail store is transient and in memory while the client is connected and is deleted when they disconnect. With such a server it is only reasonable to have a fully online client, and with the naive way to implement this that just works, easily, and for free.
> OK, what is inaccurate about it? And what are these mythical design constraints? Shouldn't they be written down somewhere?
They are, actually: mbox is an example mail store that existing systems had, and which IMAP should be able to support as a protocol. In essence, any server that previously had been supporting only POP--or even hell: a POP server being proxied to clients as if it were IMAP--isn't going to have anywhere to store UIDs lying around, and so is going to be forced to simulate the entire notion of IMAP transiently in memory. A good client will work against this server, and it doesn't require you to code specially around it or anything: it will just be inefficient, as it will delete the data and redownload it often.
IMAP simply does not standardize the mail store: it is more similar to a generic protocol like HTTP. To demonstrate, HTTP doesn't require the server have a specific filesystem to store the files that it serves; if the server's filesystem doesn't have the a concept of "last modified date", then it isn't going to be able to return a Last-Modified header; if it doesn't do so, the client's caching algorithms are going to suck, and it is going to end up redownloading files many more times than it otherwise should have to... this is actually the exact same behavior that will happen with broken UIDVALIDITY.
This really isn't a protocol issue: as I said, the protocol needs this feature even if the server works because of the concept of folder names. Yes: you could replace folder names with folder UIDs, but that's just equivalent to concatenating the folder name with the UIDVALIDITY and calling that the UID. If the UIDVALIDITY changes, it is a new folder. The guy who wrote that imap.rb really really wanted to support a good feature (stable folders) on a server that was fundamentally incapable of it with any protocol: that's not IMAP's fault.
> This seems insane to me. Live updates are all well and good, but why on Earth do we need live updates WHILE THE CLIENT HAS AN OUTSTANDING REQUEST?
Yes: IRC (RFC1459). With IRC you can send commands and get responses, and you can send a bunch of commands and get a bunch of responses. Interleaved with those responses (which from your perspective is going to look like "in the middle of a request/response") you will also be getting incoming private messages, because that's just what the protocol does: it streams data to you while also accepting commands and providing responses. You could even argue that that makes IRC two separate protocols combined on a single socket, which is actually how Mark Crispin described IMAP.
"""IMAP is two protocols; a command/response protocol that is client initiated (section 6, and tagged 7.1.1 - 7.1.3), and a data-transmission protocol that is server initiated (section 7).""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
> Then describing the protocol in terms of commands and command responses seems a bit absurd, right?
Yes, it is; in fact, I believe (but could never say for certain, and am finding this exceedingly awkward that I am needing to defend a dying guy I don't even know against silly tiny complaints that could easily have waited a little while longer) that Mark agrees with you: he at least explicitly said as much on the imap-protocol mailing list, and his disappointment with how he was forced to change the way the spec was written into that "absurd" way of description was a compromise to get buy-in. It is actually one of the reasons he asks people to go back and read IMAP2.
"""This is one of the reasons why I did NOT want to say "such-and-such response occurs as a result of such-and-such command" (I was forced to do so against my wishes); I knew that people would falsely presume that
stating such implies that such-and-such response only occurs as a result of such-and-such command. The fact that the text says "this response occurs as a result of that command" does NOT mean that the response can not occur at another time.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
Comments
That Courier FUD article does not list a single specific complaint with the specification (well, other than the version number ;P):
Um, that version number issue is pretty significant I think, and it really calls into question the competence of the IMAP committee at the most basic level:
In 2003, Appendix B of RFC 3501 identified over a hundred corrections to RFC 2060. Observe that RFC 3501 does not define a new version of the IMAP protocol. It's the same version. This is a “revised” document, which attempts to clarify and address numerous issues and inaccuracies in the original specification (some of which I noted publicly).
And, to make things even more interesting, as of this date there are already several reported errata to 3501. Furthermore, RFC 3501 actually changed the IMAP protocol, adding several new requirements, but it kept the IMAP4rev1 version. In other words: the protocol has changed, but it's still the same protocol, officially. Figure that one out.
Isn't updating version numbers when you change the protocol the most basic job of any standards organization? Is this really the sort of thing that we should be failing at?
If you still insist on using an inferior server, the result will be weird and will be, well, inferior ;P, but at least it will work.
All compliant client implementations have to be designed to deal with these "inferior servers" because they still exist. So client implementers cannot rely on usable persistent UIDs (if they want to be standards compliant, which is apparently very important to Crispin). And I'll note that you still haven't given me an explanation for why this feature exists in the spec at all: was the IMAP team really so shortsighted that they believed that no one would ever make a real IMAP server datastore?
that doesn't make it an accurate description of the state of affairs, nor certainly does it indicate anything about the design constraints of the protocol.
OK, what is inaccurate about it? And what are these mythical design constraints? Shouldn't they be written down somewhere?
the original IMAP protocol was highly asynchronous, and simply streamed protocol state to the client whenever things were updated so that the local client would have a live-updated display.
This seems insane to me. Live updates are all well and good, but why on Earth do we need live updates WHILE THE CLIENT HAS AN OUTSTANDING REQUEST?
Can you name other protocols that adopt this gloriously unstructured style? I mean, lots of protocols could in theory benefit from this 'ultra low latency' trick, right?
The result is that the protocol, to a very real extent, doesn't even have "command responses"
Then describing the protocol in terms of commands and command responses seems a bit absurd, right?
> Um, that version number issue is pretty significant I think, and it really calls into question the competence of the IMAP committee at the most basic level: ... Isn't updating version numbers when you change the protocol the most basic job of any standards organization? Is this really the sort of thing that we should be failing at?
Actually, no: these are all protocol drafts; the official RFC for HTTP/1.1 is RFC2616, but RFC2068 also documents a protocol that happens to be called HTTP/1.1. RFC2616 "is an update to RFC 2068". There were actually protocol-requirement changes (the addition of new status codes, changes to existing ones to fit what actually got implemented, modifications to compatibility modes for proxies, etc.) between RFC2068 and RFC2616: if you disagree with this practice, it is either the fault of using a "request for comment" as if it were standard or a bug in the IETF itself, not an issue specific to IMAP.
> All compliant client implementations have to be designed to deal with these "inferior servers" because they still exist.
Doing so is trivial, because if the UIDVALIDITY changes you simply delete all the data and redownload it as if it were a new folder; you would need this functionality anyway if the user deleted the folder and created a new one in its place: it just so happens that some servers are so inferior that their notion of the IMAP mail store is transient and in memory while the client is connected and is deleted when they disconnect. With such a server it is only reasonable to have a fully online client, and with the naive way to implement this that just works, easily, and for free.
> OK, what is inaccurate about it? And what are these mythical design constraints? Shouldn't they be written down somewhere?
They are, actually: mbox is an example mail store that existing systems had, and which IMAP should be able to support as a protocol. In essence, any server that previously had been supporting only POP--or even hell: a POP server being proxied to clients as if it were IMAP--isn't going to have anywhere to store UIDs lying around, and so is going to be forced to simulate the entire notion of IMAP transiently in memory. A good client will work against this server, and it doesn't require you to code specially around it or anything: it will just be inefficient, as it will delete the data and redownload it often.
IMAP simply does not standardize the mail store: it is more similar to a generic protocol like HTTP. To demonstrate, HTTP doesn't require the server have a specific filesystem to store the files that it serves; if the server's filesystem doesn't have the a concept of "last modified date", then it isn't going to be able to return a Last-Modified header; if it doesn't do so, the client's caching algorithms are going to suck, and it is going to end up redownloading files many more times than it otherwise should have to... this is actually the exact same behavior that will happen with broken UIDVALIDITY.
This really isn't a protocol issue: as I said, the protocol needs this feature even if the server works because of the concept of folder names. Yes: you could replace folder names with folder UIDs, but that's just equivalent to concatenating the folder name with the UIDVALIDITY and calling that the UID. If the UIDVALIDITY changes, it is a new folder. The guy who wrote that imap.rb really really wanted to support a good feature (stable folders) on a server that was fundamentally incapable of it with any protocol: that's not IMAP's fault.
> This seems insane to me. Live updates are all well and good, but why on Earth do we need live updates WHILE THE CLIENT HAS AN OUTSTANDING REQUEST?
Yes: IRC (RFC1459). With IRC you can send commands and get responses, and you can send a bunch of commands and get a bunch of responses. Interleaved with those responses (which from your perspective is going to look like "in the middle of a request/response") you will also be getting incoming private messages, because that's just what the protocol does: it streams data to you while also accepting commands and providing responses. You could even argue that that makes IRC two separate protocols combined on a single socket, which is actually how Mark Crispin described IMAP.
"""IMAP is two protocols; a command/response protocol that is client initiated (section 6, and tagged 7.1.1 - 7.1.3), and a data-transmission protocol that is server initiated (section 7).""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
> Then describing the protocol in terms of commands and command responses seems a bit absurd, right?
Yes, it is; in fact, I believe (but could never say for certain, and am finding this exceedingly awkward that I am needing to defend a dying guy I don't even know against silly tiny complaints that could easily have waited a little while longer) that Mark agrees with you: he at least explicitly said as much on the imap-protocol mailing list, and his disappointment with how he was forced to change the way the spec was written into that "absurd" way of description was a compromise to get buy-in. It is actually one of the reasons he asks people to go back and read IMAP2.
"""This is one of the reasons why I did NOT want to say "such-and-such response occurs as a result of such-and-such command" (I was forced to do so against my wishes); I knew that people would falsely presume that stating such implies that such-and-such response only occurs as a result of such-and-such command. The fact that the text says "this response occurs as a result of that command" does NOT mean that the response can not occur at another time.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...