This project is pretty awesome... though I'm not clear on what happens after you are setup.
Does mail-in-a-box then provide the scripts to perform regular software updates and any configuration migrations between versions? There's more to running software than the initial setup... A complete Mail solution in a Unix-like environment consists of a lot of disconnected programs with their own configurations that are difficult to get running, and even more difficult to maintain without a full time systems administrator in place.
I mentioned in another discussion recently how much I would love to see something akin to SmarterMail available as a simple package install Cross-Platform (one of the best mail server softwares out there imho, from a setup/upgrade POV) but commercial and tied to Windows for deployments... If I didn't have to work for a living, I'd probably start something like this. Mail services are usually made far more complicated than they should be, and I understand there are a lot of desired features... but I do feel that having a good module/plugin system that one could be developed that isn't the pain that current solutions are.
To me a current mail solution should provide, SMTP, POP3, IMAP, WebMail, and WebAdmin at a minimum... Value adds would be easy multi-domain support, easy to configure AV/Spam plugins, Calendars + Sharing and Group Contact Sharing. Honestly, the only solutions with a relatively easy setup for this are for Windows... All the nix solutions are cobbled together bits that are very hard to upgrade and maintain versions and require a lot more breadth of knowledge than a single product. I've tried many of the systems for nix and they mostly suck in practice.. some more than others.
Mail services are usually made far more complicated than they should be, and I understand there are a lot of desired features...
That was my impression of the whole thing, too. I've long had an exim configuration I could decorate my walls with, without understanding what most of it did or if it was secure.
Recently, I got so fed up I began writing my own mail server suite. It's still pretty basic and in development, but it does have some of the features you mention, namely
SMTP
POP3
WebMail (though rudimentary)
WebAdmin
Multi-Domain support
In the pipeline, but not yet ready
IMAP
Plugins
Some of the goals of the project are to have a mail processing suite with a clear interface between the modules, as well as easy extensibility and configuration.
Me and some people I've talked into testing it already run some instances, and so far it has proved pretty stable.
Caveat: The backend is an SQLite database, so if your use-case is serving a lot of clients, there might be some lock contention.
If you separate each account's folders/inbox into a separate sqlite database, with the accounts/configuration in another, it probably wouldn't be to bad at even moderate scale.
But for < 100 users one sqlite db would probably be sufficient. (on a relatively fast drive/ssd)
That's actually exactly what you can already do with cmail ;)
Each user can optionally be assigned a "user database", storing only the mails in her own inbox (which also allows users to have direct control over their own mail database).
If this is not used, mail is stored in the master database.
As you said, most normal deployments should not run into those limits, but its worth keeping them in mind.
Why not use Maildir, or really any other standard scalable mail storage format? The most annoying thing in the universe is trying to export mail from one client to another with incompatible formats. Well, okay, maybe Vogon poetry is worse, but e-mail is a close second.
SQLite has a pretty great C API making it really easy to programmatically store, modify and retrieve data, which made it my first choice for runtime configuration data.
Adding another storage backend for the mail data would have meant a lot more code paths to support.
SQLite also has a lot of tools surrounding and supporting it (I particularly like SQLiteStudio) which allows querying/modifying/analyzing your mails with a huge number of programs, and using SQL - something I've found really useful!
With a sufficiently sane schema (which I hope we've achieved), transforming from the database format to any other format becomes really easy!
Also, since cmail is not really being developed with the intent to have people access their mail with a shell account (the main reason being that cmail users should not have to map to system users), supporting access formats on the server side is less of a priority - preferred access is via POP (already available) and IMAP (once it's done). The user database feature allows cmail users that ARE system users to control their own database.
An exporter to convert to/from Maildir is a planned feature :)
Hear-hear. Aside from a big directory of ascii text files, I can't think of a more future proof method of storage. So sqlite has that going for it, in addition to the extension API - which is awesome. I recently had to write some integration software for Kronos and some no name security system, as I was already storing the data in sqlite I decided to handle the Kronos business logic with a loadable extension - it was a pretty pleasant experience. Actually, I'm pretty sure there is an extension out there that allows you to use a directory of CSV files for table storage...
I've been thinking of using one of their schemas as a base for doing something similiar for an email client/interactive mail archive type thing (think: hyperkitty/pipermail).
Also notmuch has done some work for mapping email to xapian for search/tagging/indexing -- could also be a good source of inspiration I think.
I was aware of dbmail, but did not peruse the source in-depth, in part because it does not in itself implement SMTP.
Archiveopteryx I did not know, but thanks for the tip!
Using their databases as the basis for new tools has the obvious benefits of cross-compatibility, though there's always the drawback of database cruft that accumulates by virtue of differences in implementation or project goals.
I'm curious about all these comments about how much time it takes to maintain these systems. I've run a Debian+Courier+Spamassassin+Roundcube setup for over a decade for friends and family, and the maintenance required has been quite minimal. I have a script that emails me when security updates are available, and apt-get upgrade just takes a few seconds. Every couple of years I have to upgrade the Debian version, but that rarely takes more than a couple of hours. Outside of that, it just runs and runs.
Having said that, I am comfortable doing all that Linux sysadmin stuff, so maybe that's the difference. And if I were offering email accounts to people I didn't know, that would be an entirely different maintenance challenge.
Thanks... I ran my own mail server for a number of years, but with only a handful of accounts, it finally became far easier to just outsource... I've been thinking of doing something like mail-in-a-box on a cloud/vps host, but the time/frustration in maintaining such a thing has kept me away.
Does mail-in-a-box then provide the scripts to perform regular software updates and any configuration migrations between versions? There's more to running software than the initial setup... A complete Mail solution in a Unix-like environment consists of a lot of disconnected programs with their own configurations that are difficult to get running, and even more difficult to maintain without a full time systems administrator in place.
If so, I'd be extremely tempted to switch over. I already use Postfix/Dovecot/Roundcube, but Roundcube is currently broken because of conf changes between upgrades and Dovecot has been _horribly_ broken (taken a week+ to sort out) by a big configuration change on their side a couple of years ago :\
Comments
This project is pretty awesome... though I'm not clear on what happens after you are setup.
Does mail-in-a-box then provide the scripts to perform regular software updates and any configuration migrations between versions? There's more to running software than the initial setup... A complete Mail solution in a Unix-like environment consists of a lot of disconnected programs with their own configurations that are difficult to get running, and even more difficult to maintain without a full time systems administrator in place.
I mentioned in another discussion recently how much I would love to see something akin to SmarterMail available as a simple package install Cross-Platform (one of the best mail server softwares out there imho, from a setup/upgrade POV) but commercial and tied to Windows for deployments... If I didn't have to work for a living, I'd probably start something like this. Mail services are usually made far more complicated than they should be, and I understand there are a lot of desired features... but I do feel that having a good module/plugin system that one could be developed that isn't the pain that current solutions are.
To me a current mail solution should provide, SMTP, POP3, IMAP, WebMail, and WebAdmin at a minimum... Value adds would be easy multi-domain support, easy to configure AV/Spam plugins, Calendars + Sharing and Group Contact Sharing. Honestly, the only solutions with a relatively easy setup for this are for Windows... All the nix solutions are cobbled together bits that are very hard to upgrade and maintain versions and require a lot more breadth of knowledge than a single product. I've tried many of the systems for nix and they mostly suck in practice.. some more than others.
That was my impression of the whole thing, too. I've long had an exim configuration I could decorate my walls with, without understanding what most of it did or if it was secure.
Recently, I got so fed up I began writing my own mail server suite. It's still pretty basic and in development, but it does have some of the features you mention, namely
In the pipeline, but not yet ready
Some of the goals of the project are to have a mail processing suite with a clear interface between the modules, as well as easy extensibility and configuration.
Me and some people I've talked into testing it already run some instances, and so far it has proved pretty stable.
Caveat: The backend is an SQLite database, so if your use-case is serving a lot of clients, there might be some lock contention.
If you're interested, check out https://github.com/cmail-mta / http://cmail.rocks/
Wow, exactly what I was looking for. Keep up the good work! I will follow the progress, as I am interested in the upcoming IMAP support.
If you separate each account's folders/inbox into a separate sqlite database, with the accounts/configuration in another, it probably wouldn't be to bad at even moderate scale.
But for < 100 users one sqlite db would probably be sufficient. (on a relatively fast drive/ssd)
That's actually exactly what you can already do with cmail ;)
Each user can optionally be assigned a "user database", storing only the mails in her own inbox (which also allows users to have direct control over their own mail database).
If this is not used, mail is stored in the master database.
As you said, most normal deployments should not run into those limits, but its worth keeping them in mind.
Why not use Maildir, or really any other standard scalable mail storage format? The most annoying thing in the universe is trying to export mail from one client to another with incompatible formats. Well, okay, maybe Vogon poetry is worse, but e-mail is a close second.
Well one reason being I really like SQLite.
SQLite has a pretty great C API making it really easy to programmatically store, modify and retrieve data, which made it my first choice for runtime configuration data. Adding another storage backend for the mail data would have meant a lot more code paths to support.
SQLite also has a lot of tools surrounding and supporting it (I particularly like SQLiteStudio) which allows querying/modifying/analyzing your mails with a huge number of programs, and using SQL - something I've found really useful!
With a sufficiently sane schema (which I hope we've achieved), transforming from the database format to any other format becomes really easy!
Also, since cmail is not really being developed with the intent to have people access their mail with a shell account (the main reason being that cmail users should not have to map to system users), supporting access formats on the server side is less of a priority - preferred access is via POP (already available) and IMAP (once it's done). The user database feature allows cmail users that ARE system users to control their own database.
An exporter to convert to/from Maildir is a planned feature :)
Hear-hear. Aside from a big directory of ascii text files, I can't think of a more future proof method of storage. So sqlite has that going for it, in addition to the extension API - which is awesome. I recently had to write some integration software for Kronos and some no name security system, as I was already storing the data in sqlite I decided to handle the Kronos business logic with a loadable extension - it was a pretty pleasant experience. Actually, I'm pretty sure there is an extension out there that allows you to use a directory of CSV files for table storage...
Did you have a look at http://dbmail.org/ ? And/or http://aox.org/ (Archiveopteryx) ?
I've been thinking of using one of their schemas as a base for doing something similiar for an email client/interactive mail archive type thing (think: hyperkitty/pipermail).
Also notmuch has done some work for mapping email to xapian for search/tagging/indexing -- could also be a good source of inspiration I think.
I was aware of dbmail, but did not peruse the source in-depth, in part because it does not in itself implement SMTP.
Archiveopteryx I did not know, but thanks for the tip!
Using their databases as the basis for new tools has the obvious benefits of cross-compatibility, though there's always the drawback of database cruft that accumulates by virtue of differences in implementation or project goals.
I'm curious about all these comments about how much time it takes to maintain these systems. I've run a Debian+Courier+Spamassassin+Roundcube setup for over a decade for friends and family, and the maintenance required has been quite minimal. I have a script that emails me when security updates are available, and apt-get upgrade just takes a few seconds. Every couple of years I have to upgrade the Debian version, but that rarely takes more than a couple of hours. Outside of that, it just runs and runs.
Having said that, I am comfortable doing all that Linux sysadmin stuff, so maybe that's the difference. And if I were offering email accounts to people I didn't know, that would be an entirely different maintenance challenge.
Yes it does.
Thanks... I ran my own mail server for a number of years, but with only a handful of accounts, it finally became far easier to just outsource... I've been thinking of doing something like mail-in-a-box on a cloud/vps host, but the time/frustration in maintaining such a thing has kept me away.
If so, I'd be extremely tempted to switch over. I already use Postfix/Dovecot/Roundcube, but Roundcube is currently broken because of conf changes between upgrades and Dovecot has been _horribly_ broken (taken a week+ to sort out) by a big configuration change on their side a couple of years ago :\