I had this problem repeatedly with both Google and Microsoft, though never at the same time and never, so far as I could tell, for the same reason.
Everything was set up properly. The domains I hosted had IPv4 and IPv6 addresses with forward and reverse DNS matching exactly. SPF was set perfectly, according to multiple test sites. DKIM was always used and always validated, again according to multiple test sites. I even signed up for a DMARC endpoint account at report-uri.com.
Didn't matter. Messages would be spontaneously rejected, sometimes in mid-conversation (this happened to my wife quite a lot). Other times, the remote end would claim to accept the message and then it would disappear into the great bit bucket in the sky. It would be like this for a day or two then spontaneously go back to working.
I finally gave up and switched entirely over to Fastmail awhile back. No more delivery problems, but no more self-hosting my own e-mail, something I'd done for twenty years prior (my domain is older than Mozilla and Google).
Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms? Sure, you can put up the server, but most people are on one of the big providers, and chances are they've got your IP range blocked, because nearly every IP range you might be using has probably been owned by spammers at some point.
Email was actually one of the first aspects of the "classic" Internet to fall to the gentrification of the tech monopolies, long before the web in general. In theory, it's an open ecosystem anyone can jump into. In reality, unless you have a lot of energy/time/money to get around the big providers banning you for "spam," it's not going to happen.
To be fair, I was also surprised to read that anyone was still using Hotmail, so maybe I just don't have my finger on the pulse of the hip new generation.
FWIW, not for me either. I run mail servers for a small(ish) organisation.
In fact, I'd say it's getting easier, not harder. There was a time when fairly arbitrary rules were applied (like "thou shall not send email from a dialup IP Address). They seem to have mostly gone now, and when they are still in place you can just self report via a web form you are not being naughty and it's all good.
But you do have to keep your nose clean. Do not send SPAM, under any circumstances. Sounds easy, and indeed it is easy if you are just running a mail server for an organisation where all mail you are sending is generated by people within that organisation. But if you are doing something that involves relaying emails, eg something like running a mailing list, it becomes harder.
To give an example - I run a mail server for a club of maybe 20 people. All email is strictly policed, eg you have to sign up, all sign ups are reviewed by a human, including any email asking to sign up. All good right? Nope, because the club has to have at least one email address (say for the secretary) that does not involve signing up. Those emails are all moderated of course - but you are notified of emails waiting for moderation via email. Those notifications contain snippets of the original. If one of the moderators has a GMail account, those snippets triggering SPAM filters can get your server banned by google. The solution is to run a imap server for secretary@ and get them to fetch it manually. Bypass Google for all relayed email in other words.
So you have to follow the rules and it's not always obvious what you have to do to follow the rules, but it ain't impossible, and in fact once you know what the rules are it isn't even hard. Any capable sysadmin could do it.
Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms?
Well I've been self hosting my emails for 10 years now, and my emails still go through. I'm sure there must be cases where you face issues, but not my in my personal experience.
Only thing I've seen once or twice are websites with "validation" rules like /.@(gmail|hotmail|yahoo)./
I have also been self-hosting my email for almost 20 years.
Until a few years ago I never had any problems, but during the last 3-4 years I had sporadic problems with messages that either were not delivered or they were delivered with a very large delay, e.g. a day, or they were delivered only if I resent them 3 or 4 times, at some of the large providers, like Gmail.
It still works fine most of the time, but when problems happen they are annoying because there is no way to know that something happened, unless I have an alternative way to ask the recipient, e.g. by phone.
In any case, now I always check that at least the final destination e-mail server has accepted the message, which should have been enough to guarantee the delivery, according to standards, but nonetheless this is not true any more.
Yep. For important emails, I tail the exim mainlog. But in six sigma cases, it just works or I get to know that my mails were not delivered days later.
It's still doable. I self host and have managed to switch IP's providers and so forth without getting into trouble. It is, however, critical that you make sure you don't get an IP that was recently used by a spammer, and it's a lot of work to get everything up and running. Warming up an IP is difficult when talking volumes that small.
Do you have any tips on warming up your IP? I’ve been going down the self-hosting rabbithole again after years and just set up a server. I’m about to set up a mail server in the coming weeks and checked that my IP is not on any blocklists, but my normal email traffic is very low. Should I automate sending an email to my accounts on major platforms every other day or something along those lines?
Send authenticated mails. Send an email or two to your own Hotmail/Gmail address each day (or at least regularly), and make sure to interact with the email (read/reply/forward/put in folder/etc.). Personally, I use Hotmail for reading my shopping lists.
If you end up in your own spam folder, "Not junk" is your friend. If you don't get through at all, do the escalation dance with Microsoft until the situation improves.
Once you reliably hit your own Hotmail inbox, start sending mails to A FEW others as well. Verify that they receive the mail, and ask them to add you to their trusted sender lists, press not junk, etc, and keep bugging Microsoft if they just discard your emails silently. Once you have a few people doing that, reputation should improve quickly.
I well remember the times when I set up sendmail and saw emails immediately sent in Netscape Mail instead waiting for minutes. This was using a 32kB dialup connection.
Was like 1997. Needless to say it was then sendmail struggling to get the mails over dialup. At least my single-threaded Netscape was responsive.
Worked until 1999 or so. Then spam became a thing and more and more mails were rejected and I folded my setup.
Lots of people self-host their email, it works fine. The worst problem I've encountered is graylisting delays. Sometimes Gmail will throw a fit, but that's true whether you're originating from @icloud.com or from @yourtinydomain.com.
A couple years ago I did some superficial analysis on the domains of the registration emails our customers were giving us. I don't remember the details, but IIRC only about 80% of them were connected to one of the big providers (google, apple, qq, microsoft, yahoo(!), etc).
Email was actually one of the first aspects of the "classic" Internet to fall to the gentrification of the tech monopolies, long before the web in general.
But isn't the federated nature of email the only thing which prevents the said monopolies from taking over email completely? If it can be proven that X giant is blocking mails from Y small email company then they're in lot of trouble at least in EU.
That's the reason chat and everything else is a lost cause; Can't send a message from Messenger to Telegram unless you use their bot API to build a specialized application[1] which is a downgrade from XMPP like interoperable protocol.
[1] A product from my previous startup which allowed communication between Messenger <-> Telegam <-> Viber <-> LINE for dating - https://hitstartup.com/about/#FindDate
Your are still allowed. I self-host personal mail in a VM and still can send mail to gmail.
The problem is that big players (Gmail/MS) care about an anti-spam false positives only when they affects a relatively large fraction of all messages. Small mail servers account for a small fraction of all messages, so big players don't care about them at all.
I will not be surprised if they have SLI/KPI which allows them reject by a mistake millions of non-spam messages daily because it is a small percent of all messages they process.
> Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms?
For some time seemingly legit mail from Azure Cloud Services also got flagged by Gmail. If that was the case -- I thought -- what changes do I have for my own hosted or non-major email provider? That is when I decided I couldnt win as a small startup and just went the managed GSuite route for email hosting.
That is when I decided I couldnt win as a small startup and just went the managed GSuite route for email hosting.
Do you realize people like you giving up is precisely why these evil corporations can get away with what they do? Together, we small providers are strong. But if some resistance fighters drop the struggle and join Dark Vader 's side, we are bound to loose.
Maybe we should start a campaign. Give a one-year notice to those big email provider, in partnership with smaller email providers, saying they have one year to comply with email standards or we'll start boycotting and promoting alternatives as a federation of free/open email providers?
We could start doing like the comment on OP does. Send a rejection notice with a link to some list of all the various blogs documenting how big providers refuse to follow the standards.
My other thought was to have my Postfix server reject all mail from Microsoft servers, with a message requesting the sender to use a different mail provider and pointing to this article.
I've often been tempted to do that. However, we would actively be hurting our own users who would be incapable to receive mails from such servers. We'd in fact be hurting the very principles of interoperability we're defending in the first place.
Adding an auto-reply or an email footer mentioning that their providers actively hurts the email ecosystem and linking to alternatives would be a good compromise in my view.
Along with a libre email provider federation, that could go a long way...
Some companies reject to communicate to you straight away if you use GMail and don't run your own company EMail. On the other hand some do the opposite and demand you use GMail.
After testing - looks like new sign-ups get a choice of @outlook.com or @hotmail.com (although users with previously available/legacy domains like @hotmail.co.uk retain them).
Apparently the service as a whole has been based on o365 since around 2015 according to wikipedia.
I knew about about but only found out about two years ago, and I'm not one who is bad with tech either. So how common of knowledge it is is debatable I would say.
I'm pretty sure 0% of MS's customers asked them to silently drop emails without either recteting the mail at sumission time or dropping it in the spam foler.
I finally gave up and switched entirely over to Fastmail awhile back. No more delivery problems, but no more self-hosting my own e-mail, something I'd done for twenty years prior (my domain is older than Mozilla and Google).
Can anyone speak to what a provider like Fastmail does behind the scenes that guarantees their delivery? Is it the threat to sue/kick up dirt/raise issues with the appropriate regulatory body if perfectly good emails don't go through? Merely having a seat at some sort of regulatory body or committee?
I'm self-hosting my mails. Microsoft expects you to send enough mails to consider your mails not as spam. One thing Fastmail does that people who self-host can't is: sending enough mails.
I don't know what other measure they may take but this alone probably goes a long way.
And yes, this is fucking frustrating. I'm not going to send thousands of mails per months just for the sake of it and don't know how I should do that without spamming. This is so fucked up.
This is definitely something I've never heard of, thanks for mentioning it -- I had no idea there was such a consideration (or even that it would be a good way to filter out scammers)
From what I can tell, there are two main reasons companies like Fastmail don't have these problems. The first is that good email providers spend a lot of time maintaining the reputation of the IP addresses they use and shut down bad actors as quickly as possible. The second is that the more people that use the service, the higher percentage of good quality emails being sent, reducing the effect of the occasional bad actor.
Comments
I had this problem repeatedly with both Google and Microsoft, though never at the same time and never, so far as I could tell, for the same reason.
Everything was set up properly. The domains I hosted had IPv4 and IPv6 addresses with forward and reverse DNS matching exactly. SPF was set perfectly, according to multiple test sites. DKIM was always used and always validated, again according to multiple test sites. I even signed up for a DMARC endpoint account at report-uri.com.
Didn't matter. Messages would be spontaneously rejected, sometimes in mid-conversation (this happened to my wife quite a lot). Other times, the remote end would claim to accept the message and then it would disappear into the great bit bucket in the sky. It would be like this for a day or two then spontaneously go back to working.
I finally gave up and switched entirely over to Fastmail awhile back. No more delivery problems, but no more self-hosting my own e-mail, something I'd done for twenty years prior (my domain is older than Mozilla and Google).
Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms? Sure, you can put up the server, but most people are on one of the big providers, and chances are they've got your IP range blocked, because nearly every IP range you might be using has probably been owned by spammers at some point.
Email was actually one of the first aspects of the "classic" Internet to fall to the gentrification of the tech monopolies, long before the web in general. In theory, it's an open ecosystem anyone can jump into. In reality, unless you have a lot of energy/time/money to get around the big providers banning you for "spam," it's not going to happen.
To be fair, I was also surprised to read that anyone was still using Hotmail, so maybe I just don't have my finger on the pulse of the hip new generation.
"Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms?"
Not for me and loads of my tiny and not so tiny customers. Just keep your nose clean and follow the rules. This is in the UK, elsewhere ymmv.
FWIW, not for me either. I run mail servers for a small(ish) organisation.
In fact, I'd say it's getting easier, not harder. There was a time when fairly arbitrary rules were applied (like "thou shall not send email from a dialup IP Address). They seem to have mostly gone now, and when they are still in place you can just self report via a web form you are not being naughty and it's all good.
But you do have to keep your nose clean. Do not send SPAM, under any circumstances. Sounds easy, and indeed it is easy if you are just running a mail server for an organisation where all mail you are sending is generated by people within that organisation. But if you are doing something that involves relaying emails, eg something like running a mailing list, it becomes harder.
To give an example - I run a mail server for a club of maybe 20 people. All email is strictly policed, eg you have to sign up, all sign ups are reviewed by a human, including any email asking to sign up. All good right? Nope, because the club has to have at least one email address (say for the secretary) that does not involve signing up. Those emails are all moderated of course - but you are notified of emails waiting for moderation via email. Those notifications contain snippets of the original. If one of the moderators has a GMail account, those snippets triggering SPAM filters can get your server banned by google. The solution is to run a imap server for secretary@ and get them to fetch it manually. Bypass Google for all relayed email in other words.
So you have to follow the rules and it's not always obvious what you have to do to follow the rules, but it ain't impossible, and in fact once you know what the rules are it isn't even hard. Any capable sysadmin could do it.
Well I've been self hosting my emails for 10 years now, and my emails still go through. I'm sure there must be cases where you face issues, but not my in my personal experience.
Only thing I've seen once or twice are websites with "validation" rules like /.@(gmail|hotmail|yahoo)./
I have also been self-hosting my email for almost 20 years.
Until a few years ago I never had any problems, but during the last 3-4 years I had sporadic problems with messages that either were not delivered or they were delivered with a very large delay, e.g. a day, or they were delivered only if I resent them 3 or 4 times, at some of the large providers, like Gmail.
It still works fine most of the time, but when problems happen they are annoying because there is no way to know that something happened, unless I have an alternative way to ask the recipient, e.g. by phone.
In any case, now I always check that at least the final destination e-mail server has accepted the message, which should have been enough to guarantee the delivery, according to standards, but nonetheless this is not true any more.
Yep. For important emails, I tail the exim mainlog. But in six sigma cases, it just works or I get to know that my mails were not delivered days later.
It's still doable. I self host and have managed to switch IP's providers and so forth without getting into trouble. It is, however, critical that you make sure you don't get an IP that was recently used by a spammer, and it's a lot of work to get everything up and running. Warming up an IP is difficult when talking volumes that small.
Do you have any tips on warming up your IP? I’ve been going down the self-hosting rabbithole again after years and just set up a server. I’m about to set up a mail server in the coming weeks and checked that my IP is not on any blocklists, but my normal email traffic is very low. Should I automate sending an email to my accounts on major platforms every other day or something along those lines?
Send authenticated mails. Send an email or two to your own Hotmail/Gmail address each day (or at least regularly), and make sure to interact with the email (read/reply/forward/put in folder/etc.). Personally, I use Hotmail for reading my shopping lists.
If you end up in your own spam folder, "Not junk" is your friend. If you don't get through at all, do the escalation dance with Microsoft until the situation improves.
Once you reliably hit your own Hotmail inbox, start sending mails to A FEW others as well. Verify that they receive the mail, and ask them to add you to their trusted sender lists, press not junk, etc, and keep bugging Microsoft if they just discard your emails silently. Once you have a few people doing that, reputation should improve quickly.
I well remember the times when I set up sendmail and saw emails immediately sent in Netscape Mail instead waiting for minutes. This was using a 32kB dialup connection.
Was like 1997. Needless to say it was then sendmail struggling to get the mails over dialup. At least my single-threaded Netscape was responsive.
Worked until 1999 or so. Then spam became a thing and more and more mails were rejected and I folded my setup.
Lots of people self-host their email, it works fine. The worst problem I've encountered is graylisting delays. Sometimes Gmail will throw a fit, but that's true whether you're originating from @icloud.com or from @yourtinydomain.com.
A couple years ago I did some superficial analysis on the domains of the registration emails our customers were giving us. I don't remember the details, but IIRC only about 80% of them were connected to one of the big providers (google, apple, qq, microsoft, yahoo(!), etc).
But isn't the federated nature of email the only thing which prevents the said monopolies from taking over email completely? If it can be proven that X giant is blocking mails from Y small email company then they're in lot of trouble at least in EU.
That's the reason chat and everything else is a lost cause; Can't send a message from Messenger to Telegram unless you use their bot API to build a specialized application[1] which is a downgrade from XMPP like interoperable protocol.
[1] A product from my previous startup which allowed communication between Messenger <-> Telegam <-> Viber <-> LINE for dating - https://hitstartup.com/about/#FindDate
Your are still allowed. I self-host personal mail in a VM and still can send mail to gmail.
The problem is that big players (Gmail/MS) care about an anti-spam false positives only when they affects a relatively large fraction of all messages. Small mail servers account for a small fraction of all messages, so big players don't care about them at all.
I will not be surprised if they have SLI/KPI which allows them reject by a mistake millions of non-spam messages daily because it is a small percent of all messages they process.
For some time seemingly legit mail from Azure Cloud Services also got flagged by Gmail. If that was the case -- I thought -- what changes do I have for my own hosted or non-major email provider? That is when I decided I couldnt win as a small startup and just went the managed GSuite route for email hosting.
Do you realize people like you giving up is precisely why these evil corporations can get away with what they do? Together, we small providers are strong. But if some resistance fighters drop the struggle and join Dark Vader 's side, we are bound to loose.
Maybe we should start a campaign. Give a one-year notice to those big email provider, in partnership with smaller email providers, saying they have one year to comply with email standards or we'll start boycotting and promoting alternatives as a federation of free/open email providers?
We could start doing like the comment on OP does. Send a rejection notice with a link to some list of all the various blogs documenting how big providers refuse to follow the standards.
I've often been tempted to do that. However, we would actively be hurting our own users who would be incapable to receive mails from such servers. We'd in fact be hurting the very principles of interoperability we're defending in the first place.
Adding an auto-reply or an email footer mentioning that their providers actively hurts the email ecosystem and linking to alternatives would be a good compromise in my view.
Along with a libre email provider federation, that could go a long way...
Some companies reject to communicate to you straight away if you use GMail and don't run your own company EMail. On the other hand some do the opposite and demand you use GMail.
After testing - looks like new sign-ups get a choice of @outlook.com or @hotmail.com (although users with previously available/legacy domains like @hotmail.co.uk retain them).
Apparently the service as a whole has been based on o365 since around 2015 according to wikipedia.
I knew about about but only found out about two years ago, and I'm not one who is bad with tech either. So how common of knowledge it is is debatable I would say.
Is big evil tech at fault though? I'm sure the vast majority of users asked Microsoft and Google to do something about spam.
Iirc at its height 90% of all email was spam.
I'm pretty sure 0% of MS's customers asked them to silently drop emails without either recteting the mail at sumission time or dropping it in the spam foler.
I spotted someone in the wild using a netzero email address this month as their day-to-day.
Can anyone speak to what a provider like Fastmail does behind the scenes that guarantees their delivery? Is it the threat to sue/kick up dirt/raise issues with the appropriate regulatory body if perfectly good emails don't go through? Merely having a seat at some sort of regulatory body or committee?
I'm self-hosting my mails. Microsoft expects you to send enough mails to consider your mails not as spam. One thing Fastmail does that people who self-host can't is: sending enough mails.
I don't know what other measure they may take but this alone probably goes a long way.
And yes, this is fucking frustrating. I'm not going to send thousands of mails per months just for the sake of it and don't know how I should do that without spamming. This is so fucked up.
This is definitely something I've never heard of, thanks for mentioning it -- I had no idea there was such a consideration (or even that it would be a good way to filter out scammers)
From what I can tell, there are two main reasons companies like Fastmail don't have these problems. The first is that good email providers spend a lot of time maintaining the reputation of the IP addresses they use and shut down bad actors as quickly as possible. The second is that the more people that use the service, the higher percentage of good quality emails being sent, reducing the effect of the occasional bad actor.
I think it's likely that they directly share an allow list of each other's server to form some sort of network of email providers.