> I'm sure there is no malicious intent behind this
How can anyone be sure of this? This is only one of Google’s practices that seems to follow a pervasive pattern of eroding open internet standards while presenting Google’s own proprietary implementation as somehow superior. Eventually, the open standard loses all meaning because the most popular implementation does not actually adhere to it. Meanwhile, Google reaps enormous benefits in the form of additional signals for its advertising business. How can this not be grounded in malicious intent?
> and that there are some very smart people working on spam prevention at Google.
There are some very smart people working on advertising at Google. The rush to forget the primary nature of Google - it’s an adtech firm - is why they’ve been allowed to skate for so long. Gmail’s spam filtering is just a pretext for passing all email through a machine learning system. Sure, one possible signal emitted by that system is whether a message is spam or not. Perhaps this determination is conflated with wether the message is useful for ad targeting: after all, when viewed from Google’s own perspective certain e-mail messages contain no information which can be used for ad targeting, so they must be spam. The user’s interests are clearly secondary to this.
So, back to the “smart people” working on this: at what point do we begin judging engineers for working at Google? There’s a lot of highly vitriolic criticism that emanates from Google’s workforce on a variety of subjects, but how many of them would actually pull the pin and leave their employer? I don’t have any statistics to offer, but it seems to me that we still have a ways to go before Google has become completely drained of engineering mindshare.
I'm also one of those oldfarts that insists on hosting my private emails for me and a few friends. I've owned the ip since early 2000's and have had zero "spam"/blacklist incidents. One trend I've been noticing is that Microsoft/Outlook have regularly started to block me. I have to contact their abuse which takes a day or so to get unblocked. To date, I've done so >10 times:
hotmail-com.olc.protection.outlook.com[104.47.1.33] said: 550 5.7.1
Unfortunately, messages from [<redacted>] weren't sent. Please contact
your Internet service provider since part of their network is on our block
list (S3150). You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors.
[VE1EUR01FT028.eop-EUR01.prod.protection.outlook.com] (in reply to MAIL
FROM command)
Reporting-MTA: dns; <redacted>
X-Postfix-Queue-ID: 14A41FEB66
X-Postfix-Sender: rfc822; <redacted>
Arrival-Date: Thu, 14 Mar 2019 14:07:42 +0200 (CEST)As someone who runs their own personal mail server, this annoys me to no end.
I set up DKIM, SPF, and reverse-DNS records and resented every moment of it. Even after all that, there's some chance that an email from my server will be marked as insecure/spam or otherwise just not be delivered because Google has come up with some new brilliant mail security/auth/permission scheme that the world has to adopt tomorrow or be cut off from all Gmail users.
>I've been running a small SMTP and IMAP mail server for [...] around 15 years [...] I have SPF records and DKIM message signing setup on the domains I use. The server is hosted on commercial static IP space (with the very same IP it first went on-line) and I've made sure with the ISP that correct reverse DNS records are in place. [...] Being a good administrator and a well-behaved player on the network is no longer enough [...] Now every time I write a mail I wonder whether Google's AI will let it through or not. [...] So far Google has blocked personal messages to friends and family in multiple languages, as well as business mail. I stopped guessing what text their algorithms deem suspicious.
Yep, I know the author's frustration very well. I made a previous comment[0] trying to warn others of personal email servers' outgoing email being spam-holed -- and yet some of the replies still argued I was overstating the difficulties.
Everybody's risk tolerance is different. Personally, I just don't have the bandwidth to administer my own private email server and constantly worry if recipients are receiving my emails.
The last time a company screwed with e-mail at scale was Earthlink.
Remember for a couple of years, every time you'd send a message to someone at Earthlink, you'd get an automated rely demanding that you verify yourself before the message would go through?
Now I can't remember the last time I saw an @earthlink.* e-mail address.
Google has apparently learned from that and put the "error" in the 550 messages, where they can't be seen by the end user, and lead to non-helpful resolutions for sysadmins.
The result is that the blame for missing messages goes to the sender, not to the recipient's email service.
If Gmail at least notified the sender that there was a problem, then a pattern of responsibility could be established. But this is just another dark pattern.
I run a mail monitoring service [0], and we hear this complaint every now and then.
A couple of things that regularly seems to trigger false positives in spam algorithms:
- no or misconfigured SPF and/or DKIM
- no or misconfigured reverse-DNS
- automatically included footer texts (confidentiality, copyright, safe a tree don't print, etc)
- regular automatic replies from the domain (such as out of office notifications)
- the use of embedded images (logos, human signatures, etc)
We sometimes joke that these triggers were built in by the algorithm developers as a means of punishing those who litter their email with pointless texts and images.
Same here. I love email for a variety of reasons, one of them for being an open standard and protocol.
So, naturally, as many of you, I went the mail/postfix, DKIM, SPF, etc way. And all is fine until you start receiving random hard bounces with no real debugable answer for Google.
It got me deeply sad and questioning my decisions: since you can’t really ignore Gmail, email isn’t in practice “open” anymore. So I might as well sign up for Facebook, WhatsApp and the likes. It’s been years and I haven’t yet, but it’s getting harder and harder.
The real issue here is that if you tell some gmail user to contact their e-mail provider to resolve delivery issues on their side, like this, they'll just stare at you, not knowing what you're even talking about.
It's like there's an assumption that gmail is perfect, and the problem is with the sender. Even if that was true, a normal mail hosting company would at least tell its customer why the mail is not being delivered, so that the customer can tell the sender what to fix.
The gmail recipient is never exposed to this side of google. So they don't know what a nightmare comapny it is to communicate with.
Why should everyone and their dog be solving gmail users's problems with receiving messages? It's such a demented system. It should be the other way round. Recipients, via their provider should be solving their issues with spam filtering and blocking.
If the gmail user would be blocked by my mail server, I would not tell them to go guess what's wrong, fix gmail, and to have fun. It should not be acceptable the other way round either.
I'm in the same situation as many people here. Started as a small personal email projects many years ago just to learn. I really enjoyed it, reading all the specs and making it always better year after year. Dmarc felt like a total achievement.
I started to host emails to many friends, small businesses and even a SaaS I developed. The subscription needs an email validation and I'm aware that the activation email ends up in the Spam folder for the new customers using Google emails. This activation email has everything from dkim, spf, dmarc, to unsubscribe link, full physical address of the business, etc and still I can't hit a good enough score.
I was thinking to start using Google service to send the activation link and hosting my personal domains, but seeing that I am not alone, I will continue to improve my little email projects.
Thanks all for cheering me up on this. I'm sure we can come up with a solution and I would be happy to help. When do we start?
I have logwatch logs that I email to my gmail address via Google's own SMTP servers using my own Google credentials for authentication.
Hilariously, Google will flag these emails sent to myself using my own credentials and their infrastructure as spam. I have no faith in them ever getting this right.
Google seems to not agree DKIM is setup properly?
https://toolbox.googleapps.com/apps/checkmx/check?domain=tab...
Recently Google has bounced replies to emails from a gmail.com address to us, with a message about it being "suspected unsolicted email" (for a reply???). That's just egregious (we're on a dedicated server at a reputable supplier, same IP for years, proper email processess, not on any blacklist and only low-volume usage - certainly no spam).
This also mirrors my experience, as I host my own email and it's solely for me. At one point, I couldn't successfully send any email to gmail, but then it changed for some reason and simply gets sent to spam folders instead. Perhaps it's entirely related to some people emailing me first that it gets through at all.
I don't send much mail to gmail, though. Sans that, my only issue has been a mail server that uses Reverse DNS, which I don't have set up, and entirely ignores my email without it.
I suppose I can understand this if some people get a great deal of spam, but requiring so much of this on an unencrypted message seems more like useless reassurances than anything. I'm not criticizing email for being unencrypted, but this seems more like another hoop to jump through than anything.
Also of note, almost all of the spam I receive is from gmail addresses and I wouldn't be surprised if the invalid addresses that send messages demanding bitcoin are also from gmail, but with fake From fields.
I'll just chip in with everyone else who is also self-hosting:
I'm in a similar boat; been running email servers since before GMail existed. My personal one I've been running out of a home server closet since 2001. I've also done everything I can to guarantee I'm not running an open relay and not sending email unsolicited. Have been mostly lucky so far, but occasionally I will have people on mailing lists I manage (people I have met IRL and put them on the list to organize group meetings IRL) not get email. Used to be other stupid mail providers (AOL comes to mind), but these days it appears to be Google, sometimes.
I've had this domain nearly twenty years and run email on it for that same amount of time. I'm not going to "just switch", especially to a privacy invading ad-spewing "alternative" that doesn't give me as much control. Fix your damn servers, Google.
I have run a private mailserver for over 20 years, and I have the same problem. There's no telling what Google will let through. I have a second account at a medium-sized email provider, basically paying them for access to the email deliverability cartel.
Add another perspective to this. Once I updated my phone, and google would not accept new google Authenticator codes after set up. Locked out of 15 years of email.
Add me to the victim list. I've been running my own mail server for 13 years and have to tell everyone I'm intending to email to check their spam as that's probably where my mail will be. I send a handful of mails a week probably. Almost always goes to the bin if it goes to gmail.
Lest anyone think Gmail is less susceptible to problems because of the sheer volume of mail they handle (so any problem gets a lot of 'eyes' on it), they had an amusing and long-standing bug that meant any email containing a link to any domain name starting with "0x" automatically went to spam.
You could literally have two long standing, legitimately used accounts send an email to each other containing a link to a URL like http://0xANYTHINGHERE.com/ and it would be insta-spammed. I suspect it was a hard coded rule to avoid people using "long IP" URLs to circumvent other filters.. except there are lots of legitimate 0x domains that aren't long IPs.
It was fixed sometime in the past year but I got a lot of use out of it in talks I've given about email deliverability over the years.
I run a highly customised PHPBB forum.
Without these customisations that forum would be overrun with all sorts of spam.
However, these customisations only stop spam postings but can't stop actual registrations.
Based on the users that I see who are registering I see a great majority of these spammers love using Gmail accounts.
So while it is good that Google Gmail is trying to fix these spam issues, from where I stand Gmail users seem to be a big part of the spamming problem.
Spammers love Gmail only because they can easily create spamming e-mail accounts.
Google? Any large organization. I help a small non-profit run their site in my free time, and Google's actually been the nicest about our oldschool forwarders and Mailman lists. Despite being on no public blacklists, AT&T domains drop us at the border, with the only appeal process being via email, and I've never heard back. AOL and Yahoo? Flip a coin.
Of course, email forwarding turns out to suck, but we're just going to suck it up and move to G Suite for organizational email addresses and let folks forward from there. E-lists, OTOH, I haven't found a good integration to automate membership in the organization vs. G-Suite; perhaps it's time to just move to a forum.
A while back I'd explored trying to own my data, especially with email, and found that the efforts involved in hosting your own email server were tantamount to a full-time job.
The amount of fighting you have to do to stay on everyone's whitelists is absurd.
Reading this thread made me realize the magnitude of this problem. It seems very sensible that small email server administrators should unite into a single effort in order to better publicize this issue, raise awareness and shame Google into compliance.
Google seems to be suffocating the internet bit by bit on all fronts and it needs to be stopped.
Imagine a phone carrier dropping your calls before the phone even rang on the other end. This is how I feel most of the time when I email people.
Every new recipient I email, if I don't hear from them within 2 days, I have to contact them out of band to ask them to check their spam folder. The problem is usually Gmail's heavy filtering.
Can confirm, my server is also not accepted by google for years now when mailing someone with a business account. I'm not going to bother fixing Google's problems and support the behaviour, though, I'll just contact them through other means and let them know their email provider is blocking mail addressed to them.
At work we use sendgrid because of this. Have to trust a centralised third party to send out api keys. It's frustrating.
In my opinion the Gmail web interface should expose a way to generate something that is similar to a password, whose delivery is then controlled via the web portal.
This is similar to the +whatevertag trick that gmail pioneered for tagging, however that can be removed by malicious parties (spammers) via a simple regex. So Google have almost all of the infrastructure but should just add a bit more to get the rest of the way there.
What I mean in specific :
1. I want to sign up to and receive your newsletter (you[re Ted) but I don't trust you yet. so I should navigate to gmail.com, click something like "generate another inbox", leave it set it to "For now deliver this mail to my inbox", add the description "for Ted's Possibly Spammy Newsletter", and then click "generate". It should give me inbox3943578423@gmail.com - similar to a phone number but a bit longer and personalized to one recipient - and then I should give that to the recipient to use, in this case the possibly spammy newsletter. It should always be delivered to my inbox, as I've set. Once one of the spammers sells my email address (for example I start getting advance payment scams) I'll be able to disable further spam from there by sending it to the trash but also know that Ted's newsletter is the one that got compromised or sold it. You can do this today by going through the steps of registering a new gmail address and turning on forwarding, but it takes like 10 minutes to do so. it should be like 10 seconds.
This should be possible because people always have easy access to the gmail web interface. There's no reason it can't be a bit more like a social network where you confirm it from the web interface as well.
that's my idea anyway.
I've also run a mail server for a long time. Yahoo and Hotmail are probably the most annoying, Yahoo because you can't do anything about it and Hotmail for blocking entire IP-ranges. I don't blame them though, preventing spam is very hard. Spammers always have spf,dkim,dmarc,reverse dns,signing, etc meanwhile many legitimate senders has none. I'ts a hard problem to solve. I've tried things like increasing the response time so the sender have to sit and wait, or used spam lists, but there are too many false positives. One easy solution is to make it illegal to send spam. Since my country made sending unsolicited e-mail an offense, spam decreased a lot. The second problem is hacked servers. Unless you are running a mail server, always block port 25 in the firewall. So at least the hackers wont be able to send spam on your behalf. Many ISP's already block port 25.
I had my personal domain hosted with Google years ago. One of the reasons I moved away is that emails that I was sending to other Google Apps domains were bouncing as spam when the destination address was an alias. The bounce came from Google Groups, which is what Google used for handling aliases within Apps domains.
e.g. https://pastebin.com/u48DAaLP
That particular example was sent via SMTP, but I had the same problem when sending via the Gmail web interface, and it occurred sending to at least three different Google Apps domains.
After I moved my domain off Google Apps (I switched to Fastmail for a variety of reasons, but that issue was the kicker), I was able to send to those same addresses without issue. In fairness to Google, I was on the Google Apps free tier at the time, so there was nowhere to go for support.
I've always thought that, I also used to operate my own email server and I've encountered the same problem with other email providers, always the big ones, yahoo (there was a time when yahoo was big in email, at least here), microsoft and now yes, google. What is even worse is that sometimes google just drops the email messages, replies with 250 ok, but the email never gets delivered to inbox or even spam. Definetly the big 4 or 5 or whatever are taking over on all aspects of our life. Actually if I think better about it, they have already done so, there was an article here on hn about blocking google, amazon and ms network blocks, nothing really works as expected without them.
Regarding the issue with mailing lists for open source projects this isn't just an issue regarding Gmail. DKIM and DMARC cause no end of problems for mailing lists and it effects mail delivery to all major providers.
Mailman, which most open source projects use for mailing lists, have developed work arounds to address some of the issues. Unfortunately my experience is that many projects run older versions that don't have these work arounds or if running newer versions they have not been enabled. Most likely because no one has revisited the configuration since initial deployment on an older version. After all they didn't start the project to spend their time being mailing list admins.
Am I the only one here experiencing that GMail does reasonably good job in filtering out the spam, while I have to visit the Spam folder approximately once a week, just to randomly correct a mail or 2 a month? (not a Google employee)
If ProtonMail would standardize a client protocol that other MUAs implement (not their "bridge" kludge), I'd start encouraging people to move there.
I'm not interested in ProtonMail's encryption (and it's potentially a liability, attracting aggressive state action). I'm mainly interested in their apparent respect for the privacy of users' private communications. And also hoping that ProtonMail has a bit more reliable delivery than GMail.
In any case, rising competition lifts all performance boats, or something like that.
I've also hosted my own email for several years now, on several domains. The one thing I have that does not appear to be listed is a DMARC policy. As the comments on the blog post suggest, configuring DMARC works. I have no problem delivering to either Gmail or Outlook.
I don't think it's unreasonable to be strict regarding DMARC delivery. My MTA has a fairly strict SPF configuration - any email with an invalid spf result is rejected. This can come about because a legitimate company has misconfigured their spf records (happened twice in all the years I have hosted, discussions via postmaster@ helped them configure their dns correctly), but 99.999% of the time it is a spammer. What is worse is that rejecting email for domains without any SPF records can still result in valid email being lost, in 2019.
In this specific case, I don't think Google are "being evil". They're trying to reduce spam in the email ecosystem and they're doing it by using standards they themselves adhere to (Gmail send me reports of dmarc statistics each day google domains receive email from my box).
On the other hand, I do of course support either self hosting, or using another provider so as to ensure we do not end up with a Gmail monopoly. If I did not self host, I would find another provider like (but may not) Fastmail, Posteo etc (I would have to seriously review the options, which I haven't done).
I'm glad I'm not the only one seeing these problems. I send mail from a personal server, and recently discovered that my messages to people @gmail were frequently sent to spam or simply dropped. I briefly thought friends were ghosting me until I figured it out. I set up SPF, DKIM, and DMARC, but it seems like the only way to get reliable email these days is to use Gmail, Hotmail, or Yahoo (at least for a little while...).
As far as I can tell, Gmail isn’t actually “real email” anymore, and you shouldn’t use it for anything important. It’s verging in being a walled garden.
Does AI compares whom data to define which message is important or No? It doesn't track any specific way on evolution of mentality when it goes to define data correlated to humans usage. Like others products on market. Therefore, if you apply this thought to AI I guess they just made with basics conglomerated informations gathering. That CAN´T move a barrier when satisfying humans usage. It is just how you confront which message is correct to say is relevant or no. AI can be freely a way companies uses their needs on market influence because they're , on a very simple association, doing by themselves, then people get a notion of arbitrariety on those machines that create discussion to the whole universe you see about robot revolution etc, but not the case. I guess they try to hard to make then appear as human to be soon accepted by costumers, its just the way it is, it is not the time to require maximum levels of perfection on how easy they're to addapt as a user
Gmail can be pretty aggressive with spam filtering, but I'm very happy with the bias toward flagging things as spam. Spam was such an infuriating problem, and lots of marketers would happily push the line if they didn't live in terror of being blacklisted by Google. If only Google could the legacy telephone networks like they run Gmail.
The black box is more mysterious to me. I had a client for whom I managed a G-Suite account with a custom domain name. The domain was not on a blacklist, and things generally worked well.
One day a colleague and I discovered that he had not received some of my emails (intra-domain - me@example.com to him@example.com).
This is all within the confines of Google. Google had flagged some messages as spam, and by what determination I could not fathom. The content seemed perfectly typical.
I have had really pleasant experiences with G-Suite human support, at least in terms of the quality of interaction. But they could not answer why some intra-domain emails were being flagged as spam. I have suspicions that it would take a whole team of G engineers to maybe identify what bit of logic in their systems (incorrectly) marked some of the emails as spam.
It seems the beast (automation) is just almost not under their control anymore.
Isn't charging people to use other carriers one of the big things that got Bell broken up? Google seems to tread in a of very dangerous waters. I wonder if they're just not aware of how close they are to being shutdown entirely, simply because so far they've always withstanded most legal challenges.
There may be a workaround here: configure your own mailserver to relay outgoing messages through a service with a known good reputation. For example relaying via sparkpost or mandrill should be trivial to do. You might be able to do that under a free developer account if you don't send a lot of email.
Thank you for sharing your experiences, insights.
#1 - Does anyone send test emails and measure delivery rates? As in send yourself a bunch of emails and see what happens.
USPS and its major customers and vendors do this with physical mail. They measure stuff like UAA (undeliverable as addressed). FWIW, their Inspector General estimates 4.3% of mail was UAA in 2013. Report Number: MS-AR-14-006 https://www.uspsoig.gov/document/undeliverable-addressed-mai...
#2 - What is the responsibility, liability for email relays to treat everyone equally? For comparison, a US retailer has to accept US currency, but can (sometimes) turn away problematic clients. Is there anything like that for electronic exchanges, transactions?
If people stopped using G in such high percentage, then they would most likely need to adapt back to a more open internet, instead of trying to wall it off around their services.
Reminds me of jabber.ccc.de that stopped providing new accounts because they felt they were ruining a federated system.
This is the reason I sadly don't recommend to customers that they run their own email servers, and instead outsource it to GSuite or O365. Being on the hook when emails aren't delivered for one or more of a dozen reasons that you can't control is no fun at all.
"Only when something like this happens you realize just how impossible it is to talk to someone on the modern internet without having Google somewhere in the middle." This says it all and is likely become a strategic imperative for Google along the way.
>I wonder. Google as a company is famously focused on machine learning through automated analytics and bare minimum of human contact
It should be noted that ultimately these efforts to "learn" result in ML, AI whatever pointed AT you, not working for you.
Interesting...
Like the author, I've been running my own mailserver for over a decade and am very conscientious about ensuring that no attackers use it as a spam relay.
While the vast majority of the people I exchange email with don't use GMail at all, so it can take a while before I notice any issues with it, I did happen to notice that GMail was rejecting my outgoing email a couple of weeks ago.
This week, I finally got around to trying to address the problem (it's not high priority because having GMail reject my emails isn't really a huge deal).
...and I found that it is working again without my changing anything. Weirdness abounds.
Running your own e-mail server for receiving SMTP makes a lot of sense in many ways. I've been doing it for close to a decade myself.
Sending SMTP yourself (directly, without an SMTP relay service) sets you up for trouble.
What boggles the mind is that they've a very string disincentive to fix it: I had to move away from my privately run email to Gmail for that very reason, and I'd gather I'm far from being alone.
Better that what I've been seeing, which is Google just sending emails sent from my private domain to spam, with no bounce back or notification of any kind. This is despite the fact that I send the emails FROM gmail using their 'send as' feature. Nor does it seem to matter if I'm writing to a contact whom I've exchanged many emails with in both directions for years. If I send from my @gmail address they always go through. Send the same content from my domain and it's a crapshoot.
Edit: And of course, I do have DKIM and SPF configured.
I tried to set up a Discourse discussion forum for our neighborhood this year, and got stuck on setting up an email server that wouldn’t get blocked automatically by all major providers. Email is not a healthy ecosystem. You have to pay to send from one of the big trusted providers or else you get blocked just because your email ip address is untrusted by default. Yes fighting spam is important, but it’s shocking how much email providers rely on simple ip filters and trust levels rather than AI analysis of email content.
This article didn't mention whether or not their servers are attempting delivery using TLS/StartTLS, which is a good thing to check. I think Google penalizes email delivered insecurely.
In the article:
> I can't tell other people to go off Gmail
I disagree. There are reasons to switch off gmail. Not just Google eating mail but also for privacy reasons. Google knows all about your banking, eCommerce orders, your media subscriptions, health issues and many other dependencies.
A good alternative is protonmail. It is private, has a mobile app, is a free but you can also pay to support the service. I also consider protonmail much more secure than gmail.
I'm quite happy with the ease of setting up and running a Mail-in-a-Box [1] instance. After seeing this article I did an experiment and while my hosted mail sent to a @gmail.com address did not bounce it went straight to the Spam folder. At least I can add other users to my email and enjoy private conversations.
Yet one more example of this. See [1], [2].
Definitely Google has done something to mess up their spam filter algorithms in the last year.
[1] https://news.ycombinator.com/item?id=19536465 [2] https://news.ycombinator.com/item?id=19500357
If I recall my time at a main-sleaze email sender, some ISPs will send SMTP retry requests on random incoming email to see if there is a functioning SMTP server on the other side of the originator's email. Often times, spammers won't care about the email that's being sent, so won't respond, or in cases where they are hiding the origin, won't be able to respond.
I don't know that this is purposeful in all cases. I've had issues on a mail host where sending an email to a different box on the same domain would reject the message as spam, refusing to even deliver it. With respect to google, however, the fact that they offer a service of their own that conveniently can avoid this issue is more than a little shady.
Its like the deep learning has identified a feature it can use to determine spam: is this from the top 10,000 domains? No? It's spam.
Spam from small domains might be pretty high as a category, but of course we don't want statistical judgements about categories to outweigh the merits of the individual. Maybe Google's algorithms have been watching too much Fox News.
I often see and hear about this issue a lot and I wonder how other email companies cracked it.
I use FastMail for instance and never have this issue, but I know so many people who gave up on running their own mail servers at the small enterprise level because of stuff like this I often wonder how FastMail does not have these issues but others do. Is it a headers thing I wonder?
Someone posted on reddit the other day about (seemingly) the exact same issue https://www.reddit.com/r/selfhosted/comments/bbrvlt/any_reco...
I use my own SMTP server for receiving, but for sending I use the ISP's server (I think I have seem to have read somewhere that not only Google but some other services as well, will not accept mail from dynamic IP addresses). I have not had problems that I know of, so far. Please tell me of whatever mistake I may have made of such thing.
Gmail is rejecting valid content as spam and it never hits my spam folder. If I was not expecting an email as I was in this case, I would be totally oblivious. Something is up with the spam filter recently. They might have decrease the tolerance for spam and as a result a lot of legitimate traffic is being hit in the cross fire.
That's the price you're paying for a) self hosting and b) relying on decentralized protocols that don't have sensible abuse protections.
People hosting their own servers enabled wide spread abuse due to misconfigurations. Because everyone could do it and because defaults were shit for decades, stuff like open relays were common. People defaulted to the wrong ports. Almost no one bothered to offer STARTTLS/Transport Encryption. Spam would have killed mail by now if it hadn't been for major players like Google, GMX, Hotmail/Outlook/etc.
Back in the day, greylisting was commonly regarded as a best practice, leading to the impression that email is unreliable and prone to latencies.
I'm sorry it's this difficult to host mail by yourself nowadays, but I'm happy to have a spam-free inbox every day and if this is the price for that, I'm sure about 1-2 billion people are willing to pay it.
I'm quite astounded that there have been no updates to mail protocols in the last couple of years to at least mitigate the most common issues, but all I see are band-aids that are complex to setup and horrible to debug in case of issues.
I've had Google blackhole me several times. Each time without any apparent reason. It's impossible to talk to a human being at Google who is able or willing to help. I've had similar problems with MS, but at least I was able to talk to a human being and get the situation resolved.
To me this is more evidence that we definitely need public decentralized platforms that can replace Google and once we have them we need to eliminate the company. Private companies should not be allowed to monopolize platforms and control our lives.
Thanks for writing this up, this mirrors my experience setting up a modern email server for personal & business use.
I currently self-host for non-mission-critical email, use FastMail for business, and continue to use google apps for personal/mission-critical.
If you run your own email server check out Postmark's DMARC monitor, it's a free service:
(tablix.org doesn't have a DMARC record)
He’s been using the same IP address for his mail server for 15 years. It’s possible his IP is on a subnet range that regularly gets blacklisted by Gmail due to other actors on the subnet sending malicious email/spam to Gmail.
Does anyone here use encrypted and signed messages on a regular basis? How does gmail handle that on either sending or receiving ends? Is delivery rate better or worse if it is fully encrypted?
Reminds me of Spamhaus. Are they still around? When I worked for an ISP we had to deal with them regularly. They'd happily hold you hostage over a single e-mail.
I use non-Google email with SpamArrest, gives me complete control over whitelist/blacklist. Legitimate users always have a way of getting through.
Ignore it and let people be angry (at Google for not receiving their mail from various places). Meanwhile move to a decent service like ProtonMail.
Self hosting email is a pain in the butt.
Try fastmail infomaniak.com (can also buy email for external domain) gandi.net (email included in domain)
I had exactly the same kind of problems when I tried to set up a new mail server.
In the end I just gave up and started using mailgun as a relay.
I am in the process to detach Google products, somehow gmail is the only product I cannot find a working alternative.
I wonder if originally this technically constitutes an illegal form of interception of mail in some jurisdictions.
Perhaps there really would be a lot more spam without such filtering, but it points to the actual problem being elsewhere. Perhaps we need some kind of cheap and userriendly (uniform but decentralized) email court system, and fine / ban email accounts that misbehave?
I have a similar problem and it sounds like some here do as well.
What can we do about Google's email monopoly?
I for one welcome our new overlords
Does google have legal obligation to receive mails from 3rd party smtp servers ?
I have a lot of success by having put my mailservers in the dnswl.org system.
Can someone explain in simple terms why, in 2019, anyone is running their own mail server? Unless you are the sysadmin, server guy/gal, whatever, this just sounds like signing up for a world of hurt doing it yourself.
Ironically Google's spam always seems to go through...
They want your users to use gmail, that's all.
Cue the *"but it's so easy to host your own email! Mine has been working fine for years!"
Until this happens. Which it does very often, for lots of reasons that are out of your control.
One more reason to de-Googlify my life. AWS has WorkMail I think it is time to try it out.
Google has become pure evil. Avoid Android. Don’t use Chrome, G-mail or Google maps, or their search. Block Google analytics and alikes.
Hacker news is fucked by google
Fair play.
I like my emails with pesto sauce, tuna, and cheese ... wicked combination.
Op should sign up for gsuite.
Google eats everything
Companies like this give whores a bad name.
Only 3 people I know using Gmail any longer. I stopped back in 2010 and "didn't feel a thing".
Have also run into this. Google is the quintessential evil tech corporation. We should all start avoiding their services when possible.
Google has no incentive to fix these kinds of problems.
It's big enough that when someone complains that a message sent wasn't received, the intended recipient will say, "I never have problems with my Gmail account. It must be you." And the sender has to switch to Gmail to reliably communicate with the outside world.
I wish this was just paranoia, but we've seen multiple discussions on HN about Google programs and policies that alter the internet in ways that only benefit Big G. It's like we're heading back to the days when people didn't know the difference between AOL and "the internet."