Microsoft is even worse. They are blocking their own customers from replying to emails because their outgoing smtp server rejects it as spam.
Despite all the advancement in machine learning, spam filtering seems to keep getting worse. And both Google and Microsoft have absolutely laughable support for when your emails are getting blocked. Could we just dial things back a bit? Some spam is fine if the alternative is having an unknown amount of legitimate messages not delivered to you.
I understand this guy's frustration but this:
> And this is happening after SPF, DKIM and DMARC provided a solution to the spam problem.
is just wrong. Tons of spam comes from servers with SPF, DKIM and DMARC now. It stopped being a trustworthy signal of not-spam many years ago.
They say "Both email servers have PTR records set up, and SPF [...] DKIM, and DMARC records[...]".
Yes. Great. Thing is that this is such a trivial barrier to entry that guess what? Spammers do it too! Email has become so utterly corrupted with spam that the reality is that an independent provider who has no existing reputation is, 99% of the time, going to be a spammer.
It would be wonderful if we could fix this - but so far noone's come up with a workable solution.
Heretical opinion: the government does a good job with postal mail. Let them handle electronic mail.
Most of the problems of spam have to do with who's an approved sender and who's abusive. Regulate the mail, it's much harder to be abusive.
You're in this country and sending spam? We arrest you. You're in another country? We rate limit your mail, report you to foreign authorities, and flag anything that looks like your mail.
You can burn me at the stake now.
Okay, instead of taking this at face value (with a big acknowledgment that it is genuinely frustrating that both Google and Microsoft don't have good contacts to just know what went wrong), I'll analyse this carefully.
This website is operated by Virtual Industries Group (https://www.vig.co.nz/), which is according to their website operates three services: School Bookings/School Interviews (https://www.schoolinterviews.co.nz/, focused on school scheduling and the service discussed here), Care Bookings (https://www.carebookings.co.nz/, which is the same service as above but focuses on day-care and other similar functions), and MessageMyWay (https://www.messagemyway.com/, which according to their website "is the communications hub for your community. It is your emergency communications plan, your telephone tree, and your email broadcast system all in one").
From a cursory glance, these three services shares this set of outbound MX servers. While it is very unlikely that someone who uses School Interview and Care Bookings would mark their message as spam, if the messages relayed by MessageMyWay is sent on the same outbound MXes then I could immediately see the problem. A large part of MX operators knows this and separates "marketing" and "operational" messages into separate servers to prevent this exact thing from happening. While I understand this dev's frustration, maybe the messages relayed by MessageMyWay are the ones marked as spam by frustrated parents who are receiving irrelevant school marketing which are sent to same MXes as their important operational messages?
If I were the developer (and still insist on using on-prem email), I'll operate three groups of servers:
Set A: purely for company-initiated messages, never for the customers
Set B: "operational" customer-initiated messages: School Bookings, Care Bookings, and MessageMyWay mails which are marked "critical" by the users (which is apparently additional cost)
Set C: "marketing" customer-initiated messages: "normal" MessageMyWay mails
I can't even get Gmail to deliver my 100% genuine, properly authenticated, sent-from-my-own-domain-which-has-never-sent-spam firebase sign-in emails - and it's the same parent company!
The get sent to spam or (worse) silently have the links stripped out a decent percentage of the time
That sucks, big time.
What I don't get is, what is Google trying to achieve with this? A rate limit that doesn't increase daily or check the spam rate or provide a support rep to lift it?
Gmail has been around a long time. Is this the first we're hearing about this?
I have a more positive experience than the OP, I worked with a news outlet that has several newsletters and sent about 200K emails/day, and we use our own servers/domain and an open source/self-hosted tool to send them (I think it was Mailman [1]).
We did all those technical bits like SPF and DKIM, put the one click unsubscribe link on the message and also on the header of the message so that clients like Gmail can put the unsubscribe link on their own UI [2], all the recommended practices. I remember using the tool Mail Tester [3] and the results where all green.
We doesn't have issues with being marked as SPAM by Gmail/Outlook, and have average open rates of 50%, which is a lot higher than the industry standard who is around 20% [4].
We have a good UX and an ethical way to treat our users, like all users have to opt-in to their desired newsletters when creating an account (or choose to receive newsletters without creating an account), one-click unsubscribe link in big text at the bottom of each newsletter, but also a one-click-no-need-to-be-logged-in link to opt-out of all the newsletters that the user was subscribed, and more stuff like that.
But one thing that I think give us a lot of reputation for the Gmail algorithm, it was that we designed a feature that if the user doesn't opened a newseltter for about 3 months, we started to send the newsletters with an alert at the top saying like "Seems that you aren't reading this newsletter anymore, you will be automatically unsubscribed in 30[n counter] days. Click here to disable the auto-unsuscribe." (the disable the auto-unsuscribe also was for people who have images disabled and we can't track the openings, but is small percentage). So with that feature, we get sure that our users where engaged with the newsletters, and we have a system to avoid sending messages to "dead" emails, maintaining a fresh and healthy database of emails, and seems that Gmail/Outlook knows and likes that.
[2] https://www.sendinblue.com/blog/list-unsubscribe-header/
[3] https://www.mail-tester.com/
[4] https://mailchimp.com/resources/email-marketing-benchmarks/
Off-topic:
> We recommend choosing a word or phrase that means something to you, and then adding one or two numbers or symbols, like "!" or "+". So something like squash:club! makes an excellent password.
That is terrible advice.
This is why Proton is extremely aggressive with its free users and close accounts seemingly arbitrarily. Any abuse of their service could cause them to be banned because they are small.
Google doesn't have to care, who is going to ban Google SMTP servers? That would be suicide.
The promise behind SPF, DKIM and DMARC is that they allowed positive reputation mechanisms, which cannot work if any domain is easy to spoof. So if you're careful, only send good email and stop sending when the recipient asks, that should be a golden star on your review, destination domains will happily let your email through and nobody will be able to freeride on your good reputation because your domain is authenticated.
However, this is still hard to establish, because it depends on the destination domain server to decide whether you're sending good stuff or not. I would like to have a mechanism by which the user can decide.
I imagine something like this: each email provider, say Gmail, issues to its users a number of single use codes like "Sor7xeik". When the user wants to subscribe to a newsletter (say news@interesting.com) it gives its own email address and one of those codes. The first email from news@interesting.com contains some header like
Authorized-Sender: authorize Sor7xeik
Authorized-Sender: yes
With this mechanism bad practices like address harvesting and selling become much less useful (because an address alone is not that useful, if the sender is not authorized; and the authorization must be initiated by the user).
BTW, I am not saying that all emails should be sent with this authorized sender mechanism. I don't expect individual users to collect authorizations for each of their contacts. Email without the Authorized-Sender header would still be subject to the usual spam filtering, but agencies that often send legitimate mass emails can have a mechanism to prove that they're doing it with the user authorization.
It's very easy to get legitimate emails false-positived as spam by gmail/gsuite or office365 MX even if your sending IP address is not in a bad-reputation IP block, your rdns, spf, dkim and dmarc are impeccable.
I know people who've been running mx/smtp servers on the internet since 1994 who have now given up on running their own, not because they're technically incapable or unwilling to continue to do so, but because they've been forced into themselves using gsuite or office365 because of the monopolistic practices of the huge email-as-a-service providers.
I tried to move off fastmail back to gmail last year because I lost a job opportunity due to a random fastmail-origin email i sent getting blocked by my interviewer's google mail. the first three went through fine, it was only the fourth one that didn't make it. they thought i ghosted and had already chosen another candidate because of it.
ultimately i didn't make the move because email is unusable without custom aliases i can completely bounce email from. but i think i might attempt again, but again I recently found myself insanely paranoid about my interviewing situation and had to make the awkward transitioning to my gmail account and explain why to my interviewer
An open system right up until your users use the junk button as the delete button.
OP here. For those thinking that Gmail MUST have valid reasons, and that we abusing the system in some way, consider...
1. We are registered with Gmail, Microsoft and Yahoo to provide feedback when someone marks our messages as spam (we DO do everything right). We get reports less than once a week, so if I've got my maths right, less than 0.002% of recipients think we're spammers.
1. Our two servers send around 10,000 messages a day, spread fairly evenly over time. That works out at three or four messages per server per minute. If we were spammers, we'd be ashamed of ourselves.
Could their behavior towards legitimate email from small-time outfits like this be useful evidence in an anti-compete lawsuit?
Alternatively, some kind of class action?
(Asking any lawyers in the room)
Disclosure: I work in Gmail.
Likely there's a reputation issue on those IPs. Consider filling out a form to investigate that: https://support.google.com/mail/contact/gmail_bulk_sender_es...
For me at least, existing completely without GMail / Google SSO is unrealistic, and the million email corpus that is attached to it would be painful to part with as well.
But I've been trying to slowly de-tangle myself, and for it's faults ProtonMail has been working out pretty OK for me as a compromise between usability and true digital freedom.
This is what all fediverse protocols and specs gaining traction today should be designing for. Email was lucky and got their foot in the door before the dotcom era figured out about walled gardens. Now we're slowly crawling back out of them, but if we're not careful we'll be out of the pot and into the frying pan.
I am running my own server @entropy.be for last 20 years. Maybe I'm lucky but it just works? The secret is only to have trusted users that do not send spam.
OK I get the occasional spam through spamassasin, but I can live with archiving 5 spam emails/day.
Meanwhile Gmail happily lets through any scam email from any hijacked account on outlook.com.
Besides Google playing nice and fixing their stuff, is there a possible fix for this?
This is anticompetitive, and should be enforced as such.
Unfortunately regulatory capture is real.
It's not just gmail, Big Email is a cartel.
i wound up developing a pretty complex email sending infrastructure because of situations like this. it involves putting a message in a queue and then sending to a list of predefined proxies (mailjet, mailgun,sendgrid, etc, etc) and using backoff algorithms per proxy per hour....
Why is server spam filtering even needed with Junk/NotJunk feature in Thunderbird?
killing?
More like murdered.
[flagged]
Yes, email has evolved from an open standard to a closed system dominated by only a few big corporations. I don't see how is this situation acceptable - such a fundamental functionality is out of reach to all but a few of us. Well, Gmail and such are free and convenient (until you get banned for no reason), so it's good enough for the most. Trying to get back to the open decentralized ecosystem is a waste of time.
But we do it anyway - and we start with the identity layer. Email based on blockchain identity, free and open. I've been working on this for a while, still WIP but check it out: https://ubikom.cc
Yep, after over a decade of trying to run independent mail servers we finally broke down and switched to SendInBlue and SendGrid last year. Even if you do everything right, it's just not possible to get gmail to reliably deliver mail unless you use one of the big services for smtp. It's really disappointing.
Edit: We can at least keep using our servers for incoming mail, and just use those for sending.