I'm the Abusix engineer in question (and actually the architect of the system in question), and you're being somewhat "economical" with what actually happened here.
Here's the actual chain of events in question:
- You recently switched ISPs and that meant you moved to a new IP block.
- Without going into massive details on how our infra works, but when we have the most serious level of listing (e.g. hitting our most secure traps as in this case), we treat the same /24 more aggressively than we normally would (because of things like snoeshoe spam that would normally spread traffic across a wide range of IPs).
We use /24s only where we cannot determine different ownership of the IPs by looking at the abuse contact registered at the RIR e.g. if the IPs are different contacts then we don't bundle them into the same bucket. In this case Hurricane Internet owns the entire range, so the /24 is used.
We do this because there is no other way to do it that isn't completely abuseable by a bad actor e.g. rDNS is completely trivial to forge and to claim multiple fake entities. If someone has a fool-proof way to do this, then I'm all ears.
Then we get to your failings in this case:
- You don't have proper, working bounce handling.
You were repeatedly sending mail to an old customer and we were rejecting 100% of the email you sent to that address - stating that you should stop sending to it in the rejection message. We always reject traffic on traps we are building so that bounce handling removes them automatically over time.
- You decided to send a marketing message to a bunch of users, including to the address described above "We (rsync.net) are experimenting with a lifetime prepayment option". This message provided no List-Unsubscribe option at all, so I could not unsubscribe it without exposing the trap to your engineer (which is my primary concern as it takes years to build traps properly).
- Your engineer said to me "we took down 600 old accounts and are reviewing our contact policies going forward" and "there are still plenty of other customers who are listed generically as bouncing so we will have work to do here".
That tells me that you knew that you were sending to old accounts and that your bounce handling was either not great or non-existent.
I take our role very seriously and I and my team go out of our way to help anyone who finds themselves listed by providing evidence and advice and we always try to find a good resolution for all legitimate senders.
That is exactly what I did here with your engineer, the problem is resolved as the specific account was removed and you know now what you were doing wrong and how to fix it so that it never happens again.
Your engineer was appreciative and I said if there are any further issues in the meantime whilst you fix things on your end, then we would help by exempting your traffic whilst you did those changes.
I can't see how we could have done anything more in this case.
On our website, we provide blog posts and videos, we take part in conferences and workshops to give advice on how to do things properly so you never have blocklisting issues.
I can easily summarise here what everyone should do to avoid issues:
- Have proper rDNS for your sending IPs that is part of your administrative domain (e.g. don't use your providers generic rDNS that contains the entire IP in the hostname). If someone visits the domain name, make sure there is a website present that has contact details as a minimum.
- Make sure your abuse@domain and postmaster@domain accounts actually go to a responsible person.
- If you send any marketing mail at all, make sure it has a working List-Unsubscribe header (preferably HTTP that allows someone to unsubscribe without having to contact you). Important note: If you use a mailto: unsubscribe, then I cannot unsubscribe a trap, even if I wanted to.
- Make sure you have working bounce handling. If you repeatedly send to an account and it either bounces or is hard rejected, then you need to stop sending to it and mark it as bad. No excuses.
- Don't send to addresses where you have not contacted them or had any interaction at all for > 1 year. If you haven't kept in touch with your customers, then that is on you.
- If you have a web form that when submitted, sends a message to an external user, then you MUST a) validate all of the input fields and disallow URLs unless the field required it and b) prevent automated submission via the use of a CAPTCHA.
- When collecting email addresses for mailing lists, always use confirm opt-in e.g. send a message containing a link that they have to click to activate the subscription. Do not send any further messages to them until this has been completed.
- Make sure you separate IP addresses being used for outbound mail from those used for outbound NAT pool. Block outbound port 25 from the NAT pools and make your firewall notify you of any port 25 activity from any hosts as this could indicate they are infected.
- When provisioning a new mail server IP, don't send more than 30 messages per minute (e.g. 0.5 messages/sec) for the first day and then increase the volume over the following week.
I'm sure some will disagree with some of these, but I guarantee that if you follow all of these, then you'll never have an issue with a blocklist like ours.
Incredibly insightful, thank you for taking the time to post this!
> - Don't send to addresses where you have not contacted them or had any interaction at all for > 1 year. If you haven't kept in touch with your customers, then that is on you.
That seems odd though. "Keeping in touch" in regular intervals is exactly the kind of thing customers may want to unsubscribe from, so a working unsubscribe function means there are customers where I cannot keep in touch. How is that on me? That means I can't send e.g. a password reset email to them if they try to log in two years later?
Apologies for not being clearer on this, I'm generally talking about marketing messages when I say "not keeping in touch".
My point here is "consent to send", e.g. I think most people would find that someone importing a years old list of email addresses and then suddenly sending marketing messages is unacceptable and constitutes spam.
If you're "doing it right", then "touching" your customers with a once a year "Hey, we haven't seen you in a while" message is absolutely fine and generally a good thing to do as it maintains consent, because they can unsubscribe if they're no longer interested (as they might have forgotten they signed-up or had an account).
If someone hasn't logged into your "service" for >1 year and now wants to do a password reset several years later, then I would say that is entirely different.
We generally try our hardest to handle this case without it causing a listing if you hit our traps with something like this.
"- Your engineer said to me "we took down 600 old accounts and are reviewing our contact policies going forward" and "there are still plenty of other customers who are listed generically as bouncing so we will have work to do here"."
(snip)
"That tells me that you knew that you were sending to old accounts and that your bounce handling was either not great or non-existent."
Yes, that is exactly right.
I have instructed everyone to be as liberal and forgiving of non-payment and failed contact as possible. These people, who were paying customers, have data stored here for safekeeping and we're not going to trash it because we haven't heard from them in 3 months or their email bounced.
I can give you hours of stories of customers who came back from military deployment, came back from depression, came back from prison, came back from financial ruin ... that contacted us, beyond all hope, to see if we still had their account. And we did.
After 21 years of doing this are there a few hundred accounts that we are giving extreme benefit of the doubt to ? There certainly are.
Bottom line: our duty of safeguarding customer data trumps this weeks fashionable spam heuristics.
> Bottom line: our duty of safeguarding customer data trumps this weeks fashionable spam heuristics.
I haven't suggested that you do anything differently to that.
Keep the data, keep the accounts, do whatever you feel is best for you and your business.
All I've said is to be smarter when it comes to sending email to old accounts that are repeatedly bouncing. I've outlined what was wrong and I worked with your engineers to resolve it.
The screenshot I got of your conversation with (Dave) was, indeed very well done and was both informative and actionable. You responded to us very quickly and with a high level of professionalism.
Further, participating here in this HN discussion is noteworthy and impressive.
FWIW, we immediately complained to he.net about the bad actor elsewhere in that /24 ... we'll see.
> The screenshot I got of your conversation with (Dave) was, indeed very well done and was both informative and actionable. You responded to us very quickly and with a high level of professionalism.
> Further, participating here in this HN discussion is noteworthy and impressive.
Thank you - much appreciated.
The quality, morals and perception of our product and how we treat people matters to me greatly which is why I'm always happy to get involved in discussions like this.
As is evidenced from other post on this topic, there are a number of competitors that don't behave in the same way and that leads people to think we are all the same and I absolutely want to challenge that perception when it comes to us.
> FWIW, we immediately complained to he.net about the bad actor elsewhere in that /24 ... we'll see.
Thanks, feel free to refer them to me and I'm happy to provide as much evidence as they need.
As a former admin of an isp & running saas apps I applaud you greatly for your approach. Some people may find these approaches are "heavy handed" but the reality this is what life is like in 2022 with email. Should we need to do this? Definitely not, but it's an arms race and each company sending email has to up their game - to both be compliant in getting email accepted, and to foster a healthy relationship with customers.
The comment about "lifetime prepayment option" is exactly something that MUST have an unsubscribe option to it. I understand everyone is trying to increase business, but this is something @rsync missed mentioning, insisting you don't need one.
One of the ways most blocklists work is by employing spam traps which can either be individual email addresses or entire domains.
It's quite a large topic, but I'll try and summarise for you.
You can't just take a spam trap and use it to block anything that hits it - that would be incredibly unfair as you might not know the history of the email address or domain and you'd generate considerable false positives if you did.
If you buy a domain or register an email address and you know for certain that it's never been used before, then this is typically called a "pristine trap" and there are then various ways to then "seed" this so it starts to receive spam and malicious traffic. This is the exception where it could be used immediately. However this process usually takes a very long time (usually years) to receive enough traffic to be useful.
Any other address or domain where you don't know the history would be called a "recycled" trap. There are various opinions on how these should be managed, but generally it's accepted that you need to reject ALL traffic on these for at least 2 years to allow genuine senders to work out they are no longer valid (which is why bounce handling is important!).
The other type of trap is a "typo" trap, which is typically an entire domain (or an email address deliberately close to another) which could easily be typo'd. I have a rule here that says we neverever use these for any blocklisting (other lists will not have the same rules!), however they can be useful to see trends and detect compromised hosts (as these typos frequently end up in compromised databases that get sold around the dark web). They can receive considerable volumes as well.
Typo traps are especially why you should always use confirmed opt-in and employ CAPTCHA on your sign-up forms (although this also helps with all traps in general).
We carefully monitor all traps to see how they are performing e.g. detections .vs. traffic from whitelisted sources and false positive reports and will immediately remove a trap if it begins to perform poorly or has been made public in any way.
It takes years to build a trap network and we have to discard all of that work should they become discovered (as they could be used maliciously), which is why a blocklist operator will never reveal the addresses to you as they are a closely guarded secret. In fact on our own network, even we don't see them - we monitor them via a hashed name! (only I and a few other select people can convert the hashes to the actual domains or addresses).
I think that covers the basics - I hope that explains it sufficiently for you
> make sure it has a working List-Unsubscribe header (preferably HTTP that allows someone to unsubscribe without having to contact you). Important note: If you use a mailto: unsubscribe, then I cannot unsubscribe a trap, even if I wanted to.
This is very interesting to me. Could you elaborate further?
- Some providers (IIRC outlook being the main one) only support mailto: and most prefer it. I have never seen advice that you "must" provide HTTP before.
- What do you mean by unsubscribe without contacting me? Isn't HTTP or SMTP just as much contacting me?
- Why can't you unsubscribe mailto: even if you wanted to? Earlier you hinted that it may have to do with revealing the trap, but while a HTTP url may not directly include the trap address it would be linked anyways. (Otherwise how would the unsubscribe work?)
Thanks for all the valuable insights. It does sound like you are taking a responsible approach here.
Sure - happy to elaborate here, re-reading my response I understand the confusion here as my choice language wasn't great ;-)
I didn't say anywhere that you MUST support HTTP unsubscribes, but I just said it was preferable (to me at least).
HTTP typically means there is some sort of form which will typically auto-fill the email address from the database (because of the HTTP GET parameters included in the link) and all the user has to do is generally click 'Unsubscribe' and the address is automatically removed from the database.
The reason I prefer this is that if I or one of my colleagues want to unsubscribe a spam trap, then we can - but only if the above is true. (Disclaimer: we only do this very occasionally, usually it's only if we've spoken to someone and are convinced they are legitimate, have fixed any issues we've identified, but are hitting traps because of a single address - if we didn't do this then we'd either have to force the sender to reconfirm their entire database by asking everyone to opt-in again, or we'd have to whitelist the sender, neither are great options - so if we ever do this, then it's done hours/days after the event so as to not tip off anyone on what the trap was).
Because spam traps have to stay secret, we only reference them by a hash value internally (and only I and a select few can convert hash to trap value to do the unsubscribe) and as the messages are received on our trap network the traps themselves are obfuscated to a know value before they are stored in our evidence system.
mailto: means we can't do this at all - a trap can never originate traffic, that's cardinal rule of them, it would be grossly unfair if they did.
> - What do you mean by unsubscribe without contacting me? Isn't HTTP or SMTP just as much contacting me?
Poor use of words from me.
What I meant is that I don't trust messages that have an unsubscribe that does not encode the recipient in it in some way e.g. HTTP method without GET parameters (as this means I would have to do considerably more work to fill out the form) or a mailto:[email protected]?subject=unsubscribe - both to me scream our that there is no automatic connection to a database and that someone on the receiving end has to manually remove the address - which could take weeks, or be never.
One other question. How do you manage domain based reputation vs IP reputation. I'm trying to get by running a service off of Cloud provider IPs and they may rotate from time to time as VPSs rotate. Part of me feels that once my domain has enough reputation it should be fine. It does seem like some of the big providers work like this (Google seems to trust me now from any IP) but it seems that a lot of smaller providers rely on blacklists more as they don't see as much traffic to build reputation. How do blacklists handle domain reputation?
From our perspective, we don't link IP and domains at all at the moment, they are totally independent.
Because spam traps get very different traffic to a real mail server, you can make a lot of assumptions that would be impossible to do on a real mail stream.
So generally speaking, if your domain name isn't in the top million domains (and there are still some exceptions to this - but I'm not going to list them all here) and you hit one of our pristine traps (see my post about the different trap types) with a message containing your domain, that will usually cause the domain to be listed.
Other lists and spam filters will have their own methods for this.
My belief is that Google and Microsoft only use AI for this (which is a bit of a nightmare if it gets it wrong).
Not all blocklist providers are the same.