• Hi all and welcome to TheWoodHaven2 brought into the 21st Century, kicking and screaming! We all have Alasdair to thank for the vast bulk of the heavy lifting to get us here, no more so than me because he's taken away a huge burden of responsibility from my shoulders and brought us to this new shiny home, with all your previous content (hopefully) still intact! Please peruse and feed back. There is still plenty to do, like changing the colour scheme, adding the banner graphic, tweaking the odd setting here and there so I have added a new thread in the 'Technical Issues, Bugs and Feature Requests' forum for you to add any issues you find, any missing settings or just anything you'd like to see added/removed from the feature set that Xenforo offers. We will get to everything over the coming weeks so please be patient, but add anything at all to the thread I mention above and we promise to get to them over the next few days/weeks/months. In the meantime, please enjoy!

IMAP email question

RogerS

RogerS

Moderator
Staff member
Joined
Jul 21, 2014
Messages
13,886
Reaction score
170
Location
Blimey! Actually finished.
I got several of these today in quick succession...relating to an email sent to the same email address.

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

redacted@btinternet.com
Domain redacted.co.uk has exceeded the max defers and failures per hour (5/5 (83%)) allowed. Message discarded.

After talking to my hosting company (Zen) they told me that it was an automatic block placed on my domain because the same email address had been sent to in quick succession (or somesuch verbage). Further discussion elicited that the destination server was not responding properly but my email client (Apple Mail) kept on resending. And because of this resending (the rate, the number within a defined timeframe...dunno) the threshold of Max Defers was reached and the outgoing server raised a block to prevent my domain sending out any more emails for an hour.

To be honest, I wasn't really that impressed with the understanding and comprehension of this issue by Zen support. For example, if there was a null response from the btinternet server then why wasn't that passed 'up the chain' to my email client ?

In the past, whenever there has been an email transmission issue I have received a message from the other server saying that it couldn't deliver because of x, y or z. I got nothing in this case.

Zen told me I 'should change the settings in my email client to refrain from resending in the event of an error' and that it was nothing to do with them, that I should speak to Apple. Apple tell me that it is down to Zen. Piggy in the middle time.

In fact as I typed in that last paragraph I was thinking 'that lass at Zen didn't have a Scoobie' but I am not an IMAP expert.

So.....help please !
 
I randomly have, perhaps 2-3 times a year, occasions where my Mac will steadfastly refuse to send emails from my long established hotmail account and its associated SMTP server. When it decides to do this the emails appear to sit in my Outbox, but I get a slew of those responses you mention above.

To be honest I don'r know what fixes it, but I generally tend to end up doing LOTS of reboots, deleting the Outlook SMTP server, rebooting again then recreating the SMTP server with the exact same (correct) settings it had before, and miraculously it just starts working again and is fine for ages.

Might be worth giving that a shot.
 
I had another try with Zen support and this time got someone who was bit more clued up. It's a massive vindication of Roger's First Law in action !

There are two 'systems' at work. The first are the IMAP protocols for sending and receiving emails. Usual method - send and wait for a reply - process reply - DoNext. If for whatever reason the receiving server cannot handle the incoming email - invalid recipient, for example - then a suitable response will be returned to the sending server which processes the response etc. The end result can be one of those cryptic email messages you get ..."recipients inbox is full", for example. But.......What If!

What if the receiving server does not reply because it is overloaded ? What does the sending server do? Why, it sends it again. And again. And again. And which is when it all goes pear-shaped.

Because most (if not all) outgoing email servers can be configured to monitor whether the (same?) email is being sent in rapid succession to the same email address within a specified timeframe. If so then the server is configured to conclude that emails coming from that domain are spam. I can't really see the logic in that but there you go. So it puts a softlock on the domain for AN HOUR during which time, no emails can be sent. To anyone. Time critical email needing to be sent ? You are so screwed.
 
Lons said:
Was that only with the ones you sent to my BT account Roger?
Click to expand...
Yup. But the soft hold applied to all outgoing emails for an hour. If time was of the essence I’d have been seriously hacked off. Maybe Pukin’s trolls were trying a DDoS attack on btinternet.
 
I'm not aware of anyone else having issues sending to my address roger and I seem to be sending and receiving as normal. Strange!
 
Lons said:
I'm not aware of anyone else having issues sending to my address roger and I seem to be sending and receiving as normal. Strange!
Click to expand...
Not strange at all. These things happen all the time and are relatively transient until the BT server boys put some more petrol in
 
I can remember a time when someone, far more technically informed than I am, was explaining to me how the "Simple" bit in SMTP means that it's full of vulnerabilities like that. You could add the lack of reliable delivery and read receipts and the ease of address spoofing to the list, along with the lack of encryption and probably many more that I can't remember.
It's a classic case of how something that is "just good enough to be useful" prevents adoption of something better specified but harder to set up. (In this example, my colleague was expounding the virtues of the X.400 message handling standard, which was designed for secure exchange of data between systems. An excellent choice for that at the time, but not for the "electronic postcards" that SMTP carries.)
How will the world ever manage to upgrade something in such widespread use?
 
RogerS said:
Because most (if not all) outgoing email servers can be configured to monitor whether the (same?) email is being sent in rapid succession to the same email address within a specified timeframe. If so then the server is configured to conclude that emails coming from that domain are spam. I can't really see the logic in that but there you go. So it puts a softlock on the domain for AN HOUR during which time, no emails can be sent. To anyone. Time critical email needing to be sent ? You are so screwed.
Click to expand...
This is referred to as greylisting, and there's a reason behind it. A spec-compliant mail transfer agent (MTA, or to most people just an email server) will have no problem handling a "not now, try again in an hour" response from a destination server. It's already got a store of mails in transit waiting to be relayed or delivered, so it just holds the message there with a retry time attached to it.

A lot of spammers, though, don't want to bother implementing the entire SMTP protocol; they just want to send out as many messages as possible as fast as possible. Those programs will treat a temporary failure the same way as any other failure, by just moving on to the next address. So, if a mail server sees a sending server behaving in a way that looks a bit odd but isn't enough to decide it's definitely spam, then instead of blacklisting a spamming server it'll greylist it - send temporary failures for a defined period of time, then go back to accepting messages. Anything that actually bothers to retry the delivery is probably a real message and not a spambot.
 
So...this is what Zen just told me and I have no idea whether what they are saying is correct.

They host my domain...I send an email from my domain and that goes to the Zen server which then sends it on to the recipients server. In this case btinternet.com.

Zen say that their server is configured to guard against abuse such as spam etc. So if I was to send an email from my domain multiple times within a short space of time then the Zen server will automatically pick this up, assume 'abuse' and create a MaxDef (or something sounding similar) which blocks any more emails from my domain for an hour. This I can understand.

However, in this instance, I only sent the one email from my domain to the Zen server. That email got queued and the Zen server tried to send it to btinternet.com. But because at that moment in time the btinternet server was overloaded, the Zen server didn't get any response and so it queued the email and sent it again. Same lack of response, same requeue. And again. And again. Note that my domain is not initiating any of this constant retry. Part of the way the mail system works. So far..so good.

But. After x retries the Zen server declares that my domain is at fault and issues the MaxDef block. Which to my mind is wrong. It is their server that has done all the retries.

Zen tell me that that is the way it works, that nothing can be tweaked etc etc blah blah blah.

Is that correct ?
 
OK, so it now sounds like there's something more complex going on.

It looks like something about that message has triggered Zen's spam defences. It's got a rule in place that if a single domain is sending out too many (for some arbitrary definition of 'too many') messages that get 'try again later' responses from the destination server, then it discards those messages and blocks sending for a time. From their perspective, this behaviour makes sense - if they're relaying messages which are causing them to get greylisted by lots of destination servers, that will affect the reputation of every message they handle or relay, not just those from that single domain but from all their customers. That's obviously bad for business.

The part that isn't clear is how a single outgoing email could have triggered that. Either it's doing its normal 'retry after a time' behaviour but btinternet thinks it's doing it too fast, or their original comment is correct and somehow your client is trying to send it multiple times. I can't see how the latter would be the case, though, because once Zen has accepted the message for relaying, as far as your client knows it's been successfully sent and there's no reason to retry anything - this is why bounce messages arrive as a new email, because by the time it reaches that point your client considers it sent and has long ago finished with the original message.

Ultimately, I don't think there's much to do right now except to sanity check your mail client settings, make sure there isn't anything telling it to try sending repeatedly, and wait to see whether it happens again. It's possible that this was just an unfortunate confluence of several minor things happening at once, and that it's a one-off.

Email is a complex beast - as Andy mentioned, the protocol was originally designed to be 'simple', for a simpler time on a simpler Internet, and the unfortunate subsequent discovery of people being less than excellent to each other has resulted in an enormous growth both of extra protocols layered on top and of weird counter-intuitive behaviour that's nonetheless become industry standard for abuse mitigation reasons. Since these things weren't all designed at the same time or by the same people, sometimes they interact in unfortunate ways at the edge cases, and we pretty much just have to shrug and carry on. The distributed nature of it only adds to the complexity; diagnosing with any confidence exactly what happened in this instance would need access to Zen's mail server logs, and possibly btinternet's as well, which they're not about to give out.
 
spb said:
.The part that isn't clear is how a single outgoing email could have triggered that. Either it's doing its normal 'retry after a time' behaviour but btinternet thinks it's doing it too fast, or their original comment is correct and somehow your client is trying to send it multiple times.
Click to expand...
But there was no response from btinternet as their server was overloaded. There would appear to be no thought given in the protocol as to 'What If' there is no response. Really should be two outgoing queues - one for nomal stuff and the other to park temporarily emails where there is no response from the other end. Not keep retrying and bashing its head against a brickwall until it triggers MaxDef and - unjustifiably - blocks outgoing emails from the originating mail server. Not that difficult to design.

Apple Mail client does not have the facility to resend (unless done by a human)
 
No response, or network problems meaning it can't be reached at all, are temporary failures. It'll go straight back into the 'retry later' queue, same as if the destination server gave it a temporary failure response. The server might have different retry times configured for different failure modes, though - I suppose it's possible that it's configured to retry every, say, 10 minutes with no backoff for a network failure, in which case if the disruption lasts more than 40 minutes then it'll hit the "five defers in an hour" threshold.

It'd be an unfortunate oversight if it allowed the same message to count multiple times against that limit, but I've seen much worse than that in production software.
 
Many thanks, Stephen. The 'retry later' queue ios exactly what I was suggesting and it sounds as if some mail servers will have this in place.

No such luck in the case of Zen, because it went straight back into the "send again ASAP" queue and so MaxDef was reached rather quickly.

I'm actually tempted to DIY it. Now what can go wrong !
 
Every email server has that 'retry later' queue - the question is just how long it waits before retrying; if that's too short a time then it can easily still trigger these sorts of rate limits. If the limit is 'five in an hour', then there's little difference from the outside between 'straight away' and 'in ten minutes'.

In all honesty, if there's one piece of online infrastructure not to try to run yourself, it's email. There are just too many extra bits that aren't technically required but will get you thrown in the spam bin if you don't get them exactly right (SPF is simple enough in a cookbook sort of a way, but DKIM is another beast entirely), and even if you set up everything perfectly on the first time, there are just too many mail servers out there that will treat any mail server it hasn't seen many times before as potential spam, or that subscribe to big 'reputation-based' lists of IP addresses that will also get you treated as spam for any or no reason. Given that one of those is Microsoft, which includes any company that uses Microsoft 365 for email, it's just headache after headache.
 
You must log in or register to reply here.
Back
Top