Latest Update:
Thu Jan 21 23:45 2016 UTC:
As of 23:45 UTC the backlog of queued email that had accumulated on the affected server has been re-delivered to original recipients.Previous Updates:
Thu Jan 21 23:00 2016 UTC:
Following our investigations, we believe the cause of today's email issues were as follows:When negotiating an SMTP transaction, MIDAS doesn't explicitly supply an "HELO" value, as ordinarily this value is determined by the host server itself and automatically added to any SMTP transactions.
However, it appears there's a bug in a Perl module which MIDAS users to conduct SMTP transactions (Net::SMTP) whereby if a "HELO" value is not explicitly set in the code, instead of determining the correct value from the server, the module simply defaults to a value of "localhost.localdomain".
In the vast majority of cases, this doesn't cause any issue/side affects with the sending of mail via SMTP, however, strictly speaking a HELO value of "localhost.localdomain" is invalid as it doesn't conform an RCF standard (RFC5321).
It appears that an email blocklist had picked up on this HELO value being transmitted as part of SMTP transactions originating from one of our client servers, and based on this value also being prevalent in spam bots, assumed our client server was also engaged in spam.
As of 20:46 UTC we have pushed an update to all our hosted clients to now explicitly set a "HELO" value when initiating SMTP transactions (this update is now also available to self-hosted customers via the usual update route).
As mentioned earlier, as of 20:52 UTC the affected client server has been de-listed from the blocklist, however it may take a few hours before mail servers around the global retrieve a fresh copy of the blocklist with the client server's IP address removed, in the meantime, the client server in question will attempt to re-deliver messages that have currently accumulated in its queue.
We sincerely apologize for any inconvenience this may have caused earlier today, and thank you for your patience as the server works to attempt re-deliver of emails it was unable to send earlier.
Thu Jan 21 22:30 2016 UTC:
The cause of this issue was due to one of our client servers becoming listed on a popular email blocklist earlier today.The server has since been de-listed and removed from this particular blocklist as of 20:52 UTC, however it may take several more hours before mail servers around the global retrieve a fresh copy of the blocklist with the client server's IP address removed.
Further technical details/specifics on the nature of this issue will follow shortly...
Thu Jan 21 21:05 2016 UTC:
The issue which caused the earlier delays in some outgoing email has now been resolved, however there are still delays as we work through the backlog of queued email that has accumulated since this issue began.Further updates to follow...
Thu Jan 21 20:24 2016 UTC:
The probable cause of the issue has now been identified, and we're working on a resolution...Further updates to follow...
Thu Jan 21 16:20 2016 UTC:
We've detected an elevated rate of emails sent through our client servers being delayed or rejected, and we're currently investigating.Customers who have configured their MIDAS systems to send email via their own external SMTP relay/server appear unaffected, and this issue appears to only affect some customers who have configured their hosted MIDAS systems to send email via the built-in "Sendmail" option.
Further updates will follow...