2023-02-21: Intermittent email errors are affecting several products - RESOLVED
On Tuesday, February 21, our third-party email server provider experienced a DDoS attack resulting in email sending errors in several Springy products. They implemented a fix Tuesday night but errors have persisted intermittently.
Affected Apps: LibAnswers, LibCal, LibConnect, LibWizard, LibGuides and LibApps.
If you see an error in LibAnswers about the email server being unreachable when trying to send an email (a ticket reply to a patron, an internal note or a ticket transfer) -- or if your Errors tab is full of error messages to the same effect -- these replies will need to be resent. Jump to this comment for more details.
Updates will be posted in this thread -- the latest updates are on top.
2023-02-27 2:58 PM ET - We have been monitoring email sending performance and it has been operational for some time now. Since performance has been consistent, we won't be providing any further updates unless another degradation occurs. Thank you for your patience as this situation was resolved!
2023-02-24 9:15 AM ET - We seem to be holding steady with no or very few email sending errors. Our mail service provider continues to monitor system performance.
2023-02-23 2:10 PM ET - Email sending has been stable for us on Thursday and for Springy customers, as far as we can tell based on reports. Fingers are crossed that we're coming through this, but we are still awaiting notice of full resolution. Our mail service provider is actively mitigating the attack at this time and are closely monitoring the performance of the system.
2023-02-22 2:50 PM ET - The fix implemented last night was working for several hours, including this Wednesday morning. But this afternoon, we -- us at Springshare, and you our customers -- are starting to report errors again. They are continuing to mitigate the issue.
2023-02-22 11:45 AM ET - The fix implemented by our third-party email service provider has improved email sending significantly for us -- we at Springshare have been able to send messages out, and we are no longer seeing Support tickets from you reporting errors with your own Springy sites. That said, our email service provider is still monitoring and the status is still listed as "degraded." We'll update again once all systems are fully in the clear, or if the performance issues return.
2023-02-21 4:55 PM ET - The initial fix has been implemented by our third-party email service provider, but they are still seeing reports of issues -- this includes us and other Springy customers. They are continuing to investigate.
2023-02-21 1:47 PM ET - Our third-party email service provider has identified the issue (DDos attack) and a fix is being implemented.
2023-02-21 10:00 AM ET - Email Alert: We are aware that some sites are experiencing intermittent email sending errors due to performance issues with our third-party email service provider. We at Springshare are also being affected with the emails we send you in response to Support and other inquiries.
We are keeping a close eye on the situation and apologize for any inconvenience this has caused!