2023-02-21: Intermittent email errors are affecting several products - RESOLVED

Springy_Meg · 2023-02-21T15:23:37+00:00

There was an error rendering this rich post.

On Tuesday, February 21, our third-party email server provider experienced a DDoS attack resulting in email sending errors in several Springy products. They implemented a fix Tuesday night but errors have persisted intermittently.

Affected Apps: LibAnswers, LibCal, LibConnect, LibWizard, LibGuides and LibApps.

If you see an error in LibAnswers about the email server being unreachable when trying to send an email (a ticket reply to a patron, an internal note or a ticket transfer) -- or if your Errors tab is full of error messages to the same effect -- these replies will need to be resent. Jump to this comment for more details.

Updates will be posted in this thread -- the latest updates are on top.

2023-02-27 2:58 PM ET - We have been monitoring email sending performance and it has been operational for some time now. Since performance has been consistent, we won't be providing any further updates unless another degradation occurs. Thank you for your patience as this situation was resolved!

2023-02-24 9:15 AM ET - We seem to be holding steady with no or very few email sending errors. Our mail service provider continues to monitor system performance.

2023-02-23 2:10 PM ET - Email sending has been stable for us on Thursday and for Springy customers, as far as we can tell based on reports. Fingers are crossed that we're coming through this, but we are still awaiting notice of full resolution. Our mail service provider is actively mitigating the attack at this time and are closely monitoring the performance of the system.

2023-02-22 2:50 PM ET - The fix implemented last night was working for several hours, including this Wednesday morning. But this afternoon, we -- us at Springshare, and you our customers -- are starting to report errors again. They are continuing to mitigate the issue.

2023-02-22 11:45 AM ET - The fix implemented by our third-party email service provider has improved email sending significantly for us -- we at Springshare have been able to send messages out, and we are no longer seeing Support tickets from you reporting errors with your own Springy sites. That said, our email service provider is still monitoring and the status is still listed as "degraded." We'll update again once all systems are fully in the clear, or if the performance issues return.

2023-02-21 4:55 PM ET - The initial fix has been implemented by our third-party email service provider, but they are still seeing reports of issues -- this includes us and other Springy customers. They are continuing to investigate.

2023-02-21 1:47 PM ET - Our third-party email service provider has identified the issue (DDos attack) and a fix is being implemented.

2023-02-21 10:00 AM ET - Email Alert: We are aware that some sites are experiencing intermittent email sending errors due to performance issues with our third-party email service provider. We at Springshare are also being affected with the emails we send you in response to Support and other inquiries.

We are keeping a close eye on the situation and apologize for any inconvenience this has caused!

Find more posts tagged with

Comments

adeutschler

Thanks for this alert! Do you know if emails sent during this period need to be re-sent or will they be sent after the fix is in place? We've seen this error quite a bit today & I want to make sure our patrons actually get their replies.

Springy_Meg

Hi, Everyone,

As @adeutschler suggests, after sending an email in LibAnswers -- whether a reply to a customer, an internal note, or a ticket transfer -- you may have seen an error message indicating that email servers were unreachable.

I'm afraid any emails that showed this message will need to be manually resent. 😕 Here's how you can tell if an email was successful and what to do if it was not. Please note, as of my writing this (2/21, at 5:40 PM ET), the email service provider's status is still "degraded" -- your attempt to resend at this point may be successful, or it might fail again.

To see if an email was successfully sent, open the ticket and view the staff reply:

To resend the email,

Click the gear icon in the upper right of the staff reply.
Select Resend.
Enter the patron's email address.
Click Add Email.
Click Resend Email.

Then, verify that the latest attempt to reply was successful by checking for the gray bar at the bottom of the reply with the text, Emailed to: [patron details]. If you don't see this gray bar and text, it was not successful, and you can try the same steps again later.

We're terribly sorry for the inconvenience here! We'll continue to post updates here as we have them.

JuliePompo

We are seeing the error again. It seemed to be fine for awhile but not anymore.

Springy_Meg

@JuliePompo, we're seeing the same. 😞 I've updated the Lounge alert with the latest, along with the top post in this thread.

Jennie

I have just hit the error again, it was ok, but now it's not.

https://massey.libanswers.com

Springy_Will

@Jennie I'm sorry for the continued issues. This issue is still ongoing, and trust me when I say it's probably just as annoying for you as it is for us. 😭

katyp

Hey Springy Peeps,

I wanted to update you on our problem. It's just slightly different.

On Tuesday, we received a reference ticket in LibAnswers (submitted through our website and Ask Us form), but no email came through to let us know that there was a ticket. When I sent test tickets through later that afternoon, they came through fine and everyone received the email notification.

I'm sure it is related to this same issue and it only happened the one time (we don't get tons of tickets). However, since it is a slightly different thing that is happening, I wanted to give you a heads up.

Thanks for your awesomeness!

Katy

Springy_Meg

Hi, @katyp,

Indeed, this is related to the same issue. 😕 Email notifications configured to auto-send when a new ticket arrives in LibAnswers go through our mail server, so these would be affected as well (in the form of, simply not showing up in your inboxes). I'm pretty sure there's not a way to manually resend these, so you'll want to keep an eye on your queue(s) while this issue persists so you're aware of what comes in and can manually send other staff notifications if needed.

What a week! 🎢 😅 Thank you all for your patience. Today -- Thursday -- has been much better, and our fingers are crossed we'll get the "All clear!" status update soon.

lampley

We're seeing new errors today. Here's an example: "Jan 30 2023, 03:29pm | Error Reply to Questioner #21514852 Failed to send to <removedforprivacy>@nb27.com, please retry | 612: unable to connect to MX servers: - hk.file.myqcloud.com: connection failed: while dialing: dial tcp4 10.3.98.191:0->43.132.105.108:25: i/o timeout"

I've seen MX errors before, but never like this. Assuming related.

Springy_Will

@lampley

MX errors are usually a sign of a different problem involving the patron's email provider. I don't think those are related to the current issues that we're having. Feel free to submit a ticket through our Ask site, and we can take a deeper dive into it.

lampley

That's what I thought, but since this error had a lot more detail to it than I usually see for MX errors I thought I'd throw it in :) Thanks!