Have some feedback for the Lounge? Let us know in our Springshare Lounge Feedback Survey here!

2023-02-21: Intermittent email errors are affecting several products - RESOLVED

Options
Springy_Meg
Springy_Meg Member, Administrator, Moderator, Springy, SpringyCamp Counselor admin

On Tuesday, February 21, our third-party email server provider experienced a DDoS attack resulting in email sending errors in several Springy products. They implemented a fix Tuesday night but errors have persisted intermittently.

Affected Apps: LibAnswers, LibCal, LibConnect, LibWizard, LibGuides and LibApps.

If you see an error in LibAnswers about the email server being unreachable when trying to send an email (a ticket reply to a patron, an internal note or a ticket transfer) -- or if your Errors tab is full of error messages to the same effect -- these replies will need to be resent. Jump to this comment for more details.

Updates will be posted in this thread -- the latest updates are on top.

2023-02-27 2:58 PM ET - We have been monitoring email sending performance and it has been operational for some time now. Since performance has been consistent, we won't be providing any further updates unless another degradation occurs. Thank you for your patience as this situation was resolved!

2023-02-24 9:15 AM ET - We seem to be holding steady with no or very few email sending errors. Our mail service provider continues to monitor system performance.

2023-02-23 2:10 PM ET - Email sending has been stable for us on Thursday and for Springy customers, as far as we can tell based on reports. Fingers are crossed that we're coming through this, but we are still awaiting notice of full resolution. Our mail service provider is actively mitigating the attack at this time and are closely monitoring the performance of the system.

2023-02-22 2:50 PM ET - The fix implemented last night was working for several hours, including this Wednesday morning. But this afternoon, we -- us at Springshare, and you our customers -- are starting to report errors again. They are continuing to mitigate the issue.

2023-02-22 11:45 AM ET - The fix implemented by our third-party email service provider has improved email sending significantly for us -- we at Springshare have been able to send messages out, and we are no longer seeing Support tickets from you reporting errors with your own Springy sites. That said, our email service provider is still monitoring and the status is still listed as "degraded." We'll update again once all systems are fully in the clear, or if the performance issues return.

2023-02-21 4:55 PM ET - The initial fix has been implemented by our third-party email service provider, but they are still seeing reports of issues -- this includes us and other Springy customers. They are continuing to investigate.

2023-02-21 1:47 PM ET - Our third-party email service provider has identified the issue (DDos attack) and a fix is being implemented.

2023-02-21 10:00 AM ET - Email Alert: We are aware that some sites are experiencing intermittent email sending errors due to performance issues with our third-party email service provider. We at Springshare are also being affected with the emails we send you in response to Support and other inquiries.

We are keeping a close eye on the situation and apologize for any inconvenience this has caused!

Comments

  • adeutschler
    adeutschler Member Springy Novice
    Options

    Thanks for this alert! Do you know if emails sent during this period need to be re-sent or will they be sent after the fix is in place? We've seen this error quite a bit today & I want to make sure our patrons actually get their replies.

  • JuliePompo
    JuliePompo Member Springy Initiate
    Options

    We are seeing the error again. It seemed to be fine for awhile but not anymore.

  • Jennie
    Jennie Member Springy Initiate
    Options

    I have just hit the error again, it was ok, but now it's not.

    https://massey.libanswers.com

  • Springy_Will
    Springy_Will Member, Administrator, Moderator, Springy, SpringyCamp Counselor admin
    Options

    @Jennie I'm sorry for the continued issues. This issue is still ongoing, and trust me when I say it's probably just as annoying for you as it is for us. 😭

  • katyp
    katyp Member Lab Coat
    Options

    Hey Springy Peeps,

    I wanted to update you on our problem. It's just slightly different.

    On Tuesday, we received a reference ticket in LibAnswers (submitted through our website and Ask Us form), but no email came through to let us know that there was a ticket. When I sent test tickets through later that afternoon, they came through fine and everyone received the email notification.

    I'm sure it is related to this same issue and it only happened the one time (we don't get tons of tickets). However, since it is a slightly different thing that is happening, I wanted to give you a heads up.

    Thanks for your awesomeness!

    Katy

  • Springy_Meg
    Springy_Meg Member, Administrator, Moderator, Springy, SpringyCamp Counselor admin
    Options

    Hi, @katyp,

    Indeed, this is related to the same issue. 😕 Email notifications configured to auto-send when a new ticket arrives in LibAnswers go through our mail server, so these would be affected as well (in the form of, simply not showing up in your inboxes). I'm pretty sure there's not a way to manually resend these, so you'll want to keep an eye on your queue(s) while this issue persists so you're aware of what comes in and can manually send other staff notifications if needed.

    What a week! 🎢 😅 Thank you all for your patience. Today -- Thursday -- has been much better, and our fingers are crossed we'll get the "All clear!" status update soon.

  • lampley
    lampley Member Lab Coat
    Options

    We're seeing new errors today. Here's an example: "Jan 30 2023, 03:29pm | Error Reply to Questioner #21514852 Failed to send to <removedforprivacy>@nb27.com, please retry | 612: unable to connect to MX servers: - hk.file.myqcloud.com: connection failed: while dialing: dial tcp4 10.3.98.191:0->43.132.105.108:25: i/o timeout"

    I've seen MX errors before, but never like this. Assuming related.

  • Springy_Will
    Springy_Will Member, Administrator, Moderator, Springy, SpringyCamp Counselor admin
    Options

    @lampley

    MX errors are usually a sign of a different problem involving the patron's email provider. I don't think those are related to the current issues that we're having. Feel free to submit a ticket through our Ask site, and we can take a deeper dive into it.

  • lampley
    lampley Member Lab Coat
    edited February 2023
    Options

    That's what I thought, but since this error had a lot more detail to it than I usually see for MX errors I thought I'd throw it in :) Thanks!

This discussion has been closed.