2025-03-17 9:00 AM - US Region - Email Sending for LibAnswers is down - RESOLVED

Springy_Will · 2025-03-17T13:20:32+00:00

There was an error rendering this rich post.

For more information: Please see this linked comment from our VP of Customer Experience for a post-mortem about the incident:

https://lounge.springshare.com/discussion/comment/10630#Comment_10630

Thanks for your patience and for being on board with Springshare!

2025-03-17 2:10 PM: Services for LibAnswers has been restored. There might be a delay in getting emails queued up and sent out, but the problem should resolve itself throughout the next hour or so.

If we need to communicate further about the situation, we will provide those updates on this post.

2025-03-17 9:00 AM - Email Sending for LibAnswers is down:

Hi all, we're having an issue this morning with our email utility in LibAnswers in the US region. If you use LibAnswers, your replies to patrons and receiving emails will likely be impacted. Our developers are working hard to get this back up and running ASAP.

Find more posts tagged with

LibAnswers

Announcements/Page Alerts

Comments

laura_percival

Just to say we've had a few issues here in the EU region as well - a few bounces when forwarding emails into a queue, and taking longer than usual to appear when no bounce.

Beu

South Africa as well since this morning.

dbottorff

Any update on this? In Chicago we're seeing what looks like the emails starting to work, but don't want to assume this has been resolved.

Springy_Will

@dbottorff @Beu @laura_percival

This issue should be resolved now. Please give our system some time to catch up with all the emails that might've been queued up by the system. :)

lampley

Does this mean we don't need to resend emails?

bjuhl

yes, do we need to resend messages that wound up in the "errors" queue - thanks!

Springy_Will

Hi @lampley

The system should try to do that on its own. It might take a bit depending on how clogged up the queue is for sending these out, but they will go out.

jlk4p

What about incoming email that would have created new tickets? Are those queued up to create new tickets?

We have a circulation email account that forwards messages to LibAnswers to create tickets. During the outage we had messages coming into the mailbox that didn't get into LibAnswers. We are wondering if those will still get generated if we give it time.

Springy_Anna

Hello everyone,

Today we experienced an issue with our email sending provider which affected LibAnswers' ability to send and receive emails.

Email sending problems occurred between 6am and 1:36pm U.S. Eastern time and affected sites on US servers (which include customers based in Central & South America and Africa) using our default sending method.
Email receiving problems occurred between the hours of 9:48am - 12:48pm U.S. Eastern time and affected all LibAnswers sites in all regions.
No email data was lost during this time.

What happened?
Our email service provider deactivated sending from our libanswers.com domain, claiming the Sinch SMTP credentials relating to that domain had been compromised. There is no evidence of this, and it is not a mechanism we use or have ever used. We do not have those credentials on any of our servers or configurations. Therefore, it is not possible for us to have leaked it.

As a cautionary measure we rotated all of our API keys and user account credentials (all of which have MFA activated) before prompting the vendor to reactivate the domain. As of approximately 1:36pm U.S. Eastern, all email processes are working again.

What about emails sent / received during this time?
The best news in all of this is that no emails were lost, either sent or received. In fact, as of approximately 2:30pm U.S. Eastern, all queued emails from LibAnswers were sent. We are working as quickly as possible to retrieve all received emails from our provider's servers.

We are also working with our email provider to ensure this never happens again. Once again, rest assured that no data was compromised and no emails were lost. We are very sorry you had to experience this delay today, due to our email provider's actions.

As always, we appreciate you being on board!
Anna

Anna Burke
VP, Customer Experience

Lou_R

Hi,

It has been a day, I was expecting the emails that got sorted into the ERRORS tab would no longer be there and be fixed, but it still has not.

Any advice/updates? Should we just manually resend everything?

New York location

jlk4p

Hi,

I can also report that someone at our institution mentioned that received emails (that we know should have come in) never did. I think there were about 12 incoming messsages as of 3pm that still haven't come in to LibAnswers. That person also noticed emails still sitting in the ERRORS tab and they wound up resending them because there were only 5 there.

Charlottesville location

Springy_Will

@Lou_R -

The emails should resend themselves but the error won't clear out on its own. If you see a little "Emailed to:" grey box at the bottom of your ticket reply, that likely means the reply has gone out.

@jlk4p - I just learned this that we're working on getting the emails that are stored within Mailgun to get released to the LibAnswers system. They're not lost, just kind of limbo while our develpers work to get these emails queued up.

Lou_R

@Springy_Will, I see the "emailed to" at the bottom, but the error message is after it so it's a bit confusing, see below.

Springy_Anna

Hi @Lou_R - those error messages are the ones Will mentions won't go away on their own. All emails that had been queued from LibAnswers were sent out by about 2:30pm Eastern on Monday. It's just that those errors persisted in the tickets / systems. We're really sorry for the confusion!

Springy_Anna

Hello all, the following email just went out to all LibAnswers Admins in the US region from our Chief Springy, Slaven, who asked that I also post it here. (The US region was the only region affected in the details below, which is why it was only sent to those admins.)

- Anna

On Monday, March 17, 2025 at 6am EDT we were notified by our email service provider, Sinch, that they had deactivated our libanswers.com domain for security reasons. As you know, Springshare is very security conscious, so this was a shock to all of us. We immediately began to ask questions of them, investigate their claims, and take required steps for reactivation of our domain. We rotated all of our API keys and user account credentials (all of which have had MFA enabled) and quickly prompted Sinch to reactivate the domain. Once they finally did so at 1:36pm EDT, all email processes resumed without issue.

As we noted in our post mortem on the Springshare Lounge (above), there is no evidence there was a security leak nor have we ever used the SMTP credentials Sinch said were compromised. We are still in conversation with them to understand why they shut down our domain with no basis and no notice. Springshare has been an exemplary customer for over a decade with no previous concerns, security or otherwise, so we are both shocked and upset by this situation.

Once the domain was active again, we were able to successfully send out all emails queued from LibAnswers during that outage period. Unfortunately, as we were working to restore emails coming into the system during that period, we found none. Typically emails are saved on their servers for 3 days, which we thought would give us an opportunity to restore incoming emails. However, Sinch confirmed that deactivating the domain not only stopped email sending, but also receiving.

This means that between 6am EDT and 1:35pm EDT on Monday, March 17, 2025, any emails sent to LibAnswers systems on US servers were rejected by Sinch's servers, so we cannot restore them to the LibAnswers systems.

Please understand that there was no data lost, as rejected emails would still be on the sending side and could be resent by the senders. But Springshare cannot do anything to fetch or restore them as that is out of our control.

What does this mean for you?

Any replies or new tickets emailed to your systems during the above time frame were rejected by our provider, therefore they cannot be restored into your system.
We cannot tell how many messages were sent in to the system, nor from whom, since our provider did not store that data.
Any emails sent to LibAnswers systems in that time frame got an error message back from our provider saying: "550 5.7.1 Relaying denied". What that email server did with that error message depends on how that email server is configured.
- It is possible that the email server tried to send the message again later.
- It is also possible that the email server sent that error along to the person writing in.
- It is equally possible that the email server did nothing.
If you are forwarding email from your own email system, there may be an opportunity to check your own email server logs / inboxes to see if any of the email / data about the email sent to your system during that time is recoverable by your IT department.
Alternately, you may want to consider posting a message to your community to request that anything they sent during that time frame be resent.

As mentioned before, we are very upset about this entire situation and are continuing to discuss with Sinch what led to our domain being deactivated. We are also investigating what our options are to ensure something like this never happens again, including alternate providers. We know you trust us to ensure our systems are working smoothly and this has damaged that trust, just as our trust in our provider has been damaged. We assure you we will continue to have the appropriate conversations, investigations, planning, and work done to ensure that LibAnswers email sending is stronger than ever.

As always, we thank you for being Springshare customers and we're here for you whenever you need us.

Sincerely,

Slaven Zivkovic
Chief Springy
Springshare

lousiekat

Hi @Springy_Anna - we're in Australia and when I read the announcements on Springshare Lounge this week my understanding was that we were unaffected.

The analysis email received today advised that the AU region was affected, and indeed I can see that a ticket we were concerned about now looks normal thanks to the mitigation actions. However, we had already reported this ticket to our University IT because we are dealing with the aftermath of a cyber incident from January and are hyper-vigilant. I will now have to explain to them what happened, and hopefully they haven't restricted the account of the employee who created the affected ticket.

I have a few questions:

When was it identified that AU region was affected and why this was not included in the original announcements, only the "post-mortem"?
What will be done to ensure that affected regions are fully identified in the future?
The resolution notification was sent to all LibAnswers admins via email. In future can we please receive the announcement notifying of the incident via email as well? I don't log onto the Springshare Lounge every day.

Springy_Will

Hi @lousiekat

Thanks your questions. We'll follow up with you via a ticket so that our LibAnswers product manager can dive into your questions.

Best,

Will - Lounge Admin, Springy Support