Hello all, the following email just went out to all LibAnswers Admins in the US region from our Chief Springy, Slaven, who asked that I also post it here. (The US region was the only region affected in the details below, which is why it was only sent to those admins.)

- Anna

On Monday, March 17, 2025 at 6am EDT we were notified by our email service provider, Sinch, that they had deactivated our libanswers.com domain for security reasons. As you know, Springshare is very security conscious, so this was a shock to all of us. We immediately began to ask questions of them, investigate their claims, and take required steps for reactivation of our domain. We rotated all of our API keys and user account credentials (all of which have had MFA enabled) and quickly prompted Sinch to reactivate the domain. Once they finally did so at 1:36pm EDT, all email processes resumed without issue.

As we noted in our post mortem on the Springshare Lounge (above), there is no evidence there was a security leak nor have we ever used the SMTP credentials Sinch said were compromised. We are still in conversation with them to understand why they shut down our domain with no basis and no notice. Springshare has been an exemplary customer for over a decade with no previous concerns, security or otherwise, so we are both shocked and upset by this situation.

Once the domain was active again, we were able to successfully send out all emails queued from LibAnswers during that outage period. Unfortunately, as we were working to restore emails coming into the system during that period, we found none. Typically emails are saved on their servers for 3 days, which we thought would give us an opportunity to restore incoming emails. However, Sinch confirmed that deactivating the domain not only stopped email sending, but also receiving.

This means that between 6am EDT and 1:35pm EDT on Monday, March 17, 2025, any emails sent to LibAnswers systems on US servers were rejected by Sinch's servers, so we cannot restore them to the LibAnswers systems.

Please understand that there was no data lost, as rejected emails would still be on the sending side and could be resent by the senders. But Springshare cannot do anything to fetch or restore them as that is out of our control.

What does this mean for you?

• Any replies or new tickets emailed to your systems during the above time frame were rejected by our provider, therefore they cannot be restored into your system.
• We cannot tell how many messages were sent in to the system, nor from whom, since our provider did not store that data.
• Any emails sent to LibAnswers systems in that time frame got an error message back from our provider saying: "550 5.7.1 Relaying denied". What that email server did with that error message depends on how that email server is configured.
  • It is possible that the email server tried to send the message again later.
  • It is also possible that the email server sent that error along to the person writing in.
  • It is equally possible that the email server did nothing.
• If you are forwarding email from your own email system, there may be an opportunity to check your own email server logs / inboxes to see if any of the email / data about the email sent to your system during that time is recoverable by your IT department.
• Alternately, you may want to consider posting a message to your community to request that anything they sent during that time frame be resent.

As mentioned before, we are very upset about this entire situation and are continuing to discuss with Sinch what led to our domain being deactivated. We are also investigating what our options are to ensure something like this never happens again, including alternate providers. We know you trust us to ensure our systems are working smoothly and this has damaged that trust, just as our trust in our provider has been damaged. We assure you we will continue to have the appropriate conversations, investigations, planning, and work done to ensure that LibAnswers email sending is stronger than ever.

As always, we thank you for being Springshare customers and we're here for you whenever you need us.

Sincerely,

Slaven Zivkovic
Chief Springy
Springshare