Ignas3 min read Behind the Scenes October 13, 2017

What happened on the nightmare Monday?

What happened on the nightmare Monday?

We made a mistake and lost some important data from the Automation Workflows. The lost data tracked the specific workflow steps a subscriber was on.

It didn’t affect subscribers who already completed the workflow. However, those who were in the queue no longer had data connected with the step they were currently on.

We’re truly sorry. We know how important the automation is for your business. We understand that our mistake might have caused lost sales or other damage. We promise to do everything possible to prevent this situation from happening in the future.

Here are more details:

How did it happen?

  • On Oct 9 an update for better system performance was released.
  • It caused some Automation Workflows to stop.
  • Our developers reacted fast and started fixing the issue.
  • However, they made a mistake in a database query that deleted some data which tracked the workflow steps of queued subscribers.
  • We tried to restore the data from backups. We do backups every day and save them for a week. However, it turned out that backups exactly for that database silently failed. It meant we didn’t have the data to restore.

What did we do next?

  • It took us almost 24 hours to try out all possible solutions data restoration approaches. We managed to restore around 50% of affected automation workflows based on the last email sent to the subscriber. However, we could not restore longer and more difficult workflows.
  • We decided to stop all automations that we couldn’t restore. This prevented the risk of sending all workflow emails from the beginning to the subscribers who were somewhere in the middle of the workflow.
  • We informed all the accounts (that had their automations stopped) about the issue, and the steps they could take to make sure their automations continue to work. Here’s the email we sent.
  • Our recommendation was to make a copy of the automation workflow and turn it ON. This guarantees that your workflows work well for all new subscribers.
  • We continue to help everyone who needs our assistance to solve this issue. In which case please contact us by live chat or email (info@mailerlite.com). We will investigate your account and determine if it’s possible to restore some of the stuck subscribers. Each workflow is unique and can have not only emails but other actions and conditions, this is why it needs individual approach.

What are we doing to prevent such cases in the future?

  • First of all, we already fixed the backups issue.
  • Now we’re working on a solution for better backup monitoring so that a silent backup failure situation never happens again.
  • Starting today our development team is switching to hackathon mode and will work through the weekend to improve our testing environments and procedures.

Thank you for your trust!

In the last few days, we received lots of supportive emails and messages from you. It really helps us to stay focused and overcome this crisis.

Some users have taken an extra step to try to help us and other users. I’d like to share a blog post by George Donnelly: Turn the MailerLite Data Loss Into a Marketing Win

Another user Alexis Anne created a video on how she solved the issue.

Again, I sincerely apologize. This won’t happen again.