Backend service interruption
Incident Report for Lilt
Postmortem

A bug in code deployed to a backend service prevented document import from succeeding for several hours.  The converter was then rolled back to a previous version, restoring service. The bug has been found and corrected.

Additionally, we are making several improvements to the backend release process both improve release quality and detect this type of failure more rapidly. These steps include:

  • Improvements to automated test coverage in the build and deployment process; a test that would have caught this bug was not run prior to the deployment.
  • Investigating ways to improve our abilities to assess the liveness/health of backend services to detect failures more quickly.
  • Improving internal build notifications to help focus investigations on services that have been redeployed recently.
Posted Nov 26, 2018 - 20:43 UTC

Resolved
The issue has been resolved.
Posted Nov 22, 2018 - 05:19 UTC
Identified
File import / export is broken due to a software release this afternoon. We are reverting the release now.
Posted Nov 22, 2018 - 05:14 UTC
This incident affected: Backend.