Service Disruption
Incident Report for Tymeshift
Postmortem

Summary

On the 24th October 2018 between 10:12 CST and 10:24 CST an incident occurred which caused our application to run in a degraded state. Functionality was fully restored at 10:25CST. Inits degraded state, the application was serving customers with 404 pages preventing users from accessing their workspace in our US2 region.

Cause

The underlying cause for this outage was a configuration change to our deployment service resulting in core application files being removed from our public web folder.

In this case we were able to rollback the configuration changes and restore the deleted files. No customer data was impacted as a result of this outage.

What we are changing going forward

This incident highlighted several gaps in our process, and we have planned the following improvements to ensure that this class of failure does not reoccur:

  • We are adjusting our change process to improve our testing of code within our staging environment
  • Introducing maintenance windows outside of peak times

We apologize for the disruption in service as a result of this incident. If you have any questions relating to this incident, please do not hesitate to contact us at support@tymeshift.com.

Thank you for your continued trust.

  • The Tymeshift Team
Posted 8 months ago. Oct 30, 2018 - 10:58 CDT

Resolved
This incident has been resolved. We apologize for the inconvenience caused. Our team is monitoring this closely.

Thank you.
Posted 8 months ago. Oct 24, 2018 - 10:32 CDT
Investigating
We are currently investigating this issue and will update shortly.
Posted 8 months ago. Oct 24, 2018 - 10:24 CDT