On the 24th October 2018 between 10:12 CST and 10:24 CST an incident occurred which caused our application to run in a degraded state. Functionality was fully restored at 10:25CST. Inits degraded state, the application was serving customers with 404 pages preventing users from accessing their workspace in our US2 region.
The underlying cause for this outage was a configuration change to our deployment service resulting in core application files being removed from our public web folder.
In this case we were able to rollback the configuration changes and restore the deleted files. No customer data was impacted as a result of this outage.
What we are changing going forward
This incident highlighted several gaps in our process, and we have planned the following improvements to ensure that this class of failure does not reoccur:
We apologize for the disruption in service as a result of this incident. If you have any questions relating to this incident, please do not hesitate to contact us at firstname.lastname@example.org.
Thank you for your continued trust.