The main reason for the disruption was an erroneous deployment update that caused a hanging connection to the main DB replica and hence impacted the performance of the application. We're rewriting our deployment procedures and infrastructure to a more robust provisioning system (Terraform) and hence expect to completely eliminate these types of issues.
Posted Jun 24, 2020 - 15:28 CDT
The issue has been fixed and we're now monitoring the situation to make sure that the whole system is stable.
Posted Jun 24, 2020 - 15:19 CDT
We've had a partial outage on our US1 instances that we've identified and are already recovering from. As soon as it's completely resolved we will update our status page and keep close monitor.
Posted Jun 24, 2020 - 14:17 CDT
This incident affected: US1 (Zendesk App, Time & Attendance, Schedule, Forecasting, Admin function, Reporting).