US1 V2 partial outage
Incident Report for Tymeshift
Postmortem

At 2pm UTC time, we've deployed a new version of our V2 product and executed changes to the database in order to optimize the stored data. However, while doing so, we've triggered a a heavy process that caused tables related to our schedule feature to not respond in a timely manner and thus impacting the platform. We've immediately proceeded to stop that process and recover functionality in a matter of minutes.

Moving forward, we will put in place checks to prevent these processes from being executed in bulk and test the impact during the pre-deployment phase.

Since data processing services are in isolation, all agent and ticket activity was collected and processed normally.

Posted Feb 09, 2023 - 09:35 CST

Resolved
This issue has been resolved and we will be following up with a post mortem.
Posted Feb 09, 2023 - 08:47 CST
Monitoring
The issue has been resolved and we're monitoring the system.
Posted Feb 09, 2023 - 08:36 CST
Investigating
We're currently investigating an incident on our V2 platform after a recent deployment.
Posted Feb 09, 2023 - 08:15 CST
This incident affected: US1 (Time & Attendance, Schedule, Forecast, Admin function, Reporting).