At 7:36am PST our engineers identified an issue with one of our regions that impacted the reporting and administrative functionality within Tymeshift. Upon investigating the incident, we discovered that approximately 1 in every 5 requests to our database instances were being refused. Diving deeper, our engineers determined that the problem was likely isolated to an internal networking issue within our infrastructure provider, Google Cloud Platform. We immediately began working with our partners at Google to make them aware of the impact and work to resolve the issue. At approximately 12:30 PM PST the refused connections dropped from 1/5 to ~1/200. And, at ~5PM PST the connections refused returned to zero.
We have been working diligently with our partners at Google to post mortem the incident, and will include Google’s post mortem below for your reference.
As per Google’s post mortem, they do not expect the incident to recur, however, we are still investigating additional precaution we can take on our side to mitigate future risk.
Please note: If your account was impacted by this incident, you would have observed issues logging into Tymeshift, running reports, and accessing scheduling. However, Time & attendance tracking and agent facing functionality was not impacted.