Intermittent connectivity issues impacting reporting & login
Incident Report for Tymeshift
Postmortem

At 7:36am PST our engineers identified an issue with one of our regions that impacted the reporting and administrative functionality within Tymeshift. Upon investigating the incident, we discovered that approximately 1 in every 5 requests to our database instances were being refused. Diving deeper, our engineers determined that the problem was likely isolated to an internal networking issue within our infrastructure provider, Google Cloud Platform. We immediately began working with our partners at Google to make them aware of the impact and work to resolve the issue. At approximately 12:30 PM PST the refused connections dropped from 1/5 to ~1/200. And, at ~5PM PST the connections refused returned to zero.

We have been working diligently with our partners at Google to post mortem the incident, and will include Google’s post mortem below for your reference.

As per Google’s post mortem, they do not expect the incident to recur, however, we are still investigating additional precaution we can take on our side to mitigate future risk.

Please note: If your account was impacted by this incident, you would have observed issues logging into Tymeshift, running reports, and accessing scheduling. However, Time & attendance tracking and agent facing functionality was not impacted.

Posted 6 days ago. Dec 05, 2018 - 23:20 CST

Resolved
This incident has been resolved.
Posted 7 days ago. Dec 05, 2018 - 19:18 CST
Monitoring
Our partners at Google have resolved the networking issues impacting our users. At this time, the incident is resolved, but we are continuing to monitor the situation and continue to work with our partners at Google to investigate the root cause and will provide a post mortem as soon as this is complete.
Posted 7 days ago. Dec 05, 2018 - 15:02 CST
Update
Our engineers are working with our cloud platform partners at Google to supply the necessary diagnostic information and Google is investigating the issue to determine the cause of the intermittent network connectivity. We'll post another update here as soon as we know more.

In the meantime -
Tracking as well as most other areas of Tymeshift are working normally, however you may be experiencing some slowness or intermittent loading issues.

Thank you for your patience and we apologize for the inconvenience.
Posted 7 days ago. Dec 05, 2018 - 14:16 CST
Update
We are continue to work with our partners at Google to resolve the intermittent network connectivity issue which is causing some parts of our application to be slow/load intermittently for users within the US2 region.
Posted 7 days ago. Dec 05, 2018 - 13:18 CST
Identified
Upon investigating this issue we believe it is due to a network services issue with our cloud provider, Google. The majority of functionality has been restored, but we are still observing some intermittent slowness and failure of certain pages to load.

We are currently working with Google on these final issues and will update you again as soon as we know more.

We apologize for the inconvenience and appreciate your patience.
Posted 7 days ago. Dec 05, 2018 - 12:12 CST
Update
We are receiving alerts about connection issues to our databases causing intermittent service disruption
We are still investigating this issue.

Thank you for your patience.
Posted 7 days ago. Dec 05, 2018 - 11:34 CST
Investigating
Users on US2 are reporting they can not login into the application
We are currently investigating this issue.

We will update you as soon as we have an update.
Posted 7 days ago. Dec 05, 2018 - 09:30 CST
This incident affected: US2 (Application).