Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Oct 13, 2019 - 16:00 CDT
This update is in regard to the issues we experienced with the release of our new app which runs on the v2 version of the Zendesk Apps Framework.
On Thursday 10/10/2019 at 4 PM Pacific Time, we deployed our new v2 app. At approximately 9 PM Pacific Time, our partners at Zendesk rolled the app back to the v1 version due to a spike in traffic coming through the Zendesk Proxy Service responsible for proxying secure requests and performance issues being reported by our mutual customers. If you or any of your agents were having trouble loading Tymeshift during this time it was likely due to this incident.
This issue was due to a security change in the Zendesk Apps Framework that required all secure requests to be passed through their proxy server. The proxy server was unable to keep up with the load our app was creating.
We believe that it’s important to share the reason that we did not catch this issue during our beta testing with customers is that the beta program did not require the use of the proxy service, and thus, everything worked smoothly. During the rollout, we identified that this was a new requirement and only identified the performance limitations of the proxy service upon release. Our team has taken note of this incident and will be implementing additional precautions in the future should a similar situation arise.
After becoming aware of the problem on Thursday evening, our engineering team has been working to architect a new solution to minimize the number of requests that need to be passed through Zendesk’s proxy service.
Additionally, our partners at Zendesk are implementing additional caching on their proxy service to further minimize the chance of potential performance impacts in the future.
We’re happy to report that we were able to resolve all issues and are planning a release do-over this Sunday (10/13/2019) at approximately 2 PM Pacific. Both Tymeshift & Zendesk’s Engineering team will be monitoring the release closely to ensure a smooth rollout.
We appreciate your patience and understanding as well as our awesome partners at Zendesk working tirelessly with us to resolve this issue over the weekend. We will continue to update this incident as new information becomes available.
UPDATE: 10/13/2019 5 PM Pacific: We have successfully redeployed our v2 application. Our team is currently monitoring the release and we will provide further updates shortly.
UPDATE: 10/13/2019: 10:30 PM Pacific: We've identified a issues with the new v2 release that we are currently working to resolve. These issues are: #1 Unable to select a General Task in the Agent Dashboard Page and Agent Dashboard does not update status properly when switching activities (however time is still tracked correctly) #2 App reloads if you haven't started your day yet and attempt to select another navigation item #3 App does not switch to Untracked time properly if you close your last remaining ticket tab
Our engineers are currently working to resolve these issues and we'll update this page again once they are corrected.
UPDATE: 10/14/2019: 10 AM Pacific: All of the above issues have been resolved.
UPDATE: 10/14/2019: 12 PM Pacific: We are receiving some reports of app slowness/the clock not tracking properly when switching activities.
UPDATE: 10/15/2019: 8 PM Pacific: We have released an update which should improve the loading speed of the app. We are continuing to investigate the issues related to switching activities although we believe this intermittent problem may have been tied to the performance issues.
Posted Oct 12, 2019 - 19:20 CDT
This scheduled maintenance affected: US2 (Zendesk App), US1 (Zendesk App), and BDC (Zendesk App).