Resolved -
After monitoring the situation overnight, we are confident that the fixes we put into place late yesterday afternoon have addressed the significant delay issues with the Google Sheets integration. The queue has returned to a normal size, and all jobs which were scheduled for processing during the backlog are now complete. Thank you for your patience and understanding.
Feb 20, 08:32 EST
Monitoring -
We have implemented a couple of fixes on our side which we are hopeful will address the queue backlog. The issue causing the extreme delay in Google Sheets integration deliveries was twofold:
1. Certain Google accounts were being throttled by Google, which caused the request time to increase significantly
2. A configuration setting in our queuing service resulted in those requests effectively "timing out" listening for the request, which then caused the service to retry those requests.
The result was that requests were scaling exponentially because the queuing service was expecting a response, and would essentially push those delayed responses to the back of the line, even if those delayed jobs were successful.
We are currently monitoring the impact of the fixes and are encouraged that the request timeouts have stopped and the queue is decreasing. We expect that the deliveries will continue to be delayed as the workers process the queued requests, however. We will update you once the queue has reached a comparatively normal size.
Feb 19, 17:03 EST
Identified -
We have identified the root cause of the severe delays for the Google Sheets integration. We process SubmitAction activities (the thing that happens when you press the submit button) in a FIFO manner; First In, First Out. Normally, as these submission requests increase, we can scale the number of workers (the things that actually process submission requests) accordingly to keep the processing time to a minimum.
In this case, what we are seeing is that Google is throttling a few customer accounts, limiting the number of submissions those accounts can use over a given time period. These accounts have queued up a sizeable number of submissions for processing. This in turn, has caused the backlog of queued jobs to increase at a faster rate than Google's throttling will allow our workers to process those jobs.
At this time, we don't have a way to change the processing order of requests. Unfortunately, that means that everyone's queued Google Sheets submissions are significantly slowed, with the delay appearing to be roughly 5.5 hours. To use a highway analogy, all of the Sheets requests are stuck behind a slow moving semi on a one-lane road, and there are no off ramps for the next 5.5 hours.
We appreciate your patience and understanding during this time. Please note that none of the submissions are being dropped and they are all being put in the queue as normal, so no work is being lost.
Feb 19, 09:33 EST
Investigating -
We are aware that the Google Sheets integration for forms is currently experiencing a significant delay, and are investigating the issue. We will update you once we have identified the cause.
Feb 19, 08:15 EST