Last Monday, Google had a large scale outage which seemed to have affected nearly all of its services. Multiple issues started at 3:47 am PT / 11:47am BST and affected Workplace apps, Gmail and YouTube as well as other services. However the issue was to do with how Google authenticates its users.

Google was in the middle of migrating some of the User ID service to a new storage solution which ran out of disk quota.

Google said:

We would like to apologize for the scope of impact that this incident had on our customers and their businesses. We take any incident that affects the availability and reliability of our customers extremely seriously, particularly incidents which span multiple regions.

Google also said:

On Monday 14 December, 2020 from 03:46 to 04:33 US/Pacific, credential issuance and account metadata lookups for all Google user accounts failed. As a result, we could not verify that user requests were authenticated and served 5xx errors on virtually all authenticated traffic. The majority of authenticated services experienced similar control plane impact: elevated error rates across all Google Cloud Platform and Google Workspace APIs and Consoles.

Google will be working on improving their own internal monitoring and alerts to avoid this from happening again.

Tags: