Reliability

On Thursday 12 June 2025, shortly before 11:00 Pacific time, requests to Google Cloud's APIs began failing with 503 errors around the world. The incident's list of affected products ran from Identity and Access Management and Cloud Storage to Vertex AI Online Prediction. The failing component was Service Control, the system that checks every API request against its authorization and quota policies before a product sees it. On 29 May a new quota-policy check had been added to it without appropriate error handling and without a feature flag. On 12 June a policy change with unintended blank fields was written to the regional Spanner tables that Service Control reads. Quota metadata is replicated globally, so within seconds Service Control everywhere reached the new code path, hit a null pointer and crashed in a loop.