Andy Garcia · freelance dev
FRENES
Back to the blog
Cloud & Reliability

Google Cloud Storage Had 18 Hours of Write Errors in Iowa. The First Public Update Came 7 Hours In.

October 19, 2026 7 min read

On Thursday, October 8, 2026, at 04:00 Pacific time, Google Cloud Storage started returning errors on writes in us-central1, its Iowa region. For 12 hours and 30 minutes, customers saw HTTP 503 UNAVAILABLE errors on WriteObject calls, client-side timeouts, and much higher latency on the writes that did go through. On Friday, October 9, it came back for 5 hours and 41 minutes in one zone, us-central1-b. Google's final incident report, posted on October 14, puts the total at 18 hours and 11 minutes. According to the report, other regions and other Google Cloud services were not affected.

The cause is the part people will quote: a change meant to relieve a capacity problem created the outage, and an incomplete rollback created the rerun. I read the report, the incident history behind it, and the pages it does not link: the SLA, the retry documentation, the location guide.

My reason is simple. BidForge, the tool I built to read French public tender files for small businesses, starts from a file. So I read this for the write path.

What broke, in Google's words

Some storage backend nodes in us-central1 were short on capacity because of increased traffic. To deal with it, Google pushed "a traffic sorting optimization configuration update" to the Cloud Storage load-balancing layer. The update "inadvertently" bypassed the capacity-aware routing logic. A small fraction of nodes, those with little spare capacity, received "an overwhelming proportion of incoming write traffic." They ran out of resources and the ingestion services behind them timed out. Separately, a problem in the underlying database service added metadata errors for part of the window.

Two lines in the remediation section stand out. Engineers first tried to add emergency capacity to the region, and that "was blocked by systemic regional capacity limits." Even Google could not add capacity in Iowa on demand. And the October 9 recurrence "was caused by an incomplete rollback of an upload planner feature, which broke the load balancing mechanism" again. What finally held was a full revert of the load-balancing change.

Google's prevention list is the lesson in short: validate load-balancing changes in smaller regions before larger ones, and make it impossible for routing to bypass capacity safeguards.

The status page trailed by hours

The final report says Google engineers detected the issue at 04:50. The first public update in the incident history is timestamped 11:24 PDT, 7 hours and 24 minutes after the start. It said the problem began at 05:19, a time Google's preliminary report of October 9 moved to 04:00. Every live update gave the workaround as "None at this time."

Some customers' status pages were faster. At 09:32 PDT, Mixpanel reported a data ingestion delay for US projects, and two hours later named Google Cloud as the upstream cause. At 09:42 PDT, Elastic warned that some Elastic Cloud Serverless projects in GCP us-central1 could see delayed or unavailable operations, and at 10:21 it named a cloud storage issue, an hour before Google's first post. At 14:53 PDT, Temporal failed its multi-region namespaces in us-central1 over to their replica regions as a precaution and reported no impact on workflow execution. IncidentHub, which tracks status pages, counted four downstream services that named the outage as a cause: Elastic, Temporal, Mixpanel and Torq.

The second day is thinner still. The public history shows no update between 13:48 PDT on October 8 and the preliminary report at 09:44 PDT on October 9, which covered October 8 only. IncidentHub's write-up of October 12 still ended the incident on October 8. The 5 hours and 41 minutes in us-central1-b first appear in the final report.

I do not read this as Google hiding anything. A provider's status page reports what the provider has confirmed, at the speed of its process. The alert on your own error rate is the only status page that speaks for your users.

Default retries are short, when they happen at all

Google's retry documentation lists 503 as retryable. Whether your client actually retries an upload depends on the language.

Object inserts are "conditionally idempotent": safe to retry only with an ifGenerationMatch precondition. Per that page, the Node.js, Java and Go libraries retry such operations only when the precondition is set. The Python library retries object inserts by default. And when they retry, it is mostly brief. Node.js defaults to 3 retries, starting at 1 second and doubling: about 7 seconds of waiting. Java stops at a 50-second total timeout, Python at a 120-second deadline. Go is the exception: no total limit by default, except 32 seconds per chunk of a multi-request resumable upload. The first occurrence lasted 12 and a half hours.

The cheap fix is one parameter. A generation-match precondition of 0 means the request proceeds only if no live object has that name. If a retried upload already succeeded, the retry gets 412 Precondition Failed instead of writing twice. With unique object names per upload, you can retry freely and treat that 412 as done.

The real fix is to stop making the user wait on the bucket. Accept the file, keep it somewhere you control, answer "received", and let a queue push it to storage with backoff for as long as it takes. A 503 then becomes a delay in your pipeline instead of an error on someone's screen.

The SLA is a credit, measured on all your traffic

The Cloud Storage SLA promises 99.9 percent monthly uptime for Standard storage in a region, and 99.95 percent in a dual-region or multi-region. Uptime is 100 percent minus the average error rate over every five-minute period of the month, per project and per region. The error rate is 5xx responses divided by all valid requests, reads included. A client-side timeout, one of the symptoms in the report, returns no 5xx response at all, so as I read the definition, it does not count.

October has 8,928 five-minute periods. The two windows cover about 218 of them. To fall below 99.9 percent, assuming a clean rest of the month, your error rate during those periods had to average above 4.1 percent. To fall below 99 percent, about 41 percent. If writes are 10 percent of your requests and a third of them failed during the windows, your rate was about 3.3 percent: no credit, because the reads that worked dilute it.

If you do qualify, the credit is 10 percent of the month's bill for that storage class, 25 percent from 95 to 99 percent, applied to future bills. Retries only count if they follow the SLA's back-off rule: at least 1 second, growing exponentially up to 32 seconds. And you must notify Google support within 30 days of becoming eligible, or lose the credit.

A second region is insurance, with a gap in the fine print

Google's location guide says that for a dual-region bucket such as NAM4, which pairs us-central1 with us-east1, "in the event of a regional outage, requests are automatically failed over to the other region."

It costs more. The pricing page lists regional Standard storage in Iowa at $0.000027397 per GiB per hour, about $0.020 per GiB per month. A dual-region is billed in both regions, $0.022 each in the page's own example (NAM4 itself is listed at $0.044), plus $0.02 per GiB replicated on write in North America. For 500 GiB stored and 100 GiB of new data a month, that is about $10 against about $24.

What the report does not say is whether dual-region buckets that include us-central1 saw errors. This was a degradation, not a region going dark, and the guide does not say when failover triggers in that case. Before paying for the second region, I would ask support that question and want the answer in writing.

What I would check before the next one

  • Every write path and the location of its bucket: uploads, exports, logs, backups.
  • For each, the client library and its default retry policy. ifGenerationMatch=0 on create-only uploads.
  • A queue between the user and the bucket for anything the user cannot easily resend.
  • An alert on your own 5xx rate, with writes counted apart from reads.
  • If you were in us-central1 on October 8 and 9: your error rate for the month, and a decision on the SLA claim before the 30 days run out.
  • Your own rollbacks. A rollback is a deploy: write it, run it somewhere small first, and check that it is complete. I orchestrate most of my work with Claude Code, and this is the step I would never let an agent call done on its own. Google's second day came from a rollback that was not complete.

The cheapest line in the report is Google's first preventive action: try the change somewhere small before the place that matters. It applies to every team, on any cloud, starting with the next config change.

Sources

A project like this one?

I design and deploy products like this. Let's talk.

Let's talk