Incident Post Mortem

Between 09:37 and 12:33 UTC on 7 September 2026, requests to the Arraylake API (most visibly, opening a repository) intermittently returned HTTP 500 errors. The trigger was an automated cross-AZ failover of our managed PostgreSQL cluster, a routine high-availability event that completed normally in under a minute. However, our API's database connection pools reconnected during the brief window in which cluster roles were still settling, and ended up holding connections to an instance that was subsequently demoted to a read replica. Because those pooled connections remained open and healthy at the network level, they were never recycled, so any operation requiring a database write continued to be rejected as read-only long after the database itself had fully recovered. This affected opening repositories in particular, since issuing the short-lived, scoped object-storage credentials returned by that call records state in the database.

No data was lost or corrupted, no incorrect data was served, and durability and access controls were unaffected at all times; the impact was limited to request availability. We restored service by cycling the API instances to establish fresh database connections.

We are adding connection-level health validation and recycling so that pools re-establish themselves automatically after a failover, making the credential-issuing path degrade gracefully rather than fail when its bookkeeping store is briefly unavailable, and adding specific alerting for this failure mode, which would have detected the incident in seconds.

The incident has been resolved.

Arraylake is currently experiencing degraded access. The following API routes are returning 500 errors.

GET /repos/{org}/{repo}/open
GET /repos/{org}/{repo}/bucket-credentials
GET /repos/icechunk/{org}/{repo}/branches
GET /repos/icechunk/{org}/{repo}/tags
POST /orgs/{org}/repos
GET /repos/icechunk/{org}/{repo}/commits
GET /repos/icechunk/{org}/{repo}/tree
GET /repos/icechunk/{org}/{repo}/virtual-chunk-status
GET /repos/icechunk/{org}/{repo}/virtual-chunk-containers
POST /orgs/{org}/buckets
POST /orgs/{org}/members
POST /user/orgs
GET /repos/icechunk/{org}/{repo}/commits/{commit_id}
POST /orgs/{org}/repos/import
GET /repos/icechunk/{org}/{repo}/dataset-node