Incidents and maintenance reported on status page for Earthmover https://status.earthmover.io/ https://d1lppblt9t2x15.cloudfront.net/logos/b85a7fd51dc4565256c60263dbca1740.png https://status.earthmover.io/ en https://status.earthmover.io/incident/1055374 Mon, 07 Sep 2026 14:27:00 -0000 https://status.earthmover.io/incident/1055374#00fcc65278953d44dbe7ccfb869dda72158073ca6c110ad988b6fbcd9e3cbcf1 Incident ## Incident Post Mortem Between 09:37 and 12:33 UTC on 7 September 2026, requests to the Arraylake API (most visibly, opening a repository) intermittently returned HTTP 500 errors. The trigger was an automated cross-AZ failover of our managed PostgreSQL cluster, a routine high-availability event that completed normally in under a minute. However, our API's database connection pools reconnected during the brief window in which cluster roles were still settling, and ended up holding connections to an instance that was subsequently demoted to a read replica. Because those pooled connections remained open and healthy at the network level, they were never recycled, so any operation requiring a database write continued to be rejected as read-only long after the database itself had fully recovered. This affected opening repositories in particular, since issuing the short-lived, scoped object-storage credentials returned by that call records state in the database. No data was lost or corrupted, no incorrect data was served, and durability and access controls were unaffected at all times; the impact was limited to request availability. We restored service by cycling the API instances to establish fresh database connections. We are adding connection-level health validation and recycling so that pools re-establish themselves automatically after a failover, making the credential-issuing path degrade gracefully rather than fail when its bookkeeping store is briefly unavailable, and adding specific alerting for this failure mode, which would have detected the incident in seconds. https://status.earthmover.io/incident/1055374 Mon, 07 Sep 2026 12:39:00 -0000 https://status.earthmover.io/incident/1055374#24e062b1e290e62deabe21126d9e30716c36d7af03b2d91628c150d58e32ed09 Incident The incident has been resolved. https://status.earthmover.io/incident/1055374 Mon, 07 Sep 2026 09:37:00 -0000 https://status.earthmover.io/incident/1055374#f5ee4bf0f08c26e559062af744a10dbf4349dc931cb49265ce66e3f977580358 Incident Arraylake is currently experiencing degraded access. The following API routes are returning 500 errors. GET /repos/{org}/{repo}/open GET /repos/{org}/{repo}/bucket-credentials GET /repos/icechunk/{org}/{repo}/branches GET /repos/icechunk/{org}/{repo}/tags POST /orgs/{org}/repos GET /repos/icechunk/{org}/{repo}/commits GET /repos/icechunk/{org}/{repo}/tree GET /repos/icechunk/{org}/{repo}/virtual-chunk-status GET /repos/icechunk/{org}/{repo}/virtual-chunk-containers POST /orgs/{org}/buckets POST /orgs/{org}/members POST /user/orgs GET /repos/icechunk/{org}/{repo}/commits/{commit_id} POST /orgs/{org}/repos/import GET /repos/icechunk/{org}/{repo}/dataset-node https://status.earthmover.io/incident/1049202 Thu, 03 Sep 2026 16:22:00 -0000 https://status.earthmover.io/incident/1049202#b706db0d9d72064cc23fdc5aa6d156f39084db9b590e258f0f6611bf9071696a Incident Resolved https://status.earthmover.io/incident/1049202 Thu, 03 Sep 2026 16:11:00 -0000 https://status.earthmover.io/incident/1049202#0bee3216105dc4a57d65a3cab66239cf33f3a3cb0f88a6110b8208b9324b9568 Incident Between 16:11 UTC and and 16:22 UTC, we saw a large number of 500 responses. This was due to a data migration on our back-end that took much longer than expected. The issue has been resolved. https://status.earthmover.io/incident/814578 Thu, 29 Jan 2026 17:17:00 -0000 https://status.earthmover.io/incident/814578#436eccebdbc33fe6dacfbe6376b600ff4298606c285a8fe52b397fdf1ffdf004 Incident We identified the root cause as a routing issue that took about 15 minutes to diagnose and fix. This was the result of rolling out a new feature. All Flux users should now have access to the new parallel deployments feature. https://status.earthmover.io/incident/814578 Thu, 29 Jan 2026 16:51:00 -0000 https://status.earthmover.io/incident/814578#f8828397d94b020a2a1067bcbed1a6e77f0c23156d37910a2c0784a024c255b5 Incident Flux has suffered an outage while we rolled out a new feature. This was an unanticipated outage and affects all Flux services. To deploy the fix, we will need to bring down all Flux services and redeploy them. https://status.earthmover.io/incident/769318 Tue, 18 Nov 2025 16:34:00 -0000 https://status.earthmover.io/incident/769318#c58dfaf9e0b2e3358e65fc8136371d500fe7f602db8231d6d45e6c22dca63dd1 Incident Issue has been resolved https://status.earthmover.io/incident/769318 Tue, 18 Nov 2025 14:57:00 -0000 https://status.earthmover.io/incident/769318#5cfa11964c744ed6646f266dc19b2205b7e9968a56c2438e47df3512a7088d63 Incident Users logging in or refreshing their credentials in the web app or the Arraylake library/CLI will not be able to complete the authentication process. Users and processes that are using an Arraylake API client will continue to be able to use our service without disruption. https://status.earthmover.io/incident/747471 Mon, 20 Oct 2025 22:02:00 -0000 https://status.earthmover.io/incident/747471#1b88d932365a1b5bf9f3c2afa65097a66f0ad5599b01d097c5e6d0529968cd60 Incident Latency and request performance have returned to normal levels, and the Earthmover platform is operating as expected again. That said, given the earlier instability in AWS us-east-1, we’re continuing to keep a close watch on system health and AWS service status. https://status.earthmover.io/incident/747471 Mon, 20 Oct 2025 21:28:00 -0000 https://status.earthmover.io/incident/747471#35d938438f3d8b30319ddc0c5b0edd23cbc1d4e945e180ca5dc513e674c4b526 Incident The Earthmover platform is reachable again, but performance is highly degraded. Latency is significantly elevated and requests are still intermittently failing due to ongoing instability in AWS us-east-1. We’re continuing to monitor and will provide updates as things stabilize. https://status.earthmover.io/incident/747471 Mon, 20 Oct 2025 19:18:00 -0000 https://status.earthmover.io/incident/747471#227611846c868961f573359300e8d25647674f1870e1b51e6b949abcb457905e Incident The AWS us-east-1 outage continues to impact the Earthmover platform with no meaningful recovery observed so far. Our team is actively monitoring system health, evaluating possible mitigation options, and tracking AWS recovery efforts. We’ll share updates as soon as there are material changes. https://status.earthmover.io/incident/747471 Mon, 20 Oct 2025 17:00:00 -0000 https://status.earthmover.io/incident/747471#4b8214fe31903d770c62fdb359a10570470d680c55adc1d6fb7c46a3a82b7d63 Incident AWS is reporting signs of recovery in us-east-1, but we have not yet observed any improvement in Earthmover systems. Service disruption remains ongoing. We’ll continue monitoring and share updates as conditions change. https://status.earthmover.io/incident/747471 Mon, 20 Oct 2025 16:35:00 -0000 https://status.earthmover.io/incident/747471#0a61c7af25ac247029361570463b8d494111454d2624953ce3397c9f7d551b39 Incident The AWS us-east-1 outage has worsened and is now causing a full disruption of the Earthmover platform. Most requests are failing or timing out. We’re monitoring closely and will update as soon as services begin to recover. https://status.earthmover.io/incident/747471 Mon, 20 Oct 2025 16:04:00 -0000 https://status.earthmover.io/incident/747471#76e6560e94834cba61a13ad9e6048542d7446f358443295d4f69c28d771029d3 Incident We are tracking a significant outage in AWS us-east-1. This is currently impacting network traffic to the Earthmover API. More details will be posted as we investigate the issue. https://status.earthmover.io/incident/736626 Thu, 02 Oct 2025 21:30:00 -0000 https://status.earthmover.io/incident/736626#d8085b3644e0cc1d596df6253843f9cd81c9c1fff92dcbe9843c20a26ac42ea0 Incident We have rolled out a fix to the issue. All systems have now returned to regular service. https://status.earthmover.io/incident/736626 Thu, 02 Oct 2025 20:45:00 -0000 https://status.earthmover.io/incident/736626#1026ef67899e753b8dffffc8111f52590acc5272ec089b98e158e99dd8ab9f91 Incident On Oct. 2 we experienced an issue which prevented customers from accessing repos between 4:45pm and 5:30pm EST. The issue originated with a service release which was not backwards compatible with existing client versions. The error you may have seen would have been `ValueError: V1 repositories can no longer be opened in the Arraylake client.` No data were lost, and the issue has been resolved. Further, this downtime was not reflected in https://status.earthmover.io/ which monitors the availability of our API but not this particular functionality. In response to this incident, we are reviewing our integration testing to understand why this issue wasn’t detected before the release. We are also augmenting our status monitoring to include deeper verification of functionality. Next week we will share a technical post mortem with deeper analysis of the root cause of the incident. We apologize sincerely for this incident, and we thank you for your continued feedback and support. https://status.earthmover.io/maintenance/619537 Mon, 14 Jul 2025 20:58:31 -0000 https://status.earthmover.io/incident/619537#42cc837034b5ad69ad4dcd8682c2c58ac5f2f6279058dd7bdad5c74dc8f0a862 Maintenance We’ll be performing scheduled maintenance on the Earthmover platform on Thursday, July 17, 2025, from 6:00 PM to 7:00 PM PDT. This maintenance includes a backend database upgrade in Arraylake, which will improve system performance and reliability. During this time, the following services will be temporarily unavailable: - Earthmover API - Web application - Flux endpoints We’ll share real-time updates throughout the maintenance window at status.earthmover.io. If you have any questions or concerns, feel free to reach out to us via Slack or email at support@earthmover.io. Thanks for your understanding! —The Earthmover Team