Cloudfleet status page

2026-07-01 17:00 UTC: We are aware of an issue in our billing system that is causing some pushes to the Cloudfleet Container Registry (CFCR) to fail. Reads (image pulls) are not impacted and all stored data is safe. Our team is actively working on a solution and we will provide updates as they become available.

UPDATE (2026-07-01 17:12 UTC): Push functionality has recovered and CFCR is operating normally again. Reads (image pulls) were never impacted and all stored data remains safe. We are keeping this incident open while we continue to monitor the system to ensure it remains stable. Thank you for your patience.

RESOLVED (2026-07-01 17:46 UTC): After a period of monitoring, CFCR push functionality has remained fully stable and the underlying billing system issue has been fixed. This incident is now resolved. Thank you for your patience.

We will be performing scheduled maintenance on the CFKE data store in the europe-central-1a region on 2026-06-08 from 11:00 to 16:00 UTC. We do not expect any downtime; however, some intermittent network bleeps might occur on the CFKE apiserver during this window. Thank you for your patience.

We will be performing scheduled maintenance on the CFKE data store in the northamerica-central-1a region on 2026-06-02 from 16:00 to 18:00 UTC. We do not expect any downtime; however, some intermittent network bleeps might occur on the CFKE apiserver during this window. Thank you for your patience.

We will be performing scheduled maintenance on our IAM system starting 2026-05-19 07:30 UTC. The work is expected to take no more than 15 minutes, and we anticipate no user-facing impact during this window. Thank you for your patience. UPDATE (07:55 UTC): Maintenance complete. No impact or downtime. We will continue monitoring the system to ensure everything remains stable.

According to Hetzner, the incident lasted from 2026-02-03 12:31 UTC to 2026-02-03 14:19 UTC and is now resolved. Cloudfleet services were not impacted. A subset of CFKE customers running Hetzner-exclusive environments experienced issues related to Hetzner load balancers, while CFKE customers operating in multi-cloud setups reported successful failover to healthy providers. For updates directly from Hetzner, follow their status page at https://status.hetzner.com/.

We identified and resolved an issue that affected a small subset of customer clusters. The issue was caused by Hetzner deprecating the cpx11, cpx21, cpx31, cpx41, and cpx51 instance families in EU and Singapore locations, while their API continued to incorrectly report these instances as available and returned valid pricing information for all locations. As a result, our node auto-provisioner could select these instance families as provisioning candidates. In this scenario, the Hetzner API returned an unexpected error type, causing the auto-provisioner to enter a retry loop. Over time, this led to rate limiting of the auto-provisioner and degraded related cluster operations, including node initialization, garbage collection, and load balancer management. We have manually disabled the affected instance families in our catalog and deployed a fix to correctly recognize this error condition and stop retrying. All systems are operating normally. We are also collaborating with the Hetzner team to address and correct this behavior in their API.

Between 15:58 and 17:45 (CET), we observed network instability affecting some clusters running on Hetzner infrastructure. During this period, an increased rate of packet drops caused intermittent connectivity issues. As a result, some workloads experienced restarts, and in a limited number of cases, systems showed elevated CPU usage due to reconnection and recovery activity. The network has since stabilized, affected systems are recovering, and we continue to closely monitor all clusters to ensure full stability. We will share a follow-up update if additional information becomes available.

We observed increased error reports from 3rd party infrastructure provider Hetzner due to no available capacity in the FSN1 region. Node provisioning failures affected customers with fleet configurations limited to this region. Cloudfleet services are not impacted. The incident started at 4:00 PM UTC and error rates returned to normal levels around 4:57 PM UTC. We recommend configuring additional regions in your fleet configurations to allow temporary node scheduling in alternative regions during capacity constraints.

On November 18th, the Cloudfleet API and Console experienced an outage between 11:20 AM UTC and 2:30 PM UTC due to a Cloudflare outage. IAM, CFKE, CFCR, and customer workloads were not impacted during this time. As a follow-up, the teams at Cloudfleet are evaluating potential future architecture to remove the reliance on 3rd party services.

We observed an increased rate of container image pull failures from registry.k8s.io affecting customers using 3rd party infrastructure provider Hetzner. Hetzner IP addresses were being rate-limited or blocked by the upstream registry. Cloudfleet services are not impacted. Error rates came back to normal at 04:30 UTC. Customers were directed to use Cloudfleet's Quay mirror for improved reliability. We recommend all production workloads use Cloudfleet's registry mirrors instead of registry.k8s.io to avoid dependency on 3rd party infrastructure providers.