Northflank - Notice history

Experiencing partial outage

99.95% uptime

Networking - Major outage

Certificates - Major outage

Services - Major outage

Addons - Major outage

Jobs - Major outage

Builds - Major outage

Logs and Metrics - Major outage

Northflank App - Operational

100.0% uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2025
Jun 2025
Jul 2025

Northflank API - Operational

100.0% uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2025
Jun 2025
Jul 2025

Marketing Site - Operational

100.0% uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2025
Jun 2025
Jul 2025

Documentation - Operational

100.0% uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2025
Jun 2025
Jul 2025

Notice history

View current status

Jul 2025

Let's Encrypt Certificate Generation Issues
Resolved11 hours 30 minutes
  • ResolvedUTC

    Certificate generation is functioning without issue. We will continue to monitor the situation.

  • MonitoringUTC

    Let's Encrypt is currently experiencing an outage leading to certificate generation delays and failures on the Northflank platform. This will impact provisioning of new addons or domains.

Jun 2025

Issues with Google Cloud platform - Affecting multiple services
ResolvedPartial outage2 hours 44 minutes
  • ResolvedUTC

    GCP BYOC cluster processing is now operational.

  • UpdateUTC
    We are observing improved operations in the US - Central region. Addon creation and backup issues are now resolved. Ongoing Issues: * GCP BYOC cluster processing
  • MonitoringUTC

    Google Cloud has acknowledged the incident. For detailed information, please visit: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW

    Most services have been restored to normal operation. However, we are still experiencing the following issues:

    • Addon creation in the US - Central region

    • Addon backups in the US - Central region

    • GCP BYOC cluster processing

    We will continue monitoring the situation and provide updates as they become available.

  • IdentifiedUTC
    We have noticed issues with Google Cloud APIs. This seems to be widespread across the Google Cloud platform. We are currently determining the impact of the outage, and we will provide updates when we know more. Currently, logs, metrics, builds, BYOC and addon backups may be affected.
Metrics ingestion and query outage
ResolvedPartial outage2 hours 16 minutes
  • ResolvedUTC

    The ingestion rate has been stable for 30 minutes and historic writes have caught up.

    We sincerely apologise for the service disruption you may have experienced. We are investigating the root cause to prevent similar issues in the future.

  • MonitoringUTC

    The fix has been rolled out and we are currently monitoring the ingestion rate and catch-up.

  • IdentifiedUTC

    Metrics query and ingestion are currently affected. Logs are unaffected.

    We are currently applying a fix for the issue and monitoring the results.

May 2025

Infrastructure stability issues
ResolvedPartial outage1 hour 9 minutes
  • ResolvedUTC

    We sincerely apologise for the outage you experienced.

    What happened:
    Yesterday, we rolled out an update to support ARM architecture across our platform.
    As part of that, a migration was initiated today to extend support to x86 architecture within our database schemas.
    This triggered a higher-than-expected volume of database rescheduling events, which in turn put excessive pressure on the underlying GCP infrastructure. Some nodes failed under this load, compounding the issue and causing degraded service for several customers.

    Timeline of events (UTC):

    • 15:36 — Northflank engineers were alerted to failing health checks on customer endpoints.

    • 15:41 — Elevated 503 errors were detected from a specific cluster.

    • 15:42 — An internal incident was formally declared.

    • 15:57 — Replacement infrastructure began provisioning to recover failed nodes.

    • 16:32 — All customer addons returned to a healthy state, with the exception of four databases which required manual intervention.


    Thank you for your patience—we know how critical uptime is, and we are reviewing and improving our systems to ensure this doesn’t happen again.

    Will Stewart
    CEO @ Northflank

  • MonitoringUTC

Previous

May 2025 to Jul 2025

Next