PNP SDK Service Outage - Incident Report

Date: October 15, 2025
Time:: 10:50 UTC - 11:50 UTC
Status: Resolved

Summary

Earlier today, a broken deployment caused a temporary service outage affecting applications utilizing the PNP SDK. The issue was identified shortly after deployment and immediately reverted, restoring normal functionality.

Impact

Only applications and services that depend on the PNP SDK experienced degraded performance or failed requests during the outage period. All other components and systems remained fully operational.

Resolution

Upon detection, the faulty deployment was quickly reverted to the previous stable version. Once reverted, all affected applications recovered automatically, and no further errors were observed.

In the past hour, we experienced an outage in the Asia region due to a failure in one of our cloud components. This issue was isolated to the Asia region and did not affect other regions. The system has since been restored, and operations are back to normal.

We are investigating Internet connectivity issue outside of the AWS network, which may be impacting connectivity between some customer networks in APAC region to US-EAST-1. Connectivity to instances and services within the Region is not impacted by the event.

In last 1 hour, We noticed a sudden huge traffic spike in europe region which triggered rate limit errors for some of the new users in europe region. Our systems are usually autoscalable but some services like distributed key generation takes some time to auto scale, we identified the spike and manually scaled it up. System seems to be back to normal now.

Old users were not impacted in this outage.