At approximately 13:25:30 UTC on Friday, April 22 a configuration change to an internal network caused an unexpected disruption of network availability to many of our internal services. This manifested to users as timeouts and 5XX errors from most of our endpoints. Our monitoring systems alerted us to the outage and our teams immediately began investigating the issue. Once the issue with the networking configuration change was identified, we rolled back the change and restored network access. Service for our JSON-RPC APIs was restored at approximately 14:52:00 UTC. Our websockets API took longer to recover and was restored at approximately 15:58:00 UTC.
In response to this incident, we restricted access to the affected network settings in order to prevent further modifications without additional validation and review. In addition, we will be conducting an audit of all our network controls and topologies to ensure that the impact of changes are isolated to a single service, and any shared infrastructure changes are closely monitored. We will also continue our efforts to compartmentalize and more extensively validate changes to production resources before they are fully rolled out.
We take the uptime and performance of the Infura platform very seriously, and view every failure as an improvement opportunity for our team.
Posted Apr 28, 2022 - 02:05 UTC
Resolved
Websocket functionality has been restored. At this point all Infura services are back to full operating capacity. Again, we apologize for the inconvenience this has caused and we will be issuing a full post-mortem once we have gathered all the details. In the meantime, if you have any questions or need help recovering your own systems in response to this incident, please reach out to our support team.
Posted Apr 22, 2022 - 16:27 UTC
Update
Full RPC functionality has been restored. Websocket functionality remains in a degraded state as additional capacity comes online.
Posted Apr 22, 2022 - 15:54 UTC
Update
We are in the process of rolling out a fix and our core systems are starting to recover. Full service functionality will be reestablished as remaining components come back online.
Posted Apr 22, 2022 - 15:08 UTC
Monitoring
We are in the process of rolling out a fix and our core systems are starting to recover. Full service functionality will be reestablished as remaining components come back online.
Posted Apr 22, 2022 - 15:05 UTC
Update
We are continuing to work on a fix for this issue.
Posted Apr 22, 2022 - 14:01 UTC
Identified
The root cause has been identified and we are working on a fix.
Posted Apr 22, 2022 - 13:57 UTC
Investigating
We are aware of an outage of some of our ETH1, Palm, Optimism, Polygon, Filecoin, and Arbitrum API endpoints. At the moment, the outage is limited to the listed components, which are temporarily unavailable. We are currently investigating to pinpoint a root cause and remediate.