All service subsystems have recovered and are back to normal operation. Work on the full post-mortem has begun and will be posted later today.
Posted Nov 11, 2020 - 14:42 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 11, 2020 - 14:28 UTC
Update
We are continuing with service recovery efforts. Most mainnet subsystems are returning to normal behavior now but we are continuing to triage and check each subsystem. We have re-enabled access from core accounts. Currently Archive requests are still disabled but we expect those to be available again shortly.
Posted Nov 11, 2020 - 13:12 UTC
Update
While we are deploying the fix we are limiting traffic at our API gateway to expedite service recovery. Core account users will temporarily receive a service disabled message while system health is restored.
Posted Nov 11, 2020 - 12:37 UTC
Update
We apologize to our users for the extended service disruption. We are continuing to work on the fix to ensure its correctness and completeness. The root cause was traced to several components within our infrastructure which were locked to an older stable version of the go-ethereum client which encountered a critical consensus bug at block 11234873. This affected several Geth versions including 1.9.9 and 1.9.13. Components running 1.9.19 and later were unaffected. A full post-mortem will be completed and shared after the incident is resolved.
Posted Nov 11, 2020 - 11:23 UTC
Identified
We've identified the root cause and are preparing a fix to restore service functionality
Posted Nov 11, 2020 - 09:47 UTC
Update
We've brought additional capacity online for the affected components and are continuing to triage the root cause of the outage. System recovery operations are in progress.
Posted Nov 11, 2020 - 09:03 UTC
Investigating
We are currently experiencing a service outage for our Ethereum Mainnet API. Our on-call team is investigating and working to restore service functionality. We will post updates here as we have them.
Posted Nov 11, 2020 - 08:12 UTC
This incident affected: Infura Ethereum API (Mainnet HTTPS JSON-RPC API, Mainnet WebSocket JSON-RPC API, Mainnet Filters).