Degraded performance after the Petersburg fork on Ropsten
Incident Report for Infura
Postmortem

What we observed

On February 2 we detected degraded performance affecting some of our Ethereum users for Ropsten network API requests, around the time the Petersburg fork block for Ropsten (4939394) was mined and shared over the peer-to-peer network. Upon further investigation we determined that, while all of our Ropsten nodes were updated ahead of time with Ethereum clients ready for the Petersburg fork, our network was having trouble receiving Petersburg blocks consistently.

What we found

After gathering as much information as we could, and coordinating with members of the Ethereum Foundation, we pinpointed the root cause to a lack of miners running updated Ethereum clients: without Petersburg-aware miners the new chain failed to progress with new blocks, while the pre-Petersburg chain continued to grow. Having competing chains further resulted in added confusion for Ropsten users, as their queries could return inconsistent data: several blockchain explorers and other service providers ended up on the non-Petersburg chain.

How we remediated

Recognizing the importance, for dapps and developers, of having the Constantinople and Petersburg updates applied to a cross-client Proof of Work test network like Ropsten, we decided to contribute by standing up several Petersburg-aware miners (we don’t routinely run Ropsten miners as part of our public service, nor of our internal infrastructure), which soon began committing new blocks. This allowed the Petersburg chain to grow and over time become the canonical Ropsten chain. With other operators also up to date, the network stabilized.

What we plan to do in the future

We had enough lead time to upgrade relevant infrastructure to be fork-aware, but we did not anticipate the need to stand up miners ahead of time to ensure that Petersburg blocks would be reliably mined on Ropsten. Our action items:

1. Further improve on our internal processes and automation so that on any future Ropsten fork we’ll be able to more quickly and effectively start new miners if necessary;

2. Continue to proactively communicate with the Ethereum Foundation, the core developers and the community to ensure any situation like this can be anticipated and fully understood;

3. As always, communicate with developers to understand what tools, APIs and infrastructure best allow them to get their job done. We continue to support Proof of Authority test networks as well, like Rinkeby, Kovan and soon the newly-released Görli, which may be a good alternative in many contexts.

We want to stress the importance of miners running up-to-date clients in preparation for forks. If you are helping maintain Ropsten by running a miner (thanks!) please keep it up to date.

Posted Feb 07, 2019 - 23:21 UTC

Resolved
With enough miners on the Petersburg side of the fork, the issues we've identified on Ropsten have been resolved.
Posted Feb 03, 2019 - 18:05 UTC
Monitoring
Miners with updated clients have been deployed, and the Petersburg chain has now progressed enough to be a valid canonical chain for Ropsten. Our API services are operating as expected and we're continuing to monitor for any additional problems.
Posted Feb 02, 2019 - 23:30 UTC
Identified
We have identified a network-wide issue with miners on the Ropsten network running outdated software. We're assisting the community in restoring network integrity.
Posted Feb 02, 2019 - 05:25 UTC
Investigating
Some users are experiencing degraded performance on Ropsten after the Petersburg fork; we are assisting the community in investigating.
Posted Feb 02, 2019 - 03:06 UTC