On February 2 we detected degraded performance affecting some of our Ethereum users for Ropsten network API requests, around the time the Petersburg fork block for Ropsten (4939394) was mined and shared over the peer-to-peer network. Upon further investigation we determined that, while all of our Ropsten nodes were updated ahead of time with Ethereum clients ready for the Petersburg fork, our network was having trouble receiving Petersburg blocks consistently.
After gathering as much information as we could, and coordinating with members of the Ethereum Foundation, we pinpointed the root cause to a lack of miners running updated Ethereum clients: without Petersburg-aware miners the new chain failed to progress with new blocks, while the pre-Petersburg chain continued to grow. Having competing chains further resulted in added confusion for Ropsten users, as their queries could return inconsistent data: several blockchain explorers and other service providers ended up on the non-Petersburg chain.
Recognizing the importance, for dapps and developers, of having the Constantinople and Petersburg updates applied to a cross-client Proof of Work test network like Ropsten, we decided to contribute by standing up several Petersburg-aware miners (we don’t routinely run Ropsten miners as part of our public service, nor of our internal infrastructure), which soon began committing new blocks. This allowed the Petersburg chain to grow and over time become the canonical Ropsten chain. With other operators also up to date, the network stabilized.
We had enough lead time to upgrade relevant infrastructure to be fork-aware, but we did not anticipate the need to stand up miners ahead of time to ensure that Petersburg blocks would be reliably mined on Ropsten. Our action items:
1. Further improve on our internal processes and automation so that on any future Ropsten fork we’ll be able to more quickly and effectively start new miners if necessary;
2. Continue to proactively communicate with the Ethereum Foundation, the core developers and the community to ensure any situation like this can be anticipated and fully understood;
3. As always, communicate with developers to understand what tools, APIs and infrastructure best allow them to get their job done. We continue to support Proof of Authority test networks as well, like Rinkeby, Kovan and soon the newly-released Görli, which may be a good alternative in many contexts.
We want to stress the importance of miners running up-to-date clients in preparation for forks. If you are helping maintain Ropsten by running a miner (thanks!) please keep it up to date.