Ethereum 2.0 core developer, Raul Jordon explained the reasons why the beacon chain started missing a lot of block proposals. He revealed that Prysm represented a large portion of Ethereum 2.0 operators and hence, naturally became the likely suspect.
This was the first major incident on Ethereum 2.0. So what exactly happened? Prysm validators failed to produce blocks for a period of 18 epochs on Friday, 23rd of April. The very next day, the validators reported the same issue again and again this time for 18 epochs. This issue prompted a reduction in validator participation.
Jordon admitted to being able to reproduce the error locally and said that the problem demonstrated an already known issue regarding ETH 1.0 data voting and validator deposits. This is the first time, that the issue had been reported to the team since it had never been widespread in any testnet nor in mainnet.
The bug was reportedly extremely tricky to diagnose, and it took the team of developers a period of 30 hours before getting to a root cause.
Impact on Ethereum 2.0
According to the developer, this event should not reduce the confidence in Ethereum 2.0. He explained that there was no consensus failure, and the impact and scope of the incident were small at the scale of the current 2.0 mainnet. Acknowledging the robustness of the ecosystem, Jordon stated,
“.. [ETH2.0].. with very high validator participation numbers since genesis, and has never skipped finality for any epoch. The ability for the network to go back to perfect operation after incident resolution instead increases confidence in the resilience of the blockchain, from our perspective.”
According to the post-mortem report, the total loss incurred from the incident is estimated to be roughly 25 ETH. All clients of the ecosystem were reportedly subjected to missed attestation penalties and the average per-validator loss was nearly 205K gwei [around $0.50 in the current market price of the crypto-asset].
In addition, the operational impact of the failure in block proposals is estimated to have surpassed 120 man-hours in the 30 hours between detection and resolution.