After an “insane quantity of data” invaded the proof-of-stake chain, pushing validators out of consensus and grinding still block production, Solana stakeholders hurried to put the network back together Saturday night.
Bots flooded Candy Machine, a popular NFT minting tool, earlier Saturday, causing an unusual flood of inbound traffic: four million transaction proposals and 100 gigabits of data per second — a network record, according to one source at the Solana Foundation.
This swarm drove validators out of an agreement for unknown reasons. At 4:32 p.m. EST, block production became impossible, and the network fell black. Validators rebooted the cluster at slot 131973970 by 11:00 p.m. EST, coordinating using Solana’s Discord channels and a Google doc generated by one of the validators.
Anatoly Yakovenko, a co-founder who was away for much of the squabble, lauded the validator community for spearheading mainnet recovery. On Saturday, he was chastised on Twitter for reportedly going “MIA” amid a network outage.
Solana outage caused by a spike in transaction volume
Unlike the 17-hour downtime in September, the hard fork restart on Saturday did not result in new-and-improved code filling throughout the validators. They just resumed where the network had left off seven hours before.
Validators debated whether to write code that would temporarily halt Candy Machine transactions as they prepared for the restart. Some in the Discord questioned whether such a move amounted to censorship. Regardless, it would only work if two-thirds of validators agreed to participate. On Saturday night, it appeared that few did.
Solana, a proof-of-stake blockchain, claimed on Twitter that it successfully completed a cluster restart at around 11 p.m. ET on Saturday, following an approximately seven-hour downtime when the network failed to establish an agreement.
Phantom wallet and decentralized exchange are two services provided by the Solana ecosystem. As RPC node providers plodded back up, Mango Markets battled to get back on its feet.
The interruption caused a bloodbath in SOL markets, which lasted only a few minutes. According to CoinGecko, SOL token hit a 24-hour low of $83.13 roughly three hours into the outage before rebounding back above $89 again.
Engineers were still looking into why the network couldn’t recover from the transaction overload that triggered the outage, according to the Twitter message.