With blockchains promising to become "world computers" upon which anyone can create projects ranging from mere toys to critical infrastructure, uptime is crucial, and a five-hour-long outage is devastating.
Solana goes down for five hours
They were doing so well! After suffering a slew of outages during 2022, Solana had seemed to get their act together in 2023 with only one major outage. Now, however, Solana has gone offline again for five hours.
Solana tries turning it off and on again (twice)
It's just like mid-2022 again! As transactions slowed to a crawl, developers embarked on a "coordinated restart" — a euphemism for the rather centralized way this supposedly decentralized network has to routinely go about fixing itself.
One "coordinated restart" apparently wasn't enough, because a second one followed later that day. Developers reportedly didn't know why the blockchain suddenly began to slow, though it followed shortly after validators began adopting a new version of Solana code, pointing to a possible culprit in the new release. The new version had reportedly operated for six months on the testnet before it began to be deployed.
Other theories were also considered, as reported by CoinDesk: "One leading theory was that a 'fat block' gunked up the blockchain's mechanics."
The outage is reminiscent of the ones that plagued the network through 2022, leading some to question whether it could be suitable for replacing critical infrastructure.
One misconfigured node apparently takes the entire Solana network offline
In the latest illustration of our marvelous new decentralized, resilient blockchain future, one single Solana node apparently was able to take down the entire Solana network. Solana outages are nothing new, and tend to end (as this one did) with Solana issuing instructions to the people who run their validators, asking them all to turn them off and on again.
A validator operator reported that "It appears a misconfigured node caused an unrecoverable partition in the network." It's a bit startling that, in a supposedly decentralized network, one single node can bring the entire network offline.
OptiFi developer accidentally closes the project contract, irretrievably locking $661,000
OptiFi, a derivatives defi project, accidentally and permanently shut down the project smart contract, irretrievably locking up $661,000 — the project's entire fund. A developer had been trying to push an update to the project, and ran into issues related to Solana network congestion (a recurring issue). While trying to clean up from a partially-executed transaction, the developer accidentally ran a command that closed the project's primary smart contract.
OptiFi has promised to return user deposits and settle all positions. In a post-mortem, they wrote that they had learned that "Every deployment needs a rigorous process and single point failure can be avoided. Please don't rush like what we did, especially for defi projects". They further outlined a "peer-surveillance approach" in which three people would be required to deploy any changes together. They also asked the Solana team to implement a two-step confirmation for such a potentially destructive command.
Solana network halted again
Solana is one of the more popular proof-of-stake blockchains, and is often trotted out as an alternative to Ethereum when people bring up Ethereum's environmental impact, slowness, or high transaction costs.
However, Solana has been plagued with stability issues, and on June 1 it was taken offline by its developers for what CryptoWhale says was the eighth time this year. This occurred only days after an incident in which the Solana blockchain clock drifted significantly behind real-world time.
Solana loses track of time
The Solana blockchain clock drifted about 30 minutes behind real-world time on May 26, as a result of slower-than-usual slot times. Solana's status page read that "this has no impact on performance or network operations", though The Block noted that this time drift could result in smaller staking payouts.
Blockchain timekeeping is also selling point of Solana, which talks up its "proof of history" algorithm in a blog post where Solana Labs co-founder Anatoly Yakovenko says, "our clocks never drift".
Solana goes down again
On April 30, NFT minting bots began flooding the Solana network with 4 million transactions per second, causing the network to lose consensus. The project tweeted that "Engineers are still investigating why the network was unable to recover, and validator operators prepare for a restart." The network was offline for seven hours.
This is hardly the first instability the network has demonstrated, much to the chagrin of its users. Transaction flooding is an issue on Solana in part because of the low transaction fees compared to networks like Bitcoin and Ethereum, which have relatively high gas fees that would make flooding extremely expensive.
Investors on Solana-based defi platforms experience mass liquidations caused by yet another outage
Solana was so overloaded with bot transactions that users couldn't transact. As the cryptocurrency market in general continued to tank, users rushed to top up the collateral they had provided to keep their loans from being liquidated and found they couldn't get the transfers to go through. One user reported spending eight hours trying unsuccessfully to add collateral, before eventually getting liquidated and losing 500 SOL (about $47,500). It took Solana 24 hours to even identify the cause of the issue, and another 24 before they were able to resolve it. Traders watching their loans get liquidated were not impressed when Solana Labs co-founder tweeted "lol", with a screenshot of a Solana node showing high amounts of duplicate packets.
Solana experiences outage or "congestion", depending on who you believe
Journalist Colin Wu reported that the Solana blockchain had an approximately four-hour-long outage due to a DDoS attack, while many others noticed enormous slowdowns. Solana later claimed there had been no DDoS and no outage, and that there was just "some congestion", a claim several crypto outlets reported at apparent face value. The "congestion" was reported to have been from the launch of the highly-hyped SolChicks NFT project, although you have to wonder how a blockchain that claims to be able to handle 50,000 transactions per second (though averaging around 1,700 in reality) could be affected so majorly by a single project. This was the third apparent network issue suffered by the Solana blockchain over the past few months.