One misconfigured node apparently takes the entire Solana network offline

In the latest illustration of our marvelous new decentralized, resilient blockchain future, one single Solana node apparently was able to take down the entire Solana network. Solana outages are nothing new, and tend to end (as this one did) with Solana issuing instructions to the people who run their validators, asking them all to turn them off and on again.

A validator operator reported that "It appears a misconfigured node caused an unrecoverable partition in the network." It's a bit startling that, in a supposedly decentralized network, one single node can bring the entire network offline.

OptiFi developer accidentally closes the project contract, irretrievably locking $661,000

OptiFi, a derivatives defi project, accidentally and permanently shut down the project smart contract, irretrievably locking up $661,000—the project's entire fund. A developer had been trying to push an update to the project, and ran into issues related to Solana network congestion (a recurring issue). While trying to clean up from a partially-executed transaction, the developer accidentally ran a command that closed the project's primary smart contract.

OptiFi has promised to return user deposits and settle all positions. In a post-mortem, they wrote that they had learned that "Every deployment needs a rigorous process and single point failure can be avoided. Please don't rush like what we did, especially for defi projects". They further outlined a "peer-surveillance approach" in which three people would be required to deploy any changes together. They also asked the Solana team to implement a two-step confirmation for such a potentially destructive command.

Solana network halted again

Solana is one of the more popular proof-of-stake blockchains, and is often trotted out as an alternative to Ethereum when people bring up Ethereum's environmental impact, slowness, or high transaction costs.

However, Solana has been plagued with stability issues, and on June 1 it was taken offline by its developers for what CryptoWhale says was the eighth time this year. This occurred only days after an incident in which the Solana blockchain clock drifted significantly behind real-world time.

Solana loses track of time

The Solana blockchain clock drifted about 30 minutes behind real-world time on May 26, as a result of slower-than-usual slot times. Solana's status page read that "this has no impact on performance or network operations", though The Block noted that this time drift could result in smaller staking payouts.

Blockchain timekeeping is also selling point of Solana, which talks up its "proof of history" algorithm in a blog post where Solana Labs co-founder Anatoly Yakovenko says, "our clocks never drift".

Solana goes down again

On April 30, NFT minting bots began flooding the Solana network with 4 million transactions per second, causing the network to lose consensus. The project tweeted that "Engineers are still investigating why the network was unable to recover, and validator operators prepare for a restart." The network was offline for seven hours.

This is hardly the first instability the network has demonstrated, much to the chagrin of its users. Transaction flooding is an issue on Solana in part because of the low transaction fees compared to networks like Bitcoin and Ethereum, which have relatively high gas fees that would make flooding extremely expensive.

Investors on Solana-based defi platforms experience mass liquidations caused by yet another outage

Tweet from aeyakovenko: "lol" with a screenshot of a spike in network trafficAnatoly Yakovenko's tweet during the outage (attribution)
Solana was so overloaded with bot transactions that users couldn't transact. As the cryptocurrency market in general continued to tank, users rushed to top up the collateral they had provided to keep their loans from being liquidated and found they couldn't get the transfers to go through. One user reported spending eight hours trying unsuccessfully to add collateral, before eventually getting liquidated and losing 500 SOL (about $47,500). It took Solana 24 hours to even identify the cause of the issue, and another 24 before they were able to resolve it. Traders watching their loans get liquidated were not impressed when Solana Labs co-founder tweeted "lol", with a screenshot of a Solana node showing high amounts of duplicate packets.

Solana experiences outage or "congestion", depending on who you believe

An illustration of a yellow chick with a large brown afro, bruised eyes, and black dress shoesSolChick #535 (attribution)
Journalist Colin Wu reported that the Solana blockchain had an approximately four-hour-long outage due to a DDoS attack, while many others noticed enormous slowdowns. Solana later claimed there had been no DDoS and no outage, and that there was just "some congestion", a claim several crypto outlets reported at apparent face value. The "congestion" was reported to have been from the launch of the highly-hyped SolChicks NFT project, although you have to wonder how a blockchain that claims to be able to handle 50,000 transactions per second (though averaging around 1,700 in reality) could be affected so majorly by a single project. This was the third apparent network issue suffered by the Solana blockchain over the past few months.

No JavaScript? That's cool too! Check out the Web 1.0 version of the site to see more entries.