A Routing Bug Took Solana 86% of the Way to Losing Finality

Solana came within 4.51 percentage points of losing the ability to finalize transactions on Wednesday morning, after a single misconfigured route at one hosting provider knocked 28.83% of staked SOL offline. The chain stops finalizing at 33.34%, meaning the network got roughly 86% of the way there.

Teraswitch, the provider, has published a writeup explaining how it happened. The company uses a default route internally to signal that an edge router can reach the internet, with each site normally preferring the one its own routers originate. A default route from its Miami site was propagated with its metric and communities stripped, and a route reflector in Amsterdam pushed it into Europe and Asia-Pacific.

Edge routers there read it as locally originated and preferred it over their real one, then passed it to the data center core, which rejected it as invalid. Twelve sites in London, Amsterdam, Dublin, Frankfurt, Singapore and Tokyo were left with nothing valid to forward to. North America was untouched. Engineers found the fault within ten minutes and service returned at 04:16:15 UTC.

Staking protocol Marinade Finance, which pulled the numbers afterwards, found the outage concentrated on a single autonomous system. AS20326 carries 118,890,767 SOL, more than a quarter of everything staked on the network, and 94% of it went offline in the same minutes.

That is already above the ceiling meant to prevent this. The Solana Foundation’s delegation program caps any one autonomous system at 25% of network stake. AS20326 sits at 27.34%.

80 million SOL sat and waited

Marinade found that 59 validators holding 80.2 million SOL came back inside the same narrow window in Amsterdam, Frankfurt and Tokyo, having waited for the routing to reconverge rather than switching to anything else. Helius, the second-largest validator on Solana, was down fore the full 33 minutes. Of 74 operators Marinade could measure, three “came back clean”: Laine and Cogent Crypto, both run by Sol Strategies, and Lion3d.

The exposure also runs wider than one provider, with a further 14.1 million SOL going offline in the same minutes across latitude.sh, Limestone, Butterfly Research and Allnodes, which Marinade said it could not explain from the data, and which suggests counting stake by hosting provider understates what fails together.

Marinade turned the same analysis on itself, reporting that four autonomous systems hold two-thirds of the stake its allocation model distributes, with one, AS395201, accounting for 36.94%. “Nobody should be comfortable with that, us included,” the firm tweeted, noting that it would review its concentration limits per network and per data center, and start publishing whether a validator runs hot swap and automatic failover, which cannot currently be determined from outside.

The 333 SOL in missed rewards will be covered by validator bonds at the end of the epoch. Had delinquency crossed a third, nothing would have finalized for any SOL holder anywhere, and no bond covers that. The last time Solana halted outright, in February 2024, it took close to five hours to restart.