Polygon zkEVM Outage: Deep Dive Reveals Ethereum Reorg and Synchronization Issues

Over a week after an outage halted block production on Polygon's zkEVM network for 14 hours, developers have released a post-mortem report explaining that an Ethereum mainnet reorg caused the sequencer to accept transactions with deprecated timestamps, leading to the outage.

Polygon zkEVM Outage: Deep Dive into the Causes and Mitigation Measures

Over a week ago, Polygon Labs' Ethereum Layer 2 solution, Polygon zkEVM, experienced an unexpected outage that halted block production on the network for several hours. While the issue has since been resolved, the specific causes of the disruption remained shrouded in uncertainty until now. In a comprehensive post-mortem report, developers have shed light on the root causes of the event and outlined proactive steps taken to prevent future occurrences.

Unraveling the Ethereum Reorg Trigger

In their initial statements, Polygon Labs attributed the Polygon zkEVM outage to an Ethereum mainnet "reorg." A reorg, short for reorganization, occurs when a consensus disagreement over transaction ordering or validity arises, forcing the network to converge on a single version of the blockchain history, prioritizing the chain with the most computational work or stake backing it.

According to the developers' post-mortem analysis, the Ethereum reorg in this instance resulted in the omission of a Polygon zkEVM deposit transaction. This critical change was not detected promptly by the Layer 2 chain's synchronizer, a component responsible for monitoring and relaying Layer 1 changes to Layer 2.

Synchronization Failure and Transaction Rejection

The synchronizer's lapse in detecting the omitted deposit transaction had a cascading effect on the Polygon zkEVM sequencer, the component responsible for ordering and batching transactions for posting to the Ethereum mainnet. Unaware of the updated mainnet state, the sequencer continued to order transactions with timestamps that had become outdated on the mainnet.

As a result, when these transactions were batched to the mainnet, they were rejected by the network due to their invalid timestamps. "Valid proofs for these transactions were generated, but the result of these transactions were no-operation, as if they were not there," the developers explained.

Impact and Remediation

Approximately 4,000 transactions were affected by the outage. The developers attributed the prolonged 14-hour downtime to the efforts required to reconcile these affected transactions.

In response to the incident, developers have released updated versions of the zkEVM node and prover components, including enhancements to the synchronizer to improve its ability to detect and respond to Layer 1 changes. Additionally, they have implemented an extra safeguard in the sequencer to cross-check transaction timestamps with the mainnet before batching them for submission.

TVL Resilient, but Some Transactions May Require Further Attention

Polygon zkEVM's total value locked (TVL) appears to have remained largely unaffected by the outage. At the time of writing, L2Beat data indicates a slight increase from approximately $150 million to $153 million since the incident.

However, Polygon Labs has cautioned that some of the affected transactions may still be pending processing. DApps that have identified affected transactions are encouraged to reach out to Polygon Labs for assistance.

Contextualizing the Outage

The Polygon zkEVM outage highlights the growing prevalence of network disruptions in the cryptocurrency space, particularly in the Ethereum Layer 2 ecosystem. In December 2023, Arbitrum, the largest Ethereum Layer 2 network by TVL, experienced an outage lasting over five hours due to a surge in network activity. Similarly, zkSync Era, the largest ZK-rollup by TVL, faced a four-hour outage in the same month.

Importance of Post-Mortem Analysis

This post-mortem report provides valuable insights into the causes and resolution of the Polygon zkEVM outage, addressing lingering uncertainties surrounding the incident. It underscores the importance of post-mortem analyses in enhancing the understanding of network disruptions, identifying areas for improvement, and building more resilient systems.