The Shwap Upgrade: Scaling the Data Availability Network by 12x

Celestia, a crypto project that builds modular blockchain networks, has announced the completion of its first upgrade, The Shwap upgrade. The upgrade comes with a new set of features that makes data availability (DA) network sampling 12 times faster than before.

Celestia, a modular blockchain network builder, has completed its first upgrade, dubbed "The Shwap Upgrade." The upgrade introduces a new set of features designed to make data availability (DA) network sampling 12 times faster than before.

The update is now live on the Arabica and Mocha testnets using celestial-node v0.18.12. Shwap enables smaller nodes and leather blocks by reducing storage requirements by sixteen times, advancing towards the main goal of the Celestia community roadmap: scaling to 1 GB blocks.

Celestia Improvement Proposal (CIP)-19 defines Shwap as a new messaging protocol and storage system for the DA network. Shwap minimizes the need for light nodes while enabling massive scaling of the DA throughput. This accelerates existing crypto applications and makes new applications easily accessible.

Lower-prerequisite applications allow anyone to run a light node in a browser or wallet, enabling fully verifiable web apps. After further testing, Shwap is expected to drop its Mainnet Beta sometime in November.

Inside the DA Network

The DA Network currently operates on two primary protocols. The first handles circulating chain headers and utilizes a go-header system, while the second manages block data, which is organized into a data square of small shares or samples.

Light nodes are tasked with sampling these shares, and the current protocol, inherited from the Devnet-era IPLG system, centers on this sampling strategy. While functional, this protocol presents several challenges to scaling, especially to accommodate one-gigabyte blocks.

Challenges:

A common problem in blockchain engineering is hash addressability, which is the ability to identify and locate data stored as hashes across the nodes in a blockchain network. Hash addressability has some inherent benefits, such as no duplicates and greater flexibility.

For example, one can store the hashes of hash-based data structures (like data square storage). When light nodes perform verifications, they download the partial data and Merkle proof of each partial data by fetching each node individually.

With a 528×528 data square, the four-step process becomes a bottleneck. A full node has the global index of the Merkle tree, making the process slower. Thus, hash addressability pointed to two major issues: storing Merkle proofs rather than recalculating them on the fly and relying on a slower data access pattern (O(log2n)) when O1 can be used to efficiently access large data volumes.

Enter Shwap:

The Shwap upgrade, deriving its name from the terms "share" and "swap," addresses the scalability challenges faced by the DA network by revamping how data is stored and accessed within the protocol.

Moreover, in Shwap, the previous, inefficient, experimental version of block reconstruction is also swapped out in favor of a more production-ready version in the upcoming upgrade. In pre-Shwap, the set time for 2 MB was 2 seconds; however, in post-Shwap, the set time for 2 MB is reduced to 10 milliseconds.