Surge

Surge is an internal project born from a simple conviction: blockchain data should be as queryable and explorable as any other dataset. Existing block explorers show transactions and balances, but the deeper questions — patterns in coretime allocation, storage evolution across runtimes, cross-chain asset flows — require infrastructure that doesn't yet exist. We're building that infrastructure, with the goal of layering on intelligence-powered analysis to let users ask questions of the data directly.

The Problem

Blockchain data is abundant but inaccessible. Archive nodes store everything, but querying them can be slow, expensive, and limited to predefined access patterns. Want to correlate validator session changes with block production metrics? Trace how storage namespaces evolve across runtime upgrades? Analyse on-demand coretime pricing trends? Each question requires custom tooling, direct RPC calls, or maintaining bespoke indexing infrastructure. The result is that most blockchain analytics stays surface-level — block heights, transaction counts, account balances — while the data that would actually inform decisions remains locked inside opaque storage tries.

Our Approach

The architecture draws from event sourcing principles: treat the blockchain itself as an immutable event log, replay it into a streaming backbone, and build purpose-specific read models (projections) from the stream. Rather than querying archive nodes directly, we replay finalised blocks and write the resulting raw data to typed topics on a message stream. This first stage effectively liberates the data — extracting it from the archive node's storage format into something that can be iterated, filtered, and projected independently.

From there, specialised projectors subscribe to stream topics and build structured views of the data. Each projector is purpose-built for its domain: blocks and their weight metrics, extrinsic outcomes with decoded call data, system events with phase tracking, coretime core assignments and on-demand spot pricing, storage namespace snapshots, validator sessions and authority rotations, token issuance, and runtime metadata evolution. Projectors produce both stateful projections (latest state, overwritten on update) and append-only projections (full history, immutable once written) — different query patterns demand different storage strategies.

The pipeline outputs to multiple backends simultaneously. ScyllaDB handles structured projection state — its wide-column model maps naturally to blockchain data where queries are partition-key driven and writes are append-heavy. GreptimeDB captures time-series metrics — block times, weight utilisation, spot prices — optimised for range queries and downsampled aggregation. The message streams themselves serve as the integration layer, decoupling data extraction from projection logic and allowing new projectors to be added without replaying the chain.

A REST API sits in front of the projection stores, with content negotiation supporting both binary and JSON responses. The web frontend provides interactive dashboards with chart visualisations for exploring block metrics, coretime analytics, storage state, and runtime metadata across chains.

What's Next

The indexing pipeline and web UI represent the foundation — not the end goal. The next phase is an agentic intelligence layer: an abstraction over the projection stores, exposed via MCP so an LLM can query them directly, allowing users to compose ad-hoc queries across the full dataset through natural language. Instead of navigating pre-built dashboards, a user could ask "show me coretime spot price correlation with block weight over the last epoch" and get a dynamically generated visualisation. The structured projections we've built are designed with this in mind — typed, well-partitioned data that an agent can reason about and query programmatically.

We're also expanding projection coverage — graph-based projections for mapping account relationships and cross-chain flows, deeper parachain support beyond the relay and system chains, and historical comparison tooling for tracking how on-chain state evolves across runtime upgrades.

Surge is currently in closed preview. If you're interested in early access, join the waiting list at surge.rs.