The Evolution of Blockchain Data Indexing Technology: From Node to AI-Powered Full-Chain Data Services

1 Introduction

Since the first batch of dApps emerged in 2017, to the present where various financial, gaming, and social dApps are flourishing, have we thought about the data sources on which these decentralized applications rely?

In 2024, AI and Web3 become hot topics. In the field of AI, data is like the source of life, driving systems to continuously learn and evolve. Without the support of massive amounts of data, even the most sophisticated algorithms cannot truly demonstrate their intelligence.

This article will delve into the development of blockchain data accessibility, analyze the evolution of data indexing technology, and compare the features of protocols such as The Graph, Chainbase, and Space and Time in terms of data services and product architecture.

2 The Complexity and Simplicity of Data Indexing: From Blockchain Nodes to Full-Chain Database

2.1 Data Source: Blockchain Node

Blockchain, as a decentralized ledger, has its nodes responsible for recording, storing, and disseminating transaction data on the chain. Each node keeps a complete copy of the blockchain data, ensuring the decentralized nature of the network. However, for ordinary users, building and maintaining a node is not an easy task, as it requires specialized skills and faces high costs.

To solve this problem, RPC Node providers have emerged. They are responsible for managing nodes and providing data access services through RPC endpoints. Public RPC endpoints are free but have limitations, while private endpoints offer better performance but still have room for improvement in efficiency. Nevertheless, the standardized API interfaces of node providers lower the threshold for users to access on-chain data, laying the foundation for subsequent data parsing and applications.

2.2 Data Analysis: From Prototype Data to Usable Data

The raw data provided by blockchain nodes is often encrypted and encoded, increasing the difficulty of parsing. The data parsing process transforms complex prototype data into a format that is easy to understand and operate, which is a key link in the entire data indexing process and directly affects the efficiency and effectiveness of blockchain data applications.

2.3 The Evolution of Data Indexers

As the amount of Blockchain data surges, the demand for data indexers is growing daily. Indexers achieve efficient querying by organizing on-chain data and sending it to a database. They provide a unified query interface, allowing developers to quickly retrieve the required information using standardized languages.

Different types of indexers each have their advantages:

1. Full Node Indexer: Extracts data directly from the Blockchain Node, ensuring completeness but with high resource consumption.
2. Lightweight Indexer: Relies on full nodes to obtain specific data, reducing storage requirements but potentially increasing query time.
3. Dedicated Indexer: Optimized for specific types of data or Blockchain, such as NFT data or DeFi transactions.
4. Aggregated Indexer: Integrates multi-chain and off-chain data, providing a unified query interface, suitable for multi-chain dApps.

Faced with a massive amount of data, mainstream indexer protocols not only support multi-chain indexing but also customize data parsing frameworks for different application needs.

The emergence of indexers has significantly improved data indexing and query efficiency. Compared to traditional RPC endpoints, indexers support complex queries, data filtering, and analysis, and can aggregate multi-chain data sources. By operating in a distributed manner, indexers provide enhanced security and performance, reducing the risk of interruptions.

2.4 Full Chain Database: Aligning to Stream Priority

As the application requirements become more complex, standardized indexing formats struggle to meet diverse query needs. In modern data pipeline architectures, the "stream-first" approach has become a solution to address the limitations of traditional batch processing, enabling real-time data ingestion, processing, and analysis.

Blockchain data service providers are also moving towards building data streams. Solutions like The Graph's Substreams, Goldsky's Mirror, and the real-time data lakes provided by Chainbase and SubSquid aim to address the needs for real-time parsing and comprehensive querying.

Redefining on-chain data management through the lens of modern data pipelines, we can envision a future where high-performance datasets are tailored for any business use case.

3 AI + Database? In-depth Comparison of The Graph, Chainbase, Space and Time

3.1 The Graph

The Graph network provides multi-chain data indexing and query services through decentralized nodes. Its core product model includes a data query execution market and a data indexing cache market, serving users' product query needs.

Subgraphs ( are the fundamental data structure of The Graph network, defining the methods of data extraction and transformation. The network consists of four roles: indexers, curators, delegators, and developers, ensuring system operation through economic incentives.

The AutoAgora, Allocation Optimizer, and AgentC tools developed by Semiotic Labs utilize AI technology to optimize index pricing, resource allocation, and user query experience, enhancing the intelligence level of the system.

![Reading, indexing to analysis, a brief overview of the Web3 data indexing track])https://img-cdn.gateio.im/webp-social/moments-cf9a002b9b094fbbe3be7f611001b5c1.webp(

) 3.2 Chainbase

Chainbase, as a full-chain data network, integrates multi-chain data, making it easier for developers to build and maintain applications. Its features include:

• Real-time Data Lake: Provides instant access to blockchain data streams.
• Dual-chain architecture: Built on Eigenlayer AVS for the execution layer, enhancing the programmability and composability of cross-chain data.
• Innovative data format: Introduce the "manuscripts" standard to optimize data structuring and utilization.
• Encrypted World Model: Combining AI technology, it has created Theia, a model for understanding and predicting Blockchain transactions.

Chainbase leverages the AI model Theia to deeply mine the value of on-chain data, providing intelligent data services and enhancing the platform's competitiveness.

![Read, Index to Analyze, Brief on Web3 Data Indexing Track]###https://img-cdn.gateio.im/webp-social/moments-b343cab5112c1a3d52f4e72122ae0df2.webp(

) 3.3 Space and Time

Space and Time ###SxT( is committed to building a verifiable computation layer that expands zero-knowledge proof technology. Its core innovation, Proof of SQL, ensures that SQL queries on decentralized data warehouses are verifiable and tamper-proof.

SxT collaborates with Microsoft AI Lab to develop generative AI tools that simplify Blockchain data processing. Users can query Blockchain data in natural language within Space and Time Studio, and the AI automatically converts it to SQL and executes the query.

![Reading, indexing to analysis, a brief overview of the Web3 data indexing track])https://img-cdn.gateio.im/webp-social/moments-97443cbd177ac4ffd1665da670ffbf12.webp(

) 3.4 Difference Comparison

The three platforms each have their own characteristics: The Graph focuses on decentralized indexing and query services, Chainbase emphasizes real-time data lakes and AI-driven data analysis, while Space and Time highlights verifiable computation and natural language queries.

![Reading, indexing to analysis, a brief overview of the Web3 data indexing track]###https://img-cdn.gateio.im/webp-social/moments-0742180b7da8a9dcddafc465a4dba9cb.webp(

Conclusion and Outlook

Blockchain data indexing technology has evolved from node data sources to AI-enabled full-chain data services, improving data access efficiency and intelligence levels. In the future, with the development of technologies such as AI and zero-knowledge proofs, blockchain data services will become further intelligent and secure, continuing to drive innovation as an industry infrastructure.