@attestate/crawler

A JavaScript library for retrieving on-chain storage and generating data derivatives.

Modular: Extract, Transform and Load stage separation makes composing simple components into complex workflows possible. Stages are reunnable to recover errors.
Embedded: Unlike Graph Protocol, Attestate’s Crawler does not have a separate server process. It writes directly to LMDB, an embedded database that can be read-from thread-safely at any time.
Batteries-included: Support for Ethereum, IPFS, Arweave, GraphQL and rate-limit pooling for Infura and Alchemy.

Attestate’s Crawler is created to build a web3 of peer to peer applications.

Comparison

Compare Attestate’s crawler with the Graph Protocol and Ponder using the table below.

Trait	Attestate Crawler	Graph Protocol	Ponder
Embeddable/Local Development	✓ (Run as your app)	✘ (Runs as a network)	✓ (Local dev server with hot reloading)
Output Formats	CSV, JSON, LMDB	GraphQL	GraphQL (Postgres?)
Web2 support	HTTP, JSON-RPC, GraphQL	Partial	N/A
Free	✓	✘ (tokens required)	✓
Complexity	A few JS files	Big code base	TypeScript/Node.js code base
Censorship-resistant	✓	Partially	N/A
Customizable	✓ (Compose & implement custom strategies)	✓ (Query many subgraphs)	✓ Same as graphql
License	GPL-3	Apache 2.0/MIT	MIT
Replicate data	✓	✓	✓
Derive data	✓	Multi-graph app	Multi-graph app
Dependencies	Node.js, LMDB (optional)	Docker, IPFS, PostgreSQL, Node.js	Node.js, PostgreSQL (optional)
Language/runtime	Node.js	AssemblyScript/WebAssembly	TypeScript/Node.js
Package ecosystem	✓ npm	✘ no npm	✓ npm
Fulltext search queries	✓ (use LMDB)	✓	(planned)

For a more in-depth discussion of uses cases, visit the use cases site.

Warning

We’re currently actively working on these documents. They’re still far from complete!

Concepts

Basics

Specifications