Configuration

Compared to other tools that may help you replicate Ethereum state locally, Attestate’s crawler comes with plenty of configuration options. This page’s purpose is to serve as a reference for all configurable options.

Environment Variables

@attestate/crawler internally uses dotenv to automatically load environment variables from a .env file in the project root into Node.js’s process.env object. However, if necessary, environment variables that have already been set in, e.g., the .env file can be overwritten by passing them before an invoking command.

The following environment variables are required for @attestate/crawler to run:

  • RPC_HTTP_HOST describes the host that Ethereum JSON-RPC extraction request are made against. It must be set to an URL to an Ethereum full node’s JSON-RPC endpoint that starts with https://. ws:// or wss:// prefixes are currently not supported. We support URLs that include the API’s bearer token as is the case with, e.g., Infura or Alchemy.

  • RPC_API_KEY is the API key for the host extraction requests are made against. It must be set if an Ethereum full node was provisioned behind an HTTP proxy that requires a bearer token authorization via the HTTP Authorization header. In this case, the header is structurally set as follows: Authorization: Bearer ${RPC_API_KEY}.

  • DATA_DIR is the directory that stores all results from extraction and transformation of the crawler. It must be set to a file system path (relative or absolute).

  • EXTRACTION_WORKER_CONCURRENCY is the number of simultaneous requests the extraction worker will make against RPC_HTTP_HOST. It must be an integer value and it cannot be smaller than 1.

  • IPFS_HTTPS_GATEWAY describes the host that IPFS extraction requests are made against. A list of publicly accessible IPFS gateways can be found here.

  • IPFS_HTTPS_GATEWAY_KEY is the API key for the IPFS host extraction requests are made against. It must be set if an IPFS node was provisioned behind an HTTP proxy that requires a bearer token authorization via the HTTP Authorization header. In this case, the header is structurally set as follows: Authorization: Bearer ${IPFS_HTTPS_GATEWAY_KEY}.

  • ARWEAVE_HTTPS_GATEWAY describes the host that Arweave extraction requests are made against. A commonly-used Arweave gateway is https://arweave.net.