substreams-js/substreams-js

SubstreamsJS

TypeScript library for consuming Substreams suitable for web browsers, Node.js and alternative runtimes.

import { createGrpcTransport } from "@connectrpc/connect-node";
import {
  createAuthInterceptor,
  createRegistry,
  createRequest,
  fetchSubstream,
  isEmptyMessage,
  streamBlocks,
  unpackMapOutput,
} from "@substreams/core";

const SUBSTREAM = "https://github.com/streamingfast/substreams-uniswap-v3/releases/download/v0.2.7/substreams.spkg";
const MODULE = "map_pools_created";

const substream = await fetchSubstream(SUBSTREAM);
const registry = createRegistry(substream);
const transport = createGrpcTransport({
  baseUrl: "https://mainnet.eth.streamingfast.io",
  httpVersion: "2",
  interceptors: [createAuthInterceptor("<YOUR TOKEN>")],
  jsonOptions: {
    typeRegistry: registry,
  },
});

const request = createRequest({
  substreamPackage: substream,
  outputModule: MODULE,
  productionMode: true,
  stopBlockNum: "+10000",
});

for await (const response of streamBlocks(transport, request)) {
  const output = unpackMapOutput(response.response, registry);
  if (output !== undefined && !isEmptyMessage(output)) {
    console.dir(output.toJson({ typeRegistry: registry }));
  }
}

Installation

Note This library is pure ESM and no longer provides CommonJS (CJS) exports. If your project uses CommonJS, you will have to convert to ESM or use the dynamic import() function. Please don't open issues for questions regarding CommonJS / ESM.

# Using pnpm
pnpm add @substreams/core

# Using yarn
yarn add @substreams/core

# Using npm
npm install @substreams/core

Packages

• @substreams/core: Provides all core functionality for interacting with substreams packages and consuming substream modules. (source code).
• @substreams/manifest: Parses and validates substreams.yaml manifests and bundles them into substream packages. (source code).
• @substreams/mermaid: Generates mermaid graph visualizations of substream module graphs. (source code).
• @substreams/react: Lightweight utilities for building React based user interfaces on top of substreams. (source code).

Authors

• @fubhy (fubhy.eth, Twitter)

License

MIT License

substreams-js's Issues

Substreams Sink js not working on Nextjs

I tried using substreams sink js in my Next.js project. I wanted to create a service to fetch data from any substream dynamically. However, when I used this service in my component, an error occurred: - error ./node_modules/undici/lib/api/api-connect.js:3:24 Module not found: Can't resolve 'async_hooks'.

TypeError: Cannot convert module.initialBlock to a BigInt

https://github.com/substreams-js/substreams-js/blob/main/packages/core/src/manifest/signature/create-module-hash.ts#L79

  new DataView(array.buffer).setBigUint64(0, module.initialBlock, true);

For Node.js v18.18.0, this line may raise TypeError, such as `Cannot convert 0 to a BigInt, should be:

  new DataView(array.buffer).setBigUint64(0, BigInt(module.initialBlock), true);
\

Migrate `@bufbuild/connect` (and relatives) dependency to `@connectrpc`

It appears that @bufbuild has moved over to a different organization and that new work is happening there. We would need to upgrade to that new version somehow.

Message I get when trying to install newest version(s):

$ yarn install
yarn install v1.22.19
warning package.json: No license field
warning [email protected]: No license field
[1/4] 🔍  Resolving packages...
warning @bufbuild/[email protected]: Connect has moved to its own org @connectrpc and has a stable v1. Run `npx @connectrpc/connect-migrate@latest` to update. See https://github.com/connectrpc/connect-es/releases/tag/v0.13.1 for details.
warning @bufbuild/[email protected]: Connect has moved to its own org @connectrpc and has a stable v1. Run `npx @connectrpc/connect-migrate@latest` to update. See https://github.com/connectrpc/connect-es/releases/tag/v0.13.1 for details.
[2/4] 🚚  Fetching packages...
...

add `${number}` for `startBlockNum` type

Currently

startBlockNum?: number | bigint | undefined;
stopBlockNum?: number | bigint | `+${number}` | undefined;

Propose

• adding ${number} to support negative & positive string type

startBlockNum?: number | bigint | `${number}` | undefined;

Example Inputs

• "+100"
• "100"
• "-100"
• Number(100)
• Number(-100)
• BigInt(100)
• BigInt(-100)

Add `applyParams` testcase for (allow `=` character)

Allow = to be included as param value

split using .split(/=(.*)/s) Regex pattern

• "mymodule=A=B"
  • module => "mymodule"
  • value => "A=B"
• add Unit tests

https://github.com/substreams-js/substreams-js/blob/main/packages/core/src/manifest/params/apply-params.test.ts

> "mymodule=A=B".split("=",2)
[ 'mymodule', 'A' ]

> "mymodule=A=B".split("=")
[ 'mymodule', 'A', 'B' ]

> "mymodule=A=B".split(/=(.*)/s)
[ 'mymodule', 'A=B', '' ]

Package path not exported

On the latest version of substreams/manifest, when trying to

import {
  readPackage
} from '@substreams/manifest';

There's an error

node:internal/errors:496
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './Bigint' is not defined by "exports" in node_modules/effect/package.json imported from node_modules/@effect/schema/mjs/Schema.mjs
    at new NodeError (node:internal/errors:405:5)
    at exportsNotFound (node:internal/modules/esm/resolve:359:10)
    at packageExportsResolve (node:internal/modules/esm/resolve:695:9)
    at packageResolve (node:internal/modules/esm/resolve:870:14)
    at moduleResolve (node:internal/modules/esm/resolve:936:20)
    at defaultResolve (node:internal/modules/esm/resolve:1129:11)
    at nextResolve (node:internal/modules/esm/loader:163:28)
    at ESMLoader.resolve (node:internal/modules/esm/loader:835:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:424:18)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:77:40) {
  code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}

Node.js v18.17.1

custom eth rpc node

Hi team, when i use substream to fetch uniswap data, i always get this error.

This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
ConnectError: [unknown] terminated

  at Function.from (file:////@[email protected]_@[email protected]/node_modules/@bufbuild/connect/dist/esm/connect-error.js:72:20)
   at abort (file:///@[email protected]_@[email protected]/node_modules/@bufbuild/connect/dist/esm/protocol/run-call.js:103:36)

I think it happens because eth_call connect error.
So can I use my custom rpc node ?

`@substreams/manifest` Schema.string.pipe is not a function

• version: "@substreams/manifest": "^0.4.0"

import { readPackage } from "@substreams/manifest";

const manifest = "https://github.com/pinax-network/subtivity-substreams/releases/download/v0.2.3/subtivity-ethereum-v0.2.3.spkg";
const substreamPackage = await readPackage(manifest);

Include `block.clock.id` to `State`

Perhaps we can include include clock in State

Before

export type State = {
  current: bigint;
  cursor: string | undefined;
  timestamp: Date | undefined;
  modules: {
    [key: string]: ModuleState;
  };
};

Proposal A

export type State = {
  current: bigint; // Current Block Number
  id?: string; // Block ID
  cursor?: string;
  timestamp?: Date;
  modules: {
    [key: string]: ModuleState;
  };
};

Proposal B

export type State = {
  current: bigint;
  cursor?: string;
  clock?: Clock;
  modules: {
    [key: string]: ModuleState;
  };
};

manifest `readPackage` should also allow relative filepath

@substreams/manifest expects absolute path, it does not handle relative file path

The following Manifest URL & file path should be functional:

// local file
MANIFEST=./src/substreams-eth-block-meta-v0.5.1.spkg
MANIFEST=substreams-eth-block-meta-v0.5.1.spkg
MANIFEST=/usr/local/share/substreams-eth-block-meta-v0.5.1.spkg

// remote URL
MANIFEST=https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.5.1/substreams-eth-block-meta-v0.5.1.spkg

Example

import { readPackage } from "@substreams/manifest";

const manifest = "substreams-eth-block-meta-v0.5.1.spkg";
const substreamPackage = await readPackage(manifest);

❌ error

Caused by

if (!isReadableLocalFile(file)) {
    throw new Error("File is not readable");
}

Related

• pinax-network/substreams-sink-csv#1

Importance of protobuf files ordering in `substreams.yaml` manifest

When using this library I've noticed it cannot create registry properly when the order of protobuf.files in the manifest is not strict.

As for example, consider a definition that imports other definitions from neighbouring files. And if the imported ones (e.g. op_rollup.proto or nitro_rollup.proto) are stated after the one that does the import (e.g. combined_rollups.proto), then such an error will be raised:

invalid FieldDescriptorProto: type_name .<CUSTOM_PKG>.NitroRollup not found"

So in order to fix it I had to repack the .SPKG by changing the manifest file from this:

protobuf:
  files:
    - combined_rollups.proto
    - op_rollup.proto
    - nitro_rollup.proto
  importPaths:
    - ./proto

to this:

protobuf:
  files:
    - op_rollup.proto
    - nitro_rollup.proto
    - combined_rollups.proto
  importPaths:
    - ./proto

I've done a bit of debugging and found that this happens in create-descriptor-set.js:447 in newField() because of cart.mapEntries doesn't yet have the child definitions added.

The substreams package itself doesn't seem to break in such a case if used directly through substreams CLI. But this library does break because of that.

If this cannot be easily fixed I suggest at least adding some ⚠️ info in the README :)

Rather than that, thanks for a very useful library!