NEAR Lake Framework is a small library companion to NEAR Lake. It allows you to build your own indexer that subscribes to the stream of blocks from the NEAR Lake data source and create your own logic to process the NEAR Protocol data.
use futures::StreamExt;
use near_lake_framework::LakeConfig;
#[tokio::main]
async fn main() -> Result<(), tokio::io::Error> {
// create a NEAR Lake Framework config
let config = LakeConfig {
s3_bucket_name: "near-lake-testnet".to_string(), // AWS S3 bucket name
s3_region_name: "eu-central-1".to_string(), // AWS S3 bucket region
start_block_height: 82422587, // the latest block height we've got from explorer.near.org for testnet
};
// instantiate the NEAR Lake Framework Stream
let stream = near_lake_framework::streamer(config);
// read the stream events and pass them to a handler function with
// concurrency 1
let mut handlers = tokio_stream::wrappers::ReceiverStream::new(stream)
.map(|streamer_message| handle_streamer_message(streamer_message))
.buffer_unordered(1usize);
while let Some(_handle_message) = handlers.next().await {}
Ok(())
}
// The handler function to take the entire `StreamerMessage`
// and print the block height and number of shards
async fn handle_streamer_message(
streamer_message: near_lake_framework::near_indexer_primitives::StreamerMessage,
) {
eprintln!(
"{} / shards {}",
streamer_message.block.header.height,
streamer_message.shards.len()
);
}
In order to be able to get objects from the AWS S3 bucket you need to provide the AWS credentials.
AWS default profile configuration with aws configure looks similar to the following:
~/.aws/credentials
[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS docs: Configuration and credential file settings
Add the following dependencies to your Cargo.toml
...
[dependencies]
futures = "0.3.5"
itertools = "0.10.3"
tokio = { version = "1.1", features = ["sync", "time", "macros", "rt-multi-thread"] }
tokio-stream = { version = "0.1" }
# NEAR Lake Framework
near-lake-framework = { git = "https://github.com/near/near-lake-framework" }
Everything should be configured before the start of your indexer application via LakeConfig
struct.
Available parameters:
s3_bucket_name: String
- provide the AWS S3 bucket name (near-lake-testnet
,near-lake-mainnet
or yours if you run your own NEAR Lake)s3_region_name: String
- provide the region for AWS S3 bucketstart_block_height: u64
- block height to start the stream from
TL;DR approximately $18.15 per month (for AWS S3 access, paid directly to AWS) for the reading of fresh blocks
Explanation:
Assuming NEAR Protocol produces accurately 1 block per second (which is really not, the average block production time is 1.3s). A full day consists of 86400 seconds, that's the max number of blocks that can be produced.
According the Amazon S3 prices list
requests are charged for $0.005 per 1000 requests and get
is charged for $0.0004 per 1000 requests.
Calculations (assuming we are following the tip of the network all the time):
86400 blocks per day * 5 requests for each block / 1000 requests * $0.0004 per 1k requests = $0.173 * 30 days = $5.19
Note: 5 requests for each block means we have 4 shards (1 file for common block data and 4 separate files for each shard)
And a number of list
requests we need to perform for 30 days:
86400 blocks per day / 1000 requests * $0.005 per 1k list requests = $0.432 * 30 days = $12.96
$5.19 + $12.96 = $18.15
The price depends on the number of shards
We use Milestones with clearly defined acceptance criteria: