hirosystems / ordhook Goto Github PK

Build indexers, standards and protocols on top of Ordinals and Inscriptions (BRC20, etc).

License: Apache License 2.0

Rust 98.15% Dockerfile 0.29% JavaScript 1.57%

ordhook's Introduction

   /     /   ▶ Ordhook
  / --- /      Ordinal indexing engine based on Chainhook.
 /     /       Build indexes, standards and protocols on top of Ordinals and Inscriptions (BRC20, etc).

Introduction

The Ordinal Theory is a protocol aiming at attributing unique identifiers to minted satoshis (sats). With its numbering scheme, satoshis can be inscribed with arbitrary content (aka inscriptions), creating bitcoin-native digital artifacts more commonly known as NFTs. Inscriptions do not require a sidechain or separate token, which makes it attractive for new entrants to adopt, extend, and use. These inscribed sats can be transferred using Bitcoin transactions, sent to Bitcoin addresses, and held in Bitcoin UTXOs. In all respects, these transactions, addresses, and UTXOs are normal Bitcoin transactions, addresses, and UTXOs, except that to send individual sats, transactions must control the order and value of inputs and outputs per Ordinal Theory.

Now that we discussed Ordinal Theory, let's dive into what ordhook attempts to solve for developers.

The ordhook is an indexer designed to help developers build new re-org-resistant applications on top of the Ordinal Theory. This indexer will make it easy for protocol developers and users of those protocols to trace and discover the ownership of Ordinal's inscriptions, along with a wealth of information about each inscription.

The ordhook uses Chainhook SDK from the Chainhook project, which is a re-org-aware transaction indexing engine for Stacks and Bitcoin. The SDK is designed with first-class event-driven principles, so it helps developers extract transactions from blocks efficiently and keeps a consistent view of the chain state.

With ordhook, Bitcoin developers can reliably implement feature-rich protocols and business models utilizing near-real-time Ordinals inscriptions and transfers events.

Quick Start

Installing `ordhook` from source

$ git clone https://github.com/hirosystems/ordhook.git
$ cd ordhook
$ cargo ordhook-install

Getting started with `ordhook`

Explore Ordinal activities in your terminal

Once ordhook is installed, Ordinals activities scanning can simply be performed using the following command:

$ ordhook scan blocks --interval 767430:767753 --mainnet
Inscription 6fb976ab49dcec017f1e201e84395983204ae1a7c2abf7ced0a85d692e442799i0 revealed at block #767430 (ordinal_number 1252201400444387, inscription_number 0)
Inscription 26482871f33f1051f450f2da9af275794c0b5f1c61ebf35e4467fb42c2813403i0 revealed at block #767753 (ordinal_number 727624168684699, inscription_number 1)

In this command, an interval of blocks to scan (starting at block 767430, ending at block 767753) is being provided. ordhook will display inscriptions and transfers activities occurring in the range of the specified blocks.

The activity for a given inscription can be retrieved using the following command:

$ ordhook scan inscription 6fb976ab49dcec017f1e201e84395983204ae1a7c2abf7ced0a85d692e442799i0 --mainnet
Inscription 6fb976ab49dcec017f1e201e84395983204ae1a7c2abf7ced0a85d692e442799i0 revealed at block #767430 (ordinal_number 1252201400444387, inscription_number 0)
Transferred in transaction bc4c30829a9564c0d58e6287195622b53ced54a25711d1b86be7cd3a70ef61ed at block 785396

Stream Ordinal activities to an indexer

ordhook is designed to help developers extract ordinals activities (inscriptions and transfers) from the Bitcoin chain and streaming these activities to their indexer / web application.

In order to get started, a bitcoind instance with access to the RPC methods getblockhash and getblock must be running. The RPC calls latency will directly impact the speed of the scans.

Note: the configuration of a bitcoind instance is out of scope for this guide.

Assuming:

1) a bitcoind node correctly configured and

2) a local HTTP server running on port 3000 exposing a POST /api/events endpoint,

A configuration file Ordhook.toml can be generated using the command:

$ ordhook config new --mainnet
✔ Generated config file Ordhook.toml

After adjusting the Ordhook.toml settings to make them match the bitcoind configuration, the following command can be ran:

$ ordhook scan blocks --interval 767430:767753 --post-to=http://localhost:3000/api/events --config-path=./Ordhook.toml

ordhook will retrieve the full Ordinals activities (including the inscriptions content) and send all these informations to the http://localhost:3000/api/events HTTP POST endpoint.

Run `ordhook` as a service for streaming blocks

ordhook can be ran as a service for streaming and processing new blocks appended to the Bitcoin blockchain.

$ ordhook service start --post-to=http://localhost:3000/api/events --config-path=./Ordhook.toml

New http-post endpoints can also be added dynamically by adding the following section in the Ordhook.toml configuration file:

[http_api]
http_port = 20456

Running ordhook with the command

$ ordhook service start --config-path=./Ordhook.toml

will spin up a HTTP API for managing events destinations.

A comprehensive OpenAPI specification explaining how to interact with this HTTP REST API can be found here.

Troubleshooting: Performance and System Requirements

The Ordinals Theory protocol is resource-intensive, demanding significant CPU, memory, and disk capabilities. As we continue to refine and optimize, keep in mind the following system requirements and recommendations to ensure optimal performance:

CPU: The ordhook tool efficiently utilizes multiple cores when detected at runtime, parallelizing tasks to boost performance.

Memory: A minimum of 16GB RAM is recommended.

Disk: To enhance I/O performance, SSD or NVMe storage is suggested.

OS Requirements: Ensure your system allows for a minimum of 4096 open file descriptors. Configuration may vary based on your operating system. On certain systems, this can be adjusted using the ulimit command or the launchctl limit command.

ordhook's People

Contributors

Stargazers

Watchers

Forkers

dmitrytorba agraebe ebkalderon coruscateor markmhendrickson criadoperez perceptforminc isparta-bitc trichards-hiro bridgit-spc defianalysis sabbyanandan hysoka15 analyticsninja 8bitsats superposition 0xspyop ordinariusprof dantegpt omahs rfk1118 haxjump 13552128609 4ker-dep script-meme l2ordinals ikigailabseth 0xtreasury ahmadou5 gopherj andrej656 jesse-21 richmix unilayerio globalboost cz-orangedx istomp looksrare cryptomummy openbitdev

ordhook's Issues

CI failing when merging branches

https://github.com/hirosystems/ordhook/actions/runs/6087613429/job/16516548524

The curious case of c30b84077464cd6870241f3a1c780b948b07b1257803a9b1cf5c342ce580543bi0 / 14957963

Log estimated time remaining when indexing

Logging for the indexing process currently only lists the latest block mined:

05/19/2023, 07:15:43 AM | Dequeuing block #790460 for processing (# blocks inboxed: 0) |  
05/19/2023, 07:21:21 AM | Dequeuing block #790461 for processing (# blocks inboxed: 0) |  
05/19/2023, 07:37:10 AM | Dequeuing block #790462 for processing (# blocks inboxed: 0) |  
05/19/2023, 08:08:15 AM | Dequeuing block #790463 for processing (# blocks inboxed: 0) |  
05/22/2023, 08:55:13 AM | Dequeuing block #790895 for processing (# blocks inboxed: 1) |  
05/22/2023, 08:55:35 AM | Dequeuing block #790896 for processing (# blocks inboxed: 0) |

We often field questions about how much longer this process will take. The ask here is to add an estimated time remaining to each of these log messages.

get inscription content error

I am receiving error not found on this request, but the ID is correct and does indeed exist

GET on this link: https://api.hiro.so/ordinals/v1/inscriptions/d88cfc96b40111d142dbdd8e855c3dc8428d2562567bb03b41f8d3ab0d98eedai0/content

Data does indeed exist: https://ordinals.com/inscription/d88cfc96b40111d142dbdd8e855c3dc8428d2562567bb03b41f8d3ab0d98eedai0

Different ordinals number from ordinals.com

Hi, I am running the ordhook server with version: v1.0.3 and v1.1.3, they all have the wrong inscription number of inscription id: f34df2c53c3411c452ebfdef6537fd14546630fca0c425d1a1641d7bf8211008i0, same problem of your server: https://ordinals.hiro.so/inscription/f34df2c53c3411c452ebfdef6537fd14546630fca0c425d1a1641d7bf8211008i0.

We all have the number: #46625971, but on ordinals.com(v0.9.0) the number is #46625972.

Thank you very much!

Ability to compute ordinal numbers from transactions in mempool

Draft PR #133

ordhook stuck in indexing inscriptions and never send any data to my endpoint

dear friends:

I am new to ordhook and trying ordhook to stream ordinal activities to my endpoint, I used a CHAINSTACK endpoint, and following is my Ordhook.toml:

[storage]
working_dir = "ordhook"

# [http_api]
# http_port = 20456

[network]
mode = "mainnet"
bitcoind_rpc_url = "http://IP:PORT"
bitcoind_rpc_username = "XXXXXXX"
bitcoind_rpc_password = "XXXXXX"

bitcoind_zmq_url = "tcp://IP:PORT"


[limits]
max_number_of_bitcoin_predicates = 100
max_number_of_concurrent_bitcoin_scans = 100
max_number_of_processing_threads = 16
bitcoin_concurrent_http_requests_max = 16
max_caching_memory_size_mb = 32000


[bootstrap]
download_url = "https://archive.hiro.so/mainnet/ordhook/mainnet-ordhook-sqlite-latest"

[logs]
ordinals_internals = true
chainhook_internals = true

because Ordhook can't use a https endpoint, therefore I used nginx as proxy, 47.128.200.237 is the ip of my nginx server. I used the following command to start the service:

ordhook service start --post-to=http://localhost:3000/api/events --config-path=./Ordhook.toml

and the service stuck in the following state and without sending any data to my endpoint:

I guest the problem lies in the field bitcoind_zmq_url = "tcp://IP:PORT", I saw from the doc that, this field is set to tcp://0.0.0.0:18543, which means the bitcoin node should locate at the same machine as ordhook, but I don't have a full bitcoin node in my local and I have to use a remote bitcoin service.

Do you know what's my problem, how can I fix that, I am highly appreciate your help, thank you very much.

Out of memory when downlaod "mainnet-ordhook-sqlite-latest.tar.gz"

$ ordhook scan blocks 767430 767753 --config-path=./Ordhook.toml
Nov 23 10:40:48.002 INFO Unable to retrieve hord.sqlite file locally
Nov 23 10:40:48.003 INFO Downloading https://archive.hiro.so/mainnet/ordhook/mainnet-ordhook-sqlite-latest.tar.gz
=> https://archive.hiro.so/mainnet/ordhook/mainnet-ordhook-sqlite-latest.tar.gz
[====================================================>.....] (19719829229/21581459625)killed

Introducing `inscriptions-sequence-consensus-hash`

Following up on ordinals/ord#2574.

Objective

The primary objective is to create a robust mechanism for ensuring data integrity and consistency across ordinal indexers. This involves introducing a consensus hash built using a Merkle root, derived from blockchain events. This enhancement aims to address the current limitations in the ordinals protocol, particularly in improving provability.

Problem

The current system is lacking of a consensus hash, limiting capabilities in detecting regressions and indexing divergences. To overcome these limitations, we propose the generation of a Merkle root. This root will be derived from specific blockchain events, namely inscription reveals and transfers. By employing cryptographic consensus hashes in the form of a Merkle root, ordinal indexers can enhance their upgrade process, establish verifiable checkpoints, and prevent the publication of inaccurate indexes.

Proposed Protocol

Event Types for Consideration

Inscription Revealed: This event occurs when an inscription's content becomes publicly known.
Inscription Transferred: This event is recorded when the ownership of an inscription changes hands.
Event Processing and Merkle Tree Construction
Events are chronologically sorted based on their on-chain inclusion timestamps.
These sorted events are then structured into a Merkle tree. This tree structure enables efficient and secure verification of the contents.
Generating the Consensus Hash
The Merkle proof, derived from the constructed tree, is combined with the consensus hash of the previous block, and the hash of the current Bitcoin block.
This combination generates the new consensus hash for the current block.
This hash serves as a reliable reference point for validating the integrity and consistency of ordinal indexers' data.

Inscription events

A satoshi being inscribed per the rules of a given sequence, would be augmenting the said sequence:

0        1        2                18               52              72                   74                  82
|--------|--------|-----------------|----------------|---------------|--------------------|-------------------|
 Sequence  Action   Ordinal Number    Inscription ID   Content hash    Output post-reveal  Offset post-reveal

Sequence : Sequence being inscribed - 0x00 if the sequence being indexed is the blessed sequence (which is the only stable sequence to date). We are versioning sequences because rules are changing overtime. Post-Jubilee, this byte will be bumped to 0x01.
Action: inscription = 0x01
Ordinal Number: Ordinal number being inscribed - uint128
Inscription ID: Id of the inscription. Composed of:
- Transaction ID - 32 bytes,
- Inscription index (intra-transaction) - uint16
  Example: 94e524cbdaf19c27491ac6fe22e7884e35f11ee3ba290ddf218efeaf5d5e9bddi0 = 0x94e524cbdaf19c27491ac6fe22e7884e35f11ee3ba290ddf218efeaf5d5e9bdd + 0x0000
Content hash: hash160 of the inscription enveloppe - 20 bytes
Output post-reveal: Outpoint pointing to the satoshi, post reveal - uint16
Offset post-reveal: Outpoint pointing to the satoshi, post reveal - uint64

Transfer events

A satoshi is being transferred:

0        1        2                  36                         80                         124
|--------|--------|-------------------|-------------------------|---------------------------|
 Sequence  Action    Inscription ID      Satpoint pre-transfer      Satpoint post-transfer

Sequence : Sequence being inscribed - 0x00 if the sequence being indexed is the blessed sequence (which is the only stable sequence to date). We are versioning sequences because rules are changing overtime. Post-Jubilee, this byte will be bumped to 0x01.
Action: transfer = 0x02
Ordinal Number: Ordinal number being inscribed - uint128
Inscription ID: Id of the inscription. Composed of:
- Transaction ID - 32 bytes,
- Inscription index (intra-transaction) - uint16
  Example: 94e524cbdaf19c27491ac6fe22e7884e35f11ee3ba290ddf218efeaf5d5e9bddi0 = 0x94e524cbdaf19c27491ac6fe22e7884e35f11ee3ba290ddf218efeaf5d5e9bdd + 0x0000
Satpoint pre-transfer: Satpoint pointing to the inscription id, pre transfer. Composed of:
- Transaction ID - 32 bytes,
- Output index - uint16
- Offset - uint64
Satpoint post-transfer: Satpoint pointing to the inscription id, post transfer. Composed of:
- Transaction ID - 32 bytes,
- Output index - uint16
- Offset - uint64

Outcome

Implementing this protocol will lead to:

Enhanced data provability and integrity within the ordinals ecosystem.
Reduced instances of regression and indexing errors.
Streamlined upgrade processes for ordinal indexers with reliable checkpoints for data validation.

`ordhook-sdk-js` experiment: from proof of concept to production-readiness

The branch feat/improve-sdk is including a solid draft of the upcoming ordinals-api re-architecture.
Instead of starting ordhook and registering ordinals-api as an http observer receiving payloads including ordinals inscriptions / transfers, the incoming version of ordinals-api will pull a NPM @hirosystems/ordhoook-sdk-js library and implement the following skeleton (present in components/ordhook-sdk-js/lib/test.ts:

import { OrdinalsIndexer } from "./index";

const indexer = new OrdinalsIndexer({
    bitcoinRpcUrl: 'http://0.0.0.0:8332',
    bitcoinRpcUsername: 'devnet',
    bitcoinRpcPassword: 'devnet',
    workingDirectory: '/Users/ludovic/ordhook-sdk-js',
    logs: false
});

indexer.applyBlock(block => {
    console.log(`Hello from JS ${JSON.stringify(block)}`);
});

indexer.undoBlock(block => {
    console.log(`Hello from JS ${JSON.stringify(block)}`);
});

indexer.streamBlocks();

The OrdinalsIndexer struct can be fully piloted in Typescript. In case of migrations, the following actions can also be performed:

// Delete blocks and inscriptions
indexer.dropBlocks([32103, 32104]);

// Re-download + index blocks
indexer.rewriteBlocks([32103, 32104]);

// Sync blocks until chain tip
indexer.syncBlocks();

// Replay a list of blocks
indexer.replayBlocks([32103, 32104]);

The ordhook-sdk-js is producing a state (rocksdb+sqlite) identical to the one being produced by ordhook-cli. So a state produced by ordhook-cli can be reused by an app being built with ordhook-sdk-js and vice-versa.

Remaining work:

Wire dropBlocks method
Wire rewriteBlocks method
Wire syncBlocks method
Wire replayBlocks method
Manually publish ordhook-sdk-js on NPM for end to end testing
Automatically publish ordhook-sdk-js via CI
Prepare docker image

`ordhook service start` can silently fail when user's machine "max per-process number of file descriptors" set by `ulimit` is too low

Summary: User must manually set ulimit -Sn 10000 so that ordhook process can handle work required when creating tables for ordinals inscription indexes, however both 1) the error messages suggesting a problem are hidden by default and 2) the error messages themselves don't contain or suggest the fix (increasing ulimit -Sn 10000).

Description of problem: With ordhook service start running, this screen would persist (indefinitely?) with no indication that certain indexing actions aren't working:

These would be the error logs that suggest an error:

Solution is to have user manually increase ulimit -Sn 10000 — maybe create a special class of error message indicating fix required on part of user?

Waiting for ZMQ messages from bitcoind

Dec 28 15:42:42.863 INFO Pipeline successfully processed sequence of blocks (823282 to 823284)
Dec 28 15:42:42.938 INFO Database up to date, service will start streaming blocks
Dec 28 15:42:43.180 INFO Observing Bitcoin chain events via ZeroMQ: tcp://0.0.0.0:18543
Dec 28 15:42:43.180 INFO Listening on port 20456 for chainhook predicate registrations
Dec 28 15:42:43.207 INFO Waiting for ZMQ connection acknowledgment from bitcoind
Dec 28 15:42:43.207 INFO Waiting for ZMQ messages from bitcoind

Update getting-started steps

After a series of trial and error, I still cannot get hord up and running locally.

It appears SQLite, Redis and RosckDB .. either all are optional or everything is required?? At least from the errors, SQLite appears to be a requirement.

Errors:

╭─sabbyanandan ~/hiro/hord ‹develop●›
╰─$ hord scan inscription 6fb976ab49dcec017f1e201e84395983204ae1a7c2abf7ced0a85d692e442799i0 --mainnet
thread 'main' panicked at 'FATAL: could not find /Users/sabbyanandan/hiro/hord/cache/hord.sqlite', components/hord-cli/src/db/mod.rs:171:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

╭─sabbyanandan ~/hiro/hord ‹develop●›
╰─$ hord scan blocks 100 111 --post-to=http://localhost:3000/api/events --config-path=./Hord.toml
Jul 19 20:24:14.705 INFO Streaming blocks from bitcoind http://localhost:18443/
thread 'main' panicked at 'FATAL: could not find cache/hord.sqlite', components/hord-cli/src/db/mod.rs:171:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

And with Redis running locally at 6379, I uncommented database_uri = "redis://localhost:6379/" in Hord.toml, and when running the DB check, I ended up with a RocksDB error.

╭─sabbyanandan ~/hiro/hord ‹develop●›
╰─$ hord db check --config-path ./Hord.toml
Jul 19 20:31:21.845 ERRO unable to open blocks_db: IO error: No such file or directory: While opening a file for sequentially reading: cache/hord.rocksdb/CURRENT: No such file or directory

Overall, I am entirely confused. I can't understand what is required to get hord up and running.

Rename `hord` to `ordhook`

codebase
k8s deployments

Speed-up ordinals indexing

having the bitcoin node and the chainhook on the same machine
we should have a flag dev to read the blockchain data from the disk instead of rpc for fast syncing
once's synced we have choice to read from rpc

db sync unable to retrieve Bitcoin chain tip

Running ordhook db sync --config-path=./Ordhook.toml to sync the db I get this error

ERRO unable to retrieve Bitcoin chain tip (JSON-RPC error: transport error: unexpected HTTP code: 400)

I previously posted this issue #232 where the sync would get stuck after saving a few thousand blocks to disk. This seemed to be fixed by upgrading the bitcoin core to version 25.0. But now after ordhook saving blocks up to ~820000 to disk this error is appearing. Any idea what might be causing this?

btcoind generate block, hord server do not start scan the latest block

Hi,

After hord service start --config-path=./Hord.toml scan the lastest block. and get info:

then I make btcoind generate new block(generate 10 blocks), but hord not start scan the lastest block.

Is this a BUG or need to use other methods to let hord start scan lastest block(scan the new 10 blocks).

Which configuration should I use to avoid exceeding the memory？

I run the Ordhook service in the machine (8C32G), but it exceeds the max.

My Ordhook.toml configuration is as follows:

[limits]
max_number_of_bitcoin_predicates = 100
max_number_of_concurrent_bitcoin_scans = 100
max_number_of_processing_threads = 16
bitcoin_concurrent_http_requests_max = 16
max_caching_memory_size_mb = **28000**

Inscription number not matched with ordinals.com on v2.0.0

I started a new ordhook v2.0.0 server, reloaded all the data from the beginning. But I got the mis-matched inscription number: 6d9f78ebf5ad60df99a76fcc21bbbfb09c83c82c80c84140b0e52effee2dd0c4i0, #53666689 from ordhook but from #53681848 ordinals.com. I think many inscriptions are mis-matched.

logs:
Jan 08 00:42:35.562 INFO Inscription 6d9f78ebf5ad60df99a76fcc21bbbfb09c83c82c80c84140b0e52effee2dd0c4i0 (#53666689) detected on Satoshi 1877042280052473 (block #824802, 3879 transfers)

Thank you very much!

Stuck on warning "no such table: observers"

When running

ordhook scan blocks --interval [first_block]:[last_block] --post-to=http://localhost:3000 --config-path=./Ordhook.toml

The first block appears to be processed correctly, but then I see

WARN unable to query hord.sqlite: no such table: observers

output over and over and no progress seems to be made.

Any idea what causes this?

Add from location on inscription transfer events

"inscription_transferred" event only have "destination" field, for example:

{
                "inscription_transferred": {
                  "destination": {
                    "type": "transferred",
                    "value": "bc1qctx9fzhzf4253ka7jd2s0sf5fqvzffnfvpk5wn"
                  },
                  "inscription_id": "26482871f33f1051f450f2da9af275794c0b5f1c61ebf35e4467fb42c2813403i0",
                  "post_transfer_output_value": 4562,
                  "satpoint_post_transfer": "a03017b45b99a61a69c8f4aa9e649ef476147396f59f0f75c03a156c3cc83aa5:0:0",
                  "satpoint_pre_transfer": "26482871f33f1051f450f2da9af275794c0b5f1c61ebf35e4467fb42c2813403:0:0",
                  "tx_index": 2296
                }
              }

It would be better to include both the "from" address and the "destination" address in the event, so that the event listener does not have to query them from the database history.

Polish CLI commands

Some commands / subcommands could be outdated / redundant / broken.
Some attention there would be great.

Waiting for ZMQ connection acknowledgment from bitcoind , been waiting.

Jan 14 02:27:48.936 INFO Block #2572752 processed, revealed 1 inscriptions [860612] and 1 transfers
Jan 14 02:27:48.953 INFO Pipeline successfully processed sequence of blocks (2572698 to 2572752)
Jan 14 02:27:48.981 INFO Database up to date, service will start streaming blocks
Jan 14 02:27:49.367 INFO Observing Bitcoin chain events via ZeroMQ: tcp://127.0.0.1:28332
Jan 14 02:27:49.384 INFO Waiting for ZMQ connection acknowledgment from bitcoind
Jan 14 02:27:49.384 INFO Waiting for ZMQ messages from bitcoind

I've been waiting here for a new block of data, and I don't see a push.

my bitcoin.conf:
testnet=1
dbcache=10240
txindex=1
daemon=1
server=1
rest=1
whitebind=127.0.0.1:8333
rpcallowip=0.0.0.0/0
rpcuser=user
rpcpassword=password999
zmqpubrawtx=tcp://:28332
zmqpubrawblock=tcp://:28332

Add an enum to specify transfer type

Inscription transfers should have an enum value that specifies the kind of transfer it represents: regular transfer, sent as fee to miner, sent to unspendable output, etc.

This will help the API discern how it should treat this transfer in BRC-20.

Cursed recursive inscription has different number on ord.io compared to ordinals.hiro.so

Describe the bug
The short inscription number of this inscription is different between ord.io and ordinals.hiro.so

https://ordinals.hiro.so/inscription/81433a25e784366a7db917af98393be6eba232a03646e1fc072904fc3b8803b2i2

https://www.ord.io/81433a25e784366a7db917af98393be6eba232a03646e1fc072904fc3b8803b2i2

I expect the short numbers to be the same between the explorers.

Extend auto-download of missing `cache/hord.sqlite` when using hord as a service

description

the single binary hord scan works as expected by downloading the cache/hord.sqlite if it does not exist, but when trying to spin up hord as a service and stream ordinal activity for the first time (if the file does not exist yet) we get an error

"FATAL: could not find cache/hord.sqlite"

the current workaround for anyone trying to stream ordinal activity is to first run the default binary, something like hord scan blocks 767430 767753 --mainnet which will download the cache/hord.sqlite if it does not exist yet.

expected behavior

run the hord service start commands and if cache/hord.sqlite does not exist auto-download - if it does exist, use it.

actual results

error: "FATAL: could not find cache/hord.sqlite"

How the ordinals-api should configure link hooks

I use the relevant information below to start the chain hook only to see the structure, but not to start port monitoring.
If I want to use ordinals-api, chainhook doesn't have listening port, I can't configure ordinals-api env.ts correctly

chainhook service start --predicate-path=./ordinals.json --config-path=./config.toml

[storage]
driver = "memory"
redis_uri = ""
cache_path = "./data2"

[chainhooks]
max_stacks_registrations = 500
max_bitcoin_registrations = 500

[network]
mode = "mainnet"
bitcoind_rpc_url = ""
bitcoind_rpc_username = ""
bitcoind_rpc_password = ""
stacks_node_rpc_url = "https://stacks-node-api.mainnet.stacks.co/"

{
"chain": "bitcoin",
"uuid": "818725a4-555d-46db-a60c-213a506b37ba",
"name": "inscription_feed",
"version": 1,
"networks": {
"mainnet": {
"start_block": 767430,
"end_block": 793083,
"if_this": {
"scope": "ordinals_protocol",
"operation": "inscription_feed"
},
"then_that": {
"file_append": {
"path": "ordinals.txt"
}
}
}
}
}

Support Bitcoin Testnet

I'm building on a test network and when running ordhook service start --post-to=http://localhost:3000/api/events --config-path=./Ordhook.toml it says Nov 14 10:29:24.226 ERRO Error standardizing block: error retrieving prevout for transaction 4a2750823d8b3111f7ceb0ab258abcdf10d481e78c7f5c3052888dc269ae4616, input #0 (block #2538412)

Off by one when resuming indexing

When hord restarts, it retrieves the last block N where we left, and start indexing a N-1, instead of N+1 - perfectly fine, but unnecessary work.

Post from block #775203 when starting service with post-to parameter

I started ordhook server V2.0.1 like this: ordhook service start --post-to=http://post_to_server/ --config-path=./Ordhook.toml. In V1.3 it will post the latest processed block to the post-to server, but in V2.0.1 it posted from block #775203. I can not figure out if it is a bug or new feature, please check it. And I want to know how to post the latest processed block to the server but not from block #775203.
Thank you very much!

logs:
Jan 16 07:13:28.919 INFO Block received from pre-processor
Jan 16 07:13:28.953 INFO Evaluating 0 bitcoin chainhooks registered
Jan 16 07:13:28.953 INFO 0 bitcoin chainhooks positive evaluations
Jan 16 07:13:28.953 INFO 0 bitcoin chainhooks will be triggered
Jan 16 07:13:28.953 INFO 0 bitcoin chainhooks to deregister
Jan 16 07:13:29.602 INFO Processing block #775203 through http://post_to_server/ predicate revealed 21 new inscriptions [5012, 5013, 5014, 5015, 5016, 5017, 5018, 5019, 5020, 5021, 5022, 5023, 5024, 5025, 5026, 5027, 5028, 5029, 5030, 5031, 5032] and 0 transfers
Jan 16 07:13:29.624 INFO Trigger http://post_to_server/ successful
Jan 16 07:13:30.457 INFO Processing block #775204 through http://post_to_server/ predicate revealed 5 new inscriptions [5033, 5034, 5035, 5036, 5037] and 0 transfers
Jan 16 07:13:30.490 INFO Trigger http://post_to_server/ successful
Jan 16 07:13:31.442 INFO Processing block #775205 through http://post_to_server/ predicate revealed 0 new inscriptions [] and 2 transfers
Jan 16 07:13:31.459 INFO Trigger http://post_to_server/ successful
Jan 16 07:13:32.228 INFO Processing block #775206 through http://post_to_server/ predicate revealed 8 new inscriptions [5038, 5039, 5040, 5041, 5042, 5043, 5044, 5045] and 2 transfers
Jan 16 07:13:32.249 INFO Trigger http://post_to_server/ successful
Jan 16 07:13:33.007 INFO Processing block #775207 through http://post_to_server/ predicate revealed 11 new inscriptions [5046, 5047, 5048, 5049, 5050, 5051, 5052, 5053, 5054, 5055, 5056] and 0 transfers
Jan 16 07:13:33.033 INFO Trigger http://post_to_server/ successful
Jan 16 07:13:33.924 INFO Processing block #775208 through http://post_to_server/ predicate revealed 7 new inscriptions [5057, 5058, 5059, 5060, 5061, 5062, 5063] and 1 transfers
Jan 16 07:13:33.940 INFO Trigger http://post_to_server/ successful

Cursed Inscription IDs

I just inscribed some parent/child cursed inscriptions. The ID in the API is ending with i1 - this makes sense as the child was indeed inscribed on index 1, but in every explorer this does not effect the ID, it should always end in i0 as far as I am aware. Bug?

See here: https://api.hiro.so/ordinals/v1/inscriptions?address=bc1ptsyzdercvzky7z2rxy8rv8njn6mpy5s72r2kgnhhuexk637c22eqz7qga0&limit=60

thread 'Block data compression' panicked at 'called `Option::unwrap()

I've installed ordhook on my local computer and i'm trying to run this as a service. On the beginning everything was looking fine, but further i was i've encountered more errors. Currently im +- 190k block and my console looks like on screen.

My config:

[storage]
working_dir = "ordhook"

# The Http Api allows you to register / deregister
# dynamically predicates.
# Disable by default.
#
# [http_api]
# http_port = 20456
# database_uri = "redis://localhost:6379/"

[network]
mode = "mainnet"
bitcoind_rpc_url = "xxx"
bitcoind_rpc_username = "xxx"
bitcoind_rpc_password = "xxx"
# Bitcoin block events can be received by Chainhook
# either through a Bitcoin node's ZeroMQ interface,
# or through the Stacks node. Zmq is being
# used by default:
bitcoind_zmq_url = "tcp://xxx:29000"
# but stacks can also be used:
# stacks_node_rpc_url = "http://0.0.0.0:20443"

[limits]
max_number_of_bitcoin_predicates = 100
max_number_of_concurrent_bitcoin_scans = 100
max_number_of_processing_threads = 2000
bitcoin_concurrent_http_requests_max = 64
max_caching_memory_size_mb = 8000

Disable the following section if the state

must be built locally

[bootstrap]
download_url = "https://archive.hiro.so/mainnet/ordhook/mainnet-ordhook-sqlite-latest"

[logs]
ordinals_internals = true
chainhook_internals = true

Tried this on 2 machines and both have similar issues - did i missed something?

cursed inscription numbers do not match ordinals.com

when I view the Dimension ordinals, I see two things that are off. The cursed numbers do not match Ordinals.com. For example, if I look at this one, it is -54,314. But in the wallet, it's in the -73,000s range. Also, the inscription ID is incorrect. https://ordinals.com/inscription/5c4cd4e1409ad051ff7768f54d37bee2fd5d27f140fff4ffe458c732214a8d3bi0

The inscription ID ends with i0 not i1

Run consistency check (+ recovery) when starting service

unable to open hord.rocksdb: IO error....No locks available

Hi Team, I'm seeing this error:

ERRO unable to open hord.rocksdb: IO error: lock hold by current process, acquire time 1700832341 acquiring thread 1826032: /usr/local/etc/ordhook/data/hord.rocksdb/LOCK: No locks available

Any clues or any additional info I can provide for debugging?

default target network fetched from Ordhook.toml file

Several times, during the ordhook usage, is need to provide the target network as an argument. This makes the input a bit verbose.

Proposal: Use the content of the Ordhook.toml file to set the target network (if not available).

Implement floating height logic for initial indexing

When hord starts, the first step of its boot sequence is to fetch the chain tip and index the missing blocks. The chain tip should be updated on a regular basis to be sure that we're fully caught up before moving on the next step.

What‘s wrong with the command `ordhook scan blocks 767430 767753 --mainnet`

I waited for this command for some time, with no result.
And I run wget https://archive.hiro.so/mainnet/chainhooks/hord.sqlite.gz, which shows that 404

Anyone help?

spin up ordhook as a servcie , occur error ''database disk image is malformed"

when spinning up ordhook as a service ,use below command:
ordhook service start --config-path=./Ordhook.toml

no such column: inscription_number

Describe the bug
When I run ordhook scan blocks --interval 767430:767753 --mainnet
The warning messages keep appearing and cannot download the latest block

Transaction ID
Address
Block# 780851
Time stamp

To Reproduce
Steps to reproduce the behavior:

run ordhook db sync --config-path=./Ordhook.toml
meesages:
➜ ordhook git:(develop) ✗ ordhook db sync --config-path=./Ordhook.toml Jan 06 01:21:51.429 WARN unable to query hord.sqlite: no such column: inscription_number Jan 06 01:21:51.468 INFO Indexing inscriptions from block #780851 to block #824545 Jan 06 01:21:54.547 INFO Processing 3 blocks Jan 06 01:21:54.547 INFO Inscriptions data computation for block #780851 started Jan 06 01:21:54.547 WARN unable to prepare query hord.sqlite: no such column: inscription_number Jan 06 01:21:55.547 WARN unable to prepare query hord.sqlite: no such column: inscription_number

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots

Desktop (please complete the following information):

OS: linux
Browser [e.g. chrome, safari]
Version [e.g. 22]

Thanks for helping.

thread 'Inscription indexing runloop' panicked at 'index out of bounds'

Hi! I'm running ordhook v1.2.0 in a k8s cluster on AWS. I've successfully passed the on-chain data scraping the from a Bitcoin daemon but currently I've getting the following error:

{"msg":"Unable to find block #337489, will retry in 2.479s","level":"WARN","ts":"2023-12-15T16:30:03.450302565Z"}
{"msg":"Unable to find block #337489, will retry in 6.8s","level":"WARN","ts":"2023-12-15T16:30:05.929453541Z"}
{"msg":"Unable to find block #337489, will retry in 15.831s","level":"WARN","ts":"2023-12-15T16:30:12.729600261Z"}
{"msg":"Unable to find block #337489, will retry in 2.372s","level":"WARN","ts":"2023-12-15T16:30:28.577820567Z"}
{"msg":"Unable to find block #337489, will retry in 6.392s","level":"WARN","ts":"2023-12-15T16:30:30.949958675Z"}
{"msg":"Unable to find block #337489, will retry in 16.042s","level":"WARN","ts":"2023-12-15T16:30:37.342022251Z"}
{"msg":"Unable to find block #337489, will retry in 2.653s","level":"WARN","ts":"2023-12-15T16:30:53.391773419Z"}
{"msg":"Unable to find block #337489, will retry in 5.461s","level":"WARN","ts":"2023-12-15T16:30:56.044894169Z"}
{"msg":"Unable to find block #337489, will retry in 13.126s","level":"WARN","ts":"2023-12-15T16:31:01.506024575Z"}
{"msg":"Unable to find block #337489, will retry in 2.14s","level":"WARN","ts":"2023-12-15T16:31:14.639585865Z"}
{"msg":"Unable to find block #337489, will retry in 6.236s","level":"WARN","ts":"2023-12-15T16:31:16.779700143Z"}
{"msg":"Unable to find block #337489, will retry in 17.141s","level":"WARN","ts":"2023-12-15T16:31:23.015829111Z"}
{"msg":"Unable to compute inscription's Satoshi: block #337489 not in database","level":"ERRO","ts":"2023-12-15T16:31:40.15924017Z"}
{"msg":"Inscriptions data computation for block #775087 collected","level":"INFO","ts":"2023-12-15T16:31:40.159593944Z"}
{"msg":"Inscriptions data computation for block #775087 ended","level":"INFO","ts":"2023-12-15T16:31:40.159627221Z"}
{"msg":"Cleanup: threadpool deallocation started","level":"INFO","ts":"2023-12-15T16:31:40.159787852Z"}
{"msg":"Cleanup: threadpool deallocation ended","level":"INFO","ts":"2023-12-15T16:31:40.161352385Z"}
{"msg":"Inscription 41a183751841ef3b86b617a2fe3acaef41e6c7cc15122540b04df5ebc0b64a04i0 (#3079) detected on Satoshi 1624155815836413 (block #775087, 1204 transfers)","level":"INFO","ts":"2023-12-15T16:31:40.171143788Z"}
{"msg":"Inscription 99d0776e89ab3cf130b60f9b81bea85d14e89449225d0e6027f8ad2e6810a505i0 (#3080) detected on Satoshi 1485271863560085 (block #775087, 632 transfers)","level":"INFO","ts":"2023-12-15T16:31:40.171240211Z"}
{"msg":"Inscription 9925d3156f0dc99bbb4847351821d2a26ae4f6a4c4377972403803b4df8f4008i0 (#3081) detected on Satoshi 720397146009797 (block #775087, 692 transfers)","level":"INFO","ts":"2023-12-15T16:31:40.171295167Z"}
{"msg":"Inscription 61bf61ae9813dbcc6d7e8fbf7cd11c53bebf9a54f211a59ac759bee564145308i0 (#3082) detected on Satoshi 1758519431978480 (block #775087, 786 transfers)","level":"INFO","ts":"2023-12-15T16:31:40.17141037Z"}
...
{"msg":"Inscription 09158c40974878b4ef4e9e53bc582fa29ae609bb0eb504b6e2145abc93e52fdei0 (#3287) detected on Satoshi 1908985664123741 (block #775087, 417 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.534917044Z"}
{"msg":"Inscription 860d096b1c115d7e24786aa1a827261ec7f16a37ee3589ec41debfbc72cedadei0 (#3288) detected on Satoshi 774306456787885 (block #775087, 872 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.534953534Z"}
{"msg":"Inscription 1cd2563d3a92de6ced6293d02411eeb8603bf520771bde6397cff7a92b9c2cdfi0 (#3289) detected on Satoshi 1342458242394567 (block #775087, 1315 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.53499789Z"}
{"msg":"Inscription 888ee3c648b81febfb663dbabae365932b99b6e3e6e0a52a8f37537ee3d28edfi0 (#3290) detected on Satoshi 1458728730856920 (block #775087, 3502 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535028494Z"}
{"msg":"Inscription 038945c67d5322db281bed0c6163d713b471f290effb16313fc38c02caa58de0i0 (#3291) detected on Satoshi 1902016825684115 (block #775087, 883 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535059486Z"}
{"msg":"Inscription c252d95f05550a77f913c342fa1372268e89e0334b1a8c969be3b590c0b6cae1i0 (#3292) detected on Satoshi 377151882205556 (block #775087, 2644 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535088073Z"}
{"msg":"Inscription 769f9a872d21cda73934f5a157b1959973126e02fe4175417068d8294e8155e5i0 (#3293) detected on Satoshi 1290287356375636 (block #775087, 851 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535118421Z"}
{"msg":"Inscription c73737b0c94d378883e87c56c38ae36568313fd0b9b87baa83dbe03e57a8a8e6i0 (#3294) detected on Satoshi 436282714939553 (block #775087, 1575 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535147701Z"}
{"msg":"Inscription 0420e80c85733c93059a280c5949f6e5a154a1cf91baed4076891544309706e9i0 (#3295) detected on Satoshi 1758519432205653 (block #775087, 775 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535178751Z"}
{"msg":"Inscription 3bec6ea7fb8d24d2ca8eedcd1a2bd4c0f20cdfb2f232fcfe1e18500f226993ebi0 (#3296) detected on Satoshi 377151881580556 (block #775087, 2599 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535209019Z"}
{"msg":"Inscription a1606f8d8b4cabf64fd462e1ba1a52ea3844ae28b78a1224c4ceb252f5eda7ebi0 (#3297) detected on Satoshi 1074548074961308 (block #775087, 603 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.53524046Z"}
{"msg":"Inscription 2570fc49be547161d9263bd9aeaa0ee043aa6c315f8b7998e37ab298b94af9edi0 (#3298) detected on Satoshi 1127826600881918 (block #775087, 900 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535269703Z"}
{"msg":"Inscription f7cd9c8a9b6bab3ccb072f68f3f821f95b0664377ad9b2238feb51ebcdcecfefi0 (#3299) detected on Satoshi 1724192072391484 (block #775087, 1452 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535299665Z"}
{"msg":"Inscription 850fadd14256b3a8655cc0791dab8320e03b27e5ba153471676ae7af649a38f0i0 (#3300) detected on Satoshi 1918870789904749 (block #775087, 451 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535329958Z"}
{"msg":"Inscription 8ada9bdfa060ce57fa76ab4530e1ee3d69976dc6530e982273bf010748bd5af2i0 (#3301) detected on Satoshi 1678009589072077 (block #775087, 157 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535360105Z"}
{"msg":"Inscription 41ce4e544e33c13607cb1c3903c61baf7f39b439c34992bbeb3a7e8713a789f2i0 (#3302) detected on Satoshi 720397145945568 (block #775087, 695 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535391211Z"}
{"msg":"Inscription 1dafd434a84ccbfc0f2ff62e3137e1d89a2dc8cc1b4a2834a57bfdb86e491bf3i0 (#3303) detected on Satoshi 436282715139553 (block #775087, 1587 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535419691Z"}
{"msg":"Inscription f280cc01ecd95c455fb8905d4625d28923cef61c422aefd56ce44d419fb863f6i0 (#3304) detected on Satoshi 1571153899235607 (block #775087, 784 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535450411Z"}
{"msg":"Inscription 215b8f12569f13249becc70849001845d5549ae43f788df50fa92b6e3022d5f8i0 (#3305) detected on Satoshi 845872662600981 (block #775087, 518 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.53548257Z"}
{"msg":"Inscription 7fa39918362ead5a7da765b8bb9ec8c84d3cef0e4800038fc9be180e5f11eaf9i0 (#3306) detected on Satoshi 1737591295501891 (block #775087, 99 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535512785Z"}
{"msg":"Inscription 6fff04f9b9dcb9f5f052e8fad7f941aa9c9f4c4f210edaaf3e06849a050d06fai0 (#3307) detected on Satoshi 1571153899894299 (block #775087, 740 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535543052Z"}
{"msg":"Inscription b7d76c64887b1ef6c801265d11033585373d087831f1a37ff93f2665b09f62fbi0 (#3308) detected on Satoshi 1410541458678424 (block #775087, 1358 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535572105Z"}
{"msg":"Inscription 9030380e3075060a4b539dcdd79bb7ca5eeb771385235ebadccb36b2d7d5e4fci0 (#3309) detected on Satoshi 1562949746879201 (block #775087, 2333 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535606273Z"}
{"msg":"Inscription 7b1bad4deac151db09cd73aa47c9025e0b4691603fd7055afebec236b41a53fdi0 (#3310) detected on Satoshi 1902016849324855 (block #775087, 831 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535635616Z"}
{"msg":"Inscription 7ba07e9712fdec9967aafadec6a49c266898a80ed0d9b1158a150de9c657f6fdi0 (#3311) detected on Satoshi 1290287357325636 (block #775087, 923 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.53566526Z"}
{"msg":"Inscription c47e9837ba0a4eb5701dc934e8dba14d963df8bebe4f47c4b7e4b6f762773bffi0 (#3312) detected on Satoshi 1127826600781918 (block #775087, 890 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.53569359Z"}
{"msg":"Inscription 892648b9ea9c58663a906b6845dc8b42cf289eaf8e050d2dc497c0db670b741ci0 (#3313) detected on Satoshi 1165465341497207 (block #775087, 78 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535723229Z"}
{"msg":"Inscription 62615d463afd164447dcae0ead77c4b7ad2d363498f366661ddaeef7f065c02bi0 (#3314) detected on Satoshi 1446379692105363 (block #775087, 1710 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535754923Z"}
{"msg":"Inscription 2f52ce0d53bfbcf703ba8540265a521db85c6bf7cf7db1345dc5c3eb23295a7bi0 (#3315) detected on Satoshi 1888308309994047 (block #775087, 173 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535786265Z"}
{"msg":"Inscription 6ea505f6077d1199a70b4dc9ae1c02ffd90689f7de38b4027ba00a6a380e8662i0 (#3316) detected on Satoshi 1883996452637432 (block #775087, 399 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535817264Z"}
{"msg":"Inscription 06a59cdc38960aad232d548b5ff873c821fc43e2e37b015685c72c18111bc59ei0 (#3317) detected on Satoshi 1217429722110751 (block #775087, 1213 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535846184Z"}
{"msg":"Inscription 832442d5b7647abf1f90e1492432bd35c6a34b414cbef8505136d685b51cad5bi0 (#3318) detected on Satoshi 1197987616032863 (block #775087, 226 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.535906476Z"}
{"msg":"Inscription a473947ee7e7e061f4c40210bfa9458859449f78b81ac6b94c3eb1300ab5a9a9i0 (#3319) detected on Satoshi 1197987615912863 (block #775087, 226 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.536785374Z"}
{"msg":"Inscription 832bba509f1bcfa3f9a22522caeea8d1d7722ba1607e6fbfa4458f79c96ba043i0 (#3320) detected on Satoshi 1197987615952863 (block #775087, 226 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.536820345Z"}
{"msg":"Inscription 2bf79c79e318cebd20738cfd4142b6308a814424395c07168294b05e3be148eei0 (#3321) detected on Satoshi 1197987615992863 (block #775087, 226 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.536847519Z"}
{"msg":"Inscription 847290d2d07bfc750c0eba8cf34372f945bec0f4d961eb90a081d845a7c28280i0 (#3322) detected on Satoshi 1197987616072863 (block #775087, 226 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.536874406Z"}
{"msg":"Inscription a93d6a3f2eb393f3f4065be754457927533e1916164a0e6963c699f6a2e0088bi0 (#3323) detected on Satoshi 1197987616112863 (block #775087, 226 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.536905265Z"}
{"msg":"Inscription da6f9ab97084de4c63fd8aaa336f1c970650cb91659f1edecd95ad957ea87282i0 (#3324) detected on Satoshi 1197987614313878 (block #775087, 238 transfers)","level":"INFO","ts":"2023-12-15T15:15:38.536936347Z"}
thread 'Inscription indexing runloop' panicked at 'index out of bounds: the len is 1 but the index is 1', /src/components/ordhook-core/src/db/mod.rs:1284:24
stack backtrace:
   0:     0x5568981fba21 - std::backtrace_rs::backtrace::libunwind::trace::he648b5c8dd376705
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x5568981fba21 - std::backtrace_rs::backtrace::trace_unsynchronized::h5da3e203eef39e9f
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x5568981fba21 - std::sys_common::backtrace::_print_fmt::h8d28d3f20588ae4c
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x5568981fba21 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd9a5b0c9c6b058c0
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x55689822981f - core::fmt::rt::Argument::fmt::h0afc04119f252b53
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/fmt/rt.rs:138:9
   5:     0x55689822981f - core::fmt::write::h50b1b3e73851a6fe
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/fmt/mod.rs:1094:21
   6:     0x5568981f77d7 - std::io::Write::write_fmt::h184eaf275e4484f0
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/io/mod.rs:1714:15
   7:     0x5568981fb835 - std::sys_common::backtrace::_print::hf58c3a5a25090e71
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x5568981fb835 - std::sys_common::backtrace::print::hb9cf0a7c7f077819
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x5568981fcfa3 - std::panicking::default_hook::{{closure}}::h066adb2e3f3e2c07
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:269:22
  10:     0x5568981fcd34 - std::panicking::default_hook::h277fa2776900ff14
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:288:9
  11:     0x5568981fd529 - std::panicking::rust_panic_with_hook::hceaf38da6d9db792
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:705:13
  12:     0x5568981fd427 - std::panicking::begin_panic_handler::{{closure}}::h2bce3ed2516af7df
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:597:13
  13:     0x5568981fbe86 - std::sys_common::backtrace::__rust_end_short_backtrace::h090f3faf8f98a395
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:151:18
  14:     0x5568981fd172 - rust_begin_unwind
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:593:5
  15:     0x556897122893 - core::panicking::panic_fmt::h4ec8274704d163a3
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:67:14
  16:     0x5568971229f2 - core::panicking::panic_bounds_check::h11601ba3567ad740
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:162:5
  17:     0x55689728903a - ordhook::db::parse_satpoint_to_watch::h556b740a80b3d32d
  18:     0x5568972804c5 - ordhook::db::insert_inscription_in_locations::h52e4bc75ec8b9e8f
  19:     0x5568972808bf - ordhook::db::update_inscriptions_with_block::h39d4c3e70b7f9620
  20:     0x5568973d77be - ordhook::core::protocol::inscription_sequencing::augment_block_with_ordinals_inscriptions_data_and_write_to_db_tx::h226f91ae249cf186
  21:     0x55689724f8c9 - ordhook::core::pipeline::processors::inscription_indexing::process_block::hcd50acee2e3831f2
  22:     0x55689724eb90 - ordhook::core::pipeline::processors::inscription_indexing::process_blocks::h4dad5361c99abb2b
  23:     0x556897378c55 - std::sys_common::backtrace::__rust_begin_short_backtrace::h04c021a1bf03dd82
  24:     0x55689732c1b2 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h2af8557af88c8ed5
  25:     0x5568982002b5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc0b1022758ecac73
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/boxed.rs:1993:9
  26:     0x5568982002b5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h0c9654ebe7ad657e
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/boxed.rs:1993:9
  27:     0x5568982002b5 - std::sys::unix::thread::thread::new::thread_start::h04c8e9c7d83d3bd5
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys/unix/thread.rs:108:17
  28:     0x7ffbe4e59ea7 - start_thread
  29:     0x7ffbe4c2fa2f - clone
  30:                0x0 - <unknown>

My current config is:

[storage]
working_dir = "/opt/ordhook"

# The Http Api allows you to register / deregister
# dynamically predicates.
# Disable by default.
#
 [http_api]
 http_port = 20456
 database_uri = "xxx"

[network]
mode = "mainnet"
bitcoind_rpc_url = "xxx"
bitcoind_rpc_username = "xxx"
bitcoind_rpc_password = "xxx"
# Bitcoin block events can be received by Chainhook
# either through a Bitcoin node's ZeroMQ interface,
# or through the Stacks node. Zmq is being
# used by default:
bitcoind_zmq_url = "xxx"
# but stacks can also be used:
# stacks_node_rpc_url = "http://0.0.0.0:20443"

[limits]
max_number_of_bitcoin_predicates = 100
max_number_of_concurrent_bitcoin_scans = 100
max_number_of_processing_threads = 200
bitcoin_concurrent_http_requests_max = 16
max_caching_memory_size_mb = 32000

# Disable the following section if the state
# must be built locally
[bootstrap]
download_url = "https://archive.hiro.so/mainnet/ordhook/mainnet-ordhook-sqlite-latest"

[logs]
ordinals_internals = true
chainhook_internals = true

config command enhancement

When issuing the command $ ordhook config new a general error messaging is raised:

Aug 30 12:10:06.841 ERRO Invalid combination of arguments

Can we make this error more self-explanable? Or even default to a pre-defined target network?

Add method to get brc20 activity based on tx_id

I think the parameter of tx_id can be added in /ordinals/v1/brc-20/activity

"internal error: entered unreachable code" when starting service on testnet

Hi, I have encountered the following error when I started the ordhook service on testnet, restarted service still get the same error.

Can someone help me to figure out where is the problem? thanks.

Oct 12 09:25:20.488 INFO Starting service...
Oct 12 09:25:20.496 INFO Indexing inscriptions from block #2436517 to block #2532329
Oct 12 09:25:23.604 INFO Processing 100 blocks
Oct 12 09:25:23.604 INFO Number of inscriptions in block #2436517 to process: 15 (L1 cache hits: 0, queue: [2436518, 2436519, 2436520, 2436521, 2436522, 2436523, 2436524, 2436525, 2436526, 2436527, 2436528, 2436529, 2436530, 2436531, 2436532, 2436533, 2436534, 2436535, 2436536, 2436537, 2436538, 2436539, 2436540, 2436541, 2436542, 2436543, 2436544, 2436545, 2436546, 2436547, 2436548, 2436549, 2436550, 2436551, 2436552, 2436553, 2436554, 2436555, 2436556, 2436557, 2436558, 2436559, 2436560, 2436561, 2436562, 2436563, 2436564, 2436565, 2436566, 2436567, 2436568, 2436569, 2436570, 2436571, 2436572, 2436573, 2436574, 2436575, 2436576, 2436577, 2436578, 2436579, 2436580, 2436581, 2436582, 2436583, 2436584, 2436585, 2436586, 2436587, 2436588, 2436589, 2436590, 2436591, 2436592, 2436593, 2436594, 2436595, 2436596, 2436597, 2436598, 2436599, 2436600, 2436601, 2436602, 2436603, 2436604, 2436605, 2436606, 2436607, 2436608, 2436609, 2436610, 2436611, 2436612, 2436613, 2436614, 2436615, 2436616], L1 cache len: 0, L2 cache len: 0)
Oct 12 09:25:23.980 INFO Completed ordinal number retrieval for Satpoint 0x5303827f29ff1486a05a3012531d2ece294995e8ee834b140620f4d92f2f6794:0:0 (block: #332855:1834831058, transfers: 3250, progress: 1/15, priority queue: true, thread: 10)
Oct 12 09:25:23.980 INFO Number of inscriptions in block #2436517 to pre-process: 0
Oct 12 09:25:23.980 INFO Completed ordinal number retrieval for Satpoint 0x8b6d1321fa24e048f510afd34bc27daf9d3fde7ba3a74e0be10a0706b7565110:0:0 (block: #332855:1834816714, transfers: 3246, progress: 2/15, priority queue: true, thread: 12)
Oct 12 09:25:23.980 INFO Number of inscriptions in block #2436517 to pre-process: 17
Oct 12 09:25:23.981 INFO Completed ordinal number retrieval for Satpoint 0x8e9b08cf7158c6483701a88c14f2791a6d8fcf474092bdc50ff24de75ee65bdf:0:0 (block: #332855:1834823886, transfers: 3248, progress: 3/15, priority queue: true, thread: 0)
Oct 12 09:25:23.985 INFO Completed ordinal number retrieval for Satpoint 0x60a0570889b360de7d77a70745fa67378ec6f150d13999a5890bafbd76d3b05c:0:0 (block: #332855:1834820300, transfers: 3247, progress: 4/15, priority queue: true, thread: 13)
Oct 12 09:25:23.985 INFO Completed ordinal number retrieval for Satpoint 0x1b90f5bfc6c135b4cafaa709fd10b246f9704753f7657903d9face1ab8875923:0:0 (block: #332855:1834827472, transfers: 3249, progress: 5/15, priority queue: true, thread: 4)
Oct 12 09:25:24.003 INFO Completed ordinal number retrieval for Satpoint 0xbc413c5dd2c45625dc9bd955bff8b982546dce3a2afc04be47abd1b885188ed6:0:0 (block: #386926:481724476, transfers: 5705, progress: 6/15, priority queue: true, thread: 11)
Oct 12 09:25:24.127 INFO Completed ordinal number retrieval for Satpoint 0x79f238e97a29fff293fb5c2237c646535017f5663f789f4b8c1bd1e72336e88b:0:0 (block: #356950:1747587355, transfers: 8442, progress: 7/15, priority queue: true, thread: 5)
Oct 12 09:25:24.128 INFO Completed ordinal number retrieval for Satpoint 0xf41354158c50a23d5b3f441b560abbf5ff805ea09e0828c21f8425f239791055:0:0 (block: #356950:1747584743, transfers: 8442, progress: 8/15, priority queue: true, thread: 8)
Oct 12 09:25:24.128 INFO Completed ordinal number retrieval for Satpoint 0x59fb59ca78961820b7509606116f0ae658bb6a3200b3e7f306a4ef4b22e8ab86:0:0 (block: #356950:1747579519, transfers: 8442, progress: 9/15, priority queue: true, thread: 14)
Oct 12 09:25:24.128 INFO Completed ordinal number retrieval for Satpoint 0xdb8d8abc5fe09a367407b4ae2cd365ec9cd9df9899d197496a05f98908633255:0:0 (block: #356950:1747580825, transfers: 8442, progress: 10/15, priority queue: true, thread: 6)
Oct 12 09:25:24.128 INFO Completed ordinal number retrieval for Satpoint 0xdb251795985cf44687e8e50219840d56e6f8c0d53b654f60f63620777cad3acc:0:0 (block: #356950:1747586049, transfers: 8442, progress: 11/15, priority queue: true, thread: 3)
Oct 12 09:25:24.129 INFO Completed ordinal number retrieval for Satpoint 0xe44e5f4cb7cb0fa9e6c0b698e99c9e1c4c50b659223751db716a9af855bb3786:0:0 (block: #356950:1747582131, transfers: 8442, progress: 12/15, priority queue: true, thread: 2)
Oct 12 09:25:24.129 INFO Completed ordinal number retrieval for Satpoint 0x815e62ba8c8ab0c46674a9345acd93da08c72ab37cc7a6537a391854d81ace1d:0:0 (block: #356950:1747578213, transfers: 8441, progress: 13/15, priority queue: true, thread: 1)
Oct 12 09:25:24.129 INFO Completed ordinal number retrieval for Satpoint 0x4bbf77556d66e9a83cfe1eb853afa630402adfba58b93c51775a49a7783ab8a2:0:0 (block: #356950:1747583437, transfers: 8442, progress: 14/15, priority queue: true, thread: 7)
thread 'Worker' panicked at 'internal error: entered unreachable code', components/ordhook-core/src/core/protocol/satoshi_numbering.rs:237:25
stack backtrace:
0: 0x55cf4e3c7810 - std::backtrace_rs::backtrace::libunwind::trace::h1e9c869ddabe891c
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x55cf4e3c7810 - std::backtrace_rs::backtrace::trace_unsynchronized::h1d559bd09dbc24c9
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x55cf4e3c7810 - std::sys_common::backtrace::_print_fmt::h3ae709387ceadd9f
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/sys_common/backtrace.rs:65:5
3: 0x55cf4e3c7810 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h19496e227970904a
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/sys_common/backtrace.rs:44:22
4: 0x55cf4e3f521c - core::fmt::rt::Argument::fmt::h18fc21a41543ef0e
at /builddir/build/BUILD/rustc-1.72.0-src/library/core/src/fmt/rt.rs:138:9
5: 0x55cf4e3f521c - core::fmt::write::h78fbcccbc63e799c
at /builddir/build/BUILD/rustc-1.72.0-src/library/core/src/fmt/mod.rs:1094:21
6: 0x55cf4e3c3c07 - std::io::Write::write_fmt::h1e345f3cfc2298b9
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/io/mod.rs:1714:15
7: 0x55cf4e3c75c1 - std::sys_common::backtrace::_print::h9d70a06057e2bba5
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/sys_common/backtrace.rs:47:5
8: 0x55cf4e3c75c1 - std::sys_common::backtrace::print::h7095f0a3537d7cab
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/sys_common/backtrace.rs:34:9
9: 0x55cf4e3c8ede - std::panicking::default_hook::{{closure}}::hfaea52c1f7cf1d21
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/panicking.rs:269:22
10: 0x55cf4e3c8bee - std::panicking::default_hook::haaa8bc63372a2b35
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/panicking.rs:288:9
11: 0x55cf4e3c94b4 - std::panicking::rust_panic_with_hook::hc69bf3eb9cc3946e
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/panicking.rs:705:13
12: 0x55cf4e3c9369 - std::panicking::begin_panic_handler::{{closure}}::h656585ee80d93332
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/panicking.rs:595:13
13: 0x55cf4e3c7c66 - std::sys_common::backtrace::__rust_end_short_backtrace::h01707de9d3720233
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/sys_common/backtrace.rs:151:18
14: 0x55cf4e3c90c7 - rust_begin_unwind
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/panicking.rs:593:5
15: 0x55cf4d3fb443 - core::panicking::panic_fmt::h968ed060d7c6cd97
at /builddir/build/BUILD/rustc-1.72.0-src/library/core/src/panicking.rs:67:14
16: 0x55cf4d3fb4d3 - core::panicking::panic::h7cbea5846c0c91a6
at /builddir/build/BUILD/rustc-1.72.0-src/library/core/src/panicking.rs:117:5
17: 0x55cf4d6d6ec7 - ordhook::core::protocol::satoshi_numbering::compute_satoshi_number::h90e5a78de084b450
18: 0x55cf4d56513a - std::sys_common::backtrace::__rust_begin_short_backtrace::h5d886db980cc4e2f
19: 0x55cf4d56caae - core::ops::function::FnOnce::call_once{{vtable.shim}}::he7465ca6664149c0
20: 0x55cf4e3cbed2 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::h01c2b804072cdb0f
at /builddir/build/BUILD/rustc-1.72.0-src/library/alloc/src/boxed.rs:1993:9
21: 0x55cf4e3cbed2 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::hebf6a458c08a34f6
at /builddir/build/BUILD/rustc-1.72.0-src/library/alloc/src/boxed.rs:1993:9
22: 0x55cf4e3cbed2 - std::sys::unix::thread::Thread::new::thread_start::h8344d8c31e93c732
at /builddir/build/BUILD/rustc-1.72.0-src/library/std/src/sys/unix/thread.rs:108:17
23: 0x7f90b287fea5 - start_thread
24: 0x7f90b20a2b0d - clone
25: 0x0 -

db sync gets stuck

Running ordhook db sync --config-path=./Ordhook.toml

Seems to get stuck after a few thousand blocks. It outputs like below :

....
Dec 15 10:56:47.970 INFO Block #50072 saved to disk
Dec 15 10:56:47.970 INFO Block #50088 saved to disk
Dec 15 10:56:47.970 INFO Block #50104 saved to disk
Dec 15 10:56:47.970 INFO Block #50120 saved to disk
Dec 15 10:56:47.970 INFO Block #50136 saved to disk
Dec 15 10:56:47.970 INFO Block #50152 saved to disk
Dec 15 10:56:47.971 INFO Block #50167 saved to disk
Dec 15 10:56:47.971 INFO Block #50183 saved to disk
Dec 15 10:56:47.971 INFO Block #50200 saved to disk

But then does not proceed.

I also see this line being output which may be a related problem:

thread 'Block data compression' panicked at 'called `Option::unwrap()` on a `None` value', components/ordhook-core/src/db/mod.rs:1583:59

Handle tracking of inscriptions spent in fees

Ordinals-api shows [FastifyError: Request body is too large] when receiving activities

ordhook scan blocks begin end --post-to=http://localhost:3000/api/events --config-path=./Ordhook.toml

The above command uses Ordhook to stream and then post ordinal activities to http://localhost:3000/api/events

When I use this feature or just start the service with --post-to, the server in Ordinals-api will meet the problem as follows:

SPIKE: Speed up ordhook CI for pull requests [4 hours]

Recent runs are taking 22 minutes for the Cargo test step and 29 minutes for the Build/Tag/Push Image step. Can we speed either step up?

Research & Implementation spike. In scope:

Spend 4 hours looking into, and if possible implementing, any changes that can speed up our CI process.

Support Jubilee

ordhook frequently running out of memory in a kubernetes container

Hi! I'm currently running ordhook as a container inside a k8s cluster on a node with the following characteristics:

OS: linux (amd64)
OS Image: Amazon Linux 2
Kernel version: 5.10.199-190.747.amzn2.x86_64
Container runtime: containerd://1.7.2
Kubelet version: v1.24.17-eks-e71965b

The latest logs can be seen below:

{"msg":"Completed ordinal number retrieval for Satpoint 0x05bbb59c01d9f3f44490c551856549d417993e426fcb26b06c5d79cbcaad13d6:0:0 (block: #89596:3943833505, transfers: 817, progress: 357/6103, priority queue: true, thread: 156)","level":"INFO","ts":"2024-01-08T15:55:08.046252351Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x1a72b28b5cc48b1764e547d19036184d9ad1cb01f3b4af5a6b66812a8e0c76d7:0:0 (block: #792260:503784574, transfers: 108, progress: 358/6103, priority queue: true, thread: 157)","level":"INFO","ts":"2024-01-08T15:55:08.046280667Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x4a9cd716190a772699c24406d9e0555cf1453295a661a02d0ea597670505a4d8:0:0 (block: #688454:299151663, transfers: 424, progress: 359/6103, priority queue: true, thread: 158)","level":"INFO","ts":"2024-01-08T15:55:08.046308098Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x0706338d6d7e13d17fd67af60309c8e351c5a0245b03c14e5cf6fe4bdb917ed9:0:0 (block: #89596:3943820770, transfers: 817, progress: 360/6103, priority queue: true, thread: 159)","level":"INFO","ts":"2024-01-08T15:55:08.046334941Z"}
{"msg":"Starting service...","level":"INFO","ts":"2024-01-08T15:55:28.536621703Z"}
{"msg":"Checking database integrity up to block #797798","level":"INFO","ts":"2024-01-08T15:55:28.871883967Z"}

My current setup is described in a manifest of a stateful-set type on a node with 30 GiB total memory. It seems to be working till it reaches the currently set memory limit and restarts due to:
Reason:Reason: OOMKilled - exit code: 137 and I'm not sure it this is a memory leak or some k8s specific issue. With different values set for the memory limit the usage seem to be near the allowed max but eventually crosses it regardless of the value in place.

I've dug through the whole source code of the configuration and there is very little to adjust. Needless to say I've tried modifying the max_caching_memory_size_mb with no apparent effect on the described above issue. Is there a way to better handle resources via ordhook? I think it would be really beneficial to have an application level defined memory limit to avoid frequent restarts and time consuming integrity checks.