bestarch-ae/cacherpc

Solana JSON-RPC caching server

cacherpc's Issues

MailboxError panics

thread 'actix-rt:worker:3' panicked at 'actor is dead: MailboxError(Mailbox has closed)

Log updated config

1.when config is reloaded, log new config
2.when config is applied, log new values (limits are already being logged)

Unix socket based, command listener implementation

For the purposes of convenient management of dynamic settings of the application, there's a proposal to implement unix socket listener, which will listen for external commands, and to dynamically change the settings accordingly. It will be quite flexible alternative for signal handling.

`filter::tests::tree_matches_overall` can randomly fail

Currently filter::tests::tree_matches_overall can randomly panic with 'Uniform::new called with `low >= high`' message, which probably means that proptest generates x..x strategy at some point. My guess is that this is related to arb_non_matching_filters strategy.

This is not a critical issue since the panic is not caused by the test logic, but is annoying. For the time being, broken CI can be fixed with a restart.

Cleanup client code in get_account_info / get_program_accounts

It's basically the same code, could be refactored into a function.

Detect infinite reconnections on websocket

Sometimes, some worker threads, that manager webosket connections, go into infinite loop trying to establish working websocket connection to validator. Need to figure out the reason, why it happens, and implement detection and correction for such cases.

Implement metrics endpoint with Prometheus format

Allow monitoring caching performance by keeping metrics and printing them in Prometheus (and/or other) format

getAccountInfo occasionally returns error

It's been reported that getAccountInfo occasionally returns

Error: failed to get info about account 2CSEjyDtAtgCjTKyXHZyfUVe7EERtL7rjYjJSgcBPYLf: Invalid params: invalid type: map, expected string at line 1 column 34

Gist to reproduce here https://gist.github.com/barry-the-sender/d877a87f23d8d52d32fdd71c01f71481.

Pubsub reconnect backoff settings

Make ExponentialBackoff max_elapsed_time infinite with max_interval = 10sec.

Cannot request Stake11111 (Gateway timeout)

When requesting getProgramAccounts(Stake11111...) without filters via the cache I get a gateway timeout error, however it works without the cache.

In logs I can see the following error:

Sep 15 12:39:24.950  WARN cache_rpc::rpc: request: Request { jsonrpc: "2.0", id: Num(1), method: "getProgramAccounts", params: Some((Pubkey([6, 161, 216, 23, 145, 55, 84, 42, 152, 52, 55, 189, 254, 42, 122, 178, 85, 127, 83, 92, 138, 120, 114, 43, 104, 164, 157, 192, 0, 0, 0, 0]), ProgramAccountsConfig { encoding: Base64, commitment: None, data_slice: None, filters: None, with_context: Some(true) })) } error: Overflow
Sep 15 12:39:24.950  INFO cache_rpc::rpc: reporting gateway timeout req.id=Num(1)

This is the RPC request:

curl api.mainnet-beta.solana.com -X POST -H Content-Type: application/json -d 
  {"jsonrpc":"2.0","id":1, "method":"getProgramAccounts", "params":["Stake11111111111111111111111111111111111111", { "encoding": "base64" }]}

Add pubsub metrics

HTTP/rpc code has metrics, websocket/pubsub has none.

Track subscriptions by key

The problem:

We use subscription_active method to check whether we can retrieve the value from cache for a particular key. Currently it's implemented by checking AtomicBool flag for a websocket connection to which this key is routed. This flag is updated in update_status by comparing number of active subscriptions to the number of desired subscriptions, which means that if one of subscription confirmations is delayed, all keys routed to this worker are considered not active, and are not retrievable from cache, which is obviously not desirable.

Proposed solution:

Track subscription status by key.

Make SLOT_DISTANCE configurable

SLOT_DISTANCE threshold is currently hardcoded at 150 (same as default for validator). Make it configurable via command-line option.

Pubsub reconnect counter (and possibly rpc)

Add metric to count reconnect attempts

Any path component should be accepted

To follow the behaviour of the solana validator and be as transparent as possible the cache should accept any path "/" component of the URL. I have seen some applications use this in the wild and it is supported by the validator.

There are also special URL paths in the validator that get translated into non RPC requests like "/health".

Currently cacherpc returns 404 on any path a part from "/". We work around it in our proxy by just overwriting the path component to any request we pass to zubr cache.

Caching getProgramAccounts with filters

It seems like filters are kinda popular, so we should find a way to cache requests with filters.
One possible solution would be to make an additional request without filter and cache the result (using a queue and a background worker, probably).

Content type with charset returns 415

We're seeing 415 errors when the content-type header specifies a charset. This happens on requests where the content-type reads:

application/json;charset=….

Implement getMultipleAccounts

It should be pretty easy to do, basically the same as getAccountInfo

Add ability to reload waf script without restart

Add command like cacherpd reload waf or something

Batch requests support implementatin in WAF

Currently cacher doesn't preprocess batch requests, simply forwarding them to validator cluster.
There's a requirement to support batch requests in WAF rules evaluation, so that the rules will apply to all requests, regardless of their type and composition.

Use proper backoff instead of handrolled loop with sleep

There're some nice backoff libraries on crates.io.

Detect situations when when ws subscriptions are active but no updates received

Cacher relies on the fact that cache entries have corresponding active ws subscriptions, in order to detect whether the cache entry might be stale or not. But sometimes it may be that subscription exists but no updates are being received from it, which possibly may indicate that the cache entry became stale.

There's a need for a way to detect such situations and not to serve request from cache, if it happens.

Slots & commitments

1. We currently store & report the latest slot seen in either validator response or received in pubsub (per commitment level). This leads to slot in our responses lagging behind validator if there're no changes for cached accounts.

We should find out how to get the latest slot for commitment via pubsub (preferably).

2.If we have account A cached with slot = N, commitment = Processed, and later we receive an update for account B with slot N+1, commitment = Finalized, it seems like it means that slot N is also Finalized, which in turn, means that we can return account A's data for a request with Finalized commitment requirement.