hubmakerlabs / replicatr Goto Github PK
View Code? Open in Web Editor NEWNostr relay with Internet Computer integration for inter-relay synchronization
Home Page: https://mleku.dev/git/replicatr
License: GNU General Public License v2.0
Nostr relay with Internet Computer integration for inter-relay synchronization
Home Page: https://mleku.dev/git/replicatr
License: GNU General Public License v2.0
a candid specification for the (envelope) datatypes is required
it is in draft still but NIP-66, relay metadata and monitoring is relevant to the ACL system:
https://github.com/dskvr/nips/blob/this-is-the-last-time/66.md
event kinds 30066 and 30166 seem like they can be self-published automatically by relays and contain useful data to clients implementing and publishing reports on this protocol feature
not high priority but would be good to implement this feature by end of MS3
requires the creation of a relay identity which represents automated reports and potentially later on could be part of a chat based interface
it is already quite simple to feed a collection of events to a relay, half of the code already exists in the export/import and ingestr but to cleanly pull the entire database over the network while hot will be an important disaster prevention feature
the owners prescribed in the relay configuration will be able to send a request with an empty filter that will return every event in the database
a small tool that sends such requests and receives the results and optionally writes to .jsonl
file, and can also read a .jsonl
file and push all of the events in it to the relay while it is live, in both directions using the owner authentication to permit this backup/restore functionality
this is not high priority but should be completed for MS3
write a test suite to make sure that the agent and canister access is working correctly
by accident due to the debugging process for the race conditions in the relay interface and eventstore for replaceable events code that used to find and delete it has been commented out, as this extra step often triggered a race condition
a simple test shows that a kind 0 event being updated currently now adds the event but the order of the results and the number that come back from a query without a limit set is undefined behaviour
in order to make this work as clients expect while exposing the ability to return the history of replaceable events, as most replaceable events are intended to be one per pubkey, the following changes are needed:
created_at
timestamp order, newest first, there isn't any way to specify this and clients usually will sort it but why make work for clients that are usually less efficientthere is the beginnings of an ACL implementation, and a chat based control interface with workaround for clients without NIP-42 - this needs to be completed
code compiles but needs to be tested to work
using new client to send messages and get debugging info back
with a working replica, can now implement the IC backend
making a note for future reference of a possible feature addition
the central idea is adding a new index that adds a reference count to events so that it can be seen how many events are referring to another event... this would probably need to be some kind of weighted and complex score, so that it not only evaluates the direct references, which will be replies, but also, the number of indirect references, to provide a weighting to evaluate whether an event should be evicted preferentially to others of similar age in terms of how recently they are accessed
currently the prune strategy purely is based on Least Recently Used LRU style, but a relevance score like this could help avoid pruning out data that has high relevance
design several data use management strategies, pruning by age, keeping track of last access times of events, and setting low- and high-water marks
creating relay accounts for access to a canister
the storage of events on the badger side should remain less than the IC but for reasons of efficiency in both time and cost of access to the IC the badger event store needs to have cache management strategies to contain the age and size of the database, as well as to immediately store everything in parallel with the IC - IC's main purpose is propagating events to multiple relays
in order to facilitate immediate propagation of new events related to existing subscriptions, the IC canister needs to have a short term storage table containing all active filters currently in open subscriptions, and when new events are stored, the filters are applied to them and if they match, logged in relays should be sent notifications of a match
this will also require some kind of filter identification scheme, probably a truncated hash of the receiver will suffice, and this key needs to be linked in to the relay side so that when the notifications are received the relevant subscriber is sent the matching event
with the candid specs created for the data types, a simple library to encode and decode them into Go runtime variables is required
an important element of the required features for this relay is to enable it to be used in a purely private environment - given a client that actually respects users relay choices...
for this, it is necessary that the relay has a private internal events storage and a class of access permission of administrator and user
the administrator will be able to push administration access control events to the relay which will alter the stored ACL that also needs to be implemented
so, a checklist:
currently relay is not being sent or not accepting follow lists and relay lists from clients, this needs to definitely be working
currently, queries repeatedly decode and decode again events that are hot in the cache, especially recent ones that are sent out in open subscriptions of current websocket connections
it would make a lot of sense to eliminate the duplication of both memory and processing required for this data by changing the event decoding library into a freelist that allows avoiding unbounded spikes of memory use and unnecessary re-decoding of the binary data in the database for events that are being heavily accessed concurrently
create indexes for search acceleration
these will require new data sets in the IC to store tables of references between search keys and events
for now these should be sufficient to improve search capability and effectively implement some level of DVM capability
a candid specification for the (filter and event) datatypes is required
may not be so good for its lack of range scanning... investigate
the relay will monitor events to catch kind 4 messages with the relay's public key as the recipient, and respond to them
this is a less complicated way to implement user access control, owners and administrators can just issue commands, when you first message the relay or message it with /help it will print out the current command options
the existing encoder for storage in the local storage database (ie badger) is extremely silly and is to be replaced with simple gob encoding of the raw event data and eliminating the intermediate type
there is a bug in the eventstore query function that seems to stall the websocket and cause the client sending events to time out the socket context (7 seconds)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.