Comments (6)
Hello @morya ,
You are talking about admin needs, but showing client protocol - I think this should not be part of it, at least at first. For now, a method for Node
could be a good start.
Presence for each channel is now kept in single Redis HASH - so there is nothing to scan actually - it's a single key.
In general I'd like to provide more scalability here, it requires more thinking and analysis than provided above.
from centrifuge.
Yes, it's a admin API, but, the client protocol has precense
call too.
what I mean with redis scan, hgetall
could block redis for a while potentially.
And, true, redis scan is not the best solution here.
I was thinking a sync method, like sync between etcd nodes, or sync/psync between redis master/slave.
It could be a heave lifting...
clients will always see all users, won't miss a single join/leave status, maybe a filter to see range of users.
BR
from centrifuge.
Another way could be shard presence to load it chunk by chunk from different keys, sth like pagination. For example, you in channel with 100k subscribers, we could set shard number == 10, and then somehow distribute information over those keys. Not sure about exact algorithm and API for this – just an idea for now.
I'd also like to mention that in Centrifugo PRO we approach the need in massive presence analysis by using ClickHouse analytics - i.e. using a system which can provide access to massive data in near real-time.
from centrifuge.
for now, we use a hack method by read from redis directly with command hscan xxx
.
but, it's not a quite an elegant nor accurate method, when doing both subscribing join/leave
messages and read from hscan
results.
there is no way to keep eventually consistency
data.
that's why I metioned sync between etcd nodes, or sync/psync between redis master/slave
I think it could be a way, like subscribing psync protocol
data from a follower redis
, and parse them, convert them into join/leave
messages.
but it seems quite complicated
from centrifuge.
Yes, it's complicated... Possibly, for such scale you need different model, with some approximation. Sth like mentioned approach with ClickHouse, or some other store. Heavily depends on the target use case – since you have not described it I am just trying to give you alternative directions of thinking.
from centrifuge.
Thanks, really appreciate
from centrifuge.
Related Issues (20)
- ServerJoin event not sent HOT 4
- client.send(data) disconnects clients on empty data HOT 2
- Occasional data race in `initMetricsRegistry` HOT 1
- Clients being disconnected due to "Slow" reason - throttling options HOT 14
- Redis broker pool size HOT 4
- KeyDB connect HOT 3
- private channel HOT 2
- Deprecated ProtocolVersion1 will be removed soon HOT 1
- Client level heartbeat config HOT 7
- Custom recover at connection level HOT 4
- High memory consumption HOT 7
- Apply suggestions from fieldalignment for frequently used objects HOT 1
- Making send call to return a reply HOT 1
- Publish() causes a runtime error when no client subscribed to then channel. HOT 2
- Remove SockJS transport HOT 1
- Embed Gorilla WebSocket HOT 1
- How Many Concurrent RPC Requests can server handle? HOT 2
- [feature] Return error when publishing with incorrect history TTL + history meta TTL configuration
- [bug] Investigate occasional panic in tests
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from centrifuge.