uvnsq API

[wlgq2]

uvnsq API

[wlgq2]

1.i delete appendSub and Added appendNsqd interface for multiple nsqd.
2. deleting NsqConsumer object will disconnect all connection.but, it's not thread-safe,you can only delete NsqConsumer object in the loop thread.
3.you need to update uv-cpp to version 1.4.8

[wlgq2]

yes，one connection can only subscribe to one topic && channel.

[wlgq2]

do you mean a NsqConsumer object append multiple tcp connection？
i think this is unreasonable. one consumer should subscribe one topic。you can new multiple consumers object to manage multiple topics。
this consumers can run on the same loop(thread). so you can share variables to pass data.

[dselivanov]

Yes, I mean 1 consumer = 1 nsq topic and multiple connections to multiple nsqd.

[wlgq2]

I will consider how to implement this feature.

[dselivanov]

I've tried to build such functionality myself using multiple NsqConsumer objects. But I'm still not sure how to properly close nsqd connections. Is it just by deleting NsqConsumer instance? Do I need somehow to "de-register" such consumer from event loop?

Sorry for these questions, I'm not an expert in llibuv...

[dselivanov]

@wlgq2 thank you very much for implementing this feature!

Couple of more questions:

• what if I need to close only a particular nsqd connection? This is a valid case where consumer polling nsqdlookup and watch for new and stale connections (which it needs to add and and remove). So may be it worth to add removeNsqd method?
• "only delete NsqConsumer object in the loop thread" - you mean where loop is running, right?
• is appending nsqd thread safe?

[wlgq2]

@dselivanov

1. not support close a particular nsqd connection. uvnsq will auto reconnect a nsqd when this nsqd disconnected.
2. i mean，delete NsqConsumer must called in libuv‘s event loop。because libuv is not thread safe. you can use uv::EvnetLoop::runInThisLoop to register a callback run in libuv’s loop。all uvnsq callback run in libuv‘s loop(this means called from uv::EventLoop::run function)。
3. i recommend appending nsqd before loop->run();

[dselivanov]

Thanks for the answers. The challenge for 1 and 3 is that nsqd producer can come and go.

For example I want to run consumer and read from topic_1.

1. I call nsqlookupd and ask what are nsqd which have topic_1. Suppose these nsqd are nsqd_1 and nsqd_2.
2. Then I connect to nsqd_1 and nsqd_2 and start handle messages
3. Time to time I poll nsqlookupd to check what are the nsqd which have topic_1. And for example at some point of time nsqlookupd says that there is new nsqd_3 which also has topic_1. Then I need to append nsqd_3 and add it to the event loop.
4. It is very good that NsqConsumer will try to reconnect to nsqd automatically. But there is possibility that some nsqd can completely finish, so it makes sense to remove such connections from consumer. So it would be nice to have a method to remove nsqd by host and port: void removeNsqd(string host, uint16_t port)

[wlgq2]

this will make the code complicated，if support for dynamically adding and deleting nsq nodes.

[dselivanov]

yeah, I know, but this is the whole purpose of the nsqlookupd within NSQ architecture. And this is the most popular pattern of NSQ usage...

[dselivanov]

I've been playing with the latest uvnsq and it seems there several parts which are necessary for dynamic nsqd:

1. in NsqConsumer each connection to nsqd shoud have some id (for example string host:port)
2. Need somehow to track from which connection message came from in order to call fin method to the right nsqd.

[wlgq2]

fin auto called in NsqConsumer now .

i will consider whether to support dynamic connect for nsqd, if I have time.

[wlgq2]

@dselivanov
i think the correct design is，one NsqConsumer- one nsqd service. you can use multiple consumers connect multiple nsqds. delete and new consumer for dynamic nsqd.

If you want to deal with multiple nsqd of the same topic，you can set the same callback function for multiple customers while have same topic.

[dselivanov]

Thanks, this makes sense. So such design is more similar to what was in previous version (one connection per nsqConsumer and explicit fin)

[wlgq2]

i update this