Giter Site home page Giter Site logo

wankdanker / node-discover Goto Github PK

View Code? Open in Web Editor NEW
229.0 17.0 59.0 162 KB

Automatic and decentralized discovery and monitoring of nodejs instances with built in support for a variable number of master processes, service advertising and channel messaging.

JavaScript 100.00%

node-discover's Introduction

node-discover

npm version

Automatic and decentralized discovery and monitoring of nodejs instances with built in support for a variable number of master processes, service advertising and channel messaging.

Why?

So, you have a whole bunch of node processes running but you have no way within each process to determine where the other processes are or what they can do. This module aims to make discovery of new processes as simple as possible. Additionally, what if you want one process to be in charge of a cluster of processes? This module also has automatic master process selection.

Compatibility

This module uses broadcast and multicast features from node's dgram module. All required features of the dgram module are implemented in the following versions of node

- v0.4.x
- v0.6.9+

Example

Be sure to look in the examples folder, especially at the distributed event emitter

var Discover = require('node-discover');

var d = Discover();

d.on("promotion", function () {
	/* 
		* Launch things this master process should do.
		* 
		* For example:
		*	- Monitior your redis servers and handle failover by issuing slaveof
		*    commands then notify other node instances to use the new master
		*	- Make sure there are a certain number of nodes in the cluster and 
		*    launch new ones if there are not enough
		*	- whatever
		* 
		*/
		
	console.log("I was promoted to a master.");
});

d.on("demotion", function () {
	/*
		* End all master specific functions or whatever you might like. 
		*
		*/
	
	console.log("I was demoted from being a master.");
});

d.on("added", function (obj) {
	console.log("A new node has been added.");
});

d.on("removed", function (obj) {
	console.log("A node has been removed.");
});

d.on("master", function (obj) {
	/*
		* A new master process has been selected
		* 
		* Things we might want to do:
		* 	- Review what the new master is advertising use its services
		*	- Kill all connections to the old master
		*/
		
	console.log("A new master is in control");
});

Installing

npm

npm install node-discover

git

git clone git://github.com/wankdanker/node-discover.git

API

Constructor

Discover(opts, callback)
  • opts - object
    • helloInterval : How often to broadcast a hello packet in milliseconds
      • Default: 1000
    • checkInterval : How often to to check for missing nodes in milliseconds
      • Default: 2000
    • nodeTimeout : Consider a node dead if not seen in this many milliseconds
      • Default: 2000
    • masterTimeout : Consider a master node dead if not seen in this many milliseconds
      • Default: 2000
    • address : Address to bind to
      • Default: '0.0.0.0'
    • port : Port on which to bind and communicate with other node-discover processes
      • Default: 12345
    • broadcast : Broadcast address if using broadcast
      • Default: '255.255.255.255'
    • multicast : Multicast address if using multicast
      • Default: null (don't use multicast, use broadcast)
    • mulitcastTTL : Multicast TTL for when using multicast
      • Default: 1
    • unicast : Comma separated String or Array of Unicast addresses of known nodes
      • It is advised to specify the address of the local interface when using unicast and expecting local discovery to work
    • key : Encryption key if your broadcast packets should be encrypted
      • Default: null (that means no encryption)
    • mastersRequired : The count of master processes that should always be available
    • weight : A number used to determine the preference for a specific process to become master
      • Default : Discover.weight()
      • Higher numbers win.
    • client : When true operate in client only mode (don't broadcast existence of node, just listen and discover)
      • Default : false
    • reuseAddr : Allow multiple processes on the same host to bind to the same address and port.
      • Default: true
      • Only applies to node v0.12+
    • ignoreProcess : If set to false, will not ignore messages from other Discover instances within the same process (on non-reserved channels), join() will receive them.
      • Default: true
    • ignoreInstance : If set to false, will not ignore messages from self (on non-reserved channels), join() will receive them.
      • Default: true
    • advertisement : The initial advertisement object which is sent with each hello packet.
    • hostname : Override the OS hostname with a custom value.
      • Default: null (use DISCOVERY_HOSTNAME env var or the OS hostname)
  • callback - function that is called when everything is up and running
    • signature : callback(err, success)

Attributes

  • nodes

Methods

promote()

Promote the instance to master.

This causes the old master to demote.

var Discover = require('node-discover');
var d = Discover();

d.promote();

demote(permanent=false)

Demote the instance from being a master. Optionally pass true to demote to specify that this node should not automatically become master again.

This causes another node to become master

var Discover = require('node-discover');
var d = Discover();

d.demote(); //this node is still eligible to become a master node.

//or

d.demote(true); //this node is no longer eligible to become a master node.

join(channel, messageCallback)

Join a channel on which to receive messages/objects

var Discover = require('node-discover');
var d = Discover();

//Pass the channel and the callback function for handling received data from that channel
var success = d.join("config-updates", function (data) {
	if (data.redisMaster) {
		//connect to the new redis master
	}
});

if (!success) {
	//could not join that channel; probably because it is reserved
}

Reserved channels

  • promotion
  • demotion
  • added
  • removed
  • master
  • hello

leave(channel)

Leave a channel

var Discover = require('node-discover');
var d = Discover();

//Pass the channel which we want to leave
var success = d.leave("config-updates");

if (!success) {
	//could leave channel; who cares?
}

send(channel, objectToSend)

Send a message/object on a specific channel

var Discover = require('node-discover');
var d = Discover();

var success = d.send("config-updates", { redisMaster : "10.0.1.4" });

if (!succes) {
	//could not send on that channel; probably because it is reserved
}

advertise(objectToAdvertise)

Advertise an object or message with each hello packet; this is completely arbitrary. Make this object/message whatever applies to your application that you want your nodes to know about the other nodes.

var Discover = require('node-discover');
var d = Discover();

d.advertise({
	localServices : [
		{ type : 'http', port : '9911', description : 'my awesome http server' },
		{ type : 'smtp', port : '25', description : 'smtp server' },
	]
});

//or

d.advertise("i love nodejs");

//or

d.advertise({ something : "something" });

start()

Start broadcasting hello packets and checking for missing nodes (start is called automatically in the constructor)

var Discover = require('node-discover');
var d = Discover();

d.start();

stop()

Stop broadcasting hello packets and checking for missing nodes

var Discover = require('node-discover');
var d = Discover();

d.stop();

eachNode(fn)

For each node execute fn, passing fn the node fn(node)

var Discover = require('node-discover');
var d = Discover();

d.eachNode(function (node) {
	if (node.advertisement == "i love nodejs") {
		console.log("nodejs loves this node too");
	}
});

Events

Each event is passed the Node Object for which the event is occuring.

promotion

Triggered when the node has been promoted to a master.

  • Could happen by calling the promote() method
  • Could happen by the current master instance being demoted and this instance automatically being promoted
  • Could happen by the current master instance dying and this instance automatically being promoted

demotion

Triggered when the node is no longer a master.

  • Could happen by calling the demote() method
  • Could happen by another node promoting itself to master

added

Triggered when a new node is discovered

removed

Triggered when a new node is not heard from within nodeTimeout

master

Triggered when a new master has been selected

helloReceived

Triggered when the node has received a hello from given one

helloEmitted

Triggered when the node sends a hello packet

Node Object

{ 
	isMaster: true,
	isMasterEligible: true,
	advertisement: null,
	lastSeen: 1317323922551,
	address: '10.0.0.1',
	port: 12345,
	id: '31d39c91d4dfd7cdaa56738de8240bc4',
	hostName : 'myMachine'
}

TODO

I have not tested large packets. The current version does not handle recombining split messages.

LICENSE

(MIT License)

Copyright (c) 2011 Dan VerWeire [email protected]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

node-discover's People

Contributors

aikar avatar dashersw avatar eser avatar goran-n avatar kostiak avatar nickytope avatar super-ienien avatar wankdanker avatar wehriam avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-discover's Issues

Dynamically adding nodes to unicast cluster

Hi,
I'm trying to use this for a current project and was wondering if it's possible to dynamically add a new node to a unicast cluster.

I've been trying for a little while to no luck, it seems I can update the destination lists but the node its self isn't added to the node list of all the siblings.

Any advice would be great, thanks!

Using Node-Discover

Hi,

I have started using node.js recently and got fascinated by its speed and processing. I was just trying all the modules I found. But can anyone tell me how to use Node-discover protocol to discover nodes. I have tried many ways (which might be wrong), resulted in null.
How to implement node-discover in-order to discover running node.js process?

Please help me.

Using Discover inside cluster causes unhandled exception on windows

According to NodeJS archives , sharing a UDP port on windows causes an ENOTSUP unhandled exception.
This ofcourse happens with Discover.

The solution offered is to mark UDP ports as exclusive. This does not solve the issue for someone wanting to have multiple cluster childs in the some discovery cluster, but it does for those using different childs for different Discovery clusters on different ports.

I managed to fix this on my development by changing
self.socket.bind(self.port, self.address, function () {
to
self.socket.bind({port:self.port, address:self.address,exclusive:true}, function () {
on Network.prototype.start in network.js

I think this can be also changed to
self.socket.bind({port:self.port, address:self.address,exclusive:!self.reuseAddr}, function () {
so that no extra option is created for this issue.

If some other solution exists for Windows, please forgive my ignorance. I am quite the beginner in nodejs-cluster development

Node Weight not honoured on new node

I also just noticed that the node weight is not being honoured when a new node is being added. It appears to always set the new node to master ahead of existing masters regardless of the weight of the node.

I am attempting to setup a system so that the older node will always be master e.g.:

var n1 = new Discover({
            weight: Date.now() * -1 
        });

I suggest either checking the weight when a new node is added, or perhaps there should be some sort of "shouldBeMaster()" function that can be used on both new node & in the checkId interval?

I only note this because even though I haven't tried it, it looks like if there should be a master count of n > 1, then it will only promote a node if it has the highest weight as opposed to being in the top n weights. I have not tested to see if this is the case though, just looks like it might be a bug.

cluster with nodes in the different subnets it is possible?

i think manually add ip-adresses of nodes from rancher-metadata (some service with node-list)

for example ip is dunamic and can be 10.42.185.29, 10.42.2.232, 10.42.69.231 ..

how i can add this nodes to cluster or make right broadcast ?

Advertising > 1218 characters

When my service foo advertises with a payload greater than 1218 characters (I'm stringifying the object to get the length) it seems that d.added doesn't get triggered from other services. Is there some maximum character limit I'm running up against?

Masters Required count doesn't include self

I have noticed that if I set mastersRequired to 1, it will allow up to 2 masters as the master count when processing a new master does not include self.me.isMaster in the count.

I presume this is not expected behaviour?

Possible wrong examples in doc?

I tried hours by using the examples, not running at all. Then I find out maybe the example should call send in Discover()'s callback function. Please check.

var Discover = require('node-discover');
var d = Discover(function(){
//this will success.
var success = d.send("someChannel", { message : "oh..." });
});
//outside the callback will cause destination array is undefined inside networks.js:153
// var success = d.send("someChannel", { message : "oh..." });
//also I try to running start() @ this point, it will emit a error "address already bind", cause Discover() already called one, maybe some prevention code should be here to accept multiply start() calls?

Message size issues

I have an array of objects, the contents does not really matter but as soon as the array length goes over 30 it wont send at all.

Could this be an issue of the encryption methods? or because its UDP?

Invalid 'main' field

I am using Cote and monitor tool logged this to console.

import cote from 'cote';
cote.MonitoringTool(3000);

DeprecationWarning: Invalid 'main' field in '..../node_modules/@dashersw/node-discover/package.json' of 'discover'. Please either fix that or report it to the module author

@wankdanker How could i help to solve this?

Problem with two nodes

In case if one node dies and two nodes remain, then almost always we have two masters. And we have these events:
1: promotion
2: promotion
2: master
1: master

Leader Election/Consensus Protocol

Which consensus protocol do you actually use under the covers for node-discover? Which published scheme is it closest to? Paxos vs Raft for example.

Thanks,

Travell

Usecase for two discovery nodes with the same processUuid

Right now, every node gets an instanceUuid and a processUuid. If you start two discovery nodes from the same process, they will get the same processUuid, for example:

var Discover = require('node-discover');

var d1 = new Discover();  
var d2 = new Discover();

d1 and d2 will have the same processUuid. They will not discover each other, but a discovery node outside that process will discovery them both. So that limits each node process to one discovery node. Is there a usecase for this, or another reason why I wouldn't be able to have multiple discovery instances (that are able to discovery each other) as part of the same node process?

I was thinking of adding something like a 'multiInstance" flag to the constructor but wanted to see if I'm missing something and what the purpose of processUuid is.

Couldn't run ddnode.js

I've tried to run ddnode-test.js, but I've found few errors in there, from line 47 [client.id]. Can anyone help me with this issue.

Thanks in advance.

Crashes if no network adaptor alive (if using multicast)

If (for some reason) the network adaptor is unavailable at start and you are using multicast then the module dies horribly with a [addMembership ENODEV] error.

Should be handled in some way - either emit the error back up - or retry in some way.

node doesn't know it's id

Hi,

I have the next question.
When some node connects to the network, the other node can identify it by id and the hostname.
Like this (console log):
node-app A new node has been added. {"isMaster":true,"isMasterEligible":true,"weight":-0.1450726893915,"address":"172.28.128.21","lastSeen":1450726903761,"hostName":"C","port":22222,"id":"99cbcfef-4d0f-435e-afab-495f69ac35b0"} +716ms

However, I haven't found the place inside the node itself, which contains the same id.

I found the:

d.broadcast.instanceUuid
d.broadcast.processUuid

However, these are not the ids which other nodes are using to identify the newly connected node.

Why I need this? I want to include this information to the channel message, so the node would know that this message was addressed to it.

Any thoughts if it is possible? Or what is the expected behaviour? I don't want to identify the nodes by the hostnames, because those are not uniq.

Regards,

don't put callback inside try block

this can be a weird bug:
https://github.com/wankdanker/node-discover/blob/master/lib/network.js#L111-L116

if there is a runtime error when the callback in called (and that is in other code, so anything could happen) then the try will catch it, and then call the callback again.
so the callback will be called twice, once with a success, then again with an error,
except with an error that was actually IN the CALLBACK.

this will be very weird, so once you are bitten by that you never forget.

Incoming data not fully exposed

When a data packet arrives at the underlying network.js (line 56) if gets emitted along with the complete object and the rinfo.

In the join function in the layer above (discover.js) when that get's handled ( lines 285-7 ) only the data part gets passed on... Is there any good reason not to extend that function to pass them all on ? (or at least the object ?

    self.broadcast.on(channel, function (data, obj, rinfo) {
        self.emit(channel, data, obj, rinfo);
    });

(Though in fact all I am after is the obj.hostName but hey.)

Security warning from node-uuid

node-discover is using the deprecated / no-longer-maintained node-uuid module. That is resulting in console.warn messages being logged:

[SECURITY] node-uuid: crypto not usable, falling back to insecure Math.random()

I believe the solution is to switch from node-uuid to uuid.

.advertise network interface binding?

how can I set on which interface the advertise packets should be sent out while using .advertise()?
Currently it seems that node-discover uses only one of the available network interfaces in a multi network interface scenario.
did I miss something or is binding to a specific interface not available?

New Leadership module introduces undesirable master flopping

If you run node examples/master.js in three separate terminals, you'll see that the last node added becomes a master and then gets demoted shortly after.

This happens because the BasicLeadershipElection.prototype.check is being called on each BasicLeadershipElection.prototype.helloReceived. So, if the first packet the newest node sees is from not a master it will promote itself. There should be a timeout so that the new node can have some time to learn about all the nodes in the network before determining if it should be promoted.

related: #34, #35

remove a client from the cluster

Is it possible to remove a client from a cluster?

Two possibilities that I can think of:

  1. Some one/process/computer undesireable has joined the cluster
  2. The client software does not have the required "code" to listen to a broadcast and "unjoin" the network.

is there a way to get this in node discover?
is there a way to go about it? I can develop it.
give me some insights

node-discover for nodes which are not in the same LAN

Hi,

Sorry for asking the question which is probably obvious, but - as I understand this discovery thing is working perfect, when all the nodes are located within one LAN network.

What if I want to run the nodes in the different LANs, but still want them to find each other. Is there anything which is possible to configure using node-discover for such case?

For example, using ElasticSearch, I am able to state the IPs (where I expect the other nodes to be run) in the configuration, so cluster will gather itself:

discovery.zen.ping.unicast.hosts: ["xxx","yyy","zzz"]

Regards,

Encryption issue

I don't know if this an error or the desired behavior for node-discover:

I have at least 2 apps with multiple instances with different keys (the rest of the conf is the same). I get this error:

TypeError: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
crypto.js:287:27 • Cipher.final
node_modules/node-discovery/lib/streams.js:168:21 • _transform
....

I know that if I set different ports this works fine but if that's the case i need one additional port per app instead of just one for discovery?
Thanks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.