independent starting of master and workers

[unias]

independent starting of master and workers

[SourceZh]

@leebaok
I have something to say.
The method getnode that I write in etcdlib.py is aimed at getting both key and value(particularly key) that are stored in the etcd. And It has no relationship about docklet. Maybe the method name getnode is confused. But it is just because that the method getkey has been defined in the etcdlib.py, so I simply change the method name to getnode.
So, is there any necessary to move the getnode from etcdlib.py to worker.py?

[caodg]

I recommend you move getnode out from etcdlib.py. In fact, you only called getnode once in worker.py. Therefore, there is no need to encapsulate it as a function, even in worker.py.

[leebaok]

@SourceZh

I agree with @caodg ^_^

[SourceZh]

OK, I get it.

[leebaok]

@SourceZh

We merged your pull request of independent starting of master and worker.

But for init mode of startup, there is a bug. In worker.py, you set key of runnodes/IP to 'waitting' in line 57 and than test key of runnodes/IP in line 78. Of cause, it exists and the mode is always recovery.

I think you can use allnodes/IP to decide the mode. For example, self.key in line 60 is sat to allnodes/IP and etcd.listdir in line 76 is to get content of 'machines/allnodes'.

What do you think ?

Waiting for your new pull request !

[SourceZh]

@leebaok
Yes, it is a bug. I will fix it tonight.

[caodg]

There is still a bug. If we init master, the worker will crash. The reason is the node var not defined.

logger.error("get key failed. %s" % node)

There is still a logic error here. If master inits, the worker will definitely fail to get the value. Therefore, the worker should try to register again assuming master has initiated itself. I changed the code as following and it seems working. Please check. @SourceZh @leebaok

diff --git a/src/worker.py b/src/worker.py
index 4a43007..acdaf11 100755
--- a/src/worker.py
+++ b/src/worker.py
@@ -170,8 +170,8 @@ class Worker(object):
                 if value=='ok':
                     self.etcd.setkey("machines/runnodes/"+self.addr, "ok", ttl = 2)
             else:
-                logger.error("get key failed. %s" % node)
-                sys.exit(1)
+                logger.warning("get key %s failed, trying register" % self.addr)
+                self.etcd.setkey("machines/runnodes/"+self.addr, "work")

 if __name__ == '__main__':

[leebaok]

@caodg

init master without stopping workers is much more complex. If we just restart master with init mode but not stopping workers, much data and status will become not consistent.

So, if we want to init our docklet, restarting master as well as restarting workers is necessary.

And I think restarting master in recovery mode without stopping workers is enough for us.

More: there is a way to init master without stopping workers by hand that master calls worker to restart itself.

[caodg]

Agree.

I have fixed the "var not found" bug.

So I will close this issue, and will create a new one in OS Practice milestone, addressing the issue of restarting workers automatically when initialising master.