Priam never marks a node as dead

[netflix]

Priam never marks a node as dead

[matthewbogner]

Not trying to debate the test you ran or that you didn't see what you saw :)

But here is where Priam marks an instance in SimpleDB with the "-dead" suffix.
https://github.com/Netflix/Priam/blob/master/priam/src/main/java/com/netflix/priam/identity/InstanceIdentity.java#L119

[sagarl]

Looks like the issue is resolved, hence closing it.

[aryanet]

I think I should rephrase this. I figured Priam marks the node as Dead in Simple DB, but the problem is that it does not automatically remove the dead token from the ring. What happened is that I killed a node to see what happens. A new node came up by auto scaling group, Priam picked the dead node's token from Simple DB but couldn't gracefully start the cassandra service as cassandra complained the token still exist and must run decommission or removetoken. I had to manually run removetoken from a healthy node nad restart priam in the new node so that it joins the ring.

[sagarl]

Hi Arya,

I ran the test in following order:

1. Created a New 9 node cluster ( 3 Nodes in each zone)

2. Terminated one of the instances (out of 9 running instances)
   a. Ran "ring" command -- which displays the Terminated node as Down

3. ASG brought up another instance
   a. After the node is up & running, again ran "ring" command -- which displayed the new node using the same
   token as that of dead node.

Let me know if this works for you.

[sagarl]

Hopefully the above explanation has resolved the issue, hence closing it.

[dehora]

Created #161 instead of commenting on closed ticket.

[aryanet]

Sagar, this is still an issue. I am thinking that Netflix tooling maybe hiding the nodetool removetoken step for you. I cannot find where this is done in Priam and as you see dehora found the same issue. The replacement mechanism is not by the docs and it has hit me in the past. Please take a look at #159. I think a replacement node, should never claim the same token.