Doubling the Size of Cluster Doesn't Create a Balanced Ring

[aryanet]

Doubling the Size of Cluster Doesn't Create a Balanced Ring

[jasobrown]

Hi Arya,

I've been [poking around with the double ring functionality myself for the
last few days, and things have seemed sane for me. I started with a three
node cluster, went to 6, then 24 and the distribution was legit. I'll try
to reproduce your case (starting with 9 nodes and doubling to 18). In the
mean time, can you share the steps you went through to double the ring?

Thanks,

-Jason

On Sun, Jul 29, 2012 at 11:14 PM, Arya Goudarzi <
[email protected]

wrote:

I have tried to double the size of cluster by doubling the number of
minimum instance in 3 participating auto scaling group each from 3 to 6 to
upgrade the ring from 9 nodes to 18 nodes. However I end up with an
imbalanced ring. Please advice what I am doing wrong. From the code of
PriamInstanceFactory.java I can tell the calculation is done based on the
maximum number of instances in ASG, but I cannot figure why scale up,
creates an imbalance. I suppose there is a right way to do this which is
not clear as Netflix always mentioned in their blog posts that they
benchmark doubling the size of cluster with Priam.

Old Ring:
Address DC Rack Status State Load
Effective-Ownership Token

             132332031580364958013534569558607324498

****.39.144 us-east 1a Up Normal 12.13 MB
33.33% 1808575600
***__157.165 us-east 1b Up Normal 12.15 MB
33.33% 9452287970026068429538183541579914807
*85.183 us-east 1c Up Normal 12.15 MB
33.33% 18904575940052136859076367081351254014
***__60.61 us-east 1a Up Normal 12.16 MB
33.33% 56713727820156410577229101240436610842
*67.5 us-east 1b Up Normal 12.14 MB
33.33% 66166015790182479006767284780207950049
***__110.200 us-east 1c Up Normal 12.17 MB
33.33% 75618303760208547436305468319979289256
***77.115 us-east 1a Up Normal 12.13 MB
33.33% 113427455640312821154458202479064646084
***__234.163 us-east 1b Up Normal 12.15 MB
33.33% 122879743610338889583996386018835985291
****26.159 us-east 1c Up Normal 12.15 MB
33.33% 132332031580364958013534569558607324498

New Ring:

Address DC Rack Status State Load
Effective-Ownership Token

             151236607520417094872610936638150002896

***39.144 us-east 1a Up Normal 18.71 MB
16.67% 1808575600
***__157.165 us-east 1b Up Normal 18.7 MB
19.44% 9452287970026068429538183541579914807
*.85.183 us-east 1c Up Normal 18.74 MB
22.22% 18904575940052136859076367081351254014
***__60.61 us-east 1a Up Normal 12.17 MB
33.33% 56713727820156410577229101240436610842
*67.5 us-east 1b Up Normal 12.15 MB
33.33% 66166015790182479006767284780207950049
***__110.200 us-east 1c Up Normal 12.18 MB
33.33% 75618303760208547436305468319979289256
*252.67 us-east 1a Up Normal 9.44 MB
16.67% 85070591730234615865843651859750628454
***__60.86 us-east 1b Up Normal 7.87 MB
13.89% 89796735715247650080612743629636298057
*123.155 us-east 1c Up Normal 6.29 MB
11.11% 94522879700260684295381835399521967660
***__38.58 us-east 1a Up Normal 18.75 MB
16.67% 113427455640312821154458202479064646072
*.77.115 us-east 1a Up Normal 18.78 MB
0.00% 113427455640312821154458202479064646084
***__24.218 us-east 1b Up Normal 9.38 MB
16.67% 118153599625325855369227294248950315675
*185.221 us-east 1c Up Normal 9.4 MB
16.67% 122879743610338889583996386018835985278
***__234.163 us-east 1b Up Normal 18.74 MB
2.78% 122879743610338889583996386018835985291
*26.159 us-east 1c Up Normal 18.71 MB
5.56% 132332031580364958013534569558607324498
***__.145.200 us-east 1a Up Normal 9.37 MB
16.67% 141784319550391026443072753098378663690
*.7.166 us-east 1b Up Normal 7.82 MB
13.89% 146510463535404060657841844868264333293
***.108.241 us-east 1c Up Normal 6.3 MB
11.11% 151236607520417094872610936638150002896

---

Reply to this email directly or view it on GitHub:
#45

[jasobrown]

Also, looking at your original 9 node cluster, the tokens are not equally
spaced around the token range. I just brought up a 9 node cluster and the
nodetool ring output looks lie:

Address DC Rack Status State Load Owns
Token

151236607520417094872610936638150002911
.185.179 us-east 1e Up Normal 11.29 KB 11.11%
1808575600
.91.185 us-east 1c Up Normal 11.29 KB 11.11%
18904575940052136859076367081351254013
.114.206 us-east 1d Up Normal 11.29 KB 11.11%
37809151880104273718152734160893932427
.212.118 us-east 1e Up Normal 11.29 KB 11.11%
56713727820156410577229101240436610841
.159.98 us-east 1c Up Normal 11.29 KB 11.11%
75618303760208547436305468319979289255
.151.139 us-east 1d Up Normal 11.29 KB 11.11%
94522879700260684295381835399521967669
.184.164 us-east 1e Up Normal 11.29 KB 11.11%
113427455640312821154458202479064646083
.78.202 us-east 1c Up Normal 11.29 KB 11.11%
132332031580364958013534569558607324497
.162.21 us-east 1d Up Normal 11.32 KB 11.11%
151236607520417094872610936638150002911

Each node has a token range size of 18904575940052136859076367079542678414
(plus or minus < 10). The original tokens your cluster began with has gaps.
How did you create the first 9 nodes for the cluster?

On Mon, Jul 30, 2012 at 10:26 AM, Jason Brown [email protected] wrote:

Hi Arya,

I've been [poking around with the double ring functionality myself for the
last few days, and things have seemed sane for me. I started with a three
node cluster, went to 6, then 24 and the distribution was legit. I'll try
to reproduce your case (starting with 9 nodes and doubling to 18). In the
mean time, can you share the steps you went through to double the ring?

Thanks,

-Jason

On Sun, Jul 29, 2012 at 11:14 PM, Arya Goudarzi <
[email protected]

wrote:

I have tried to double the size of cluster by doubling the number of
minimum instance in 3 participating auto scaling group each from 3 to 6 to
upgrade the ring from 9 nodes to 18 nodes. However I end up with an
imbalanced ring. Please advice what I am doing wrong. From the code of
PriamInstanceFactory.java I can tell the calculation is done based on the
maximum number of instances in ASG, but I cannot figure why scale up,
creates an imbalance. I suppose there is a right way to do this which is
not clear as Netflix always mentioned in their blog posts that they
benchmark doubling the size of cluster with Priam.

Old Ring:
Address DC Rack Status State Load
Effective-Ownership Token

             132332031580364958013534569558607324498

****.39.144 us-east 1a Up Normal 12.13 MB
33.33% 1808575600
***__157.165 us-east 1b Up Normal 12.15 MB
33.33% 9452287970026068429538183541579914807
*85.183 us-east 1c Up Normal 12.15 MB
33.33% 18904575940052136859076367081351254014
***__60.61 us-east 1a Up Normal 12.16 MB
33.33% 56713727820156410577229101240436610842
*67.5 us-east 1b Up Normal 12.14 MB
33.33% 66166015790182479006767284780207950049
***__110.200 us-east 1c Up Normal 12.17 MB
33.33% 75618303760208547436305468319979289256
***77.115 us-east 1a Up Normal 12.13 MB
33.33% 113427455640312821154458202479064646084
***__234.163 us-east 1b Up Normal 12.15 MB
33.33% 122879743610338889583996386018835985291
****26.159 us-east 1c Up Normal 12.15 MB
33.33% 132332031580364958013534569558607324498

New Ring:

Address DC Rack Status State Load
Effective-Ownership Token

             151236607520417094872610936638150002896

***39.144 us-east 1a Up Normal 18.71 MB
16.67% 1808575600
***__157.165 us-east 1b Up Normal 18.7 MB
19.44% 9452287970026068429538183541579914807
*.85.183 us-east 1c Up Normal 18.74 MB
22.22% 18904575940052136859076367081351254014
***__60.61 us-east 1a Up Normal 12.17 MB
33.33% 56713727820156410577229101240436610842
*67.5 us-east 1b Up Normal 12.15 MB
33.33% 66166015790182479006767284780207950049
***__110.200 us-east 1c Up Normal 12.18 MB
33.33% 75618303760208547436305468319979289256
*252.67 us-east 1a Up Normal 9.44 MB
16.67% 85070591730234615865843651859750628454
***__60.86 us-east 1b Up Normal 7.87 MB
13.89% 89796735715247650080612743629636298057
*123.155 us-east 1c Up Normal 6.29 MB
11.11% 94522879700260684295381835399521967660
***__38.58 us-east 1a Up Normal 18.75 MB
16.67% 113427455640312821154458202479064646072
*.77.115 us-east 1a Up Normal 18.78 MB
0.00% 113427455640312821154458202479064646084
***__24.218 us-east 1b Up Normal 9.38 MB
16.67% 118153599625325855369227294248950315675
*185.221 us-east 1c Up Normal 9.4 MB
16.67% 122879743610338889583996386018835985278
***__234.163 us-east 1b Up Normal 18.74 MB
2.78% 122879743610338889583996386018835985291
*26.159 us-east 1c Up Normal 18.71 MB
5.56% 132332031580364958013534569558607324498
***__.145.200 us-east 1a Up Normal 9.37 MB
16.67% 141784319550391026443072753098378663690
*.7.166 us-east 1b Up Normal 7.82 MB
13.89% 146510463535404060657841844868264333293
***.108.241 us-east 1c Up Normal 6.3 MB
11.11% 151236607520417094872610936638150002896

---

Reply to this email directly or view it on GitHub:
#45

[aryanet]

I have multi-region enabled and I supposed the gap is due to the offset hash added to the region tokens. I learned that from Vijay in Issue #29 where he linked this:
http://www.datastax.com/dev/blog/deploying-cassandra-across-multiple-data-centers

I haven't spawn the other region just yet. Here are the steps I use to deploy:

1. I have 3 auto-scaling groups (1 for each zone) in each region which use a launch config that has Priam and Cassandra 1.1.2 cooked in;
2. The min and max instances in my ASGs initially are 3.
3. Priam configuration multiregion.enable is true;
4. Priam configuration zones.available is: us-east-1a,us-east-1b,us-east-1c,us-west-2a,us-west-2b,us-west-2c
5. So I fired up the auto-scaling groups in us-east region and they created 9 instances as expected.
6. I loaded some garbage data to them using the cassandra-stress tool with NetworkTopologyStrategy and StrategyOption of us-east:3. So, each node started to have even amount of data resulting in the first ring.
7. I changed the min/max number of instances in my ASGs from 3 to 6, and I ended up with the final ring.

I supposed something has to do with the other region not being up yet but I'd like to understand this well. I am going to kill my cluster and start small like you did and do both regions first. Then I scale up both regions and will report back.

[aryanet]

OK, here is an original 6 node I made with 1 instance in each zone in 2 regions:

Address DC Rack Status State Load Effective-Ownership Token
141784319550391026443072753098537460998
.174.52 us-east 1a Up Normal 50.2 MB 100.00% 1808575600
.155.60 us-east 1b Up Normal 50.1 MB 100.00% 28356863910078205288614550621122593221
.65.40 us-east 1c Up Normal 50.09 MB 100.00% 56713727820156410577229101240436610842
.87.16 us-west-2 2a Up Normal 50.07 MB 100.00% 85070591730234615865843651859909425756
.226.195 us-west-2 2b Up Normal 49.92 MB 100.00% 113427455640312821154458202479223443377
.60.231 us-west-2 2c Up Normal 50.07 MB 100.00% 141784319550391026443072753098537460998

I loaded it with 1M random rows using the stress tool with NetworkTopologyStrategy and StrategyOptions us-east:3,us-west-2:3, so as you see all nodes have the full data set.

In the next step I doubled the number of nodes in each ASG in each region. I ended up with this ring:

Address DC Rack Status State Load Effective-Ownership Token
155962751505430129087380028408194469803
.174.52 us-east 1a Up Normal 56.16 MB 50.00% 1808575600
.155.60 us-east 1b Up Normal 56.14 MB 58.33% 28356863910078205288614550621122593221
.65.40 us-east 1c Up Normal 56.15 MB 66.67% 56713727820156410577229101240436610842
145.103 us-east 1a Up Normal 28.11 MB 50.00% 85070591730234615865843651859750628460
.87.16 us-west-2 2a Up Normal 56.14 MB 75.00% 85070591730234615865843651859909425756
.7.119 us-east 1b Up Normal 23.46 MB 41.67% 99249023685273718510150927169407637270
.122.145 us-east 1c Up Normal 18.76 MB 33.33% 113427455640312821154458202479064646080
.226.195 us-west-2 2b Up Normal 56.15 MB 83.33% 113427455640312821154458202479223443377
.90.245 us-west-2 2a Up Normal 14.07 MB 25.00% 127605887595351923798765477788880452183
.236.245 us-west-2 2b Up Normal 9.38 MB 16.67% 141784319550391026443072753098537460993
.60.231 us-west-2 2c Up Normal 56.16 MB 91.67% 141784319550391026443072753098537460998
.65.203 us-west-2 2c Up Normal 4.71 MB 8.33% 155962751505430129087380028408194469803

What am I doing wrong in my steps that I end up with an imbalanced ring every time I scale up. Please advice.

[aryanet]

Jason,

Please read my comment here: f747696#commitcomment-1655561

/double_ring api is not documented anywhere, so I figured it out from code. I am running from master on cassandra-1.1.2 and I have a feeling that does not work since i have found the exception. THough, haven't had a chance to find a way to fix it. Please advice.

[jasobrown]

Hey Arya,

Sorry for the long delay on this one. Please make sure (and I'm not sure if this is documented nor how well - I'll have to double check), that the property Priam configuration zones.available is different for each region and only lists the zones for that region. Thus for instances running in us-east-1, you should only have entries like 'us-east-1a,us-east-1b,us-east-1c' , and no entries for zones for other regions.

Let me know if this helps, or if this is still an issue.

-Jason

[sagarl]

Looks like this issue is resolved, hence closing it.