Sometimes script dies on AssertationError

[thejj]

Sometimes script dies on AssertationError

[TheJJ]

sure thing :) maybe i did fix something meanwhile

[TheJJ]

Thanks for the report! I'm not sure why this happens, but I guess it's because you have racks (and i haven't tested the script in an environment where there's more than hosts).
Can you run again with debug loggin, i.e. -vv, and send me the output for the whole movement calculation (starting at the last time it prints TRY-0 moving)?
The error basically means that we used an item (a bucket like rack or host, or an osd) not as often as a layer in the crush hierarchy should be used.

[Dragonn]

Here you are, as requested:

[2021-12-06 13:56:18,252] TRY-0 moving pg 8.d9 (1/100) with 13.2G from osd.23
[2021-12-06 13:56:18,252]   OK => taking pg 8.d9 from source osd.23 since it has too many of pool=8 (28 > 25.611671723449557)
[2021-12-06 13:56:18,253] prepare crush check for rule
{'name': 'replicated_rule',
 'steps': [{'item': -1, 'item_name': 'default', 'op': 'take'},
           {'num': 0, 'op': 'chooseleaf_firstn', 'type': 'rack'},
           {'op': 'emit'}]}
[2021-12-06 13:56:18,253] allowed reuses per rule step, starting at root: [3, 1, 1]
[2021-12-06 13:56:18,253] processing crush step {'op': 'take', 'item': -1, 'item_name': 'default'} with tree_depth=0, rule_depth=0
[2021-12-06 13:56:18,253] processing crush step {'op': 'chooseleaf_firstn', 'num': 0, 'type': 'rack'} with tree_depth=1, rule_depth=1
[2021-12-06 13:56:18,253] processing crush step {'op': 'emit'} with tree_depth=4, rule_depth=2
Traceback (most recent call last):
  File "./placementoptimizer.py", line 1917, in <module>
    try_pg_move.prepare_crush_check()
  File "./placementoptimizer.py", line 984, in prepare_crush_check
    assert reuses == uses
AssertionError

Also I tried to add debug code print(idx, reuses, item, uses) on line 984 it outputed following right before Traceback:

0 3 -1 3
1 1 -18 2

Not sure if it helps

[TheJJ]

I think the assertion triggers because the crush map itself fails to validate. Meanwhile I've created support for collecting the cluster state which you can send me then. See #5 :)

[Dragonn]

Thanks for info, I am currently unable to replicate this issue with latest version of script. I tried to generate more PG moves and it didn't crush either.

Are you okay if I close this issue and reopen it later if error reoccurs?