Comments (5)
sure thing :) maybe i did fix something meanwhile
from ceph-balancer.
Thanks for the report! I'm not sure why this happens, but I guess it's because you have racks (and i haven't tested the script in an environment where there's more than hosts).
Can you run again with debug loggin, i.e. -vv
, and send me the output for the whole movement calculation (starting at the last time it prints TRY-0 moving
)?
The error basically means that we used an item (a bucket like rack or host, or an osd) not as often as a layer in the crush hierarchy should be used.
from ceph-balancer.
Here you are, as requested:
[2021-12-06 13:56:18,252] TRY-0 moving pg 8.d9 (1/100) with 13.2G from osd.23
[2021-12-06 13:56:18,252] OK => taking pg 8.d9 from source osd.23 since it has too many of pool=8 (28 > 25.611671723449557)
[2021-12-06 13:56:18,253] prepare crush check for rule
{'name': 'replicated_rule',
'steps': [{'item': -1, 'item_name': 'default', 'op': 'take'},
{'num': 0, 'op': 'chooseleaf_firstn', 'type': 'rack'},
{'op': 'emit'}]}
[2021-12-06 13:56:18,253] allowed reuses per rule step, starting at root: [3, 1, 1]
[2021-12-06 13:56:18,253] processing crush step {'op': 'take', 'item': -1, 'item_name': 'default'} with tree_depth=0, rule_depth=0
[2021-12-06 13:56:18,253] processing crush step {'op': 'chooseleaf_firstn', 'num': 0, 'type': 'rack'} with tree_depth=1, rule_depth=1
[2021-12-06 13:56:18,253] processing crush step {'op': 'emit'} with tree_depth=4, rule_depth=2
Traceback (most recent call last):
File "./placementoptimizer.py", line 1917, in <module>
try_pg_move.prepare_crush_check()
File "./placementoptimizer.py", line 984, in prepare_crush_check
assert reuses == uses
AssertionError
Also I tried to add debug code print(idx, reuses, item, uses)
on line 984
it outputed following right before Traceback:
0 3 -1 3
1 1 -18 2
Not sure if it helps
from ceph-balancer.
I think the assertion triggers because the crush map itself fails to validate. Meanwhile I've created support for collecting the cluster state which you can send me then. See #5 :)
from ceph-balancer.
Thanks for info, I am currently unable to replicate this issue with latest version of script. I tried to generate more PG moves and it didn't crush either.
Are you okay if I close this issue and reopen it later if error reoccurs?
from ceph-balancer.
Related Issues (20)
- IndexError: list index out of range HOT 3
- can't run it with python < 3.9 HOT 1
- Add ability to handle invoking the Ceph client via Cephadm? HOT 1
- no trace found for 2147483647 in default~hdd HOT 6
- JSONDecodeError in Reef HOT 4
- Please consider type datacenter as a root bucket HOT 1
- JSON TypeError on show command when json output is used. HOT 4
- ./placementoptimizer.py showremapped balks at cluster topology changes HOT 3
- Feature request: take bluestore_min_alloc_size into account HOT 3
- KeyError: 66 (in used += self.cluster.osd_transfer_remainings[osdid]) HOT 5
- ZeroDivisionError: division by zero (osd_objs_acting) HOT 3
- RuntimeError: pg 18.6a to be moved to osd.117 is misplaced with -198781.0<0 objects already transferred
- ./placementoptimizer.py showremapped --by-osd throws KeyError if there is recovery
- key error - on healthy cluster
- it's breaking CRUSH rule HOT 3
- Sorting in heterogeneous clusters
- OSD utilization is very different from ceph osd df HOT 2
- crush root X not known HOT 8
- Failing when using --source-osds
- The "balance" subcommand does not balance if there was a recent "upmap-remapped.py" run
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ceph-balancer.