JSONDecodeError in Reef

[debuggercz]

JSONDecodeError in Reef

[TheJJ]

Can you please send me the output of ceph osd dump --format json? It seems it's not json :) You can send a mail to jj at sft.lol

[TheJJ]

It appears to be caused by inf used as invalid json values...

"read_balance":
{"score_acting":inf,
"score_stable":inf,
"optimal_score":0,
"raw_score_acting":3,
"raw_score_stable":3,
"primary_affinity_weighted":0.99999600648880005,
"average_primary_affinity":1,
"average_primary_affinity_weighted":1}

Hmm, it should probably be fixed in upstream-ceph, but we can hopefully hack around this bug somehow.
Python's json parser doesn't allow inf, it just accepts Infinity, so my first attempt in dc654ec is to just convert the value...

Can you please test and send me placementoptimizer.py gather yourfile.xz also via mail? then i can see if reef produces more funny output which I haven't tested so far (don't have a reef cluster yet :))

[golpa]

Have a look at this bug report we encountered the same problem. Do you have a lot of OSDs and is the pool causing the inf only stored on very few of them? If so, the workaround that worked for us is to increase the number of OSDs the pool is stored on. We did this by increasing the PG count until num OSDs / total OSDs was greater than .005

[TheJJ]

dc654ec seems to be good enough, but it would of course be better if ceph emitted compliant json... but I think its ok for now. Thanks for the state dump again!