BUG: None value in MultiIndex can't be found

[MentalMegalodon]

BUG: None value in MultiIndex can't be found

[chaoyihu]

Thanks for raising the issue. The problem is with None value handling in MultiIndex::insert.

In the example code, df["one", None, "yes"] = 1 calls MultiIndex::insert, and the MultiIndex object gets updated to:

MultiIndex::insert 1 ('one', None, 'yes')
# before
levels [['one'], ['a'], ['yes']]
codes [[0], [0], [0]]
# updated
new_levels [['one'], ['a', None], ['yes']]
new_codes [[0, 0], [0, 1], [0, 0]]

While if the None value is inserted correctly, for example, using df.concat as you have mentioned, the MultiIndex object should be updated to:

# df
MultiIndex([('one', 'a', 'yes')],
           )
levels: [['one'], ['a'], ['yes']], codes: [[0], [0], [0]]

# df_add
MultiIndex([('one', nan, 'yes')],
           )
levels: [['one'], [], ['yes']], codes: [[0], [-1], [0]]

# df_concat
MultiIndex([('one', 'a', 'yes'),
            ('one', nan, 'yes')],
           )
levels: [['one'], ['a'], ['yes']], codes: [[0, 0], [0, -1], [0, 0]]

, where if key is NA value, location of index unify as -1.

I have submitted PR #59069 which will hopefully resolve this issue.