FeatureBinarizer: existence of missing values in binary features throws an error

[trusted-ai]

FeatureBinarizer: existence of missing values in binary features throws an error

[dennislwei]

@rahulnair23 Thanks for raising this issue and clearly pinpointing it. I think a one-line fix would be to use data[c].dropna().unique() instead of data[c].unique() in FeatureBinarizer L70.

However, your issue has made me think that the "NaN indicator" that can be created for ordinal columns should be extended to binary (and unary) columns.

• In the fit() method, lines 90-92 could be copied after line 70 to identify binary columns with NaNs.
• Then in transform(), something like lines 168-175 could be copied after line 136 to create the NaN indicator, but it would have to be modified. Perhaps like:

                if c in NaN:
                    # Ensure that rows corresponding to NaN values are zeroed out
                    indNull = data[c].isnull()
                    A.loc[indNull, (str(c), '', '')] = 0
                    # or maybe A[(str(c), '', '')].fillna(0, inplace=True) works and is easier
                    if self.negations:
                        A.loc[indNull, (str(c), 'not', '')] = 0
                    # Add NaN indicator column
                    A[(str(c), '==', 'NaN')] = indNull.astype(int)
                    if self.negations:
                        A[(str(c), '!=', 'NaN')] = (~indNull).astype(int)

Since you're working with data for which this problem arises, could I ask you to test at least the one-line fix and if it works, submit a pull request? If you're up for it (and if it might help you to encode NaNs), it would be great if you could also try the NaN indicator extension above. Thanks in advance.

[rahulnair23]

Sure, turned in a pull request #80 with the one-liner fix (verified that it works on my data). Thanks!