validation errors Extra(nan) or Invalid(nan)

[shawnbrown]

validation errors Extra(nan) or Invalid(nan)

[shawnbrown]

I've been looking at this closely and discovered a handful of un-handled corner cases related to NaN values. Until I get this sorted, NaN values will have to be handled using a workaround—e.g., using the fillna() method to replace them with a proxy value.

As a stopgap, you could do the following:

NAN = object()

# Include NAN in the validation set.
data = df['A'].fillna(NAN)
validate.superset(data, {'x', 'y', 'z', NAN})

# Accept NAN as a difference.
data = df['A'].fillna(NAN)
with accepted(Invalid(NAN)):
    validate(data, str)

Going forward, I will file a related issue/bug for this with the goal of allowing the use of NaN values directly:

# Include NaN in the validation set.
validate.superset(data, {'x', 'y', 'z', np.nan})

# Accept NAN as a difference.
with accepted(Invalid(np.nan)):
    validate(data, str)

I'll post a follow-up to this issue once I have patched this behavior.

[upretip]

Thanks. I will follow this.

[shawnbrown]

This is done:
ce71b34: Update predicate handling to better support NaN values.
bee6aa8: Add NaN handling idioms to test_usecases.py.
32d3bb9: Add test_numbers_equal() to verify numeric comparison.
e8435b1: Update difference behavior to support tuples containing NaNs.
c962e04: Change RequiredInterval to fail if arguments are NaN.
c78f390: Fix RequiredInterval to properly handle NaN differences.
4995510: Update NaN use cases to highlight recommended pattern.
fa2646e: Add how-to documentation for working with NaN values.

[shawnbrown]

@upretip, I've just pushed some new "how to" docs that give detail regarding NaN validation and behavior. You can view it in the latest docs here:

How to Deal With NaN Values
https://datatest.readthedocs.io/en/latest/how-to/nan-values.html

[upretip]

Thanks for the help. Closing this issue now!