tfdv.validate_statistics drops existing column and adds it as a new column

[tensorflow]

tfdv.validate_statistics drops existing column and adds it as a new column

[mzinkevi]

That looks really weird. No, in general it shouldn't have a new column and a column dropped at the same time. I have never seen such errors before.

If you can produce a small statistics object and schema object that produces these kinds of errors, that would help in the diagnosis of what is wrong.

[sudroy]

One possibility for why this could happen is if your eval data has whitespace character that causes our data validation code to treat is as a separate feature. Can you please check if the edits that you made to your eval data inadvertently also changed the feature column name for the feature "f_key" in the CSV header to have a whitespace character ?

It will also be useful if you provide us the snippets of the statistics for this particular feature "f_key" from your eval statistics (i.e., the FeatureNameStatistics with name field set to "f_key"), and also make sure that there is a single instance of FeatureNameStatistics with this name.

[hanneshapke]

Thank you @mzinkevi and @sudroy for your comments. I tried to reproduce the odd behavior with a smaller dataset, but I didn't see the same errors. I checked the data source for whitespaces, but the file looks good.

I also noticed that validate_statistics reported a bunch of type changes for a large number of columns, when I only added a few new categorical tokens to some of the columns.

I think more and more that the odd behavior is related to my initial csv data, rather than related to tfdv. Let me test a few more cases in the coming days and report back if I encounter similar errors.|
Thank you for your help investigating the behavior.

[paulgc]

@hanneshapke Closing this issue for now. Feel free to reopen this in case you are able to reproduce the odd behavior.

[jackhawa]

I get the same issue even a year after this issue has been closed.

@hanneshapke did you end up solving your issue? If so it would be great if you can share that. Thanks.

[npoly]

@jackhawa Can you provide a minimal repro so that we can investigate further?