Can we detect an image that does not have a true class?

[cleanlab]

Can we detect an image that does not have a true class?

[cgnorthcutt]

Yes. Cleanlab does this automatically. If your classifier is appropriate for the data (reasonably high cross validation accuracy), it will include any image that is not labeled correctly in the error set, regardless of the underlying true class.

This is easy to test. Add an image of the letter A into the MNIST digit dataset and see if it identified the letter as an error.

If your dataset contains lots of birds or if your model isn't very good (like if you use a simple naive Bayes on image data), then your model won't be able to train well enough on cats and dogs to have good predicted probabilities, which will affect cleanlab's ability to find the errors. But for the most part, if you use a reasonable model that has high accuracy (> 90 percent) on the data when it's clean and the data has no more than 40 percent error, cleanlab will find most of the birds in your cat / dog set.

[offchan42]

Thank you. This answered my question!