how to test fid？

[knazeri]

how to test fid？

[knazeri]

@ljjcoder This is the right script to test FID. However, since it's using Inception model, make sure that you run the script on GPU using --gpu flag:

python ./scripts/fid_score.py --path [path_to_ground_truth] [path_to_output] --gpu 1

The warning message you are receiving is because the pre-trained Inception model is trained using PyTorch < 1.0.0 and can be ignored for now!

Measuring FID is a very CPU intensive process and it requires a lot of RAM, and you normally need 10,000+ images to get an acceptable result! In our experiments, on a Titan V GPU and an Intel Xeon with 8 cores, it takes more than 2 minutes to calculate and it eats up almost 25G of RAM.

[ljjcoder]

@knazeri ，thanks for your reply！when test the fid on places2 ，the score is reasonable（the better result from a human point of view will get lower fid scores）.But when test it on celebA, the score seems to not depict the quality of the generated image（the better result from a human point of view will get higher fid scores). Maybe the the inception model trained on imagenet is not suitalbe for test faces result?

[knazeri]

@ljjcoder The inception model is only used to extract deep features from input images. The Frechet distance measures the distance between two multivariate normals, that means the input distribution has to be diverse (large) enough to be considered normal! We used 10,000 images to evaluate FID on CelebA and Figure 13 in our paper shows the efficacy of FID!
How many images did you test with? What are the FID values you are receiving?

[ljjcoder]

@knazeri , On celebA, the mask hole is 128x128 square, and the image is 256x256, I test 1000 images, I get fid score is 14.24 use CA model. but I use another model to produce the inpainted results which are better than CA, geting higher result 14.86. there are some CA result:


there are some another model result:

[knazeri]

@ljjcoder First off, 1,000 images are not enough to capture an entire distribution. I've seen in papers that the FID is reported for more than 10,000 (sometimes 25,000) images!
Second, please note that FID takes an entire distribution into account and measures the distance between the mean and covariance of two distributions. That means even though some images might not be visually pleasing, the overall quality (of the entire set) might be good!
Having said that, none of these quantitative measures (FID included) are perfect! Still, the human study remains the best qualitative measure to evaluate generative models!

[codinglin]

Hello, when calculating FID, do you compare 10,000 result pictures with 10,000 GT pictures, or compare 10,000 result pictures with all GT pictures?