Reproducing results

[qinliuliuqin]

Reproducing results

[YouHuang67]

Thank you for your interest in our work and for reaching out with your questions.

Regarding the issue with case 008.jpg, it seems that the discrepancy arises from the fact that the DAVIS dataset I utilized was sourced from the FocusCut repository, which can be found here:
https://github.com/frazerlin/focuscut#focuscut
The corresponding DAVIS dataset link is:
https://drive.google.com/file/d/1-ZOxk3AJXb4XYIW-7w1-AXtB9c8b3lvi/view

This version of the dataset has pre-processed ground truths where all objects are merged, which is different from the one provided by the RITM repository (the link that is mistakenly mentioned in our repository). This explains why I didn't encounter the crash issue you experienced with case 008.jpg.

Our code is indeed tailored for the FocusCut dataset format, and it may encounter issues if there are multiple objects as GT in the format provided by other sources.

I apologize for any confusion caused and appreciate your understanding. Should you have any further questions, please do not hesitate to ask.

[YouHuang67]

Thank you for your follow-up and for sharing your test results.

I have noticed slight variations in test outcomes as well. Achieving completely deterministic results in testing can be challenging, and I suspect a couple of factors could be contributing to these minor fluctuations:

The click generation process in our model uses certain approximations, especially during distance transforms and center point identification. For example, in our implementation, we use conditions like dist_map > max_dist / (sfc_inner_k + eps) to identify the center when sfc_inner_k = 1, instead of using a condition like dist_map == max_dist. This method also generates random clicks during training when sfc_inner_k > 1. These approaches might introduce a small degree of randomness. Additionally, not using a fixed random seed during the testing phase could be another factor.

There is also some inherent randomness in PyTorch computations, like in convolution operations which might use approximation for faster processing. Though the effect of this randomness is expected to be very small, it might contribute to the slight variability, as the evaluations were not run in a deterministic mode.

These factors together could be the reason behind the observed variances. However, I believe their impact on the overall results is minimal.

I hope this sheds some light on the issue. If you have more questions, feel free to ask.

[qinliuliuqin]

Thanks, You. The following are my script and reproduced results using your provided model and data.

CUDA_VISIBLE_DEVICES=0,1,2,3 \
    bash tools/dist_clicktest.sh \
    work_dirs/interformer_light_coco_lvis_320k/iter_320000.pth 4 \
    --dataset DAVIS \
    --size_divisor 32

Results:
NoC85: 4.54
NoC90: 5.57
NoC95: 12.41 

I also observed that the result numbers varied slightly for different runs. Is there any uncontrolled randomness in the evaluation pipeline? It is common that results may differ in other environments, but the evaluation pipeline has to be deterministic. Do you have any insights on this? Thank you so much!

[qinliuliuqin]

Thanks for your detailed explanation. Yes, randomness in evaluation can be tolerated if it's small and bounded. Another way to circumvent this issue is to report the mean and std of multiple runs. I may reopen this issue if I have something to discuss.
Thank you again!