Comments (11)
hi,
I download the visa and trained first, then tested. Reasults are really good.
from inctrl.
Same lower reproduction twice @Diana1026, please check the few-shot sample set for selection. BTW, is the performance high affected by few-shot sample?
![image](https://private-user-images.githubusercontent.com/32570671/320272479-28132637-539d-47a7-95ec-ce78939057b4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIyNDc3MTgsIm5iZiI6MTcyMjI0NzQxOCwicGF0aCI6Ii8zMjU3MDY3MS8zMjAyNzI0NzktMjgxMzI2MzctNTM5ZC00N2E3LTk1ZWMtY2U3ODkzOTA1N2I0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzI5VDEwMDMzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWM1YTU0YzU1YTc4ODcwZThiNThhZTE2YmUwNTg4YzMxN2E1NjI2MTZlYTE0ZjBlMWQyMjU1NThjOWZkODkxOTMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.uFinWo-BTzr7rK1HorXgISvZm29D-fHofjmCnRdTn7A)
from inctrl.
Could you please check if your pre-trained model and few-shot samples are correct? I downloaded the repository and tested the model again, and there was no problem. In addition, the test data statistics are provided in the Appendix and you can check your annotation JSON files based on it.
from inctrl.
thanks for your reply!
I added the pre-trained model path in _C.TEST_CHECKPOINT_FILE_PATH, and the run command is: python test.py --val_normal_json_path ../../visa/camdle_val_normal.json --val_outlier_json_path ../../visa/candle_val_outlier.json --category candle --few_shot_dir ../../visa/2
i checked the annotation json files, 100 normal images, 100 defect images, it seems everything is ok... 0.0
from inctrl.
I also got lower performance (for visa candle, AUC-ROC: 0.8850, AUC-PR: 0.8706), about three points lower than the paper(AUPOC: 0.916±0.006, AUPRC: 0.920±0.008)
from inctrl.
Hi, I downloaded the project and run directly, below is my current reproduced result:
from inctrl.
Since our report's results are based on averages across various independent runs, there might be some deviation. I will release another few-shot sample set for selection later. However, for the current few-shot file, the results should be like the given image.
from inctrl.
Same lower reproduction twice @Diana1026, please check the few-shot sample set for selection. BTW, is the performance high affected by few-shot sample?
from inctrl.
The results shouldn't change when the parameters(pre-trained model) and few-shot samples are fixed, and this is the reason why you got the same results twice. I used the same few-shot samples but the performance is the same as the image I uploaded. Please check if the hardware and the PyTorch version are aligned with us.
Indeed, selecting different few-shot samples will influence the performance as the selected normal samples might not be a good representative of the normal pattern, we clarify the problem in Sec 4.4 Failure Cases of our paper.
from inctrl.
for shot=2 and candle dataset, i got AUC_ROC=08027, AUC_PR=0.8096, with default setting.
and i tried to set batchsize from bs=8 to bs=2, then the performance raised to below
from inctrl.
lower performance for 2-shot candle class of ViSA
from inctrl.
Related Issues (20)
- could you share how to visualize the segment result when inference? HOT 1
- Hi, you can use **_torch.save()_** to generate .pt file for your own few-shot samples. HOT 2
- The performance of using 4-shot or 8-shot on the Visa dataset is similar to that of 2-shot HOT 2
- Can you provide the test set about Visa, ELPV, SDD, AITEX? HOT 3
- the training process HOT 1
- lower performance on 2-shot visa-candle with the default setting (e.g., pre-trained model and few-shot prompts)
- Is the model trained on the full dataset of MVTec available?
- Few-shot Normal Images for Inference. HOT 5
- License HOT 1
- 在测试过程中,可以不传入--few_shot_dir这个参数吗?
- Is it possible to not pass in the --few_shot_dir parameter during testing? HOT 1
- how to test one img? HOT 2
- json generation fails HOT 6
- Permission denied error because of CUDA?
- Few-shot Normal Samples for Inference - MVTec HOT 2
- INT8
- Problems reproducing performance HOT 1
- How is x.pt file generated with the extension python test.py --few_shot_dir HOT 1
- Can you provide code for visual inspection HOT 3
- Guidance on Training and Testing with Custom Dataset Similar to MVTec Format
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from inctrl.