can you all get paper's result?

[zkm98]

can you all get paper's result?

[ma-xu]

@zkm98

I suppose you are talking about the performance on ModelNet40.
The performance on ModelNet40 of almost all methods are not stable, see (CVMI-Lab/PAConv#9 (comment)).
If you run the same codes for several times, you will get different results.

Here is the log we got the testing 94.5 acc with voting:
https://drive.google.com/file/d/1z-QbWLMlGxfMVPImcLeaDOVOJGvm64ny/view?usp=sharing

Here is the log we got the testing 94.1 acc without voting: https://drive.google.com/file/d/1T8ZljhO2evsxQ0aL7l0xbd_dVHgargzT/view?usp=sharing

Also, the randomness of ModelNet40 is our motivation to experiment on ScanObjectNN benchmark, and to report the mean/std results of several runs.

Please let me know if you have any questions or concerns.

Best,
Xu

[ma-xu]

@coallar

COULD YOU PLEASE READ WHAT I POSTED PREVIOUSLY IN THIS ISSUE?

Again, the performance on ModelNet40 is not deterministic，the perofmrance has a large variance. This also motivates us to consider ScanObjectNN as a benchmark. Please see above and the paper.

[zkm98]

It seems it's hard to train for me...

[ma-xu]

It seems it's hard to train for me...

@zkm98 Could you please clarify what do you mean by "hard to train"?
Empirically, pointmlp is easy to train in terms of deployment, speed, and loss curve, etc.

[zkm98]

about hard to train, I mean, it is hard to get the results in your papers, sorry that I dont clearly define it. For most experiments I trained, I just got the result of 92.8% using pointmlp Elite, I was trying to train from scratch, although maybe I can get the result as you say using the pretrained models. It cant talk me into it.

[ma-xu]

@zkm98 I uploaded checkpoint/logs for PointMLP and PointMLP-elite (also, the bug-fixed results and checkpoints are also uploaded). Please see the README file.

As I mentioned previously, the results on ModelNet40 are not stable (but always greater than 93.x for both pointmlp and pointmlp-elite). The randomness is the motivation for us to consider ScanObjectNN benchmark.

[le-cheng]

Thank you very much for your wonderful work. I tried testing with your pre-trained model and with the best parameters I only got 93.6. I would like to know what this deals with, theoretically with the same model, using the same parameters, with only validation I should get 94.1.

[ma-xu]

@le-cheng Thanks for your interest. I will double-check and reply to you by Wednesday.

[ma-xu]

@le-cheng
I just double-checked the testing result, there is no problem, and I can get the same results as I claimed. See screenshot.

For your convenience, I uploaded a simple test code.

[le-cheng]

@ma-xu
OK, my problem has been solved！

I would like to ask if your training method is just multiple training sessions, or is there some other parameter adjustment. Am I using your code under what circumstances it is possible to get the same results as you.

[ma-xu]

@le-cheng Sry that I did not understand you question. If you are talking about the unstable peroformance?
I trained with the default parameters (see log files).

The performance on ModelNet40 of almost all methods are not stable, see (CVMI-Lab/PAConv#9 (comment)).
If you run the same codes for several times, you will get different results.

Also, the randomness of ModelNet40 is our motivation to experiment on ScanObjectNN benchmark, and to report the mean/std results of several runs.

[le-cheng]

Well, thanks.

[coallar]

only 93.071 can I get！

[yanghu819]

For the experiment itself is unstable, could you produce multi-times running logs or the exact random seed to reproduce the best checkpoint.

[ma-xu]

@yanghu819
Sure, I upload the 6 runs of PointMLP (last_checkpoints is removed to save space, model names is our fixed developing name, ignore it). Here is the link: https://drive.google.com/drive/folders/1SUuMDjaWhxURaUO7VtPpJ_vnxvaBA9eO?usp=sharing
The folder name pattern is [modelname]-[time]-[randgeneratedseed]. That is, we trained the model (bug fixed version, we can also provide unfixed version for the submission if needed) on Feb/09/2022, with 6 randomly generated seeds.

Train with the exactly same seed, one propobaly still cannot get the deterministic result, we tested that. You can also have a try, the seed is provided in the folder name.

Btw, you can easily get the same result using our pre-triained checkpoint.

Besides, a good choice is to report the mean and std of several runs as in Table 3 on ScanObjectNN benchmark. You can easily get a better PointMLP result than we reported.

[ma-xu]

@zkm98 @yanghu819 @le-cheng @coallar
I will close this issue since there is no further discussion. Feel free to reopen it if necessary.