Low training accuracy with pointnet2

[horizon-research]

Low training accuracy with pointnet2

[Boyuan-Tian]

Hi Dohee,

Thank you for your interest in our work.

I reproduced the training procedure and obtained the training log shown in the following figure, which matches well with our previous result.

My test experiment is:

• GTX 1080
• Ubuntu 18.04
• CUDA 10.1
• CUDNN 7.6.5
• Tensorflow-GPU 1.14.0
• g++ 7.5.0

The original model was developed on CUDA 10.2, so that wouldn’t be a problem, then the only difference should be the g++ version.

If there was nothing changed in the code, and the only thing you did the following commands:

python launch.py --download pointnet2
python launch.py --compile pointnet2
python launch.py --train pointnet2

I would suggest beginning with the compiled TensorFlow operators.

We provided the pre-trained model in the directory named “log*,” you can run our evaluation script with our checkpoints to see if you can get anticipated results.

If the evaluation failed, then there must be something wrong with the compiled operators, given that both the code and model are not changed and passed my test.

If this is the case, I would suggest you upgrade the compiler and recompile the operators.

Please let me know if there are further details and keep me posted.

Best regards,
Boyuan Tian

[kdheejb7]

Thank you for your rapid response!

The reason that I used low version g++ is that if I use g++ 7, I have some problems with 'WithRank' during compile step

I got a hint from your test experiment and just solved the problem!
I experimented with your code using a multi-GPU machine, and the machine has various GPUs like GTX 1080, RTX 2080ti, and RTX 3090.
Because RTX 3090 is the most powerful of them, so RTX 3090 is set as a default.
However, RTX 3090 does not support CUDA 10.x version.

I solved the low accuracy problem by changing the GPU used for training to a different one(RTX 2080TI).

Thank you so much :)

[Boyuan-Tian]

Hi Dohee,

Glad to hear that. I didn’t realize that 3090 could work with CUDA 10.

Before I moved to GTX 1080 today, I initially reproduced the experiment on 3090 + CUDA 10.
The GPU was not recognized, and the computations were allocated to CPUs.

Since the issue is resolved, I will close this.

Best regards,
Boyuan Tian