Does ppl.nn support fp16 or int8?

[openppl-public]

Does ppl.nn support fp16 or int8?

[openppl-public]

Currently PPLNN CUDA supports fp16 only.

Just want to know if ppl.nn support half-precision or int8 inference.

Looking forward to your reply.

[zerollzeng]

Can we test the fp16 performance with pplnn-build/tools/pplnn?

[ltj2013]

Can we test the fp16 performance with pplnn-build/tools/pplnn?

Yes, you can test the fp16 performance with pplnn in tesla T4.

[openppl-public]

Can we test the fp16 performance with pplnn-build/tools/pplnn?

Benchmark instructions are listed here: Benchmark（中文版）

[zerollzeng]

Yeah, I read it before, eg running with:

./pplnn --onnx-model model.onnx --inputs input.bin --in-shapes 1_3_224_224 --dims 1_3_224_224 --warmuptimes 100 --runningtimes 100

will run the model on gpu with fp32 precision right? looks like we can not specify precision thought args.

[ltj2013]

Yeah, I read it before, eg running with:

./pplnn --onnx-model model.onnx --inputs input.bin --in-shapes 1_3_224_224 --dims 1_3_224_224 --warmuptimes 100 --runningtimes 100

will run the model on gpu with fp32 precision right? looks like we can not specify precision thought args.

CUDA only support fp16 for conv and gemm op right now. PPL will convert data type before excute. So the results will show the fp16 performance for cuda.