CPU vs GPU Inference

[creativeikep]

CPU vs GPU Inference

[BlueprintBen]

My apologies - I think you're absolutely right that Deep Profiling used with Barracuda produces flawed results and that a non-deep-profiling build appears to match your numbers more closely. Will see if the code change makes a difference but I'm much happier with the performance now.

[creativeIKEP]

Hello, @BlueprintBen !

Currently, BlazePoseBarracuda is optimized for GPU inference.
In my environment, BlazePoseBarracuda is running at about 200 FPS or more performance on the high performance GPU (ex: RTX2080 super).
On the other hand, BlazePoseBarracuda is running at about 20 FPS or less performance on the not high performance computer (ex: Macbook Pro Early 2015).
Thus, depending on the GPU performance of the computer, there are cases where real-time operation is difficult.

Thanks.

[BlueprintBen]

Is there anything that I have to enable in the example scene to utilize the GPU rather than the CPU? I'm using an RTX 6000 but am seeing ~30ms of Barracuda.Execute.

Edit: Actually taking ~45ms on Full, ~30ms on Lite (had to count the PoseDetector and the PoseLandmarker). My CPU is an i7-7700k.

Edit 2: Numbers only drop a few ms if in a Standalone build with Deep Profiling. When running in Editor or Standalone, my CPU usage is 30% while my GPU usage is 15%.

Edit 3: On my laptop I'm seeing (i7-6700HQ / GTX960M), I'm seeing 70ms of CPU time per frame (Full model) used by Barracuda.Execute.

[creativeIKEP]

FPS drops when using Deep Profile not only in Standalone build but also in Unity Editor.
Therefore, try to check the FPS from the Stats in the Game Window without using the Deep Profile.

Also, in PoseDetectionBarracuda and PoseLandmarkBarracuda, the device used for inference is not explicitly specified, but is automatically set by Unity Barracuda.
Therefore, can you please clone PoseDetectionBarracuda and PoseLandmarkBarracuda to your local PC and explicitly specify GPU inference?
The changes in the source code are as follows.
Change lines:

• https://github.com/creativeIKEP/PoseDetectionBarracuda/blob/33279c9945e16244a62edcdc8804ee284f58e3af/Packages/PoseDetectionBarracuda/Script/PoseDetecter.cs#L44
• https://github.com/creativeIKEP/PoseLandmarkBarracuda/blob/e57ab453b61efebdadeb94589a13336734f78af2/Packages/PoseLandmarkBarracuda/Script/PoseLandmarker.cs#L152

Code change:

woker = model.CreateWorker();

↓

woker = model.CreateWorker(WorkerFactory.Device.GPU);

Reference: https://docs.unity3d.com/Packages/[email protected]/api/Unity.Barracuda.ModelExtensions.html#Unity_Barracuda_ModelExtensions_CreateWorker_Unity_Barracuda_Model_Unity_Barracuda_WorkerFactory_Device_System_Boolean_