[Frigate 0.13] libyolo_layer.so in the tensorrt container is compiled only for "modern" GPUs

[jaharkes]

[Frigate 0.13] libyolo_layer.so in the tensorrt container is compiled only for "modern" GPUs

[blakeblackshear]

Can a patched file be mounted over the old one for now? There are a relatively small number of downloads of the tensorrt builds and an even smaller number with this issue. I would target dev for now. We could always add it to master later.

[NickM-27]

@NateMeyer we can defer to @blakeblackshear but currently I don't believe another bug fix update is planned so probably dev is my guess.

[NateMeyer]

@jaharkes Can you proof read this gist and make sure this works for you? Thanks!

https://gist.github.com/NateMeyer/a689b4462e57b3de0ebcc40e6538fc03

[NickM-27]

CC @NateMeyer

In general some issues with older GPUs are not unexpected and we have seen some others with similar issues. The updat was done because 40 series GPUs were not working. I don't know that we would want to use a custom compiled version of the library as that may cause other issues for other users.

[jaharkes]

I'm not an nvidia compiler expert, but from my brief research it looks like it should be possible to build a single binary that at least supports all architectures (although it may not be optimized for any specific one) by setting computes=50 60 70 80 90 in the Makefile that builds libyolo_layer.so.
https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#virtual-architecture-feature-list

https://kaixih.github.io/nvcc-options/

Compatibility:
-arch=compute_Xa is compatible with -code=sm_Xb when a≤b
-arch=compute_X* is incompatible with -code=sm_Y*

[NateMeyer]

Good find. This wasn't an issue before because the libyolo_layer was built as part of the setup instructions for TRT on 0.12, so it always used the user's GPU compute level. I'll take a look at the nvcc flags and see if there is a good way to do that universally for the CI build. One bloated library is sure to be better than making every user download the whole toolchain to recompile it.

[NateMeyer]

I don't know that we would want to use a custom compiled version of the library as that may cause other issues for other users.

We already build libyolo_layer.so as part of the docker build process. It is mostly an extra cuda kernel to transform the output of the yolo model to something more usable.

[NickM-27]

ah good to know, thanks for the clarification

[NateMeyer]

@NickM-27 , if I submitted a PR that adds the computes="50 60 70 80 90" to the libyolo_layer.so script, should that target master or dev? I view it a little more as a bug fix for maxwell cards on 0.13. We'll need a couple people who've had problematic Maxwell cards to test also.

This computes value would only be set for the amd64 trt build, not jetpack. I don't think those have had issues with this? But it wouldn't be hard to also add a unique set of compute flags for each of those either.

[posssson]

@jaharkes Thank you very much for your solution, I was stuck for 4 days :)

[NateMeyer]

Can a patched file be mounted over the old one for now? There are a relatively small number of downloads of the tensorrt builds and an even smaller number with this issue. I would target dev for now. We could always add it to master later.

I can throw together a gist with instructions to rebuild and mount the updated library for v0.13

[jaharkes]

Works like a charm.