RuntimeError: CUDA error: device kernel image is invalid

[otamajakusi]

RuntimeError: CUDA error: device kernel image is invalid

[xrstokes]

ok here is my dockerfile for future reference. works with latest builds. and has ffmpeg support.
FROM nvcr.io/nvidia/l4t-pytorch:r32.5.0-pth1.6-py3

ENV DEBIAN_FRONTEND=noninteractive

RUN apt update
RUN apt install -y cmake libgtk2.0-dev wget

RUN python3 -m pip install --upgrade pip

RUN python3 -m pip install ffmpeg
RUN python3 -m pip install opencv-python

RUN wget https://nvidia.box.com/shared/static/p57jwntv436lfrd78inwl7iml6p13fzh.whl -O torch-1.8.0-cp36-cp36m-linux_aarch64.whl
RUN python3 -m pip install torch-1.8.0-cp36-cp36m-linux_aarch64.whl
RUN git clone https://github.com/pytorch/vision torchvision
RUN cd torchvision && git checkout v0.9.0 && python3 setup.py install

RUN python3 -m pip install scikit-build
RUN python3 -m pip install ffmpeg
RUN python3 -m pip install opencv-python
RUN git clone https://github.com/ultralytics/yolov5.git
RUN cd yolov5 && python3 -m pip install -r requirements.txt
WORKDIR /yolov5

[xrstokes]

Thanks mate! From: otamajakusi ***@***.***> Sent: Wednesday, 5 May 2021 5:04 PM To: otamajakusi/dockerfile-yolov5-jetson ***@***.***> Cc: xrstokes ***@***.***>; Mention ***@***.***> Subject: Re: [otamajakusi/dockerfile-yolov5-jetson] RuntimeError: CUDA error: device kernel image is invalid (#4) @xrstokes<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fxrstokes&data=04%7C01%7C%7C1a98664dee124854382e08d90f9401f4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637557950626032488%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=JU5QUXkrcXBHzDK4vkjv3GS382xOEBRoWUpn%2Fg1wDnA%3D&reserved=0> 77c133a<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fotamajakusi%2Fdockerfile-yolov5-jetson%2Fcommit%2F77c133ad646b917ca0c48260a668a816b10b9702&data=04%7C01%7C%7C1a98664dee124854382e08d90f9401f4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637557950626042482%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cQiNimfziFzU5JyoHPNEs3WmObcxFS%2BFEN0MLxFXG6Q%3D&reserved=0> supports ffmpeg. - You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fotamajakusi%2Fdockerfile-yolov5-jetson%2Fissues%2F4%23issuecomment-832462087&data=04%7C01%7C%7C1a98664dee124854382e08d90f9401f4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637557950626042482%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nBLWvpJCAqgWwwwamcD3xpOnAH3m62%2Fh6hJP%2F3bTOn0%3D&reserved=0>, or unsubscribe<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHOLECKQIRPT42BY5PZMNG3TMDUXJANCNFSM43WBONTA&data=04%7C01%7C%7C1a98664dee124854382e08d90f9401f4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637557950626052477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ckkm6tJr0aj735Rkeo8gX2gmefR5nXBfhAABH6%2BTDWY%3D&reserved=0>.

[xrstokes]

Changing back to the 5.0 image fixed the problem. But now I'm having trouble testing on an MP4. Do i need to install a lib for the MP4 format?

[xrstokes]

Another day and no progress. No matter what i try i can't open an mp4 in the container.
video = cv2.VideoCapture('all_colors.MP4')
fps = video.get(cv2.CAP_PROP_FPS)
print(fps)

^works on the jetson. but always fails in the container returning 0. That means i can't run the container with a video as the source. Any advice?

[xrstokes]

rebuilt opencv with -D OPENCV_FFMPEG_SKIP_BUILD_CHECK=ON \

^ fixed problem i think.

[otamajakusi]

My environment is:

jetson@jetson-desktop:~/dockerfile-yolov5-jetson$ cat /etc/nv_tegra_release 
# R32 (release), REVISION: 5.1, GCID: 26202423, BOARD: t210ref, EABI: aarch64, DATE: Fri Feb 19 16:45:52 UTC 2021

And I'm not good at torch, but on my environment, the instruction you mentioned doesn't show any error.

root@jetson-desktop:/yolov5# python3
Python 3.6.9 (default, Oct  8 2020, 12:12:24) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> device = torch.device("cuda")
>>> device 
device(type='cuda')
>>> torch.rand(10)
tensor([0.9158, 0.7678, 0.5396, 0.6999, 0.3756, 0.8283, 0.8684, 0.6938, 0.5391,
        0.3479])
>>> torch.rand(10)
tensor([0.9311, 0.3135, 0.5782, 0.3391, 0.7750, 0.6705, 0.5390, 0.0953, 0.6380,
        0.4314])
>>> torch.rand(10).to(device)
tensor([0.4091, 0.0234, 0.1496, 0.8245, 0.9412, 0.9183, 0.3809, 0.7819, 0.1235,
        0.5110], device='cuda:0')

[otamajakusi]

rebuilt opencv with -D OPENCV_FFMPEG_SKIP_BUILD_CHECK=ON \

^ fixed problem i think.

I'm not sure, but I think this is not a issue for this repository, right?

[xrstokes]

Thanks for your input, It'll take me a few days to process it sorry. Just a noob tying to nut this out and there isn't a ton of information out there.
While i couldn't get it to work running the latest downloaded image from nvidia, going back to 5.0 solved it but i couldn't work out why. The running the command i mentioned on an mp4 failed because there was no link from opencv to ffmpeg. Hopefully I'm on top of it now. I'll post back here a step by step once i nail it down.

Thanks again for sharing your work.

[otamajakusi]

@xrstokes
print(cv2.getBuildInformation()) says:

...
  Video I/O:
    DC1394:                      NO
    FFMPEG:                      NO
      avcodec:                   NO
      avformat:                  NO
      avutil:                    NO
      swscale:                   NO
      avresample:                NO
    GStreamer:                   NO
    v4l/v4l2:                    YES (linux/videodev2.h)

So actually FFMPEG does not seem to be enabled.

[xrstokes]

any idea how i can make ffmpeg enabled? I've been trying for 2 days. I need to be able to run tests on mpegs as well as on camera.

[xrstokes]

cmake shows this.
-- Video I/O:
-- DC1394: YES (2.2.5)
-- FFMPEG: YES
-- avcodec: YES (57.107.100)
-- avformat: YES (57.83.100)
-- avutil: YES (55.78.100)
-- swscale: YES (4.8.100)
-- avresample: YES (3.7.0)
-- GStreamer: YES (1.14.5)
-- v4l/v4l2: YES (linux/videodev2.h)

but then when it's finished. The version has changed wo i know it installed. but print(cv2.getBuildInformation()) says: NO.

weird.

[xrstokes]

DOH! It wasn't working when i installed ffmpeg with apt. If i install it with pip it works. Sorry to bother you.

[otamajakusi]

@xrstokes 77c133a supports ffmpeg.