ryoyo-nv/gaze-analysis-system

Gaze analysis system for Jetson Xavier NX, and AGX Xavier.

gaze-analysis-system's Issues

Cannot execute run_gaze_sequential.py to the end

I tried to run this script without Kibana, on Jetson AGX Xavier with Jetpack 4.4 flashed by SDK Manager.
But the script cannot executed to the end, and I found following error.

Error: Failed to read a frame data because gst thread is done already
Error: Gst thread is dead

The full log when executed the script is attached.
Could you tell me something wrong?

Thanks.

log.txt

[Request] Add LD_LIBRARY_PATH to .bashrc

I forget to set LD_LIBRARY_PATH every time.

I think I can just append the following to setup.sh ~/.bashrc.

export LD_LIBRARY_PATH=/opt/nvidia/deepstream/deepstream/lib/cvcore_libs:$LD_LIBRARY_PATH

You also might want to add the following entry to README.md.

• (optional) Add LD_LIBRARY_PATH to .bashrc

When I run setup.sh in a Docker container, I get the following error.

When I run setup.sh in a Docker container, I get the following error

root@8b6a2355b7d7:/opt/nvidia/deepstream/Gaze-Analysis-System# ls -la
total 228
drwxr-xr-x 1 root root   4096 Jun 17 03:47 .
drwxr-xr-x 1 root root   4096 Jun 17 03:38 ..
drwxrwxr-x 8 root root   4096 Jun 17 03:31 .git
-rw-rw-r-- 1 root root    292 May 27 04:38 .gitignore
-rw-rw-r-- 1 root root    132 Jun 17 03:47 Dockerfile
-rw-rw-r-- 1 root root 168557 May 27 04:38 Gaze-Analysis-System-License.pdf
-rw-rw-r-- 1 root root   3744 May 27 04:38 README.md
-rw-rw-r-- 1 root root   1627 May 27 04:38 azure_config.ini
drwxrwxr-x 4 root root   4096 Jun 17 03:31 ds
drwxrwxr-x 3 root root   4096 Jun 17 03:31 kibana
drwxrwxr-x 7 root root   4096 Jun 17 03:31 model
-rw-rw-r-- 1 root root   3865 May 27 04:38 run.py
-rwxrwxr-x 1 root root   5667 Jun 17 03:44 setup.sh
drwxrwxr-x 2 root root   4096 Jun 17 03:31 src
drwxrwxr-x 3 root root   4096 Jun 17 03:31 utils
root@8b6a2355b7d7:/opt/nvidia/deepstream/Gaze-Analysis-System# bash ./setup.sh
Checking DeepStream installation...
done.

Installing dependencies...
Reading package lists... Done
Building dependency tree
Reading state information... Done
ca-certificates is already the newest version (20210119~18.04.2).
<<<snip>>>
Err:1 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 python3.6 arm64 3.6.9-1~18.04ubuntu1.6
  404  Not Found [IP: 185.125.190.39 80]
Err:2 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libpython3.6-stdlib arm64 3.6.9-1~18.04ubuntu1.6
  404  Not Found [IP: 185.125.190.39 80]
Err:3 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 python3.6-minimal arm64 3.6.9-1~18.04ubuntu1.6
  404  Not Found [IP: 185.125.190.39 80]
Err:4 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libpython3.6-minimal arm64 3.6.9-1~18.04ubuntu1.6
  404  Not Found [IP: 185.125.190.39 80]
Err:12 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 dbus arm64 1.12.2-1ubuntu1.2
  404  Not Found [IP: 185.125.190.39 80]
Err:63 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe arm64 libavutil-dev arm64 7:3.4.8-0ubuntu0.2
  404  Not Found [IP: 185.125.190.39 80]
Err:64 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe arm64 libswresample-dev arm64 7:3.4.8-0ubuntu0.2
  404  Not Found [IP: 185.125.190.39 80]
Err:65 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe arm64 libavcodec-dev arm64 7:3.4.8-0ubuntu0.2
  404  Not Found [IP: 185.125.190.39 80]
Err:66 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe arm64 libavformat-dev arm64 7:3.4.8-0ubuntu0.2
  404  Not Found [IP: 185.125.190.39 80]
Err:67 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe arm64 libavresample-dev arm64 7:3.4.8-0ubuntu0.2
  404  Not Found [IP: 185.125.190.39 80]
Err:69 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libexpat1-dev arm64 2.2.5-3ubuntu0.2
  404  Not Found [IP: 185.125.190.39 80]
Err:107 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libmysqlclient20 arm64 5.7.37-0ubuntu0.18.04.1
  404  Not Found [IP: 185.125.190.39 80]
Err:112 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libnss3 arm64 2:3.35-2ubuntu2.13
  404  Not Found [IP: 185.125.190.39 80]
Err:114 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libpq5 arm64 10.19-0ubuntu0.18.04.1
  404  Not Found [IP: 185.125.190.39 80]
Err:164 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe arm64 libswscale-dev arm64 7:3.4.8-0ubuntu0.2
  404  Not Found [IP: 185.125.190.39 80]
Err:167 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libtiffxx5 arm64 4.0.9-5ubuntu0.4
  404  Not Found [IP: 185.125.190.39 80]
Err:168 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libtiff5-dev arm64 4.0.9-5ubuntu0.4
  404  Not Found [IP: 185.125.190.39 80]
Err:169 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libtiff-dev arm64 4.0.9-5ubuntu0.4
  404  Not Found [IP: 185.125.190.39 80]
Err:207 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libpython3.6 arm64 3.6.9-1~18.04ubuntu1.6
  404  Not Found [IP: 185.125.190.39 80]
Err:208 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libpython3.6-dev arm64 3.6.9-1~18.04ubuntu1.6
  404  Not Found [IP: 185.125.190.39 80]
Err:223 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 python3.6-dev arm64 3.6.9-1~18.04ubuntu1.6
  404  Not Found [IP: 185.125.190.39 80]
Fetched 94.4 MB in 26s (3686 kB/s)
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/p/python3.6/python3.6_3.6.9-1~18.04ubuntu1.6_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/p/python3.6/libpython3.6-stdlib_3.6.9-1~18.04ubuntu1.6_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/p/python3.6/python3.6-minimal_3.6.9-1~18.04ubuntu1.6_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/p/python3.6/libpython3.6-minimal_3.6.9-1~18.04ubuntu1.6_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/d/dbus/dbus_1.12.2-1ubuntu1.2_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/universe/f/ffmpeg/libavutil-dev_3.4.8-0ubuntu0.2_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/universe/f/ffmpeg/libswresample-dev_3.4.8-0ubuntu0.2_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/universe/f/ffmpeg/libavcodec-dev_3.4.8-0ubuntu0.2_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/universe/f/ffmpeg/libavformat-dev_3.4.8-0ubuntu0.2_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/universe/f/ffmpeg/libavresample-dev_3.4.8-0ubuntu0.2_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/e/expat/libexpat1-dev_2.2.5-3ubuntu0.2_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/m/mysql-5.7/libmysqlclient20_5.7.37-0ubuntu0.18.04.1_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/n/nss/libnss3_3.35-2ubuntu2.13_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/p/postgresql-10/libpq5_10.19-0ubuntu0.18.04.1_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/universe/f/ffmpeg/libswscale-dev_3.4.8-0ubuntu0.2_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/t/tiff/libtiffxx5_4.0.9-5ubuntu0.4_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/t/tiff/libtiff5-dev_4.0.9-5ubuntu0.4_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/t/tiff/libtiff-dev_4.0.9-5ubuntu0.4_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/p/python3.6/libpython3.6_3.6.9-1~18.04ubuntu1.6_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/p/python3.6/libpython3.6-dev_3.6.9-1~18.04ubuntu1.6_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/p/python3.6/python3.6-dev_3.6.9-1~18.04ubuntu1.6_arm64.deb  404  Not Found [IP: 185.125.190.39 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Setup failed. Please check error messages.
exit.

After running apt udpate, I was able to run setup.sh without any problems.
I think you need to add apt update to setup.sh.
How do you think?

Error with 'python3 run.py /dev/video0 --media v4l2 --cam-fps 10'

If I try to run it with /dev/video0 --media v4l2 --cam-fps 10, I get an error.

deepstream pipeline initializing...
[DEBUG] DeepStream pipeline properties:  {'filesrc': {'location': ''}, 'v4l2src': {'device': '/dev/video0'}, 'v4l2filter': {'caps': 'video/x-raw, width=1280, height=720, framerate=10/1, parsed=True'}, 'v4l2jpgdec': {'mjpeg': True}, 'filter1': {'caps': 'video/x-raw(memory:NVMM), width=1280, height=720, format=NV12'}, 'streammux': {'width': 1280, 'height': 720, 'batch-size': 1, 'batched-push-timeout': 4000000, 'live-source': False, 'frame-num-reset-on-eos': True}, 'pgie': {'config-file-path': 'ds/ds_pgie_facedetect_config.txt'}, 'tracker': {'tracker-width': 640, 'tracker-height': 384, 'gpu-id': 0, 'll-lib-file': '/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so', 'll-config-file': 'ds/tracker_config.yml', 'enable-batch-process': True}, 'sgie1': {'config-file-path': 'ds/ds_sgie_age_config.txt'}, 'sgie2': {'config-file-path': 'ds/ds_sgie_gender_config.txt'}, 'sgie3': {'config-file-path': 'ds/ds_sgie_faciallandmarks_config.txt'}, 'tgie': {'customlib-name': 'ds/lib/libnvds_gazeinfer.so', 'customlib-props': 'config-file:ds/lib/gazenet_config.txt'}, 'filter2': {'caps': 'video/x-raw, format=RGBA, width=1280, height=720'}, 'filter3': {'caps': 'video/x-raw, format=BGR'}, 'sink': {'emit-signals': True, 'caps': 'video/x-raw, format=BGR, width=1280, height=720'}}
---< snip > ---
ds_pgie_facedetect_config.txt sucessfully
Error: Failed to read a frame data from gst pipeline
Error: gst-stream-error-quark: Internal data stream error. (1): gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2-source:
streaming stopped, reason not-negotiated (-4)
Error: Failed to read a frame data because gst thread is done already
Error: Gst thread is not alive
exiting application...

However, it worked when I specified the Width and Height and cam-fps as follows.

python3 run.py /dev/video0 --media v4l2 --codec mjpg --width 1280 --height 720 --cam-fps 16

Cannot start the application

Environment:

• HW: Jetson Xavier NX
• OS: JetPack 4.6.1

Expected behaviour

I can run the application.

Actual behaviour

I got the following error and could not start.

$ python3 run.py /dev/video0 --media v4l2 --cam-fps 10 
Traceback (most recent call last):
  File "run.py", line 10, in <module>
    from ds.ds_pipeline import DsVideo
  File "/home/t-yano/Projects/Gaze-Analysis-System/ds/ds_pipeline.py", line 20, in <module>
    import dscprobes
ImportError: libnvcv_faciallandmarks.so: cannot open shared object file: No such file or directory

Step reprocedure.

1. Run upgrade
   sudo apt upgrade
2. Install Deepstream-6.0.1
   sudo apt install -y deepstream-6.0
3. Clone Git Repository & exec setup.sh

git clone https://github.com/Ryoyo-NV/Gaze-Analysis-System.git
git checkout -b jp4.6.2_ds6.0.1 refs/tags/jp4.6.2_ds6.0.1
chmod +x setup.sh
./setup.sh

The setup.sh shell has completed without any problems.
5. Start application

I have run the following

$ python3 run.py /dev/video0 --media v4l2 --cam-fps 10 

failed compile on 'Building gazeinfer library...'

Environment:

• HW: Jetson Xavier NX
• OS: JetPack 4.6.1

Expected behaviour

The following Install procedure completes without problems.
https://github.com/Ryoyo-NV/Gaze-Analysis-System#installation

Actual behaviour

The compile of gazeinfer will fail

Building gazeinfer library...
-fPIC -DDS_VERSION="6.1.0" -I /usr/local/cuda-10.2/include -I ../../../includes -I /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdsvideotemplate/includes -I /opt/nvidia/deepstream/deepstream/sources/includes -I /opt/nvidia/deepstream/deepstream/sources/includes/cvcore_headers -I ../../deepstream-faciallandmark-app -I ../ -I ../../common/ -pthread -I/usr/include/gstreamer-1.0 -I/usr/include/orc-0.4 -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0-I/usr/lib/aarch64-linux-gnu/glib-2.0/include -Wno-deprecated-declarations
g++ -c -o gazeinfer.o -fPIC -DDS_VERSION=\"6.1.0\" -I /usr/local/cuda-10.2/include -I ../../../includes -I /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdsvideotemplate/includes -I /opt/nvidia/deepstream/deepstream/sources/includes -I /opt/nvidia/deepstream/deepstream/sources/includes/cvcore_headers -I ../../deepstream-faciallandmark-app -I ../ -I ../../common/ -pthread -I/usr/include/gstreamer-1.0 -I/usr/include/orc-0.4 -I/usr/include/gstreamer-1.0-I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -Wno-deprecated-declarations gazeinfer.cpp
In file included from gazeinfer.cpp:52:0:
../../common/ds_yml_parse.h:33:10: fatal error: nvds_yml_parser.h: No such file or directory
 #include "nvds_yml_parser.h"
          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:70: recipe for target 'gazeinfer.o' failed
make: *** [gazeinfer.o] Error 1
Setup failed. Please check error messages.
exit.

Error Running in Deepstream 6.0

This is the below error I am facing while running on jetpack 4.6 and deepstream 6.0.

Traceback (most recent call last):
  File "run_gaze_sequential.py", line 11, in <module>
    from face_landmark import FaceLandmarkEngine, FaceLandmarkPost, FaceLandmarkDraw, leye_boxes, reye_boxes
  File "/home/cvision/Gaze-Analysis-System/face_landmark.py", line 11, in <module>
    ctypes.CDLL(os.path.join(os.getcwd(), "libflnetsoftargmax.so"))
  File "/usr/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/cvision/Gaze-Analysis-System/libflnetsoftargmax.so: cannot open shared object file: No such file or directory

failed compile on 'Building Python wrapper library for gazeinfer'

Environment:

• HW: Jetson Xavier NX
• OS: JetPack 4.6.1

Expected behaviour

The following Install procedure completes without problems.
https://github.com/Ryoyo-NV/Gaze-Analysis-System#installation

Actual behaviour

The compile of gazeinfer will fail

Building Python wrapper library for gazeinfer...
Cloning into 'deepstream_tao_apps'...
remote: Enumerating objects: 1248, done.
remote: Counting objects: 100% (187/187), done.
remote: Compressing objects: 100% (135/135), done.
remote: Total 1248 (delta 89), reused 114 (delta 52), pack-reused 1061
Receiving objects: 100% (1248/1248), 26.31 MiB | 7.28 MiB/s, done.
Resolving deltas: 100% (646/646), done.
g++ -c -o deepstream_faciallandmark_meta.o -Wall -Werror -DPLATFORM_TEGRA -I /opt/nvidia/deepstream/deepstream/sources/includes -I /opt/nvidia/deepstream/deepstream/sources/includes/cvcore_headers -I /usr/local/cuda-10.2/include -I deepstream_tao_apps/apps/tao_others/deepstream-faciallandmark-app -I deepstream_tao_apps/apps/tao_others/deepstream-gaze-app `pkg-config --cflags gstreamer-1.0 python3 pygobject-3.0` -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-sign-compare -fPIC deepstream_tao_apps/apps/tao_others/deepstream-faciallandmark-app/deepstream_faciallandmark_meta.cpp
g++ -c -o call_probes_from_py.o -fpermissive -Wall -Werror -DPLATFORM_TEGRA -I /opt/nvidia/deepstream/deepstream/sources/includes -I /opt/nvidia/deepstream/deepstream/sources/includes/cvcore_headers -I /usr/local/cuda-10.2/include -Ideepstream_tao_apps/apps/tao_others/deepstream-faciallandmark-app -I deepstream_tao_apps/apps/tao_others/deepstream-gaze-app `pkg-config --cflags gstreamer-1.0 python3 pygobject-3.0` -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-sign-compare -fPICcall_probes_from_py.cpp
cc -shared -o dscprobes.so deepstream_faciallandmark_meta.o call_probes_from_py.o `pkg-config --libs gstreamer-1.0 python3 pygobject-3.0` -L/opt/nvidia/deepstream/deepstream/lib/ -lnvdsgst_meta -lnvds_meta -lnvds_inferutils -lnvds_utils -lm -lstdc++ -L/usr/local/cuda-10.2/lib64/ -lcudart -lcuda -L/opt/nvidia/deepstream/deepstream/lib/cvcore_libs -lnvcv_faciallandmarks -lnvcv_core -lnvcv_tensorops -lnvcv_trtbackend -Wl,-rpath,/opt/nvidia/deepstream/deepstream/lib/
done.

Building gazeinfer library...
-fPIC -DDS_VERSION="6.1.0" -I /usr/local/cuda-10.2/include -I ../../../includes -I /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdsvideotemplate/includes -I /opt/nvidia/deepstream/deepstream/sources/includes -I /opt/nvidia/deepstream/deepstream/sources/includes/cvcore_headers -I ../../deepstream-faciallandmark-app -I ../ -I ../../common/ -pthread -I/usr/include/gstreamer-1.0 -I/usr/include/orc-0.4 -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0-I/usr/lib/aarch64-linux-gnu/glib-2.0/include -Wno-deprecated-declarations
g++ -c -o gazeinfer.o -fPIC -DDS_VERSION=\"6.1.0\" -I /usr/local/cuda-10.2/include -I ../../../includes -I /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdsvideotemplate/includes -I /opt/nvidia/deepstream/deepstream/sources/includes -I /opt/nvidia/deepstream/deepstream/sources/includes/cvcore_headers -I ../../deepstream-faciallandmark-app -I ../ -I ../../common/ -pthread -I/usr/include/gstreamer-1.0 -I/usr/include/orc-0.4 -I/usr/include/gstreamer-1.0-I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -Wno-deprecated-declarations gazeinfer.cpp
In file included from gazeinfer.cpp:52:0:
../../common/ds_yml_parse.h:32:10: fatal error: gst/rtsp-server/rtsp-server.h: No such file or directory
 #include <gst/rtsp-server/rtsp-server.h>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:70: recipe for target 'gazeinfer.o' failed
make: *** [gazeinfer.o] Error 1
Setup failed. Please check error messages.
exit.

Step reprocedure.

1. Run upgrade
   sudo apt upgrade
2. Install Deepstream-6.0.1
   sudo apt install -y deepstream-6.0
3. Clone Git Repository & exec setup.sh

git clone https://github.com/Ryoyo-NV/Gaze-Analysis-System
chmod +x setup.sh
./setup.sh