smellslikeml/actionai

Real-Time Spatio-Temporally Localized Activity Detection by Tracking Body Keypoints

ActionAI 🤸

ActionAI is a python library for training machine learning models to classify human action. It is a generalization of our yoga smart personal trainer, which is included in this repo as an example.

Getting Started

These instructions will show how to prepare your image data, train a model, and deploy the model to classify human action from image samples. See deployment for notes on how to deploy the project on a live stream.

Installation

Add the smellslikeml PPA and install with the following:

sudo add-apt-repository ppa:smellslikeml/ppa
sudo apt update

# Install with:
sudo apt-get install actionai

Make sure to configure the working directory with:

actionai configure

Using the CLI

Organize your training data in subdirectories like the example below. The actionai cli will automatically create a dataset from subdirectories of videos where each subdirectory is a category label.

.
└── dataset/
    ├── category_1/
    │   └── *.mp4
    ├── category_2/
    │   └── *.mp4
    ├── category_3/
    │   └── *.mp4
    └── ...

Then you can train a model with:

actionai train --data=/path/to/your/data/dir --model=/path/to/your/model/dir

And then run inference on a video with:

actionai predict --model=/path/to/your/model/dir --video=/path/to/your/video.mp4

View the default config.ini file included in this branch for additional configurations. You can pass your own config file using the --cfg flag.

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

License

This project is licensed under the GNU General Public License v3.0 - see the LICENSE.md file for details

References

• Hackster post
• YogAI article
• Convolutional Pose Machine
• Pose estimation for mobile
• Pose estimation tensorflow implementation

actionai's Issues

Other devices

@cclauss @smellslikeml @mayorquinmachines Hi all thanks for opensourcing and sharing your work its really good work , i have few queries

1. can we test the work on cpu and other embedded devices
2. can i train my model on the opensource dataset and test it for the real time environment

How to improve FPS?

I've been working on to detect my own set of exercises, specifically push ups and sit ups. However, when running inference on my CSI camera or on a video file with my Jetson Nano, the FPS flickers and ranges from 2-10. I want to be able to run inference with an FPS of around 20, how can I do this?

Edit: I figured out that it can only be increased by changing my hardware, nothing to do with my software! Jetson Hacks has a video on how to improve FPS on the Nano itself on Youtube.

online_finetune.py without PS3 Controller

I'd like to use online_finetune.py without PS3 Controller.
Please let me know how to useonline_finetune.py without PS3 Controller.

draw_object return None

Hi,

I'm quite new on Jetson TX2. Then I have a stupid question. ;-)
I meet a TypeError while running python3 iva.py 0 on my laptop.
draw_object returns None body_dict
Any idea ?
Thanks,

Traceback (most recent call last):
File "iva.py", line 238, in
for body in pose_list:
TypeError: 'NoneType' object is not iterable

bounding box and pose dots on video

Hello!
Thanks a lot for your great work!

When I run inference.py on Yoga weights, on video it doesn't show the bounding box of the person and does not show dots of Pose Estimation, as in your GIFs. How can I put BB and Dots on the person on video, like in your GIFs?

IndexError when running iva.py on Jetson Nano

I was trying to run iva.py on my Jetson Nano with Jetpack 4.2, TensorFlow 2.0.0, when i got a "IndexError: index 0 is out of bounds for axis 0 with size 0" error. Does anyone have an idea what's causing the problem and how i can resolve it? Thanks in advance. Full logs below.

2021-02-22 14:26:13.848469: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-02-22 14:26:35.213896: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-02-22 14:26:35.514677: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2021-02-22 14:26:35.579968: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2021-02-22 14:26:35.580367: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-02-22 14:26:35.580668: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-02-22 14:26:35.580848: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-02-22 14:26:35.581061: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-02-22 14:26:36.455909: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-02-22 14:26:36.456797: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-02-22 14:26:36.548857: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-02-22 14:26:36.549570: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2021-02-22 14:26:36.550118: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2021-02-22 14:26:36.550409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-02-22 14:26:38.018770: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2021-02-22 14:26:38.090364: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x8d0a5870 executing computations on platform Host. Devices:
2021-02-22 14:26:38.090460: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2021-02-22 14:26:38.359351: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2021-02-22 14:26:38.373631: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x8d0a3330 executing computations on platform CUDA. Devices:
2021-02-22 14:26:38.373717: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2021-02-22 14:26:38.493701: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2021-02-22 14:26:38.494358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2021-02-22 14:26:38.495665: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-02-22 14:26:38.552145: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-02-22 14:26:38.552472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-02-22 14:26:38.552586: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-02-22 14:26:39.194542: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-02-22 14:26:39.195029: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-02-22 14:26:39.221961: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-02-22 14:26:39.238405: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2021-02-22 14:26:39.239452: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2021-02-22 14:26:39.272974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-02-22 14:27:27.904387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-22 14:27:28.167470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2021-02-22 14:27:28.167578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2021-02-22 14:27:28.871407: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2021-02-22 14:27:28.934040: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2021-02-22 14:27:28.989680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 225 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
Traceback (most recent call last):
  File "iva.py", line 275, in <module>
    matched, unmatched_trackers, unmatched_detections = tracker_match(track_boxes, [b[0] for b in bboxes])
  File "iva.py", line 139, in tracker_match
    if(IOU_mat[m[0],m[1]] < iou_thrd):
IndexError: index 0 is out of bounds for axis 0 with size 0

iva.py TypeError issue

Hi,
I'm quite new on Jetson TX2. Then I have a stupid question. ;-)
I meet a TypeError while running python3 iva.py 0 on my laptop.
draw_object returns None body_dict

Traceback (most recent call last):
File "iva.py", line 238, in
for body in pose_list:
TypeError: 'NoneType' object is not iterable

IndexError: index 0 is out of bounds for axis 0 with size 0

I run iva.py on Jetson Nano using this command: python3 iva.py 0
But there is an error:

WARNING:tensorflow:Layer lstm will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
Traceback (most recent call last):
File "iva.py", line 275, in
matched, unmatched_trackers, unmatched_detections = tracker_match(track_boxes, [b[0] for b in bboxes])
File "iva.py", line 139, in tracker_match
if(IOU_mat[m[0],m[1]] < iou_thrd):
IndexError: index 0 is out of bounds for axis 0 with size 0

Any ideas? Thanks!

TypeError: 'dict' object is not callable while running train_legday.py

Thanks for the wonderful code.

I have got this error while running yogai on Linux PC, not on Raspberry PI.

• Ubuntu 18.04.3

What I did:

• cd examples/yogai
• ./install.sh (changed the tensorflow part to pip3 install tensorflow==1.11)
• python3 train_legday.py

Traceback (most recent call last):
  File "train_legday.py", line 61, in <module>
    X_train, X_test, y_train, y_test = load_data(args.data)
  File "train_legday.py", line 32, in load_data
    y_train = keras.utils.to_categorical(list(map(lbl_dict, y_train)), num_class)
TypeError: 'dict' object is not callable

I searched a little bit, I couldn't find any solution. The error is due to this part:

list(map(lbl_dict, y_train))

Any suggestion?

The engine plan file is not compatible with this version of TensorRT, expecting library version 5.1.5 got 5.1.6, please rebuild.

[TensorRT] ERROR: The engine plan file is not compatible with this version of TensorRT, expecting library version 5.1.5 got 5.1.6, please rebuild. Traceback (most recent call last): File "experimental/teachable_machine.py", line 7, in <module> import poses File "/home/xxxx/ActionAI/experimental/poses.py", line 19, in <module> model_trt.load_state_dict(torch.load(OPTIMIZED_MODEL)) File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 824, in load_state_dict load(self) File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 819, in load state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs) File "/usr/local/lib/python3.7/site-packages/torch2trt/torch2trt.py", line 309, in _load_from_state_dict self.context = self.engine.create_execution_context() AttributeError: 'NoneType' object has no attribute 'create_execution_context'
how to suppose fix issue mentioned above? actually i tried to run on laptop which is installed tensorrt 5.1.5, or this cannot be run on device other than jetson nano?

Serialization Error in verifyHeader: 0

i done the installation and ran python3 iva.py 0, errors occur：

[TensorRT] ERROR: ../rtSafe/coreReadArchive.cpp (31) - Serialization Error in verifyHeader: 0 (Magic tag does not match)
[TensorRT] ERROR: INVALID_STATE: std::exception
[TensorRT] ERROR: INVALID_CONFIG: Deserialize the cuda engine failed.
Traceback (most recent call last):
File "iva.py", line 39, in
model_trt.load_state_dict(torch.load(OPTIMIZED_MODEL))
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 815, in load_state_dict
load(self)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 810, in load
state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
File "/usr/local/lib/python3.6/dist-packages/torch2trt/torch2trt.py", line 309, in _load_from_state_dict
self.context = self.engine.create_execution_context()
AttributeError: 'NoneType' object has no attribute 'create_execution_context'

Any ideas?

seems related to the model...did you test your model on jetson nano?

Issue in init of motionclassifier

Could you please clarify what this part of code is doing. There are no files with the prefix "motion_" in the models directory.

self.motion_path = sorted(glob('./models/motion_*'))[-1]
self.N = int(os.path.splitext(self.motion_path)[0].split('_')[-1])
self.motion_model = load_model(self.motion_path)

pose.tflite

Hello,

Has anyone tried using pose.tflite model in some other project? Is it even possible to do that? I tried using in some other flutter project, however it crashed.

This says some of them are successful however most of them are facing similar issue.

Can anyone suggest what is the best way to do that?

• AB

Jetson nano tx1 configuration

hi,
can you please tell us about the setup of your system,which versions of requirements you have used.like version of tensorflow ,cuda,etc.
i am facing much errors due to miss matching.
Regards,

Jetson Nano: AttributeError: 'NoneType' object has no attribute 'create_execution_context

System

Library	Version
Jetson Nano on Jetpack	4.5.1
CUDA	10.2
TensorRT	7.1.3
PyTorch	1.8
TorchVision	0.9.0
Tensorflow	2.4.0

Verified USB Webcam is working with OpenCV and Tensorflow TRT Yolo model in same virtual environment.

Installed system requirements, installed trt_pose repo, downloaded model assets to the /models directory, and getting this error when running:
python iva.py 0

optimized model path: /home/nvidia/ActionAI/models/resnet18_baseline_att_224x224_A_epoch_249_trt.pth
[TensorRT] ERROR: coreReadArchive.cpp (31) - Serialization Error in verifyHeader: 0 (Magic tag does not match)
[TensorRT] ERROR: INVALID_STATE: std::exception
[TensorRT] ERROR: INVALID_CONFIG: Deserialize the cuda engine failed.
Traceback (most recent call last):
  File "iva.py", line 42, in <module>
    model_trt.load_state_dict(torch.load(OPTIMIZED_MODEL))
  File "/home/nvidia/.virtualenvs/py3tf23cv4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1209, in load_state_dict
    load(self)
  File "/home/nvidia/.virtualenvs/py3tf23cv4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1204, in load
    state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
  File "/home/nvidia/.virtualenvs/py3tf23cv4/lib/python3.6/site-packages/torch2trt-0.2.0-py3.6-linux-aarch64.egg/torch2trt/torch2trt.py", line 454, in _load_from_state_dict
    self.context = self.engine.create_execution_context()
AttributeError: 'NoneType' object has no attribute 'create_execution_context'

There is an article on nVIdia which says this could be to mismatching input sizes, though I don't see how that could be. The other thing is it could be a TensorRT incompatibility.

Has anyone been able to get this running on a Jetson Nano with the Jetpack 4.5? Any ideas out there?

Re: Custom Dataset Training

@smellslikeml
Hi,
I am working on detection of custom action recognition using my own dataset however i am unable to train due to some sklearn pipeline error. The error image is attached below.
I am using jetson tx2 for the same.

Any help would be appreciated .

AttributeError: module 'tensorflow_core.contrib' has no attribute 'lite'

On a fresh install:

python3 demo.py
Using TensorFlow backend.
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

• https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
• https://github.com/tensorflow/addons
• https://github.com/tensorflow/io (for I/O related ops)
  If you depend on functionality not listed there, please file an issue.

Traceback (most recent call last):
File "demo.py", line 124, in
mC = motionClassifier()

File "demo.py", line 19, in init
self.interpreter = tf.contrib.lite.Interpreter(model_path=self.model_path)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/util/lazy_loader.py", line 63, in getattr
return getattr(module, item)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/init.py", line 51, in getattr
return getattr(module, item)
AttributeError: module 'tensorflow_core.contrib' has no attribute 'lite'

Setup Configuration to run this repository?

Hello,

I'm having trouble running this repository on Cuda 10.1, tensorrt 6.0.1.5, torch 1.7.1, trt-pose 0.0.1, tensorflow 2.0.0

Could you provide the versions for all these libraries that the current branch works on?

Running on Windows

can we run ActionAI on windows? If yes, can you please share installation steps

Project dependencies may have API risk issues

Hi, In ActionAI, inappropriate dependency versioning constraints can cause risks.

Below are the dependencies and version constraints that the project is using

tensorflow>=2.6.2
scipy
scikit-learn
opencv-contrib-python
pandas
pillow

The version constraint == will introduce the risk of dependency conflicts because the scope of dependencies is too strict.
The version constraint No Upper Bound and * will introduce the risk of the missing API Error because the latest version of the dependencies may remove some APIs.

After further analysis, in this project,
The version constraint of dependency scikit-learn can be changed to >=0.15.0,<=0.20.4.
The version constraint of dependency pandas can be changed to >=0.4.0,<=0.6.0.
The version constraint of dependency pandas can be changed to >=0.13.0,<=0.23.4.
The version constraint of dependency pillow can be changed to ==9.2.0.
The version constraint of dependency pillow can be changed to >=2.0.0,<=9.1.1.

The above modification suggestions can reduce the dependency conflicts as much as possible,
and introduce the latest version as much as possible without calling Error in the projects.

The invocation of the current project includes all the following methods.

The calling methods from the scikit-learn

sklearn.utils.linear_assignment_.linear_assignment
random.choice

The calling methods from the pandas

pandas.DataFrame.to_csv
pandas.read_csv

The calling methods from the pillow

PIL.Image.fromarray
PIL.Image.open

The calling methods from the all methods

numpy.where
pickle.load
transformer.PoseExtractor.transform
feat_array.append
subprocess.Popen.communicate
numpy.zeros.tolist
body_labels.items
keras.optimizers.RMSprop
tracker.annotate.sub_
datetime.datetime.now
self.cubit_q.append
utils.source_capture
cv2.resize
tensorflow.contrib.lite.Interpreter.get_tensor
person.PersonTracker.set_pose
pygame.joystick.Joystick.init
pandas.read_csv.to_csv
cv2.imread.sub_
cv2.VideoCapture.release
actionModel
cv2.drawMarker
tensorflow.keras.models.Sequential
cv2.VideoCapture.set
a_idx.data.cpu.numpy
sample.reshape.reshape
lstm_model.compile
a_idx.data.cpu
self.parse_objects
screen_size.getScreenDims
lstm_model.summary
load_data
numpy.random.randint
numpy.vstack
model.lstm_model.fit
open
self.body_labels.values
motionClassifier.pose
str
model.lstm_model
logging.StreamHandler
utils.img_obj.annotate
humans
self.q.append
cv2.rectangle
lstm_model.fit
ky.pose_dict.values
augment
numpy.random.choice
sys.exit
numpy.sum
self.motion_path.os.path.splitext.split
isinstance
control.getButton
self.interpreter.get_tensor
cv2.threshold
self.interpreter.get_output_details
self.get_keypoints.append
self.formatPose
cv2.VideoCapture.read
cv2.dilate
numpy.sin
bbox.append
random.uniform
body_dict.get
lstm_model.add
X_train.reshape.reshape
self.motion.pose
get_bbox.append
trackers.append
get_bbox
image.astype.astype
mean.image.sub_.div_
pandas.read_csv.iterrows
self.cam.read
PoseExtractor
model.lstm_model.predict
idx.trackers.set_bbox
tracker.annotate
tensorflow.keras.models.load_model.save
generate_stickfig
os.path.splitext
keras.models.Sequential
tensorflow.keras.models.load_model
keras.layers.Dropout
unmatched_trackers.append
ss.N.x.reshape.kp_rotate.np.apply_along_axis.flatten
idx.trackers.set_pose
rand_shift.copy
numpy.concatenate.append
cv2.imwrite
trackers.pop
torch2trt.TRTModule
body.values
argparse.ArgumentParser.add_argument
self.motion_model.predict
self.interpreter.get_input_details
ffwd_n_avg
utils.source_capture.read
numpy.linalg.norm
ky.pose_dict.keys
self.flow.index
trainModel
cv2.imread.resize
self.run_inference
logging.getLogger.info
image.np.expand_dims.astype
numpy.empty
open.close
argparse.ArgumentParser
PIL.Image.fromarray
tensorflow.keras.models.Sequential.summary
pose.replace.replace
type
pygame.joystick.Joystick.get_button
get_data
time.time
random.randint
cv2.VideoWriter
tensorflow.keras.models.load_model.add
motionClassifier
paf.detach.cpu
dataset_train.str.replace
X_test.reshape.reshape
torch.Tensor
dataset_test.str.replace
bs4.BeautifulSoup
tuple
humans.append
self.interpreter.set_tensor
model.RMSprop
transformer.PoseExtractor
tensorflow.keras.models.load_model.predict
self.get_keypoints
torch.load
pandas.read_csv.head
logging.Formatter
labeled_images.append
tensorflow.contrib.lite.Interpreter
set
tensorflow.contrib.lite.Interpreter.get_output_details
url.split
FaceDetector.process_frame
pandas.DataFrame.to_csv
flow.Flow
xgboost.XGBClassifier
keras.layers.Dense
resolution_string.split.decode.split
self.preprocess
tensorflow.contrib.lite.Interpreter.invoke
keras.utils.to_categorical
tensorflow.keras.models.load_model.compile
data_lst.append
numpy.zeros
sorted
model.to_categorical
pygame.joystick.Joystick.get_name
self.sayPhrase
pickle.dump
cv2.dilate.copy
activity_dict.values
list.append
actionModel.get_params
source.isdigit.isdigit
rand_shift
model_trt
cv2.imread
vert_swap
cmap.detach
collections.defaultdict
tensorflow.keras.layers.LSTM
utils.id_gen
csv.writer
numpy.array
cv2.namedWindow
join
mC.result.np.sum.reshape
numpy.arctan
sklearn.utils.linear_assignment_.linear_assignment
rand_flow
person.PersonTracker.set_bbox
cv2.VideoCapture
b_idx.data.cpu.numpy
cv2.VideoWriter_fourcc
utils.update_trackers
os.path.basename
json.load
actionModel.fit
datetime.datetime.now.strftime
importlib.import_module.classifier
dict
torch.Tensor.cuda
extractor.transform.reshape
numpy.asarray
os.path.exists
self.model_trt
unmatched_detections.append
sorted.append
cv2.VideoCapture.isOpened
bs4.BeautifulSoup.find_all
min
body_list.append
cv2.accumulateWeighted
motionClassifier.motion
os.path.join
time.sleep
self.interpreter.allocate_tensors
os.walk
tensorflow.contrib.lite.Interpreter.get_input_details
cv2.absdiff
preprocess
subprocess.call
cv2.VideoWriter.release
collections.defaultdict.keys
self.motion.ft_ext
cv2.findContours
idx.trackers.update_pose
subprocess.Popen
poses.inference
cv2.cvtColor
utils.img_obj
IOU
collections.deque.append
numpy.hstack
lst.append
keras.models.load_model
self.ffwd_n_avg.list.np.concatenate.reshape
dictOfElements.items
motionClassifier.motion_detector
importlib.import_module
paho.mqtt.client.Client
tensorflow.keras.models.load_model.summary
numpy.apply_along_axis
lstm_model.save
tensorflow.keras.optimizers.RMSprop
pygame.joystick.init
format
print
resolution_string.split
self.motion_path.rstrip.split
tensorflow.contrib.lite.Interpreter.allocate_tensors
gray.copy.astype
pygame.init
cv2.CascadeClassifier
cv2.VideoCapture.get
numpy.concatenate
tensorflow.keras.models.Sequential.add
cv2.destroyAllWindows
pygame.joystick.Joystick
numpy.float32
max
collections.deque
self.getPose
PersonTracker
person.PersonTracker
self.getPose.replace
ListHumans
self.faces.append
getScreenDims
PIL.Image.open
pose_model.predict
body_dict.items
p.stdout.close
draw_objects
tensorflow.contrib.lite.Interpreter.set_tensor
output_data.np.sum.reshape
sklearn.model_selection.train_test_split
argparse.ArgumentParser.parse_args
cv2.line
feature_vec.tolist.append
list
logging.StreamHandler.setFormatter
lstm_model
tensorflow.keras.layers.Dropout
pose.replace.replace.strip
dd.find
cv2.imshow
pandas.read_csv
glob.glob
torch.device
paf.detach
tracker_match
run_inference
coords.np.concatenate.reshape
tensorflow.keras.models.load_model.evaluate
cv2.contourArea
tensorflow.keras.utils.to_categorical
faces.FaceDetector.process_frame
inference
pygame.event.pump
image.transforms.functional.to_tensor.to
divmod
GetKeypoints
self.mean.image.sub_.div_
self.pose_model.predict
requests.get
numpy.flip
keras.layers.LSTM
logging.getLogger.addHandler
trt_pose.parse_objects.ParseObjects
pandas.read_csv.pop
multiprocessing.cpu_count
float
aggs
cv2.VideoWriter.write
logging.getLogger.setLevel
logging.getLogger
utils.source_capture.release
sys.path.append
cv2.putText
m.reshape
round
q.ffwd_n_avg.list.np.concatenate.reshape
cv2.waitKey
random.choice
sklearn.pipeline.Pipeline
faces.FaceDetector
model.lstm_model.compile
getKeysByValue
img.np.expand_dims.astype
DrawObjects
numpy.cos
augment.to_csv
self.interpreter.invoke
map
self.interpreter.get_tensor.reshape
utils.inference.motionClassifier
image.np.asarray.astype
os.listdir
bboxes.append
f
pprint.pprint
image.astype.reshape
len
cmap.detach.cpu
pickle.load.predict
cv2.warpAffine
torch2trt.TRTModule.load_state_dict
id_gen
tensorflow.lite.Interpreter
ord
image.transforms.functional.to_tensor.to.sub_
trt_pose.coco.coco_category_to_topology
parse_objects
b_idx.data.cpu
image.np.zeros_like.astype
PoseExtractor.transform
paho.mqtt.client.Client.connect
utils.get_bbox
resolution_string.split.decode
lstm_model.evaluate
self.model_trt.load_state_dict
logging.StreamHandler.setLevel
cv2.convertScaleAbs
int
enumerate
tensorflow.keras.layers.Dense
apply_vec
img.copy
self.ffwd_n_avg
coords.np.vstack.T.reshape
cv2.GaussianBlur
self.detector.detectMultiScale
paho.mqtt.client.Client.loop_forever
paho.mqtt.client.Client.subscribe
next
numpy.argmax
motionClassifier.ft_ext
range
numpy.expand_dims
keras.models.load_model.predict
xgboost.XGBClassifier.fit
cv2.circle
operator.itemgetter
load_image
feat_array.np.array.squeeze
x.reshape
csv.writer.writerow
cv2.setWindowProperty
tensorflow.keras.models.load_model.fit
subdir.split
PersonTracker.update_pose
flow.Flow.run
PersonTracker.set_bbox
pandas.DataFrame
self.motion_path.rstrip
numpy.prod
FaceDetector
torchvision.transforms.functional.to_tensor
cv2.GaussianBlur.copy
self.result.reshape
numpy.zeros_like

@developer
Could please help me check this issue?
May I pull a request to fix it?
Thank you very much.

Model trained on walking/loitering data

Hi
Thanks for the nice repo , can you please provide the trained model for the loitering / walking dataset ?
Thanks

train action like walking, carrying, etc

hello thank you for sharing a great work, i just pretty new at computer vision task, if you would mind explaining more detail does this project able to do certain task like walking, carrying, standing, etc. and if you would mind to give me more guidance about how to obtain 'lstm.h5' files when doing train sequential, thank you

classifier.sav

I would like try out an inference, could you please share classifier.sav file.

Error while training

Hello,
I recorded 3 different JPEG sequences of around 300 images each (repeating multiple time a movement, so maybe 20 images per movement multiple times). I ran the preprocess.py and the data seems to be correct in the data/data.csv. Now when I try to run train.py I get this error :

/home/hugo/anaconda3/envs/tensorrt/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of f AND g EVALUATIONS EXCEEDS LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

I tried modyfing train.py by modifying this line from

    pipeline = actionModel(config.classifier())

to

    pipeline = actionModel(config.classifier(max_iter=1000000))

Without success.

I also tried reducing the number of pictures if it could help the algorithm. They are 1280x720pixels, so maybe that's the issue? Also for a sequence of event, do I need to run train.py first then train_sequential.py aftewards, or can I just skip directly to train_sequential.py. Right now running the script throws this error :

Traceback (most recent call last):
  File "train_sequential.py", line 60, in <module>
    model = tf.keras.models.load_model('./models/lstm.h5')
  File "/home/hugo/anaconda3/envs/tensorrt/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/save.py", line 149, in load_model
    loader_impl.parse_saved_model(filepath)
  File "/home/hugo/anaconda3/envs/tensorrt/lib/python3.6/site-packages/tensorflow_core/python/saved_model/loader_impl.py", line 83, in parse_saved_model
    constants.SAVED_MODEL_FILENAME_PB))
OSError: SavedModel file does not exist at: ./models/lstm.h5/{saved_model.pbtxt|saved_model.pb}

Thanks!

Bus error when running iva.py

python3 iva.py inout/testvid.mp4

Bus error (core dumped)

On closer inspection this happens when iva.py imports torch
It might be related to pytorch/pytorch#4111

Using TensorFlow backend ... [Errno 111] Connection refused

Hello There,

I got an error when I tried to run "app.py" on my raspberry 3B+, which shows "Connection Refuse". Please see the log below. Thank you very much for the help!

pi@raspberrypi:~ $ cd YogAI/
pi@raspberrypi:~/YogAI $ python3 app.py
Using TensorFlow backend.
Traceback (most recent call last):
File "app.py", line 50, in
client.connect(HOST, PORT, 60)
File "/usr/local/lib/python3.5/dist-packages/paho/mqtt/client.py", line 839, in connect
return self.reconnect()
File "/usr/local/lib/python3.5/dist-packages/paho/mqtt/client.py", line 962, in reconnect
sock = socket.create_connection((self._host, self._port), source_address=(self._bind_address, 0))
File "/usr/lib/python3.5/socket.py", line 712, in create_connection
raise err
File "/usr/lib/python3.5/socket.py", line 703, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

Updating the Website Link

The current url (https://whttps//www.hackster.io/actionai/actionai-custom-tracking-multiperson-activity-recognition-fa5cb5ww.hackster.io/yogai/yogai-smart-personal-trainer-f53744) is invalid, and when cleaned, returns a 404.

You might want to consider replacing the URL with the currently functional one, https://www.hackster.io/actionai/actionai-custom-tracking-multiperson-activity-recognition-fa5cb5.

Cool work!

no module named 'screen_size'

when I run the inference.py ,I caught the problem no module named 'screen_size' ?
Would you please give me some help?

PoseNet from Google

Hello,
Do you think poses data can be taken using PoseNet model that Google released instead of trt_pose & TensorRT?

There are too many dependencies and installation process in too complex for TensorRT. I am unable to make progress because of it.

Please guide.

Thanks,
AB

This model is not working properly in my flutter application.

I have implemented this model but it doesn't seem to work. Is it compatible with flutter?

Is this able to classify actions like sitting, standing, and lying down?

First, thank you for sharing this amazing work.
I'm using Jetson TX2 and usb webcam.
I'm wondering that this library automatically classifies normal actions like I mentioned without additional training.
If it doesn't, does it only classify squatting and spinning?
I look forward to your reply!

train_sequential dataset

Hi @smellslikeml ,

I have read through the README.md provided but would like to clarify on some things that are not mentioned in it.

1. For the dataset, inside each subdirectory which is the label for which we want to classify the action, do we put in the sequence of images that constitute the action (eg: squatting) or do we only put in images of people in the squat position?
2. Related to the 1st Qns above, are we able to put in more than 1 sequence of squatting if provided we need to put in a sequence of images?
3. Do we only have to change the conf.py file when using train_sequential? what are the list of things we need to modify?

Thank you!

train_sequential.py is missing the PoseExtractor, or requires a different preprocess.py

Hello,

Just wondering if I'm missing something or if train_sequential.py is currently a template?

dataset = pd.read_csv('data/data.csv', index_col=None)
If data.csv is generated by preprocess.py it is a csv list of all image paths. Reshaping a data.csv with:

X_test = X_test.reshape(X_test.shape[0], pose_vec_dim, window)
How is it reshaped unless data.csv is constructed in a different format like an image being transformed and its feature vector being recorded.

docker

can i run this program without docker?

does this works for multiperson?

python script for general action recognition

I run iva.py and get the output for spin or squat.
But, I'd like to run the general action recognition.
Then, which python script should I use for the general action recognition?

[Question]Parsing good poses and poses from LSTM module ?

Hi , I am trying to extend theLSTM moduleand was wondering if it could be used to make a simple parser ,to differentiate Good posture and Bad Posture.

What i have is lets say Ref Video which lets say is a sequence of situps done correctly and test video where some situups are wrong (ex 3/10 ) . I am extracting the normalized keypoints in json format x, y co-ordinates and confidence score (Coco -posenet format )per frame for both reference and test video .

I want to extract the those 3 bad situps. Is it possible to do so ?? If so could you provide me the guidance.

Install Tensorflow 2.0

Hi,

Do you know how to install tensorflow 2.0 on Jetson. I did not find it. Thanks a lot.

change the motion identification dataset used

can anyone provide a tutorial on changing the motion identification dataset used?

TypeError: tuple indices must be integers or slices, not tuple

I am using jetson nano tx1.
can you please debug this error
here is my full terminal status
[TensorRT] WARNING: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[ WARN:0] global /home/hamza/opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
[ WARN:0] global /home/hamza/opencv/modules/videoio/src/cap_gstreamer.cpp (1184) setProperty OpenCV | GStreamer warning: GStreamer: unhandled property
2021-07-03 18:43:26.535555: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-07-03 18:43:42.980931: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-07-03 18:43:43.216263: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-03 18:43:43.252280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1742] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 194.55MiB/s
2021-07-03 18:43:43.252747: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-07-03 18:43:43.266841: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-07-03 18:43:43.267047: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-07-03 18:43:43.267235: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-07-03 18:43:43.267471: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-07-03 18:43:43.267641: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-07-03 18:43:43.277523: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-07-03 18:43:43.277755: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-03 18:43:43.277983: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-03 18:43:43.292821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1884] Adding visible gpu devices: 0
2021-07-03 18:43:44.054252: W tensorflow/core/platform/profile_utils/cpu_utils.cc:108] Failed to find bogomips or clock in /proc/cpuinfo; cannot determine CPU frequency
2021-07-03 18:43:44.090574: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ae52e50 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-07-03 18:43:44.090673: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-07-03 18:43:44.202240: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-03 18:43:44.202702: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ae71050 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-07-03 18:43:44.202767: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2021-07-03 18:43:44.242034: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-03 18:43:44.242247: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1742] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 194.55MiB/s
2021-07-03 18:43:44.242830: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-07-03 18:43:44.242999: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-07-03 18:43:44.243104: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-07-03 18:43:44.243232: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-07-03 18:43:44.243363: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-07-03 18:43:44.243472: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-07-03 18:43:44.243576: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-07-03 18:43:44.243784: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-03 18:43:44.244020: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-03 18:43:44.244120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1884] Adding visible gpu devices: 0
2021-07-03 18:44:09.863146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1283] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-07-03 18:44:10.010804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1289] 0
2021-07-03 18:44:10.010977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1302] 0: N
2021-07-03 18:44:10.386921: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-03 18:44:10.505899: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-03 18:44:10.606799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1428] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 92 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
WARNING:tensorflow:Layer lstm will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
Traceback (most recent call last):
File "iva.py", line 275, in
matched, unmatched_trackers, unmatched_detections = tracker_match(track_boxes, [b[0] for b in bboxes])
File "iva.py", line 129, in tracker_match
if(d not in matched_idx[:,1]):
TypeError: tuple indices must be integers or slices, not tuple

Error when try to run teachable_machine.py

None Traceback (most recent call last): File "teachable_machine.py", line 54, in <module> for body in pose_list: TypeError: 'NoneType' object is not iterable Segmentation fault (core dumped)
i got this error when run the program, i already make sure that video contain person and exist.

It seems that train.py can't save a model

I got model but it has nothing and is just 2.0kB size.
I think that's why I got this error running iva.py
OSError: SavedModel file does not exist at: models/classifier.sav/{saved_model.pbtxt|saved_model.pb}
Has anyone experienced the same thing as me?

Not supported for Ubuntu 22.04.1 LTS

command: sudo add-apt-repository ppa:smellslikeml/ppa

Error message
Reading package lists... Done
E: The repository 'https://ppa.launchpadcontent.net/smellslikeml/ppa/ubuntu jammy Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

activity recognition on a live cctv camera.

I would like to do activity recognition on a live cctv camera. In the Read.md section it is mentioned that the deployment file has reference codes to do it, where can I find this file.

Fail to converge warning when training on custom dataset

I've followed the steps on the readme. train.py was able to load the images from dataset, but it outputs the following message:

INFO: Initialized TensorFlow Lite runtime.
/home/username/anaconda3/envs/actionai/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

Any advice on how to solve this?

OS: Ubuntu 18.04 LTS

image = np.expand_dims(np.asarray(image).astype(self.input_details[0]['dtype'])[:, :, :3], axis=0)

return self.fit(X, y, **fit_params).transform(X) File "/home/cccxx/sss/Act/ActionAI/transformer.py", line 46, in transform image = np.expand_dims(np.asarray(image).astype(self.input_details[0]['dtype'])[:, :, :3], axis=0)
hello i got sort of trouble during running train.py, how suppose to solve this?

Which version of TensorRT is used for this project

When I try to run iva.py, I get an error, and the specific error returned is
[05/23/2023-08:41:34] [TRT] [E] 1: [stdArchiveReader.cpp::StdArchiveReader::30] Error Code 1: Serialization (Serialization assertion magicTagRead == magicTag failed.Magic tag does not match) [05/23/2023-08:41:34] [TRT] [E] 4: [runtime.cpp::deserializeCudaEngine::50] Error Code 4: Internal Error (Engine deserialization failed.) Traceback (most recent call last): File "iva.py", line 8, in <module> from model import * File "/home/wheeltec/ActionAI/model.py", line 73, in <module> model_trt.load_state_dict(torch.load(MODEL_PATH)) File "/home/wheeltec/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1037, in load_state_dict load(self) File "/home/wheeltec/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1032, in load state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs) File "/usr/local/lib/python3.6/dist-packages/torch2trt-0.4.0-py3.6-linux-aarch64.egg/torch2trt/torch2trt.py", line 591, in _load_from_state_dict self.context = self.engine.create_execution_context() AttributeError: 'NoneType' object has no attribute 'create_execution_context'

I have a question...

Does this code run in windows?
If anyone sees this i would appreciate it if you answer thank you!!

Please don't waste time on this repo.

Please correct me if I am wrong, but it seems to me that the code is very incomplete, and there are many mistakes in the README file. I have no idea why this repo is getting so many stars.

For example, train.py uses a classifier.sav for classifying actions, which is not provided. The model definition is not provided as well so it's impossible to train from scratch either.
Also, train_sequential.py doesn't include a pose estimator, so it doesn't work with the data.csv generated by preprocessing.py.

One will need to modify a lot if he or she follows the instructions in README.

An error when installing the xgboost

hi, I got an error when I install the xgboost on my raspberry 3B+,
the error code is : command"python setup.py egg_info"failed with error code 1 in /tmp/pip-install-v8ptnvpp/xgboost/
Could you give me some help?