ibaigorordo/pykinectazure

Python library to run Kinect Azure DK SDK functions

pyKinectAzure

Python 3 library for the Azure Kinect DK sensor-SDK.

Similar solutions

Part of the ideas in this repository are taken from following repositories:

• pyk4a: Really nice and clean Python3 wrapper for the Kinect Azure SDK.

• Azure-Kinect-Python: More complete library using ctypes as in this repository, however, examples about how to use the library are missing and the library is harder to use.

The objective of this repository is to combine the strong points of both repositories by creating a easy to use library that allows the use of most of the functions of the Kinect Azure. Also, to create sample programs to showcase the uses of the library

Prerequisites

• Azure-Kinect-Sensor-SDK: required to build this library. To use the SDK, refer to the installation instructions here.
• ctypes: required to read the library.
• numpy: required for the matrix calculations
• opencv-python: Required for the image transformations and visualization.

Installation

pip install pykinect_azure

How to use this library

• The library has been tested in Windows 10 and Ubuntu 20.04 with the Kinect Azure SDK 1.4.0 and 1.4.1, it should also work with other operating systems.

  • Windows: When using the pyKinectAzure class, it requires the path to the k4a.dll module, make sure that the path is the correct one for your Kinect Azure SDK version. By default the path (module_path) is set to C:\\Program Files\\Azure Kinect SDK v1.4.0\\sdk\\windows-desktop\\amd64\\release\\bin\\k4a.dll.

  • Linux: When using the pyKinectAzure class, it requires the path to the k4a.so module, make sure that the path is the correct one for your Kinect Azure SDK version. When using Linux set module_path to /usr/lib/x86_64-linux-gnu/libk4a.so, please follow the instruction from microsoft to install the right packages.

  • Nvidia Jetson: When using the pyKinectAzure class, it requires the path to the k4a.so module, make sure that the path is the correct one for your Kinect Azure SDK version. When using Nvidia Jetson set module_path to to '/usr/lib/aarch64-linux-gnu/libk4a.so', please follow the instruction from microsoft to install the right packages.

• The pyKinectAzure class is a wrapper around the _k4a.py module to make the library more understandable. However, the pyKinectAzure class still contains few methods from the Kinect Azure SDK

• The _k4a.py module already contains all the methods in the Kinect Azure SDK. So, for more advanced of the Kinect Azure SDK check the _k4a.py module.

Examples

For an example on how to obtain and visualize the depth data from the Azure Kinect DK check the exampleDepthImageOpenCV.py script.

git clone https://github.com/ibaiGorordo/pyKinectAzure.git
cd pyKinectAzure/examples
python exampleDepthImageOpenCV.py

Also, there is an example to obtain and visualize the smooth depth from the Azure Kinect DK check the exampleSmoothDepthImageOpenCV.py script.

python exampleSmoothDepthImageOpenCV.py

note: when you are dealing on the linux platform, please insure that the user have permission to the usb devices, or always execute under the root permission by adding sudo ahead.

Contribution

Feel free to send pull requests.

Bug reports are also appreciated. Please include as much details as possible.

TODO:

Wrappers for the Kinect Azure data

• Create wrapper to read depth images.
• Create wrapper to read Infrared images.
• Create wrapper to read IMU data.
• Create function to convert image buffer to image depending on the image type.
• Create wrapper to transform depth image to color image.
• Create wrapper to transform depth image to 3D point cloud.
• Create funtion to visualize 3D point cloud.

Create examples

• Example to visualize depth images.
• Example to visualize passive IR images.
• Example to plot IMU data.
• Example to visualize Depth as color image.
• Example to overlay depth color with alpha over real image.
• Example to visualize 3D point cloud

Body tracking

• Create library for the body tracking SDK similar the same way as the current library.
• Combine image and skeleton data.
• Generate 3D skeleton visualization.

Future ideas

• Run Deep Learning models on Kinect data (Openpose 3D skeleton, semantic segmentation with depth, monocular depth estimation validation)
• Track passive infrared marker for motion capture analysis

pykinectazure's Issues

Why I get the depth image with a 2D array?

Hello. I run your project on my jeston agx xavier which is a great job! But I confused that the depth image is a 2D array

Update to Body Tracking v1.1.0

Hi, any idea or advice on updating the body tracking part to support the latest v1.1.0 API? They seems to add some new things to the DLLs. I have tried to implement it but no didn't get it work :(

I wish you all good health and all the best

nothing

How to use device_get_serialnum

Hi,
I want to get device serialnum,but got this error.
What's wrong about it? I have no idea about ctypes.

def device_get_serialnum(self):
serial_number = 0
serial_number_size = 0
self.k4a.k4a_device_get_serialnum(self.device_handle,serial_number,serial_number_size)

[2020-07-02 15:43:50.392] [error] [t=23504] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (292): Invalid argument to depth_get_device_serialnum(). serial_number_size == NULL
[2020-07-02 15:43:50.392] [error] [t=23504] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (292): depth_get_device_serialnum() returned failure.
[2020-07-02 15:43:50.392] [error] [t=23504] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (962): depth_get_device_serialnum(device->depth, serial_number, serial_number_size) returned failure in k4a_device_get_serialnum()

Is it possible to display the coordinates of the joint as a value?

Hello!

Thank you very much for the sample!

I would like to use this library to create a program that displays the coordinates of joints and use those coordinates to do various things.

The question is, is it possible to display joint coordinates as values ​​in this wonderful library you created?

Also, I am currently trying this and borrowed a little Sample and wrote it like this.
I will post some of them, but I am sorry for the dirty code.

# Initialize the library, if the library is not found, add the library path as argument
pykinect.initialize_libraries(track_body=True)

# Modify camera configuration
device_config = pykinect.default_configuration
device_config.color_resolution = pykinect.K4A_COLOR_RESOLUTION_1080P
device_config.depth_mode = pykinect.K4A_DEPTH_MODE_WFOV_2X2BINNED
#print(device_config)

# Start device
device = pykinect.start_device(config=device_config)

# Start body tracker
bodyTracker = pykinect.start_body_tracker()

cv2.namedWindow('Color image with skeleton',cv2.WINDOW_NORMAL)
while True:
	
	# Get capture
	capture = device.update()

	# Get body tracker frame
	body_frame = bodyTracker.update()

	# Get the color image
	ret, color_image = capture.get_color_image()

	# Get the num bodies
	num_bodies_frame = body_frame.get_num_bodies()
	#print(num_bodies_frame)

	# Get the index map
	#body_frame_index_map = body_frame.get_transformed_body_index_map

	## Get Joint position
	for i in range(num_bodies_frame):
		num_body_frame_skeleton = body_frame.get_body_skeleton(index = i)
		num_body_frame_id = body_frame.get_body_id(index = i)
		#print(num_body_frame_id)
		for a in range(num_body_frame_id):
			get_body_joint = body_frame.get_body(bodyIdx = a)
			print(get_body_joint)`

When I ran it, I got an error like this.

PS C:\Users\U4\pyKinectAzure\examples> python c:\Users\U4\pyKinectAzure\examples\exampleBodyTrackingColorCamera.py Traceback (most recent call last): File "c:\Users\U4\pyKinectAzure\examples\exampleBodyTrackingColorCamera.py", line 54, in <module> print(get_body_joint) File "C:\Users\U4\pyKinectAzure\examples\..\pykinect_azure\k4abt\body.py", line 39, in __str__ f"\tposition: [{self.position.x},{self.position.y},{self.position.z}]\n" AttributeError: 'Body' object has no attribute 'position

If you don't mind, please let me know the solution.
I'm sorry, but I would appreciate it if you could answer as soon as possible.
Thank you!

Convert_2d_to_3d

How to obtain 3d coordinates (x, y ,z) of a pixel (x,y) in the image.

Invalid argument to image_dec_ref()

[2020-06-29 14:26:57.835] [error] [t=16976] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS: 16077 type:Depth
[2020-06-29 14:26:57.871] [error] [t=16976] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS: 49422 type:Depth
[2020-06-29 14:26:57.895] [error] [t=24284] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\image\image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t 0000000000000000
[2020-06-29 14:26:57.895] [error] [t=24284] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\image\image.c (363): Invalid argument to image_dec_ref(). image_handle (0000000000000000) is not a valid handle of type k4a_image_t

Anyway to crop out image for specific body parts such as head, torso?

Looking at the original SDK documentation, I've not found a way to find pixel values in a depth/color image for specific body parts such as the head. I can relate the specific joints for the body parts using the skeleton API, but how do I get the general area of pixels for combined joints?

How to save the result of Body Tracking as .json or .skeleton?

How to save the result of Body Tracking as .json or .skeleton?

_k4arecord.k4arecord.k4a_playback_open ctype error

Hi, thank you for this excellent work. I'm trying to use the repo to do body tracking using recorded MKV.

However, I failed with ctype data-type check at a strange point, _k4arecord.k4arecord.k4a_playback_open.

Here is my code (not well tested):

_k4a.k4a.setup_library(modulePath)

k4a_record = _k4arecord.k4arecord(modulePath)
capture_handle = _k4a.k4a_capture_t()
calibration_handle = _k4a.k4a_calibration_t()

playback_handle = _k4arecordTypes.k4a_playback_t()

"""
my error occurs here:
"""
k4a_record.k4a_playback_open(ctypes.create_string_buffer(os.path.join(root_path, "kinect", device, "out.mkv").encode('utf-8')), playback_handle)

# k4a.k4a_calibration_get_from_raw(ctypes.create_string_buffer(raw_calibration.encode("utf-8")), ctypes.c_size_t(len(raw_calibration)), _k4atypes.K4A_DEPTH_MODE_NFOV_UNBINNED, _k4atypes.K4A_COLOR_RESOLUTION_1536P, calibration_t)

k4a_record.k4a_playback_get_calibration(playback_handle, calibration_handle)
body_tracker = kinectBodyTracker(bodyTrackingModulePath, calibration_handle, _k4abt.K4ABT_DEFAULT_MODEL)
while True:
    k4a_record.k4a_playback_get_next_capture(playback_handle, capture_handle)

    body_tracker.enqueue_capture(capture_handle)
    body_tracker.detectBodies()

    for body in body_tracker.bodiesNow:
        body_tracker.printBodyPosition(body)
        print(body.skeleton)

If argument 2 is playback_handle, it says

argument 2: <class 'TypeError'>: expected LP_LP__handle_k4a_playback_t instance instead of LP__handle_k4a_playback_t

If ctypes.pointer(playback_handle), it says

argument 2: <class 'TypeError'>: expected LP_LP__handle_k4a_playback_t instance instead of LP_LP__handle_k4a_playback_t

Which is very confusing, because it seems like ctypes.pointer(playback_handle) just have the right data type.

I also tried

isinstance(_k4arecordTypes.k4a_playback_t, ctypes.POINTER(_k4arecordTypes.k4a_playback_t))

True

What should I do?

Thanks!

TypeError: shape must be a tuple buffer_array = np.ctypeslib.as_array(buffer_pointer,shape=[image_size])

I met problem with running the example. Please help me?

'''

    disable_streaming_indicator: False
    (True or False). Streaming indicator automatically turns on when the color or depth camera's are in use.

Traceback (most recent call last):
File "exampleColorImageOpenCV.py", line 40, in
color_image = pyK4A.image_convert_to_numpy(color_image_handle)
File "../pyKinectAzure\pyKinectAzure.py", line 490, in image_convert_to_numpy
buffer_array = np.ctypeslib.as_array(buffer_pointer,shape=[image_size])
File "D:\Anaconda3\lib\site-packages\numpy\ctypeslib.py", line 436, in as_array
return array(obj, copy=False)
TypeError: shape must be a tuple
'''

exampleRecording.py not working

Hi,

I am currently working on Linux Ubuntu 20.04

I trying to make work the exampleRecoring.py but I get an error which is :

Which makes me believe that a library is wrongly called. This is strange since all other example worked for me.

Best Regards,

William

A memory leak problem on the new refactor version.

Hi, thanks for updating this refactor version. The playback feature with body tracking works well for me.

But I think I got a memory leaking problem. On my computer, all example codes would cause a memory leak.
For the exampleColorImage.py I manually added the capture.reset() method to solve the problem. However, I find it is a bit tricky to release a body_frame, I need also to create a new Frame after each release, which significantly drops the performance.

Can you help me? And may I know your computer OS and python version? I'm afraid this may be just because of my legacy version of python (I'm working on Windows 10 x64 with python 3.7.7).

[BodyTrackingColorCamera] 'Body tracker capture enqueue failed!'

Hi I tried running the exampleBodyTrackingColorCamera.py
It did work, and video window was opened, but it shutdown after a while
it occurred below these errors

[2022-03-31 11:01:59.335] [error] [t=9820] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\image\image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t 0000000000000000
[2022-03-31 11:01:59.358] [error] [t=9820] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\image\image.c (389): Invalid argument to image_get_buffer(). image_handle (0000000000000000) is not a valid handle of type k4a_image_t
[2022-03-31 11:01:59.359] [error] [t=9820] [K4ABT] ..\src\TrackerHost\DepthFrameBlobK4A.cpp (13): Initialize(). Get depth buffer from the capture handle failed!
[2022-03-31 11:01:59.360] [error] [t=9820] [K4ABT] ..\src\TrackerHost\TrackerHost.cpp (275): EnqueueCapture(). Initialize DepthFrameBlob failed!
Body tracker capture enqueue failed!
  File "BodyTrackingColorCamera.py", line 32, in <module>
    body_frame = bodyTracker.update()
  File "..\pykinect_azure\k4abt\tracker.py", line 38, in update
    self.enqueue_capture(Device.capture.handle(), timeout_in_ms)
  File "..\pykinect_azure\k4abt\tracker.py", line 43, in enqueue_capture
    _k4abt.VERIFY(_k4abt.k4abt_tracker_enqueue_capture(self._handle, capture_handle, timeout_in_ms), "Body tracker capture enqueue failed!")
  File "..\pykinect_azure\k4abt\_k4abt.py", line 157, in VERIFY
    traceback.print_stack()

when i tried running the exampleBodyTracking.py, these error won't happen

Did anyone know how to solve this problem?
thank you lot!

[BodyTracking] AttributeError: 'Tracker' object has no attribute '_handle'

My Desktop information

• OS with Version: Ubuntu 20.04.3 LTS 64-bit
• CPU: Intel® Core™ i7-10700 CPU @ 2.90GHz × 16 (No independent GPU)
• SDK Version: 1.4.1
• Packages version: libk4a1.4, libk4abt1.1.0, k4a-tools1.4.1

I can run the k4abt_simple_3d_viewer by calling k4abt_simple_3d_viewer CPU in my terminal. Also no problem with the running of k4aviewer by calling it in the terminal.
Python files except for body tracking parts can be run normally, such as exampleColorImage.py and exampleDepthImage.py and so on.

To reproduce:

1. Replace the k4abt.so in utils/utils.py/def get_k4abt_module_path with libk4abt.so
2. Replace the k4abt_tracker_default_configuration.processing_mode in k4abt/_k4abtTypes.py with K4ABT_TRACKER_PROCESSING_MODE_CPU
3. Run any python file for Body Tracking.

Error log:

[2022-02-11 11:08:38.189] [error] [t=3332] [K4ABT] ../src/TrackerHost/TrackerHost.cpp (185): Create(). /tmp/onnxruntime/onnxruntime/core/providers/cuda/cuda_call.cc:123 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cudaError; bool THRW = true] /tmp/onnxruntime/onnxruntime/core/providers/cuda/cuda_call.cc:117 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cudaError; bool THRW = true] CUDA failure 35: CUDA driver version is insufficient for CUDA runtime version ; GPU=32745 ; hostname=nmed-OptiPlex-5080 ; expr=cudaSetDevice(device_id_); 

[2022-02-11 11:08:38.189] [error] [t=3332] [K4ABT] ../src/sdk/k4abt.cpp (38): tracker->Create(sensor_calibration, config) returned failure in k4abt_tracker_create()
Body tracker initialization failed!
  File "/home/nmed/OneDrive/CODE/Python/Azure-Kinect-python/examples/exampleBodyTracking.py", line 22, in <module>
    bodyTracker = pykinect.start_body_tracker()
  File "/home/nmed/OneDrive/CODE/Python/Azure-Kinect-python/examples/../pykinect_azure/pykinect.py", line 63, in start_body_tracker
    return Tracker(Device.calibration, model_type)
  File "/home/nmed/OneDrive/CODE/Python/Azure-Kinect-python/examples/../pykinect_azure/k4abt/tracker.py", line 11, in __init__
    self._handle = self.create(model_type)
  File "/home/nmed/OneDrive/CODE/Python/Azure-Kinect-python/examples/../pykinect_azure/k4abt/tracker.py", line 66, in create
    _k4abt.VERIFY(_k4abt.k4abt_tracker_create(self.calibration.handle(), tracker_config, tracker_handle), "Body tracker initialization failed!")
  File "/home/nmed/OneDrive/CODE/Python/Azure-Kinect-python/examples/../pykinect_azure/k4abt/_k4abt.py", line 157, in VERIFY
    traceback.print_stack()
Exception ignored in: <function Tracker.__del__ at 0x7fe997559d30>
Traceback (most recent call last):
  File "/home/nmed/OneDrive/CODE/Python/Azure-Kinect-python/examples/../pykinect_azure/k4abt/tracker.py", line 15, in __del__
    self.destroy()
  File "/home/nmed/OneDrive/CODE/Python/Azure-Kinect-python/examples/../pykinect_azure/k4abt/tracker.py", line 27, in destroy
    if self.is_valid():
  File "/home/nmed/OneDrive/CODE/Python/Azure-Kinect-python/examples/../pykinect_azure/k4abt/tracker.py", line 18, in is_valid
    return self._handle
AttributeError: 'Tracker' object has no attribute '_handle'

I would appreciate it if any suggestion you can give to me.

help!AttributeError: undefined symbol: k4a_calibration_color_2d_to_depth_2d

runfile('/home/zhengzh/DK/DK_C++/pyKinectAzure-master/examples/exampleDepthImageOpenCV.py', wdir='/home/zhengzh/DK/DK_C++/pyKinectAzure-master/examples')
Traceback (most recent call last):

File "", line 1, in
runfile('/home/zhengzh/DK/DK_C++/pyKinectAzure-master/examples/exampleDepthImageOpenCV.py', wdir='/home/zhengzh/DK/DK_C++/pyKinectAzure-master/examples')

File "/home/zhengzh/anaconda3/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 705, in runfile
execfile(filename, namespace)

File "/home/zhengzh/anaconda3/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "/home/zhengzh/DK/DK_C++/pyKinectAzure-master/examples/exampleDepthImageOpenCV.py", line 16, in
pyK4A = pyKinectAzure(modulePath)

File "../pyKinectAzure/pyKinectAzure.py", line 18, in init
self.k4a = _k4a.k4a(modulePath)

File "../pyKinectAzure/_k4a.py", line 481, in init
self.k4a_calibration_color_2d_to_depth_2d = dll.k4a_calibration_color_2d_to_depth_2d

File "/home/zhengzh/anaconda3/lib/python3.6/ctypes/init.py", line 361, in getattr
func = self.getitem(name)

File "/home/zhengzh/anaconda3/lib/python3.6/ctypes/init.py", line 366, in getitem
func = self._FuncPtr((name_or_ordinal, self))

AttributeError: /home/zhengzh/Azure-Kinect-Sensor-SDK/build/bin/libk4a.so: undefined symbol: k4a_calibration_color_2d_to_depth_2d

hope can provide some examples for visualize multi-cameras' synching output

I use pyAzureKinect to Get Calibration but it always null, could you tell me how to get calibration's Intrinsic

microsoft/Azure-Kinect-Sensor-SDK#1488

Unable to read multiple cameras data

I have two kinect cameras and I want to read their data at the same time and display it, but the data I can only read is the same camera. I don't know where is my mistake. My code is as follows.

Audio support

Hey there,

is there a way to capture the soundtrack from the Kinect?
In this sense, is it also possible to activate or deactivate corresponding sensors, such as the microphone?
If not, is this possibly planned for the future?

Nice work btw.
Thank you.

Failed to load body tracker library k4abt.so: cannot open shared object file: No such file or directory

Hi everyone

we have trouble with this error, I've checked the PATH, and to be sure, we've export PATH="/path/to/module"

can anybody help? I appreciate that, I guess it should be easy to resolve

How do I extract depth data?

I can't seem to find a way to extract the depth data from kinect DK.
I found a few blogs that talk about the use of getRawDepth() but it's not in Kinect DK I think.

Running Playback on synced .mkv files.

Hi! I tried running the examplePlaybackBodyTracker.py file using an mkv file from a synced recording I did using k4arecorder.

I'm getting the following error:
[2022-03-15 11:59:44.460] [error] [t=24870] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil) [2022-03-15 11:59:44.460] [error] [t=24870] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/image/image.c (389): Invalid argument to image_get_buffer(). image_handle ((nil)) is not a valid handle of type k4a_image_t [2022-03-15 11:59:44.460] [error] [t=24870] [K4ABT] ../src/TrackerHost/DepthFrameBlobK4A.cpp (13): Initialize(). Get depth buffer from the capture handle failed! [2022-03-15 11:59:44.460] [error] [t=24870] [K4ABT] ../src/TrackerHost/TrackerHost.cpp (275): EnqueueCapture(). Initialize DepthFrameBlob failed! Body tracker capture enqueue failed! File "examplePlaybackBodyTracker.py", line 32, in <module> body_frame = bodyTracker.update(capture=capture) File "../pykinect_azure/k4abt/tracker.py", line 36, in update self.enqueue_capture(capture.handle(), timeout_in_ms) File "../pykinect_azure/k4abt/tracker.py", line 43, in enqueue_capture _k4abt.VERIFY(_k4abt.k4abt_tracker_enqueue_capture(self._handle, capture_handle, timeout_in_ms), "Body tracker capture enqueue failed!") File "../pykinect_azure/k4abt/_k4abt.py", line 157, in VERIFY traceback.print_stack()

For some reason when it starts capturing frames from the mkv file it is empty. This only happen with synced mkv files, if I do it with a recording without synching the kinects, it works fine.

I'm just getting familiar with the code, do you ahve any idea how it could be fixed?

Thanks!

Projecting depth image to color space using Playback API

The transformed depth image seems to have that hexagonal artifact. Is this expected behaviour?

Here is the transformed-depth samples:

And here is the original color one:

Let me know if you need anymore info.

How to get point cloud and camera internal parameters?

Thank you very much for your pyKinectAzure, but I have encountered some problems when using it.
I had a problem calling the function transform_depth_image_to_point_cloud and the point_cloud_image I got was None.
Finally, I wonder if pyKinectAzure can read the camera's internal parameters directly?
Hope to get your answer, thank you!

Here's the code I used to get the point cloud:
pyK4A.device_open()

# Modify camera configuration
device_config = pyK4A.config
device_config.color_resolution = _k4a.K4A_COLOR_RESOLUTION_1080P
device_config.depth_mode = _k4a.K4A_DEPTH_MODE_WFOV_2X2BINNED
print(device_config)

# Start cameras using modified configuration
pyK4A.device_start_cameras(device_config)
    pyK4A.device_get_capture()
    depth_image_handle = pyK4A.capture_get_depth_image()
    point_cloud_image_handle = pyK4A.transform_depth_image_to_point_cloud(depth_image_handle)
    point_cloud_image = pyK4A.image_convert_to_numpy(point_cloud_image_handle)

Body tracking?

Does this library support body tracking? The Readme seems to suggest so by suggesting that other similar solutions lack body tracking, and I see a branch to "add body tracking" but can't find any documentation for how to use body tracking, if indeed it is supported.

Can we save the onnxruntime tensorrt engine for body tracking?

Hi Ibai, many thanks for provising this brilliant azure kinect python library. It works well for me!

I'm wondering if it's possible to save the onnxruntime tensorrt engine cache for the body tracking? Tensorrt always takes a long time to optimize and build engine, and I think it would be better if we can directly load a saved engine for running.

onnxruntime do have this feature, however, how can we do it here? Do you have a suggestion?
Thanks

Problems with multiprocessing

I am using this module, it works fine for me. Thanks for your great work.

However, it seems that the pyK4A.device_start_cameras() doesn't work in sub process or sub thread, while using it in main process will cause a block which is not acceptable.

Could you please give me some advice about this? Or are there any other solution works with multi-process?

Thanks!

Linux: wrong library name

In utils.py, lines 30 - 32, you have:

# For non-Arm chips, first check if it is running linux
if platform.system().lower() == 'linux':
    return 'k4abt.so'

However, this is not the correct file name. The correct name should be:

# For non-Arm chips, first check if it is running linux
if platform.system().lower() == 'linux':
    return 'libk4abt.so'

utils.py.zip

capture.get_pointcloud(): BGRA32 data unexpected + causing reshape failure?

Hello! I am trying to do some basic I/O with a very short and simple 5-second .mkv recording. I am simply trying to get a point cloud from a single frame of the recording.

Here is the code I am using:

import numpy as np
from pykinect_azure import pykinect

# Initialize the library, if the library is not found, add the library path as argument
pykinect.initialize_libraries()

# Start playback
playback = pykinect.start_playback("test.mkv")

playback_config = playback.get_record_configuration()
print(playback_config)

# Get camera capture
while(playback.isOpened()):
	capture = playback.update()
	_, depth = capture.get_depth_image()
	_, color = capture.get_color_image()
	if(np.any(depth) and np.any(color)):
		break

capture.get_pointcloud()

Here is the error:

Somehow, the code is finding BGRA32-formatted data in the recording. I don't really understand why, especially given the following output of printing playback.get_record_configuration():

Record configuration: 
	color_format: 0 
	(0:JPG, 1:NV12, 2:YUY2, 3:BGRA32)

	color_resolution: 5 
	(0:OFF, 1:720p, 2:1080p, 3:1440p, 4:1536p, 5:2160p, 6:3072p)

	depth_mode: 2 
	(0:OFF, 1:NFOV_2X2BINNED, 2:NFOV_UNBINNED,3:WFOV_2X2BINNED, 4:WFOV_UNBINNED, 5:Passive IR)

	camera_fps: 2 
	(0:5 FPS, 1:15 FPS, 2:30 FPS)

	color_track_enabled: True 
	(True of False). If Color camera images exist

	depth_track_enabled: True 
	(True of False). If Depth camera images exist

	ir_track_enabled: True 
	(True of False). If IR camera images exist

	imu_track_enabled: True 
	(True of False). If IMU samples exist

	depth_delay_off_color_usec: 0 us. 
	Delay between the color image and the depth image

	wired_sync_mode: 0
	(0:Standalone mode, 1:Master mode, 2:Subordinate mode)

	subordinate_delay_off_master_usec: 0 us.
	The external synchronization timing.

	start_timestamp_offset_usec: 200244 us. 
	Start timestamp offset.

I am using Ubuntu 20.04, VS Code (remote/SSH into Ubuntu system), and I installed the Azure Kinect DK SDK using the apt install commands (I did not compile or copy any files myself). Am I doing something wrong?

The test.mkv file was recorded using the k4arecorder command line tool.

This is SO GREAT

What an amazing python project and this can be used on linux too!
I have tested on Ubuntu 20.04, just modify the modulePath into "/usr/lib/x86_64-linux-gnu/libk4a.so" and it works smoothly just like on windows.

Feature Request: Process video using the Kinect body tracker on image/depth sequences

I have seen the record playback feature. However, could we add a feature To utilize the body tracker from a different data source —-particularly, a sequence of RGBD images (either pngs or open3d objects). Intrinsics could be passed into the camera configuration.

Question: How was this tested on Ubuntu 20.04

Hi,

I am trying to setup my Azure Kinect with Ubuntu 20.04 but it seems that there are no official Ubuntu binaries for the SDK yet (microsoft/Azure-Kinect-Sensor-SDK#1263). Also commonly found suggestions to go for the Ubuntu 18.04 version does not work for me.

So basically, I was wondering how I can setup the Azure Kinect on Ubuntu 20.04 ... would really like to use your tool.

Thanks in advance

Why the coordinate of the depth camera is the opposite of the RGB?

I confused that the coordinate of the depth camera after transformed_depth() is the opposite of the RGB, for example, the depth value of the coordinate color[x][y] is depth[y][x].

Memory overflow problem

Hi, I am trying to run some code on a remote server using ssh.
I think the code cannot access the window information in this case.
So, I tried to run a code without lines corresponding to window visualization (e.g cv2.namedWindow('Depth Image',cv2.WINDOW_NORMAL)).

But, it caused the memory overflow to my computer with 16GB CPU even if I used the only 40 seconds mkv video recorded using Kinect DK camera.

I got some error messages as follows when I run examplePlayback.pywithout lines of 20 and 30.

[2022-02-07 03:48:24.141] [error] [t=9425] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/tewrapper/tewrapper.c (61): transform_engine_start_helper(). Transform engine create and initialize failed with error code: 204.
[2022-02-07 03:48:24.142] [error] [t=9425] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/tewrapper/tewrapper.c (68): teresult == K4A_DEPTH_ENGINE_RESULT_SUCCEEDED returned failure in transform_engine_start_helper()
[2022-02-07 03:48:24.142] [error] [t=9425] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/tewrapper/tewrapper.c (86): transform_engine_start_helper(tewrapper) returned failure in transform_engine_thread()
[2022-02-07 03:48:24.142] [error] [t=9204] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/tewrapper/tewrapper.c (313): tewrapper_create(). Transform Engine thread failed to start
[2022-02-07 03:48:24.142] [error] [t=9204] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/transformation/transformation.c (637): transformation_context->tewrapper != NULL returned failure in transformation_create()

Can somebody know how to solve this problem?

Typo in k4abt_types

When specifying the list of joint names, the right eye is "rigth eye" rather than "right eye".

Just wanted to point this typo out.

I love all the work you do.

How to get the original depth value?

In the code, the saved depth image is the format of uint8, I want to know how to get the real depth value for this uint8 pcitures.

Using code causes memory leaks.

test.zip

this is minimum code for test

I'm going to create a streaming program using 'ibaiGorodo/pyKinectAzure'. Images need to be received from the device on a frame basis, but memory leakage occurs when images are received.

Memory leakage occurs again when using 'fromImage()' in pyside.

how can i solve this problem?

also I'm sorry for my poor English.

Do you have example about color to depth?

Could you give me a example or a function of transform_color_to_depth?I tried to imitate the writing style of transform_depth_to_color, but failed.

Body tracker initialization failed!

Hello

I am trying to run your demos. Everything is working except of
exampleBodySegmentation
exampleBodyTracking
exampleBodyTrackingColorCamera

I get this error:

Body tracker initialization failed!
File "", line 1, in
File "C:\Users\Interrupt\AppData\Local\Programs\Python\Python37\lib\idlelib\run.py", line 155, in main
ret = method(*args, **kwargs)
File "C:\Users\Interrupt\AppData\Local\Programs\Python\Python37\lib\idlelib\run.py", line 548, in runcode
exec(code, self.locals)
File "C:\Users\Interrupt\pyKinectAzure\examples\exampleBodyTrackingColorCamera.py", line 35, in
pyK4A.bodyTracker_start(bodyTrackingModulePath)
File "../pyKinectAzure\pyKinectAzure.py", line 49, in bodyTracker_start
self.body_tracker = kinectBodyTracker(bodyTrackerModulePath, depthSensorCalibration, modelType)
File "../pyKinectAzure\kinectBodyTracker.py", line 41, in init
self.initializeTracker()
File "../pyKinectAzure\kinectBodyTracker.py", line 103, in initializeTracker
_k4abt.VERIFY(self.k4abt.k4abt_tracker_create(self.sensor_calibration, self.tracker_config, self.tracker_handle), "Body tracker initialization failed!")
File "../pyKinectAzure_k4abt.py", line 117, in VERIFY
traceback.print_stack()

My aim is to print the joint coordinates. Maybe you can help me?

(I've installed Azure Kinect SDK v1.4.1 and Azure Kinect Body Tracking SDK 1.1.0 + made the PATH entrys)

Is possible do Body Tracking from Record?

Hi I have a question, is it possible to get body tracking functionalities from a record?

transformation_depth_image_to_color_camera

Hi:
I want to transform depth_image to color camera, Can you give me example?
Thanks

How to read a record?

Hello I have a question, how to read data from the video record? Because I check the samples and didn't see a sample of offline processing. Thank you for your help.

Error while using exampleBodySegmentation.py

Hi I am currently working on a Azure Kinect project and found your library.
I tried using exampleBodySegmentation.py but I got a error saying

[2021-01-25 13:51:02.859] [error] [t=23784] [K4ABT] D:\a\1\s\src\TrackerHost\TrackerHost.cpp (467): VerifyDependencies(). Cannot locate onnxruntime.dll
[2021-01-25 13:51:02.859] [error] [t=23784] [K4ABT] D:\a\1\s\src\TrackerHost\TrackerHost.cpp (129): Create(). At least one dependent library is missing!
[2021-01-25 13:51:02.859] [error] [t=23784] [K4ABT] D:\a\1\s\src\sdk\k4abt.cpp (38): tracker->Create(sensor_calibration, config) returned failure in k4abt_tracker_create()
Body tracker initialization failed!

I have Azure Kinect Body Tracking SDK v1.0.0 and Azure Kinect SDK v1.4.1 installed at the correct path.
I added the files to the Path variable in Environment variable and it still didn't work.

It would be thankful if you could find the reasons to these problem.

Thank you for you time.

How can we use the API in this repo to get the realtime depth image infomation and turn it into .bin file?

Glad to learn your excellent jod. How can we use the API in this repo to get the realtime depth image infomation and turn it into .bin file? @ibaiGorordo

would you add record and playback function？

Hi：
I added record_create function in _k4a.py, but got error，function not found
would you add record and playback function please？Thank You

        self.k4a_record_create = dll.k4a_record_create
        self.k4a_record_create.restype = k4a_result_t
        self.k4a_record_create.argtypes = (ctypes.POINTER(ctypes.c_char), \
                                           k4a_device_t, \
                                           k4a_device_configuration_t, \
                                           k4a_record_t, \
                                           )

  File "../pyKinectAzure\pyKinectAzure.py", line 14, in __init__
    self.k4a = _k4a.k4a(modulePath)
  File "../pyKinectAzure\_k4a.py", line 560, in __init__
    self.k4a_record_create = dll.k4a_record_create
  File "D:\soft\Anaconda3\lib\ctypes\__init__.py", line 369, in __getattr__
    func = self.__getitem__(name)
  File "D:\soft\Anaconda3\lib\ctypes\__init__.py", line 374, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'k4a_record_create' not found

Body Tracker Initialization Failed due to "onnxruntime.dll"

I am using your library with with Azure Kinect DK for Pose Estimation. On of the my laptop - it works perfectly.

When I replicated the development environment on another laptop (with the same hardware specification as the first laptop), maintaining the exact same version of NVidia Drivers, CUDA, cuDNN, Azure Kinect SDK and Azure Kinect Body Tracking SDK; I get exception in Body Tracker Initialization due to "onnxruntime.dll"

Below is the log -

2021-04-16 13:11:46.1972431 [E:onnxruntime:, inference_session.cc:1268 onnxruntime::InferenceSession::Initialize::<lambda_d0b43ece64526f87430c2f161129335c>::operator ()] Exception during initialization: C:\source\ort-restore\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(1487)\onnxruntime.dll!00007FF8634BF2AC: (caller: 00007FF86350EE66) Exception(3) tid(4030) 80070057 The parameter is incorrect.

[2021-04-16 13:11:46.222] [error] [t=16432] [K4ABT] ..\src\TrackerHost\TrackerHost.cpp (185): Create(). Exception during initialization: C:\source\ort-restore\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(1487)\onnxruntime.dll!00007FF8634BF2AC: (caller: 00007FF86350EE66) Exception(3) tid(4030) 80070057 The parameter is incorrect.

[2021-04-16 13:11:46.222] [error] [t=16432] [K4ABT] ..\src\sdk\k4abt.cpp (38): tracker->Create(sensor_calibration, config) returned failure in k4abt_tracker_create()
Body tracker initialization failed!
Exception during initialization: C:\source\ort-restore\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(1487)\onnxruntime.dll!00007FF8634BF2AC: (caller: 00007FF86350EE66) Exception(3) tid(4030) 80070057 The parameter is incorrect.

File "D:\MY_PROJECT_PATH\MY_PYTHON_SCRIPT.py", line 1383, in
pyK4A.bodyTracker_start(bodyTrackingModulePath)
File "../pyKinectAzure\pyKinectAzure.py", line 34, in bodyTracker_start
self.body_tracker = kinectBodyTracker(bodyTrackerModulePath, depthSensorCalibration)
File "../pyKinectAzure\kinectBodyTracker.py", line 26, in init
self.initializeTracker()
File "../pyKinectAzure\kinectBodyTracker.py", line 73, in initializeTracker
_k4abt.VERIFY(self.k4abt.k4abt_tracker_create(self.sensor_calibration, self.tracker_config, self.tracker_handle), "Body tracker initialization failed!")
File "../pyKinectAzure_k4abt.py", line 107, in VERIFY
traceback.print_stack()

Process finished with exit code 1

Can you please suggests a solution that I can try to fix this.

I have double checked all the version of the dependencies and everything is matching to the dependencies on the 1st laptop on which this Library is working flawlessly.

Thanks for the help.

Body tracker initialization failed! - due to CUDA Out of Memory Error

Hello:

I am using the Body Tracking SDK in a Flask API to perform Pose Estimation on selected body parts.

For Example, with FLASK API_1, I am performing tracking of the Hands, with FLASK API_2, I am performing tracking of the head etc. So, I have created multiple APIs and have return call back functions which do body tracking for that specific API call.

Now, I have used code from the example > exampleBodySegmentation.py.

I am doing the initialization of the pyK4A at each API call function using

pyK4A = pyKinectAzure(modulePath))

and then I am releasing/disposing the instance before exiting the API callback function using

pyK4A.image_release(depth_image_handle) pyK4A.image_release(pyK4A.body_tracker.segmented_body_img) pyK4A.device_stop_cameras() pyK4A.device_close()

I am using NVidia RTX 2060 GPU and I have observed that with each API Call approx 1 GB VRAM of the GPU is used. So, if I call the API 3 times, on each API Call approx 1 GB VRAM is consumed and it adds up to around 3.5 GB. Due to this - if I make 6 API calls, all the GPU Memory gets consumed (approx 5.2 GB) and then we get the below mentioned error -

`

    Device configuration:
    color_format: 0
    (0:JPG, 1:NV12, 2:YUY2, 3:BGRA32)

    color_resolution: 1
    (0:OFF, 1:720p, 2:1080p, 3:1440p, 4:1536p, 5:2160p, 6:3072p)

    depth_mode: 2
    (0:OFF, 1:NFOV_2X2BINNED, 2:NFOV_UNBINNED,3:WFOV_2X2BINNED, 4:WFOV_UNBINNED, 5:Passive IR)

    camera_fps: 2
    (0:5 FPS, 1:15 FPS, 2:30 FPS)

    synchronized_images_only: True
    (True of False). Drop images if the color and depth are not synchronized

    depth_delay_off_color_usec: 0 ms.
    Delay between the color image and the depth image

    wired_sync_mode: 0
    (0:Standalone mode, 1:Master mode, 2:Subordinate mode)

    subordinate_delay_off_master_usec: 0 ms.
    The external synchronization timing.

    disable_streaming_indicator: False
    (True or False). Streaming indicator automatically turns on when the color or depth camera's are in use.


    2021-05-17 11:53:17.9269568 [E:onnxruntime:, inference_session.cc:626 onnxruntime::InferenceSession::Initialize] Exception during initialization: C:\agent\_work\11\s\onnxruntime\core\providers\cuda\cuda_call.cc:97 onnxruntime::CudaCall C:\agent\_work\11\s\onnxruntime\core\providers\cuda\cuda_call.cc:91 onnxruntime::CudaCall CUDA failure 2: out of memory ; GPU=0 ; hostname=XYZ ; expr=cudaMalloc((void**)&p, size);


    Exception during initialization: C:\agent\_work\11\s\onnxruntime\core\providers\cuda\cuda_call.cc:97 onnxruntime::CudaCall C:\agent\_work\11\s\onnxruntime\core\providers\cuda\cuda_call.cc:91 onnxruntime::CudaCall CUDA failure 2: out of memory ; GPU=0 ; hostname=NIBLP746 ; expr=cudaMalloc((void**)&p, size);


    [2021-05-17 11:53:17.953] [error] [t=12268] [K4ABT] D:\a\1\s\src\TrackerHost\TrackerHost.cpp (157): Create(). Exception during initialization: C:\agent\_work\11\s\onnxruntime\core\providers\cuda\cuda_call.cc:97 onnxruntime::CudaCall C:\agent\_work\11\s\onnxruntime\core\providers\cuda\cuda_call.cc:91 onnxruntime::CudaCall CUDA failure 2: out of memory ; GPU=0 ; hostname=XYZ ; expr=cudaMalloc((void**)&p, size);
    
    [2021-05-17 11:53:17.953] [error] [t=12268] [K4ABT] D:\a\1\s\src\sdk\k4abt.cpp (38): tracker->Create(sensor_calibration, config) returned failure in k4abt_tracker_create()

     Body tracker initialization failed!

`

After performing some diagnostics by using breakpoints and debugger, as well as monitoring the GPU usage in the Task Manager and with the command 'nvidia-smi'; I was able to conclude that - may be multiple body tracking trackers/instances are being created at each API calls and they are not getting released/disposed. Hence the GPU memory is exhausting after 5 to 6 API calls. Additionally, if we terminate the Flask Server, the GPU Memory gets Flushed out. So, my hypothesis seems reasonable.

I did some more research and I found a link where in people faced similar issue. Please check the last comment by YZR821 on this link -
microsoft/Azure-Kinect-Sensor-SDK#576

I have tried to delete pyK4A instance and explicitly call the Garbage Collector before exiting the function; but that does not help.

Additionally, I experiment with the sample example "exampleBodySegmentation.py" by running it 5 times and observed the GPU Memory usage in the Task Manager; but since the Script Terminate when we press ESC Key, I was not able to confirm if some tracker is still utilizing the GPU Memory.

So, can you please check and let me know - how can we properly dispose/release the GPU memory once we are done with the Body Tracking SDK.

Thanks .

Error while using exampleBodySegmentation.py

Hello!
It may be difficult to understand in English from the translation, but I'm sorry.
I'm a college student who is now starting to touch Azure Kinect. I was looking for Azure Kinect to work with python and found your great library.

So, I tried using "exampleBodySegmentation.py", but the following error occurred.

In order to solve this problem, I saw a case of a similar error like URL in github memory before and decided to see it and put it into practice.

https://githubmemory.com/repo/ibaiGorordo/pyKinectAzure/issues/15
Here, it was said that it was solved by updating Windows and lowering the AzureKinectSDK to v10.0, so I thought I would do it, but I am in trouble because the AzureKinectSDK v10.0 can not be installed.
I would be grateful if you could tell me how to solve it as soon as possible. Thank you.

Feature request: setup.py

Hi, thank you for writing this comprehensive python wrapper over the Kinect Azure SDK. I found it extremely useful. Is it possible to add setup.py functionality so it can be used in other projects without sys path inserts?