konstantinosang/pykinect2-mapper-functions

ICoordinateMapper functions using PyKinectV2 wrapper for python

Hi, I am Konstantinos! 👋.

I tell computers to do things. Sometimes they listen. 🚀

~~~~> CHECK OUT MY BIO <~~~~

~~~> AUTO GITHUB RESUME BIO <~~~

~~~> PORTFOLIO <~~~

Tools & Languages

Links

pykinect2-mapper-functions's Issues

About Aligned Image

Hello, I can't display the picture when I use uint16 in the depth of About Aligned Image.
It can only be displayed if it is changed to uint8, but it does not seem to be very complete. Do you have any ideas?
Thank you

Convertted depth image should be an uint16 array

Thanks for these functions! They are really useful.

However, in the depth_2_color_space function, the returned depth gets converted to uint8 which I think is not optimal. The original data is in uint16 range and numpy astype may cause loss in precision, even plain erroneous depth value.

Best regards!

What is the requirement of the Python Version?

Typo in depth_img = depth_frame.reshape((kinect.depth_frame_desc.Height, kinect.depth_frame_desc.Width, 4)).astype(np.uint8)

Thanks for compiling this set of useful mapper functions!

I think the value 4 in this line

PyKinect2-Mapper-Functions/mapper.py

Line 58 in bc63d39

depth_img = depth_frame.reshape((kinect.depth_frame_desc.Height, kinect.depth_frame_desc.Width, 4)).astype(np.uint8)

should be changed to 1

mapper.py is a slow in converting between color space to depth space

I am using python 3.7 64bit.
i did not do pip install since it seems to not work in that way. rather using the given files directly via import.

However the issue is that running mapper.py is very slow. 0.5 fps is the frame rate. I think waitKey(3000) is the reason. But that still does not fix the issue:

When i set show = False that improves but still slow and not smooth 25/30fps. Rather somewhere around 8/10fps.

Any solution to this issue? When i do not call depth_2_color_space the speed is just back to 25/30fps i can say.

Canwe use mapper functions on already acquired images

I have already acquired rgb-d dataset from kinect2. Can this library be used for already acquired images?

Mapping in the Original PyKinect

First, I know that this is for PyKinect2 and it doesn't work in the original one. I wish to do something similar for the original PyKinect, but all information on how to do it has seemed to have vanished. I'm asking here as you seem to have figured out how to do it in this version and I'm wondering if you could point me in the right direction.

SDK for 1.8: https://learn.microsoft.com/en-us/previous-versions/windows/kinect-1.8/hh855347(v=ieb.10)
The pykinect docs seem to have been lost at some point.

when using function depth_point_2_world_point(), a problem will occur.

Hello. When I use the function depth_point_2_world_point(), I find there may be a bug.
As you can see in these picture, no matter what the input coordinates are, the output X and y change all the time, but the Z value remains unchanged at 20.48.

I really want to know why this problem occurs. Thanks very much.

About Mapper "color_2_depth_space"

Hello, when I use mapper, the color_2_depth_space containing imshow will not have a window display.
Can you help me? Thank you

something wrong in depth map to color, it seems not accuracy

Related to 1.getting same values of pixel in entire columna and 2. frame resolution

[hello] @KonstantinosAng

    Actually I am using kinect/pykinect2 lib to connect my kinect v2 with python. I am getting the depth image also and I  did some processing also in opencv but I am facing issue when I am going to print the captured frame matrix it shows me the same values in in entire column so I am not able to detect any object coming in front of my kinect sensor. also I am attaching the screenshot of matrix output.

And my second issue is about frame resolution , In kinect/pykinect2 i am not able to reduce the resolution properly. if i want 15x15 resolution then it converts height only 15 and width is 120. when i am going to increase the resolution above 120x120 then it work properly but below 120x120 it only decreases the height and the width is 120 as it is .

my code:

import cv2

from pykinect2 import PyKinectV2
from pykinect2 import PyKinectRuntime
import ctypes
import _ctypes
import pygame
import numpy as np
from PIL import Image
MARGIN =10
WIDTH = 10
HEIGHT = 10
mog = cv2.createBackgroundSubtractorMOG2(history=2000, varThreshold=16, detectShadows=True)
class DepthRuntime(object):
def init(self): # constructor initialize(assign values) to the data members of the class
x = pygame.init() # initialize all imported pygame modules

    # Used to manage how fast the screen updates
    self._clock = pygame.time.Clock()

    # Loop until the user clicks the close button.
    self._done = False

    # Used to manage how fast the screen updates
    self._clock = pygame.time.Clock()

    # Kinect runtime object, we want only color and body frames
    self._kinect = PyKinectRuntime.PyKinectRuntime(PyKinectV2.FrameSourceTypes_Depth)

    # back buffer surface for getting Kinect infrared frames, 8bit grey, width and height equal to the Kinect color frame size
    self._frame_surface = pygame.Surface((self._kinect.depth_frame_desc.Width, self._kinect.depth_frame_desc.Height), 0, 24)
    # here we will store skeleton data
    self._bodies = None

    # Set the width and height of the screen [width, height]

    self._screen = pygame.display.set_mode( (120,120),pygame.HWSURFACE | pygame.DOUBLEBUF | pygame.RESIZABLE, 32)
    self._infoObject = pygame.display.Info()
    print(self._infoObject)


    pygame.display.set_caption("Kinect for Windows v2 Depth")

def draw_Depth_newframe(self, frame, target_surface):
    if frame is None:  # some usb hub do not provide the infrared image. it works with Kinect studio though
        return
    target_surface.lock()
    f7= np.uint8(frame.clip(1, 4000)/ 16.)  #clip to change the minimum value to 0 and maximum value to 250
    frame8bit = np.dstack((f7,f7,f7)) #change all RGB values to the same value
    address = self._kinect.surface_as_array(target_surface.get_buffer())
    ctypes.memmove(address,frame8bit.ctypes.data, frame8bit.size) #Moving data (frame) to that particular address
    #frame = cv2.cvtColor(frame8bit, cv2.COLOR_GRAY2RGB)
    # cv2.imshow('KINECT Video Stream', frame)
    frame8bit = frame8bit.astype(np.uint8)
    frame8bit = cv2.resize(frame8bit, (20,20))
    new_Arr = cv2.cvtColor(frame8bit, cv2.COLOR_BGR2GRAY)
    print(new_Arr)
    fgmask = mog.apply(new_Arr)
    # threshold this and clean it up using dilation with a elliptical mask
    fgthres = cv2.threshold(fgmask.copy(), 128, 255, cv2.THRESH_BINARY)[1]
    fgdilated = cv2.dilate(fgthres, kernel=cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (3, 3)), iterations=3)
    bgmodel = mog.getBackgroundImage()
    cv2.imshow("bgmodel",bgmodel)
    cv2.imshow("fgdilated",fgdilated)
    cv2.imshow("fgthres",fgthres)
    cv2.imshow("new_Arr", new_Arr)

    # print(new_Arr.ndim)
    # print(new_Arr.size)
    # print(new_Arr.shape)

   #frame = frame.astype(np.uint8)
    #reshaped_arr = np.reshape(frame8bit,(1,651264))
    del address
    target_surface.unlock()

def run(self):
        # -------- Main Program Loop -----------
        while not self._done:
            # --- Main event loop
            for event in pygame.event.get():  # User did something
                if event.type == pygame.QUIT:  # If user clicked close
                    self._done = True  # Flag that we are done so we exit this loop

                elif event.type == pygame.VIDEORESIZE:  # window resized

                    self._screen = pygame.display.set_mode((0,0),pygame.HWSURFACE | pygame.DOUBLEBUF | pygame.RESIZABLE, 32)

            # --- Getting frames and drawing
            if self._kinect.has_new_depth_frame():
                frame = self._kinect.get_last_depth_frame()
                self.draw_Depth_newframe(frame, self._frame_surface)
                frame = None

                self._screen.blit(self._frame_surface, [0,0])
                pygame.display.update()


            # --- Go ahead and update the screen with what we've drawn.
            pygame.display.flip()

            # --- Limit to 60 frames per second
            self._clock.tick(1)


        # Close our Kinect sensor, close the window and quit.
        self._kinect.close()
        pygame.quit()

main = "Kinect v2 Depth"
game = DepthRuntime()
game.run()

please help me in this issues.
Thank you

when using color_point_2_depth_point, sometime an error will occur

as you can see in this picture, this is my code.

and there will always be an error:

Traceback (most recent call last):
  File "video_kinect.py", line 462, in <module>
    x,y = color_point_2_depth_point(a._kinect,_DepthSpacePoint,a._kinect._depth_frame_data,[int(center_x),int(center_y)])
  File "video_kinect.py", line 365, in color_point_2_depth_point
    return [int(depth_x), int(depth_y)]
OverflowError: cannot convert float infinity to integer

I really can not understand why this error occurs. Thank you!

Hello！Here is a question about 'color_2_depth_space'.

Thank you for providing this method !
I succeed in excuting mapper.py but I found that the edge of the aligned image is overlapped sometimes, how can I solve the problem to get a perfect aligned image?
Thank you again!