xenon-19/gesture-controlled-virtual-mouse

Virtually controlling computer using hand-gestures and voice commands. Using MediaPipe, OpenCV Python.

Gesture Controlled Virtual Mouse  

Gesture Controlled Virtual Mouse makes human computer interaction simple by making use of Hand Gestures and Voice Commands. The computer requires almost no direct contact. All i/o operations can be virtually controlled by using static and dynamic hand gestures along with a voice assistant. This project makes use of the state-of-art Machine Learning and Computer Vision algorithms to recognize hand gestures and voice commands, which works smoothly without any additional hardware requirements. It leverages models such as CNN implemented by MediaPipe running on top of pybind11. It consists of two modules: One which works direct on hands by making use of MediaPipe Hand detection, and other which makes use of Gloves of any uniform color. Currently it works on Windows platform.

Video Demonstration: link
Note: Use Python version: 3.8.5

Features

click on dropdown to know more

Gesture Recognition:

Neutral Gesture

Neutral Gesture. Used to halt/stop execution of current gesture.

Move Cursor

Cursor is assigned to the midpoint of index and middle fingertips. This gesture moves the cursor to the desired location. Speed of the cursor movement is proportional to the speed of hand.

Left Click

Gesture for single left click

Right Click

Gesture for single right click

Double Click

Gesture for double click

Scrolling

Dynamic Gestures for horizontal and vertical scroll. The speed of scroll is proportional to the distance moved by pinch gesture from start point. Vertical and Horizontal scrolls are controlled by vertical and horizontal pinch movements respectively.

Drag and Drop

Gesture for drag and drop functionality. Can be used to move/tranfer files from one directory to other.

Multiple Item Selection

Gesture to select multiple items

Volume Control

Dynamic Gestures for Volume control. The rate of increase/decrease of volume is proportional to the distance moved by pinch gesture from start point.

Brightness Control

Dynamic Gestures for Brightness control. The rate of increase/decrease of brightness is proportional to the distance moved by pinch gesture from start point.

Voice Assistant ( Proton ):

Launch / Stop Gesture Recognition

• Proton Launch Gesture Recognition
  Turns on webcam for hand gesture recognition.
• Proton Stop Gesture Recognition
  Turns off webcam and stops gesture recognition. (Termination of Gesture controller can also be done via pressing Enter key in webcam window)

Google Search

• Proton search {text_you_wish_to_search}
  Opens a new tab on Chrome Browser if it is running, else opens a new window. Searches the given text on Google.

Find a Location on Google Maps

1. Proton Find a Location
   Will ask the user for the location to be searched.
2. {Location_you_wish_to_find}
   Will find the required location on Google Maps in a new Chrome tab.

File Navigation

   

• Proton list files / Proton list
  Will list the files and respective file_numbers in your Current Directory (by default C:)
• Proton open {file_number}
  Opens the file / directory corresponding to specified file_number.
• Proton go back / Proton back
  Changes the Current Directory to Parent Directory and lists the files.

Current Date and Time

• Proton what is today's date / Proton date
Proton what is the time / Proton time
  Returns the current date and time.

Copy and Paste

• Proton Copy
  Copies the selected text to clipboard.
  
• Proton Paste
  Pastes the copied text.

Sleep / Wake up Proton

• Sleep
  Proton bye
  Pauses voice command execution till the assistant is woken up.
• Wake up
  Proton wake up
  Resumes voice command execution.

Exit

• Proton Exit
  Terminates the voice assisstant thread. GUI window needs to be closed manually.

Getting Started

Pre-requisites

Python: (3.6 - 3.8.5)
Anaconda Distribution: To download click here.

Procedure

git clone https://github.com/xenon-19/Gesture-Controlled-Virtual-Mouse.git

For detailed information about cloning visit here.

Step 1:

conda create --name gest python=3.8.5

Step 2:

conda activate gest

Step 3:

pip install -r requirements.txt

Step 4:

conda install PyAudio

conda install pywin32

Step 5:

cd to the GitHub Repo till src folder

Command may look like: cd C:\Users\.....\Gesture-Controlled-Virtual-Mouse\src

Step 6:

For running Voice Assistant:

python Proton.py

( You can enable Gesture Recognition by using the command "Proton Launch Gesture Recognition" )

Or to run only Gesture Recognition without the voice assisstant:

Uncomment last 2 lines of Code in the file Gesture_Controller.py

python Gesture_Controller.py

Collaborators

Viral Doshi	GitHub	Email	LinkedIn	Instagram
Nishiket Bidawat	Github	Email	LinkedIn	Instagram
Ankit Sharma	GitHub	Email	LinkedIn	Instagram
Parth Sakariya	Github	Email	LinkedIn	Instagram

gesture-controlled-virtual-mouse's Issues

Error while running proton.py

After configuring all the files and requirements.txt when i go to run the file named proton.py it is showing error that the libraries which are written for import is saying that it is not found.
I have attached the screenshot for the reference.

Regarding Library

How to install screen brightness control library

Cursor is invisible

Describe the bug
When I start the Project, it works perfectly, but the mouse cursor is invisible. So I can't see where I'm clicking.

Desktop :

• OS: Windows 11 Pro
• Browser [Opera GX]
• GPU: RTX 2070
• CPU: Intel I9-9900k
• RAM: DDR4 32GB 3000Mhz

Ubuntu adaptation/version.

Hello!) I want to ask - will there be any attempts to add the ubuntu version?) Looking forwards for your reply))) Thanks!

This is the error i got

make it runable on non window system like linux parrot or debian 11

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
i dont have comport on my system as i use linux
Describe the solution you'd like
A clear and concise description of what you want to happen.
use module that are compatible with linux and mac
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Proton isn't working

Hi,
I tried executing https://github.com/xenon-19/Gesture-Controlled-Virtual-Mouse but proton isn’t working. It is not recognising my commands. Please help as soon as possible. It is also not launching the Gesture Controller.

Chirag Nahata
[phone]

Error

Can you help me to solve this error?

Able to run proton but not taking voice commands

I am able to run the code of proton but it is not taking the command given by the user and it is showing the output as cant recognize

IMG-20220627-WA0012

i ran the proton.py code but the application never opened. Any solutions?

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

process finished with exit code 0 but I haven't seen any interface

I have installed all the modules. But getting 5 errors
I have run the code successfully
I got 'Process finished with exit code 0'
But cant see the result

(most likely due to a circular import) Process finished with exit code 1

File "C:\Users\ADMIN\Desktop\Gesture-Controlled-Virtual-Mouse-main\src\Gesture_Controller.py", line 3, in
import cv2
File "C:\Users\ADMIN\Desktop\Gesture-Controlled-Virtual-Mouse-main\venv\lib\site-packages\cv2_init_.py", line 181, in
bootstrap()
File "C:\Users\ADMIN\Desktop\Gesture-Controlled-Virtual-Mouse-main\venv\lib\site-packages\cv2_init_.py", line 175, in bootstrap
if _load_extra_py_code_for_module("cv2", submodule, DEBUG):
File "C:\Users\ADMIN\Desktop\Gesture-Controlled-Virtual-Mouse-main\venv\lib\site-packages\cv2_init.py", line 28, in _load_extra_py_code_for_module
py_module = importlib.import_module(module_name)
File "C:\Program Files\Python38\lib\importlib_init.py", line 127, in import_module
return _bootstrap.gcd_import(name[level:], package, level)
File "C:\Users\ADMIN\Desktop\Gesture-Controlled-Virtual-Mouse-main\venv\lib\site-packages\cv2\gapi_init.py", line 301, in
cv.gapi.wip.GStreamerPipeline = cv.gapi_wip_gst_GStreamerPipeline
AttributeError: partially initialized module 'cv2' has no attribute 'gapi_wip_gst_GStreamerPipeline' (most likely due to a circular import)

Process finished with exit code 1

OPENCV error, what is correct version of opencv we need to use?

python Proton.py
Traceback (most recent call last):
File "Proton.py", line 8, in
import pyautogui
File "C:\Users\Manoj.conda\envs\gest\lib\site-packages\pyautogui_init_.py", line 183, in
import pyscreeze
File "C:\Users\Manoj.conda\envs\gest\lib\site-packages\pyscreeze_init_.py", line 27, in
import cv2
File "C:\Users\Manoj.conda\envs\gest\lib\site-packages\cv2_init_.py", line 181, in
bootstrap()
File "C:\Users\Manoj.conda\envs\gest\lib\site-packages\cv2_init_.py", line 175, in bootstrap
if _load_extra_py_code_for_module("cv2", submodule, DEBUG):
File "C:\Users\Manoj.conda\envs\gest\lib\site-packages\cv2_init.py", line 28, in _load_extra_py_code_for_module
py_module = importlib.import_module(module_name)
File "C:\Users\Manoj.conda\envs\gest\lib\importlib_init.py", line 127, in import_module
return _bootstrap.gcd_import(name[level:], package, level)
File "C:\Users\Manoj.conda\envs\gest\lib\site-packages\cv2\gapi_init.py", line 301, in
cv.gapi.wip.GStreamerPipeline = cv.gapi_wip_gst_GStreamerPipeline
AttributeError: partially initialized module 'cv2' has no attribute 'gapi_wip_gst_GStreamerPipeline' (most likely due to a circular import)

speech_recognition module is not installing

No output/error while running proton.py [HELP]

Hello there,
I'm trying to run proton.py for voice control asssitant, however when i try running it, it ends up opening gesture control instead.

I took the advice from previous ticket with similiar issue and I'm left with no output when i run proton.py now.

It's just stuck. No error, no output.

Any kind of help is mcuh appreciated

bugs in the code

getting errors in the code

error installing mediapipe and "dist1" is not defined please help me ASAP

not showing output

mot showing expected output just the executed without error screen , even after removing the last two
comment lines in Gesture-controller.py
can anyone help me out
@xenon-19

Scrolling control not working

Hi . I want to ask a question about your project on github. I am getting an error as seen in the file, what is the reason for this?

I will be glad if you answer.

I was going through your project Gesture-Controlled-Virtual-Mouse on github and tried to run it on my local machine but it's not working......I followed every step from github to run it, but still....I'm getting the following error when I'm trying to run this code. I tried everything I could do. Help me with that issue and find attached ss of errors. reply ASAP

I cant run the codes it shows

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Now output showm

Out put not shown

Proton not responding

Proton doesn't respond. The input taken from the user is shown in the command prompt but is not shown in the Proton interface.

error in pip install -r requirements.txt

couldn't run proton.py

I'm getting the following error when I'm trying to run this code
Could you plz help me

change in brightness

can you check and rectify, there is an issue with the brightness control, pinch gesture is not working when changing the brightness. sometimes the camera windows automatically closes while using the pinch gesture for changing the brightness by getting the following statement in the terminal "INFO: Created TensorFlow Lite XNNPACK delegate for CPU."

error while running proyton.py

ERROR: Could not find a version that satisfies the requirement mediapipe (from versions: none)
ERROR: No matching distribution found for mediapipe
WARNING: You are using pip version 20.1.1; however, version 23.3.2 is available.
You should consider upgrading via the 'c:\users\ubaid\appdata\local\programs\python\python38-32\python.exe -m pip install --upgrade pip' command.

i try to download any mediapipe version but the error is same

No output

I am not getting any errors.
But still, python Gesture_Controller.py is not showing the expected output.

Pre-install Module

Hi sir,
I'm a new coder but I'm facing issues regarding launching this file
how may i fix the errors regarding module?

Not able to run Gesture_Controller.py

After uncommenting the last two lines of the code in Gesture_Controller.py, when I tried to run the code, I was facing the following error:

Can someone help me in resolving this issue?

Windows Interface "Gui"

Hi, I don't understand programming could you make a basic executable Windows Interface "Gui" version of this for me?

I got the same error athe s you, I followed these steps and now I can run the project.

          I got the same error as you, I followed these steps and now I can run the project.

1. For this error
   AttributeError: partially initialized module 'cv2' has no attribute 'gapi_wip_gst_GStreamerPipeline' (most likely due to a circular import)
   I did this

pip uninstall opencv-python
pip uninstall opencv-contrib-python
pip uninstall opencv-contrib-python-headless
and then
pip3 install opencv-contrib-python==4.5.5.62

reference: https://stackoverflow.com/questions/72706073/attributeerror-partially-initialized-module-cv2-has-no-attribute-gapi-wip-gs

2. When I executed after making the above changes, I got the following error.

that is solved by
pip install protobuf==3.20.*
reference: https://stackoverflow.com/questions/72441758/typeerror-descriptors-cannot-not-be-created-directly

Finally when I run this, I got the correct output
python Gesture_Controller.py

I hope it Helps.

Originally posted by @padminipriya69 in #17 (comment)

error when running Gesture_Controller.py

I encounter the following error when running Gesture_Controller.py:

File "C:\Users\yiyun.zy\anaconda3\envs\py385\lib\importlib_init_.py", line 127, in import_module
return _bootstrap.gcd_import(name[level:], package, level)
File "C:\Users\yiyun.zy\Desktop\xuelangyun\gesture_control\venv\lib\site-packages\cv2\gapi_init.py", line 290, in
cv.gapi.wip.GStreamerPipeline = cv.gapi_wip_gst_GStreamerPipeline
AttributeError: partially initialized module 'cv2' has no attribute 'gapi_wip_gst_GStreamerPipeline' (most likely due to a circular import)

How can I solve it? Thank you so much

Bro after proton getting open it is not recognising command

Proton is showing can't recognise command again and again

Volume and brightness control not working

Describe the bug

After getting thumb and index close to each other, code stops running, could not perform brightness and volume control. Im running this in Python V-3.8.5. I made no changes to the code
To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.
Every other feature works really good, but controlling fails for me.
Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS] Windows 10
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Traceback error

(gest) C:\Users\vigne\Gesture-Controlled-Virtual-Mouse\src>python Gesture_Controller.py
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Traceback (most recent call last):
File "Gesture_Controller.py", line 600, in
gc1.start()
File "Gesture_Controller.py", line 596, in start
cv2.destroyAllWindows()
cv2.error: OpenCV(4.2.0) C:\projects\opencv-python\opencv\modules\highgui\src\window.cpp:645: error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Cocoa support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function 'cvDestroyAllWindows'

HELP WITH THE GLOVE GESTURE CONTROL

Is your feature request related to a problem? Please describe.
i have been able to make the code work for the hand gesture but i dont understand how the glove version works

Describe the solution you'd like
can you pls expalin what i need to do make it work

Describe alternatives you've considered
i try a diffrent color of some object but it doesnt works , Do i need to wear glove
Additional context
pls i will be greateful if you reply

AttributeError: partially initialized module 'cv2' has no attribute 'gapi_wip_gst_GStreamerPipeline' (most likely due to a circular import)

File "C:\Users\ADMIN\Desktop\Gesture-Controlled-Virtual-Mouse-main\src\Gesture_Controller.py", line 3, in
import cv2
File "C:\Users\ADMIN\Desktop\Gesture-Controlled-Virtual-Mouse-main\venv\lib\site-packages\cv2_init_.py", line 181, in
bootstrap()
File "C:\Users\ADMIN\Desktop\Gesture-Controlled-Virtual-Mouse-main\venv\lib\site-packages\cv2_init_.py", line 175, in bootstrap
if _load_extra_py_code_for_module("cv2", submodule, DEBUG):
File "C:\Users\ADMIN\Desktop\Gesture-Controlled-Virtual-Mouse-main\venv\lib\site-packages\cv2_init.py", line 28, in _load_extra_py_code_for_module
py_module = importlib.import_module(module_name)
File "C:\Program Files\Python38\lib\importlib_init.py", line 127, in import_module
return _bootstrap.gcd_import(name[level:], package, level)
File "C:\Users\ADMIN\Desktop\Gesture-Controlled-Virtual-Mouse-main\venv\lib\site-packages\cv2\gapi_init.py", line 301, in
cv.gapi.wip.GStreamerPipeline = cv.gapi_wip_gst_GStreamerPipeline
AttributeError: partially initialized module 'cv2' has no attribute 'gapi_wip_gst_GStreamerPipeline' (most likely due to a circular import)

Process finished with exit code 1