everythingishacked/semaphore

A full-body keyboard using gestures to type through computer vision

Semaphore

A full-body keyboard

View a fuller demo and more background on the project at https://youtu.be/h376W93gQq4

The next iteration of this project, designed as a full-body game controller, is also available at https://github.com/everythingishacked/Gamebody

Semaphore uses OpenCV and MediaPipe's Pose detection to perform real-time detection of body landmarks from video input. From there, relative differences are calculated to determine specific positions and translate those into keys and commands sent via keyboard.

The primary input is to "type" letters, digits, and symbols via flag semaphore by extending both arms at various angles. Rather than waiting a set time after every signal, you can jump to repeat the last sent symbol.

See the SEMAPHORES dictionary in the code for a full set of angles, which mostly conform to standard US semaphore with some custom additions. Most of the rest of the keyboard is included as other modifier gestures, such as:

• shift: open both hands, instead of fists
• backspace: both hands over mouth
• digits and other extra symbols: squat while signaling
• command: lift left leg to ~horizontal thigh
• control: lift right leg to ~horizontal thigh
• arrow left/right/up/down: cross arms and raise each straight leg LEG_ARROW_ANGLE degrees
• repeat previous letter/command: jump

Running on latest MacOS from Terminal, toggle the following for keyboard access: System Settings -> Privacy & Security -> Accessibility -> Terminal -> slide to allow

For Mac, this uses a custom keyboard library. This is built for a Mac keyboard, but you can also swap e.g. Windows key for Command simply enough.

semaphore's Issues

Add functionality to insert newline upon a backflip

We for sure a newline on backflip of the user.

Improve FPS for faster input

Semaphore starts to skip letters at ~100 WPM.

Touching base for collaboration

Hi @everythingishacked ,
We had a brief conversation about your tool on Hackernews; I couldn't find your contact, so I'm writing you here.

I've just released a PoC for a body gamepad: https://github.com/mristin/body-gamepad. Please contact me if you are still interested in collaboration -- I believe there is a lot of potential for such a tool (or rather a set of tools) in sports & education.

You can find my email on my GitHub profile.

Cheers,
Marko

Support emoji

Use facial expression + hand gesture to type emoji 🤔

Missing LICENSE

I see you have no LICENSE file for this project. The default is copyright.

I would suggest releasing the code under the GPL-3.0-or-later or AGPL-3.0-or-later license so that others are encouraged to contribute changes back to your project.

Edit: Thank you! Cool project!

error

3871 Segmentation fault