sarvagnya/finguitar

This is our repo for CodeOdyssey HackaThon

Finguitar 🎸

Finguitar - This is an online instrument which you can play using just your fingers.
CodeOdyssey Hack-a-thon Winners || Team NeoStar Aaghadi

---

This web-app is called Finguitar (Finger-Guitar). It uses Computer Vision to build an online instrument which you can play using just your fingers. You can hold up a gesture with your hand and the web-app will record it using your webcam and play a music note for every gesture! The idea is to then give the user the ability to combine these gestures to make a tune out of it. So, it's like playing a guitar... but using Computer Vision! Currently we recognize 10 gestures to play different sounds: Palm Fist Thumb Index Middle 2 fingers Pinky Call Rock LShape(finger+thumb) Ok

Resources

• Demo Video
• Devfolio Submission

🤖Tech-Stack

Front-end

• ReactJS

Back-end

• Flask

Computer vision

• OpenCV

📲Communication b/w the web-app and the model

The communication is carried out on a WebSocket endpoint run by the server. The web-app uses the browser's Stream API to emit the current frame encoded as a base64 string via the WebSocket. The server receives the string, decodes it and runs the OpenCV model on it. The gesture recognized is then relayed back on the same endpoint as a response.

🔮Further additions

• Host the web-app and the model
• Improve the animations on the web-app
• Add a gamified experience for the user, and provide him with a score as a feedback
• Add a tutorial run, where the user can first train on the set of gestures and get familiar with the sounds.

👨‍💻Team Members

• Sarvagnya Purohit
• Jash Shah
• Pratiksha Sankhe
• Alisha Kamat