bijection/sistine

Turn a MacBook into a Touchscreen with $1 of Hardware

Project Sistine

We turned a MacBook into a touchscreen using only $1 of hardware and a little bit of computer vision. The proof-of-concept, dubbed “Project Sistine” after our recreation of the famous painting in the Sistine Chapel, was prototyped by Anish Athalye, Kevin Kwok, Guillermo Webster, and Logan Engstrom in about 16 hours.

Basic Principle

The basic principle behind Sistine is simple. Surfaces viewed from an angle tend to look shiny, and you can tell if a finger is touching the surface by checking if it’s touching its own reflection.

Kevin, back in middle school, noticed this phenomenon and built ShinyTouch, utilizing an external webcam to build a touch input system requiring virtually no setup. We wanted to see if we could miniaturize the idea and make it work without an external webcam. Our idea was to retrofit a small mirror in front of a MacBook’s built-in webcam, so that the webcam would be looking down at the computer screen at a sharp angle. The camera would be able to see fingers hovering over or touching the screen, and we’d be able to translate the video feed into touch events using computer vision.

(Read the rest of our blog post, including a video demo and a high-level explanation of the algorithm, here)

Installation (with Homebrew Python)

• First, make sure you have Mac Homebrew installed on your computer. If not, you can install it by running /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

• Install Python 2 via Homebrew with brew install python2

• Install OpenCV 3 via Homebrew with brew install opencv3

• Install PyObjC via Pip with pip2 install pyobjc

Running

Run python2 sistine.py

License

Copyright (c) 2016-2018 Anish Athalye, Kevin Kwok, Guillermo Webster, and Logan Engstrom. Released under the MIT License. See LICENSE.md for details.

sistine's Issues

[No Bug] [ Appreciation] _Sistine_

There are some moments, where you are so high, you think anything is possible. I think this project and the completion time it took(16 hours) has literally blown me away. Does it work on windows too? I want to try it. Regards and a big note of thanks from India.

Is this project still active? Or is there anything else like it? Also, I saw some stuff about Windows?

Heya, this project looks super cool! but, there haven't been any updates since '18.

Does it still work?

Also, is it true that people used it on Windows? if so, how?

OR are there any alternative projects that make a windows Webcam into a touch screen? (I'm sure there are but, knowing if there are any good ones would be primo! :-)

Thanks

Error: no module named cv2

I tried installing opencv-python and opencv3 but still get error

[Praise]大巧不工

God have to use idioms to express my admiration for you only！
in chinese :
只有用这个成语才能表达对你们这个创意的赞叹！

Compatibility with other laptops

I wanted to fork and try it out on my old Acer V3 371 (i5, 32Gb Ram, 1TB ssd), but im not sure if it going to work. What if i took a pen and and draw apple, would your code consider it as a macbook? Is my four year specs enough to run this code?

About parameters

Hello, I want to know what the parameters that are fixed at the beginning here represent

Orientation error

It works great at the beginning where it has me touch and hold on 6 or so green dots. But then it puts 6 or seven random red lines on the screen and I can't progress. I've touched all over, ran my finger along the lines, but nothing happens. It seems to try and track my finger but I don't know what it's looking for.

[Not bug] - [Idea] - Touch bar

Hey!

I saw your touchscreen project this is a great idea made me think about simulating the Touch Bar. If you need only one specific icon set, you could print it and the camera (thats already on because of touch sceen) could grab wich one is hidden by your finger. If more than one is hidden it doesnt do anything so if you are using the touchscreen it doesnt get confused.

Or you could just print numbers or icons or QR code on it and you could change the fake touchbar layout in the software mapping.

Best regards,
Tom

License?

There's no information about the type of license for this project.

Sistine.py in crash when I run it

I run with the command python2 sistine.py but the window with the picture of webcam, remains open for only 2 seconds.
In the terminal i can see this message:
Traceback (most recent call last): File "sistine.py", line 397, in <module> main() File "sistine.py", line 383, in main if not currStage(segmented, debugframe, options, ticks, drawframe, calib, state): File "sistine.py", line 223, in _calibration x, y, touch = find(segmented, debugframe=drawframe, options=options) File "sistine.py", line 152, in find _, cnts, _ = cv2.findContours(segmented_image.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) ValueError: need more than 2 values to unpack

Can't I run this code on windows without Linux VM?

I am unable to install Quartz. My friend suggests I use a VM with a Linux OS to run this code.
Can this issue be resolved from your side? I mean, the idea and the current code is great. May be we can make it more accessible!

gif broken in readme eom

Explain parameters

It's not much of an issue
Just that, can you explain what comp dimensions xy and capture dimensions xy are? Please!

Added numpy to pip install

Hey, figured y'all should add pip2 install numpy to the instructions or just flat out create a requirements.txt file and do pip2 install -r requirements.txt

[No Bug] Windows enabled code

I have taken the pain to modify this to be used on windows as will.
https://github.com/ashishgupta1350/Windows-Touch
Please add stimulate.py from this to your main repo and pull changes.
Regards.
Ashish.

[Not bug] - [Idea] - Full touch screen

Hey there,
I was discussing this awesome project with one of my friends and he asked me why it was limited to a certain portion of the screen. I told him it's because of the field of view of the camera and mirror combined is limited.

So he gave me a piece of advice that you should look at this problem from the industrial design point of view.
Basically ask someone who is good in the industrial design field, whether is it possible to design a mirror which can cover the whole screen and reflect it in webcam!!

And I must tell you this, that this is one of the coolest outcomes I've seen of a hackathon. Good job dude.
Let me know if you have any question regarding this idea.

[function- segment] [Why RGB2LAB & !BGR2LAB]

Why did you segment image using RGB2LAB when you could have used BGR2LAB and set the threshold values on the third layer as 140-230. It gives the same result.