stc/semantic-vision

Semantic computer vision examples with p5js, ml5js & A-Frame

Semantic Vision

Overview

Introduction to creative coding and computer vision using modern web based technologies. Computer vision is with us since many years. Reading and tracking light, pixels, blobs, edges, colors are common techniques in interactive environments. However, in recent years, the field is just shifting to next level states with successful learning systems, like machine learning and deep neural networks. Semantic content became available to deal with, thanks for the extended databases and training algorithms out there. Our machines now understand drawings, objects, human poses, gestures, and many other semantically meaningful components, that are related to vision. This workshop covers some basic methods on how to use these advances, also, it is intended to find novel ways of cognitive interaction modalities based on these tools.

Tools

We are using the free and open source p5js library to create shapes and forms on the canvas, in Javascript language. The simplest way to get started is to use the excellent online p5 editor, however if you plan to bring your pieces offline and continue working on them in a more elaborative way, it is useful to clone this repository and use the open source, cross platform editor VSCodium with your projects.

Folder Structure

001 - uses simple webcam input + p5js

002 - 006 using ml5js and neural networks to learn to find custom features in the image

007 - using mindAR (on top of A-Frame) as the computer vision engine and displays content in mixed reality

Examples

001 - color tracking

002 - object detection

003 - posenet

004 - hand pose

005 - feature extraction

006 - pixel regression

007 - augmented reality + image tracking - open this on your phone, then show this image to your device. Multiple videos example

Class Works

Tarot Card Divination (Boszorkány Project). Works with this this marker image shown to the device

Skill Cards Works with this this marker image shown to the device

Turing Book.

Garbage Identifier Works with this this marker image shown to the device

Zodiac Jewelery.

Augmented Posters. Works with this this marker image shown to the device

Community Garden. Works with this this marker image shown to the device

Campus Monsters.

Classroom Schedule. Works with this this marker image shown to the device

Piggy Card Game

Movie Story

Inspiration

Human Pose and motion: an extensive overview of the topic in media history, by Daito Manabe

Open Source Machine learning Projects: what people are doing with ML in 2018

History of Machine Learning: general overview starting from 1642

Artists and Machine Intelligence: medium publications from different artists working in the field