wendychen0521/creativeml2020

Creative Machine Learning Examples

CreativeML2020

Creative Machine Learning Examples for 2020

Teaching resources for projects and workshops exploring machine learning (ML)

Main teaching webpage

Detailed explanations, walk-throughs and direct links to each teaching exercise sketch live on the teaching website but collections and corresponding Arduino code can be downloaded from this repo.

Teachable Machine

These examples all run in the P5JS editor and import simple image classification models trained using the teachable Machine

To view the full list of examples see the P5JS Teachable Machine Collection

Key examples include:

• Image Classification Template
• Sound Classification template
• Image Classification + Emoji Output
• Image Classification + Image Output
• Image Classification + Sound Output

ML5

These examples run in the P5JS editor and explore varying ML5 functions such as classification, regression and pre-trained models.

To view the full list of examples see the ML5 Collection

Key examples include:

• Webcam Image Classifier
• Webcam Classifier sound trigger
• MobileNet classification
• WebCam Regression
• PoseNet Classifier

DIY Neural Nets in ML5

These examples live in the P5JS editor and explore how to use the ML5 Neural Net function to build templates for training any input, in particular, Physical Computing inputs.

To view the full list of examples see the DIY Neural Net Collection

Key examples include:

• Basic template
• Mouse XY Classifier

Physical Teachable Machines

These examples work with a range of methods:

• Training directly in ML5 with the neural net function
• Training in the Arduino IDE with the Simple_KNN library.
• Training with Tensorflow Lite using the Arduino_TensorflowLite library and Google Colabs

Each method will be outlined below.

Training directly in ML5 with the neural net function

You can send serial messages from any Arduino input directly into P5JS and use the ML5 neural net function to train these. You will need the corresponding Arduino code uplaoded on to your board for each example to work. The corresponding Arduino code can be found in the Physical teachable Machines folder on this repo or each .ino file is directly referenced at the top of P5JS code.

To view the full list of examples see the Physical Teachable Machine collection

Key examples include:

• Classify 3 serial inputs
• Classify 3 web USB inputs
• Classify an array of LDR inputs
• Classify Capacitive Touch
• Classify Gestures with IMU sensors and visula confidence output

If you are using the serial examples you will also need to run the P5 Serial App alongside them.

If you are using the webUSB examples you do not require any 3rd party apps. However, you will need to make sure your Arduino board is compatible. A list of compatible boards can be found here

Arduino IDE and the Simple_KNN library.

In the past year is has become more accessible to run Machine Learning algrithms directly on the micro-controller (Arduino) using TensorflowLite and KNN libraries.

This repo contains examples to get you started using KNN algorithms all self-contained in Arduino. You can also find examples taking the basic structure one or two steps further to trigger different physical outputs.

Key examples include:

• Simple KNN template
• Classify Colour
• Classify Orientation
• Classify Color to physical output
• Classify Orientation to LED output
• Classify Color to neopixel output

These examples require the Arduino Nano 33 BLE Sense board. A guide to starting with Arduino and libraries can be found here.

Training with Tensorflow Lite using the Arduino_TensorflowLite library and Google Colabs.

You can train more detailed inputs using TensorFlow Lite and Google CoLabs. They provide detailed tutorials and pre-trained models to train on top of. These examples are a little more advanced and require some familiarity with Google Colab, file structures and working with multiple .ino files.

The example listed are referenced from Dominic Pajak and Sandeep Mistry

Key examples include:

• Gesture to Emojis
• Fruit to Emoji
• Magic Wand
• Full list

RunwayML

These examples build upon the existing RunwayML Processing library and P5JS examples taking these basic network structures further to explore experimental interactions with chat bot generating imagery, capacitive touch latent walks, speech feedback loops and motorised outputs to name a few. Have an explore!

Key Processing examples include:

• Trill Capacitive Touch to StyleGAN latent space
• ElizaChatBot to AttGAN
• GPT2- SpokenOutput

Key P5JS examples include:

• StyleGAN Latent Walk
• Speech Recognition to GPT2
• Hosted Model Template

The P5JS Runway Collection houses all the examples including new ways of connecting via hosted models.

Acknowledgements and references:

The code used in the examples above reference, adapt and expanded upon functionality from original sources by...

• Rebecca Fiebrink and Wekinator
• Andreas Refsgaard
• Sandeep Mistry
• Dominic Pajak
• Dan Shiffman and The Coding Train
• Bérenger Recoules
• Gene Kogan and ML4A
• Runway + Cristóbal Valenzuela
• George Profenza
• Yining Shi