ryanknauer/cpsc448

CPSC448

Project Proposal

https://docs.google.com/document/d/1puxoi1hOgQyhx2ozLxz7JmSdaY13pf-dHCr5eZlqh5Q/edit?usp=sharing

Description

I am currently taking CPSC 491 working on a project with CDM to build an interactive toy robot to help children with Autism. For this directed studies course, I will be taking on an extension to this project, building on concepts covered in CPSC 340 and CPSC 425. This project extension would be to implement a Chrome Extension that processes face emotions in youtube videos being watched in real time. This feature will be integrated with the toy robot we are building in the project in order to highlight emotions in the video. The goal is for this to be used as an aid to help teach emotion interpretation and engagement at a young age, and in a very inexpensive and accessible way. This is partially inspired from the Autism Glass Project, hopefully providing another aid for emotion interpretation development in an environment that is both comfortable and engaging for the children. For this directed studies course, I will solely be focusing on the Facial and Emotion Recognition portion of this application as this portion is out of the scope of my teams 491 project.

Learning Goals

• Understand the high + mid level intuition behind the emotion recognition models that would be used in this application
• Gain hands on application experience using open source models for facial recognition and emotion recognition
• Learn to fine tune and improve available tools to solve a specific, real world problem
• Gain experience using tensorflow
• Understand workings behind more advanced models used in industry such as Long Short Term Memory
• Understand intuition behind implementation of models used for emotion and facial recognition
• Explore potential options for improvement for this application

Table of Contents/Breakdown

This course was broken down into two main parts in order to maintain a balance of both implementation and reading/learning. The first part stems from the 491 course, implementing and improving the chrome extension outlined above. Here I was able to apply and build upon skills covered in CPSC322 + CPSC 422 to integrate spatial locality into the emotion recognition system using Markov Models. The second was to dive deeper into reading papers to get a better idea for how the open source ML tools used are actually working. This allowed me to strengthen my knowledge of Neural Networks and CNNs covered briefly in CPSC340 and 425. Because the readings and papers have a lot of overlap in terms of material and learning goals I have provided references to these overlaps throughout both sections. For high level layout purposes I have chosen to keep them seperated.

Youtube Chrome Extension

• Demo
• Outline of Implementation
  • Setting Up Chrome Extension
  • Capturing Frames
  • Face Recognition
  • Emotion Recognition
• Improving Emotion Recognition Using Markov Models
  • First Attempt Markov Model
    • Results
    • Next Steps
  • Hidden Markov Models
    • Transition Model
    • Observation Model
    • Single Value Observation
    • Vectorized Observations
    • Results
  • Resources
    • CPSC 322 + 422
    • Bert Huang - Virginia Tech HMM lecture

Paper Readings

• Reading #1 - A Brief Review of Facial Emotion Recognition Based on Visual Information

• Reading #2 - Neural Networks, Convolutional Neural Networks, ImageNet

• Reading #3 - Recurrent Neural Networks, Long Short Term Memory

Future Learning

• Further Implementations
• Loose Ends
• Further Explorations

cpsc448's Issues

Injected Youtube button not showing on first page load

Does not continue facial recognition after skipping in video.