dynamicgenetics/the-learning-machine

Public engagement project using interactive human-in-the-loop online machine learning to teach machines to recognise human emotion (in development)

Can We Teach Computers to Understand Emotion?

Researching the Digital Generation

Each of us lays down invisible digital footprints as we move around and interact with the modern world. Our movements are logged by our phones and sat navs, our activity is monitored by wearable sensors, and our social media interactions form part of a worldwide conversation.

This shared digital information allows us to engage with the world in ways that were unthinkable for previous generations. Businesses use this huge amount of data to understand their customers and offer services such as finding the best route through a busy city or suggesting products you might like based on what you’ve just bought. But our digital footprints are useful for health researchers too.

They can use digital information shared by people taking part in their research to understand important issues such as the spread of disease, or the development of mental health problems. Some types of information, such as the emotions we express on social media, are easy for most humans to understand. But health researchers often work with groups of thousands of people, like Bristol’s famous Children of the Nineties study. It would be impossible and intrusive for researchers to read and interpret every social media post themselves. Computers, on the other hand, work extremely quickly, but have difficulty with some tasks that humans typically find straightforward.

Can human and artificial intelligence work together on this, combining the strengths of both to improve human health? Or, to put it another way, can we teach computers to understand emotion?

The Learning Machine

The learning machine is one component of our Curiosity Toolkit element which aims at demonstrating how computers can work alongside humans to learn how to understand emotion, using both emotional faces and a curated set of tweets.

As a standalone experience, each visitor begins with a naïve machine that randomly sorts faces or words to different categories.

They begin to sort the faces or words into categories and the machine begins to learn how to recognise emotions in faces or words, updating its understanding of the categories each time the visitor makes a correction.

This is an example of how machines and humans can work together: humans have expert knowledge of emotions, but machines are faster at processing images and words.

With human-in-the-loop online machine learning we can get the best from both humans and machines.

Demos

Instructions on how to setup and initialise demos, are reported here

Credits and Acknowledgments

This project and the Curiosity Toolkit have been developed by the team at Dynamic Genetics Lab in partnership with the Jean Golding Institute, the Public Engagement team at the University of Bristol, and We The Curious.

The project is part of the Curiosity Challenge Competition promoted by the Jean Golding Institute for Data Science and the Alan Turing Institute.

The Curiosity toolkits will contribute to the forthcoming Open City Lab exhibition at We The Curious. More information here.

the-learning-machine's Issues

Donuts do not update in Frontend for images on board

The donuts do not change nor updates after image annotation, despite values from backend are passed to frontend changed.

This is particularly easy to spot if default number of images is decreased (e.g. to 2 ) and a non-retrained model is used (as it's the case of having heavy shifts in performance)

After a quick check, I noticed that the refreshNodeArray function in d8r.js updates links from server response, but doesn't do anything with donuts.

Isn't there were the donut update should be performed ?

Frontend: faces could go beyond new hexagon boundaries

First public shout out to @OliverDavis for the new improvements to the UI (see branch: launch)
It is simply OUTSTANDING 🤩

(See screenshots below - cc/ @ninadicara )

Also (from Screenshot 1):
faces should be constrained within the new Hexagon Boundaries.

Backend: Add `stats` JSON object to `BackendResponse`

Continuing in this issue the conversation started in the latest PR

(@OliverDavis comment)

How about this as a return object for both the annotate and forget endpoints?

{ "data": [current return object], "stats": {name, value for model fit stats} }

RE: I think that makes lots of sense. Also, appending the new stats objects to the current backend response is super quick to do too.

I think we should just decide what the stats object should contain, and which metric we want to calculate.

One first hypothesis can be to generate a per-class score (e.g. per-class precision), along with an overall score for all the faces currently on the board.

In other words, the stats object can have the following structure:

{
  stats: { 
      happy: happy score in [0, 1]
      sad: sad score  in [0, 1]
      angry: angry score  in [0, 1]
      disgust: disgust score  in [0, 1]
      fear: fear score  in [0, 1]
      surprise: surprise score  in [0, 1]
      overall: overall global score  in [0, 1]
}

When a new face will come in, updated scores for old 24 + 1 faces on the board will be returned.

Would that make any sense?

Frontend: Not-Human Classification also updates Bar Charts

Dragging a face to "not-human" also updates the bar charts of performance.
We should check this out when we'll link backend integration.

Branch: launch