robotpt/vision-project

A robot interaction to help low vision people read more.

vision-project's Issues

Figure out how we will use audio features for robot interaction adaptation

This issue will be a place for discussion and collecting resources for using audio features for robot interaction adaptation

It is one goal of the project to use audio data from the participant reading various texts in some meaningful way to cause the robot to adapt the interaction. The first part of this is to understand what features we can sense and from there we can think about what we can adapt based on this information.

Sensing

Possible features

• Their accuracy in speaking the words
  • Perhaps compare a transcription of them through Amazon Polly to the text they're reading
  • It will be hard to do this on a phoneme level outside of a transcription service because language is complex and it will likely be challenging to find a similar dataset
• Their reading speed
  • We could find their reading rate in words per minute by starting and stoping the interaction - we could also see if they read most of the text by looking at the transcript of the recording
• Their reading confidence
  • Measured by features like how shakey is their voice, etc.

Possible tools

• openSmile

Adaptation

Possible ways of adapting the interaction

• Giving simple performance feedback
  • For example, "You are reading 25% faster than last week."
• Giving advice
  • For example, "Maybe you can try slowing down to make sure that you're reading clearly."

Make a script to run the interaction from Dataplicity

We are having trouble restarting interactions on QT likely because of how it's network is setup. Instead can make a script to start the interaction from Dataplicity. This is an easy workaround for now and will let us focus on getting the full interaction together.

Interaction folder is in the top directory

The ros-vision-interaction folder is in the top directory. It would be more consistent to put the directory in the src folder.

The interaction doesn't restart correctly on QT

For some reason (probably network reasons), the Docker container doesn't restart correctly on QT. A possible approach to solving this is by installing the required software locally.

Extend the multistage dockerfile to include running the interaction

It would be great to run the interaction directly, as we did in the ABM project.

If we want the interaction to continue on restarts, we can do step 7 here.

Misleading documentation header

The title of "On your local machine" is kind of confusing with Docker, since Docker can also be installed on the local machine but not native.

Install project dependencies in Dockerfile

This project uses packages including robotpt_common_utils, schedule, and freezegun that need to be included in the Dockerfile.

Fix CI after Github Actions update

Fixes #17. Github seems to have disabled the set-env command (failing tests, here). We should look at how to fix CI. I see the same issue in cordial.

Add state database

We will need to have a persistent state database of some sort. This database should handle concurrency issues, that is being written to by several sources at the same time.

Make a way to get feedback by email on videos

This can be accomplished by creating a node that uses audrow/email-keyword-matcher (which requires Python 3.6 or newer) and having that node listen for file upload events from robotpt/ros-data-capture's aws_uploader. When a new file is uploaded (possibly after regexing for video formats), an email is sent with possible modes of feedback. Once an email reply is recieved, that feedback can be sent to the main controller to use.

Also, robotpt/ros-data-capture's directory_size_manager should be used to make sure that the folder where the videos are saved doesn't grow too large if videos are being recorded but not uploaded, say, due to slow internet.