sharathbhushan/speech_module_client

#Research Project on Implementation of Speech Module on Ohmni Telepresence Robot

This documents serves as a complete description to setup and start a speech module on ohmni robot

Maintainer: Sharath Nataraj Email id: [email protected]

Filetree

src

• cogrob_chatbot/
  • .rasa/
    • cache/
  • actions/
    • pycache/
    • init.py
    • actions.py
  • data/
    • nlu.yml
    • rules.yml
    • stories.yml
  • models/
    • 20220926-120721-accepting-quarter.tar.gz
  • tests/
    • test_stories.yml
  • config.yml
  • credentials.yml
  • domain.yml
  • endpoints.yml
• efficient_wakeword/
  • src/
    • hello_kai/
    • sample_refs/
    • init.py
    • audio_processing.py
    • baseModel.tflite
    • effnet_word.py
    • engine.py
    • generate_reference.py
    • ibm_generate.py
    • logmelcalc.tflite
    • package_installation_scripts.py
    • streams.py
    • tO_README
  • CMakeLists.txt
  • package.xml
• flask_ros_demo/
  • flask_demo/
    • launch/
    • msg/
    • scripts/
    • CMakeLists.txt
    • package.xml
    • requirements.txt
  • README.md
• porcupine_wake_word/
  • src/
    • kai_wake_word.py
  • CMakeLists.txt
  • [To Readme.txt](./src/porcupine_wake_word/To Readme.txt)
  • package.xml
• porcupine_wakeword_model/
  • 112b040e-7860-46d0-bf6e-2437c35ad026.zip
  • a47f61cc-1c94-4869-99bf-0d88d98472d0.zip
• rasa_ros/
  • launch/
    • dialogue.xml
  • scripts/
    • dialogue_interface.py
    • dialogue_server.py
    • rasa_action.sh
    • rasa_server.sh
  • srv/
    • Dialogue.srv
  • CMakeLists.txt
  • package.xml
• tts/
  • src/
    • pycache/
    • nix/
    • pyflite/
    • .gitignore
    • LICENSE
    • README.md
    • ed_tts.py
    • espeak.py
    • mbrola.py
    • nix-ljspeech-stochastic-v0.1-20220916T150228Z-001.zip
    • nix.py
    • pico.py
    • requirements.txt
    • sound.raw
    • test.py
  • CMakeLists.txt
  • package.xml
• vosk_ros/
  • config/
    • params.yaml
  • launch/
    • start_dialogflow_service.launch
    • start_rasa_service.launch
  • models/
    • vosk-model-small-en-us-0.15/
  • msg/
    • speech_recognition.msg
  • src/
    • tts_engine.py
    • vosk_node.py
  • CMakeLists.txt
  • README.md
  • package.xml
• CMakeLists.txt
• Ohmni_kai.zip
• Our_data.db
• mailer.py
• requirements.txt
• webbhook.py

Robot Setup

• pull this repo into robot by using linux shell follow instructions from : https://docs.ohmnilabs.com/

• cd speech_module

• pip install -r src/requirements.txt

A google mail id has been created to manage multiple services for the robot

email id: [email protected]

password: Ohmni@123@iaas

This module allows you to choose between two wake word recognition models 1 and 2

1.Procupine Wake word Recognition Setup

• This repo contains pre-trained model in the folder porcupine_wakeword_model/ Extract the contents and save it outside the repository since the model is quite big.
• This model can be directly used to for inference from kai_wake_word.py
• The path to extracted model can be passed while astarting the wake word, Remember the porcupine wake word must be started before launching the other nodes.
• picovoice console: https://console.picovoice.ai/signup
• Login through the given google account
• training and creating access key: https://picovoice.ai/docs/porcupine/

launcing the Porcupine wake word

• working with pre-trained models: rosrun porcupine_wake_word kai_wake_word.py --access_key 7segjTjkamk6oOQb/SLE8OBex4vHMdGxW+Z2cxX1EthTIJ13wNPnqg== --keyword_path path_to_extracted file/Hello-Kai_en_linux_v2_1_0.ppn
• Custom trained model: rosrun porcupine_wake_word kai_wake_word.py --access_key access_key_generated_from the previous steps --keyword_path path_to_extracted file/Hello-Kai_en_linux_v2_1_0.ppn

2.Efficient Net Wake Word Recognition Setup

• some extra requirements, pip install librosa && pip install ffmpeg
• This repo also contains a pre-trained efficent-net model sample_refs/,
• remember to change the directorty path of the pre-trained model in line 12 of effnet_word.py
• it is also possible to train a new wake word using the instructions https://github.com/Ant-Brain/EfficientWord-Net#generating-custom-wakewords
• simply replace json modle inside the sample_refs directory and change the directory path of newly trained model at line 12 of effnet_word.py
• remember to run the wake word node before starting other nodes.
• node can be started by rosrun efficient_wakeword effnet_word.py
• rostopic pub -1 /eff_wake_word_status std_msgs/String "data: 'on'"

Next is the speech recognition models

by default this branch contains only vosk as STT model, switch to devel branch to find more models

Vosk Speech Recognition Engine

• Vosk STT is pre-compiled with models and can be started easily with a simle command
• rosrun vosk_ros vosk_node.py

Choosing Different NLU models

• This repository provides two differet NLU models which can be choosen by choosing different launch files in launch/

1.Dialogflow Launch

• login to google cloud platform from the given google credentials
• create a servcice key using the email id given and sownload the json file: https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-console
• add this to bot environment shell by export GOOGLE_APPLICATION_CREDENTIALS='/path_to_doownloaded_file/service_key.json'
• roslaunch vosk_ros start_dialogflow_service.launch

2.RASA Launch

• navigate to cogrob_chatbot/ and start the rasa server
• rasa run -m models --endpoints endpoints.yml --port 5002 --credentials credentials.yml --enable-api
• roslaunch vosk_ros start_rasa_service.lauch
• this node is currently under development this can be used to get replies on the frontend but some features like apointment scheduling system and controlling IOT systems may not work, refer rasa docs to change intents and rules inside cogrob_chatbot/

Quick start

Robot Setup

• contact [email protected] for dialogflow service key.
• clone this repository onto the robo
• add the service key to evironment using export GOOGLE_APPLICATION_CREDENTIALS='/path_to_doownloaded_file/service_key.json'
• catkin_make
• rosrun rosrun porcupine_wake_word kai_wake_word.py --access_key 7segjTjkamk6oOQb/SLE8OBex4vHMdGxW+Z2cxX1EthTIJ13wNPnqg== --keyword_path path_to_porcupine_model/Hello-Kai_en_linux_v2_1_0.ppn &
• roslaunch vosk_ros start_dialogflow_service.launch
• this starts all the required nodes for the operation

Local Setup

Configuring Dialogflow service

• this configuration can be made from a host computer and not on ohmni robot
• login to google dialogflow with the given credentials at the start of the readme file
• install ngrok: https://ngrok.com/download
• start ngrok service ngrok http 8000 this connects dialogflow with webhook.py application
• copy the url generated and paste it in fullfilment section of dialogflow console with a route /webhook, the url looks like : www.example_url.com/webhook
• save the changes
• the console can be used to add changes to intents

Conecting to Home Assistant

• Use the SD card provided with the deliverables, and insert it to Raspberry pi 4 with atleast 2GB of RAM, the SD card is configured to automatically connect to Service lab's WiFi. and boot Home Assistant.
• Check the IP adddress of Raspberry pi and make a note of it
• add the IP address of Raspberry pi in line 20 of webbhook.py and line 17 of mailer.py, mailer.py provides an example schema of how the employee database can be structured, more customization and development can be made to suite the needs.
• start both the applications python3 webook.py and python3 mailer.py
• setup of mailer.py is rather simple nacvigate to app passwords in google account setup and create a password with other application. copy and paste the password at line 15 of mailer.py
• This ensures mails can be sent through this python script.