scalability4all/voice-enabled-chatbot

Implementing a voice enabled chatbot which converses with a user via their voice in natural language. Website: https://scalability4all.github.io/

Voice Enabled Chatbot

Implementing a voice enabled chatbot which converses with a user via their voice in natural language. The user should be able to interact with the application like a voice assistant and appropriate response will be returned by the application (also through voice). The number of topics to converse upon will be fixed however the user should be able to converse through natural language.

If we have topics like the weather, location information, inventory information etc the application should be able to converse in that particular topic. Like, if we have questions such as:-

Hey, what is the weather today? - Weather Information

I want to know if there are any shopping stores near me? - Location Information

Will it rain today? - Weather Information

Can you tell me availability of item in store X? - Inventory Information

Initially the application should be able to identify each particular topic and then return the appropriate response. After that, the chatbot should carry on the conversation regarding that particular topic. For eg:

User - I want to know if there are any shopping stores near me.

Chatbot - These are the nearest stores:- A, B, C. Where do you wanna go?

User - I want to go to the most popular one among them

Chatbot- All right, showing you directions to C.

So, mainly the chatbot should formulate appropriate responses to incoming questions and carry on the conversation. Keep in mind, the conversation would be in natural language and the user has returned sufficient information.

Concept

Before starting conversation, bot will fetch the location of the user and other details to give personalized results.

Step 1: Speech-2-Text: Given a speech through Microphone, store it and convert it using SpeechRecognition and PyAudio libraries.

Step 2: Topic Modelling: Get Entity and Intent of chat using model with a corpora. To get the trained model, we will use the classifier to categorize it into weather, location and inventory. After that using RASA-NLU with Spacy library, we will extract the entities.

Step 3: After Finding Intent and Entity, we will set the model using the following method: Intent = Weather: Based on entity specified, We will use weather API to fetch data of location. Intent = Location: Following conversation flow: Get Stores located or Any Nearby Stores Choose Store Inventory Details about Store

Step 4: Use cache mechanism to give result about recently used query.

Changelog

• v0.1.1

  • Added support for speech to text
  • Added support of weather and google places api
  • Added basic introduction chat
  • Added voice support to chat

Usage

To change the language, enter the BCP-47 code from language support. If you want the language to be English (default), press enter.

Arguments:

         For now, you can use it for following domain:
         Introduction: Hi, Hello..
         Weather: Get weather info
         Google Search: get any google search
         Wikipedia Search: What is "query"?
         Google Places: Get me best places.

You can quit by pressing Ctrl+C

Build the application locally

• Clone the repo
  • Clone the voice-enabled-chatbot repo locally. In the terminal, run :
    git clone https://github.com/satyammittal/voice-enabled-chatbot.git
• Install the dependencies
  • We need PyAudio, a cross-platform audio I/O library.For this run :
    sudo apt-get install portaudio19-dev (linux)
    brew install portaudio (mac)
    
  • Further, install other requirements using :
    pip install -r requirements.txt
    
  • Using windows, install other requirements using:
    pip install -r requirements_windows.txt
    
  • Install english stopwords using :
    python -c "import nltk; nltk.download('stopwords')"
    
  • The pyowm is supposed to be instable in windows.
    
• Configure Google API Key
  • Go to the Google Cloud Platform Console.
  • Select or create the project for which you want to add an API key.
  • Click the menu button and select APIs & Services > Credentials.
  • On the Credentials page, click Create credentials > API key.
  • Copy and paste the API key in config.py file.

Run the application

Run the application using command - python chatbot.py

Milestones

1. Completing chat bot so that it works on multiple domain specified through config.
2. Adding classification techniques for intent seperation.
3. Automated method for Entity creation from sentences.
4. Use cache mechanism to give result about recently used query.
5. At the end, the deliverable will be to implement user interface for a sample chatbot implemented.
6. We will also extend it to create plugin for companies requiring chatbot. They can put their domain in config file and data separately to give personalized result.
7. Multi Language Support

Sample output

References

• Speech To Text - Python
• Topic-Focused Summarization of Chat Conversations

voice-enabled-chatbot's Issues

Group conversation comments classification

Is your feature request related to a problem? Please describe.
Yes. Typically in group conversation of chat apps, we find that one of person says for example 'I got into university X' and there will be 50 messages like - 'wow', 'nice', 'congrats', 'you did it', etc. All of these messages have equal status that of a "message" so when another user comes online 3 hrs later, he has to scroll up through 50 such messages to find out what the main message actually was. So in active group people miss a lot of imp messages.

Describe the solution you'd like
Ideally such messages should be identified as comment and by default shown in collapsed mode under the main post. In above scenario, main message would be "I got into university X" and below it should be "50 comments made" and no more, clicking on this link should open them all.

Describe alternatives you've considered
Providing a default comment option but that is user dependent and will require educating users to use chat app more responsibly which is not always practical. Its better if we can do this using NLP.

Additional context
One challenge using NLP approach to solving this problem is, for multiple languages, we will need multiple trained modules which is not scalable across multiple languages. This is an open problem for us at the moment and it will be amazing if someone can come up with a solution that is scalable across multiple low resource languages.

[Workspace] Need to Integrate our chatbot to Slack

Is your feature request related to a problem? Please describe.
It will be nice experience and members can experience chatbot features directly in channel.

Describe the solution you'd like
Will be setting up infra for this. Ideas are welcome. Assigning it to myself for operation work.

Describe alternatives you've considered
If you have something in your mind which will make it more interesting, cheer it up.

Additional context
Add any other context or screenshots about the feature request here.

Modify the weather API

Is your feature request related to a problem? Please describe.
When using weather API, It is giving us Banglore weather as it is hardcoded in the api call. It would be better if we can get the User location Weather

Describe the solution you'd like
First, we can get the Latitude and Longitude of the User location, then we can use those to make an api call. By that, we can provide the user with their location weather

[Feature] Multi Language Support

Currently We could only speak to chat bot in english.

For first phase, I need language to be setup in config file.
For next phase, We can research whether multi language can be given.

Basic runtime errors and text input issues

I found some issues in runtime as well as while giving some inputs.

1. Error while working with pyttsx3 module.
2. Getting "AttributeError: module 'urllib' has no attribute 'urlopen' " while working with python 3. urlopen is a function for python 2.7.
3. Detected case insensitivity for text input. Only accepts "How are you?", does Wikipedia search for "how are you?" , "how are you." and other variants. The output should be "I am fine." for all the cases.

Desktop
Ubuntu 18.04.4 LTS

Modify the jokes feature

The jokes feature is not working properly. I would like to improve it. Assign me this issue.

[Feature] [UI] Need some sample User Interface

Currently It is just using command line interface.

I need a sample user interface that take our code and users can see our chat running on small domains.

1st step : I would recommend using Flask application.

Next Step: Android Application will also be appreciated.

[Feature] Chatbot for Signup form

Is your feature request related to a problem? Please describe.
A chatbot in place of Sign up forms are more interactive. Providing clients the way to register in interactive manner rather than with forms

Describe the solution you'd like
API and storage at backend. We can use a table storing which forms are required by client.

Describe alternatives you've considered
Api from the Google form or any other agent

Additional context

Features to improve project

Is your feature request related to a problem? Please describe.

1. When the project is executed, files like .pyc are created, that need to be ignored when uploading to git.
2. Python2 has been updated to python 3, and when running the project with python 3 they give few errors.
3. README file should say how to complete Google API KEY and Weather API KEY
4. The command 'tell me a joke' is not working and give an error NameError: name 'jokes' is not defined

Describe the solution you'd like

1. Add a .gitignore file and venv for virtual environments
2. Migrate to Python 3, there are a few differences like the method raw_input.
3. Modify the README file.
4. Modify joke feature

I would like to improve it.

[Microbot] Talk It Out - bot which help you heal from saddness

Is your feature request related to a problem? Please describe.
actually there is no unique solution for depression and sadness. Every person has its own reason and its own path which leads to its situation.

Most difficult things people face while reliving their mood is:

a) Trust: whom to share their thoughts and get ack.
b) how to maintain anonymous status and get solutions.

Thats what i think we need a bot which can be an approx solution for emotional healers.

Describe the solution you'd like
I want to have emoticons being used in rchat

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Alternate Opensource geocoding API

Is your feature request related to a problem? Please describe.
To use google geolocation api, User has to fill the credit details. Instead, we can use opensource geocoding api available.

Describe the solution you'd like
We can use alternate geocoding api available to get the same feature.

[Feature] [MicroBot] Add COVID Info Microbot

Is your feature request related to a problem? Please describe.
Taking current situation into consideration, It would be good to have a bot for COVID. it should tell current active/ total cases based on your location.

Describe the solution you'd like

"Am I safe from COVID":
based on location, it tells whether you are in green/yellow/red zone

"Can I go outside?"
You are under lockdown. Please stay home.

"How many cases are there?"
There are 289 total cases, 50 recovered and 0 deaths.

Describe alternatives you've considered
API parsing and a simple tool.

Additional context
Add any other context or screenshots about the feature request here.

[FEATURE] Add Testing for successful build for commits and PRs

Currently for PRs and commits, It will be easier for reviewers and users to see whether the program is still successful building.

It will also enable us to see which version of python we are supporting.

I will recommend travis/jenkin jobs but Recommendations are welcome.

[Feature] Auto complete Or Suggestion of questions as user type

Is your feature request related to a problem? Please describe.
Later on when we have integration of Microsoft, to have a good user experience. It would be nice to have auto completion or suggestion of questions while user type

Describe the solution you'd like
We could take help of some search engines like elasticsearch, spark-hadoop

Describe alternatives you've considered
Dataset could be union of all Microsoft training set and could use online learning

Additional context
Nice to have but not high priority currently.

[Feature] - News Query Bot

Is your feature request related to a problem? Please describe.
Our chat application has news bot which sends news based on category. It would be good where we get bot answering question from that bot.

Describe the solution you'd like
Lets say:

bot send news "Price hike of petrol in India"
user can ask question: how much hike, from when it is applicable

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[Feature] Automate conditions using Topic Modelling

Currently we use a lot of if conditions for separate lot of varieties in chat.

We need to automate it using topic modelling given in paper:
https://link.springer.com/content/pdf/10.1007%2F978-3-642-36973-5_88.pdf

Any other idea is also welcomed.

[Feature] Intent Detection using classification

Currently we are using rule based learning to learn intent.

We need to use supervised learning techniques to create automated intent.

Preferred Steps:
a) Add domains in config file
b) Each domain has data in [domain].txt
c) for now, we will use label classification techniques to separate intent.

References:
https://towardsdatascience.com/a-brief-introduction-to-intent-classification-96fda6b1f557
https://mrbot.ai/blog/natural-language-processing/understanding-intent-classification/

Semantic similarity between two sentences

Hello,

As per the requirement of a chatbot, I feel it is important to have an algorithm which calculates the semantic similarity between the two sentences.
For example, In the existing code, similar sentences like, "how are you?" and "how are you doing" have been quoted separately, while the reply will be the same in both cases. Also, these sentences can be framed in many different ways which depends on the user and will not be possible to quote individually. Therefore, we can implement a system which can calculate the semantic similarity between the two sentences.
Thus, the code will be more efficient and open to varied queries.

Autocorrect feature for input.

I think as humans we often tend to make spelling errors while typing and it would be great if the chatbot could understand our request/question despite of the spelling errors. So an autocorrect feature would really help.

[BUG] FIX pep8 conflicts

python files are not yet written clean. I want it to support pep8.

A PR doing it is appreciated.

[Feature] More APIs added

Currently we use following APIs

a) Weather: OpenWeather
b) Places/Maps/Paths: Google Places API

More APIs can be research out that can do other domain like rainfall, pizza order.

This will help us adding more type of chatbot.

[Feature] FaceFilter Bot

Is your feature request related to a problem? Please describe.
Yes. when users identified as in 'strong moods', even intelligent replies are not always very effective. A creative response that can possibly alter their strong mood positively showing them their Face with filter that is relevant to situation in the conversation.

Describe the solution you'd like
Using Dlib for live face detection with OpenCV and processed photo to make filter. When to use this microbot will be decided by Intent classifier as per architecture.

Describe alternatives you've considered
For face detection, Dlib works better than OpenCV and other similar libraries but still face detection contour can be made much more robust and stable with custom modifications.

Additional context