davquar/voicecrm

Project for the Human Computer Interaction on the Web 2021 course at Sapienza University of Rome

VoiceCRM

Project for the Human Computer Interaction on the Web 2021 course at Sapienza University of Rome, thought by Prof. Emanuele Panizzi.

Content

• Report.
• Presentation.

Authors

Federico Cernera, Davide Modenese, Davide Quaranta, Eugenio Fernandes, Pietro Andrea Liberati.

voicecrm's Issues

Minor dialog/intent fixes

• When more actions are done (like new activity --> new contact) we can remove "finishing.dialog/end-new-contact.dialog" at the end of the first action and substitute it with an intermediate intent

• When more contacts are found, "self.speak_dialog("generic-data-done-repeat", {"data": utt_name})" is useless

• Maybe we can add a dialog in "last-activities", before reading the activities list

Remove "what do you want to do" in "list activities"

Add name, surname to dialogs whenever possible and useful

In dialogs like "a nickname that you use to refer to elon?", include at least the surname.

Test the "new reminder" compact intents

The compact intent set for "new reminder" may skip our skill and directly call the reminder-skill of mycroft, therefore we should fine tune our intent files to prevent it.

E.g. this should be avoided: "remind me to call michael jordan tomorrow morning → Sure, saving a reminder for tomorrow at eight o'clock"

Depends on #30

Sometimes the date parser fails

Normally extract_datetime should return None if there is no datetime in the utterance, but sometimes it fails.

Example saying "nevermind":

Traceback (most recent call last):
  File "/opt/mycroft/mycroft/skills/mycroft_skill/event_container.py", line 73, in wrapper
    handler(message)
  File "/opt/mycroft/skills/crm-skill/__init__.py", line 267, in handle_new_activity
    parsed_datetime = parse.extract_datetime(utt)
  File "/opt/mycroft/mycroft/util/parse.py", line 122, in extract_datetime
    default_time)
  File "/opt/mycroft/.venv/lib/python3.6/site-packages/lingua_franca/internal.py", line 597, in call_localized_function
    return _call_localized_function(func, *args, **kwargs)
  File "/opt/mycroft/.venv/lib/python3.6/site-packages/lingua_franca/internal.py", line 575, in _call_localized_function
    in kwargs.items()
  File "/opt/mycroft/.venv/lib/python3.6/site-packages/lingua_franca/lang/parse_en.py", line 707, in extract_datetime_en
    words = clean_string(text)
  File "/opt/mycroft/.venv/lib/python3.6/site-packages/lingua_franca/lang/parse_en.py", line 641, in clean_string
    s = _convert_words_to_numbers_en(s, ordinals=None)
  File "/opt/mycroft/.venv/lib/python3.6/site-packages/lingua_franca/lang/parse_en.py", line 48, in _convert_words_to_numbers_en
    tokens = tokenize(text)
  File "/opt/mycroft/.venv/lib/python3.6/site-packages/lingua_franca/lang/parse_common.py", line 255, in tokenize
    for index, word in enumerate(Normalizer.tokenize(text))]
  File "/opt/mycroft/.venv/lib/python3.6/site-packages/lingua_franca/lang/parse_common.py", line 34, in tokenize
    utterance = re.sub(r"([0-9]+)([\%])", r"\1 \2", utterance)
  File "/usr/lib/python3.6/re.py", line 191, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

Continue the task execution after the user adds a contact that doesn't exist

When the user asks to register something about a contact that doesn't exist, and mycroft starts the nested task, it should continue from where it was interrupted.

The handle_new_contact function may return the just created contact :)

[New contact] the "back" action word is not handled

Successful compact intents stop at the first state

Even if all the entities are recognized by the intent parser, the execution stops at the state 0, because it is not incremented when using compact intents.

Last activities takes the option "continue" even if there are no more activities

[New contact] compact intent takes "surname" into the surname field.

Bug in the intent parser (#72)

Update docker-compose to use a local build

The official mycroft image is outdated. We should use a locally cooked Mycroft build.

Pronounce real contact names instead of utterances, when possible

See:

self.speak_dialog("generic-data-done-repeat", {"data": utt_person})

Instead of utt_person, we should use the real contact name (whenever possible).

Ask for the nickname when adding a contact with the same name+surname

Use different phrases when repeating the same question

Example:

\- you already have jack sparrow. I'll stop here.
\- are you sure this is another person?

Support deletion of the last registered object

The user should be able to delete the last registered thing (contact, activity, reminder)

Add contact disambiguation

• Support having multiple contacts with the same name+surname.
• Implement a "nickname" field, and support it for searches.

[New relationship]: "I don't know this person yet" should specify who

set that elon and mike horse are friends                                                                                                         
 >> I don't know this person yet . should I add them? 

The phrase should include the contact that you don't have ("i don't know mike horse [...]")

Improve activity listing

When mycroft reads you a list of activities, the time is unnatural. It should be improved.

Repeat the extracted data when asking the next question

This implicitly tells the user whether mycroft has correctly understood the request or not, and at the same time proceeds with the task.

Migrate to Adapt

The Adapt intent parser is more powerful and flexible; it also doesn't have the infamous bugs of Padatious.

To migrate it should be sufficient to:

• convert the intent strings into regexes.
• appropriately tune the intent builder with require and optionally to allow both long and compact intents.

As an alternative to the last point, it is also possible to use both parsers: Adapt for compact intents and Padatious for simple ones.

Add "compact" intents

• New contact
• New activity
• Add reminder
• List activities

3 possible ways:

1. Define "natural" intents with fields --> nice but with possible errors
2. Define "robotic" intents with fields --> ugly but with less errors
3. Use the Adapt intent parser to validate fields and pass results to the already defined functions --> cool but ?

Add automatic behavior tests

• https://mycroft-ai.gitbook.io/docs/mycroft-technologies/mycroft-core/testing
• https://mycroft-ai.gitbook.io/docs/skill-development/voight-kampff/test-steps

Ask for contact disambiguation in new contact task

Handle repeat name for task new activity

Handle the repetition of the name when it asks "whom". Replace ask_yesno with wrap_get_response and voc_match.

Handle "empty" user inputs that result in None

When the user doesn't speak in time, or doesn't speak at all, mycroft understands None

Support relationships

Contacts can have relationships between them, e.g. son, father, friend, spouse, etc...

"relationships": [
            "<contact_id>": {"<relationship>", ...}
            ....
]

Relationships:

• friend
• spouse
• partner
• colleague
• collaborator
• son-daugther/parent
• sibling
• cousin

Improve vocal interaction - usability tests

Improve user experience with a better and satisfying vocal interaction:

• Check dialog in speak instructions

• Complete intent files

• Optimize, if possible, Mycroft vocal comprehension and adjust the speak interval time

Extend intent phrases

New activity

• record
• "can you add an activity"
• "add new reminder with chiara"

Remove stopwords from contact searches

Right now if you say something like "with elon musk", it does not match "elon musk". The contact search function should remove stopwords from the utterance.

Handle alternative task flows

Allow the user to stop the interaction, to go back, rephrase or repeat something, etc.

• New contact
• Add reminder
• New activity
• Latest activities

Put pronounced strings/intent in the locale directory

Instead of hard-coding phrases or user intents in the code, put them in files under the locale directory.
This can allow us to define variations and to translate the skill.

Delete relationship when deleting a contact

Blocked by #64 and #47

[New contact] the nickname field can be None

If the user doesn't provide a nickname, it becomes None. We had fixed it, but apparently something has broken it again.

Allow the user to add another info to/with the same contact

Ask the user "do you want to add another activity/reminder with/about this contact?" before exiting.

• New activity
• Add reminder

Extend intent phrases

Intents

• add N S to my contact list
• can you add someone to my contacts
• an you add N S
• can you add a contact
• another activity
• can you tell me the last activities

Action words

• i don't know X --> skip
• #40

"List activities" should read from the most recent to the oldest (within a group)

Ignore question repetition in the answer

Ignore things like "the name is X", etc.

General code cleanup

• Enforce Python code style
• Move utility code outside of the main file, if possible
• Add comments when needed; remove unnecessary comments

The "list activities" function doesn't ask to continue

Better interaction for the deletion of an action

When the user wants to delete the last action, the interaction between her/him and mycroft is very poor. We need to expand the set of phrases that mycroft says during this interaction.

Improve the action words recognition

See:

• https://github.com/MycroftAI/mycroft-core/blob/dev/test/unittests/util/test_parse.py
• https://github.com/MycroftAI/lingua-franca/blob/master/lingua_franca/parse.py

lingua_franca match_one/fuzzy_match

E.g. when the user says something containing "stop" or any action word, but there is no exact match, we can ask the user for confirmation.

Support birthday reminders

• If the birth date is set, the skill should remind you about the birthday by itself.
• The user should be able to ask "what are the upcoming birthdays", and the skill tells the birthdays of the next 7 days.

Add diagrams

UML diagrams to describe the interaction flow.

Fix error when extract_datetime is None

date = parse.extract_datetime(utt)[0]

The problem was introduced after adding [0]

Add validation

The user should not be able to insert invalid things.

Dates/times

• New reminder --> future
• New activity --> past
• Birth date --> past

Genre

Male, female, other

General data

Ignore:

• Yes ==> ~treat as "repeat"
• No ==> ~treat as "skip"

---

Also, validation code should be put - if possible - inside the validator field of mycroft methods, like get_response.

The nickname is always saved

The nickname is always saved even if the user skipped the nickname question

Say multiple things in contact disambiguation

Everything useful to disambiguate someone.

Add possibility to go back when listing activities

The supported actions should be extended to {repeat, back*, continue*, exit}

*: when needed

Activity listing asks 2 times for [continue, repeat, exit]

It should ask only one time.

Reminders should also tell you the person who is related to

The utterance that we pass to skill-reminder via the message bus should somewhere include the person who the reminder is related to. E.g. remind me to {activity} related to {person} on {datetime}