Giter Site home page Giter Site logo

nuance-communications / mix-demo-client-azstaticwebapps Goto Github PK

View Code? Open in Web Editor NEW
13.0 9.0 10.0 3.29 MB

Nuance Mix Demo Client for use with Azure Static Web Apps

Home Page: https://demo.mix.nuance.com

License: Apache License 2.0

Python 3.51% JavaScript 95.37% Shell 0.33% CSS 0.49% Dockerfile 0.07% Makefile 0.24%
nuance-mix conversational-ai text-to-speech dialog natural-language-understanding reactjs azure-functions azure-static-web-apps nlu chat-bots

mix-demo-client-azstaticwebapps's Introduction

Nuance Mix Demo Client

Develop and Troubleshoot Nuance Mix AI Powered Chat Bots locally and on Microsoft Azure.

Table of contents

Overview

This demo client offers a sample integration to Nuance Mix's Conversational AI Runtime Services, specifically

The client uses the HTTP/1.1 transcoded, or WebSocket transport version of the APIs (rather than the native HTTP/2 gRPC versions).

The prime purpose of this tool is to assist in Bot development and troubleshooting, including offering controlled hosted availability using Azure StaticWebApps.

Azure Functions with Core Tools are used throughout the development and deployment lifecycle.

For more information on how to leverage the client across various environments, see networking options, and be sure to secure your functions. Monitoring is made available through Application Insights.

Disclaimer: This sample client is not intended to illustrate a production-ready implementation; it is intended to aid development, demonstration, troubleshooting, and other such use cases.

About the Client

Setup

Please see the Exchanging data section in the Mix documentation. It is crucial to understand the concepts as you design your Mix projects and leverage this client -- it acts more like a gateway.

Functionality

  • Bot engagements using Nuance Mix's DLGaaS HTTP/1.1 or WebSockets API (web-gRPC not HTTP/2 gRPC)
    • Audio or Text Input, Audio or Text Output
    • PRESENTATION LAYER: Rich UI through conventions
    • DATA: Functions for development through Client Fetch, and when deployed, enables External Fetch usage
    • DATA: Client Fetch handler for Location Data supplied in userData
    • LOGS: Log Events, Filtering and Timeline Visualization
    • CHANNEL SIMULATION: Visual VA, IVR, SmartSpeaker, SMS (audio in/out, text in/out, or any permutation)
    • CHAT: Standalone mode for launching without logs and for data collection (/chat/ path)
    • THEMES: Supports branding with use of logo, assets like fonts and stylesheets
  • Speech Recognition using ASRaaS WebSockets API (web-gRPC not HTTP/2 gRPC)
  • Natural Language Understanding using NLUaaS HTTP/1.1 API (not HTTP/2 gRPC)
  • Text to Speech Synthesis using TTSaaS HTTP/1.1 API (not HTTP/2 gRPC)
  • Event data using the Log Events HTTP/1.1 API

Intended Local Use

Use Data Access nodes with the client_fetch configuration and write the integration layer locally with the intention of separate hosted Functions (through external_fetch). These Functions would be referenced and configured within Mix.dialog and Mix.dashboard respectively.

DevExample

Intended Hosted Use

The following illustrates a scenario where the client is deployed to Azure, and Data Access nodes have been configured to use external_fetch within Mix.

This simplifies the client handling, deferring to the Functions themselves, and offers lifecycle controls within Mix.

ProdExample

⚠️ There may be scenarios where client_fetch is appropriate; this has been set up such that dlgaas.js will invoke local ClientFetchHandlers.

Pre-Requisites

Quick Start 🚀

git clone [email protected]:nuance-communications/mix-demo-client-azstaticwebapps.git

Pre-requisite for either Docker or Native Host based:

make certs-setup

Assumes running the application using Docker:

make launch

If using Native Host, run two processes:

make native-run-api-secure
make native-run-app-secure

Getting Started

Makefile

Various opertions are described in a Makefile.

make help

CERTS:  certs-(prep|setup)
*LAUNCH:  launch
DOCKER:  containers-(build|run|restart|stop|status|logs|clean)
NATIVE.BUILD:  native-build-(app|api)
NATIVE.RUN:  native-run-(app|api)-(secure|insecure)
NATIVE.CLEANUP:  native-clean
DATA.ACCESS:  new-data-access-endpoint

There are two main modes of operation intended: docker and native host. Launch defaults to docker.

Certificate Related

make certs-prep

Installs the pre-requisities for certificate creation, namely openssl and mkcert. A package manager is used here: brew or choco depending on your platform.

make certs-setup

Creates certificates and stores password as needed for local use.

Quick Start

make launch

Primary command, sets up certificates and runs with docker compose.

Docker Host

make containers-build

Builds the containers needed to run the app. Reflects the full static web app, including the APIs.

make containers-run

Runs the containers for the app. Reflects the full static web app, including the APIs.

make containers-restart

Restarts the containers of the app.

make containers-stop

Stops the containers running.

make containers-status

Provides status for the containers.

make containers-logs

Follow the logs for the running containers.

make containers-clean

Removes all the images for the app.

Native Host

make native-build-app

Builds the client application package and brings in associated dependencies.

make native-build-api

Builds the functions package and sets up a virtual environment.

make native-run-app-secure

Runs the native application securely (leveraging certs).

make native-run-api-secure

Runs the native functions securely (leveraging certs).

make native-run-app-insecure

Runs the native application insecurely. Must update local.settings.json accordingly. Not default mode.

make native-run-api-insecure

Runs the native functions insecurely. Must update local.settings.json accordingly. Not default mode.

make native-clean

Cleans all application and functions related resources.

Data Access Endpoint Related

make new-data-access-endpoint

Helper to bootstrap a data access endpoint, leveraging a template.

Setting Up Certificates

If you do not have mkcert or openssl installed:

make certs-prep

To set up the necessary certificates:

make certs-setup

Option 1: Run with Docker

make containers-run

Option 2: Run Natively

Run the two processes.

App:

make native-run-api-secure

API:

make native-run-api-secure

Create a New Data Access Endpoint

make new-data-access-endpoint

See the official Azure Functions Reference - Python for more information.

Configuration

If using Docker, update the values in .env.

If running Natively, update the following environment variables to override the default values:

export oauth_server_url="https://auth.crt.nuance.com/oauth2"
export base_url_dlgaas="https://dlg.api.nuance.com/dlg/v1"
export base_url_nluaas="https://nlu.api.nuance.com/nlu/v1"
export base_url_ttsaas="https://tts.api.nuance.com/api/v1"
export base_url_logapi="https://log.api.nuance.com"
export oauth_scope="dlg nlu tts log"

When deployed, these can be set per deployment environment in the Azure Portal for the Static Web App.

local.settings.json

See resources/local.settings.json for the base file.

The default setting enable CORS for the Functions, and sets a worker count as desired. Be mindful of the http vs https distinction. By default, the application is set up to use https.

Email capabilities (Optional)

This sample offers a stub integration with SendGrid for email capabilities.

This can be used if creating a Data Access node with the name Server_Send_Email, which in turn will call the email-api-send locally.

To configure this integration, provide the following environment variables:

export sendgrid_api_key="<REPLACE_ME>"
export sendgrid_from_email="<REPLACE_ME>"
export sendgrid_custom_token="<REPLACE_ME>"

Note: Override values as needed in .env when using docker-compose ⚠️ The custom token must be provided in the Mix project named SENDGRID_TOKEN. This is to thwart unintentional usage.

Conventions for Rich UI

This client employs certain conventions within Mix.dialog to offer special types of rendering in the presentation layer.

To take advantage of these, navigate to:

  • QA node > Node Properties
    • User Input > Optional > Interactivity

Provide the following strings in the "type" field.

Special Input Types for QA Nodes:

  • email
  • phone
  • currency
  • date

To provide hints (aka placeholders), the sendData must include a variable with the type followed by 'Hint', ie. 'emailHint', 'phoneHint' and so on.

For Interactivity (aka selectables):

  • carousel
  • buttons
  • colorpicker (description should contain the HEX value)

Rich Text Modality

Note: when using Rich Text markup in messages, if elements contains classes from Bootstrap they will be rendered accordingly in this client. See chat.js to see the allowed DOM elements and attributes.

Stub Images for Carousels with Selectables

Images can be rendered when using selectables and the carousel type. Edit STUB_SELECTABLE_IMAGES in shared.js to experiment with resources.

Conventions for Link Handling

This client employes certain conventions within Mix.dialog to offer special types of event handling in the presentation layer.

To take advantage of these, simply mark up your Rich Text messages appropriately with the data attributes, depending on the intended interaction.

Intent or Entity

To trigger an intent or entity selection, leverage the following attributes:

  • data-mix-action="selectable"
  • data-mix-selectable-id - can be INTENT or the entity name
  • data-mix-selectable-value

Sample

<a href="#"
  data-mix-action="selectable"
  data-mix-selectable-id="INTENT"
  data-mix-selectable-value="iBuyPhone">
    Buy a Phone
</a>
<a href="#"
  data-mix-action="selectable"
  data-mix-selectable-id="ePhoneCapacity"
  data-mix-selectable-value="128GB">
    128GB
</a>

Simulated Inputs

To trigger a simulated user input, leverage the following attributes:

  • data-mix-action="input"
  • data-mix-input

Sample

<a href="#"
  data-mix-action="input"
  data-mix-input="$500">
    $500
</a>

Data Access Integrations

Using Client Fetch during development

This client offers developers the ability to use the client_fetch mode of the Data Accss node in Mix.dialog and integrate with an Azure Function.

This pattern would apply in a gateway-style integration, however the intention of this set up, is to eventually have the integrations use external_fetch pointing to deployed Functions.

How-To: Add Handlers for Data Access Nodes

Create a new Function specifying the endpoint URL and the data access node name to use if client-driven:

./scripts/create-da-handler.sh "weather-api-city-conditions" "Server_Weather_CityConditions"

Navigate to api/weather-api-city-conditions/__init__.py and start integrating.

By default, POST requests are expected with the body containing the sendData payload. Update api/weather-api-city-conditions/function.json if other methods are desired.

⚠️ Important: Once the function has been set up, add a handler to ExternalFetchHandlers in dlgaas.js. Create a function with the Data Access node's name, pointing to the newly defined endpoint.

Data Access Requests intended for the Client

There are cases where the use of Mix.dialog Data Access node's client fetch is intended for the end client (vs. the gateway pattern employed here).

How-To: Add Handlers for Client Actions

Add a handler to ClientFetchHandlers in dlgaas.js named with the Data Access node's name.

Client Location Capability

At this time, geolocation (lat/lng) is supported through the HTML5 API, when the client is running securely.

To request the client to provide it's location, set the Data Access node's ID to Client_Location_LatLng, and this will return a location object.

Sample Data Integrations

The data integration layer in this client is handled through Functions, and complimented by sample integrations in api/providers.py.

Each data access access is represented by it's own Function, in preparation for use when deployed and hosted.

This client illustrates integrations with: Yahoo Finance, AccuWeather, SendGrid and a Mock Phone Store.

Yahoo Finance

This API does not require authentication, in this case: the requests are made with no further checks.

  • POST /api/finance-api-asset-price

AccuWeather

Third party service authentication is required and done through passing the token in the payload of the request. In this case: apiKeyAccuweather is used with the service and must be saved in the session (stored in a variable).

  • POST /api/weather-api-city-search
  • POST /api/weather-api-city-conditions

SendGrid

Third party service authentication is configured as part of the Function, but a SENDGRID_TOKEN is required to execute calls from known sources.

In this case: the token must been assigned in Mix at the PROJECT level.

  • POST /api/email-api-send

Mock Phone Store

This integration illustrates the use of Dynamic Wordsets, consistently filtering options based on the user's selection, derived from static data.

  • POST /api/store-api-filter-phone-wordset
  • POST /api/store-api-request-purchase
  • POST /api/store-api-purchase

Branding

The chat skin can be branded.

Resources

A typical brand definition includes:

  • Logo (image)
  • Fonts
  • Stylesheet

Assets are located in /app/src/static/brands/

  • Fonts can be stored here as well

Configuration

The query parameter brand can be used to preload this configuration. Assign the value to the name of the subfolder.

Here is a sample CSS, assuming that the font files are also located in the directory.

/* Brand Assets */
@font-face {
  font-family: 'Font-Bold';
  src: url('./Font-Bold.woff2') format('woff');
  font-weight: bold;
  font-style: normal;
}
@font-face {
  font-family: 'Font';
  src: url('./Font-Regular.woff2') format('woff');
  font-weight: normal;
  font-style: normal;
}

:root {
  /* MyBrand*/
  --skin-logo: url("/brands/mybrand/logo.svg");
  --skin-logo-header: url("/brands/mybrand/logo-header.svg");
  --skin-logo-header-width: 30px;
  --skin-logo-header-height: 30px;
  --skin-logo-main-width: 25%;
  --skin-logo-main-height: auto;
  --skin-color: #336699;
  --skin-font: 'Font', 'Helvetica';
  --skin-font-size: 14px;
  --skin-chat-bg-color: #fff;
  --skin-header-text-color: #fff;
  --skin-header-bg-color: #27251F;
  --skin-footer-bg-color: #fff;
  --skin-message-text-color: #27251F;
}

Deployment

Publishing to Azure

To deploy this client, follow the Azure StaticWebApps deployment guide to publish. Update app/static/staticwebapp.config.json as needed.

Essentially:

  1. Create a GitHub Repo
  2. Create an Azure StaticWebApp pointing to the GitHub Repo (use: gatsby, point to app/, and api/)
  3. Configure accordingly in Azure
    • Create a Application Insights resource and link to StaticWebApp (APPINSIGHTS_INSTRUMENTATIONKEY)
    • Add environment variables (sengrid_api_key, sendgrid_from_email)

Considerations

  1. update content-security-policy for client - staticwebapp.config.json
  2. update client host (ASR_SERVICE_URL, DLG_SERVICE_URL) - shared.js
  3. update environment variables for functions - .env

License

This source code is licensed under the Apache-2.0 license found in the LICENSE.md file in the root directory of this source tree.

mix-demo-client-azstaticwebapps's People

Contributors

dependabot[bot] avatar nirvanatikku avatar sunilthorat09 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mix-demo-client-azstaticwebapps's Issues

Chat Panel should not AutoScroll when in focus

Use mouse events - up and down - to detect when the user is intentionally scrolling.
Prevent auto scrolling when the user is actively scrolling.
This behavior requires the user to effectively press and hold to avoid auto scrolling the chat panel.

ASRaaS wordsets feature

In the ASRaaS feature, selecting inline wordsets, it seems I can modify the values but at the end they are not really saved/used.

Add NTTSaaS support to demo client

Please update the demo client to use NTTSaaS and retire the old TTSaaS voices - this is for both standalone TTS speech generation, as well as within a DLGaaS flow.

Update title: include more insight

The window title should be meaningful such that a copy paste preserved as much information as possible.

  • Add "AppID"
  • Add "Tag"
  • Add "Language"
  • Add "Channel"
  • Add "Simulate"

While providing client id and secret id when we are creating new token in DLGaaS getting below error

Issue {"message":"Network Error","name":"AxiosError","stack":"AxiosError: Network Error\n at XMLHttpRequest.handleError (webpack-internal:///./node_modules/axios/lib/adapters/xhr.js:168:14)","config":{"transitional":{"silentJSONParsing":true,"forcedJSONParsing":true,"clarifyTimeoutError":false},"transformRequest":[null],"transformResponse":[null],"timeout":0,"xsrfCookieName":"XSRF-TOKEN","xsrfHeaderName":"X-XSRF-TOKEN","maxContentLength":-1,"maxBodyLength":-1,"env":{},"headers":{"Accept":"application/json, text/plain, /","Content-Type":"application/json"},"method":"post","url":"https://localhost:7071/api/oauth2-get-token","data":"{"data":{"clientId":"appID%3ANMDPTRIAL_ravi_khurana_servion_com_20230607T133003393380_origin_us%3Ageo%3Aus%3AclientName%3Adefault","clientSecret":"[redacted]","scope":"dlg asr tts log"},"headers":{"User-Agent":"mix-demo-client/2.1.0"}}"},"code":"ERR_NETWORK","status":null}

[feature request] profiles

It would be great to have profiles, as there used to be on the desktop demo client.

A profile would ideally hold the following information:

  • Client ID
  • Client secret
  • URNS
  • Config Tag
  • Scopes
  • Language
  • Voice
  • Start Data

Value cannot be null. (Parameter 'provider') - issue when launching Functions

Having issues when launching the Functions:

[2021-08-11T14:19:09.743Z] Looking for extension bundle Microsoft.Azure.Functions.ExtensionBundle at /Users/nirvana_tikku/.azure-functions-core-tools/Functions/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle
[2021-08-11T14:19:09.747Z] Found a matching extension bundle at /Users/nirvana_tikku/.azure-functions-core-tools/Functions/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/1.5.1
[2021-08-11T14:19:09.748Z] Fetching information on versions of extension bundle Microsoft.Azure.Functions.ExtensionBundle available on https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/index.json
[2021-08-11T14:19:10.294Z] Downloading extension bundle from https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/1.8.1/Microsoft.Azure.Functions.ExtensionBundle.1.8.1_any-any.zip to /var/folders/3p/g7l3vf152pn27xw80zsfk228fswycg/T/5d885179-f1af-4db9-9fe5-2f7324e7e160/Microsoft.Azure.Functions.ExtensionBundle.1.8.1.zip
Value cannot be null. (Parameter 'provider')
[2021-08-11T14:21:23.890Z] Stopping host...
[2021-08-11T14:21:23.890Z] Host shutdown completed.

Worked around this by dropping the extensionBundle property in api/host.json.

Bot Message is not read intermittently

Hi all, I have a re-entry message configured that gets called On Return of an intent mapper node. Intermittently this message isn’t being read by the bot. The log events table shows that the interaction was visited and there is no error there. I noticed the “AxiosError” on the left hand side which I suspect is related to this but I didn’t find any helpful details in the error.

In a given day, without making any changes to the bot and by following the exact same script, I can have a couple conversations where the message is read out as expected, followed by conversations where it isn't, finally followed by conversations where it is again read as expected.

I've first documented this issue as far back as two months ago. Most recently I've experienced this issue yesterday. Ex session id where the message was not read: 2558a53e-18b6-42da-84c8-10a227ed374b

Please let me know what other info you require. Here is the Mix Community thread https://community.mix.nuance.com/t/message-is-not-read-intermittently/6467

messsageNotRead

TTS issue within DLGaS (French LP)

Can't hear TTS within the DLG in demo client (TTSaaS only tab works fine). French language.
Project: 14901 (TTS modality is on in my project)

I do see the following error:

image

Any idea?

Thanks.
Caroline-

[feature request] Upload start-data as JSON

Currently, start-data can be added manually by entering key-value pairs.

This process is cumbersome, especially if we are working on different projects with different start-data. It would be extremely useful (much faster) to be able to upload the start data from a JSON file, and potentially to store them so that they can be switched easily.

DLGaaS with Voice Input (ASRaaS)

  • DLGaaS to use start, execute, and executeStream via web-gRPC
  • DLGaaS to offer Voice Input (live mic) through orchestration API

Allow Event Parameters to be passed in for TTSaaS

It would be a nice feature to enable the ability to pass Event Parameters, i.e. send_visemes, within the TTS request and then display the output in the response section once the TTS Audio is returned.

AutoStart Chat view without Auth screen

When loading up /chat/, and a sessionId= query param exists, the chat will auto start.
Do not show the auth screen in this scenario and jump right into the chat view.

Mix Test Tool Enhancements for Sales Demos

The Demo Team is requesting the following enhancements for customer demos.

Goal: Enable SE to demonstrate Mix IVR where the prospect is able to see the conversation transcript in real-time is a chat bubble format.

  1. Enhance the chat window pop-out to support IVR experience without having to change the URL setting.
  2. Add the ability to minimize the dial pad so the demonstrator can show more of the back-&-forth dialog. My ideal experience is that demonstrator starts with the dial pad to show that we support speech and touch tone. They can then minimize the key pad during the demo so the prospect can see more of the back-&-forth dialog. PS: I notice you can do this after the call ends.
  3. Make the IVR prompting easier to read by 1) place the IVR content within a chat bubble and 2) changing the font (gray is a little hard for remote participants to see).
  4. Ability to turn off key value details to maximize screen real estate.
    MixTestTool_IVR_UX_enhancement_#2
    MixTestTool_IVR_UX_enhancement_#3
    MixTestTool_IVR_UX_enhancement_#4

Dedicated Log Events tab

Add a new tab, to complement: Profile, DLGaaS, ASRaaS, NLUaaS, TTSaaS, and now Log Events.
Log events will focus on DLGaaS payloads; which have a session_id. Others are stateless.
See https://docs.nuance.com/mix/apis/event_logs/ for more.

  • This tab should offer the ability to filter by sessionID.
  • The table of events should be rendered as in DLGaaS.
  • It should be possible to seek through sessions.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.