nfreear/dictation

An adaptive dictation-mode speech recognizer ponyfill compatible with WebChat that gives the user time to think and stutter!

@nfreear/speech-dictation

An adaptive dictation-mode speech recognizer ponyfill compatible with WebChat that gives the user time to think and stutter (stammer)!

Mastering 'endSilenceTimeoutMs' in Microsoft Speech SDK dictation mode!

(08-Oct-2020)

Ponyfill

See Integrating with Cognitive Services Speech Services.

import { createDictationRecognizerPonyfill } from './createDictationRecognizerPonyfill.js';

const asrPonyfill = await createDictationRecognizerPonyfill({ region, key });

// ... Combine speech synthesis from default
// 'createCognitiveServicesSpeechServicesPonyfillFactory()' ...

renderWebChat(
  {
    directLine: createDirectLine({ ... }),
    // ...
    webSpeechPonyfillFactory: await createCustomHybridPonyfill({ ... })
  },
  document.getElementById('webchat')
);

Dictation mode

The key lines in createDictationRecognizerPonyfill to force dictation mode, and enable the setting of initialSilenceTimeoutMs and endSilenceTimeoutMs:

const initialSilenceTimeoutMs = 5 * 1000;
const endSilenceTimeoutMs = 5 * 1000;
// Scroll to right! → →
const url = `wss://${region}.stt.speech.microsoft.com/speech/recognition/dictation/cognitiveservices/v1?initialSilenceTimeoutMs=${initialSilenceTimeoutMs || ''}&endSilenceTimeoutMs=${endSilenceTimeoutMs}&`;
const urlObj = new URL(url);

const speechConfig = SpeechConfig.fromEndpoint(urlObj, subscriptionKey);

speechConfig.enableDictation();

// ...

const recognizer = new SpeechRecognizer(speechConfig, audioConfig);

Usage

npm install
npm start
npm test

Useful links

• Gist: speech-sdk.dictate.js.html;
• Bug: 'Can i set silence duration at end of speech ?' (#131);
• Bug: '... I need to lengthen the dictation / "Listening" timeout' (#3466)
• GitHub: Microsoft/cognitive-services-speech-sdk-js;
• GitHub: @compulim/web-speech-cognitive-services ../create...Ponyfill.js;
• GitHub: @compulim/react-dictate-button ../Composer.js;

Credit

Developed in IET at The Open University for the ADMINS project, funded by Microsoft.

---

dictation's Issues

Open-source the project / repo.

Hi,

Here are the steps which I think need to be taken to open-source this project / repo :~

• – Agree open-sourcing in principle (agreed in principle at meeting on 24th August 2021, by Tony S. in RES, Tim C. and Kevin M.);
• – Agree license – permissive MIT ? (to match e.g. https://github.com/Microsoft/BotFramework-WebChat);
• – Agree repo. location – https://github.com/IET-OU/ ;
• – Agree package name. Suggestions: "patient-speech-recognizer" OR "adaptive-speech-recognizer" (US spelling?) (E.g. https://github.com/IET-OU/patient-speech-recognizer and https://npmjs.com/package/patient-speech-recognizer )
• – Replace "__TODO_NAME__" with package name (package.json, README.md, index.html)
• – Replace "__TODO_LICENSE__" with license ID ("MIT" ?) (package.json, README.md)
• – Add "LICENSE.txt" file to repo. with text from, e.g. mit-license.org | choosealicense.com/.. – Add: "Copyright © 2021 The Open University (IET)." to the LICENSE;
• – Review the "version" field in package.json – I suggest "2.0.1";
• – Review "description", "keywords", "author", "contributors", "repository" and other fields in package.json (See: NPM docs: package.json);
• – When happy, commit then run git tag 2.0.1;
• – Experiment with npm publish --dry-run (See: NPM docs: npm publish);
• – Finally, run npm publish !

Best of luck!

---

Compare with:

• GitHub: compulim/web-speech-cognitive-services;
• Npmjs: web-speech-cognitive-services;

Set a static or dynamic dictionary of "stop phrases"

...

• Option A :~ set the phrase dictionary once on initialization?

• Option B :~ dynamically add stop-phrases via an Event?

---

note: OK to use `AudioConfig.fromDefaultMicrophoneInput()`

..

• GitHub: compulim/web-speech-cognitive-services ~ ../createSpeechRecognitionPonyfill.js;
• MS Docs: ...-speech-sdk/audioconfig;

const {
    audioConfig = AudioConfig.fromDefaultMicrophoneInput(),
    // ...
}

---

fix: Recognizer only returns results once; not on second start ...

...?

• Bug is present in Speech SDK version 1.14.0;
• NOT present in SDK version 1.13.1;
• Apparently will be fixed in next SDK release;
• SDK releases :~ https://github.com/microsoft/cognitive-services-speech-sdk-js/releases;
• Related bugs :~ microsoft/cognitive-services-speech-sdk-js#299; Azure-Samples/cognitive-services-speech-sdk#881;

---

Align `DictationRecognizer` class with Web API `SpeechRecognition` class

.. To hopefully get createPonyfill() working in MS BotFramework / ADMINS Web-Chat ... ;)

• https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition;
• Gist: https://gist.github.com/nfreear/3b9de19a76df43b3f34b2380f5309f4b;

..

fix: Use 'displayText', use confidences, fix double full-stops ...

...

• Use the displayText property in recognized callback events;
• Use result.json.NBest[0].Confidence;
• Fix the double full-stop bug, in toSentence()!

...

feat: Personalize / adapt / change the `endSilenceTimeoutMs` mid-session

Dispatch and handle a custom setTimeout event ... recreate the internal speech recognizer in mid-session ...

Dispatch setTimeout event

    const timeoutEvent = new Event('recognition:set_timeout');
    timeoutEvent.data = { endSilenceTimeoutMs, text };

    window.dispatchEvent(timeoutEvent);

---

Recreate speech recognizer

    PRIV.resetRecognizer = async () => {
          if (PRIV.setTimeoutEvent) {
            // ...

            const { recognizer, OPT } = await createRecognizer(resetOpt);

            // ...
          }
    };

---

See :~ compulim/web-speech-cognitive-services@038dccb ~ recognizer.dispose() ~ 05-Dec-2018;

..

Call `setProfanity()` ... call `enableTelemetry(false)`

..

• speechConfig.setProfanity("raw");
• ProfanityOption enum;
• sdk.Recognizer.enableTelemetry(false);
• Docs :~ https://docs.microsoft.com/en-us/javascript/api/microsoft-cognitiveservices-speech-sdk/speechconfig?view=azure-node-latest#

..

fix: Start adding unit tests ...

... Using jest !

Re-factor, influenced by 'compulim/web-speech-cognitive-services' ~ 'createSpeechRecognitionPonyfill'

..

• GitHub: compulim/web-speech-cognitive-services ../createSpeechRecognitionPonyfill.js;
• https://github.com/compulim/react-dictate-button/blob/master/packages/component/src/Composer.js#L134

---

import { SpeechRecognizer } from 'SpeechSDK';

function createSpeechRecognitionFromRecognizer (recognizer: SpeechRecognizer): SpeechRecognition {

  class SpeechRecognition extends EventTarget { // Web API definition!
    // TODO ...!
  }

  return SpeechRecognition;
}

function createCognitiveRecognizer (options: DictationOptions): SpeechRecognizer {
  // ...

  const recognizer = new SpeechRecognizer(); // MS Azure SpeechSDK

  return recognizer;
}

export function createDictationRecognizerPonyfill (options: DictationOptions): Ponyfill {
  const recognizer = createCognitiveRecognizer(options);

  const SpeechRecognition = createSpeechRecognitionFromRecognizer(recognizer);

  return {
    SpeechGrammarList,
    SpeechRecognition
  }
}

---

fix: Remove full-stop "." from end of text results in Dictation Recognizer

... Especially for "suggested action" buttons ...

... consistency with rest of Chat-bot ..!!

Move `example-app` Javascript from `src` directory, etc.

..

Update ms-cog-speech-sdk version; devDependencies versions; Node version

..

• — Update ms-cog-speech-sdk version from 1.15.1 to 1.33.1,
• — Update outdated devDependencies versions (jest, semistandard, servor),
• — Node version in CI workflow from 12.* to 18.*,
• — Add deployment to CI workflow,

---

25 vulnerabilities (12 moderate, 9 high, 4 critical)

..

fix: Use the 'CancellationErrorCode' enum

..

• https://docs.microsoft.com/en-us/javascript/api/microsoft-cognitiveservices-speech-sdk/cancellationerrorcode?view=azure-node-latest;

For example:

TooManyRequests = 3 |
Indicates that the number of parallel requests exceeded the number of allowed concurrent transcriptions for the subscription.

---

fix: Upgrade from speech SDK v1.13.1 to v1.15.1

... Relates to :~ #8

..

fix: Support for Safari (desktop ... iOS?)

..

1. "TypeError: function is not a constructor (evaluating 'super()')"

   • SpeechRecognition - createDict...js:122

2. "Error: Browser does not support Web Audio API (AudioContext is not available)."

   • https://github.com/microsoft/BotFramework-WebChat/blob/master/packages/bundle/src/createCognitiveServicesSpeechServicesPonyfillFactory.js#L57-L99;

..