catalystcode/cordova-plugin-cogsvcsspeech

Cordova plugin for Microsoft Cognitive Services speech services.

Ionic Application

The Ionic application was build with the Ionic framework, an open-source front-end development kit for building cross-platform mobile apps with HTML5 and Javascript.

Getting started

First, install Node.js. The cognitive services plugin supports Node version 10.9 or greater.

Next, install Ionic using the following command:

npm install -g ionic

Note: Visual Studio Code was used for development and ionic was installed through VS Code.

Dependencies

Create a Speech resource in Azure

To add a Speech Services resource to your Azure account:

1. Sign in to the Azure portal using your Microsoft account.

2. Select Create a resource at the top left of the portal.

   

3. In the New window, search for speech.

4. In the search results, select Speech.

   

5. Under Speech, select the Create button.

   

6. Under Create, enter:

   • A name for the new resource. The name helps you distinguish among multiple subscriptions to the same service.

   • Choose the Azure subscription that the new resource is associated with to determine how the fees are billed.

   • Choose the region where the resource will be used.

   • Choose either a free or paid pricing tier.

   • Create a new resource group for this Speech subscription or assign the subscription to an existing resource group. Resource groups help you keep your various Azure subscriptions organized.

     • Select Create.

     

     It takes a moment to create and deploy your new Speech resource. Select Quickstart to see information about your new resource.

     

7. Under Quickstart, select the Keys link under step 1 to display your subscription keys. Each subscription has two keys; you can use either key in your application. Select the button next to each key to copy it to the clipboard for pasting into the Environment.ts file.

8. Copy the name of the Speech Service and the region of the service into the Environments.ts file.

Install packages using NPM

In the folder of the Ionic Application (ie examples/ionic-angular), run the following steps.

Run the following command to install all packages within the app. In a terminal window, browse to the examples/ionic-angular folder and run the following:

npm install

Install Native-Run so that the application can run the cordova native platforms (android, ios, or browser)

npm i -g native-run

Install Cordova

npm i -g cordova

To include the wrapper for the Cognitive Services plugin as an observable, run the following command:

cp -r ../../cognitiveservices node_modules/@ionic-native/

While this should install all components, there have been a few changes to cordova recently and errors were being generated while devloping on the Mac.
If you get errors, specifically using toLowerCase, the run the following commands:

npm uninstall cordova
npm install cordova@latest
cordova platform remove ios
cordova platform add ios

Run the Application

Run on Browser

To run the application, in a terminal window, browse to the examples/ionic-angular folder and run the following:

ionic cordova run browser

Run in Android Emulator

Setting up Android environment

These steps are to set up your SDK environment to build, test, and deploy Android apps.

1. Install Java Development Kit (JDK) 8 or later.
2. Install Gradle. Go through each steps on the Gradle Installation page.
3. Install Android Studio.
4. After installing Android Studio, run the Android Studio Setup Wizard to download some standard components that are required for development.

Creating an emulator

1. In Android Studio, click the AVD Manager in the Toolbar.

   

2. In the Select Deployment Target window, click Create New Virtual Device.

   

3. Select a device configuration of your choice and press Next.

   

4. Download and select a system image. Press next.

   

5. Verify configuration and press finish.

   

6. In the Virtual Device Manager, select the run button to start the emulator.

   

Run Android Application

To run the application, in a terminal window, browse to the speecthtoText folder and run the following:

ionic cordova run android

Run in iOS Emulator

Setting up iOS environment

These steps are to set up your SDK environment to build, test, and deploy iOS apps. This will only work on a Mac.

1. Install XCode with Apple Developer account, or download from the App Store.

Run iOS Application

To run the application, in a terminal window, browse to the speecthtoText folder and run the following:

ionic cordova run ios

By participating in this project, you agree to abide by the Microsoft Open Source Code of Conduct

cordova-plugin-cogsvcsspeech's Issues

Add startSpeaking to the Plugin and Example App

Description

As an application developer, I'd like to use the startSpeaking method, in order to have the application speak the text given to the plugin.

As an user of the example application, I'd like to press the Start Talking button and have the application speak the text in the text box.

Acceptance Criteria

• In the Android plugin code, within the CognitiveServices.java file, rename the SpeakText method to startSpeaking.
• In the iOS plugin code, within the CDVCognitiveServices.h and CDVCognitiveServices.m files, rename the SpeakText method to startSpeaking.
• In the plugin interface code, within the cognitiveservices.js file, rename the SpeakText method to startSpeaking.
• In the Native code, all files under the cognitiveservices folder, rename the SpeakText method to startSpeaking.
• In the Example app, add the startSpeaking functionality.
• Update the Plugin README description and examples with the startSpeaking method.

Resources

The SDK uses the SpeakText method SpeechSynthesizer.SpeakText, so find and replace will not work.

Add demo Ionic app starting point

Add existing Ionic demo application to the repository. The demo Ionic application will provide an example implementation of the Cordova Plugin

Download latest Cognitive Services Android Library

Description

As a plugin developer, I'd like to implement the process of downloading the Cognitive Services AAR during installation, in order to use the latest Speech SDK.

As an application developer, I'd like to have the Cognitive Services AAR downloaded during installation without interaction, in order to use the latest Speech SDK.

Acceptance Criteria

• Should remove the AAR library from the Cognitive Services Plugin files in github and still have all functionality
• Should allow the Cognitive Services Plugin to be installed with the AAR without interaction.

Resources

Quickstart: Setup development environment
Speech recognition sample for Android

Add React Ionic example application

SSML Voice Customization

Description

As an application developer and a plugin user, I'd like to have the ability to send in SSML properties, in order to customize the voice in both Android and iOS.

Acceptance Criteria

• When calling the Cordova plugin for text to speech, provide an option for sending the variables used in creating the SSML xml string.
• The voice should match the variables given when speaking.

Resources

Speech Synthesis Markup Language (SSML)

Android Recognize from Microphone

Description

As a developer, I'd like to call the Cognitive Services SDK through the AAR, in order to be able to use the microphone to capture speech.

Acceptance Criteria

• When the speech recognizing function is called on the Android platform, the plugin should use a single callback object for the recognizing and recognized events.
• The callback schema need to include an 'isFinal' boolean value and a 'result' string or match the callback schema of the iOS code.
• Update relevant documentation

Resources

Java Speech Reference: com.microsoft.cognitiveservices.speech
Quickstart: Setup development environment
Example of Observable in Cordova

Add dual purpose (start/stop) to button elements

Add config.xml configuration for cordova-plugin-whitelist

In the Cognitive Services plugin code, we take a dependency on cordova-plugin-whitelist. Curious what the purpose of that dependency is.

Stop Listening

Description

As an application user, I'd like to have stop listening code on the iOS and Android, in order to have an event that stops the microphone from listening.

Acceptance Criteria

• Should have a function in the Cognitive Services plugin that implements the stop listening functionality for Android.
• Should have a function in the Cognitive Services plugin that implements the stop listening functionality for iOS.
• README documentation in the plugin should be updated to include changes to the stopListening function and example.

Convert Cordova Plugin "www" content to TypeScript

See https://github.com/ionic-team/cordova-plugin-ionic for examples of how to build a Cordova plugin with TypeScript

Cordova Plugin Apache License

Within the Cordova Plugin, there is a license file that was generated when the plugin was created with the Apache License. What is our policy for generated license files? Can we change it to MIT License?

Create abstraction that can be shared between iOS and Android approaches for STT and TTS

Is your feature request related to a problem? Please describe.

We have two different approaches for STT and TTS using Cognitive Services on iOS vs. Android. On iOS, we're using an Ionic Native plugin, which will convert a series of native method callbacks to an Observable. On Android, we're using the JS SDK, which just has two separate event callbacks, one for "Recognizing" events and the other for "Recognized" events.

The goal would be to create an abstraction that supports both.

Describe the solution you'd like
A speech service interface that can be implemented differently for iOS and Android with the following methods:

interface SpeechService {
  init(speechKey: string, speechRegion: string): void
  startListening(): Observable<{ isFinal: boolean, text: string }>
  stopListening(): Promise<void>
  speakSsml(): Promise<void>
  stopSpeaking(): Promise<void> 
}

Describe alternatives you've considered
We've considered two alternatives

1. Implement the native Cordova plugin for Android instead
2. Push platform specific logic into the application layer

For 1., we assume over time the Speech JS SDK will work as expected on iOS, so it's good to have parallel implementations.

For 2., it's good to write an abstraction once that others can reuse in their application layer.

Download iOS MicrosoftFramework during Installation

Description

As a plugin developer, I'd like to implement the process of downloading the Cognitive Services Microsoft Framework during installation, in order to use the latest Speech SDK.

As an application developer, I'd like to have the Cognitive Services AAR downloaded during installation without interaction, in order to use the latest Speech SDK.

Acceptance Criteria

• Should remove the MicrosoftFramework code from the Cognitive Services Plugin files in github and still have all functionality
• Should allow the Cognitive Services Plugin to be installed with the MicrosoftFramework without interaction.

Resources

Quickstart: Synthesize speech in Swift on iOS using the Speech
Install the SDK as a CocoaPod

Update Cognitive Services SDK to 1.9.0

Would like to update the Cognitive Services SDK to 1.9.0 or use the 1.8.0 or greater option when downloading from gradle for Android and Cocoapods for iOS.

Full testing would need to occur on the plugin.

Understanding AVAudioSession

Description

As an architect, I want to understand the AVAudioSession functionality, specifically Categories and Modes, so that I can correctly advise the the design of the cognitive services plugin.

Acceptance Criteria

• Write a hypothesis that details the approaches and outcomes of different AVAudioSession categories and modes
• Update README.md with final decision of how to implement AVAudioSession

Cannot install and run plugin on iOS

I'm having some issues installing the plugin through iOS.
When running pod install after installing the cordova plugin to my project, it says there's no dependencies. So I had to manually update the Podfile to include the following lines before end

pod 'MicrosoftCognitiveServicesSpeech-iOS', '1.8.0'
use_frameworks!

Running pod install then installs the Microsoft Framework for iOS successfully.
When I try to run the app in XCode, I get the following errors.

So I tried adding the following User-Defined under Build Settings to try and solve the build error.

After doing so, I get the following errors instead when trying to run.

Now I'm pretty stuck.
I do not encounter build issues when running cordova on android.

XCode Version: 11.2.1
Mac Version: macOS Mojave 10.14.6
Cordova Version: 7.1.0

Add CI actions for Ionic Angular example application

As a developer, I want a way to lint, build, and test commits, so I can vaiidate my changes to the example application's codebase.

Should run linting, build the project and run any tests when code changes are checked in. Should fail on any errors.

'cordova build ios' gives errors

Steps to reproduce the behavior:

1. clone 'CatalystCode/cordova-plugin-cogsvcsspeech'
2. create a Cordova project by 'cordova create ...'
3. add ios platform by 'cordova platform add [email protected]'
4. add this cognitiveServices plugin by 'cordova plugin add
5. then build by 'cordova build ios'

The build gives errors when archiving:
Undefined symbols for architecture armv7: "_OBJC_CLASS_$_SPXSpeechSynthesisCancellationDetails", referenced from: objc-class-ref in CDVCognitiveServices.o "_OBJC_CLASS_$_SPXSpeechSynthesizer", referenced from: objc-class-ref in CDVCognitiveServices.o "_OBJC_CLASS_$_SPXSpeechRecognizer", referenced from: objc-class-ref in CDVCognitiveServices.o "_OBJC_CLASS_$_SPXCancellationDetails", referenced from: objc-class-ref in CDVCognitiveServices.o "_OBJC_CLASS_$_SPXSpeechConfiguration", referenced from: objc-class-ref in CDVCognitiveServices.o ld: symbol(s) not found for architecture armv7 clang: error: linker command failed with exit code 1 (use -v to see invocation)

If it is built to emulator like 'cordova build ios --emulator', it builds successfully.
Any idea how to resolve this?

Stop Speaking

Description

As an application user, I'd like to have stop speaking code on the iOS and Android platform, in order to have an event that stops the speaker from speaking.

Acceptance Criteria

• Should have a function in the Cognitive Services plugin that implements the stop speaking functionality for Android.
• Should have a function in the Cognitive Services plugin that implements the stop speaking functionality for iOS.
• README documentation in the plugin should be updated to include changes to the stopSpeak function and example.

Request Permission

Description

As an architect, I want to determine if the plugin needs to request permissions or require that the calling application handle the permission request.

Acceptance Criteria

• Review other Cordova plugins to determine if the plugin handles required permissions. Be sure to include plugins that use the microphone and speakers of the phones.
• Create a chart that details and how many plugins handle the permissions vs leaving the functionality in the calling application
• Update README to detail the findings and determine the functionality for the plugin.

Add plugin codebase to repository

Move the exisiting plugin codebase from the private repository over to this repository.

Async Functionality in Cordova

Description

As an architect, I want to understand how Cordova implements Async functionality, so that I can determine if plugin should implement the Speech SDK async functions.

Acceptance Criteria

• Find document that details the how Cordova works with async functionality so that we can review to make a decision.
• Find document that details how the Cognitive Services Speech SDK implements async servcies so that we can review to make a decision on whether or not to use the functions in the plugin.
• Update README.md with final decision of if and how to implement async functions from the Speech SDK