nexmo-community/voice-azure-speechtotext-py

Sample Code for Realtime Transcription using Nexmo, Microsoft Azure Speech Services & websockets

Speech To Text with Nexmo Voice API & Microsoft Azure

You can use this code as a base for doing real time transcription of a phone call using Azure Speech Services.

An audio stream is sent over a websocket to your server and you then relay that on to the Azure websocket interface where recognition is performed and the phrases returned to the console.

Azure Speech Services

You'll need to signup for Azure Speech Services and make a note of two pieces of information - the first service API key, and the regional location of the Speech API service you deployed (eg. westeurope).

Running the app

Using Docker

To run the app using Docker run the following command in your terminal:

docker-compose up

This will create a new image with all the dependencies and run it at http://localhost:8000.

You can declare the required environment variables by editing the docker-compose.yml file.

Local Install

To run this on your machine you'll need an up-to-date version of Python 3.

Start by installing the dependencies with:

pip install --upgrade -r requirements.txt

Then copy the .env.example file to a new file called .env:

cp .env.example > .env

Edit the .env file to add in your own service credentials from Azure and other settings specific to your instance of the Azure Speech Service API.

HOSTNAME = "yourhostname.ngrok.io"
LANGUAGE = "en-GB"
KEY1 = "3234gh3gh34ghj32hj"
REGIONAL_API_ENDPOINT = "westeurope" # eg. "westeurope", "southeastasia", "uswest"

By default the server runs on port 8000.

Tools like ngrok are great for exposing ports on your local machine to the internet. If you haven't done this before, check out this guide.

If you aren't going to be working in the en-GB language then you can change the language to any of the other supported languages listed in the Speech Service API documentation.

The Azure Speech Service API can run across multiple regions. When you initially set it up you will specify which region your service will run in, you will then need to change the REGIONAL_API_ENDPOINT environment variable to match.

Running the example

If you are working with a local install you can run the server using this command:

python ./server.py

Linking the app to Nexmo

You will need to create a new Nexmo application in order to work with this app:

Create a Nexmo Application Using the Command Line Interface

Install the CLI by following these instructions. Then create a new Nexmo application that also sets up your answer_url and event_url for the app running locally on your machine.

nexmo app:create ms-speech-to-text http://<your_hostname>/ncco http://<your_hostname>/event

This will return an application ID. Make a note of it.

Rent a New Virtual Number

If you don't have a number already in place, you will need to rent one. This can also be achieved using the CLI by running this command:

nexmo number:buy

Link the Virtual Number to the Application

Finally, link your new number to the application you created by running:

nexmo link:app YOUR_NUMBER YOUR_APPLICATION_ID

Try it out

With your app running, call the number you assigned to it and start speaking. After a brief pause you will see whatever you say written out to the console, in real time.

Extending

This example code simply prints the reponses from Azure to the console, however to integrate it with your own application you should extend the on_return_message function in server.py

voice-azure-speechtotext-py's Issues

When we publish to azure, site not working. Error : You do not have permission to view this directory or page.

Clicked on the Deployed to Azure button and followed all steps and deployment successfully completed.

but When I navigate to the site, getting You do not have permission to view this directory or page. error as below

Websocket and azure STT sudden issue

HI,

I was following this sample and started my implementation based on that and make a live app in last month. It was working fine until yesterday afternoon but now its not working. So i downloaded this again and change my configuration related to azure and i see the same issue here. It answer calls, then print "Websocket connected" but then i speak anything it doesn't go beyond that place. I see all headers of Azure STT are receiving in on_return_message:

header value: turn.start
header value: speech.startDetected
header value: speech.hypothesis
header value: speech.hypothesis
header value: speech.endDetected
header value: speech.phrase
header value: turn.end

but it doesn't hit this statement: https://github.com/nexmo-community/voice-microsoft-speechtotext/blob/master/server.py#L128

Again, it was working fine until yesterday afternoon around 3 PM (Indian Standard time), but all of sudden its just not working. even this sample is also not working.

So any thoughts on this?

requests-2.21.0-py2.py3-none-any.whl: 7 vulnerabilities (highest severity is: 9.8)

Vulnerable Library - requests-2.21.0-py2.py3-none-any.whl

Python HTTP for Humans.

Library home page: https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Found in HEAD commit: b88d98414c9a21e5fcd0323060b1a29e132685e0

Vulnerabilities

CVE	Severity	CVSS	Dependency	Type	Fixed in (requests version)	Remediation Possible**
CVE-2023-37920	Critical	9.8	certifi-2022.9.24-py3-none-any.whl	Transitive	2.22.0	✅
CVE-2023-43804	High	8.1	urllib3-1.24.3-py2.py3-none-any.whl	Transitive	2.25.0	✅
CVE-2022-23491	High	7.5	certifi-2022.9.24-py3-none-any.whl	Transitive	2.22.0	✅
CVE-2021-33503	High	7.5	urllib3-1.24.3-py2.py3-none-any.whl	Transitive	2.25.0	✅
CVE-2020-26137	Medium	6.5	urllib3-1.24.3-py2.py3-none-any.whl	Transitive	2.22.0	✅
CVE-2023-32681	Medium	6.1	requests-2.21.0-py2.py3-none-any.whl	Direct	requests -2.31.0	✅
CVE-2023-45803	Medium	4.2	urllib3-1.24.3-py2.py3-none-any.whl	Transitive	2.25.0	✅

**In some cases, Remediation PR cannot be created automatically for a vulnerability despite the availability of remediation

Details

CVE-2023-37920

Vulnerable Library - certifi-2022.9.24-py3-none-any.whl

Python package for providing Mozilla's CA Bundle.

Library home page: https://files.pythonhosted.org/packages/1d/38/fa96a426e0c0e68aabc68e896584b83ad1eec779265a028e156ce509630e/certifi-2022.9.24-py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• requests-2.21.0-py2.py3-none-any.whl (Root Library)
  • ❌ certifi-2022.9.24-py3-none-any.whl (Vulnerable Library)

Found in HEAD commit: b88d98414c9a21e5fcd0323060b1a29e132685e0

Found in base branch: main

Vulnerability Details

Certifi is a curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts. Certifi prior to version 2023.07.22 recognizes "e-Tugra" root certificates. e-Tugra's root certificates were subject to an investigation prompted by reporting of security issues in their systems. Certifi 2023.07.22 removes root certificates from "e-Tugra" from the root store.

Publish Date: 2023-07-25

URL: CVE-2023-37920

CVSS 3 Score Details (9.8)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: High
  • Integrity Impact: High
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-xqr8-7jwr-rhp7

Release Date: 2023-07-25

Fix Resolution (certifi): 2023.7.22

Direct dependency fix Resolution (requests): 2.22.0

⛑️ Automatic Remediation will be attempted for this issue.

CVE-2023-43804

Vulnerable Library - urllib3-1.24.3-py2.py3-none-any.whl

HTTP library with thread-safe connection pooling, file post, and more.

Library home page: https://files.pythonhosted.org/packages/01/11/525b02e4acc0c747de8b6ccdab376331597c569c42ea66ab0a1dbd36eca2/urllib3-1.24.3-py2.py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• requests-2.21.0-py2.py3-none-any.whl (Root Library)
  • ❌ urllib3-1.24.3-py2.py3-none-any.whl (Vulnerable Library)

Found in HEAD commit: b88d98414c9a21e5fcd0323060b1a29e132685e0

Found in base branch: main

Vulnerability Details

urllib3 is a user-friendly HTTP client library for Python. urllib3 doesn't treat the Cookie HTTP header special or provide any helpers for managing cookies over HTTP, that is the responsibility of the user. However, it is possible for a user to specify a Cookie header and unknowingly leak information via HTTP redirects to a different origin if that user doesn't disable redirects explicitly. This issue has been patched in urllib3 version 1.26.17 or 2.0.5.

Publish Date: 2023-10-04

URL: CVE-2023-43804

CVSS 3 Score Details (8.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: Low
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: High
  • Integrity Impact: High
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://www.cve.org/CVERecord?id=CVE-2023-43804

Release Date: 2023-10-04

Fix Resolution (urllib3): 1.26.17

Direct dependency fix Resolution (requests): 2.25.0

⛑️ Automatic Remediation will be attempted for this issue.

CVE-2022-23491

Vulnerable Library - certifi-2022.9.24-py3-none-any.whl

Python package for providing Mozilla's CA Bundle.

Library home page: https://files.pythonhosted.org/packages/1d/38/fa96a426e0c0e68aabc68e896584b83ad1eec779265a028e156ce509630e/certifi-2022.9.24-py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• requests-2.21.0-py2.py3-none-any.whl (Root Library)
  • ❌ certifi-2022.9.24-py3-none-any.whl (Vulnerable Library)

Found in HEAD commit: b88d98414c9a21e5fcd0323060b1a29e132685e0

Found in base branch: main

Vulnerability Details

Certifi is a curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts. Certifi 2022.12.07 removes root certificates from "TrustCor" from the root store. These are in the process of being removed from Mozilla's trust store. TrustCor's root certificates are being removed pursuant to an investigation prompted by media reporting that TrustCor's ownership also operated a business that produced spyware. Conclusions of Mozilla's investigation can be found in the linked google group discussion.

Publish Date: 2022-12-07

URL: CVE-2022-23491

CVSS 3 Score Details (7.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: High
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://www.cve.org/CVERecord?id=CVE-2022-23491

Release Date: 2022-12-07

Fix Resolution (certifi): 2022.12.7

Direct dependency fix Resolution (requests): 2.22.0

⛑️ Automatic Remediation will be attempted for this issue.

CVE-2021-33503

Vulnerable Library - urllib3-1.24.3-py2.py3-none-any.whl

HTTP library with thread-safe connection pooling, file post, and more.

Library home page: https://files.pythonhosted.org/packages/01/11/525b02e4acc0c747de8b6ccdab376331597c569c42ea66ab0a1dbd36eca2/urllib3-1.24.3-py2.py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• requests-2.21.0-py2.py3-none-any.whl (Root Library)
  • ❌ urllib3-1.24.3-py2.py3-none-any.whl (Vulnerable Library)

Found in HEAD commit: b88d98414c9a21e5fcd0323060b1a29e132685e0

Found in base branch: main

Vulnerability Details

An issue was discovered in urllib3 before 1.26.5. When provided with a URL containing many @ characters in the authority component, the authority regular expression exhibits catastrophic backtracking, causing a denial of service if a URL were passed as a parameter or redirected to via an HTTP redirect.

Publish Date: 2021-06-29

URL: CVE-2021-33503

CVSS 3 Score Details (7.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-q2q7-5pp4-w6pg

Release Date: 2021-06-29

Fix Resolution (urllib3): 1.26.5

Direct dependency fix Resolution (requests): 2.25.0

⛑️ Automatic Remediation will be attempted for this issue.

CVE-2020-26137

Vulnerable Library - urllib3-1.24.3-py2.py3-none-any.whl

HTTP library with thread-safe connection pooling, file post, and more.

Library home page: https://files.pythonhosted.org/packages/01/11/525b02e4acc0c747de8b6ccdab376331597c569c42ea66ab0a1dbd36eca2/urllib3-1.24.3-py2.py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• requests-2.21.0-py2.py3-none-any.whl (Root Library)
  • ❌ urllib3-1.24.3-py2.py3-none-any.whl (Vulnerable Library)

Found in HEAD commit: b88d98414c9a21e5fcd0323060b1a29e132685e0

Found in base branch: main

Vulnerability Details

urllib3 before 1.25.9 allows CRLF injection if the attacker controls the HTTP request method, as demonstrated by inserting CR and LF control characters in the first argument of putrequest(). NOTE: this is similar to CVE-2020-26116.

Publish Date: 2020-09-30

URL: CVE-2020-26137

CVSS 3 Score Details (6.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-26137

Release Date: 2020-09-30

Fix Resolution (urllib3): 1.25.9

Direct dependency fix Resolution (requests): 2.22.0

⛑️ Automatic Remediation will be attempted for this issue.

CVE-2023-32681

Vulnerable Library - requests-2.21.0-py2.py3-none-any.whl

Python HTTP for Humans.

Library home page: https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• ❌ requests-2.21.0-py2.py3-none-any.whl (Vulnerable Library)

Found in HEAD commit: b88d98414c9a21e5fcd0323060b1a29e132685e0

Found in base branch: main

Vulnerability Details

Requests is a HTTP library. Since Requests 2.3.0, Requests has been leaking Proxy-Authorization headers to destination servers when redirected to an HTTPS endpoint. This is a product of how we use rebuild_proxies to reattach the Proxy-Authorization header to requests. For HTTP connections sent through the tunnel, the proxy will identify the header in the request itself and remove it prior to forwarding to the destination server. However when sent over HTTPS, the Proxy-Authorization header must be sent in the CONNECT request as the proxy has no visibility into the tunneled request. This results in Requests forwarding proxy credentials to the destination server unintentionally, allowing a malicious actor to potentially exfiltrate sensitive information. This issue has been patched in version 2.31.0.

Publish Date: 2023-05-26

URL: CVE-2023-32681

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: High
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: High
  • Integrity Impact: None
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-j8r2-6x86-q33q

Release Date: 2023-05-26

Fix Resolution: requests -2.31.0

⛑️ Automatic Remediation will be attempted for this issue.

CVE-2023-45803

Vulnerable Library - urllib3-1.24.3-py2.py3-none-any.whl

HTTP library with thread-safe connection pooling, file post, and more.

Library home page: https://files.pythonhosted.org/packages/01/11/525b02e4acc0c747de8b6ccdab376331597c569c42ea66ab0a1dbd36eca2/urllib3-1.24.3-py2.py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• requests-2.21.0-py2.py3-none-any.whl (Root Library)
  • ❌ urllib3-1.24.3-py2.py3-none-any.whl (Vulnerable Library)

Found in HEAD commit: b88d98414c9a21e5fcd0323060b1a29e132685e0

Found in base branch: main

Vulnerability Details

urllib3 is a user-friendly HTTP client library for Python. urllib3 previously wouldn't remove the HTTP request body when an HTTP redirect response using status 301, 302, or 303 after the request had its method changed from one that could accept a request body (like POST) to GET as is required by HTTP RFCs. Although this behavior is not specified in the section for redirects, it can be inferred by piecing together information from different sections and we have observed the behavior in other major HTTP client implementations like curl and web browsers. Because the vulnerability requires a previously trusted service to become compromised in order to have an impact on confidentiality we believe the exploitability of this vulnerability is low. Additionally, many users aren't putting sensitive data in HTTP request bodies, if this is the case then this vulnerability isn't exploitable. Both of the following conditions must be true to be affected by this vulnerability: 1. Using urllib3 and submitting sensitive information in the HTTP request body (such as form data or JSON) and 2. The origin service is compromised and starts redirecting using 301, 302, or 303 to a malicious peer or the redirected-to service becomes compromised. This issue has been addressed in versions 1.26.18 and 2.0.7 and users are advised to update to resolve this issue. Users unable to update should disable redirects for services that aren't expecting to respond with redirects with redirects=False and disable automatic redirects with redirects=False and handle 301, 302, and 303 redirects manually by stripping the HTTP request body.

Publish Date: 2023-10-17

URL: CVE-2023-45803

CVSS 3 Score Details (4.2)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Adjacent
  • Attack Complexity: High
  • Privileges Required: High
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: High
  • Integrity Impact: None
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-g4mx-q9vg-27p4

Release Date: 2023-10-17

Fix Resolution (urllib3): 1.26.18

Direct dependency fix Resolution (requests): 2.25.0

⛑️ Automatic Remediation will be attempted for this issue.

---

⛑️Automatic Remediation will be attempted for this issue.

update tutorial

Please make this tutorial up-to-date.

tornado-4.4.2.tar.gz: 2 vulnerabilities (highest severity is: 6.1)

Vulnerable Library - tornado-4.4.2.tar.gz

Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.

Library home page: https://files.pythonhosted.org/packages/1e/7c/ea047f7bbd1ff22a7f69fe55e7561040e3e54d6f31da6267ef9748321f98/tornado-4.4.2.tar.gz

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Vulnerabilities

CVE	Severity	CVSS	Dependency	Type	Fixed in (tornado version)	Remediation Possible**
CVE-2023-28370	Medium	6.1	tornado-4.4.2.tar.gz	Direct	6.3.2	✅
WS-2023-0296	Medium	5.6	tornado-4.4.2.tar.gz	Direct	6.3.3	✅

**In some cases, Remediation PR cannot be created automatically for a vulnerability despite the availability of remediation

Details

CVE-2023-28370

Vulnerable Library - tornado-4.4.2.tar.gz

Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.

Library home page: https://files.pythonhosted.org/packages/1e/7c/ea047f7bbd1ff22a7f69fe55e7561040e3e54d6f31da6267ef9748321f98/tornado-4.4.2.tar.gz

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• ❌ tornado-4.4.2.tar.gz (Vulnerable Library)

Found in base branch: main

Vulnerability Details

Open redirect vulnerability in Tornado versions 6.3.1 and earlier allows a remote unauthenticated attacker to redirect a user to an arbitrary web site and conduct a phishing attack by having user access a specially crafted URL.

Publish Date: 2023-05-25

URL: CVE-2023-28370

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Release Date: 2023-05-25

Fix Resolution: 6.3.2

⛑️ Automatic Remediation will be attempted for this issue.

WS-2023-0296

Vulnerable Library - tornado-4.4.2.tar.gz

Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.

Library home page: https://files.pythonhosted.org/packages/1e/7c/ea047f7bbd1ff22a7f69fe55e7561040e3e54d6f31da6267ef9748321f98/tornado-4.4.2.tar.gz

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• ❌ tornado-4.4.2.tar.gz (Vulnerable Library)

Found in base branch: main

Vulnerability Details

Tornado vulnerable to HTTP request smuggling via improper parsing of Content-Length fields and chunk lengths

Publish Date: 2023-08-15

URL: WS-2023-0296

CVSS 3 Score Details (5.6)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: High
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: Low

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-qppv-j76h-2rpx

Release Date: 2023-08-15

Fix Resolution: 6.3.3

⛑️ Automatic Remediation will be attempted for this issue.

---

⛑️Automatic Remediation will be attempted for this issue.

not able to print "display text" from Azure

I tried to run the server.py, and I couldn't see any text response shown in my screen. After have a look on the code, I found that this IF statement is the cause:

if headers["Path"] == "speech.phrase"

Tried to see the content of 'headers', found out the "Path": "speech.phrase\r", thus I change the IF statement in order to be able to print the data["DisplayText"] becomes:

if headers["Path"] == "speech.phrase\r":