twilio-labs / call-gpt Goto Github PK
View Code? Open in Web Editor NEWGenerative AI phone call toolkit using Twilio Media Streams.
License: MIT License
Generative AI phone call toolkit using Twilio Media Streams.
License: MIT License
There is no firebase configuration file (it's listed in the .gitignore) and no instructions on how to create said file. The app throws an error (below) when you attempt to start w/out that configuration file.
node:internal/modules/cjs/loader:1147
throw err;
^
Error: Cannot find module '../firebase-db.json'
After taking the latest , getting this error a lot
To help you decide, let me ask: Do you need noise cancellation in your headphones?
GPT -> user context length: 7
RangeError: Maximum call stack size exceeded
at TextToSpeechService.generate (/Users/gauravkesharwani/Repo/call-gpt/services/tts-service.js:36:44)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Right now these are my logs from a recent call
Server running on port 3000
Twilio -> Starting Media Stream for MZ1cbaeb64da6297ccb08a8cf316d2fe6c
Interaction 1: TTS -> TWILIO: Hello! I understand you're looking for a pair of AirPods, is that correct?
Twilio -> Audio completed mark (6): a85266e3-9158-4d27-8b46-6d72faa1416e
UtteranceEnd received before speechFinal, emit the text collected so far: Hi there. Can you hear me?
Interaction 0 – STT -> GPT: Hi there. Can you hear me?
Interaction 0: GPT -> TTS: Yes, I can hear you loud and clear •
Interaction 0: TTS -> TWILIO: Yes, I can hear you loud and clear •
Twilio -> Audio completed mark (272): dc2fe939-fbc0-4f0f-93c0-595cd7613ed9
Interaction 0: GPT -> TTS: How may I assist you today with your AirPods purchase?
GPT -> user context length: 5
Interaction 0: TTS -> TWILIO: How may I assist you today with your AirPods purchase?
Twilio -> Audio completed mark (273): 3149e0b8-52d4-420a-a6b2-e9c7d1c80fcc
STT -> Deepgram connection closed
Twilio -> Media stream MZ1cbaeb64da6297ccb08a8cf316d2fe6c ended.
[nodemon] restarting due to changes...
[nodemon] starting `ts-node src/app.ts`
Server running on port 3000
Twilio -> Starting Media Stream for MZ60a84bb1246cfd7160de45c4fbce614d
Interaction 1: TTS -> TWILIO: Hello! I understand you're looking for a pair of AirPods, is that correct?
Twilio -> Audio completed mark (10): 5f5b7ca0-e6fa-4d0d-bb6c-3507eaea5928
STT -> Deepgram connection closed
Twilio -> Media stream MZ60a84bb1246cfd7160de45c4fbce614d ended.
I get no audio on the call, like no audio whatsoever but I do see some transcripts in the console, I honestly can't seem to understand what the issue might be as there are no errors, it seems like deepgram closes conection out of the blue for one thing and also TTS never sends the audio to the actual call. Any ideas?
How would one create a function to forward the call to another number? Let's say gpt hears some emergency keywords (like "I'm hurt" or something like that). How would i forward the using twilio webhook or restapi via the twilio Studio or another way.
Is there any way that we can give a knowledgebase from pdf or any other format
I am getting only media event in websocket.
Can anyone help me what is the issue.
I am getting start and connected event on my local development server but not in deployed server. Because of this voice is not going through twilio
https://www.twilio.com/docs/voice/media-streams/websocket-messages#start-message
anyone can quickly tell me how i can run this code that i can conversation to gpt on my phone number
i also follow the steps of readme.MD i have run the server of using (npm run dev
) this command then i run this command (twilio phone-numbers:update +1[your-twilio-number] --voice-url=https://your-server.ngrok.io/incoming
) and then i make call of my number to twilio number but after one 1-2 second call has drop and chatgpt
donot connect with me ?? any one that can tell how i can run the code and how i talk to gpt
Currently, responses get increasingly slower as the chat context grows. We should implement a gradual rolloff of older queries that get summarized by GPT and then added to the end of the conversation.
This may require some testing to understand the optimal conversation context size based on quality of results and speed to response.
setting up Call-gpt went great. but when i call my twilio number i keeps ringing ,no answer then hangs up. On ngrok im getting 200 OK. i did run npm test everything was great. Am i missing something or what could be the problem?
node --no-deprecation app.js
Server running on port 3000
Twilio -> Starting Media Stream for MZeb5d94dbc813cf20f227ed2a7163312c
Interaction 1: TTS -> TWILIO: Hello! I understand you're looking for a pair of AirPods, is that correct?
Twilio -> Interruption, Clearing stream
Interaction 0 – STT -> GPT: Hello?
STT -> Speech was already final when UtteranceEnd recevied
Twilio -> Audio completed mark (195): 3898e4b5-4a15-4b41-871f-fd3c47b8ee77
Interaction 0: GPT -> TTS: Hi there! Thanks for calling in today. •
Interaction 0: GPT -> TTS: How can I assist you with your AirPods selection? •
Interaction 0: TTS -> TWILIO: Hi there! Thanks for calling in today. •
Interaction 0: GPT -> TTS: Do you prefer headphones that go in your ear •
Interaction 0: GPT -> TTS: or over the ear?
GPT -> user context length: 5
Interaction 0: TTS -> TWILIO: How can I assist you with your AirPods selection? •
Interaction 0: TTS -> TWILIO: Do you prefer headphones that go in your ear •
Interaction 0: TTS -> TWILIO: or over the ear?
Twilio -> Audio completed mark (467): 0ece6179-11ff-4a59-a4e9-eda039dabe4b
Twilio -> Audio completed mark (596): accf6e83-7e3c-4af4-b8d4-27bf3eb5ffa2
Twilio -> Audio completed mark (697): 9c2d998d-a0b5-43d6-860a-b4fd98ea9bf7
Twilio -> Audio completed mark (754): bb3228de-18f9-4019-b31d-f4119f48bc44
Interaction 1 – STT -> GPT: Can you tell me
Interaction 1: GPT -> TTS: Absolutely! •
Interaction 1: TTS -> TWILIO: Absolutely! •
Interaction 1: GPT -> TTS: Would you like to know more about the in-ear AirPods and AirPods Pro, •
Twilio -> Interruption, Clearing stream
Twilio -> Audio completed mark (1109): bd21b254-3f01-49b1-8119-08d3dcddffa0
Interaction 1: GPT -> TTS: or the over-the-ear AirPods Max?
GPT -> user context length: 7
Interaction 1: TTS -> TWILIO: Would you like to know more about the in-ear AirPods and AirPods Pro, •
Twilio -> Interruption, Clearing stream
Interaction 1: TTS -> TWILIO: or the over-the-ear AirPods Max?
Twilio -> Audio completed mark (1171): 44c7d4e8-1820-42e1-8456-8087fe4b8b92
UtteranceEnd received before speechFinal, emit the text collected so far: one headphone price?
Interaction 2 – STT -> GPT: one headphone price?
Twilio -> Audio completed mark (1280): ea654842-6155-4e89-a724-dca99bdbd21b
Warning: Double function arguments returned by OpenAI: {"model": "airpods"}{"model": "airpods pro"}{"model": "airpods max"}
Interaction 2: GPT -> TTS: Let me check the price, one moment.
GPT -> called checkPrice function
**C:\Python Important\call-gpt-main\node_modules\openai\error.js:43
return new BadRequestError(status, error, message, headers);
^
BadRequestError: 400 'checkPrice' is not one of ['system', 'assistant', 'user', 'function'] - 'messages.8.role'**
at APIError.generate (C:\Python Important\call-gpt-main\node_modules\openai\error.js:43:20)
at OpenAI.makeStatusError (C:\Python Important\call-gpt-main\node_modules\openai\core.js:251:33)
at OpenAI.makeRequest (C:\Python Important\call-gpt-main\node_modules\openai\core.js:290:30)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async GptService.completion (C:\Python Important\call-gpt-main\services\gpt-service.js:55:20)
at async GptService.completion (C:\Python Important\call-gpt-main\services\gpt-service.js:114:9) {
status: 400,
headers: {
'access-control-allow-origin': '*',
'alt-svc': 'h3=":443"; ma=86400',
'cf-cache-status': 'DYNAMIC',
'cf-ray': '877c5ba83ad621e3-KHI',
connection: 'keep-alive',
'content-length': '203',
'content-type': 'application/json',
date: 'Sun, 21 Apr 2024 09:30:54 GMT',
'openai-organization': 'user-gntjrfo4a4rpvvm5p9udpbnq',
'openai-processing-ms': '40',
'openai-version': '2020-10-01',
server: 'cloudflare',
'set-cookie': '__cf_bm=iKylCws.LiQAZjpYO7fLbZyeMgMRa8pUZT7aUFV6k9Q-1713691854-1.0.1.1-kKpI9bMcsYZXFTmzd2m5KozjqvGMkwwPYslFKJnFMLX.Fw4md0fCxEe0B0RWxPXenKhGQw6MkE7maRZ_osYOpg; path=/; expires=Sun, 21-Apr-24 10:00:54 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None, _cfuvid=DFu1pwBHgbgfvXVgiE_thT8EI4OpFrDNR6GdEkJB8zA-1713691854483-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None',
'strict-transport-security': 'max-age=15724800; includeSubDomains',
'x-ratelimit-limit-requests': '500',
'x-ratelimit-limit-tokens': '300000',
'x-ratelimit-remaining-requests': '499',
'x-ratelimit-remaining-tokens': '299618',
'x-ratelimit-reset-requests': '120ms',
'x-ratelimit-reset-tokens': '76ms',
'x-request-id': 'req_669c8e2baa1c400655a23fd6ab91b696'
},
error: {
message: "'checkPrice' is not one of ['system', 'assistant', 'user', 'function'] - 'messages.8.role'",
type: 'invalid_request_error',
param: null,
code: null
},
code: null,
param: null,
STT -> Speech was already final when UtteranceEnd recevied
Twilio -> Audio completed mark (195): 3898e4b5-4a15-4b41-871f-fd3c47b8ee77
Interaction 0: GPT -> TTS: Hi there! Thanks for calling in today. •
Interaction 0: GPT -> TTS: How can I assist you with your AirPods selection? •
Interaction 0: TTS -> TWILIO: Hi there! Thanks for calling in today. •
Interaction 0: GPT -> TTS: Do you prefer headphones that go in your ear •
Interaction 0: GPT -> TTS: or over the ear?
GPT -> user context length: 5
Interaction 0: TTS -> TWILIO: How can I assist you with your AirPods selection? •
Interaction 0: TTS -> TWILIO: Do you prefer headphones that go in your ear •
Interaction 0: TTS -> TWILIO: or over the ear?
Twilio -> Audio completed mark (467): 0ece6179-11ff-4a59-a4e9-eda039dabe4b
Twilio -> Audio completed mark (596): accf6e83-7e3c-4af4-b8d4-27bf3eb5ffa2
Twilio -> Audio completed mark (697): 9c2d998d-a0b5-43d6-860a-b4fd98ea9bf7
Twilio -> Audio completed mark (754): bb3228de-18f9-4019-b31d-f4119f48bc44
Interaction 1 – STT -> GPT: Can you tell me
Interaction 1: GPT -> TTS: Absolutely! •
Interaction 1: TTS -> TWILIO: Absolutely! •
Interaction 1: GPT -> TTS: Would you like to know more about the in-ear AirPods and AirPods Pro, •
Twilio -> Interruption, Clearing stream
Twilio -> Audio completed mark (1109): bd21b254-3f01-49b1-8119-08d3dcddffa0
Interaction 1: GPT -> TTS: or the over-the-ear AirPods Max?
GPT -> user context length: 7
Interaction 1: TTS -> TWILIO: Would you like to know more about the in-ear AirPods and AirPods Pro, •
Twilio -> Interruption, Clearing stream
Interaction 1: TTS -> TWILIO: or the over-the-ear AirPods Max?
Twilio -> Audio completed mark (1171): 44c7d4e8-1820-42e1-8456-8087fe4b8b92
UtteranceEnd received before speechFinal, emit the text collected so far: one headphone price?
Interaction 2 – STT -> GPT: one headphone price?
Twilio -> Audio completed mark (1280): ea654842-6155-4e89-a724-dca99bdbd21b
Warning: Double function arguments returned by OpenAI: {"model": "airpods"}{"model": "airpods pro"}{"model": "airpods max"}
Interaction 2: GPT -> TTS: Let me check the price, one moment.
GPT -> called checkPrice function
C:\Python Important\call-gpt-main\node_modules\openai\error.js:43
return new BadRequestError(status, error, message, headers);
^
BadRequestError: 400 'checkPrice' is not one of ['system', 'assistant', 'user', 'function'] - 'messages.8.role'
at APIError.generate (C:\Python Important\call-gpt-main\node_modules\openai\error.js:43:20)
at OpenAI.makeStatusError (C:\Python Important\call-gpt-main\node_modules\openai\core.js:251:33)
at OpenAI.makeRequest (C:\Python Important\call-gpt-main\node_modules\openai\core.js:290:30)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async GptService.completion (C:\Python Important\call-gpt-main\services\gpt-service.js:55:20)
at async GptService.completion (C:\Python Important\call-gpt-main\services\gpt-service.js:114:9) {
status: 400,
headers: {
'access-control-allow-origin': '*',
'alt-svc': 'h3=":443"; ma=86400',
'cf-cache-status': 'DYNAMIC',
'cf-ray': '877c5ba83ad621e3-KHI',
connection: 'keep-alive',
'content-length': '203',
'content-type': 'application/json',
date: 'Sun, 21 Apr 2024 09:30:54 GMT',
'openai-organization': 'user-gntjrfo4a4rpvvm5p9udpbnq',
'openai-processing-ms': '40',
'openai-version': '2020-10-01',
server: 'cloudflare',
'set-cookie': '__cf_bm=iKylCws.LiQAZjpYO7fLbZyeMgMRa8pUZT7aUFV6k9Q-1713691854-1.0.1.1-kKpI9bMcsYZXFTmzd2m5KozjqvGMkwwPYslFKJnFMLX.Fw4md0fCxEe0B0RWxPXenKhGQw6MkE7maRZ_osYOpg; path=/; expires=Sun, 21-Apr-24 10:00:54 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None, _cfuvid=DFu1pwBHgbgfvXVgiE_thT8EI4OpFrDNR6GdEkJB8zA-1713691854483-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None',
'strict-transport-security': 'max-age=15724800; includeSubDomains',
'x-ratelimit-limit-requests': '500',
'x-ratelimit-limit-tokens': '300000',
'x-ratelimit-remaining-requests': '499',
'x-ratelimit-remaining-tokens': '299618',
'x-ratelimit-reset-requests': '120ms',
'x-ratelimit-reset-tokens': '76ms',
'x-request-id': 'req_669c8e2baa1c400655a23fd6ab91b696'
},
error: {
message: "'checkPrice' is not one of ['system', 'assistant', 'user', 'function'] - 'messages.8.role'",
type: 'invalid_request_error',
param: null,
code: null
type: 'invalid_request_error'
}
STT -> Speech was already final when UtteranceEnd recevied
Twilio -> Audio completed mark (195): 3898e4b5-4a15-4b41-871f-fd3c47b8ee77
Interaction 0: GPT -> TTS: Hi there! Thanks for calling in today. •
Interaction 0: GPT -> TTS: How can I assist you with your AirPods selection? •
Interaction 0: TTS -> TWILIO: Hi there! Thanks for calling in today. •
Interaction 0: GPT -> TTS: Do you prefer headphones that go in your ear •
Interaction 0: GPT -> TTS: or over the ear?
GPT -> user context length: 5
Interaction 0: TTS -> TWILIO: How can I assist you with your AirPods selection? •
Interaction 0: TTS -> TWILIO: Do you prefer headphones that go in your ear •
Interaction 0: TTS -> TWILIO: or over the ear?
Twilio -> Audio completed mark (467): 0ece6179-11ff-4a59-a4e9-eda039dabe4b
Twilio -> Audio completed mark (596): accf6e83-7e3c-4af4-b8d4-27bf3eb5ffa2
Twilio -> Audio completed mark (697): 9c2d998d-a0b5-43d6-860a-b4fd98ea9bf7
Twilio -> Audio completed mark (754): bb3228de-18f9-4019-b31d-f4119f48bc44
Interaction 1 – STT -> GPT: Can you tell me
Interaction 1: GPT -> TTS: Absolutely! •
Interaction 1: TTS -> TWILIO: Absolutely! •
Interaction 1: GPT -> TTS: Would you like to know more about the in-ear AirPods and AirPods Pro, •
Twilio -> Interruption, Clearing stream
Twilio -> Audio completed mark (1109): bd21b254-3f01-49b1-8119-08d3dcddffa0
Interaction 1: GPT -> TTS: or the over-the-ear AirPods Max?
GPT -> user context length: 7
Interaction 1: TTS -> TWILIO: Would you like to know more about the in-ear AirPods and AirPods Pro, •
Twilio -> Interruption, Clearing stream
Interaction 1: TTS -> TWILIO: or the over-the-ear AirPods Max?
Twilio -> Audio completed mark (1171): 44c7d4e8-1820-42e1-8456-8087fe4b8b92
UtteranceEnd received before speechFinal, emit the text collected so far: one headphone price?
Interaction 2 – STT -> GPT: one headphone price?
Twilio -> Audio completed mark (1280): ea654842-6155-4e89-a724-dca99bdbd21b
Warning: Double function arguments returned by OpenAI: {"model": "airpods"}{"model": "airpods pro"}{"model": "airpods max"}
Interaction 2: GPT -> TTS: Let me check the price, one moment.
GPT -> called checkPrice function
**C:\Python Important\call-gpt-main\node_modules\openai\error.js:43
return new BadRequestError(status, error, message, headers);
^
BadRequestError: 400 'checkPrice' is not one of ['system', 'assistant', 'user', 'function'] - 'messages.8.role'
at APIError.generate (C:\Python Important\call-gpt-main\node_modules\openai\error.js:43:20)**
at OpenAI.makeStatusError (C:\Python Important\call-gpt-main\node_modules\openai\core.js:251:33)
at OpenAI.makeRequest (C:\Python Important\call-gpt-main\node_modules\openai\core.js:290:30)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async GptService.completion (C:\Python Important\call-gpt-main\services\gpt-service.js:55:20)
at async GptService.completion (C:\Python Important\call-gpt-main\services\gpt-service.js:114:9) {
status: 400,
headers: {
'access-control-allow-origin': '*',
'alt-svc': 'h3=":443"; ma=86400',
'cf-cache-status': 'DYNAMIC',
'cf-ray': '877c5ba83ad621e3-KHI',
connection: 'keep-alive',
'content-length': '203',
'content-type': 'application/json',
date: 'Sun, 21 Apr 2024 09:30:54 GMT',
'openai-organization': 'user-gntjrfo4a4rpvvm5p9udpbnq',
'openai-processing-ms': '40',
'openai-version': '2020-10-01',
server: 'cloudflare',
'set-cookie': '__cf_bm=iKylCws.LiQAZjpYO7fLbZyeMgMRa8pUZT7aUFV6k9Q-1713691854-1.0.1.1-kKpI9bMcsYZXFTmzd2m5KozjqvGMkwwPYslFKJnFMLX.Fw4md0fCxEe0B0RWxPXenKhGQw6MkE7maRZ_osYOpg; path=/; expires=Sun, 21-Apr-24 10:00:54 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None, _cfuvid=DFu1pwBHgbgfvXVgiE_thT8EI4OpFrDNR6GdEkJB8zA-1713691854483-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None',
'strict-transport-security': 'max-age=15724800; includeSubDomains',
'x-ratelimit-limit-requests': '500',
'x-ratelimit-limit-tokens': '300000',
'x-ratelimit-remaining-requests': '499',
'x-ratelimit-remaining-tokens': '299618',
'x-ratelimit-reset-requests': '120ms',
'x-ratelimit-reset-tokens': '76ms',
'x-request-id': 'req_669c8e2baa1c400655a23fd6ab91b696'
},
error: {
message: "'checkPrice' is not one of ['system', 'assistant', 'user', 'function'] - 'messages.8.role'",
type: 'invalid_request_error',
param: null,
code: null
Node.js v21.7.3
Hi, I am facing an issue.
When I deploy this to EC2 voice is not coming from server to my phone.
It is working on local environment.
I am getting response from gpt and text to speech service is also working I think.
What could be the issue?
The project is very impressive.
I will break it down to two issues,
I have observed that the current implementation of the AI voice agent, which uses OpenAI, Deepgram, and Twilio, experiences a delay of 4-5 seconds before responding when a call begins. This is despite using the stream = true feature. It appears that the response is delayed until the stream is completed.
In the current implementation, there is a loop that buffers the audio:
while(Object.prototype.hasOwnProperty.call(this.audioBuffer, this.expectedAudioIndex)) { const bufferedAudio = this.audioBuffer[this.expectedAudioIndex]; this.sendAudio(bufferedAudio); this.expectedAudioIndex++; } } else { this.audioBuffer[index] = audio; }
I believe this delay can be reduced by utilizing WebSocket within the while loop to stream the audio in chunks, rather than waiting for the entire stream to connect.
By implementing WebSocket for chunk-by-chunk streaming, the AI voice agent can respond more promptly, significantly enhancing the user experience.
Please let me know if I make sense or there is any reason for you to handle the stream in this way?
Could you please create another option for speech to text using Deepgram Aurora? It has lower latency that ElevenLabs, improving the overall experience.
i can't get it to work. i have set it up on fly.io, got all the api keys, upgraded eleven labs, everything is set in .env. when i call it, it establishes a websocket connection, it gets an incoming request from twilio, it response with the connection endpoint, it opens a new websocket connection, and then nothing happens, or at least i am not getting any reponses. tried everything with chatgpt but i am out of ideas on what the issue could be. any pointers?
I got Deepgram to somewhat work for TTS but it only supports "mulaw" which works with twilio but there is an odd click/noise each time a new audio file starts playback.
Anyone been able to get it to work smoothly? maybe encoding the chunks and then sending to base64 ?
Deegram these days is a much better choices then 11lab, cheaper and faster for TTS.
I am getting just click sounds and static sounds the one you get when you tap the mic and then nothing.
Any reason why this is happening?
I am trying to deploy to vercel, followed all the instructions available on the internet, but not able to do deploy on vercel, could you please provide config setting to deploy to vercel?
So i tried few combination from - https://www.twilio.com/docs/voice/twiml/stream
I tried having the flag set in different location while editing app.js, either errors the app, or does not record on Twilio.
app.post('/incoming', (req, res) => {
res.status(200);
res.type('text/xml');
res.end(`
<Response>
// tried here
<Connect>
// tried here
<Stream url="wss://${process.env.SERVER}/connection" Record="true" />
<Record transcribe="true" playBeep="false" />
</Connect>
</Response>
`);
});
whenever I try to execute the test code twilio makes the call to me but then it says an application error has occurred . I dont know if I am doing something wrong let me know how can I fix this
Hi, I just come across this project and tested it out. It worked wonder. I am wondering if we are able to train the Chatgpt with our own data such as PDF, Doc , text files or Website Scraper?
Thank you
Would it be possible to use a ubuntu VPS or EC2 to create a webhook and forego ngrok?
Tried out the project, very impressed. Thanks for open sourcing. Quick question on latency.
Noticed a minimum latency of at least 3-4s. I am measuring latency as delay between when the human speaks and when the AI responds. This was with everything deployed on fly.io in Ashburn using the exact demo as instructed.
Looks like the biggest bottleneck is the request from Twilio -> Fly.io and Fly.io -> Twilio. Second biggest bottleneck looks like transcription via deepgram.
The ReadMe suggests a latency of 1s—can you clarify the definition of latency here? Is that just looking at gpt response + TTS?
Any ideas on how to reduce latency? Is there a roadmap for this project we can follow somewhere?
Wanted to ask if I can make a PR adding typescript and typings in general, also moving away from require to import, and adding a dist and build folders, just tested and it seems to run fine, who can I ask permissions to push a branch and open a PR if this is a wanted feature?
Great work, thank you for your contribution!
For cost purposes I tried using Neets.ai and Deepgram's TTS but cant get them to work. Terminal seems to log GPT-->TTS but not TTS --> Twilio despite following the docs and matching encoding. Any advice would be greatly appreciated.
Thanks again!
Abe
Instead of functions. Can we use post api convert api response into voice than send to call ? My idea is I don't want to limit user to limited questions like a voice chat ai
We send chunks of text to ElevenLabs that are different lengths. Generally speaking, the STT recordings come back in the right order. But in some circumstances a very short recording might be processed faster than a longer one and arrive first, even though it was meant to be played later.
We should create a cache for ElevenLabs recordings and re-order outgoing recordings to Twilio Media Streams as necessary so that they follow the correct sequence.
call-gpt/services/gpt-service.js
Line 111 in 9968ade
The function updateUserContext(name, role, text)
has the following order of params, but in the function call when using tools within the gpt-service, you have the ordering as this.updateUserContext('function', functionName, functionResponse);
which is (role, name, text)
. This causes all of the tool fns to throw since it's expecting a role type of function
but you are passing in the function name instead.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.