Commit Graph

62 Commits

Author SHA1 Message Date
Hoan Luu Huu
1cdd0cf611 feat Audult call session should have its own requestor (#535)
* feat Audult call session should have its own requestor

* fix eslint

* fix eslint

* if user authenticate from http server instead of databse

* wip

* fix custom STT

* fix custom STT

* fix custom STT
2023-11-16 08:22:35 -05:00
Dave Horton
6d625d87ad Feat/assemblyai testing (#516)
* handle errors from assemblyai

* wip

* fix alert

* normalizeAssemblyai
2023-11-02 17:05:28 -04:00
Hoan Luu Huu
7fee2ba2dc feat assembly (#515)
* fix

* wip

* wip

* wip

* wip

* fix review comments
2023-11-02 09:25:04 -04:00
Dave Horton
f43a5c1491 deepgram: rework continuous asr, and resolve on speech_final not is_f… (#501)
* deepgram: rework continuous asr, and resolve on speech_final not is_final (wip)

* wip

* deepgram: empty final transcript should trigger resolve with speech if we have buffered transcripts

* wip

* fixes for deepgram compiling multiple transcripts

* test deepgram utteranceEndMs

* more handling of utteranceEndMs

* wip

* better handling of digit strings collected over multiple deepgram responses

* wip

* add support for deepgramOptions.shortUtterance which triggers off of is_final instead of speech_final

* apply deepgram fixes to transcribe

* cleanup continnuous asr

* more continuous asr fixes for deepgram

* update to verb-specifications for handling SttTask properties

* set log level for tests back to error
2023-10-30 13:57:25 -04:00
Hoan Luu Huu
2285ec5329 fix transcribe credential (#480) 2023-10-10 21:06:46 -04:00
Hoan Luu Huu
09ae083c9a fix: transcribe 2 channels (#479)
* fix: transcribe 2 channels

* fix

* fix
2023-10-10 08:46:50 -04:00
Hoan Luu Huu
ae9386791f fix cobalt model (#469) 2023-09-26 07:41:43 -04:00
Dave Horton
6a8a2aa955 update to provide hostport and model on the command line not via env (#465) 2023-09-13 13:53:00 -04:00
Dave Horton
a1793ac359 Speech vendor/cobalt (#463)
* initial changes for cobalt speech

* wip

* wip

* update to drachtio-fsmrf that supports cobalt

* update to verb-specifications with cobalt speech support

* more wip

* lint

* use node 18 with gh actions

* support for compiling cobalt hints

* fix bug in uuid_cobalt_compile_context

* update verb-specifications

* remove repeated code

* cobalt support for transcribe

* update to verb specs
2023-09-13 09:47:30 -04:00
Hoan Luu Huu
1477605e66 fix custom stt transcribe (#457) 2023-09-08 08:09:03 -04:00
Hoan Luu Huu
2935574440 feat pause resume transcribe (#438)
* feat pause resume transcribe

* wip

* fix jslint

* update fsmrf
2023-08-30 22:43:50 -04:00
Hoan Luu Huu
b1ecf069bf Feat/fallback speech 02 (#429)
* feat fallback speech

* feat fallback speech

* feat fallback speech

* gather verb

* fix

* wip

* wip

* wip

* wip

* wip

* transcribe

* transcribe

* fix

* wip

* wip

* wip

* wip

* fix

* wip

* fix
2023-08-22 09:22:01 -04:00
Hoan Luu Huu
f8c5abe9e9 feat: multi speech credential diff labels but same vendor (#426)
* feat: multi speech credential diff labels but same vendor

* update sql

* fix

* fix

* fix jslint

* fix review comment

* update verb spec version
2023-08-15 08:57:49 -04:00
Hoan Luu Huu
d7dcdb1d0c Continuos ASR for transcribe (#398)
* asrTimeout

* fix jslint

* change log

* fix interrim
2023-08-03 09:49:44 -04:00
Dave Horton
fbd0782258 #388 - support custom speech vendor in transcribe verb (#414)
Co-authored-by: Hoan Luu Huu <110280845+xquanluu@users.noreply.github.com>
2023-08-02 19:06:31 -04:00
Dave Horton
a7d07ce7ae add channel to transcribe, gather, and dtmf spans (#376) 2023-06-13 09:12:26 -04:00
Dave Horton
30beb9c093 transcribe: default hints and altLanguages (#371) 2023-06-06 13:41:31 -04:00
Dave Horton
01260ad054 transcribe: create otel child spans for each stt result that is returned in a long-running transcribe (#368) 2023-06-02 14:25:32 -04:00
Dave Horton
bd911c88f9 in verb transcribe, recognizer should be optional (default to application default) but transcriptionHook mudt be mandatory (#367) 2023-06-02 13:15:32 -04:00
Dave Horton
5131d524ce bugfix: allow for empty transcripts that nuance returns 2023-03-27 14:13:50 -04:00
Hoan Luu Huu
7be3c64116 feat: update speech-ultil version 1.0.1 (#275)
* feat: update speech-ultil version 1.0.1

* feat: update speech-ultil version 1.0.1

* more fixes for custom stt

* more fixes

* fixes

* update drachtio-fsmrf

* pass url to mod_jambonz_transcribe

* transcription utils: handle custom results

* handle custom speech vendor errors

* add support for hints to custom speech

* change to custom speech options

* send hints as an array for custom speech

* update latest speech-utils

* transcribe: changes to support soniox

* bugfix: soniox transcribe

---------

Co-authored-by: Quan HL <quanluuhoang8@gmail.com>
Co-authored-by: Dave Horton <daveh@beachdognet.com>
2023-03-12 19:38:36 -04:00
Dave Horton
1c683f1142 initial changes for soniox (#270)
* initial changes for soniox

* changes to gather for soniox

* parse soniox stt results

* handle <end> token for soniox

* soniox: handle empty array of words

* support for soniox hints

* add soniox storage options

* update to verb specs

* add support for transcribe

* compile soniox transcripts

* gather: kill no input timer for soniox when we get interim results

* fix buffering of soniox transcripts

* fix for compiling soniox transcript

* another fix for compiling soniox transcript

* another fix

* handling of <end> token

* fix soniox bug

* gather: fixes for soniox continous asr

* fix undefined variable reference

* fix prev commit

* bugfix: allow verb_status requests

* gather: for soniox no need to restart transcription after final transcription received

* update verb specs

* update verb specs, fixes for continuous asr:
2023-03-03 13:37:55 -05:00
Hoan Luu Huu
c09425fa89 feat: use verb-specifications (#262)
* feat: use verb-specifications

* feat: use verb-specifications

* fix: verb specification v2

* remove irrelevant tests

* fix: verb-scpecification

* update to use @jambonz/verb-specifications

---------

Co-authored-by: Quan HL <quanluuhoang8@gmail.com>
Co-authored-by: Dave Horton <daveh@beachdognet.com>
2023-02-15 09:56:23 -05:00
Dave Horton
0fdcb3a6d6 Feature/nvidia speech (#261)
* initial changes for nvidia speech

* allow nvidia speech credentials to be set at runtime

* update drachtio-fsmrf

* fix handling of nvidia-specific options

* fix nvidia custom config

* fix nvidia word time offsets

* fix nvidia custom configuration

* normalize nvidia transcripts

* update to @jambonz/realtime-dbhelpers with nvidia tts support
2023-02-12 14:06:01 -05:00
Dave Horton
567b03fd36 bugfix: transcribe/gather using default as vendor 2023-01-11 15:31:24 -05:00
Dave Horton
d5c04d2133 transcribe and gather: silently discard listening events from ibm stt 2023-01-11 14:59:15 -05:00
Dave Horton
71a2435c63 Feature/ibm watson (#193)
* initial changes to support ibm watson

* update specs.json for ibm

* update to drachtio-fsmrf with support for ibm

* bugfix: set access token for ibm stt, not api_key

* fix name of api_key

* normalize ibm transcription results

* rework ibm credentials

* bugfix setting runtime speech creds

* bugfix: ibm region

* typo

* changes to transcribe for ibm watson

* implement connect handler

* bugfix: bind error

* proper use of result_index

* ibm error handling
2022-11-21 22:09:37 -05:00
Dave Horton
8686348454 Feature/deepgram stt (#190)
* initial changes to support deepgram stt

* fixes for normalizing vendor-specific transcriptions

* update to latest drachtio-fsmrf with support for deepgram stt

* deepgram parsing error

* hints support for deepgram

* handling deepgram errors

* ignore late arriving transcripts for deepgram

* handling of empty transcripts

* transcribe changes

* allow deepgram stt credentials to be provided at run time

* bind channel in transcription handler

* fixes for transcribe when handling empty transcripts

* more empty transcript fixes

* update tests to latest modules

* add test cases for deepgram speech recognition
2022-11-12 19:48:59 -05:00
Dave Horton
706cd4b94b bugfix: handle gather/transcribe where vendor not explicitly specified #187 2022-11-07 09:31:51 -05:00
Dave Horton
509bb065bb Feature/nuance stt (#185)
* initial changes to gather to support nuance stt

* updateSpeechCredentialLastUsed could be called without a speech_credential_sid if credentials are passed in the flow

* fix bugname

* typo

* added handlers for nuance

* logging

* major refactor of parsing transcriptions

* initial support for nuance in transcribe verb

* updates from testing

* cleanup some tests

* update action

* typo

* gather: start nuance timers after say/play completes

* update drachtio-fsrmf

* refactor some code

* typo

* log nuance error detail

* timeout handling

* typo

* handle nuance 413 response when recognition times out

* typo in specs.json

* add support for nuance resources

* fixes and tests for transcribe

* remove logging from test

* initial support for kryptonEndpoint

* try getting access token even when using krypton

* typo in kryptonEndpoint property

* add support for Nuance tts

* parse nuance voice and model for tts

* use nuance credentials from db

* update to db-helpers@0.7.0 with caching option

* add support for azure audio logging in gather/transcribe

* sync package-lock.json
2022-11-01 12:23:49 -04:00
Dave Horton
b25f92e17a Feature/azure custom stt (#171)
* gather/transcribe: support for azure custom speech models (endpoint id)

* allow azure stt custom speech endpoint id to be passed as property in recognizer

* fix to add custom stt endpoint to session speech credentials object
2022-10-07 09:46:25 +01:00
Dave Horton
90cb5e1348 bugfix: typo in bugname was causing transcripts to be ignored 2022-10-04 12:59:58 +01:00
Dave Horton
bd49dacac4 Say length text (#165)
* typo for media bug name in azure and punctuation fix

* say: split very long text intelligently

* more fixes from testing

* update to latest synthAudio
2022-09-14 17:17:29 +02:00
Dave Horton
c88163fe11 Bugfix/config stt punctuation (#164)
* support recognizer.punctuation in config verb (#163)

* fixes from testing
2022-09-13 11:45:36 +02:00
Dave Horton
887c6243e2 handle altLanguages set at the session level via config verb; fix azure stt race condition with final transcripts from stopped recognition 2022-08-25 22:43:38 +02:00
Dave Horton
6346954e7a session-level speech hints, strip trailing punctuation on continuous asr (#151) 2022-08-18 23:18:24 +02:00
Dave Horton
3298918322 Feature/siprec server (#143)
* fixes from testing

* modify Task#exec to take resources as an object rather than argument list

* pass 2 endpoints to Transcribe when invoked in a SipRec call session

* logging

* change siprec invite to sendrecv just so freeswitch does not try to reinvite (TODO: block outgoing media at rtpengine)

* Config: when enabling recording, block until siprec dialog is established

* missed play verb in commit 031c79d

* linting

* bugfix: get final transcript in siprec call
2022-08-09 15:23:55 +02:00
Dave Horton
2882fa2d0a Feature/vm detection (#137)
* initial changes for amd

* wip

* fix bug where transcripts were discarded

* a bit of refactoring, and adding support for avmd in config verb

* bug fixes
2022-07-27 17:46:52 +01:00
Dave Horton
c3e5ffa52d bugfix: transcribe of a dialed call can now occur on both legs 2022-05-15 13:45:55 -04:00
Dave Horton
6d34850dc6 bugfix: transcribe Azure interim transcripts were missing 2022-05-11 19:22:14 -04:00
Dave Horton
182ad8c716 expose model and singleUtterance to gather/transcribe when using google 2022-05-08 12:29:55 -04:00
Dave Horton
b37881a059 bugfix: second part of outbound dial fix over wss 2022-05-07 11:52:29 -04:00
Dave Horton
72aaf80335 add support for multiple languages when using Azure STT 2022-04-26 15:07:55 -04:00
Dave Horton
359cb82d80 per recommendation from microsoft, do NOT sort transcripts by confidence: first transcript in the returned list is 'best' 2022-04-17 17:53:16 -04:00
Dave Horton
a950f9f738 Feature/trace propagation (#96)
* add b3 header for trace propagation on initial webhook

* logging

* add tracing context to all webhooks

* Add span parameter to Task.getTracingPropagation. Pass proper span to getTracingPropagation calls in Task methods to propagate the proper spanId (#91)

* some tracing cleanup

* bugfix: azure stt results need to be ordered by confidence level before processing (#92)

* fix assertion

* bugfix: vad was not enabled on config verb, restart STT on empty transcript in gather

* gather: dont send webhook if call is gone

* rest outdial: handle 302 redirect so we can later cancel request if needed (#95)

* gather: restart if we get an empty transcript (looking at you, Azure)

Co-authored-by: javibookline <98887695+javibookline@users.noreply.github.com>
2022-04-01 14:48:27 -04:00
Dave Horton
172dc1aaa7 Feature/config verb (#77)
* remove cognigy verb

* initial implementation of config verb

* further updates to config

* Bot mode alex (#75)

* do not use default as value for TTS/STT

* fix gather listener if no say or play provided

Co-authored-by: akirilyuk <a.kirilyuk@cognigy.com>

* gather: listenDuringPrompt requires a nested play/say

* fix exception

* say: fix exception where caller hangs up during say

* bugfix: sip refer was not ending if caller hungup during refer

* add support for sip:request to ws commands

* gather: when bargein is set and minBargeinWordCount is zero, kill audio on endOfUtterrance

* gather/transcribe: add support for google boost and azure custom endpoints

* minor logging changes

* lint error

Co-authored-by: akirilyuk <45361199+akirilyuk@users.noreply.github.com>
Co-authored-by: akirilyuk <a.kirilyuk@cognigy.com>
2022-03-06 15:09:45 -05:00
Dave Horton
3c5d392407 Feature/ws api (#72)
initial changes to support websockets as an alternative to webhooks
2022-02-26 14:06:52 -05:00
Dave Horton
30ed5b6a02 add support for vad to gather and transcribe (#67) 2022-02-10 08:45:16 -05:00
Dave Horton
752eed428f cognigy: when use azuyre tts, request detailed output format 2022-01-14 08:48:55 -05:00
Dave Horton
afb874aabc minor logging change 2022-01-14 07:56:11 -05:00