Commit Graph

63 Commits

Author SHA1 Message Date
Hoan Luu Huu
dbdc1cd43d support voxist stt (#1066)
* support voxist stt

* wip
2025-02-05 08:33:35 -05:00
Hoan Luu Huu
2ddcd53d6b support elevenlabs s2s (#1052)
* support elevenlabs s2s

* wip

* wip

* wip
2025-02-02 10:29:48 -05:00
Hoan Luu Huu
ba282d775d support rimelabs tts streaming (#1047) 2025-01-18 08:17:33 -05:00
Dave Horton
a194ba833e Feat/1041 (#1045)
* initial changes for stream synonym to listen

* listen on B endpoint if nested listen in dial has channel === 2
2025-01-17 08:48:39 -05:00
rammohan-y
77f3d9d7ec feat/1034: sending socket close code when there is no response from the websocket app (#1035) 2025-01-16 10:13:00 -05:00
Hoan Luu Huu
ca8cace284 support custom tts streaming (#1023)
* support custom tts streaming

* wip

* wip

* wip

* wip

* wip

* wip

* fix review comments
2025-01-14 07:24:06 -05:00
Hoan Luu Huu
499c800213 Feat/ultravox s2s (#1032)
* support ultravox_s2s

* support ultravox_s2s

* support ultravox_s2s

* wip

* wip

* wip

* wip

* fix ultravox toolcall

* wip
2025-01-14 07:11:55 -05:00
Hoan Luu Huu
564f6c9e55 support kill dial if sd ep is media timeout (#1001)
* support kill dial if sd ep is media timeout

* support kill dial if sd ep is media timeout

* support kill dial if sd ep is media timeout

* add media timeout reason header to bye message

* wip

* wip

* make configuration for freeswitch media timeout

* make configuration for freeswitch media timeout

* wip
2024-12-23 07:19:41 -05:00
Dave Horton
ba3f46df64 Feat/tts streaming (#994)
* wip

* add TtsStreamingBuffer class to abstract handling of streaming tokens

* wip

* add throttling support

* support background ttsStream (#995)

* wip

* add TtsStreamingBuffer class to abstract handling of streaming tokens

* wip

* support background ttsStream

* wip

---------

Co-authored-by: Dave Horton <daveh@beachdognet.com>

* wip

* dont send if we have nothing to send

* initial testing with cartesia

* wip

---------

Co-authored-by: Hoan Luu Huu <110280845+xquanluu@users.noreply.github.com>
2024-12-18 14:44:37 -05:00
Dave Horton
5e04db82bf Feat/deepgram voice agent (#1006)
* wip

* wip

* wip
2024-12-13 10:05:23 -05:00
rammohan-y
0458bb7d6c Feat/884: Capture system_alert when feature-server is online or offline (#950)
* writing alerts during startup and shutdown of feature-server

* feat/884: created constants for system component name and state

* feat/88: added 0.2.11 version of time-series

* feat/884: renamed constant, and added GracefulShutdownInProgress system alert
2024-12-05 09:23:03 -05:00
Dave Horton
24a66fed64 wip (#979) 2024-11-19 09:37:00 -05:00
Dave Horton
c1330d4651 fix transcribe fixes for speechmatics (#978)
* fix transcribe fixes for speechmatics

* update to verb-specs with fixes for speechmatics

* add support for speechmatics translation

* add handlers for receiving translations

* call translation hookd

* gather: no need to restart speechmatics after a final transcript during continuous asr

* graceful shutdown

* wip

* wip

* wip

* wip

* wip
2024-11-16 10:21:04 -05:00
Dave Horton
f09722a5b5 Feat/llm verb (#931)
* wip

* working version for openai realtime beta

* lint

* tests: update db to latest 0.9.2 schema
2024-10-12 19:26:27 -04:00
Dave Horton
f84b3793e1 Feat/speechmatics (#932)
* wip

* initial working version of speechmatics

* linting
2024-10-12 18:42:53 -04:00
Hoan Luu Huu
c5e19bf775 support verbio speech (#757)
* support verbio speech

* wip

* wip

* wip

* update speech utils

* update verb specification
2024-05-29 07:55:46 -04:00
Hoan Luu Huu
498dd64025 support mod_vad_detect (#762)
* support mod_vad_detect

* wip

* update verb spec and drachtio fsmrf

* Update example-voicemail-greetings.json (#761)

Update voicemail english greetings

* wip

* stopvad if playdone

---------

Co-authored-by: Vinod Dharashive <vdharashive@gmail.com>
2024-05-29 07:31:59 -04:00
Dave Horton
8999c85a71 Fixes/ws testing dh (#704)
* fixes from testing with translator app

* more updates

* linting

* update gh actions to node 20

* add support for google v2 preconfigured recognizer

* add support for google voice activity events

* update to speech-utils@0.0.45

* update speech-utils to support caching azure tts

* transcribe must buffer transcripts for channel 1 and 2 separately

* further fix for accumulating transcripts

* linting

* deepgram sends transcripts with empty alternatives array

* fix deepgram returning an empty array
2024-04-03 14:30:49 -04:00
Hoan Luu Huu
5b1d8a8ff3 Feat/ambient sounds (#678)
* initial support for coaching mode in conference

* wip

* wip

* add support for answer verb

* wip

* wip

* wip

* wip

* wip

* updates to rename option to dub

* wip

* wip

* wip

* update verb-specs

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* add option to boost audio signal in main channel

* wip

* wip

* wip

* wip

* wip

* wip

* for now, bypass use of streaming apis when generating tts audio for dub tracks

* add nested dub to dial

* wip

* add support for filler noise

* kill filler noise when gather killed

* wip

* wip

* while using sayOnTrack, we have to enclose the say command in double quotes

* disableTtsStreaming = false

* allow transcribe of b leg only on dial verb

* dub.say can either be text or object like say verb with text and synthesizer

* remove loop for sayOnTrack

* update speech-utils

* fixes for testing transcribe verb and support for dub and boostAudioSignal in lcc commands

* add dial.boostAudioSignal

* fix bug where session-level recognizer settings incorrectly overwrite verb-level settings

* update verb specs

* update dial to support array of dub verbs

* fix bug setting gain

* lint

* wip

* update speech-utils

* use new endpoint methods for mod_dub

---------

Co-authored-by: Dave Horton <daveh@beachdognet.com>
2024-03-23 16:23:57 -04:00
Dave Horton
fc3655c9bd fixes for confirm session (#663)
* fixes for confirm session

* allow empty dialconfirm array
2024-02-22 12:33:35 -05:00
Dave Horton
6d625d87ad Feat/assemblyai testing (#516)
* handle errors from assemblyai

* wip

* fix alert

* normalizeAssemblyai
2023-11-02 17:05:28 -04:00
Hoan Luu Huu
7fee2ba2dc feat assembly (#515)
* fix

* wip

* wip

* wip

* wip

* fix review comments
2023-11-02 09:25:04 -04:00
Dave Horton
cb78627e66 tag is allowed in siprec (#475) 2023-10-06 08:53:13 -04:00
Dave Horton
a1793ac359 Speech vendor/cobalt (#463)
* initial changes for cobalt speech

* wip

* wip

* update to drachtio-fsmrf that supports cobalt

* update to verb-specifications with cobalt speech support

* more wip

* lint

* use node 18 with gh actions

* support for compiling cobalt hints

* fix bug in uuid_cobalt_compile_context

* update verb-specifications

* remove repeated code

* cobalt support for transcribe

* update to verb specs
2023-09-13 09:47:30 -04:00
Hoan Luu Huu
2935574440 feat pause resume transcribe (#438)
* feat pause resume transcribe

* wip

* fix jslint

* update fsmrf
2023-08-30 22:43:50 -04:00
Hoan Luu Huu
7be3c64116 feat: update speech-ultil version 1.0.1 (#275)
* feat: update speech-ultil version 1.0.1

* feat: update speech-ultil version 1.0.1

* more fixes for custom stt

* more fixes

* fixes

* update drachtio-fsmrf

* pass url to mod_jambonz_transcribe

* transcription utils: handle custom results

* handle custom speech vendor errors

* add support for hints to custom speech

* change to custom speech options

* send hints as an array for custom speech

* update latest speech-utils

* transcribe: changes to support soniox

* bugfix: soniox transcribe

---------

Co-authored-by: Quan HL <quanluuhoang8@gmail.com>
Co-authored-by: Dave Horton <daveh@beachdognet.com>
2023-03-12 19:38:36 -04:00
Dave Horton
1c683f1142 initial changes for soniox (#270)
* initial changes for soniox

* changes to gather for soniox

* parse soniox stt results

* handle <end> token for soniox

* soniox: handle empty array of words

* support for soniox hints

* add soniox storage options

* update to verb specs

* add support for transcribe

* compile soniox transcripts

* gather: kill no input timer for soniox when we get interim results

* fix buffering of soniox transcripts

* fix for compiling soniox transcript

* another fix for compiling soniox transcript

* another fix

* handling of <end> token

* fix soniox bug

* gather: fixes for soniox continous asr

* fix undefined variable reference

* fix prev commit

* bugfix: allow verb_status requests

* gather: for soniox no need to restart transcription after final transcription received

* update verb specs

* update verb specs, fixes for continuous asr:
2023-03-03 13:37:55 -05:00
Dave Horton
0fdcb3a6d6 Feature/nvidia speech (#261)
* initial changes for nvidia speech

* allow nvidia speech credentials to be set at runtime

* update drachtio-fsmrf

* fix handling of nvidia-specific options

* fix nvidia custom config

* fix nvidia word time offsets

* fix nvidia custom configuration

* normalize nvidia transcripts

* update to @jambonz/realtime-dbhelpers with nvidia tts support
2023-02-12 14:06:01 -05:00
Dave Horton
71a2435c63 Feature/ibm watson (#193)
* initial changes to support ibm watson

* update specs.json for ibm

* update to drachtio-fsmrf with support for ibm

* bugfix: set access token for ibm stt, not api_key

* fix name of api_key

* normalize ibm transcription results

* rework ibm credentials

* bugfix setting runtime speech creds

* bugfix: ibm region

* typo

* changes to transcribe for ibm watson

* implement connect handler

* bugfix: bind error

* proper use of result_index

* ibm error handling
2022-11-21 22:09:37 -05:00
Dave Horton
8686348454 Feature/deepgram stt (#190)
* initial changes to support deepgram stt

* fixes for normalizing vendor-specific transcriptions

* update to latest drachtio-fsmrf with support for deepgram stt

* deepgram parsing error

* hints support for deepgram

* handling deepgram errors

* ignore late arriving transcripts for deepgram

* handling of empty transcripts

* transcribe changes

* allow deepgram stt credentials to be provided at run time

* bind channel in transcription handler

* fixes for transcribe when handling empty transcripts

* more empty transcript fixes

* update tests to latest modules

* add test cases for deepgram speech recognition
2022-11-12 19:48:59 -05:00
Dave Horton
509bb065bb Feature/nuance stt (#185)
* initial changes to gather to support nuance stt

* updateSpeechCredentialLastUsed could be called without a speech_credential_sid if credentials are passed in the flow

* fix bugname

* typo

* added handlers for nuance

* logging

* major refactor of parsing transcriptions

* initial support for nuance in transcribe verb

* updates from testing

* cleanup some tests

* update action

* typo

* gather: start nuance timers after say/play completes

* update drachtio-fsrmf

* refactor some code

* typo

* log nuance error detail

* timeout handling

* typo

* handle nuance 413 response when recognition times out

* typo in specs.json

* add support for nuance resources

* fixes and tests for transcribe

* remove logging from test

* initial support for kryptonEndpoint

* try getting access token even when using krypton

* typo in kryptonEndpoint property

* add support for Nuance tts

* parse nuance voice and model for tts

* use nuance credentials from db

* update to db-helpers@0.7.0 with caching option

* add support for azure audio logging in gather/transcribe

* sync package-lock.json
2022-11-01 12:23:49 -04:00
Dave Horton
3298918322 Feature/siprec server (#143)
* fixes from testing

* modify Task#exec to take resources as an object rather than argument list

* pass 2 endpoints to Transcribe when invoked in a SipRec call session

* logging

* change siprec invite to sendrecv just so freeswitch does not try to reinvite (TODO: block outgoing media at rtpengine)

* Config: when enabling recording, block until siprec dialog is established

* missed play verb in commit 031c79d

* linting

* bugfix: get final transcript in siprec call
2022-08-09 15:23:55 +02:00
Dave Horton
2882fa2d0a Feature/vm detection (#137)
* initial changes for amd

* wip

* fix bug where transcripts were discarded

* a bit of refactoring, and adding support for avmd in config verb

* bug fixes
2022-07-27 17:46:52 +01:00
Dave Horton
9aa0df256d initial changes to support siprec recording (#120)
* initial changes to support siprec recording

* include additional params on SIP INFO to start recording

* add support for maniupulating recording via REST API

* fixes from testing pause/resume recording
2022-06-23 16:21:35 -04:00
Dave Horton
12ad7e556f added support for sip:request verb, used to send SIP INFO/NOTIFY etc during call (#116) 2022-06-15 13:31:32 -04:00
Dave Horton
4f2fc70383 add new type dial:confirm 2022-04-03 22:12:14 -04:00
Dave Horton
3dd703411c kill audio on vad when bargein is true and minBargeinWordCount is zero 2022-03-17 08:51:44 -04:00
Dave Horton
172dc1aaa7 Feature/config verb (#77)
* remove cognigy verb

* initial implementation of config verb

* further updates to config

* Bot mode alex (#75)

* do not use default as value for TTS/STT

* fix gather listener if no say or play provided

Co-authored-by: akirilyuk <a.kirilyuk@cognigy.com>

* gather: listenDuringPrompt requires a nested play/say

* fix exception

* say: fix exception where caller hangs up during say

* bugfix: sip refer was not ending if caller hungup during refer

* add support for sip:request to ws commands

* gather: when bargein is set and minBargeinWordCount is zero, kill audio on endOfUtterrance

* gather/transcribe: add support for google boost and azure custom endpoints

* minor logging changes

* lint error

Co-authored-by: akirilyuk <45361199+akirilyuk@users.noreply.github.com>
Co-authored-by: akirilyuk <a.kirilyuk@cognigy.com>
2022-03-06 15:09:45 -05:00
Dave Horton
3c5d392407 Feature/ws api (#72)
initial changes to support websockets as an alternative to webhooks
2022-02-26 14:06:52 -05:00
Dave Horton
8593f12b51 add custom headers to outdial, save unique uuid for running FS to redis 2022-01-08 11:50:18 -05:00
Dave Horton
3bf1984854 K8s changes (#55)
* K8S: dont send OPTIONS pings

* fix missing ref

* k8s pre-stop hook added

* k8s pre-stop hook changes

* chmod +x utility

* more k8s pre-stop changes

* pre stop

* fix healthcheck

* k8s pre-stop working

* add readiness probe

* fix bug in pre-stop

* logging

* revamp k8s pre-stop a bit

* initial support for cognigy bot

* more cognigy changes

* switch to use transcribe for cognigy

* #54 include callInfo in dialogflow event payload
2022-01-06 12:41:14 -05:00
Dave Horton
1e93973419 Feature/azure recognition (#46)
* add support for microsoft speech recognition

* update to drachtio-fsmrf that support microsoft stt

* gather and transcribe now support microsoft
2021-11-26 16:40:25 -06:00
Dave Horton
fe1778e9ae Feature/sip refer (#44)
* changes to support sip REFER

* implement actionhook

* changes from testing

* minor logging
2021-11-20 11:39:10 -05:00
Dave Horton
19f3cbaa43 initial support for Rasa 2021-09-07 13:43:40 -04:00
Dave Horton
1a2aaf9845 Feature/queue webhooks (#34)
* initial changes for queue webhooks

* send queue leave webhook when dequeued

* bugfix: if enqeue task is killed because it is being replaced with new app supplied by LCC, ignore any app returned from the actionHook as LCC takes precedence

* remove leftover merge brackets
2021-07-31 13:32:40 -04:00
Dave Horton
756db59671 update transcribe to support google v1p1beta1 and aws 2021-01-31 15:49:19 -05:00
Dave Horton
91f507bf3f add dmtf verb 2020-10-12 09:59:50 -04:00
Dave Horton
95e4c22969 add lex support 2020-10-09 08:28:36 -04:00
Dave Horton
c02aa94500 add sms messaging support 2020-10-09 08:00:17 -04:00
Dave Horton
a4bcfca9e6 added initial support for dialogflow 2020-07-08 14:16:37 -04:00