Compare commits

..

49 Commits

Author SHA1 Message Date
Dave Horton
8938bf25dc race condition where gather transcribe is restarted after final transcript 2024-05-21 19:04:55 -04:00
Dave Horton
f7134d8fe7 more logging on restart of transcribing during gather 2024-05-10 14:12:34 -04:00
Dave Horton
a23dc50c20 lint 2024-05-03 08:56:55 -04:00
Dave Horton
888fddff37 possible fix for race condition in gather which ends but lets transcription continue 2024-05-03 08:55:16 -04:00
Dave Horton
e1497f90a8 update with various deepgram fixes, including for #700 2024-04-01 13:03:52 -04:00
Dave Horton
ec58232b61 Fix/replace application issue (#692)
* fix scenario where ws replace application from gather while awaiting command and no tasks on execution stack

* lint

* remove some debug logging
2024-03-23 16:14:16 -04:00
Hoan Luu Huu
65c241bcd1 gather verb should clean dtmf listerner even dtmfBargein=false (#686) 2024-03-23 16:01:41 -04:00
Hoan Luu Huu
75b6f89e0c add log to get more detail for AMD issue (#687)
* add log to check issue

* update drachtio-fsmrf 3.0.38
2024-03-21 09:14:32 -04:00
Hoan Luu Huu
b80d39d205 fix asrtimer always return vendor=deepgram (#682) 2024-03-13 12:57:55 -04:00
Hoan Luu Huu
40f70e3531 update speech utils version 0.0.63 (#681) 2024-03-12 09:12:18 -04:00
Hoan Luu Huu
1914b88af9 support azure language id mode (#674) 2024-03-12 08:35:01 -04:00
Hoan Luu Huu
c946a5d14d fix actionHookDelay feature is not working properly if there is no de… (#679)
* fix actionHookDelayAction when no actions is defnied

* terminated by jambonz for giveuptimeout
2024-03-12 08:33:03 -04:00
Hoan Luu Huu
878578fe0f Fix/issue 676 (#680)
* fix bargin is not working

* fix bargin is not working
2024-03-11 08:46:38 -04:00
Hoan Luu Huu
9b3be6c0b9 allow custom header on pause, resume recording (#670)
* allow custom header on pause, resume recording

* fix review comments
2024-03-05 18:01:32 -05:00
Hoan Luu Huu
4ae661daea remove unnecessary code for cleanup disableBotMode (#673) 2024-03-04 18:03:32 -05:00
Dave Horton
dbd3b59901 fix #666 2024-02-26 09:39:49 -05:00
Hoan Luu Huu
06b066a3f2 update speech util to support whisper stream (#657)
* update speech util to support whisper stream

* minor editing of span attributes

* more span attrs cleanup

---------

Co-authored-by: Dave Horton <daveh@beachdognet.com>
2024-02-22 14:17:29 -05:00
Dave Horton
fc3655c9bd fixes for confirm session (#663)
* fixes for confirm session

* allow empty dialconfirm array
2024-02-22 12:33:35 -05:00
dependabot[bot]
1b5f801830 Bump undici from 5.26.2 to 5.28.3 (#647)
Bumps [undici](https://github.com/nodejs/undici) from 5.26.2 to 5.28.3.
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v5.26.2...v5.28.3)

---
updated-dependencies:
- dependency-name: undici
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-22 10:13:19 -05:00
Dave Horton
d0ebe3f99f fix possible undefined reference in precache audio (#662)
* fix possible undefined reference in precache audio

* fix parsing of JAMBONES_EAGERLY_PRE_CACHE_AUDIO
2024-02-22 07:58:41 -05:00
Dave Horton
51a379998f fix #655 (#658)
* fix #655

* fix race condition
2024-02-22 07:46:53 -05:00
dependabot[bot]
c2ae42a456 Bump ip from 1.1.8 to 1.1.9 (#660)
Bumps [ip](https://github.com/indutny/node-ip) from 1.1.8 to 1.1.9.
- [Commits](https://github.com/indutny/node-ip/compare/v1.1.8...v1.1.9)

---
updated-dependencies:
- dependency-name: ip
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-22 07:31:01 -05:00
Hoan Luu Huu
c187685054 feat actionHook delay action (#470)
* feat actionHook delay action

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip
2024-02-20 21:09:19 -05:00
Hoan Luu Huu
81234a583c support update record from application ws connection (#645) 2024-02-19 07:53:39 -05:00
Hoan Luu Huu
206849fa25 create outbound dial from webhook ws (#581)
* wip, create outbound dial from webhook ws

* wip, create outbound dial from webhook ws

* clean
2024-02-13 07:58:39 -05:00
Dave Horton
662b6d3d95 fix: elevenlabs caching with streaming 2024-02-12 21:08:41 -05:00
Anton Voylenko
5c070597cf tag outdial session (#643) 2024-02-12 13:16:43 -05:00
Dave Horton
42be9ff1ca update to speech-utils with env to disable elevenlabs streaming (default is on) 2024-02-12 12:49:45 -05:00
Dave Horton
f0533c881b deepgram gather: if both endpointing and utterance_end_ms are set (bu… (#644)
* deepgram gather: if both endpointing and utterance_end_ms are set (but not continous asr) return either when we get speech_final or UtteranceEnd.  This is the belt-and-suspenders apprach deepgram is recommending

* include verb id in action hook if one was provided in the verb set

* minor
2024-02-12 12:32:43 -05:00
Hoan Luu Huu
c894369a13 fix pause resume background transcribe (#586)
* fix pause resume background transcribe

* fix review comments
2024-02-12 10:38:07 -05:00
Dave Horton
565478cc0a #573 address race condition in pause/resume recording (#584) 2024-02-12 10:26:34 -05:00
Hoan Luu Huu
cdd25ca33d Fix/gather timeout (#594)
* fix gather verb timeout does not work

* wip

* wip

* wip

* wip

* fix review comments
2024-02-12 10:13:02 -05:00
Markus Frindt
ef2306e558 Improve Deepgram default modely by language (#641)
Co-authored-by: Markus Frindt <m.frindt@cognigy.com>
2024-02-12 09:53:14 -05:00
Dave Horton
9c33a790bd update to latest speech-utils (#639) 2024-02-08 15:54:45 -05:00
Dave Horton
9f9a9ec598 initial changes for deepgram on-prem (#636)
* initial changes for deepgram on-prem

* typo

* fixes for selecting deepgram model

* update some property names

* wip

* wip

* wip
2024-02-07 14:21:05 -05:00
Dave Horton
75566bb268 bump to start 0.8.6 2024-02-07 08:51:05 -05:00
Hoan Luu Huu
a55f81676b Tts/elevenlabs streaming (#629)
* update to fsmrf with fix

* changes to support elevenlabs tts streaming

* say: add vendor data to span

* bug: tts spans must include cached property

* add env for JAMBONES_USE_FREESWITCH_TIMER_FD

* fix bug in prev commit

* wip

* linting

* wip - caching files generating by streaming tts

* wip caching

* cleanup some logs

* handle tts streaming failure, write alert

* update node version dependency

* set timerfd on outbound call scenarios

* default model to nova-2-phonecall when using deepgram

---------

Co-authored-by: Dave Horton <daveh@beachdognet.com>
2024-02-07 08:49:36 -05:00
Hoan Luu Huu
48a81072e8 fix gather should not play audio if gather already resolved (#638) 2024-02-06 07:42:44 -05:00
Hoan Luu Huu
74ede31cd3 fix ws reconnect does not send verb:hook data (#633) 2024-01-31 07:20:57 -08:00
Anton Voylenko
048229f019 fix(dequeue): retrieve by callsid (#630) 2024-01-31 07:06:08 -08:00
Hoan Luu Huu
71e266ae32 Merge pull request #632 from jambonz/fix/issue_631
fix default gather input is digits and gather dtmf should not require speech
2024-01-31 12:01:36 +07:00
Quan HL
5b607693dc fix default gather input is digits and gather dtmf should not require speech 2024-01-31 11:46:29 +07:00
Dave Horton
0491c5ce25 minor logging changes 2024-01-27 12:59:23 -05:00
Vinod Dharashive
a7fa2f95dd Change regex to have fqdn and IP (#625) 2024-01-25 09:13:30 -05:00
Dave Horton
901e412343 fix bug where final transcript with finished header results in timeout (#624) 2024-01-25 08:48:22 -05:00
Dave Horton
e57c7ba90a fix for #627 (#628) 2024-01-25 08:46:57 -05:00
Hoan Luu Huu
b867395d87 fix aldulting call does not send status callback when hhangup (#623) 2024-01-23 07:12:43 -05:00
Hoan Luu Huu
1a80910f91 fix pause transcribe cannot close transcription on 2nd leg (#621) 2024-01-18 11:21:25 -05:00
Hoan Luu Huu
5d4f25622d fixed call hangup as call is await for new task and received ws command (#619)
* fixed call hangup as call is await for new task and received ws command

* wi
2024-01-18 11:12:50 -05:00
29 changed files with 3678 additions and 4320 deletions

1
.gitignore vendored
View File

@@ -42,3 +42,4 @@ ecosystem.config.js
test/credentials/*.json
run-tests.sh
run-coverage.sh
.vscode

17
.vscode/launch.json vendored
View File

@@ -1,17 +0,0 @@
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"type": "node",
"request": "launch",
"name": "Launch Program",
"program": "${workspaceFolder}/test/index.js",
"env": {
"NODE_ENV": "test"
}
}
]
}

View File

@@ -130,7 +130,9 @@ const JAMBONZ_RECORD_WS_PASSWORD = process.env.JAMBONZ_RECORD_WS_PASSWORD || pro
const JAMBONZ_DISABLE_DIAL_PAI_HEADER = process.env.JAMBONZ_DISABLE_DIAL_PAI_HEADER || false;
const JAMBONES_DISABLE_DIRECT_P2P_CALL = process.env.JAMBONES_DISABLE_DIRECT_P2P_CALL || false;
const JAMBONES_EAGERLY_PRE_CACHE_AUDIO = process.env.JAMBONES_EAGERLY_PRE_CACHE_AUDIO;
const JAMBONES_EAGERLY_PRE_CACHE_AUDIO = parseInt(process.env.JAMBONES_EAGERLY_PRE_CACHE_AUDIO, 10) || 0;
const JAMBONES_USE_FREESWITCH_TIMER_FD = process.env.JAMBONES_USE_FREESWITCH_TIMER_FD;
module.exports = {
JAMBONES_MYSQL_HOST,
@@ -213,5 +215,6 @@ module.exports = {
JAMBONZ_RECORD_WS_USERNAME,
JAMBONZ_RECORD_WS_PASSWORD,
JAMBONZ_DISABLE_DIAL_PAI_HEADER,
JAMBONES_DISABLE_DIRECT_P2P_CALL
JAMBONES_DISABLE_DIRECT_P2P_CALL,
JAMBONES_USE_FREESWITCH_TIMER_FD
};

View File

@@ -97,7 +97,7 @@ module.exports = function(srf, logger) {
if (req.has('X-MS-Teams-Tenant-FQDN')) req.locals.msTeamsTenant = req.get('X-MS-Teams-Tenant-FQDN');
if (req.has('X-Cisco-Recording-Participant')) {
const ciscoParticipants = req.get('X-Cisco-Recording-Participant');
const regex = /sip:[\d]+@[\d]+\.[\d]+\.[\d]+\.[\d]+/g;
const regex = /sip:[a-zA-Z0-9]+@[a-zA-Z0-9.-_]+/g;
const sipURIs = ciscoParticipants.match(regex);
logger.info(`X-Cisco-Recording-Participant : ${sipURIs} `);
if (sipURIs && sipURIs.length > 0) {

View File

@@ -53,16 +53,24 @@ class AdultingCallSession extends CallSession {
}
_callerHungup() {
this._hangup('caller');
}
_jambonzHangup() {
this._hangup();
}
_hangup(terminatedBy = 'jambonz') {
if (this.dlg.connectTime) {
const duration = moment().diff(this.dlg.connectTime, 'seconds');
this.rootSpan.setAttributes({'call.termination': 'hangup by caller'});
this.callInfo.callTerminationBy = 'caller';
this.rootSpan.setAttributes({'call.termination': `hangup by ${terminatedBy}`});
this.callInfo.callTerminationBy = terminatedBy;
this.emit('callStatusChange', {
callStatus: CallStatus.Completed,
duration
});
}
this.logger.info('InboundCallSession: caller hung up');
this.logger.info(`InboundCallSession: ${terminatedBy} hung up`);
this._callReleased();
this.req.removeAllListeners('cancel');
}

View File

@@ -7,7 +7,8 @@ const {
TaskName,
KillReason,
RecordState,
AllowedSipRecVerbs
AllowedSipRecVerbs,
AllowedConfirmSessionVerbs
} = require('../utils/constants');
const moment = require('moment');
const assert = require('assert');
@@ -21,7 +22,9 @@ const {
JAMBONES_INJECT_CONTENT,
JAMBONES_EAGERLY_PRE_CACHE_AUDIO,
AWS_REGION,
JAMBONES_USE_FREESWITCH_TIMER_FD
} = require('../config');
const bent = require('bent');
const BackgroundTaskManager = require('../utils/background-task-manager');
const BADPRECONDITIONS = 'preconditions not met';
const CALLER_CANCELLED_ERR_MSG = 'Response not sent due to unknown transaction';
@@ -447,6 +450,47 @@ class CallSession extends Emitter {
this._sipRequestWithinDialogHook = url;
}
// Bot Delay (actionHook delayed)
get actionHookDelayEnabled() {
return this._actionHookDelayEnabled;
}
set actionHookDelayEnabled(e) {
this._actionHookDelayEnabled = e;
}
get actionHookNoResponseTimeout() {
return this._actionHookNoResponseTimeout;
}
set actionHookNoResponseTimeout(e) {
this._actionHookNoResponseTimeout = e;
}
get actionHookNoResponseGiveUpTimeout() {
return this._actionHookNoResponseGiveUpTimeout;
}
set actionHookNoResponseGiveUpTimeout(e) {
this._actionHookNoResponseGiveUpTimeout = e;
}
get actionHookDelayRetries() {
return this._actionHookDelayRetries;
}
set actionHookDelayRetries(e) {
this._actionHookDelayRetries = e;
}
get actionHookDelayActions() {
return this._actionHookDelayActions;
}
set actionHookDelayActions(e) {
this._actionHookDelayActions = e;
}
hasGlobalSttPunctuation() {
return this._globalSttPunctuation !== undefined;
}
@@ -578,8 +622,10 @@ class CallSession extends Emitter {
const res = await this.dlg.request({
method: 'INFO',
headers: {
'X-Reason': 'pauseCallRecording'
}
'X-Reason': 'pauseCallRecording',
...(this.recordOptions.headers && {'Content-Type': 'application/json'})
},
...(this.recordOptions.headers && {body: JSON.stringify(this.recordOptions.headers) + '\n'})
});
if (res.status === 200) {
this._recordState = RecordState.RecordingPaused;
@@ -600,8 +646,10 @@ class CallSession extends Emitter {
const res = await this.dlg.request({
method: 'INFO',
headers: {
'X-Reason': 'resumeCallRecording'
}
'X-Reason': 'resumeCallRecording',
...(this.recordOptions.headers && {'Content-Type': 'application/json'})
},
...(this.recordOptions.headers && {body: JSON.stringify(this.recordOptions.headers) + '\n'})
});
if (res.status === 200) {
this._recordState = RecordState.RecordingOn;
@@ -636,7 +684,8 @@ class CallSession extends Emitter {
}
task = await this.backgroundTaskManager.newTask('bargeIn', gather);
task.sticky = autoEnable;
task.once('bargeIn-done', () => {
// listen to the bargein-done from background manager
this.backgroundTaskManager.once('bargeIn-done', () => {
if (this.requestor instanceof WsRequestor) {
try {
this.kill(true);
@@ -689,7 +738,7 @@ class CallSession extends Emitter {
(type === 'stt' && credential.use_for_stt)
)) {
this.logger.info(
`Speech vendor: ${credential.vendor} ${credential.label ? `, label: ${credential.label}` : ''} selected`);
`${type}: ${credential.vendor} ${credential.label ? `, label: ${credential.label}` : ''} `);
if ('google' === vendor) {
if (type === 'tts' && !credential.tts_tested_ok ||
type === 'stt' && !credential.stt_tested_ok) {
@@ -750,7 +799,9 @@ class CallSession extends Emitter {
else if ('deepgram' === vendor) {
return {
speech_credential_sid: credential.speech_credential_sid,
api_key: credential.api_key
api_key: credential.api_key,
deepgram_stt_uri: credential.deepgram_stt_uri,
deepgram_stt_use_tls: credential.deepgram_stt_use_tls
};
}
else if ('soniox' === vendor) {
@@ -826,12 +877,14 @@ class CallSession extends Emitter {
const taskNum = ++this.taskIdx;
const stackNum = this.stackIdx;
const task = this.tasks.shift();
task._stackNum = `${stackNum}:${taskNum}`;
this.logger.info(`CallSession:exec starting task #${stackNum}:${taskNum}: ${task.name}`);
this._notifyTaskStatus(task, {event: 'starting'});
// Register verbhook span wait for end
task.on('VerbHookSpanWaitForEnd', ({span}) => {
this.verbHookSpan = span;
});
task.on('ActionHookDelayActionOptions', this._onActionHookDelayActions.bind(this));
try {
const resources = await this._evaluatePreconditions(task);
let skip = false;
@@ -888,7 +941,6 @@ class CallSession extends Emitter {
// all done - cleanup
this.logger.info('CallSession:exec all tasks complete');
this._stopping = true;
this.disableBotMode();
this._onTasksDone();
this._clearResources();
@@ -974,6 +1026,64 @@ class CallSession extends Emitter {
}
}
/**
* perform live call control - create rest:dial
* @param {obj} opts create call options
*/
async _lccCallDial(opts) {
try {
const restDialUrl = `${this.srf.locals.serviceUrl}/v1/createCall`;
await this.transformInputIfRequired(opts);
const resp = bent('POST', 'json', 201)(restDialUrl, opts);
this.logger.info(resp.body, 'successfully create outbound call');
return resp.body;
} catch (err) {
if (err.json) {
err.body = await err.json();
}
this.logger.error(err, 'failed to create outbound call from ' + this.callSid);
this._notifyTaskError(err.body);
}
}
async transformInputIfRequired(opts) {
const {
lookupAppBySid
} = this.srf.locals.dbHelpers;
opts.account_sid = this.accountSid;
if (opts.application_sid) {
this.logger.debug(`Callsession:_validateCreateCall retrieving application ${opts.application_sid}`);
const application = await lookupAppBySid(opts.application_sid);
Object.assign(opts, {
call_hook: application.call_hook,
app_json: application.app_json,
call_status_hook: application.call_status_hook,
speech_synthesis_vendor: application.speech_synthesis_vendor,
speech_synthesis_language: application.speech_synthesis_language,
speech_synthesis_voice: application.speech_synthesis_voice,
speech_recognizer_vendor: application.speech_recognizer_vendor,
speech_recognizer_language: application.speech_recognizer_language
});
this.logger.debug({opts, application}, 'Callsession:_validateCreateCall augmented with application settings');
}
if (typeof opts.call_hook === 'string') {
const url = opts.call_hook;
opts.call_hook = {
url,
method: 'POST'
};
}
if (typeof opts.call_status_hook === 'string') {
const url = opts.call_status_hook;
opts.call_status_hook = {
url,
method: 'POST'
};
}
}
/**
* perform live call control -- set a new call_hook
* @param {object} opts
@@ -1034,12 +1144,20 @@ class CallSession extends Emitter {
const t = normalizeJambones(this.logger, tasks).map((tdata) => makeTask(this.logger, tdata));
this.logger.info({tasks: listTaskNames(t)}, 'CallSession:_lccCallHook new task list');
this.replaceApplication(t);
if (this.wakeupResolver) {
//this.logger.debug({resolution}, 'CallSession:_onCommand - got commands, waking up..');
this.wakeupResolver({reason: 'lcc: new tasks'});
this.wakeupResolver = null;
}
}
else {
/* we started a new app on the child leg, but nothing given for parent so hang him up */
this.currentTask.kill(this);
}
this._endVerbHookSpan();
// clear all delay action hook timeout if there is
this._clearActionHookNoResponseGiveUpTimer();
this._clearActionHookNoResponseTimer();
}
/**
@@ -1065,6 +1183,9 @@ class CallSession extends Emitter {
* @param {string} opts.transcribe_status - 'pause' or 'resume'
*/
async _lccTranscribeStatus(opts) {
if (this.backgroundTaskManager.isTaskRunning('transcribe')) {
this.backgroundTaskManager.getTask('transcribe').updateTranscribe(opts.transcribe_status);
}
const task = this.currentTask;
if (!task || ![TaskName.Dial, TaskName.Transcribe].includes(task.name)) {
return this.logger.info(`CallSession:_lccTranscribeStatus - invalid transcribe_status in task ${task.name}`);
@@ -1224,6 +1345,14 @@ Duration=${duration} `
task.whisper(tasks, callSid).catch((err) => this.logger.error(err, 'CallSession:_lccWhisper'));
}
/**
* perform call hangup by jambonz
*/
async hangup() {
return this._callerHungup();
}
/**
* perform live call control
@@ -1297,6 +1426,19 @@ Duration=${duration} `
tasks = pruned;
}
}
else if (this.isConfirmCallSession) {
const pruned = tasks.filter((t) => AllowedConfirmSessionVerbs.includes(t.name));
if (0 === pruned.length) {
this.logger.info({tasks},
'CallSession:replaceApplication - filtering verbs allowed on an confirmSession call');
return;
}
if (pruned.length < tasks.length) {
this.logger.info(
'CallSession:replaceApplication - removing verbs that are not allowed for confirmSession call');
tasks = pruned;
}
}
this.tasks = tasks;
this.taskIdx = 0;
this.stackIdx++;
@@ -1306,6 +1448,11 @@ Duration=${duration} `
this.currentTask.kill(this, KillReason.Replaced);
this.currentTask = null;
}
else if (this.wakeupResolver) {
this.logger.debug('CallSession:replaceApplication - waking up');
this.wakeupResolver({reason: 'new tasks'});
this.wakeupResolver = null;
}
}
kill(onBackgroundGatherBargein = false) {
@@ -1330,7 +1477,8 @@ Duration=${duration} `
this.logger.info('CallSession:kill - found bargein disabled in the stack, clearing to that point');
break;
}
this.tasks.shift();
const rem = this.tasks.shift();
this.logger.debug(`CallSession:kill - clearing task ${rem.summary}`);
}
}
else this.tasks = [];
@@ -1412,7 +1560,7 @@ Duration=${duration} `
_onCommand({msgid, command, call_sid, queueCommand, data}) {
this.logger.info({msgid, command, queueCommand, data}, 'CallSession:_onCommand - received command');
const resolution = {reason: 'received command', queue: queueCommand, command};
let resolution;
switch (command) {
case 'redirect':
if (Array.isArray(data)) {
@@ -1433,7 +1581,11 @@ Duration=${duration} `
this.tasks.push(...t);
this.logger.info({tasks: listTaskNames(this.tasks)}, 'CallSession:_onCommand - updated task list');
}
resolution = {reason: 'received command, new tasks', queue: queueCommand, command};
resolution.command = listTaskNames(t);
// clear all delay action hook timeout if there is
this._clearActionHookNoResponseGiveUpTimer();
this._clearActionHookNoResponseTimer();
}
else this._lccCallHook(data);
break;
@@ -1442,6 +1594,14 @@ Duration=${duration} `
this._lccCallStatus(data);
break;
case 'dial':
this._lccCallDial(data);
break;
case 'record':
this.notifyRecordOptions(data);
break;
case 'mute:status':
this._lccMuteStatus(call_sid, data);
break;
@@ -1483,7 +1643,7 @@ Duration=${duration} `
default:
this.logger.info(`CallSession:_onCommand - invalid command ${command}`);
}
if (this.wakeupResolver) {
if (this.wakeupResolver && resolution) {
//this.logger.debug({resolution}, 'CallSession:_onCommand - got commands, waking up..');
this.wakeupResolver(resolution);
this.wakeupResolver = null;
@@ -1666,6 +1826,8 @@ Duration=${duration} `
this.rootSpan && this.rootSpan.end();
// close all background tasks
this.backgroundTaskManager.stopAll();
this._clearActionHookNoResponseGiveUpTimer();
this._clearActionHookNoResponseTimer();
}
/**
@@ -1676,6 +1838,14 @@ Duration=${duration} `
assert(false, 'subclass responsibility to override this method');
}
/**
* called when the jambonz has hung up. Provided for subclasses to override
* in order to apply logic at this point if needed.
*/
_jambonzHangup() {
assert(false, 'subclass responsibility to override this method');
}
/**
* get a media server to use for this call
*/
@@ -2027,8 +2197,12 @@ Duration=${duration} `
}
_configMsEndpoint() {
if (this.onHoldMusic) {
this.ep.set({hold_music: `shout://${this.onHoldMusic.replace(/^https?:\/\//, '')}`});
const opts = {
...(this.onHoldMusic && {holdMusic: `shout://${this.onHoldMusic.replace(/^https?:\/\//, '')}`}),
...(JAMBONES_USE_FREESWITCH_TIMER_FD && {timer_name: 'timerfd'})
};
if (Object.keys(opts).length > 0) {
this.ep.set(opts);
}
}
@@ -2087,6 +2261,69 @@ Duration=${duration} `
this.verbHookSpan = null;
}
}
// actionHook delay actions
_onActionHookDelayActions(options) {
this._actionHookDelayRetryCount = 0;
this._startActionHookNoResponseTimer(options);
this._startActionHookNoResponseGiveUpTimer(options);
}
_startActionHookNoResponseTimer(options) {
this._clearActionHookNoResponseTimer();
if (options.noResponseTimeoutMs) {
this.logger.debug(`CallSession:_startActionHookNoResponseTimer ${options.noResponseTimeoutMs}`);
this._actionHookNoResponseTimer = setTimeout(() => {
if (this._actionHookDelayRetryCount >= options.retries) {
this._jambonzHangup();
}
const verb = options.actions[this._actionHookDelayRetryCount % options.actions.length];
// Inject verb to main stack
const t = normalizeJambones(this.logger, [verb])
.map((tdata) => makeTask(this.logger, tdata));
if (t.length) {
t[0].on('playDone', (err) => {
if (err) this.logger.error({err}, `Call-Session:exec Error delay action, play ${verb}`);
this._startActionHookNoResponseTimer(options);
});
}
this.tasks.push(...t);
if (this.wakeupResolver) {
this.wakeupResolver({reason: 'actionHook no response, applied delay actions', verb});
this.wakeupResolver = null;
}
this.logger.debug(`CallSession:_startActionHookNoResponseTimer, executing verb ${JSON.stringify(verb)}`);
this._actionHookDelayRetryCount++;
}, options.noResponseTimeoutMs);
}
}
_clearActionHookNoResponseTimer() {
if (this._actionHookNoResponseTimer) {
clearTimeout(this._actionHookNoResponseTimer);
}
this._actionHookNoResponseTimer = null;
}
_startActionHookNoResponseGiveUpTimer(options) {
this._clearActionHookNoResponseGiveUpTimer();
if (options.noResponseGiveUpTimeoutMs) {
this.logger.debug(`CallSession:_startActionHookNoResponseGiveUpTimer ${options.noResponseGiveUpTimeoutMs}`);
this._actionHookNoResponseGiveUpTimer = setTimeout(() => {
this.logger.debug('CallSession:_startActionHookNoResponseGiveUpTimer Timeout');
this._jambonzHangup();
this._actionHookNoResponseGiveUpTimer = null;
}, options.noResponseGiveUpTimeoutMs);
}
}
_clearActionHookNoResponseGiveUpTimer() {
if (this._actionHookNoResponseGiveUpTimer) {
clearTimeout(this._actionHookNoResponseGiveUpTimer);
}
this._actionHookNoResponseGiveUpTimer = null;
}
}
module.exports = CallSession;

View File

@@ -34,6 +34,9 @@ class ConfirmCallSession extends CallSession {
_callerHungup() {
}
_jambonzHangup() {
}
}

View File

@@ -67,15 +67,27 @@ class InboundCallSession extends CallSession {
* This is invoked when the caller hangs up, in order to calculate the call duration.
*/
_callerHungup() {
this._hangup('caller');
}
_jambonzHangup() {
this._hangup();
}
_hangup(terminatedBy = 'jambonz') {
if (this.dlg === null) {
this.logger.info('InboundCallSession:_hangup - race condition, dlg cleared by app hangup');
return;
}
assert(this.dlg.connectTime);
const duration = moment().diff(this.dlg.connectTime, 'seconds');
this.rootSpan.setAttributes({'call.termination': 'hangup by caller'});
this.callInfo.callTerminationBy = 'caller';
this.rootSpan.setAttributes({'call.termination': `hangup by ${terminatedBy}`});
this.callInfo.callTerminationBy = terminatedBy;
this.emit('callStatusChange', {
callStatus: CallStatus.Completed,
duration
});
this.logger.info('InboundCallSession: caller hung up');
this.logger.info(`InboundCallSession: ${terminatedBy} hung up`);
this._callReleased();
this.req.removeAllListeners('cancel');
}

View File

@@ -49,13 +49,21 @@ class RestCallSession extends CallSession {
* This is invoked when the called party hangs up, in order to calculate the call duration.
*/
_callerHungup() {
this._hangup('caller');
}
_jambonzHangup() {
this._hangup();
}
_hangup(terminatedBy = 'jamboz') {
if (this.restDialTask) {
this.restDialTask.turnOffAmd();
}
this.callInfo.callTerminationBy = 'caller';
this.callInfo.callTerminationBy = terminatedBy;
const duration = moment().diff(this.dlg.connectTime, 'seconds');
this.emit('callStatusChange', {callStatus: CallStatus.Completed, duration});
this.logger.debug('RestCallSession: called party hung up');
this.logger.debug(`RestCallSession: called party hung up by ${terminatedBy}`);
this._callReleased();
}

View File

@@ -10,7 +10,8 @@ class TaskConfig extends Task {
'bargeIn',
'record',
'listen',
'transcribe'
'transcribe',
'actionHookDelayAction'
].forEach((k) => this[k] = this.data[k] || {});
if ('notifyEvents' in this.data) {
@@ -249,6 +250,14 @@ class TaskConfig extends Task {
cs.stopBackgroundTask('transcribe');
}
}
if (this.actionHookDelayAction) {
cs.actionHookDelayEnabled = this.actionHookDelayAction.enabled || false;
cs.actionHookNoResponseTimeout = this.actionHookDelayAction.noResponseTimeout || 0;
cs.actionHookNoResponseGiveUpTimeout = this.actionHookDelayAction.noResponseGiveUpTimeout || 0;
cs.actionHookDelayRetries = this.actionHookDelayAction.retries || 1;
cs.actionHookDelayActions = this.actionHookDelayAction.actions || [];
}
if (this.data.sipRequestWithinDialogHook) {
cs.sipRequestWithinDialogHook = this.data.sipRequestWithinDialogHook;
}

View File

@@ -73,7 +73,8 @@ class TaskDequeue extends Task {
try {
let url;
if (this.callSid) {
url = await retrieveByPatternSortedSet(this.queueName, `*${this.callSid}`);
const r = await retrieveByPatternSortedSet(this.queueName, `*${this.callSid}`);
url = r[0];
} else {
url = await retrieveFromSortedSet(this.queueName);
}

View File

@@ -100,6 +100,7 @@ class TaskDial extends Task {
this.referHook = this.data.referHook;
this.dtmfHook = this.data.dtmfHook;
this.proxy = this.data.proxy;
this.tag = this.data.tag;
if (this.dtmfHook) {
const {parentDtmfCollector, childDtmfCollector} = parseDtmfOptions(logger, this.data.dtmfCapture || {});

View File

@@ -27,9 +27,13 @@ class TaskGather extends SttTask {
[
'finishOnKey', 'input', 'numDigits', 'minDigits', 'maxDigits',
'interDigitTimeout', 'partialResultHook', 'bargein', 'dtmfBargein',
'speechTimeout', 'timeout', 'say', 'play'
'speechTimeout', 'timeout', 'say', 'play', 'actionHookDelayAction'
].forEach((k) => this[k] = this.data[k]);
// gather default input is digits
if (!this.input) {
this.input = ['digits'];
}
/* when collecting dtmf, bargein on dtmf is true unless explicitly set to false */
if (this.dtmfBargein !== false && this.input.includes('digits')) this.dtmfBargein = true;
@@ -103,7 +107,7 @@ class TaskGather extends SttTask {
}
async exec(cs, {ep}) {
this.logger.debug({options: this.data}, 'Gather:exec');
this.logger.debug({options: this.data}, `Gather:exec ${this.stackNum}`);
await super.exec(cs, {ep});
const {updateSpeechCredentialLastUsed} = require('../utils/db-utils')(this.logger, cs.srf);
@@ -134,6 +138,29 @@ class TaskGather extends SttTask {
this.interim = true;
this.logger.debug('Gather:exec - early hints match enabled');
}
// actionHook delay
this._hookDelayEn = cs.actionHookDelayEnabled || !!this.actionHookDelayAction;
this._hookDelayActions = this.actionHookDelayAction?.actions || cs.actionHookDelayActions || [];
// Only enable NoResponseTimeout if there is _hookDelayActions
this._hookNoResponseTimeout = (this._hookDelayActions?.length ?
(this.actionHookDelayAction?.noResponseTimeout || cs.actionHookNoResponseTimeout || 0)
: 0) * 1000;
this._hookNoResponseGiveUpTimeout = (this.actionHookDelayAction?.noResponseGiveUpTimeout ||
cs.actionHookNoResponseGiveUpTimeout || 0) * 1000;
this._hookDelayRetries = this.actionHookDelayAction?.retries || cs.actionHookDelayRetries || 1;
this._hookDelayRetryCount = 0;
this.hookDelayActionOpts = {
enabled: this._hookDelayEn,
actions: this._hookDelayActions,
noResponseTimeoutMs: this._hookNoResponseTimeout,
noResponseGiveUpTimeoutMs: this._hookNoResponseGiveUpTimeout,
retries: this._hookDelayRetries
};
const startListening = async(cs, ep) => {
this._startTimer();
if (this.isContinuousAsr && 0 === this.timeout) this._startAsrTimer();
@@ -144,6 +171,7 @@ class TaskGather extends SttTask {
this.logger.info('Gather:exec - task was quickly killed so do not transcribe');
return;
}
this.logger.debug('Gather:exec - going to start transcribing (startListening)');
this._startTranscribing(ep);
return updateSpeechCredentialLastUsed(this.sttCredentials.speech_credential_sid);
} catch (e) {
@@ -208,9 +236,15 @@ class TaskGather extends SttTask {
if (this.input.includes('speech') && this.listenDuringPrompt) {
await this._setSpeechHandlers(cs, ep);
this._startTranscribing(ep);
updateSpeechCredentialLastUsed(this.sttCredentials.speech_credential_sid)
.catch(() => {/*already logged error */});
if (!this.resolved && !this.killed) {
this.logger.debug(`Gather:exec ${this.stackNum} - going to start transcribing (listenDuringPrompt)`);
this._startTranscribing(ep);
updateSpeechCredentialLastUsed(this.sttCredentials.speech_credential_sid)
.catch(() => {/*already logged error */});
}
else {
this.logger.info(`Gather:exec ${this.stackNum} - task was killed or resolved before starting transcription`);
}
}
if (this.input.includes('digits') || this.dtmfBargein || this.asrDtmfTerminationDigit) {
@@ -227,6 +261,7 @@ class TaskGather extends SttTask {
kill(cs) {
super.kill(cs);
this._killAudio(cs);
this._killActionHookDelayAction();
this.ep.removeAllListeners('dtmf');
clearTimeout(this.interDigitTimer);
this._clearAsrTimer();
@@ -351,9 +386,6 @@ class TaskGather extends SttTask {
this.addCustomEventListener(ep, DeepgramTranscriptionEvents.Connect, this._onVendorConnect.bind(this, cs, ep));
this.addCustomEventListener(ep, DeepgramTranscriptionEvents.ConnectFailure,
this._onVendorConnectFailure.bind(this, cs, ep));
/* if app sets deepgramOptions.utteranceEndMs they essentially want continuous asr */
if (opts.DEEPGRAM_SPEECH_UTTERANCE_END_MS) this.isContinuousAsr = true;
break;
case 'soniox':
@@ -457,7 +489,7 @@ class TaskGather extends SttTask {
locale: this.language,
interim: this.interim,
bugname: this.bugname
}, 'Gather:_startTranscribing');
}, `Gather:_startTranscribing ${this.stackNum}`);
/**
* Note: we don't need to ask deepgram for interim results, because they
@@ -487,9 +519,8 @@ class TaskGather extends SttTask {
this._clearTimer();
this._timeoutTimer = setTimeout(() => {
if (this.isContinuousAsr) this._startAsrTimer();
else if (this.interDigitTimeout <= 0 ||
this.digitBuffer.length < this.minDigits ||
this.needsStt && this.digitBuffer.length === 0) {
if (this.interDigitTimer) return; // let the inter-digit timer complete
else {
this._resolve(this.digitBuffer.length >= this.minDigits ? 'dtmf-num-digits' : 'timeout');
}
}, this.timeout);
@@ -499,7 +530,9 @@ class TaskGather extends SttTask {
if (this._timeoutTimer) {
clearTimeout(this._timeoutTimer);
this._timeoutTimer = null;
return true;
}
return false;
}
_startAsrTimer() {
@@ -508,7 +541,7 @@ class TaskGather extends SttTask {
this._clearAsrTimer();
this._asrTimer = setTimeout(() => {
this.logger.debug('_startAsrTimer - asr timer went off');
const evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language);
const evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language, this.vendor);
this._resolve(this._bufferedTranscripts.length > 0 ? 'speech' : 'timeout', evt);
}, this.asrTimeout);
this.logger.debug(`_startAsrTimer: set for ${this.asrTimeout}ms`);
@@ -519,6 +552,104 @@ class TaskGather extends SttTask {
this._asrTimer = null;
}
_hangupCall() {
this.logger.debug('_hangupCall');
this.cs.hangup();
}
_actionHookDelaySayAction(verb) {
delete verb.verb;
this.logger.debug(`_actionHookDelaySayAction ${verb}`);
this._actionHookDelaySayTask = makeTask(this.logger, {say: verb}, this);
const {span, ctx} = this.startChildSpan(`actionHookDelayAction:${this._actionHookDelaySayTask.summary}`);
this._actionHookDelaySayTask.span = span;
this._actionHookDelaySayTask.ctx = ctx;
this._actionHookDelaySayTask.exec(this.cs, {ep: this.ep});
this._actionHookDelaySayTask.on('playDone', (err) => {
this._actionHookDelaySayTask = null;
span.end();
if (err) this.logger.error({err}, 'Gather:actionHookDelay Error playing tts');
});
}
_killActionHookDelayAction() {
this.logger.debug('_killActionHookDelayAction');
if (this._actionHookDelaySayTask && !this._actionHookDelaySayTask.killed) {
this._actionHookDelaySayTask.removeAllListeners('playDone');
this._actionHookDelaySayTask.kill(this.cs);
this._actionHookDelaySayTask.span.end();
this._actionHookDelaySayTask = null;
}
if (this._actionHookDelayPlayTask && !this._actionHookDelayPlayTask.killed) {
this._actionHookDelayPlayTask.removeAllListeners('playDone');
this._actionHookDelayPlayTask.kill(this.cs);
this._actionHookDelayPlayTask.span.end();
this._actionHookDelayPlayTask = null;
}
}
_actionHookDelayPlayAction(verb) {
delete verb.verb;
this.logger.debug(`_actionHookDelayPlayAction ${verb}`);
this._actionHookDelayPlayTask = makeTask(this.logger, {play: verb}, this);
const {span, ctx} = this.startChildSpan(`actionHookDelayAction:${this._actionHookDelayPlayTask.summary}`);
this._actionHookDelayPlayTask.span = span;
this._actionHookDelayPlayTask.ctx = ctx;
this._actionHookDelayPlayTask.exec(this.cs, {ep: this.ep});
this._actionHookDelayPlayTask.on('playDone', (err) => {
this._actionHookDelayPlayTask = null;
span.end();
if (err) this.logger.error({err}, 'Gather:actionHookDelay Error playing tts');
});
}
_startActionHookNoResponseTimer() {
assert(this._hookNoResponseTimeout > 0);
this._clearActionHookNoResponseTimer();
this.logger.debug('startActionHookNoResponseTimer');
this._actionHookNoResponseTimer = setTimeout(() => {
if (this._hookDelayRetryCount >= this._hookDelayRetries) {
this._hangupCall();
return;
}
const verb = this._hookDelayActions[this._hookDelayRetryCount % this._hookDelayActions.length];
if (verb.verb === 'say') {
this._actionHookDelaySayAction(verb);
} else if (verb.verb === 'play') {
this._actionHookDelayPlayAction(verb);
}
this._hookDelayRetryCount++;
this._startActionHookNoResponseTimer();
}, this._hookNoResponseTimeout);
}
_clearActionHookNoResponseTimer() {
if (this._actionHookNoResponseTimer) {
clearTimeout(this._actionHookNoResponseTimer);
}
this._actionHookNoResponseTimer = null;
}
_startActionHookNoResponseGiveUpTimer() {
assert(this._hookNoResponseGiveUpTimeout > 0);
this._clearActionHookNoResponseGiveUpTimer();
this.logger.debug('startActionHookNoResponseGiveUpTimer');
this._actionHookNoResponseGiveUpTimer = setTimeout(() => {
this._hangupCall();
}, this._hookNoResponseGiveUpTimeout);
}
_clearActionHookNoResponseGiveUpTimer() {
if (this._actionHookNoResponseGiveUpTimer) {
clearTimeout(this._actionHookNoResponseGiveUpTimer);
}
this._actionHookNoResponseGiveUpTimer = null;
}
_startFastRecognitionTimer(evt) {
assert(this.fastRecognitionTimeout > 0);
this._clearFastRecognitionTimer();
@@ -539,7 +670,7 @@ class TaskGather extends SttTask {
this._clearFinalAsrTimer();
this._finalAsrTimer = setTimeout(() => {
this.logger.debug('_startFinalAsrTimer - final asr timer went off');
const evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language);
const evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language, this.vendor);
this._resolve(this._bufferedTranscripts.length > 0 ? 'speech' : 'timeout', evt);
}, 1000);
this.logger.debug('_startFinalAsrTimer: set for 1 second');
@@ -576,11 +707,15 @@ class TaskGather extends SttTask {
// make sure this is not a transcript from answering machine detection
const bugname = fsEvent.getHeader('media-bugname');
const finished = fsEvent.getHeader('transcription-session-finished');
this.logger.debug({evt, bugname, finished}, `Gather:_onTranscription for vendor ${this.vendor}`);
this.logger.debug({
evt,
bugname,
finished,
vendor: this.vendor
}, `Gather:_onTranscription ${this.stackNum} raw transcript`);
if (bugname && this.bugname !== bugname) return;
if (this.vendor === 'ibm' && evt?.state === 'listening') return;
if (this.vendor === 'deepgram' && evt.type === 'UtteranceEnd') {
/* we will only get this when we have set utterance_end_ms */
if (this._bufferedTranscripts.length === 0) {
@@ -588,15 +723,21 @@ class TaskGather extends SttTask {
}
else {
this.logger.debug('Gather:_onTranscription - got UtteranceEnd event from deepgram, return buffered transcript');
evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language);
evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language, this.vendor);
this._bufferedTranscripts = [];
this._resolve('speech', evt);
}
return;
}
if (this.vendor === 'deepgram' && evt.type === 'Metadata') {
this.logger.debug('Gather:_onTranscription - discarding Metadata event from deepgram');
return;
}
evt = this.normalizeTranscription(evt, this.vendor, 1, this.language,
this.shortUtterance, this.data.recognizer.punctuation);
//this.logger.debug({evt, bugname, finished, vendor: this.vendor}, 'Gather:_onTranscription normalized transcript');
if (evt.alternatives.length === 0) {
this.logger.info({evt}, 'TaskGather:_onTranscription - got empty transcript, continue listening');
return;
@@ -622,7 +763,9 @@ class TaskGather extends SttTask {
if (evt.is_final) {
if (evt.alternatives[0].transcript === '' && !this.callSession.callGone && !this.killed) {
emptyTranscript = true;
if (finished === 'true' && ['microsoft', 'deepgram'].includes(this.vendor)) {
if (finished === 'true' &&
['microsoft', 'deepgram'].includes(this.vendor) &&
this._bufferedTranscripts.length === 0) {
this.logger.debug({evt}, 'TaskGather:_onTranscription - got empty transcript from old gather, disregarding');
return;
}
@@ -656,20 +799,27 @@ class TaskGather extends SttTask {
this._clearTimer();
if (this._finalAsrTimer) {
this._clearFinalAsrTimer();
const evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language);
const evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language, this.vendor);
return this._resolve(this._bufferedTranscripts.length > 0 ? 'speech' : 'timeout', evt);
}
this._startAsrTimer();
/* some STT engines will keep listening after a final response, so no need to restart */
if (!['soniox', 'aws', 'microsoft', 'deepgram'].includes(this.vendor)) this._startTranscribing(ep);
if (!['soniox', 'aws', 'microsoft', 'deepgram'].includes(this.vendor)) {
this.logger.debug('Gather:_onTranscription - going to start transcribing again (continuous asr)');
this._startTranscribing(ep);
}
}
else {
if (this.bargein && (words + bufferedWords) < this.minBargeinWordCount) {
this.logger.debug({evt, words, bufferedWords},
'TaskGather:_onTranscription - final transcript but < min barge words');
this._bufferedTranscripts.push(evt);
this._startTranscribing(ep);
if (!['soniox', 'aws', 'microsoft', 'deepgram'].includes(this.vendor)) {
this.logger.debug(
`Gather:_onTranscription - start transcribing again (min bargein words=${this.minBargeinWordCount}`);
this._startTranscribing(ep);
}
return;
}
else {
@@ -682,17 +832,11 @@ class TaskGather extends SttTask {
else if (this.vendor === 'deepgram') {
/* compile transcripts into one */
if (!emptyTranscript) this._bufferedTranscripts.push(evt);
if (this.data.recognizer?.deepgramOptions?.utteranceEndMs) {
this.logger.debug('TaskGather:_onTranscription - got speech_final waiting for UtteranceEnd event');
return;
}
/* deepgram can send an empty and final transcript; only if we have any buffered should we resolve */
if (this._bufferedTranscripts.length === 0) return;
this.logger.debug({evt}, 'TaskGather:_onTranscription - compiling deepgram transcripts');
evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language);
evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language, this.vendor);
this._bufferedTranscripts = [];
this.logger.debug({evt}, 'TaskGather:_onTranscription - compiled deepgram transcripts');
}
/* here is where we return a final transcript */
@@ -701,14 +845,26 @@ class TaskGather extends SttTask {
}
}
else {
this._clearTimer();
this._startTimer();
if (this.bargein && (words + bufferedWords) >= this.minBargeinWordCount) {
if (!this.playComplete) {
this.logger.debug({transcript: evt.alternatives[0].transcript}, 'killing audio due to speech');
this.emit('vad');
/* deepgram can send a non-final transcript but with words that are final, so we need to buffer */
let emptyTranscript = false;
if (this.vendor === 'deepgram') {
const originalEvent = evt.vendor.evt;
if (originalEvent.is_final && evt.alternatives[0].transcript !== '') {
this.logger.debug({evt}, 'Gather:_onTranscription - buffering a completed (partial) deepgram transcript');
this._bufferedTranscripts.push(evt);
}
if (evt.alternatives[0].transcript === '') emptyTranscript = true;
}
if (!emptyTranscript) {
if (this._clearTimer()) this._startTimer();
if (this.bargein && (words + bufferedWords) >= this.minBargeinWordCount) {
if (!this.playComplete) {
this.logger.debug({transcript: evt.alternatives[0].transcript}, 'killing audio due to speech');
this.emit('vad');
}
this._killAudio(cs);
}
this._killAudio(cs);
}
if (this.fastRecognitionTimeout) {
this._startFastRecognitionTimer(evt);
@@ -726,18 +882,10 @@ class TaskGather extends SttTask {
this._sonioxTranscripts.push(evt.vendor.finalWords);
}
}
/* deepgram can send a non-final transcript but with words that are final, so we need to buffer */
if (this.vendor === 'deepgram') {
const originalEvent = evt.vendor.evt;
if (originalEvent.is_final && evt.alternatives[0].transcript !== '') {
this.logger.debug({evt}, 'Gather:_onTranscription - buffering a completed (partial) deepgram transcript');
this._bufferedTranscripts.push(evt);
}
}
}
}
_onEndOfUtterance(cs, ep) {
this.logger.debug('TaskGather:_onEndOfUtterance');
this.logger.debug(`TaskGather:_onEndOfUtterance ${this.stackNum}`);
if (this.bargein && this.minBargeinWordCount === 0) {
this._killAudio(cs);
}
@@ -752,6 +900,7 @@ class TaskGather extends SttTask {
* since we dont have a final transcript yet.
*/
if (!this.resolved && !this.killed && !this._bufferedTranscripts.length && this.wantsSingleUtterance) {
this.logger.debug('Gather:_onEndOfUtterance - start transcribing again (end of utterance/wantsSingleUtterance)');
this._startTranscribing(ep);
}
}
@@ -777,6 +926,7 @@ class TaskGather extends SttTask {
try {
await this._fallback();
await this._initSpeech(cs, ep);
this.logger.debug('Gather:_onJambonzError - going to start transcribing again (jambonz error)');
this._startTranscribing(ep);
updateSpeechCredentialLastUsed(this.sttCredentials.speech_credential_sid);
return;
@@ -826,7 +976,7 @@ class TaskGather extends SttTask {
this.logger.debug('TaskGather:_onNoSpeechDetected for old gather, ignoring');
}
else {
this.logger.debug('TaskGather:_onNoSpeechDetected - listen again');
this.logger.debug('Gather:_onNoSpeechDetected - going to start transcribing again');
this._startTranscribing(ep);
}
return;
@@ -834,14 +984,25 @@ class TaskGather extends SttTask {
}
async _resolve(reason, evt) {
this.logger.debug(`TaskGather:resolve with reason ${reason}`);
this.logger.debug(`TaskGather:resolve ${this.stackNum} with reason ${reason}`);
if (this.needsStt && this.ep && this.ep.connected) {
this.ep.stopTranscription({
vendor: this.vendor,
bugname: this.bugname
})
.catch((err) => {
if (this.resolved) return;
this.logger.error({err}, 'Error stopping transcription');
});
}
if (this.resolved) return;
this.resolved = true;
// Clear dtmf event
if (this.dtmfBargein) {
this.ep.removeAllListeners('dtmf');
}
// If bargin is false and ws application return ack to verb:hook
// the gather should not play any audio
this._killAudio(this.cs);
// Clear dtmf events, to avoid any case can leak the listener, just clean it
this.ep.removeAllListeners('dtmf');
clearTimeout(this.interDigitTimer);
this._clearTimer();
this._clearFastRecognitionTimer();
@@ -851,13 +1012,6 @@ class TaskGather extends SttTask {
'stt.resolve': reason,
'stt.result': JSON.stringify(evt)
});
if (this.needsStt && this.ep && this.ep.connected) {
this.ep.stopTranscription({
vendor: this.vendor,
bugname: this.bugname
})
.catch((err) => this.logger.error({err}, 'Error stopping transcription'));
}
if (this.callSession && this.callSession.callGone) {
this.logger.debug('TaskGather:_resolve - call is gone, not invoking web callback');
@@ -865,6 +1019,15 @@ class TaskGather extends SttTask {
return;
}
// Enabled action Hook delay timer to applied actions
if (this._hookNoResponseTimeout > 0) {
this._startActionHookNoResponseTimer();
}
if (this._hookNoResponseGiveUpTimeout > 0) {
this._startActionHookNoResponseGiveUpTimer();
}
try {
if (reason.startsWith('dtmf')) {
if (this.parentTask) this.parentTask.emit('dtmf', evt);
@@ -895,6 +1058,11 @@ class TaskGather extends SttTask {
}
}
} catch (err) { /*already logged error*/ }
// Gather got response from hook, cancel all delay timers if there is any
this._clearActionHookNoResponseTimer();
this._clearActionHookNoResponseGiveUpTimer();
this.notifyTaskDone();
}
}

View File

@@ -123,8 +123,6 @@ class TaskListen extends Task {
ci,
this.metadata);
if (this.hook.auth) {
this.logger.debug({username: this.hook.auth.username, password: this.hook.auth.password},
'TaskListen:_startListening basic auth');
await this.ep.set({
'MOD_AUDIO_BASIC_AUTH_USERNAME': this.hook.auth.username,
'MOD_AUDIO_BASIC_AUTH_PASSWORD': this.hook.auth.password

View File

@@ -23,6 +23,12 @@ const breakLengthyTextIfNeeded = (logger, text) => {
}
};
const parseTextFromSayString = (text) => {
const closingBraceIndex = text.indexOf('}');
if (closingBraceIndex === -1) return text;
return text.slice(closingBraceIndex + 1);
};
class TaskSay extends Task {
constructor(logger, opts, parentTask) {
super(logger, opts);
@@ -60,7 +66,7 @@ class TaskSay extends Task {
}
async _synthesizeWithSpecificVendor(cs, ep, {vendor, language, voice, label, preCache = false}) {
const {srf} = cs;
const {srf, accountSid:account_sid} = cs;
const {updateSpeechCredentialLastUsed} = require('../utils/db-utils')(this.logger, srf);
const {writeAlerts, AlertType, stats} = srf.locals;
const {synthAudio} = srf.locals.dbHelpers;
@@ -97,11 +103,17 @@ class TaskSay extends Task {
voice = this.options.voice_id || voice;
}
ep.set({
tts_engine: vendor,
tts_voice: voice,
cache_speech_handles: 1,
}).catch((err) => this.logger.info({err}, 'Error setting tts_engine on endpoint'));
if (!preCache) this.logger.info({vendor, language, voice, model}, 'TaskSay:exec');
try {
if (!credentials) {
writeAlerts({
account_sid: cs.accountSid,
account_sid,
alert_type: AlertType.TTS_NOT_PROVISIONED,
vendor
}).catch((err) => this.logger.info({err}, 'Error generating alert for no tts'));
@@ -120,18 +132,17 @@ class TaskSay extends Task {
if (text.startsWith('silence_stream://')) return text;
/* otel: trace time for tts */
let otelSpan;
if (!preCache) {
const {span} = this.startChildSpan('tts-generation', {
'tts.vendor': vendor,
'tts.language': language,
'tts.voice': voice
});
otelSpan = span;
this.otelSpan = span;
}
try {
const {filePath, servedFromCache, rtt} = await synthAudio(stats, {
account_sid: cs.accountSid,
account_sid,
text,
vendor,
language,
@@ -141,30 +152,40 @@ class TaskSay extends Task {
salt,
credentials,
options: this.options,
disableTtsCache : this.disableTtsCache
disableTtsCache : this.disableTtsCache,
preCache
});
this.logger.debug(`file ${filePath}, served from cache ${servedFromCache}`);
if (filePath) cs.trackTmpFile(filePath);
if (!servedFromCache && !lastUpdated) {
lastUpdated = true;
updateSpeechCredentialLastUsed(credentials.speech_credential_sid)
.catch(() => {/*already logged error */});
if (!filePath.startsWith('say:')) {
this.logger.debug(`file ${filePath}, served from cache ${servedFromCache}`);
if (filePath) cs.trackTmpFile(filePath);
if (this.otelSpan) {
this.otelSpan.setAttributes({'tts.cached': servedFromCache});
this.otelSpan.end();
this.otelSpan = null;
}
if (!servedFromCache && !lastUpdated) {
lastUpdated = true;
updateSpeechCredentialLastUsed(credentials.speech_credential_sid).catch(() => {/* logged error */});
}
if (!servedFromCache && rtt && !preCache) {
this.notifyStatus({
event: 'synthesized-audio',
vendor,
language,
characters: text.length,
elapsedTime: rtt
});
}
}
if (otelSpan) otelSpan.setAttributes({'tts.cached': servedFromCache});
if (otelSpan) otelSpan.end();
if (!servedFromCache && rtt && !preCache) {
this.notifyStatus({
event: 'synthesized-audio',
vendor,
language,
characters: text.length,
elapsedTime: rtt
});
else {
this.logger.debug('a streaming tts api will be used');
const modifiedPath = filePath.replace('say:{', `say:{session-uuid=${ep.uuid},`);
return modifiedPath;
}
return filePath;
} catch (err) {
this.logger.info({err}, 'Error synthesizing tts');
if (otelSpan) otelSpan.end();
if (this.otelSpan) this.otelSpan.end();
writeAlerts({
account_sid: cs.accountSid,
alert_type: AlertType.TTS_FAILURE,
@@ -186,6 +207,11 @@ class TaskSay extends Task {
}
async exec(cs, {ep}) {
const {srf, accountSid:account_sid} = cs;
const {writeAlerts, AlertType} = srf.locals;
const {addFileToCache} = srf.locals.dbHelpers;
const engine = this.synthesizer.engine || 'standard';
await super.exec(cs);
this.ep = ep;
@@ -235,16 +261,55 @@ class TaskSay extends Task {
}
this.notifyStatus({event: 'start-playback'});
while (!this.killed && (this.loop === 'forever' || this.loop--) && this.ep?.connected) {
while (!this.killed && (this.loop === 'forever' || this.loop--) && ep?.connected) {
let segment = 0;
while (!this.killed && segment < filepath.length) {
if (cs.isInConference) {
const {memberId, confName, confUuid} = cs;
await this.playToConfMember(this.ep, memberId, confName, confUuid, filepath[segment]);
await this.playToConfMember(ep, memberId, confName, confUuid, filepath[segment]);
}
else {
this.logger.debug(`Say:exec sending command to play file ${filepath[segment]}`);
if (filepath[segment].startsWith('say:{')) {
const arr = /^say:\{.*\}\s*(.*)$/.exec(filepath[segment]);
if (arr) this.logger.debug(`Say:exec sending streaming tts request: ${arr[1].substring(0, 64)}..`);
}
else this.logger.debug(`Say:exec sending ${filepath[segment].substring(0, 64)}`);
ep.once('playback-start', (evt) => {
this.logger.debug({evt}, 'got playback-start');
if (this.otelSpan) {
this._addStreamingTtsAttributes(this.otelSpan, evt);
this.otelSpan.end();
this.otelSpan = null;
if (evt.variable_tts_cache_filename) cs.trackTmpFile(evt.variable_tts_cache_filename);
}
});
ep.once('playback-stop', (evt) => {
this.logger.debug({evt}, 'got playback-stop');
if (evt.variable_tts_error) {
writeAlerts({
account_sid,
alert_type: AlertType.TTS_FAILURE,
vendor,
detail: evt.variable_tts_error
}).catch((err) => this.logger.info({err}, 'Error generating alert for no tts'));
}
if (evt.variable_tts_cache_filename) {
const text = parseTextFromSayString(this.text[segment]);
addFileToCache(evt.variable_tts_cache_filename, {
account_sid,
vendor,
language,
voice,
engine,
text
}).catch((err) => this.logger.info({err}, 'Error adding file to cache'));
}
});
await ep.play(filepath[segment]);
if (filepath[segment].startsWith('say:{')) {
const arr = /^say:\{.*\}\s*(.*)$/.exec(filepath[segment]);
if (arr) this.logger.debug(`Say:exec complete playing streaming tts request: ${arr[1].substring(0, 64)}..`);
}
this.logger.debug(`Say:exec completed play file ${filepath[segment]}`);
}
segment++;
@@ -255,7 +320,7 @@ class TaskSay extends Task {
async kill(cs) {
super.kill(cs);
if (this.ep.connected) {
if (this.ep?.connected) {
this.logger.debug('TaskSay:kill - killing audio');
if (cs.isInConference) {
const {memberId, confName} = cs;
@@ -265,8 +330,40 @@ class TaskSay extends Task {
this.notifyStatus({event: 'kill-playback'});
this.ep.api('uuid_break', this.ep.uuid);
}
this.ep.removeAllListeners('playback-start');
this.ep.removeAllListeners('playback-stop');
}
}
_addStreamingTtsAttributes(span, evt) {
const attrs = {'tts.cached': false};
for (const [key, value] of Object.entries(evt)) {
if (key.startsWith('variable_tts_')) {
let newKey = key.substring('variable_tts_'.length)
.replace('whisper_', 'whisper.')
.replace('elevenlabs_', 'elevenlabs.');
if (spanMapping[newKey]) newKey = spanMapping[newKey];
attrs[newKey] = value;
}
}
delete attrs['cache_filename']; //no value in adding this to the span
span.setAttributes(attrs);
}
}
const spanMapping = {
'elevenlabs.reported_latency_ms': 'elevenlabs.latency_ms',
'elevenlabs.request_id': 'elevenlabs.req_id',
'elevenlabs.history_item_id': 'elevenlabs.item_id',
'elevenlabs.optimize_streaming_latency': 'elevenlabs.optimization',
'elevenlabs.name_lookup_time_ms': 'name_lookup_ms',
'elevenlabs.connect_time_ms': 'connect_ms',
'elevenlabs.final_response_time_ms': 'final_response_ms',
'whisper.reported_latency_ms': 'whisper.latency_ms',
'whisper.request_id': 'whisper.req_id',
'whisper.name_lookup_time_ms': 'name_lookup_ms',
'whisper.connect_time_ms': 'connect_ms',
'whisper.final_response_time_ms': 'final_response_ms',
};
module.exports = TaskSay;

View File

@@ -110,7 +110,12 @@ class SttTask extends Task {
this.data.recognizer.model = cs.speechRecognizerLanguage;
}
if (!this.sttCredentials) {
if (
// not gather task, such as transcribe
(!this.input ||
// gather task with speech
this.input.includes('speech')) &&
!this.sttCredentials) {
try {
this.sttCredentials = await this._initSpeechCredentials(this.cs, this.vendor, this.label);
} catch (error) {

View File

@@ -160,6 +160,7 @@ class Task extends Emitter {
const httpHeaders = b3 && {b3};
span.setAttributes({'http.body': JSON.stringify(params)});
try {
if (this.id) params.verb_id = this.id;
const json = await this.cs.requestor.request(type, this.actionHook, params, httpHeaders);
span.setAttributes({'http.statusCode': 200});
const isWsConnection = this.cs.requestor instanceof WsRequestor;
@@ -172,12 +173,18 @@ class Task extends Emitter {
* first new set of verbs arrive after sending a transcript
* */
this.emit('VerbHookSpanWaitForEnd', {span});
// If actionHook delay action is configured, and ws application have not responded yet any verb for actionHook
// We have to transfer the task to call-session to await on next ws command verbs, and also run action Hook
// delay actions
if (this.hookDelayActionOpts) {
this.emit('ActionHookDelayActionOptions', this.hookDelayActionOpts);
}
}
if (expectResponse && json && Array.isArray(json)) {
const makeTask = require('./make_task');
const tasks = normalizeJambones(this.logger, json).map((tdata) => makeTask(this.logger, tdata));
if (tasks && tasks.length > 0) {
this.logger.info({tasks: tasks}, `${this.name} replacing application with ${tasks.length} tasks`);
this.callSession.replaceApplication(tasks);
}
}

View File

@@ -31,6 +31,11 @@ class TaskTranscribe extends SttTask {
this.separateRecognitionPerChannel = this.data.recognizer.separateRecognitionPerChannel;
}
/* for nested transcribe in dial, unless the app explicitly says so we want to transcribe both legs */
if (this.parentTask?.name === TaskName.Dial && this.separateRecognitionPerChannel !== false) {
this.separateRecognitionPerChannel = true;
}
this.childSpan = [null, null];
// Continuous asr timeout
@@ -41,6 +46,7 @@ class TaskTranscribe extends SttTask {
/* buffer speech for continuous asr */
this._bufferedTranscripts = [];
this.bugname_prefix = 'transcribe_';
this.paused = false;
}
get name() { return TaskName.Transcribe; }
@@ -95,7 +101,7 @@ class TaskTranscribe extends SttTask {
}
if (this.separateRecognitionPerChannel && this.ep2 && this.ep2.connected) {
stopTranscription = true;
this.ep2.stopTranscription({vendor: this.vendor})
this.ep2.stopTranscription({vendor: this.vendor, bugname: this.bugname})
.catch((err) => this.logger.info(err, 'Error TaskTranscribe:kill'));
}
@@ -117,9 +123,11 @@ class TaskTranscribe extends SttTask {
this.logger.info(`TaskTranscribe:updateTranscribe status ${status}`);
switch (status) {
case TranscribeStatus.Pause:
this.paused = true;
await this._stopTranscription();
break;
case TranscribeStatus.Resume:
this.paused = false;
await this._startTranscribing(this.cs, this.ep, 1);
if (this.separateRecognitionPerChannel && this.ep2) {
await this._startTranscribing(this.cs, this.ep2, 2);
@@ -295,7 +303,11 @@ class TaskTranscribe extends SttTask {
async _onTranscription(cs, ep, channel, evt, fsEvent) {
// make sure this is not a transcript from answering machine detection
const bugname = fsEvent.getHeader('media-bugname');
const finished = fsEvent.getHeader('transcription-session-finished');
if (bugname && this.bugname !== bugname) return;
if (this.paused) {
this.logger.debug({evt}, 'TaskTranscribe:_onTranscription - paused, ignoring transcript');
}
if (this.vendor === 'ibm' && evt?.state === 'listening') return;
@@ -306,9 +318,10 @@ class TaskTranscribe extends SttTask {
}
else {
this.logger.debug('Gather:_onTranscription - got UtteranceEnd event from deepgram, return buffered transcript');
evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language);
evt = this.consolidateTranscripts(this._bufferedTranscripts, 1, this.language, this.vendor);
evt.is_final = true;
this._bufferedTranscripts = [];
this._resolve('speech', evt);
this._resolve(channel, evt);
}
return;
}
@@ -322,31 +335,89 @@ class TaskTranscribe extends SttTask {
return;
}
if (evt.alternatives[0]?.transcript === '' && !cs.callGone && !this.killed) {
if (['microsoft', 'deepgram'].includes(this.vendor)) {
this.logger.info({evt}, 'TaskTranscribe:_onTranscription - got empty transcript, continue listening');
let emptyTranscript = false;
if (evt.is_final) {
if (evt.alternatives[0].transcript === '' && !cs.callGone && !this.killed) {
emptyTranscript = true;
if (finished === 'true' &&
['microsoft', 'deepgram'].includes(this.vendor) &&
this._bufferedTranscripts.length === 0) {
this.logger.debug({evt}, 'TaskGather:_onTranscription - got empty transcript from old gather, disregarding');
return;
}
else if (this.vendor !== 'deepgram') {
this.logger.info({evt}, 'TaskGather:_onTranscription - got empty transcript, continue listening');
return;
}
else if (this.isContinuousAsr) {
this.logger.info({evt},
'TaskGather:_onTranscription - got empty deepgram transcript during continous asr, continue listening');
return;
}
else if (this.vendor === 'deepgram' && this._bufferedTranscripts.length > 0) {
this.logger.info({evt},
'TaskGather:_onTranscription - got empty transcript from deepgram, return the buffered transcripts');
}
}
if (this.isContinuousAsr) {
/* append the transcript and start listening again for asrTimeout */
const t = evt.alternatives[0].transcript;
if (t) {
/* remove trailing punctuation */
if (/[,;:\.!\?]$/.test(t)) {
this.logger.debug('TaskGather:_onTranscription - removing trailing punctuation');
evt.alternatives[0].transcript = t.slice(0, -1);
}
}
this.logger.info({evt}, 'TaskGather:_onTranscription - got transcript during continous asr');
this._bufferedTranscripts.push(evt);
this._startAsrTimer(channel);
/* some STT engines will keep listening after a final response, so no need to restart */
if (!['soniox', 'aws', 'microsoft', 'deepgram', 'google']
.includes(this.vendor)) this._startTranscribing(cs, ep, channel);
}
else {
this.logger.info({evt}, 'TaskTranscribe:_onTranscription - got empty transcript, listen again');
this._transcribe(ep);
}
return;
}
if (this.vendor === 'soniox') {
/* compile transcripts into one */
this._sonioxTranscripts.push(evt.vendor.finalWords);
evt = this.compileSonioxTranscripts(this._sonioxTranscripts, 1, this.language);
this._sonioxTranscripts = [];
}
else if (this.vendor === 'deepgram') {
/* compile transcripts into one */
if (!emptyTranscript) this._bufferedTranscripts.push(evt);
if (this.vendor === 'soniox') {
/* compile transcripts into one */
this._sonioxTranscripts.push(evt.vendor.finalWords);
if (evt.is_final) {
evt = this.compileSonioxTranscripts(this._sonioxTranscripts, 1, this.language);
this._sonioxTranscripts = [];
/* deepgram can send an empty and final transcript; only if we have any buffered should we resolve */
if (this._bufferedTranscripts.length === 0) return;
evt = this.consolidateTranscripts(this._bufferedTranscripts, channel, this.language);
this._bufferedTranscripts = [];
}
/* here is where we return a final transcript */
this.logger.debug({evt}, 'TaskTranscribe:_onTranscription - sending final transcript');
this._resolve(channel, evt);
/* some STT engines will keep listening after a final response, so no need to restart */
if (!['soniox', 'aws', 'microsoft', 'deepgram', 'google']
.includes(this.vendor)) this._startTranscribing(cs, ep, channel);
}
}
else {
/* interim transcript */
if (this.isContinuousAsr && evt.is_final) {
this._bufferedTranscripts.push(evt);
this._startAsrTimer(channel);
} else {
await this._resolve(channel, evt);
/* deepgram can send a non-final transcript but with words that are final, so we need to buffer */
if (this.vendor === 'deepgram') {
const originalEvent = evt.vendor.evt;
if (originalEvent.is_final && evt.alternatives[0].transcript !== '') {
this.logger.debug({evt}, 'Gather:_onTranscription - buffering a completed (partial) deepgram transcript');
this._bufferedTranscripts.push(evt);
}
}
if (this.interim) {
this.logger.debug({evt}, 'TaskTranscribe:_onTranscription - sending interim transcript');
this._resolve(channel, evt);
}
}
}
@@ -399,7 +470,8 @@ class TaskTranscribe extends SttTask {
}
_onNoAudio(cs, ep, channel) {
this.logger.debug(`TaskTranscribe:_onNoAudio restarting transcription on channel ${channel}`);
this.logger.debug(`TaskTranscribe:_onNoAudio on channel ${channel}`);
if (this.paused) return;
if (this.childSpan[channel - 1] && this.childSpan[channel - 1].span) {
this.childSpan[channel - 1].span.setAttributes({
channel,
@@ -415,7 +487,8 @@ class TaskTranscribe extends SttTask {
}
_onMaxDurationExceeded(cs, ep, channel) {
this.logger.debug(`TaskTranscribe:_onMaxDurationExceeded restarting transcription on channel ${channel}`);
this.logger.debug(`TaskTranscribe:_onMaxDurationExceeded on channel ${channel}`);
if (this.paused) return;
if (this.childSpan[channel - 1] && this.childSpan[channel - 1].span) {
this.childSpan[channel - 1].span.setAttributes({
channel,
@@ -440,6 +513,7 @@ class TaskTranscribe extends SttTask {
async _onJambonzError(cs, _ep, evt) {
this.logger.info({evt}, 'TaskTranscribe:_onJambonzError');
if (this.paused) return;
if (this.isHandledByPrimaryProvider && this.fallbackVendor) {
_ep.stopTranscription({
vendor: this.vendor,
@@ -496,7 +570,7 @@ class TaskTranscribe extends SttTask {
this._clearAsrTimer(channel);
this._asrTimer = setTimeout(() => {
this.logger.debug(`TaskTranscribe:_startAsrTimer - asr timer went off for channel: ${channel}`);
const evt = this.consolidateTranscripts(this._bufferedTranscripts, channel, this.language);
const evt = this.consolidateTranscripts(this._bufferedTranscripts, channel, this.language, this.vendor);
this._bufferedTranscripts = [];
this._resolve(channel, evt);
}, this.asrTimeout);

View File

@@ -65,18 +65,17 @@ class BackgroundTaskManager extends Emitter {
// Remove task from managed List
this.tasks.delete(type);
} else {
this.logger.info(`stopping background task, ${type} is not running, skipped`);
this.logger.debug(`stopping background task, ${type} is not running, skipped`);
}
}
stopAll() {
this.logger.info('BackgroundTaskManager:stopAll');
this.logger.debug('BackgroundTaskManager:stopAll');
for (const key of this.tasks.keys()) {
this.stop(key);
}
}
// Initiate Task
// Initiate Listen
async _initListen(opts, bugname = 'jambonz-background-listen', ignoreCustomerData = false, type = 'listen') {
let task;
@@ -134,8 +133,7 @@ class BackgroundTaskManager extends Emitter {
async _initRecord() {
if (this.cs.accountInfo.account.record_all_calls || this.cs.application.record_all_calls) {
if (!JAMBONZ_RECORD_WS_BASE_URL || !this.cs.accountInfo.account.bucket_credential) {
this.logger.error(`_initRecord: invalid configuration,
missing JAMBONZ_RECORD_WS_BASE_URL or bucket configuration`);
this.logger.error('_initRecord: invalid cfg - missing JAMBONZ_RECORD_WS_BASE_URL or bucket config');
return undefined;
}
const listenOpts = {
@@ -176,7 +174,7 @@ class BackgroundTaskManager extends Emitter {
}
_taskCompleted(type, task) {
this.logger.info({type, task}, 'BackgroundTaskManager:_taskCompleted: task completed');
this.logger.debug({type, task}, `BackgroundTaskManager:_taskCompleted: task completed, sticky: ${task.sticky}`);
task.removeAllListeners();
task.span.end();
this.tasks.delete(type);
@@ -189,7 +187,8 @@ class BackgroundTaskManager extends Emitter {
}
_bargeInTaskCompleted(evt) {
this.logger.info({evt}, 'BackgroundTaskManager:_bargeInTaskCompleted on event from background bargeIn');
this.logger.debug({evt},
'BackgroundTaskManager:_bargeInTaskCompleted on event from background bargeIn, emitting bargein-done event');
this.emit('bargeIn-done', evt);
}
}

View File

@@ -30,6 +30,7 @@
"Transcribe": "transcribe"
},
"AllowedSipRecVerbs": ["config", "gather", "transcribe", "listen", "tag"],
"AllowedConfirmSessionVerbs": ["config", "gather", "plays", "say", "tag"],
"CallStatus": {
"Trying": "trying",
"Ringing": "ringing",

View File

@@ -75,6 +75,8 @@ const speechMapper = (cred) => {
else if ('deepgram' === obj.vendor) {
const o = JSON.parse(decrypt(credential));
obj.api_key = o.api_key;
obj.deepgram_stt_uri = o.deepgram_stt_uri;
obj.deepgram_stt_use_tls = o.deepgram_stt_use_tls;
}
else if ('soniox' === obj.vendor) {
const o = JSON.parse(decrypt(credential));

View File

@@ -176,6 +176,7 @@ function installSrfLocals(srf, logger) {
const registrar = new Registrar(logger, client);
const {
synthAudio,
addFileToCache,
getNuanceAccessToken,
getIbmAccessToken,
} = require('@jambonz/speech-utils')({}, logger);
@@ -215,6 +216,7 @@ function installSrfLocals(srf, logger) {
listCalls,
deleteCall,
synthAudio,
addFileToCache,
createHash,
retrieveHash,
deleteKey,

View File

@@ -16,6 +16,9 @@ const uuidv4 = require('uuid-random');
const HttpRequestor = require('./http-requestor');
const WsRequestor = require('./ws-requestor');
const {makeOpusFirst} = require('./sdp-utils');
const {
JAMBONES_USE_FREESWITCH_TIMER_FD
} = require('../config');
class SingleDialer extends Emitter {
constructor({logger, sbcAddress, target, opts, application, callInfo, accountInfo, rootSpan, startSpan, dialTask,
@@ -192,6 +195,10 @@ class SingleDialer extends Emitter {
callSid: this.callSid,
traceId: this.rootSpan.traceId
});
if (this.dialTask && this.dialTask.tag !== null &&
typeof this.dialTask.tag === 'object' && !Array.isArray(this.dialTask.tag)) {
this.callInfo.customerData = this.dialTask.tag;
}
this.logger = srf.locals.parentLogger.child({
callSid: this.callSid,
parentCallSid: this.parentCallInfo.callSid,
@@ -324,8 +331,12 @@ class SingleDialer extends Emitter {
}
_configMsEndpoint() {
if (this.onHoldMusic) {
this.ep.set({hold_music: `shout://${this.onHoldMusic.replace(/^https?:\/\//, '')}`});
const opts = {
...(this.onHoldMusic && {holdMusic: `shout://${this.onHoldMusic.replace(/^https?:\/\//, '')}`}),
...(JAMBONES_USE_FREESWITCH_TIMER_FD && {timer_name: 'timerfd'})
};
if (Object.keys(opts).length > 0) {
this.ep.set(opts);
}
}
@@ -342,6 +353,7 @@ class SingleDialer extends Emitter {
const json = await this.requestor.request('dial:confirm', confirmHook, this.callInfo.toJSON());
if (!json || (Array.isArray(json) && json.length === 0)) {
this.logger.info('SingleDialer:_executeApp: no tasks returned from confirm hook');
this.emit('accept');
return;
}
const tasks = normalizeJambones(this.logger, json).map((tdata) => makeTask(this.logger, tdata));
@@ -415,6 +427,8 @@ class SingleDialer extends Emitter {
this.accountInfo.account.webhook_secret);
else app.notifier = {request: () => {}, close: () => {}};
}
// Replace old application with new application.
this.application = app;
const cs = new AdultingCallSession({
logger: newLogger,
singleDialer: this,

View File

@@ -102,41 +102,44 @@ const stickyVars = {
]
};
/**
* @see https://developers.deepgram.com/docs/models-languages-overview
*/
const optimalDeepramModels = {
zh: ['base', 'base'],
'zh-CN':['base', 'base'],
'zh-TW': ['base', 'base'],
da: ['enhanced', 'enhanced'],
en: ['nova-2-conversationalai', 'nova-2'],
'en-US': ['nova-2-conversationalai', 'nova-2'],
'en-AU': ['nova-2-conversationalai', 'nova-2'],
'en-GB': ['nova-2-conversationalai', 'nova-2'],
'en-IN': ['nova-2-conversationalai', 'nova-2'],
'en-NZ': ['nova-2-conversationalai', 'nova-2'],
nl: ['nova-2-conversationalai', 'nova-2'],
fr: ['nova-2-conversationalai', 'nova-2'],
'fr-CA': ['nova-2-conversationalai', 'nova-2'],
de: ['nova-2-conversationalai', 'nova-2'],
hi: ['nova-2-conversationalai', 'nova-2'],
'hi-Latn': ['nova-2-conversationalai', 'nova-2'],
en: ['nova-2-phonecall', 'nova-2'],
'en-US': ['nova-2-phonecall', 'nova-2'],
'en-AU': ['nova-2', 'nova-2'],
'en-GB': ['nova-2', 'nova-2'],
'en-IN': ['nova-2', 'nova-2'],
'en-NZ': ['nova-2', 'nova-2'],
nl: ['nova-2', 'nova-2'],
fr: ['nova-2', 'nova-2'],
'fr-CA': ['nova-2', 'nova-2'],
de: ['nova-2', 'nova-2'],
hi: ['nova-2', 'nova-2'],
'hi-Latn': ['nova-2', 'nova-2'],
id: ['base', 'base'],
it: ['enhanced', 'enhanced'],
it: ['nova-2', 'nova-2'],
ja: ['enhanced', 'enhanced'],
ko: ['enhanced', 'enhanced'],
no: ['enhanced', 'enhanced'],
pl: ['enhanced', 'enhanced'],
pt: ['nova-2-conversationalai', 'nova-2'],
'pt-BR': ['nova-2-conversationalai', 'nova-2'],
'pt-PT': ['base', 'base'],
ru: ['base', 'base'],
es: ['nova-2-conversationalai', 'nova-2'],
'es-419': ['nova-2-conversationalai', 'nova-2'],
ko: ['nova-2', 'nova-2'],
no: ['nova-2', 'nova-2'],
pl: ['nova-2', 'nova-2'],
pt: ['nova-2', 'nova-2'],
'pt-BR': ['nova-2', 'nova-2'],
'pt-PT': ['nova-2', 'nova-2'],
ru: ['nova-2', 'nova-2'],
es: ['nova-2', 'nova-2'],
'es-419': ['nova-2', 'nova-2'],
'es-LATAM': ['enhanced', 'enhanced'],
sv: ['enhanced', 'enhanced'],
sv: ['nova-2', 'nova-2'],
ta: ['enhanced', 'enhanced'],
taq: ['enhanced', 'enhanced'],
tr: ['base', 'base'],
uk: ['base', 'base']
tr: ['nova-2', 'nova-2'],
uk: ['nova-2', 'nova-2']
};
const selectDefaultDeepgramModel = (task, language) => {
@@ -144,9 +147,10 @@ const selectDefaultDeepgramModel = (task, language) => {
const [gather, transcribe] = optimalDeepramModels[language];
return task.name === TaskName.Gather ? gather : transcribe;
}
return 'base';
};
const consolidateTranscripts = (bufferedTranscripts, channel, language) => {
const consolidateTranscripts = (bufferedTranscripts, channel, language, vendor) => {
if (bufferedTranscripts.length === 1) return bufferedTranscripts[0];
let totalConfidence = 0;
const finalTranscript = bufferedTranscripts.reduce((acc, evt) => {
@@ -187,7 +191,7 @@ const consolidateTranscripts = (bufferedTranscripts, channel, language) => {
totalConfidence / bufferedTranscripts.length;
finalTranscript.alternatives[0].transcript = finalTranscript.alternatives[0].transcript.trim();
finalTranscript.vendor = {
name: 'deepgram',
name: vendor,
evt: bufferedTranscripts
};
return finalTranscript;
@@ -484,6 +488,13 @@ module.exports = (logger) => {
if ('google' === vendor) {
const model = task.name === TaskName.Gather ? 'command_and_search' : 'latest_long';
/**
* When we support google v2 the models are different and we will want something like:
* const useV2 = sttCredentials?.credentials?.project_id; //TODO: v2 pref should be set in googleOptions
* const model = task.name === TaskName.Gather ?
* (useV2 ? 'telephony_short' : 'command_and_search') :
* (useV2 ? 'long' : 'latest_long');
*/
opts = {
...opts,
...(sttCredentials && {GOOGLE_APPLICATION_CREDENTIALS: JSON.stringify(sttCredentials.credentials)}),
@@ -516,6 +527,12 @@ module.exports = (logger) => {
...{GOOGLE_SPEECH_MODEL: rOpts.model || model},
...(rOpts.naicsCode > 0 && {GOOGLE_SPEECH_METADATA_INDUSTRY_NAICS_CODE: rOpts.naicsCode}),
GOOGLE_SPEECH_METADATA_RECORDING_DEVICE_TYPE: 'phone_line',
/*
...(useV2 && {
GOOGLE_SPEECH_RECOGNIZER_PARENT: `projects/${sttCredentials.credentials.project_id}/locations/global`,
GOOGLE_SPEECH_CLOUD_SERVICES_VERSION: 'v2'
}),
*/
};
}
else if (['aws', 'polly'].includes(vendor)) {
@@ -554,6 +571,8 @@ module.exports = (logger) => {
...{AZURE_USE_OUTPUT_FORMAT_DETAILED: 1},
...(azureOptions.speechSegmentationSilenceTimeoutMs &&
{AZURE_SPEECH_SEGMENTATION_SILENCE_TIMEOUT_MS: azureOptions.speechSegmentationSilenceTimeoutMs}),
...(azureOptions.languageIdMode &&
{AZURE_LANGUAGE_ID_MODE: azureOptions.languageIdMode}),
...(sttCredentials && {
...(sttCredentials.api_key && {AZURE_SUBSCRIPTION_KEY: sttCredentials.api_key}),
...(sttCredentials.region && {AZURE_REGION: sttCredentials.region}),
@@ -614,18 +633,24 @@ module.exports = (logger) => {
};
}
else if ('deepgram' === vendor) {
let {model} = rOpts;
const {deepgramOptions = {}} = rOpts;
if (!deepgramOptions.model) {
deepgramOptions.model = selectDefaultDeepgramModel(task, language);
const deepgramUri = deepgramOptions.deepgramSttUri || sttCredentials.deepgram_stt_uri;
const useTls = deepgramOptions.deepgramSttUseTls || sttCredentials.deepgram_stt_use_tls;
/* default to a sensible model if not supplied */
if (!model) {
model = selectDefaultDeepgramModel(task, language);
}
opts = {
...opts,
DEEPGRAM_SPEECH_MODEL: model,
...(deepgramUri && {DEEPGRAM_URI: deepgramUri}),
...(deepgramUri && useTls && {DEEPGRAM_USE_TLS: 1}),
...(sttCredentials.api_key) &&
{DEEPGRAM_API_KEY: sttCredentials.api_key},
...(deepgramOptions.tier) &&
{DEEPGRAM_SPEECH_TIER: deepgramOptions.tier},
...(deepgramOptions.model) &&
{DEEPGRAM_SPEECH_MODEL: deepgramOptions.model},
...(deepgramOptions.punctuate) &&
{DEEPGRAM_SPEECH_ENABLE_AUTOMATIC_PUNCTUATION: 1},
...(deepgramOptions.smartFormatting) &&

View File

@@ -326,7 +326,9 @@ class WsRequestor extends BaseRequestor {
'WsRequestor:_onSocketClosed time to reconnect');
if (!this.ws && !this.connectInProgress) {
this.connectInProgress = true;
this._connect().catch((err) => this.connectInProgress = false);
return this._connect()
.catch((err) => this.logger.error('WsRequestor:_onSocketClosed There is error while reconnect', err))
.finally(() => this.connectInProgress = false);
}
}, this.backoffMs);
this.backoffMs = this.backoffMs < 2000 ? this.backoffMs * 2 : (this.backoffMs + 2000);

6918
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,9 +1,9 @@
{
"name": "jambonz-feature-server",
"version": "0.8.5",
"version": "0.8.6",
"main": "app.js",
"engines": {
"node": ">= 10.16.0"
"node": ">= 18.x"
},
"keywords": [
"sip",
@@ -31,10 +31,10 @@
"@jambonz/http-health-check": "^0.0.1",
"@jambonz/mw-registrar": "^0.2.4",
"@jambonz/realtimedb-helpers": "^0.8.7",
"@jambonz/speech-utils": "^0.0.33",
"@jambonz/speech-utils": "^0.0.42",
"@jambonz/stats-collector": "^0.1.9",
"@jambonz/time-series": "^0.2.8",
"@jambonz/verb-specifications": "^0.0.50",
"@jambonz/verb-specifications": "^0.0.63",
"@opentelemetry/api": "^1.4.0",
"@opentelemetry/exporter-jaeger": "^1.9.0",
"@opentelemetry/exporter-trace-otlp-http": "^0.35.0",
@@ -47,11 +47,11 @@
"bent": "^7.3.12",
"debug": "^4.3.4",
"deepcopy": "^2.1.0",
"drachtio-fsmrf": "^3.0.33",
"drachtio-fsmrf": "^3.0.38",
"drachtio-srf": "^4.5.31",
"express": "^4.18.2",
"express-validator": "^7.0.1",
"ip": "^1.1.8",
"ip": "^1.1.9",
"moment": "^2.29.4",
"parse-url": "^8.1.0",
"pino": "^8.8.0",
@@ -61,7 +61,7 @@
"short-uuid": "^4.2.2",
"sinon": "^15.0.1",
"to-snake-case": "^1.0.0",
"undici": "^5.26.2",
"undici": "^5.28.3",
"uuid-random": "^1.3.2",
"verify-aws-sns-signature": "^0.1.0",
"ws": "^8.9.0",

View File

@@ -42,7 +42,7 @@ services:
ipv4_address: 172.38.0.7
drachtio:
image: drachtio/drachtio-server:0.8.24
image: drachtio/drachtio-server:0.8.25-rc8
restart: always
command: drachtio --contact "sip:*;transport=udp" --mtu 4096 --address 0.0.0.0 --port 9022
ports:
@@ -57,7 +57,7 @@ services:
condition: service_healthy
freeswitch:
image: drachtio/drachtio-freeswitch-mrf:0.6.1
image: drachtio/drachtio-freeswitch-mrf:0.6.2
restart: always
command: freeswitch --rtp-range-start 20000 --rtp-range-end 20100
environment:

View File

@@ -347,8 +347,7 @@ test('\'transcribe\' test - deepgram config options altLanguages', async(t) => {
"en-US"
],
"deepgramOptions": {
"model": "2-ea",
"tier": "nova",
"model": "nova-2",
"numerals": true,
"ner": true,
"vadTurnoff": 10,
@@ -408,8 +407,7 @@ test('\'transcribe\' test - deepgram config options altLanguages', async(t) => {
"en-US"
],
"deepgramOptions": {
"model": "2-ea",
"tier": "nova",
"model": "nova-2",
"numerals": true,
"ner": true,
"vadTurnoff": 10,