From 5cf7ce584799deae918e049467a9ad0daa470585 Mon Sep 17 00:00:00 2001 From: let5sne Date: Sat, 25 Jul 2026 23:58:10 +0800 Subject: [PATCH] feat(practice): run realtime practice on App-Plus through renderjs App-Plus has no WebRTC or getUserMedia in the logic layer, so realtime practice was H5-only and the native entry had to advertise a record-then- transcribe downgrade. Move the media/WebRTC engine into `realtime-browser-engine.js` and drive it from `RealtimePracticeAppBridge.vue`, whose renderjs script runs inside the system WebView where those APIs do exist. The logic layer keeps the authenticated calls: SDP exchange and `search_knowledge` tool invocation stay server-proxied, so the access token is never handed to the view layer. Both sides talk over the renderjs bridge only in session payloads the server already assembled. Verified against a real App build (`app-renderjs.js` holds RTCPeerConnection and getUserMedia; `app-service.js` holds the SDP endpoint, renderjs side has zero hits). Microphone permission, live transcription, remote playback, foreground/background switching and disconnect fallback still require Android and iOS device verification. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 2 +- docs/API_INTEGRATION.md | 2 +- docs/MOBILE_APP_DEVICE_ACCEPTANCE.md | 2 +- docs/MOBILE_APP_PACKAGING.md | 2 +- .../practice/RealtimePracticeAppBridge.vue | 220 +++++++++++ mobile-uni/src/pages/user/practice/index.vue | 86 ++-- .../src/services/realtime-browser-engine.d.ts | 30 ++ .../src/services/realtime-browser-engine.js | 302 +++++++++++++++ mobile-uni/src/services/realtime-practice.ts | 366 ++---------------- mobile-uni/tests/realtime-practice.test.mjs | 51 ++- 10 files changed, 673 insertions(+), 390 deletions(-) create mode 100644 mobile-uni/src/components/practice/RealtimePracticeAppBridge.vue create mode 100644 mobile-uni/src/services/realtime-browser-engine.d.ts create mode 100644 mobile-uni/src/services/realtime-browser-engine.js diff --git a/AGENTS.md b/AGENTS.md index 61f1375b..359989c2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,7 +66,7 @@ - 候选面试复用 `/api/recruit/interview/**`；补充资料走认证的移动端接口，先写 `sys_oss`/MinIO，再写 `aihr_candidate_material`，状态仅为 `待审核/已通过/已驳回`，由管理端 `/recruit/interview` 审核。上传接口不得放入 class-level `@SaIgnore`。 - 后端 API 对接先看 `docs/API_INTEGRATION.md`。三角色对练 `/turn`/`/finish` 使用 `AihrPracticeLlmService`，必须保留“未配置/失败回退 seed”的降级链。 - 对练语音复用 `/api/ai/asr` 与 `/api/ai/tts`，TTS 保持旧 `voice` 兼容并支持角色音色。浏览器录音不可用或权限失败时，用 `audio/*` file input 继续走同一 ASR，不退回纯文字；新增 multipart 外发同样清洗 filename/contentType 的 CRLF。 -- 实时对练 Beta（WebRTC）的人设与工具声明只由服务端 `AihrRealtimePersonaRegistry` 白名单组装，浏览器原样转发 session.update，不得恢复前端自组 instructions 或音色 picker；工具执行按登录态 + Redis 会话归属 + 人设白名单三重校验，仅只读知识检索且 `source=realtime` 不落 SOP 评审/知识缺口表；实时 Beta 不计分、不持久化、仅 H5、需 APP 登录态，接口契约见 `docs/API_INTEGRATION.md`。 +- 实时对练 Beta（WebRTC）的人设与工具声明只由服务端 `AihrRealtimePersonaRegistry` 白名单组装，客户端原样转发 session.update，不得恢复前端自组 instructions 或音色 picker；工具执行按登录态 + Redis 会话归属 + 人设白名单三重校验，仅只读知识检索且 `source=realtime` 不落 SOP 评审/知识缺口表；实时 Beta 不计分、不持久化，支持 H5 与 App-Plus（App 通过 renderjs WebView 执行媒体/WebRTC，并由逻辑层代理认证 SDP 和工具请求），需 APP 登录态，接口契约见 `docs/API_INTEGRATION.md`。 - 后端业务代码不要塞进上游 `ruoyi-demo`；自有 API 放在 `backend/ruoyi-modules/ruoyi-aihr`，再接入 `ruoyi-admin`。 - AI 面试、三角色对练、案例、SOP 和三端首页保持真实 API/模型优先与既有降级。知识检索以 MySQL 为事实源，Qdrant/vector/rerank 是可选增强；向量维度不一致必须经状态与重建入口处理，不得静默换成本地向量。 - 批量资料和所有视频只走 `/api/knowledge/doc/upload-async` 异步队列；同步 `/api/knowledge/doc/upload` 仅供 SOP 单文件即时预览，禁止把重加工塞回请求线程。格式、大小、超时、OCR/视频解析和重试契约以 `docs/API_INTEGRATION.md` 为准；图片无可用视觉结果时保持「待处理」，视频关键帧不得退回纯 OCR。 diff --git a/docs/API_INTEGRATION.md b/docs/API_INTEGRATION.md index d5c29f0e..83d337b4 100644 --- a/docs/API_INTEGRATION.md +++ b/docs/API_INTEGRATION.md @@ -13,7 +13,7 @@ | M2 训练预习与求助 | `GET /api/train/practice/scenarios/{id}/prep-card`、`POST /api/train/practice/sessions/{sessionId}/help` | 均要求已登录；预习卡只使用当前租户已启用的 `prep_card/json_prep` 模板，模型返回必须严格为 3 条要点、3 条红线、2 条话术，否则回退场景卡。求助接口只允许 APP 员工访问其本人、同租户的活动训练会话；客户端可传 `roundIndex` 仅为兼容字段，服务端按会话实际进度落库，返回 `{sessionId,scenarioId,roundIndex,recordedAt}`。迁移表为 `aihr_practice_help_event`，不在请求路径执行 DDL。该表已随 2026-07-24 完整包发布并纳入远端 `62/62` 结构预检；完整训练写入与正式试点仍需另验。 | | 正式试点数据导出 | `GET /api/train/practice/export?startDate=YYYY-MM-DD&endDate=YYYY-MM-DD` | 起止日期必填且包含结束日；只统计窗口内能通过唯一手机号或外部 ID 映射到在职组织快照的正式会话，排除重复手机号和身份碰撞。完训定义为每人至少 10 次，校准必须关联同一窗口内正式会话；CSV 同时给出校准命中数、SOP 可用数、满意度响应数/平均分，以及明细级 AI 分、人工校准分、校准人、校准时间、最终采用分、满意度分和意见，避免用四舍五入后的比率反推门禁状态；汇总和明细均携带正式人员及项目口径，不混入历史 seed/开发身份 | | 对练语音 | `POST /api/ai/asr`（multipart 字段 `file`，≤5MB）、`POST /api/ai/tts`（JSON `{text≤300字, voice?, voiceProfile?:{role,voice?,speed?,emotion?,dialect?}, practiceContext?:{sessionId,turnIndex,role:'customer'}}`，`dialect` 仅接受 `mandarin/cantonese/sichuanese`，非法值返回业务码 `400`；旧 `voice` 兼容；成功返回 `ossId`，客户端播放地址为受控的内联 `data:audio/*`，不返回原始 OSS URL）、`GET /api/aihr/mobile/oss/{ossId}` | 走 OpenAI-compatible audio 接口；生产 SiliconFlow CosyVoice2 按角色映射老师傅/业主/面试官音色，业主对练再按已有情绪分切换平静/严肃/强烈语气和语速；dialect 作为语气提示传给支持表现力提示词的 TTS，不含克隆，也不等同于方言正式验收。`practiceContext` 仅允许已认证 APP 员工把本次服务端生成的业主 TTS 绑定到本人当前或刚完成会话的指定原话术回合；服务端重新校验租户、员工身份、回合和文本，不接受员工端传 OSS ID 或任意角色。设备语音降级按方言选择语言：粤语使用 `zh-HK`，四川话无法由设备语音可靠模拟时明确提示用户转用服务端语音或文字，不伪装成普通话。模型管理需启用 `category=asr/tts` 配置；移动端优先用浏览器录音，`getUserMedia/MediaRecorder` 不可用或麦克风权限失败时，用 `audio/*` file input 选择/录制音频后继续调同一 ASR 接口；ASR/TTS 未配置或失败返回 fail，前端降级设备语音或文本；训练/每日题录音只通过 `audioOssId` 走受保护下载，历史客户端传入的 HTTP `audioUrl` 不再回显；TTS 成功音频仍写入 `sys_oss` 留痕，同时用内联 data URL 保持旧客户端可播放；移动端 OSS 下载只允许经本人或主管项目范围验证的 APP 身份，后台账号若需回放必须使用未来独立、受审计的管理端契约。 | -| 实时对练 Beta（WebRTC） | `POST /api/train/practice/realtime/sdp`、`GET /realtime/personas`、`POST /realtime/session`、`POST /realtime/tools/invoke` | 浏览器 WebRTC 直连阿里云（音频不过服务端），后端仅代理 SDP 交换（3 次/分/账号，sessionId 仅作日志关联且限字符）。人设为服务端 `AihrRealtimePersonaRegistry` 白名单：`owner-calm` 业主·常规（默认）、`owner-impatient` 业主·急躁、`digital-mentor` 数字师傅；人设绑定默认音色，客户端只选不编。`session.update`（instructions/音色/turn_detection/tools 声明）由 `/realtime/session` 组装下发，浏览器原样转发；personaId 空白回落默认人设；会话记录写 Redis 30min（成功工具调用滑动续期），创建限流 6 次/分/账号。`digital-mentor` 声明 `search_knowledge` 工具：模型 function calling 命中（`response.function_call_arguments.done`）时浏览器回传 `/realtime/tools/invoke`，服务端按登录态 + Redis 会话归属 + 人设工具白名单三重校验后走 `searchAuthorized` 浅层检索（不生成 LLM 答案、不落 `aihr_agent_run`；`source=realtime` 跳过 SOP 评审与知识缺口写入），output 拼接 ≤3 条片段并截断 1200 字；工具按账号限流 20 次/分，无授权/无结果/超时一律返回软着陆文案让模型如实作答，不中断语音会话。实时 Beta 不计分、不持久化训练数据；仅 H5（App-Plus 不支持），需 APP 员工登录态；不启用模型 `enable_search`。配置 `AIHR_QWEN_REALTIME_ENDPOINT` / `AIHR_QWEN_REALTIME_API_KEY` / `AIHR_QWEN_REALTIME_MODEL`（模型白名单 `qwen3.5-omni-(flash|plus)-realtime`），密钥只放 `.env.local` 或外部环境变量。 | +| 实时对练 Beta（WebRTC） | `POST /api/train/practice/realtime/sdp`、`GET /realtime/personas`、`POST /realtime/session`、`POST /realtime/tools/invoke` | H5 浏览器及 App-Plus renderjs WebView 内的 WebRTC 直连阿里云（音频不过服务端），后端仅代理 SDP 交换（3 次/分/账号，sessionId 仅作日志关联且限字符）。人设为服务端 `AihrRealtimePersonaRegistry` 白名单：`owner-calm` 业主·常规（默认）、`owner-impatient` 业主·急躁、`digital-mentor` 数字师傅；人设绑定默认音色，客户端只选不编。`session.update`（instructions/音色/turn_detection/tools 声明）由 `/realtime/session` 组装下发，客户端原样转发；personaId 空白回落默认人设；会话记录写 Redis 30min（成功工具调用滑动续期），创建限流 6 次/分/账号。`digital-mentor` 声明 `search_knowledge` 工具：模型 function calling 命中（`response.function_call_arguments.done`）时客户端回传 `/realtime/tools/invoke`，服务端按登录态 + Redis 会话归属 + 人设工具白名单三重校验后走 `searchAuthorized` 浅层检索（不生成 LLM 答案、不落 `aihr_agent_run`；`source=realtime` 跳过 SOP 评审与知识缺口写入），output 拼接 ≤3 条片段并截断 1200 字；工具按账号限流 20 次/分，无授权/无结果/超时一律返回软着陆文案让模型如实作答，不中断语音会话。实时 Beta 不计分、不持久化训练数据；H5 与 App-Plus 均支持，App-Plus 在 renderjs 视图层执行媒体/WebRTC 并通过逻辑层复用受认证 SDP 与工具接口；需 APP 员工登录态，不启用模型 `enable_search`。配置 `AIHR_QWEN_REALTIME_ENDPOINT` / `AIHR_QWEN_REALTIME_API_KEY` / `AIHR_QWEN_REALTIME_MODEL`（模型白名单 `qwen3.5-omni-(flash|plus)-realtime`），密钥只放 `.env.local` 或外部环境变量。 | | 案例沉淀 `/knowledge/cases` | `GET /api/knowledge/case/capabilities`、`POST /api/knowledge/case/upload`、`/organize`、`/curate`、`GET /records`、`GET /records/{caseId}`、`POST /records/{caseId}/review`、`GET /records/{caseId}/media` | `/capabilities` 返回服务端判定的案例提交/查看能力，移动端不再向普通员工展示无权提交的素材表单；`/upload` 改为 multipart 真实语音上传并走 ASR，服务端只接受 MP3/WAV/M4A/WebM/OGG/AAC/FLAC，成功后原始音频写入 `sys_oss`，案例记录只保存 `mediaOssId` 供受保护媒体接口读取，不向客户端回传原始 `mediaUrl`；`/organize` 用真实转写调 chat 模型整理案例，未配置模型时按真实 transcript 本地结构化，并从背景外的真实摘要项提取学习点；APP 用户必须先以认证手机号精确校验在职 `person_phone`，再由可信主体解析 `aihr_org_snapshot` 项目范围；案例 SQL 不得把 `person_phone` 与 `ext_party_id` 作为替代查询键。上传、整理、入库、列表和详情均按该项目范围校验，未完成正式组织映射时安全拒绝，不接受前端伪造项目范围；员工列表/详情只返回 `已入库` 案例，管理端系统用户保留全局运营视图；移动端和管理端案例详情通过受保护媒体接口回放原始音频，主管/项目负责人可提交脱敏点评；预渲染视频样片仍待正式媒体资产接入 | | 案例媒体安全 | `GET /api/knowledge/case/records/{caseId}/media` | 案例详情只返回 `mediaOssId`，不返回原始 `sys_oss.url`；媒体下载会重复执行登录、后台角色或 APP 项目范围校验，再由服务端流式读取 OSS。管理端与 `mobile-uni` 通过鉴权 blob/temp 文件播放，关闭详情页时释放本地对象 URL | | 问·数字师傅工作 Agent `/pages/user/sop/index` | `POST /api/aihr/agent/messages`、`POST /api/aihr/agent/messages/media`、`POST /api/aihr/agent/actions/{draftId}/confirm`、`POST /api/aihr/agent/actions/{draftId}/dismiss` | 移动端文字、ASR 转写和图片/视频统一进入 Agent。服务端根据消息规划 `KNOWLEDGE_QA/RESOURCE_DELIVERY/LIVE_MY_WORK/LIVE_TEAM_WORK/PRACTICE_COACHING/CAPTURE_FACT/MEDIA_UNDERSTANDING/WEB_RESEARCH/CLARIFY/SOCIAL`，再由固定策略校验当前登录身份和工具权限；客户端不提交 `toolCode`，模型也不能自由指定身份、SQL、URL 或任意工具。知识、训练概况、当前待办、确认式记忆、媒体分析和全网查询均复用现有领域服务；普通图片问答只做本次视觉分析，不要求先具备知识空间，用户明确问现场制度/流程时才校验授权空间并补充 SOP。全网查询先返回 `NEEDS_INPUT`，只有用户明确同意后才调用外部服务。统一响应使用结构化 `status/sourceSummary/citations/resources/actionDraft/clarification`；按钮不从答案文本推断。写入动作只接受 30 分钟有效的不透明 `draftId`，最终确认继续复用既有版本、幂等与可见性规则。每次运行只审计租户、主体、意图、工具、状态、来源类型、耗时和错误码，不保存原始问题、答案或附件。 | diff --git a/docs/MOBILE_APP_DEVICE_ACCEPTANCE.md b/docs/MOBILE_APP_DEVICE_ACCEPTANCE.md index 746f4041..89f852ba 100644 --- a/docs/MOBILE_APP_DEVICE_ACCEPTANCE.md +++ b/docs/MOBILE_APP_DEVICE_ACCEPTANCE.md @@ -14,7 +14,7 @@ - [ ] 首次启动先显示原生隐私弹窗；两个协议链接可打开，拒绝后退出，同意后才进入登录。 - [ ] 手机号登录、短信验证码错误、登录失效重登均符合预期。 - [ ] 首次录音允许麦克风后，可完成转写；拒绝麦克风后不循环申请，能选择本地音频继续。 -- [ ] 对练入口在原生端明确降级为录音后转写，不出现不可用的浏览器 WebRTC 实时通话承诺。 +- [ ] 原生实时对练可完成麦克风授权、实时转写与远端语音播放；WebRTC/WebView 不可用或断线时明确提示并保留录音对练。 - [ ] 图片、视频、文档和音频可选择并上传；受控附件下载后可打开，不泄露 OSS 地址。 - [ ] 前后台切换、网络断开/恢复、窄屏和底部手势区无白屏、遮挡或重复提交。 - [ ] 再次安装同版本后，桌面图标、启动图和 Android 导航栏不闪白。 diff --git a/docs/MOBILE_APP_PACKAGING.md b/docs/MOBILE_APP_PACKAGING.md index 33fe2600..afeb180f 100644 --- a/docs/MOBILE_APP_PACKAGING.md +++ b/docs/MOBILE_APP_PACKAGING.md @@ -36,7 +36,7 @@ npm --prefix mobile-uni run build:app - `androidPrivacy.json.example` 是**未启用的法务模板**，其中 `__TERMS_URL__` 与 `__PRIVACY_URL__` 必须替换为法务确认的 HTTPS 地址。复制为被 Git 忽略的 `androidPrivacy.json` 后，`npm --prefix mobile-uni run verify:app-release` 才会通过。 - iOS 已按实际功能填写麦克风和照片库用途说明；不申请相机、定位、通讯录、蓝牙或广告追踪权限。 - `build:app` 依赖 `@dcloudio/uni-app-plus`（须与其他 `@dcloudio` 包同版本）。缺少该编译器时命令**不报错**，只静默产出带 `/h5/` 基路径的 H5 空壳，产物里没有 `app-service.js` 与 `app-renderjs.js`，App 平台代码等于从未编译。判定 `build:app` 是否真的成功要看产物中是否存在 `app-service.js`、`app-renderjs.js`、`manifest.json`，不能只看 `DONE Build complete.`。 -- App-Plus 当前不启用浏览器 WebRTC 实时对练；界面必须降级为录音后转写，不能将该路径宣传为实时语音通话。 +- App-Plus 实时对练通过 `renderjs` 在系统 WebView 中执行媒体/WebRTC；认证 SDP 与工具请求仍由逻辑层代理。发布前必须在 Android/iOS 真机验证麦克风授权、实时转写、远端语音播放、前后台切换和断线降级。 - 真机步骤和证据口径见 `docs/MOBILE_APP_DEVICE_ACCEPTANCE.md`。资源预检通过不等于 Android/iOS 真机或应用商店验收通过。 ## 更新策略 diff --git a/mobile-uni/src/components/practice/RealtimePracticeAppBridge.vue b/mobile-uni/src/components/practice/RealtimePracticeAppBridge.vue new file mode 100644 index 00000000..d62bb08c --- /dev/null +++ b/mobile-uni/src/components/practice/RealtimePracticeAppBridge.vue @@ -0,0 +1,220 @@ + + + + + + + diff --git a/mobile-uni/src/pages/user/practice/index.vue b/mobile-uni/src/pages/user/practice/index.vue index 18eefa16..a4fa0d8b 100644 --- a/mobile-uni/src/pages/user/practice/index.vue +++ b/mobile-uni/src/pages/user/practice/index.vue @@ -485,6 +485,18 @@ + + + + @@ -492,6 +504,7 @@ import { computed, nextTick, ref } from 'vue'; import { onHide, onShow, onUnload } from '@dcloudio/uni-app'; import ChatComposer from '@/components/chat/ChatComposer.vue'; +import RealtimePracticeAppBridge from '@/components/practice/RealtimePracticeAppBridge.vue'; import type { PracticeFinishResponse, PracticePrepCard, @@ -513,7 +526,7 @@ import { searchKnowledge } from '@/services/knowledge'; import { resetPageScroll } from '@/services/navigation'; import { ensureEmployeePosition } from '@/services/position'; import { chooseSpeechAudio, createSpeechPlaybackController, customerVoiceProfile, measureAudioDuration, MENTOR_VOICE_PROFILE, normalizeOssId, transcribeSpeechCapture, transcribeSpeechFile } from '@/services/speech'; -import { startSpeechCapture } from '@/services/speech-capture'; +import { isNativeAppRuntime, startSpeechCapture } from '@/services/speech-capture'; import type { SpeechCapture, SpeechCaptureSession } from '@/services/speech-capture'; import { createPageRequestScope, currentAccountKey, type RequestScopeTicket } from '@/services/request-scope'; import { createRealtimePracticeController, createRealtimeSession, fetchRealtimePersonas, invokeRealtimeTool, type RealtimePersonaOption } from '@/services/realtime-practice'; @@ -572,6 +585,11 @@ const realtimeAudioBlocked = ref(false); const realtimeSessionId = ref(''); const realtimePersonaOptions = ref([]); const selectedRealtimePersonaId = ref(typeof storedRealtimePersona === 'string' ? storedRealtimePersona : ''); +const realtimeAppBridge = ref<{ + start: (sessionUpdate: Record, sessionId?: string) => Promise; + stop: () => void; + resumeAudio: () => Promise; +} | null>(null); const growthDetailsExpanded = ref(false); const assignments = ref([]); const practiceHistory = ref([]); @@ -1010,34 +1028,40 @@ const appendTurn = (role: PracticeRole, text?: string, emotion = 0, synthesizeCu scrollThreadToBottom(); }; +const handleRealtimeStatus = (nextStatus: typeof realtimeStatus.value) => { + realtimeStatus.value = nextStatus; +}; +const handleRealtimeTraineeTranscript = (text: string) => { + if (realtimeActive.value) appendTurn('trainee', text); +}; +const handleRealtimeCustomerTranscript = (text: string) => { + if (realtimeActive.value) appendTurn('customer', text, 0, false); +}; +const handleRealtimeAudioPlaybackBlocked = (blocked: boolean) => { + if (realtimeActive.value) realtimeAudioBlocked.value = blocked; +}; +const handleRealtimeTerminated = () => { + if (!realtimeActive.value) return; + realtimeActive.value = false; + realtimeStatus.value = 'idle'; + realtimeAudioBlocked.value = false; + realtimeSessionId.value = ''; + sessionId.value = ''; + activeAssignmentId.value = undefined; + status.value = 'idle'; +}; +const handleRealtimeError = (text: string) => { + if (realtimeActive.value) message.value = text; +}; const realtimePractice = createRealtimePracticeController({ - onStatus: (nextStatus) => { - realtimeStatus.value = nextStatus; - }, - onTraineeTranscript: (text) => { - if (realtimeActive.value) appendTurn('trainee', text); - }, - onCustomerTranscript: (text) => { - if (realtimeActive.value) appendTurn('customer', text, 0, false); - }, - onAudioPlaybackBlocked: (blocked) => { - if (realtimeActive.value) realtimeAudioBlocked.value = blocked; - }, + onStatus: handleRealtimeStatus, + onTraineeTranscript: handleRealtimeTraineeTranscript, + onCustomerTranscript: handleRealtimeCustomerTranscript, + onAudioPlaybackBlocked: handleRealtimeAudioPlaybackBlocked, onToolCall: (callId, name, argsJson) => invokeRealtimeTool(realtimeSessionId.value, callId, name, argsJson).then((res) => res.output), - onTerminated: () => { - if (!realtimeActive.value) return; - realtimeActive.value = false; - realtimeStatus.value = 'idle'; - realtimeAudioBlocked.value = false; - realtimeSessionId.value = ''; - sessionId.value = ''; - activeAssignmentId.value = undefined; - status.value = 'idle'; - }, - onError: (text) => { - if (realtimeActive.value) message.value = text; - } + onTerminated: handleRealtimeTerminated, + onError: handleRealtimeError }); const selectRealtimePersona = (item: RealtimePersonaOption) => { @@ -1062,7 +1086,8 @@ const loadRealtimePersonas = async (request: RequestScopeTicket = requests.begin const stopRealtime = () => { if (!realtimeActive.value && realtimeStatus.value === 'idle') return; - realtimePractice.stop(); + if (isNativeAppRuntime()) realtimeAppBridge.value?.stop(); + else realtimePractice.stop(); realtimeActive.value = false; realtimeStatus.value = 'idle'; realtimeAudioBlocked.value = false; @@ -1093,7 +1118,9 @@ const startRealtime = async () => { if (!realtimeActive.value) return; realtimeSessionId.value = session.sessionId; activeScenarioName.value = session.personaName || '实时陪练'; - await realtimePractice.start(session.sessionUpdate, session.sessionId); + const controller = isNativeAppRuntime() ? realtimeAppBridge.value : realtimePractice; + if (!controller) throw new Error('实时语音模块尚未就绪，请稍后重试'); + await controller.start(session.sessionUpdate, session.sessionId); if (!realtimeActive.value) return; status.value = 'active'; message.value = `已连接「${session.personaName}」，请直接开口说话。`; @@ -1107,7 +1134,8 @@ const startRealtime = async () => { }; const resumeRealtimeAudio = async () => { - if (await realtimePractice.resumeAudio()) { + const controller = isNativeAppRuntime() ? realtimeAppBridge.value : realtimePractice; + if (controller && await controller.resumeAudio()) { message.value = '已恢复实时语音播放。'; } else { message.value = '仍无法播放实时语音，请确认设备媒体权限后重试。'; diff --git a/mobile-uni/src/services/realtime-browser-engine.d.ts b/mobile-uni/src/services/realtime-browser-engine.d.ts new file mode 100644 index 00000000..c0c04baf --- /dev/null +++ b/mobile-uni/src/services/realtime-browser-engine.d.ts @@ -0,0 +1,30 @@ +export type RealtimeBrowserStatus = + | 'idle' + | 'connecting' + | 'connected' + | 'listening' + | 'speaking' + | 'calling-tool'; + +export interface RealtimeBrowserCallbacks { + onStatus: (status: RealtimeBrowserStatus) => void; + onTraineeTranscript: (text: string) => void; + onCustomerTranscript: (text: string) => void; + onAudioPlaybackBlocked: (blocked: boolean) => void; + onToolCall: (callId: string, name: string, argsJson: string) => Promise; + onTerminated: () => void; + onError: (message: string) => void; +} + +export interface RealtimeBrowserEngine { + start: (sessionUpdate: Record, sessionId?: string) => Promise; + stop: () => void; + resumeAudio: () => Promise; +} + +export const isRealtimeBrowserSupported: () => boolean; + +export const createRealtimeBrowserEngine: ( + callbacks: RealtimeBrowserCallbacks, + exchangeSdp: (offerSdp: string, sessionId?: string) => Promise +) => RealtimeBrowserEngine; diff --git a/mobile-uni/src/services/realtime-browser-engine.js b/mobile-uni/src/services/realtime-browser-engine.js new file mode 100644 index 00000000..62ff4c33 --- /dev/null +++ b/mobile-uni/src/services/realtime-browser-engine.js @@ -0,0 +1,302 @@ +const trimTranscript = (value) => (value || '').trim(); +const normalizeSdp = (sdp) => `${sdp.replace(/\r\n|\r|\n/g, '\r\n').replace(/\r\n$/, '')}\r\n`; +const TOOL_CALL_FALLBACK_OUTPUT = '工具暂时不可用，请直接回应员工'; + +export const isRealtimeBrowserSupported = () => + typeof navigator !== 'undefined' + && Boolean(navigator.mediaDevices?.getUserMedia) + && typeof RTCPeerConnection !== 'undefined'; + +const waitForIceComplete = (connection) => + new Promise((resolve) => { + if (connection.iceGatheringState === 'complete') { + resolve(); + return; + } + const previous = connection.onicegatheringstatechange; + let timeout; + const done = () => { + clearTimeout(timeout); + connection.onicegatheringstatechange = previous; + resolve(); + }; + timeout = setTimeout(done, 3000); + connection.onicegatheringstatechange = () => { + previous?.call(connection, new Event('icegatheringstatechange')); + if (connection.iceGatheringState === 'complete') done(); + }; + }); + +export const createRealtimeBrowserEngine = (callbacks, exchangeSdp) => { + let activeAttempt = 0; + let connection = null; + let microphone = null; + let speaker = null; + let controlChannel = null; + let cancelSessionWait = null; + let sessionWaitAttempt = null; + let toolCallChain = Promise.resolve(); + + const isCurrent = (attempt) => activeAttempt === attempt; + + const releaseResources = () => { + controlChannel?.close(); + controlChannel = null; + microphone?.getTracks().forEach((track) => track.stop()); + microphone = null; + if (speaker) { + speaker.pause(); + speaker.srcObject = null; + speaker = null; + } + connection?.close(); + connection = null; + }; + + const sendSessionUpdate = (channel, sessionUpdate) => { + if (channel.readyState !== 'open') return; + channel.send(JSON.stringify({ event_id: `event_${Date.now()}`, ...sessionUpdate })); + }; + + const handleToolCall = (attempt, event) => { + const callId = trimTranscript(event.call_id); + const name = trimTranscript(event.name); + if (!callId || !name) return; + const argsJson = typeof event.arguments === 'string' ? event.arguments : '{}'; + toolCallChain = toolCallChain.then(async () => { + if (!isCurrent(attempt)) return; + const channel = controlChannel; + if (!channel || channel.readyState !== 'open') return; + callbacks.onStatus('calling-tool'); + let output; + try { + output = await callbacks.onToolCall(callId, name, argsJson); + } catch { + output = TOOL_CALL_FALLBACK_OUTPUT; + } + if (!isCurrent(attempt) || channel.readyState !== 'open') return; + channel.send(JSON.stringify({ + event_id: `event_${Date.now()}`, + type: 'conversation.item.create', + item: { type: 'function_call_output', call_id: callId, output } + })); + channel.send(JSON.stringify({ event_id: `event_${Date.now()}`, type: 'response.create' })); + callbacks.onStatus('connected'); + }).catch(() => { + // 链内任何意外抛错不得毒化后续工具调用；本次调用软失败，对话继续 + }); + }; + + const handleEvent = (raw, attempt, provider) => { + if (!isCurrent(attempt)) return; + let event; + try { + event = JSON.parse(raw); + } catch { + return; + } + if (event.type === 'session.created') provider.onSessionCreated(); + if (event.type === 'session.updated') provider.onSessionUpdated(); + if (event.type === 'input_audio_buffer.speech_started') callbacks.onStatus('listening'); + if (event.type === 'response.created') callbacks.onStatus('speaking'); + if (event.type === 'response.done') callbacks.onStatus('connected'); + if (event.type === 'conversation.item.input_audio_transcription.completed') { + const transcript = trimTranscript(event.transcript); + if (transcript) callbacks.onTraineeTranscript(transcript); + } + if (event.type === 'response.audio_transcript.done') { + const transcript = trimTranscript(event.transcript); + if (transcript) callbacks.onCustomerTranscript(transcript); + } + if (event.type === 'response.function_call_arguments.done') { + handleToolCall(attempt, event); + } + if (event.type === 'error') { + const error = new Error(event.error?.message || '实时陪练连接异常，请改用录音对练'); + callbacks.onError(error.message); + provider.onSessionError(error); + } + }; + + const bindChannel = (channel, attempt, provider) => { + channel.onmessage = (event) => handleEvent(String(event.data || ''), attempt, provider); + channel.onerror = () => { + if (!isCurrent(attempt)) return; + const error = new Error('实时陪练数据通道异常，请改用录音对练'); + callbacks.onError(error.message); + provider.onSessionError(error); + }; + }; + + const stop = () => { + activeAttempt += 1; + cancelSessionWait?.(); + cancelSessionWait = null; + sessionWaitAttempt = null; + toolCallChain = Promise.resolve(); + releaseResources(); + callbacks.onAudioPlaybackBlocked(false); + callbacks.onStatus('idle'); + }; + + const terminate = (attempt, message) => { + if (!isCurrent(attempt)) return; + callbacks.onError(message); + stop(); + callbacks.onTerminated(); + }; + + const resumeAudio = async () => { + if (!speaker) return false; + try { + await speaker.play(); + callbacks.onAudioPlaybackBlocked(false); + return true; + } catch { + callbacks.onAudioPlaybackBlocked(true); + return false; + } + }; + + const start = async (sessionUpdate, sessionId) => { + if (!isRealtimeBrowserSupported()) throw new Error('当前设备的 WebView 不支持实时语音，请更新系统 WebView 后重试'); + stop(); + const attempt = ++activeAttempt; + callbacks.onStatus('connecting'); + + let localMicrophone = null; + let localConnection = null; + let localChannel = null; + let localSpeaker = null; + let readyTimeout; + let disconnectTimeout; + let sessionUpdateSent = false; + let resolveReady; + let rejectReady; + let sessionReady = null; + const releaseLocalResources = () => { + if (disconnectTimeout) clearTimeout(disconnectTimeout); + localChannel?.close(); + localMicrophone?.getTracks().forEach((track) => track.stop()); + if (localSpeaker) { + localSpeaker.pause(); + localSpeaker.srcObject = null; + } + localConnection?.close(); + }; + const ensureCurrent = () => { + if (isCurrent(attempt)) return true; + releaseLocalResources(); + return false; + }; + + try { + localMicrophone = await navigator.mediaDevices.getUserMedia({ audio: true }); + if (!ensureCurrent()) return; + microphone = localMicrophone; + localConnection = new RTCPeerConnection({ iceServers: [] }); + connection = localConnection; + const audioSender = localConnection.addTransceiver('audio', { direction: 'sendrecv' }).sender; + const provider = { + onSessionCreated: () => { + if (!isCurrent(attempt) || !localChannel || sessionUpdateSent) return; + sessionUpdateSent = true; + sendSessionUpdate(localChannel, sessionUpdate); + }, + onSessionUpdated: () => { + if (!isCurrent(attempt) || !sessionUpdateSent) return; + const track = localMicrophone?.getAudioTracks()[0] || null; + void audioSender.replaceTrack(track) + .then(() => { + if (isCurrent(attempt)) resolveReady?.(); + }) + .catch((error) => rejectReady?.(error instanceof Error ? error : new Error('实时陪练麦克风启动失败'))); + }, + onSessionError: (error) => { + rejectReady?.(error); + terminate(attempt, error.message); + } + }; + + localChannel = localConnection.createDataChannel('oai-events'); + controlChannel = localChannel; + bindChannel(localChannel, attempt, provider); + localConnection.ondatachannel = (event) => bindChannel(event.channel, attempt, provider); + localConnection.ontrack = (event) => { + if (!isCurrent(attempt)) return; + const stream = event.streams[0] || new MediaStream([event.track]); + localSpeaker = new Audio(); + speaker = localSpeaker; + localSpeaker.autoplay = true; + localSpeaker.setAttribute('playsinline', 'true'); + localSpeaker.srcObject = stream; + void localSpeaker.play() + .then(() => { + if (isCurrent(attempt)) callbacks.onAudioPlaybackBlocked(false); + }) + .catch(() => { + if (isCurrent(attempt)) callbacks.onAudioPlaybackBlocked(true); + }); + }; + localConnection.onconnectionstatechange = () => { + if (!isCurrent(attempt)) return; + if (localConnection?.connectionState === 'failed' || localConnection?.connectionState === 'closed') { + terminate(attempt, '实时陪练已断开，请改用录音对练'); + return; + } + if (localConnection?.connectionState === 'disconnected') { + disconnectTimeout ??= setTimeout(() => { + if (localConnection?.connectionState === 'disconnected') { + terminate(attempt, '实时陪练已断开，请改用录音对练'); + } + }, 3000); + } else if (disconnectTimeout) { + clearTimeout(disconnectTimeout); + disconnectTimeout = undefined; + } + }; + + const offer = await localConnection.createOffer(); + if (!ensureCurrent()) return; + await localConnection.setLocalDescription(offer); + if (!ensureCurrent()) return; + await waitForIceComplete(localConnection); + if (!ensureCurrent()) return; + const offerSdp = localConnection.localDescription?.sdp; + if (!offerSdp) throw new Error('实时陪练连接请求创建失败'); + const answerSdp = await exchangeSdp(normalizeSdp(offerSdp), sessionId); + if (!ensureCurrent()) return; + sessionReady = new Promise((resolve, reject) => { + resolveReady = resolve; + rejectReady = reject; + readyTimeout = setTimeout(() => reject(new Error('实时陪练会话初始化超时，请使用录音对练')), 15000); + }); + void sessionReady.catch(() => undefined); + sessionWaitAttempt = attempt; + cancelSessionWait = () => { + if (readyTimeout) clearTimeout(readyTimeout); + rejectReady?.(new Error('实时陪练已结束')); + }; + await localConnection.setRemoteDescription({ type: 'answer', sdp: normalizeSdp(answerSdp) }); + if (!ensureCurrent()) return; + await sessionReady; + if (!ensureCurrent()) return; + callbacks.onStatus('connected'); + } catch (error) { + if (!isCurrent(attempt)) { + releaseLocalResources(); + return; + } + stop(); + throw error instanceof Error ? error : new Error('实时陪练连接失败，请改用录音对练'); + } finally { + if (readyTimeout) clearTimeout(readyTimeout); + if (sessionWaitAttempt === attempt) { + sessionWaitAttempt = null; + cancelSessionWait = null; + } + } + }; + + return { start, stop, resumeAudio }; +}; diff --git a/mobile-uni/src/services/realtime-practice.ts b/mobile-uni/src/services/realtime-practice.ts index cb108017..72028a21 100644 --- a/mobile-uni/src/services/realtime-practice.ts +++ b/mobile-uni/src/services/realtime-practice.ts @@ -1,7 +1,11 @@ import { apiRequest } from './api'; import { isNativeAppRuntime } from './speech-capture'; - -type RealtimeStatus = 'idle' | 'connecting' | 'connected' | 'listening' | 'speaking' | 'calling-tool'; +import { createRealtimeBrowserEngine } from './realtime-browser-engine.js'; +import type { + RealtimeBrowserCallbacks, + RealtimeBrowserEngine, + RealtimeBrowserStatus +} from './realtime-browser-engine.js'; export interface RealtimePersonaOption { id: string; @@ -28,36 +32,11 @@ interface RealtimeSdpResponse { model: string; } -interface RealtimeEvent { - type?: string; - transcript?: string; - call_id?: string; - name?: string; - arguments?: string; - error?: { message?: string }; +export interface RealtimePracticeCallbacks extends RealtimeBrowserCallbacks { + onStatus: (status: RealtimeBrowserStatus) => void; } -interface ProviderEventHandlers { - onSessionCreated: () => void; - onSessionUpdated: () => void; - onSessionError: (error: Error) => void; -} - -export interface RealtimePracticeCallbacks { - onStatus: (status: RealtimeStatus) => void; - onTraineeTranscript: (text: string) => void; - onCustomerTranscript: (text: string) => void; - onAudioPlaybackBlocked: (blocked: boolean) => void; - onToolCall: (callId: string, name: string, argsJson: string) => Promise; - onTerminated: () => void; - onError: (message: string) => void; -} - -export interface RealtimePracticeController { - start: (sessionUpdate: Record, sessionId?: string) => Promise; - stop: () => void; - resumeAudio: () => Promise; -} +export interface RealtimePracticeController extends RealtimeBrowserEngine {} export const fetchRealtimePersonas = () => apiRequest({ @@ -82,316 +61,27 @@ export const invokeRealtimeTool = (sessionId: string, callId: string, name: stri timeout: 20000 }); -const isSupported = () => - !isNativeAppRuntime() - && typeof navigator !== 'undefined' - && Boolean(navigator.mediaDevices?.getUserMedia) - && typeof RTCPeerConnection !== 'undefined'; - -const waitForIceComplete = (connection: RTCPeerConnection) => - new Promise((resolve) => { - if (connection.iceGatheringState === 'complete') { - resolve(); - return; - } - const previous = connection.onicegatheringstatechange; - let timeout: ReturnType; - const done = () => { - clearTimeout(timeout); - connection.onicegatheringstatechange = previous; - resolve(); - }; - timeout = setTimeout(done, 3000); - connection.onicegatheringstatechange = () => { - previous?.call(connection, new Event('icegatheringstatechange')); - if (connection.iceGatheringState === 'complete') done(); - }; +export const exchangeRealtimeSdp = async (offerSdp: string, sessionId?: string) => { + const response = await apiRequest({ + url: '/api/train/practice/realtime/sdp', + method: 'POST', + data: { offerSdp, sessionId: sessionId || null }, + timeout: 30000 }); - -const normalizeSdp = (sdp: string) => `${sdp.replace(/\r\n|\r|\n/g, '\r\n').replace(/\r\n$/, '')}\r\n`; -const trimTranscript = (value?: string) => (value || '').trim(); - -const TOOL_CALL_FALLBACK_OUTPUT = '工具暂时不可用，请直接回应员工'; - -export const createRealtimePracticeController = (callbacks: RealtimePracticeCallbacks): RealtimePracticeController => { - let activeAttempt = 0; - let connection: RTCPeerConnection | null = null; - let microphone: MediaStream | null = null; - let speaker: HTMLAudioElement | null = null; - let controlChannel: RTCDataChannel | null = null; - let cancelSessionWait: (() => void) | null = null; - let sessionWaitAttempt: number | null = null; - let toolCallChain: Promise = Promise.resolve(); - - const isCurrent = (attempt: number) => activeAttempt === attempt; - - const releaseResources = () => { - controlChannel?.close(); - controlChannel = null; - microphone?.getTracks().forEach((track) => track.stop()); - microphone = null; - if (speaker) { - speaker.pause(); - speaker.srcObject = null; - speaker = null; - } - connection?.close(); - connection = null; - }; - - // 服务端下发的 session.update 载荷原样转发，仅补 event_id - const sendSessionUpdate = (channel: RTCDataChannel, sessionUpdate: Record) => { - if (channel.readyState !== 'open') return; - channel.send(JSON.stringify({ event_id: `event_${Date.now()}`, ...sessionUpdate })); - }; - - const handleToolCall = (attempt: number, event: RealtimeEvent) => { - const callId = trimTranscript(event.call_id); - const name = trimTranscript(event.name); - if (!callId || !name) return; - const argsJson = typeof event.arguments === 'string' ? event.arguments : '{}'; - // 多次工具调用串行化，避免打断竞态下回传交错 - toolCallChain = toolCallChain.then(async () => { - if (!isCurrent(attempt)) return; - const channel = controlChannel; - if (!channel || channel.readyState !== 'open') return; - callbacks.onStatus('calling-tool'); - let output: string; - try { - output = await callbacks.onToolCall(callId, name, argsJson); - } catch { - output = TOOL_CALL_FALLBACK_OUTPUT; - } - if (!isCurrent(attempt) || channel.readyState !== 'open') return; - channel.send(JSON.stringify({ - event_id: `event_${Date.now()}`, - type: 'conversation.item.create', - item: { type: 'function_call_output', call_id: callId, output } - })); - channel.send(JSON.stringify({ event_id: `event_${Date.now()}`, type: 'response.create' })); - callbacks.onStatus('connected'); - }).catch(() => { - // 链内任何意外抛错不得毒化后续工具调用；本次调用软失败，对话继续 - }); - }; - - const handleEvent = (raw: string, attempt: number, provider: ProviderEventHandlers) => { - if (!isCurrent(attempt)) return; - let event: RealtimeEvent; - try { - event = JSON.parse(raw) as RealtimeEvent; - } catch { - return; - } - if (event.type === 'session.created') provider.onSessionCreated(); - if (event.type === 'session.updated') provider.onSessionUpdated(); - if (event.type === 'input_audio_buffer.speech_started') callbacks.onStatus('listening'); - if (event.type === 'response.created') callbacks.onStatus('speaking'); - if (event.type === 'response.done') callbacks.onStatus('connected'); - if (event.type === 'conversation.item.input_audio_transcription.completed') { - const transcript = trimTranscript(event.transcript); - if (transcript) callbacks.onTraineeTranscript(transcript); - } - if (event.type === 'response.audio_transcript.done') { - const transcript = trimTranscript(event.transcript); - if (transcript) callbacks.onCustomerTranscript(transcript); - } - if (event.type === 'response.function_call_arguments.done') { - handleToolCall(attempt, event); - } - if (event.type === 'error') { - const error = new Error(event.error?.message || '实时陪练连接异常，请改用录音对练'); - callbacks.onError(error.message); - provider.onSessionError(error); - } - }; - - const bindChannel = (channel: RTCDataChannel, attempt: number, provider: ProviderEventHandlers) => { - channel.onmessage = (event) => handleEvent(String(event.data || ''), attempt, provider); - channel.onerror = () => { - if (!isCurrent(attempt)) return; - const error = new Error('实时陪练数据通道异常，请改用录音对练'); - callbacks.onError(error.message); - provider.onSessionError(error); - }; - }; - - const stop = () => { - activeAttempt += 1; - cancelSessionWait?.(); - cancelSessionWait = null; - sessionWaitAttempt = null; - toolCallChain = Promise.resolve(); - releaseResources(); - callbacks.onAudioPlaybackBlocked(false); - callbacks.onStatus('idle'); - }; - - const terminate = (attempt: number, message: string) => { - if (!isCurrent(attempt)) return; - callbacks.onError(message); - stop(); - callbacks.onTerminated(); - }; - - const resumeAudio = async () => { - if (!speaker) return false; - try { - await speaker.play(); - callbacks.onAudioPlaybackBlocked(false); - return true; - } catch { - callbacks.onAudioPlaybackBlocked(true); - return false; - } - }; - - const start = async (sessionUpdate: Record, sessionId?: string) => { - if (!isSupported()) throw new Error('当前设备不支持实时语音，请使用录音对练'); - stop(); - const attempt = ++activeAttempt; - callbacks.onStatus('connecting'); - - let localMicrophone: MediaStream | null = null; - let localConnection: RTCPeerConnection | null = null; - let localChannel: RTCDataChannel | null = null; - let localSpeaker: HTMLAudioElement | null = null; - let readyTimeout: ReturnType | undefined; - let disconnectTimeout: ReturnType | undefined; - let sessionUpdateSent = false; - let resolveReady: (() => void) | undefined; - let rejectReady: ((error: Error) => void) | undefined; - let sessionReady: Promise | null = null; - const releaseLocalResources = () => { - if (disconnectTimeout) clearTimeout(disconnectTimeout); - localChannel?.close(); - localMicrophone?.getTracks().forEach((track) => track.stop()); - if (localSpeaker) { - localSpeaker.pause(); - localSpeaker.srcObject = null; + return response.answerSdp; +}; + +export const createRealtimePracticeController = ( + callbacks: RealtimePracticeCallbacks +): RealtimePracticeController => { + const engine = createRealtimeBrowserEngine(callbacks, exchangeRealtimeSdp); + return { + ...engine, + start: async (sessionUpdate, sessionId) => { + if (isNativeAppRuntime()) { + throw new Error('APP 实时语音应由 WebView 桥接启动'); } - localConnection?.close(); - }; - const ensureCurrent = () => { - if (isCurrent(attempt)) return true; - releaseLocalResources(); - return false; - }; - - try { - localMicrophone = await navigator.mediaDevices.getUserMedia({ audio: true }); - if (!ensureCurrent()) return; - microphone = localMicrophone; - localConnection = new RTCPeerConnection({ iceServers: [] }); - connection = localConnection; - const audioSender = localConnection.addTransceiver('audio', { direction: 'sendrecv' }).sender; - const provider: ProviderEventHandlers = { - onSessionCreated: () => { - if (!isCurrent(attempt) || !localChannel || sessionUpdateSent) return; - sessionUpdateSent = true; - sendSessionUpdate(localChannel, sessionUpdate); - }, - onSessionUpdated: () => { - if (!isCurrent(attempt) || !sessionUpdateSent) return; - const track = localMicrophone?.getAudioTracks()[0] || null; - void audioSender.replaceTrack(track) - .then(() => { - if (isCurrent(attempt)) resolveReady?.(); - }) - .catch((error: unknown) => rejectReady?.(error instanceof Error ? error : new Error('实时陪练麦克风启动失败'))); - }, - onSessionError: (error) => { - rejectReady?.(error); - terminate(attempt, error.message); - } - }; - - localChannel = localConnection.createDataChannel('oai-events'); - controlChannel = localChannel; - bindChannel(localChannel, attempt, provider); - localConnection.ondatachannel = (event) => bindChannel(event.channel, attempt, provider); - localConnection.ontrack = (event) => { - if (!isCurrent(attempt)) return; - const stream = event.streams[0] || new MediaStream([event.track]); - localSpeaker = new Audio(); - speaker = localSpeaker; - localSpeaker.autoplay = true; - localSpeaker.setAttribute('playsinline', 'true'); - localSpeaker.srcObject = stream; - void localSpeaker.play() - .then(() => { - if (isCurrent(attempt)) callbacks.onAudioPlaybackBlocked(false); - }) - .catch(() => { - if (isCurrent(attempt)) callbacks.onAudioPlaybackBlocked(true); - }); - }; - localConnection.onconnectionstatechange = () => { - if (!isCurrent(attempt)) return; - if (localConnection?.connectionState === 'failed' || localConnection?.connectionState === 'closed') { - terminate(attempt, '实时陪练已断开，请改用录音对练'); - return; - } - if (localConnection?.connectionState === 'disconnected') { - disconnectTimeout ??= setTimeout(() => { - if (localConnection?.connectionState === 'disconnected') { - terminate(attempt, '实时陪练已断开，请改用录音对练'); - } - }, 3000); - } else if (disconnectTimeout) { - clearTimeout(disconnectTimeout); - disconnectTimeout = undefined; - } - }; - - const offer = await localConnection.createOffer(); - if (!ensureCurrent()) return; - await localConnection.setLocalDescription(offer); - if (!ensureCurrent()) return; - await waitForIceComplete(localConnection); - if (!ensureCurrent()) return; - const offerSdp = localConnection.localDescription?.sdp; - if (!offerSdp) throw new Error('实时陪练连接请求创建失败'); - const response = await apiRequest({ - url: '/api/train/practice/realtime/sdp', - method: 'POST', - data: { offerSdp: normalizeSdp(offerSdp), sessionId: sessionId || null }, - timeout: 30000 - }); - if (!ensureCurrent()) return; - sessionReady = new Promise((resolve, reject) => { - resolveReady = resolve; - rejectReady = reject; - readyTimeout = setTimeout(() => reject(new Error('实时陪练会话初始化超时，请使用录音对练')), 15000); - }); - void sessionReady.catch(() => undefined); - sessionWaitAttempt = attempt; - cancelSessionWait = () => { - if (readyTimeout) clearTimeout(readyTimeout); - rejectReady?.(new Error('实时陪练已结束')); - }; - await localConnection.setRemoteDescription({ type: 'answer', sdp: normalizeSdp(response.answerSdp) }); - if (!ensureCurrent()) return; - const waitForSessionReady = sessionReady; - if (!waitForSessionReady) throw new Error('实时陪练会话初始化失败'); - await waitForSessionReady; - if (!ensureCurrent()) return; - callbacks.onStatus('connected'); - } catch (error) { - if (!isCurrent(attempt)) { - releaseLocalResources(); - return; - } - stop(); - throw error instanceof Error ? error : new Error('实时陪练连接失败，请改用录音对练'); - } finally { - if (readyTimeout) clearTimeout(readyTimeout); - if (sessionWaitAttempt === attempt) { - sessionWaitAttempt = null; - cancelSessionWait = null; - } + await engine.start(sessionUpdate, sessionId); } }; - - return { start, stop, resumeAudio }; }; diff --git a/mobile-uni/tests/realtime-practice.test.mjs b/mobile-uni/tests/realtime-practice.test.mjs index 5a3090d3..b44f5c82 100644 --- a/mobile-uni/tests/realtime-practice.test.mjs +++ b/mobile-uni/tests/realtime-practice.test.mjs @@ -5,39 +5,52 @@ import test from 'node:test'; const source = (path) => readFile(new URL(path, import.meta.url), 'utf8'); test('实时陪练通过受认证代理接入，前端不携带供应商密钥、不自组会话配置', async () => { - const [service, page, backendService, backendSessionService, backendPersonaRegistry] = await Promise.all([ + const [service, engine, appBridge, page, backendService, backendSessionService, backendPersonaRegistry] = await Promise.all([ source('../src/services/realtime-practice.ts'), + source('../src/services/realtime-browser-engine.js'), + source('../src/components/practice/RealtimePracticeAppBridge.vue'), source('../src/pages/user/practice/index.vue'), source('../../backend/ruoyi-modules/ruoyi-aihr/src/main/java/org/dromara/aihr/service/AihrRealtimePracticeService.java'), source('../../backend/ruoyi-modules/ruoyi-aihr/src/main/java/org/dromara/aihr/service/AihrRealtimeSessionService.java'), source('../../backend/ruoyi-modules/ruoyi-aihr/src/main/java/org/dromara/aihr/service/AihrRealtimePersonaRegistry.java') ]); + const transport = `${service}\n${engine}`; // 传输与事件协议 assert.match(service, /\/api\/train\/practice\/realtime\/sdp/); - assert.match(service, /conversation\.item\.input_audio_transcription\.completed/); - assert.match(service, /response\.audio_transcript\.done/); - assert.match(service, /session\.created/); - assert.match(service, /session\.updated/); - assert.match(service, /activeAttempt/); - assert.match(service, /resumeAudio/); + assert.match(transport, /conversation\.item\.input_audio_transcription\.completed/); + assert.match(transport, /response\.audio_transcript\.done/); + assert.match(transport, /session\.created/); + assert.match(transport, /session\.updated/); + assert.match(transport, /activeAttempt/); + assert.match(transport, /resumeAudio/); // 方案B 架构：session.update 由服务端组装，浏览器只原样转发；instructions/音色不下放前端 assert.match(service, /\/api\/train\/practice\/realtime\/session/); - assert.match(service, /sendSessionUpdate/); - assert.doesNotMatch(service, /instructions/); - assert.doesNotMatch(service, /REALTIME_PRACTICE_VOICES|createRealtimePracticeSessionUpdate|normalizeRealtimePracticeVoice/); - assert.doesNotMatch(service, /semantic_vad/); - assert.doesNotMatch(service, /DASHSCOPE_API_KEY|Authorization:\s*Bearer/); + assert.match(transport, /sendSessionUpdate/); + assert.doesNotMatch(transport, /instructions/); + assert.doesNotMatch(transport, /REALTIME_PRACTICE_VOICES|createRealtimePracticeSessionUpdate|normalizeRealtimePracticeVoice/); + assert.doesNotMatch(transport, /semantic_vad/); + assert.doesNotMatch(transport, /DASHSCOPE_API_KEY|Authorization:\s*Bearer/); // function calling 工具桥：事件分支、串行链、软着陆兜底、回传协议 assert.match(service, /\/api\/train\/practice\/realtime\/tools\/invoke/); - assert.match(service, /response\.function_call_arguments\.done/); - assert.match(service, /function_call_output/); - assert.match(service, /response\.create/); - assert.match(service, /toolCallChain/); - assert.match(service, /工具暂时不可用，请直接回应员工/); - assert.match(service, /calling-tool/); + assert.match(transport, /response\.function_call_arguments\.done/); + assert.match(transport, /function_call_output/); + assert.match(transport, /response\.create/); + assert.match(transport, /toolCallChain/); + assert.match(transport, /工具暂时不可用，请直接回应员工/); + assert.match(transport, /calling-tool/); + + // App-Plus：WebRTC 在 renderjs/WebView 执行，鉴权 SDP 与工具调用回到逻辑层。 + assert.match(appBridge, /lang="renderjs"/); + assert.match(appBridge, /createRealtimeBrowserEngine/); + assert.match(appBridge, /exchangeRealtimeSdp/); + assert.match(appBridge, /invokeRealtimeTool/); + assert.match(appBridge, /owner.*callMethod\('onBridgeEvent'/); + assert.match(page, /RealtimePracticeAppBridge/); + assert.match(page, /isNativeAppRuntime\(\) \? realtimeAppBridge\.value : realtimePractice/); + assert.doesNotMatch(transport, /!isNativeAppRuntime\(\)/); // 服务端是会话配置的唯一组装点 assert.match(backendSessionService, /session\.update/); @@ -65,7 +78,7 @@ test('实时陪练通过受认证代理接入，前端不携带供应商密钥 }); test('工具调用串行化与竞态防护：链式执行、attempt 双检、stop 重置', async () => { - const service = await source('../src/services/realtime-practice.ts'); + const service = await source('../src/services/realtime-browser-engine.js'); // 串行链必须基于上一次调用收尾，且带末端 catch 防毒化 assert.match(service, /toolCallChain = toolCallChain\.then/);