AI Chat Integration

This guide explains how to connect an IoT device to AI Chat through TiRTC. Your application server gets the role, requests short-lived connection credentials, and sends them to the device. The device then connects with TiRTC, starts a session, sends microphone audio, and plays the generated response.

1. Overall Flow

sequenceDiagram
    autonumber
    participant D as Device
    participant B as Application server
    participant P as Tange platform

    B->>P: Create or query role (console / OpenAPI), obtain role_id
    D->>B: Request AI chat
    B->>B: Verify user, device, service plan
    B->>P: POST /v1/token/aichat
    P-->>B: peer_id, token
    B-->>D: Send peer_id, token
    D->>P: TiRtcWhipConnect(peer_id, token)
    P-->>D: TiRTC connected
    D->>P: start_session(device_id, role_id)
    P-->>D: session_id and audio formats
    D->>P: Upload microphone audio
    P-->>D: Response audio, captions, events

role_id is required. Do not generate peer_id on the device, and never put server signing secrets on the device.

2. AI Talk Console

Before integration, open the AI Talk Console and complete these configurations:

1. Create an application and obtain AppId, AccessKeyId, and SecretKeyId.
2. Register devices and configure their application or product binding.
3. Enable the AI Chat service or plan.
4. Create and configure the AI role in the console.
5. Obtain the role_id used by this session, or bind devices to roles through OpenAPI.

For role management, device-role binding, and device-role query APIs, see the Apifox documentation.

3. Server Integration

The application server's goal is to determine the device and role, request connection credentials from the Tange platform, and securely deliver them to the device.

3.1 Authentication

The signing method is the same as Tange Cloud OpenAPI server-side signing. See 服务端API接口签名算法Demo.

3.2 Request Connection Credentials

Only the connection credential API uses the https://api-tirtc.tange365.com domain:

POST https://api-tirtc.tange365.com/v1/token/aichat
Content-Type: application/json
Authorization: TGV1-HMAC-SHA256 ...
X-Tg-App-Id: {AppId}

{
  "device_id": "DEMO_DEVICE_01",
  "role_id": "role_xxx"
}

Response:

{
  "code": 0,
  "msg": "ok",
  "data": {
    "peer_id": "whips://aichat?device_id=DEMO_DEVICE_01&role_id=role_xxx",
    "token": "v1.{payload}.{signature}"
  }
}

For complete details, see Server API.

3.3 Deliver Credentials to the Device

After obtaining the peer_id and token from the platform, your server should send them to the device via a secure channel (e.g., MQTT, long connection). The device only needs these two values and must never hold the server signing secrets.

Payload example:

{
  "type": "aichat_join",
  "device_id": "DEMO_DEVICE_01",
  "role_id": "role_xxx",
  "peer_id": "whips://aichat?device_id=DEMO_DEVICE_01&role_id=role_xxx",
  "token": "v1.{payload}.{signature}"
}

Security recommendations:

• Never log full tokens.
• Use peer_id and token as soon as possible to avoid expiration.
• Implement business logic to reject, queue, or end existing sessions if a device requests multiple simultaneous sessions.

4. Device Integration

4.1 Connect with TiRTC

#include "tiRTC.h"
#include <stdbool.h>
#include <string.h>

static tirtc_conn_t g_ai_conn = NULL;
static bool g_tirtc_ready = false;

static int ai_tirtc_runtime_init(const char *device_id)
{
    if (g_tirtc_ready) {
        return 0;
    }

    int ret = TiRtcInit();
    if (ret != 0) {
        return ret;
    }

    TIRTCCALLBACKS cbs;
    memset(&cbs, 0, sizeof(cbs));
    ret = TiRtcStart(device_id, &cbs);
    if (ret != 0) {
        TiRtcUninit();
        return ret;
    }

    g_tirtc_ready = true;
    return 0;
}

static void on_ai_whip(int err, tirtc_conn_t hconn, void *user_data)
{
    (void)user_data;
    if (err != 0) {
        return;
    }

    g_ai_conn = hconn;
}

int start_ai_tirtc_connect(const char *device_id, const char *peer_id, const char *token)
{
    int ret = 0;

    if (device_id == NULL || device_id[0] == '\0' ||
        peer_id == NULL || peer_id[0] == '\0' ||
        token == NULL || token[0] == '\0') {
        return -1;
    }

    ret = ai_tirtc_runtime_init(device_id);
    if (ret != 0) {
        return ret;
    }

    /* 0 means request submitted; final result is in on_ai_whip callback */
    return TiRtcWhipConnect(peer_id, token, on_ai_whip, NULL);
}

void stop_ai_tirtc_connect(void)
{
    if (g_ai_conn != NULL) {
        TiRtcClose(g_ai_conn);
        g_ai_conn = NULL;
    }

    if (g_tirtc_ready) {
        TiRtcStop();
        TiRtcUninit();
        g_tirtc_ready = false;
    }
}

TiRtcWhipConnect returning 0 only means the request was submitted. Treat the callback with err == 0 as the success signal.

4.2 Start the Session

Send start_session after the TiRTC connection is ready. For complete details and defaults, see the Event Protocol.

{
  "jsonrpc": "2.0",
  "id": "start-session-001",
  "method": "start_session",
  "params": {
    "device_id": "DEMO_DEVICE_01",
    "role_id": "role_xxx",
    "input_audio": {
      "codec": "opus",
      "sample_rate": 16000,
      "channels": 1
    },
    "output_audio": {
      "codec": "opus",
      "sample_rate": 16000,
      "channels": 1
    }
  }
}

After the success response, use the returned input_audio and output_audio as the actual audio formats for this session, then start uploading audio.

4.3 Send Audio

Send audio frames from the capture thread at a fixed interval. Keep each frame within 100 ms.

#include <stdint.h>
#include <string.h>
#include "tiRTC.h"

static const uint8_t kAiAudioStreamId = 1;

int ai_send_audio_opus(tirtc_conn_t hconn,
                       const void *opus_data,
                       uint32_t len,
                       uint32_t ts_ms)
{
    TIRTCFRAMEINFO fi;
    memset(&fi, 0, sizeof(fi));
    fi.stream_id = kAiAudioStreamId;
    fi.media = TIRTC_AUDIO_OPUS;
    // CRITICAL: Must set audio sample flags consistent with input_audio
    fi.flags = TIRTC_AUDIOSAMPLE_16K16B1C;
    fi.ts = ts_ms;
    fi.length = len;
    return TiRtcSendAudioStream(hconn, &fi, opus_data);
}

Audio Parameter Mapping: Ensure the following three configurations match perfectly to avoid recognition or decoding failures:

1. Intent in start_session.input_audio.
2. fi.media assigned in C code.
3. Audio sample specification fi.flags assigned in C code.

To let the agent understand the device camera, send H.264 or H.265 frames on stream_id = 0 after start_session succeeds. The first video frame must be a key frame. See Device Integration for the complete contract and example.

4.4 Handle Downlink Audio and Events

Decode and play downlink audio according to start_session.output_audio. The playback pipeline should support streaming playback and should stop buffered audio when an interruption or session end event is received.

Implement the event protocol described in Event Protocol, especially caption, round_start, round_end, interrupt, submit_speech, update_config, device_action, and end_session.

4.5 Minimal Run-Through Path

To quickly eliminate environmental and configuration issues, it is recommended to run the minimal end-to-end path:

1. Server Request Test: Send a signed TGV1-HMAC-SHA256 request to /v1/token/aichat using curl with your AppId/AK/SK, and ensure it returns an HTTP 200 status with a valid peer_id and token.
2. Device Connection Test: Provide the device_id, peer_id, and token to start_ai_tirtc_connect on the device and verify that on_ai_whip callback returns err == 0.
3. Session Handshake Test: Send a start_session message as soon as the connection succeeds.
4. Validation Criteria: The device should receive a JSON-RPC success response (with session_id). An error response usually implies incorrect parameters.
5. Audio Loopback Verification: Send a short looping PCM audio track and verify that the device receives downlink caption events and TIRTC_AUDIO_OPUS downlink audio data.