Device Integration

The device uses the peer_id and token delivered by the application server to establish a TiRTC connection, sends start_session, uploads audio, plays downlink audio, and handles events.

This page assumes your device has already completed TiRTC SDK integration. If not, refer to the TiRTC device integration documentation first.

Connect with TiRTC

#include "tiRTC.h"
#include <stdbool.h>
#include <string.h>

static tirtc_conn_t g_ai_conn = NULL;
static bool g_tirtc_ready = false;

static int ai_tirtc_runtime_init(const char *device_id)
{
    if (g_tirtc_ready) {
        return 0;
    }

    int ret = TiRtcInit();
    if (ret != 0) {
        return ret;
    }

    TIRTCCALLBACKS cbs;
    memset(&cbs, 0, sizeof(cbs));
    ret = TiRtcStart(device_id, &cbs);
    if (ret != 0) {
        TiRtcUninit();
        return ret;
    }

    g_tirtc_ready = true;
    return 0;
}

static void on_ai_whip(int err, tirtc_conn_t hconn, void *user_data)
{
    (void)user_data;
    if (err != 0) {
        return;
    }

    g_ai_conn = hconn;
}

int start_ai_tirtc_connect(const char *device_id, const char *peer_id, const char *token)
{
    int ret = 0;

    if (device_id == NULL || device_id[0] == '\0' ||
        peer_id == NULL || peer_id[0] == '\0' ||
        token == NULL || token[0] == '\0') {
        return -1;
    }

    ret = ai_tirtc_runtime_init(device_id);
    if (ret != 0) {
        return ret;
    }

    /* 0 means request submitted; final result is in on_ai_whip callback */
    return TiRtcWhipConnect(peer_id, token, on_ai_whip, NULL);
}

void stop_ai_tirtc_connect(void)
{
    if (g_ai_conn != NULL) {
        TiRtcClose(g_ai_conn);
        g_ai_conn = NULL;
    }

    if (g_tirtc_ready) {
        TiRtcStop();
        TiRtcUninit();
        g_tirtc_ready = false;
    }
}

TiRtcWhipConnect returning 0 only means the request was submitted. Treat the callback with err == 0 as the success signal.

Start the Session

Send start_session after the TiRTC connection is ready. For complete details and defaults, see the Event Protocol.

role_id is required and must be the role ID delivered by the application server for this session.

{
  "jsonrpc": "2.0",
  "id": "start-session-001",
  "method": "start_session",
  "params": {
    "device_id": "DEMO_DEVICE_01",
    "role_id": "role_xxx",
    "input_audio": {
      "codec": "opus",
      "sample_rate": 16000,
      "channels": 1
    },
    "output_audio": {
      "codec": "opus",
      "sample_rate": 16000,
      "channels": 1
    }
  }
}

After the success response, use the returned input_audio and output_audio as the actual audio formats for this session.

Supported Audio Formats and Sample Rates

codec	Sample rate (Hz)	Channels
opus	16000	1
pcm	16000, 8000	1
g711a	16000, 8000	1
amr	8000 (NB), 16000 (WB)	1

Use the input_audio and output_audio fields in the start_session success response as the actual session formats. See Event Protocol for complete details.

Send Audio

#include <stdint.h>
#include <string.h>
#include "tiRTC.h"

static const uint8_t kAiAudioStreamId = 1;

int ai_send_audio_opus(tirtc_conn_t hconn,
                       const void *opus_data,
                       uint32_t len,
                       uint32_t ts_ms)
{
    TIRTCFRAMEINFO fi;
    memset(&fi, 0, sizeof(fi));
    fi.stream_id = kAiAudioStreamId;
    fi.media = TIRTC_AUDIO_OPUS;
    // CRITICAL: Must set audio sample flags consistent with input_audio
    fi.flags = TIRTC_AUDIOSAMPLE_16K16B1C;
    fi.ts = ts_ms;
    fi.length = len;
    return TiRtcSendAudioStream(hconn, &fi, opus_data);
}

Ensure start_session.input_audio, fi.media, and fi.flags match perfectly.

Upload Device Video

To let the agent understand the camera scene, start sending video frames after start_session succeeds. The platform uses selected frames as visual context for the agent, while replies continue to return through downlink audio, captions, and events.

Before adding video, confirm that:

• The camera and encoder can produce complete H.264 or H.265 video frames.
• The active role uses a vision-capable model and has image input enabled.
• The device already handles microphone capture, downlink audio playback, and session cleanup.

Media Stream Convention

AI Chat uses these default stream_id values:

Media	Direction	stream_id
Microphone audio	Device -> platform	1
Response audio	Platform -> device	1
Camera video	Device -> platform	0

Video must use stream_id = 0 to match the platform's current video receiver configuration. Audio and video cannot share one stream_id within the same TiRTC connection.

start_session negotiates audio formats only; it does not return a video format. Use the video encoding and stream_id convention in this section unless Tange provides a different integration-specific contract.

Send Video Frames

After start_session succeeds, start the camera and encoder. Put each complete encoded frame in TIRTCFRAMEINFO, then call TiRtcSendVideoStream().

#include <stdbool.h>
#include <stdint.h>
#include <string.h>

#include "tiRTC.h"

static const uint8_t kAiVideoStreamId = 0;

/**
 * Sends one H.264 video frame to AI Chat.
 *
 * A positive return value means the frame entered the send queue.
 * A negative return value is a TiRTC error code.
 */
int ai_send_video_h264(tirtc_conn_t hconn,
                       const void *frame_data,
                       uint32_t frame_len,
                       uint32_t ts_ms,
                       bool is_key_frame)
{
    TIRTCFRAMEINFO fi;

    if (hconn == NULL || frame_data == NULL || frame_len == 0) {
        return TIRTC_E_INVALID_PARAMETER;
    }

    memset(&fi, 0, sizeof(fi));
    fi.stream_id = kAiVideoStreamId;
    fi.media = TIRTC_VIDEO_H264;
    fi.flags = is_key_frame ? TIRTC_FRAME_FLAG_KEY_FRAME : 0;
    fi.ts = ts_ms;
    fi.length = frame_len;

    return TiRtcSendVideoStream(hconn, &fi, frame_data);
}

For H.265 output, set fi.media to TIRTC_VIDEO_H265. Do not send an MP4 container as one frame payload; extract complete encoded video frames with the encoder or demuxer first.

Key Frames and Send Timing

• The first frame of the video stream must be a key frame with TIRTC_FRAME_FLAG_KEY_FRAME set.
• Set the key-frame flag on every later key frame. Use flags = 0 for non-key frames.
• Use a monotonically increasing millisecond timestamp in fi.ts.
• TIRTC_E_BUSY means the send buffer is full. Drop following non-key frames and ask the encoder for a new key frame before resuming.
• Stop camera capture and encoding when the session ends, the connection closes, or an error occurs. Never keep sending through a stale hconn.

Video Success Signals

• TiRtcSendVideoStream() continues returning positive values without repeated TIRTC_E_BUSY or parameter errors.
• Device logs show video stream_id = 0, codec, key-frame flag, timestamp, and frame length.
• When asked about the current scene, the agent's answer reflects visible camera content.

Frames are sent, but the agent cannot understand the scene

Check that:

• Video uses stream_id = 0; the platform ignores video on other stream IDs.
• fi.media is TIRTC_VIDEO_H264 or TIRTC_VIDEO_H265.
• The first frame and every later key frame carry TIRTC_FRAME_FLAG_KEY_FRAME.
• The active role uses a vision-capable model with image input enabled.
• Each payload is a complete encoded video frame, not an MP4 file, RTP packet, or incomplete fragment.

If sending does not recover after TIRTC_E_BUSY, do not keep queuing ordinary frames. Drop them, request a new key frame from the encoder, and resume from that key frame.

Handle Downlink Audio and Events

The default downlink audio stream_id for AI Chat is 1. Filter incoming audio frames by stream_id == 1, then decode and play according to start_session.output_audio. Implement the event protocol described in Event Protocol, especially caption, round_start, round_end, interrupt, submit_speech, update_config, device_action, and end_session.

Minimal Run-Through Path

1. The server calls /v1/token/aichat and obtains peer_id and token.
2. The device calls TiRtcWhipConnect and confirms err == 0.
3. The device sends start_session and receives a success response with session_id.
4. The device sends a short audio sample and receives downlink caption.
5. For video talkback, the device sends a key frame and later frames on stream_id = 0, then verifies that the agent can understand the current scene.
6. Test interrupt and end_session to confirm audio, camera, encoder, and connection resources are cleaned up correctly.