Device Integration

The device uses the peer_id and token delivered by the application server to establish a TiRTC connection, sends start_session, uploads audio, plays downlink audio, and handles events.

This page assumes your device has already completed TiRTC SDK integration. If not, refer to the TiRTC device integration documentation first.

Connect with TiRTC

#include "tiRTC.h"
#include <stdbool.h>
#include <string.h>

static tirtc_conn_t g_ai_conn = NULL;
static bool g_tirtc_ready = false;

static int ai_tirtc_runtime_init(const char *device_id)
{
    if (g_tirtc_ready) {
        return 0;
    }

    int ret = TiRtcInit();
    if (ret != 0) {
        return ret;
    }

    TIRTCCALLBACKS cbs;
    memset(&cbs, 0, sizeof(cbs));
    ret = TiRtcStart(device_id, &cbs);
    if (ret != 0) {
        TiRtcUninit();
        return ret;
    }

    g_tirtc_ready = true;
    return 0;
}

static void on_ai_whip(int err, tirtc_conn_t hconn, void *user_data)
{
    (void)user_data;
    if (err != 0) {
        return;
    }

    g_ai_conn = hconn;
}

int start_ai_tirtc_connect(const char *device_id, const char *peer_id, const char *token)
{
    int ret = 0;

    if (device_id == NULL || device_id[0] == '\0' ||
        peer_id == NULL || peer_id[0] == '\0' ||
        token == NULL || token[0] == '\0') {
        return -1;
    }

    ret = ai_tirtc_runtime_init(device_id);
    if (ret != 0) {
        return ret;
    }

    /* 0 means request submitted; final result is in on_ai_whip callback */
    return TiRtcWhipConnect(peer_id, token, on_ai_whip, NULL);
}

void stop_ai_tirtc_connect(void)
{
    if (g_ai_conn != NULL) {
        TiRtcClose(g_ai_conn);
        g_ai_conn = NULL;
    }

    if (g_tirtc_ready) {
        TiRtcStop();
        TiRtcUninit();
        g_tirtc_ready = false;
    }
}

TiRtcWhipConnect returning 0 only means the request was submitted. Treat the callback with err == 0 as the success signal.

Start the Session

Send start_session after the TiRTC connection is ready. For complete details and defaults, see the Event Protocol.

role_id is required and must be the role ID delivered by the application server for this session.

{
  "jsonrpc": "2.0",
  "id": "start-session-001",
  "method": "start_session",
  "params": {
    "device_id": "DEMO_DEVICE_01",
    "role_id": "role_xxx",
    "input_audio": {
      "codec": "opus",
      "sample_rate": 16000,
      "channels": 1
    },
    "output_audio": {
      "codec": "opus",
      "sample_rate": 16000,
      "channels": 1
    }
  }
}

After the success response, use the returned input_audio and output_audio as the actual audio formats for this session.

Supported Audio Formats and Sample Rates

codec	Sample rate (Hz)	Channels
opus	16000	1
pcm	16000, 8000	1
g711a	16000, 8000	1
amr	8000 (NB), 16000 (WB)	1

Use the input_audio and output_audio fields in the start_session success response as the actual session formats. See Event Protocol for complete details.

Send Audio

#include <stdint.h>
#include <string.h>
#include "tiRTC.h"

static const uint8_t kAiAudioStreamId = 1;

int ai_send_audio_opus(tirtc_conn_t hconn,
                       const void *opus_data,
                       uint32_t len,
                       uint32_t ts_ms)
{
    TIRTCFRAMEINFO fi;
    memset(&fi, 0, sizeof(fi));
    fi.stream_id = kAiAudioStreamId;
    fi.media = TIRTC_AUDIO_OPUS;
    // CRITICAL: Must set audio sample flags consistent with input_audio
    fi.flags = TIRTC_AUDIOSAMPLE_16K16B1C;
    fi.ts = ts_ms;
    fi.length = len;
    return TiRtcSendAudioStream(hconn, &fi, opus_data);
}

Ensure start_session.input_audio, fi.media, and fi.flags match perfectly.

Upload Device Video

To let the agent understand the camera scene, start sending video frames after start_session succeeds. The platform uses selected frames as visual context for the agent, while replies continue to return through downlink audio, captions, and events.

Before adding video, confirm that:

• The camera and encoder can produce complete H.264 or H.265 video frames, or complete JPEG/MJPEG images.
• The active role uses a vision-capable model and has image input enabled.
• The device already handles microphone capture, downlink audio playback, and session cleanup.

Media Stream Convention

AI Chat uses these default stream_id values:

Media	Direction	stream_id
Microphone audio	Device -> platform	1
Response audio	Platform -> device	1
Camera video	Device -> platform	0

Video must use stream_id = 0 to match the platform's current video receiver configuration. Audio and video cannot share one stream_id within the same TiRTC connection.

start_session negotiates audio formats only; it does not return a video format. Use the video encoding and stream_id convention in this section unless Tange provides a different integration-specific contract.

AI Chat supports these TiRTC video media types:

Video format	TIRTCFRAMEINFO.media	Payload contract
H.264	TIRTC_VIDEO_H264	Each payload is one complete encoded frame. Set TIRTC_FRAME_FLAG_KEY_FRAME on key frames.
H.265	TIRTC_VIDEO_H265	Each payload is one complete encoded frame. Set TIRTC_FRAME_FLAG_KEY_FRAME on key frames.
MJPEG (MJPG)	TIRTC_VIDEO_JPEG	Each payload is one complete, independently decodable JPEG image.

Send Video Frames

After start_session succeeds, start the camera and encoder. Put each complete encoded frame in TIRTCFRAMEINFO, then call TiRtcSendVideoStream().

#include <stdbool.h>
#include <stdint.h>
#include <string.h>

#include "tiRTC.h"

static const uint8_t kAiVideoStreamId = 0;

/**
 * Sends one H.264 video frame to AI Chat.
 *
 * A positive return value means the frame entered the send queue.
 * A negative return value is a TiRTC error code.
 */
int ai_send_video_h264(tirtc_conn_t hconn,
                       const void *frame_data,
                       uint32_t frame_len,
                       uint32_t ts_ms,
                       bool is_key_frame)
{
    TIRTCFRAMEINFO fi;

    if (hconn == NULL || frame_data == NULL || frame_len == 0) {
        return TIRTC_E_INVALID_PARAMETER;
    }

    memset(&fi, 0, sizeof(fi));
    fi.stream_id = kAiVideoStreamId;
    fi.media = TIRTC_VIDEO_H264;
    fi.flags = is_key_frame ? TIRTC_FRAME_FLAG_KEY_FRAME : 0;
    fi.ts = ts_ms;
    fi.length = frame_len;

    return TiRtcSendVideoStream(hconn, &fi, frame_data);
}

For H.265 output, set fi.media to TIRTC_VIDEO_H265. For MJPEG/MJPG output, set fi.media to TIRTC_VIDEO_JPEG and make every payload a complete JPEG image. The platform treats each image as an independent frame and does not rely on TIRTC_FRAME_FLAG_KEY_FRAME.

Do not send an MP4 container or a multipart MJPEG stream as one frame payload. Extract complete encoded video frames or individual JPEG images with the encoder or demuxer first.

Key Frames and Send Timing

• For H.264/H.265, the first frame of the video stream must be a key frame with TIRTC_FRAME_FLAG_KEY_FRAME set.
• For H.264/H.265, set the key-frame flag on every later key frame. Use flags = 0 for non-key frames.
• For MJPEG/MJPG, every payload is an independent frame and flags can be 0.
• Use a monotonically increasing millisecond timestamp in fi.ts.
• TIRTC_E_BUSY means the send buffer is full. For H.264/H.265, drop following non-key frames and ask the encoder for a new key frame before resuming. For MJPEG/MJPG, drop queued images and wait for the next complete JPEG.
• Stop camera capture and encoding when the session ends, the connection closes, or an error occurs. Never keep sending through a stale hconn.

Video Success Signals

• TiRtcSendVideoStream() continues returning positive values without repeated TIRTC_E_BUSY or parameter errors.
• Device logs show video stream_id = 0, codec, key-frame flag, timestamp, and frame length.
• When asked about the current scene, the agent's answer reflects visible camera content.

Frames are sent, but the agent cannot understand the scene

Check that:

• Video uses stream_id = 0; the platform ignores video on other stream IDs.
• fi.media is TIRTC_VIDEO_H264, TIRTC_VIDEO_H265, or TIRTC_VIDEO_JPEG.
• For H.264/H.265, the first frame and every later key frame carry TIRTC_FRAME_FLAG_KEY_FRAME. For MJPEG/MJPG, every payload is a complete JPEG image.
• The active role uses a vision-capable model with image input enabled.
• Each payload is a complete encoded video frame or individual JPEG image, not an MP4 file, multipart MJPEG stream, RTP packet, or incomplete fragment.

If sending does not recover after TIRTC_E_BUSY, do not keep queuing old frames. For H.264/H.265, drop ordinary frames, request a new key frame, and resume from it. For MJPEG/MJPG, drop queued images and resume with the next complete JPEG.

Handle Downlink Audio and Events

The default downlink audio stream_id for AI Chat is 1. Filter incoming audio frames by stream_id == 1, then decode and play according to start_session.output_audio. Implement the event protocol described in Event Protocol, especially caption, round_start, round_end, interrupt, submit_speech, update_config, device_action, and end_session.

Minimal Run-Through Path

1. The server calls /v1/token/aichat and obtains peer_id and token.
2. The device calls TiRtcWhipConnect and confirms err == 0.
3. The device sends start_session and receives a success response with session_id.
4. The device sends a short audio sample and receives downlink caption.
5. For video talkback, send H.264/H.265 key frames and later frames, or complete MJPEG/MJPG images, on stream_id = 0, then verify that the agent can understand the current scene.
6. Test interrupt and end_session to confirm audio, camera, encoder, and connection resources are cleaned up correctly.