Show Rive in a call

Your agent can show a Rive file during a call: a character, a quiz, a game or a chart. The person’s phone draws it whenever no video from your agent is arriving, and your agent changes it live through Rive data binding.

Before you start

Input	Requirement
Relay	An Agent Token, and a `call.created` event to answer (answer a call)
Rive file	A `.riv` of at most 10 MB with every image and font embedded, and a View Model whose properties and triggers your agent drives

Show and drive the file

Put the file on your Contact Card

Upload the .riv with the Attachments API, the same way as a profile photo. Then set it with the upload’s id. The names select what the phone draws; leave one out to use the file’s default.

await relay.contactCard.update({
  handle: "my_assistant",
  rive: { attachment_id: upload.id, artboard: "Call", state_machine: "Main", view_model: "Main" },
});

curl -sS -X PATCH 'https://api.relayapp.im/v1/contact_card?handle=<agent handle>' \
  -H "Authorization: Bearer $RELAY_AGENT_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"rive":{"attachment_id":"<upload id>","artboard":"Call","state_machine":"Main","view_model":"Main"}}'

The card, your agent’s chat handles and the Call object then carry rive.file, the URL where Relay hosts the file, so the phone loads it while the call rings. Send {"rive": {"file": "<that URL>", "artboard": "Quiz"}} to change only the names, or "rive": null to remove the file.

Open the rive channel in the call

After connect(), open the channel. Every later call returns the same handle.

TypeScript
Python

await transport.connect();
const rive = await transport.rive();

await call.connect()
rive = await call.rive()

If Relay cannot open the channel, rive() fails with the code media_unavailable, and the call goes on.

Receive Rive in a person-side client

The person’s client must send {"type":"rive"} on its call-room socket to opt into receiving. The agent sends the same frame to publish. Use the person’s connected socket and pc from the call room.Once the agent publishes, Relay may send an offer with track: "rive" to establish the SCTP transport. Answer it like any other offer; Relay then sends a rive frame with the channel id. Add these cases to your existing serialized room-frame handler:

Person-side WebRTC

async function onRoomFrame(frame) {
  if (frame.type === "offer") {
    await pc.setRemoteDescription(frame.session_description);
    const answer = await pc.createAnswer();
    await pc.setLocalDescription(answer);
    socket.send(JSON.stringify({
      type: "answer",
      session_description: { type: answer.type, sdp: answer.sdp },
    }));
  } else if (frame.type === "rive") {
    const channel = pc.createDataChannel("rive", {
      negotiated: true,
      id: frame.id,
      ordered: false,
      maxRetransmits: 0,
    });
    channel.onmessage = ({ data }) => { /* Apply JSON.parse(data) to Rive. */ };
  }
}

socket.send(JSON.stringify({ type: "rive" }));

After a restart onto a new Session, Relay sends a new id. Open the channel again with that id on the current peer connection.

Change the file and hear the person

set writes View Model properties by name or path, trigger fires a trigger, and show switches to another Relay-hosted file with optional starting values. The phone sends back what the person changes.

rive.set({ mood: "happy", "score/value": 3 });
rive.trigger("wave");
rive.on("view_model", (values) => { /* the person changed { answer: "B" } */ });
rive.on("trigger", (name) => { /* the person fired a trigger */ });

Python uses the same names: rive.set({...}), rive.trigger("wave"), rive.on("view_model", handler).

Time changes to your agent's speech

writeAudio resolves with where that audio starts on your agent’s audio track, in milliseconds. Pass that time plus an offset inside the audio as at, and the phone applies the change when that sample plays. Audio held until the person can hear it gets its real start too.

import { visemesFromAlignment } from "@relaymessenger/sdk/calls";

const start = await transport.writeAudio(frame);
if (start !== undefined) {
  // Character timings for that audio, such as ElevenLabs' `alignment`.
  for (const cue of visemesFromAlignment(alignment)) {
    rive.set({ viseme: cue.viseme }, { at: start + cue.t });
  }
}

visemesFromAlignment turns character timings into Preston Blair’s ten mouth shapes, from 0 (rest) to 9 (W and Q), for a viseme number in your file. In Python, write_audio returns the start once the person receives your agent’s audio, and None before that.

Let your framework drive the mouth

Framework	What sends the timing
Pipecat	`RelayRiveProcessor(transport)`, placed right after text-to-speech, turns word timestamps into `viseme` and the bot’s speaking frames into `speaking`.
LiveKit Agents, Python	`await RelayRive().start(session, call)` sets `speaking` for each reply. With `AgentSession(use_tts_aligned_transcript=True)` and a text-to-speech service that reports word times, it also sends `viseme`.
LiveKit Agents, TypeScript	`await new RelayRive().start(session, call)` sets `speaking` for each reply. Send `viseme` yourself with `rive.set`.
ElevenLabs Agents	`@relaymessenger/elevenlabs` sends `viseme` and `speaking` from each reply’s alignment.

Messages

Each message is JSON of at most 1 KB. The channel is unordered and does not resend, so a lost message is gone. A set replaces only the properties it names: a later value for the same property corrects a lost one, and nothing resends a lost trigger. After a media restart the transport opens the channel again and resends the last show and the latest value of every property.

Message	Meaning
`{"t": 1840, "view_model": {"viseme": 3, "speaking": true}}`	Set View Model properties when your agent’s audio reaches 1840 ms.
`{"trigger": "nod"}`	Fire a trigger at once.
`{"file": "<Relay-hosted .riv>", "artboard": "Quiz", "view_model": {"question": "Capital of France?"}}`	Show another file with starting values.
`{"view_model": {"answer": "B"}}`	From the phone: a value the person changed.
`{"trigger": "tapped_start"}`	From the phone: a trigger the person fired.

Overview

Getting started

Integrations

Coding agents

Frameworks and protocols

Voice & video

Build it yourself

Messaging

Interactions

Chats

Calls

Agents

Contacts

Message requests

Webhooks

WebSocket

Console

Reliability

Resources

Show Rive in a call

Before you start

Show and drive the file

Let your framework drive the mouth

Messages

Next steps

​Before you start

​Show and drive the file

​Let your framework drive the mouth

​Messages

​Next steps

Before you start

Show and drive the file

Let your framework drive the mouth

Messages

Next steps