Server events

These are events emitted from the OpenAI Realtime WebSocket server to the client.

conversation.created

Returned when a conversation is created. Emitted right after session creation.

conversation: object { id, object }

The conversation resource.

id: optional string

The unique ID of the conversation.

object: optional string

The object type, must be realtime.conversation.

event_id: string

The unique ID of the server event.

type: "conversation.created"

The event type, must be conversation.created.

OBJECT

conversation.created

{
    "event_id": "event_9101",
    "type": "conversation.created",
    "conversation": {
        "id": "conv_001",
        "object": "realtime.conversation"
    }
}

conversation.item.created

Returned when a conversation item is created. There are several scenarios that produce this event:

• The server is generating a Response, which if successful will produce either one or two Items, which will be of type message (role assistant) or type function_call.
• The input audio buffer has been committed, either by the client or the server (in server_vad mode). The server will take the content of the input audio buffer and add it to a new user message Item.
• The client has sent a conversation.item.create event to add a new Item to the Conversation.

event_id: string

The unique ID of the server event.

item: ConversationItem

A single item within a Realtime conversation.

One of the following:

RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

content: array of object { text, type }

The content of the message.

text: optional string

The text content.

type: optional "input_text"

The content type. Always input_text for system messages.

role: "system"

The role of the message sender. Always system.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemUserMessage object { content, role, type, 3 more }

A user message item in a Realtime conversation.

content: array of object { audio, detail, image_url, 3 more }

The content of the message.

audio: optional string

Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

detail: optional "auto" or "low" or "high"

The detail level of the image (for input_image). auto will default to high.

One of the following:

"auto"

"low"

"high"

image_url: optional string

Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

formaturi

text: optional string

The text content (for input_text).

transcript: optional string

Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

type: optional "input_text" or "input_audio" or "input_image"

The content type (input_text, input_audio, or input_image).

One of the following:

"input_text"

"input_audio"

"input_image"

role: "user"

The role of the message sender. Always user.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

An assistant message item in a Realtime conversation.

content: array of object { audio, text, transcript, type }

The content of the message.

audio: optional string

Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

text: optional string

The text content.

transcript: optional string

The transcript of the audio content, this will always be present if the output type is audio.

type: optional "output_text" or "output_audio"

The content type, output_text or output_audio depending on the session output_modalities configuration.

One of the following:

"output_text"

"output_audio"

role: "assistant"

The role of the message sender. Always assistant.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

A function call item in a Realtime conversation.

arguments: string

The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

name: string

The name of the function being called.

type: "function_call"

The type of the item. Always function_call.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

call_id: optional string

The ID of the function call.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

A function call output item in a Realtime conversation.

call_id: string

The ID of the function call this output is for.

output: string

The output of the function call, this is free text and can contain any information or simply be empty.

type: "function_call_output"

The type of the item. Always function_call_output.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }

A Realtime item responding to an MCP approval request.

id: string

The unique ID of the approval response.

approval_request_id: string

The ID of the approval request being answered.

approve: boolean

Whether the request was approved.

type: "mcp_approval_response"

The type of the item. Always mcp_approval_response.

reason: optional string or null

Optional reason for the decision.

RealtimeMcpListTools object { server_label, tools, type, id }

A Realtime item listing tools available on an MCP server.

server_label: string

The label of the MCP server.

tools: array of object { input_schema, name, annotations, description }

The tools available on the server.

input_schema: unknown

The JSON schema describing the tool's input.

name: string

The name of the tool.

annotations: optional unknown or null

Additional annotations about the tool.

description: optional string or null

The description of the tool.

type: "mcp_list_tools"

The type of the item. Always mcp_list_tools.

id: optional string

The unique ID of the list.

RealtimeMcpToolCall object { id, arguments, name, 5 more }

A Realtime item representing an invocation of a tool on an MCP server.

id: string

The unique ID of the tool call.

arguments: string

A JSON string of the arguments passed to the tool.

name: string

The name of the tool that was run.

server_label: string

The label of the MCP server running the tool.

type: "mcp_call"

The type of the item. Always mcp_call.

approval_request_id: optional string or null

The ID of an associated approval request, if any.

error: optional RealtimeMcpProtocolError { code, message, type } or RealtimeMcpToolExecutionError { message, type } or RealtimeMcphttpError { code, message, type } or null

The error from the tool call, if any.

One of the following:

RealtimeMcpProtocolError object { code, message, type }

code: number

message: string

type: "protocol_error"

RealtimeMcpToolExecutionError object { message, type }

message: string

type: "tool_execution_error"

RealtimeMcphttpError object { code, message, type }

code: number

message: string

type: "http_error"

output: optional string or null

The output from the tool call.

RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }

A Realtime item requesting human approval of a tool invocation.

id: string

The unique ID of the approval request.

arguments: string

A JSON string of arguments for the tool.

name: string

The name of the tool to run.

server_label: string

The label of the MCP server making the request.

type: "mcp_approval_request"

The type of the item. Always mcp_approval_request.

type: "conversation.item.created"

The event type, must be conversation.item.created.

previous_item_id: optional string or null

The ID of the preceding item in the Conversation context, allows the client to understand the order of the conversation. Can be null if the item has no predecessor.

OBJECT

conversation.item.created

{
    "event_id": "event_1920",
    "type": "conversation.item.created",
    "previous_item_id": "msg_002",
    "item": {
        "id": "msg_003",
        "object": "realtime.item",
        "type": "message",
        "status": "completed",
        "role": "user",
        "content": []
    }
}

conversation.item.deleted

Returned when an item in the conversation is deleted by the client with a conversation.item.delete event. This event is used to synchronize the server's understanding of the conversation history with the client's view.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the item that was deleted.

type: "conversation.item.deleted"

The event type, must be conversation.item.deleted.

OBJECT

conversation.item.deleted

{
    "event_id": "event_2728",
    "type": "conversation.item.deleted",
    "item_id": "msg_005"
}

conversation.item.input_audio_transcription.completed

This event is the output of audio transcription for user audio written to the user audio buffer. Transcription begins when the input audio buffer is committed by the client or server (when VAD is enabled). Transcription runs asynchronously with Response creation, so this event may come before or after the Response events.

Realtime API models accept audio natively, and thus input transcription is a separate process run on a separate ASR (Automatic Speech Recognition) model. The transcript may diverge somewhat from the model's interpretation, and should be treated as a rough guide.

content_index: number

The index of the content part containing the audio.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the item containing the audio that is being transcribed.

transcript: string

The transcribed text.

type: "conversation.item.input_audio_transcription.completed"

The event type, must be conversation.item.input_audio_transcription.completed.

usage: object { input_tokens, output_tokens, total_tokens, 2 more } or object { seconds, type }

Usage statistics for the transcription, this is billed according to the ASR model's pricing rather than the realtime model's pricing.

One of the following:

Tokens object { input_tokens, output_tokens, total_tokens, 2 more }

Usage statistics for models billed by token usage.

input_tokens: number

Number of input tokens billed for this request.

output_tokens: number

Number of output tokens generated.

total_tokens: number

Total number of tokens used (input + output).

type: "tokens"

The type of the usage object. Always tokens for this variant.

input_token_details: optional object { audio_tokens, text_tokens }

Details about the input tokens billed for this request.

audio_tokens: optional number

Number of audio tokens billed for this request.

text_tokens: optional number

Number of text tokens billed for this request.

Duration object { seconds, type }

Usage statistics for models billed by audio input duration.

seconds: number

Duration of the input audio in seconds.

formatdouble

type: "duration"

The type of the usage object. Always duration for this variant.

languages: optional array of TranscriptionLanguage { code }

The languages detected in the audio. Returned by gpt-transcribe. An empty array indicates that no language could be reliably detected.

code: string

The code of a language detected in the audio.

logprobs: optional array of LogProbProperties { token, bytes, logprob } or null

The log probabilities of the transcription.

token: string

The token that was used to generate the log probability.

bytes: array of number

The bytes that were used to generate the log probability.

logprob: number

The log probability of the token.

OBJECT

conversation.item.input_audio_transcription.completed

{
  "type": "conversation.item.input_audio_transcription.completed",
  "event_id": "event_CCXGRvtUVrax5SJAnNOWZ",
  "item_id": "item_CCXGQ4e1ht4cOraEYcuR2",
  "content_index": 0,
  "transcript": "Hey, can you hear me?",
  "usage": {
    "type": "tokens",
    "total_tokens": 22,
    "input_tokens": 13,
    "input_token_details": {
      "text_tokens": 0,
      "audio_tokens": 13
    },
    "output_tokens": 9
  }
}

conversation.item.input_audio_transcription.delta

Returned when the text value of an input audio transcription content part is updated with incremental transcription results.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the item containing the audio that is being transcribed.

type: "conversation.item.input_audio_transcription.delta"

The event type, must be conversation.item.input_audio_transcription.delta.

content_index: optional number

The index of the content part in the item's content array.

delta: optional string

The text delta.

logprobs: optional array of LogProbProperties { token, bytes, logprob } or null

The log probabilities of the transcription. These can be enabled by configurating the session with "include": ["item.input_audio_transcription.logprobs"]. Each entry in the array corresponds a log probability of which token would be selected for this chunk of transcription. This can help to identify if it was possible there were multiple valid options for a given chunk of transcription.

token: string

The token that was used to generate the log probability.

bytes: array of number

The bytes that were used to generate the log probability.

logprob: number

The log probability of the token.

OBJECT

conversation.item.input_audio_transcription.delta

{
  "type": "conversation.item.input_audio_transcription.delta",
  "event_id": "event_CCXGRxsAimPAs8kS2Wc7Z",
  "item_id": "item_CCXGQ4e1ht4cOraEYcuR2",
  "content_index": 0,
  "delta": "Hey",
  "obfuscation": "aLxx0jTEciOGe"
}

conversation.item.input_audio_transcription.failed

Returned when input audio transcription is configured, and a transcription request for a user message failed. These events are separate from other error events so that the client can identify the related Item.

content_index: number

The index of the content part containing the audio.

error: object { code, message, param, type }

Details of the transcription error.

code: optional string

Error code, if any.

message: optional string

A human-readable error message.

param: optional string

Parameter related to the error, if any.

type: optional string

The type of error.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the user message item.

type: "conversation.item.input_audio_transcription.failed"

The event type, must be conversation.item.input_audio_transcription.failed.

OBJECT

conversation.item.input_audio_transcription.failed

{
    "event_id": "event_2324",
    "type": "conversation.item.input_audio_transcription.failed",
    "item_id": "msg_003",
    "content_index": 0,
    "error": {
        "type": "transcription_error",
        "code": "audio_unintelligible",
        "message": "The audio could not be transcribed.",
        "param": null
    }
}

conversation.item.retrieved

Returned when a conversation item is retrieved with conversation.item.retrieve. This is provided as a way to fetch the server's representation of an item, for example to get access to the post-processed audio data after noise cancellation and VAD. It includes the full content of the Item, including audio data.

event_id: string

The unique ID of the server event.

item: ConversationItem

A single item within a Realtime conversation.

One of the following:

RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

content: array of object { text, type }

The content of the message.

text: optional string

The text content.

type: optional "input_text"

The content type. Always input_text for system messages.

role: "system"

The role of the message sender. Always system.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemUserMessage object { content, role, type, 3 more }

A user message item in a Realtime conversation.

content: array of object { audio, detail, image_url, 3 more }

The content of the message.

audio: optional string

Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

detail: optional "auto" or "low" or "high"

The detail level of the image (for input_image). auto will default to high.

One of the following:

"auto"

"low"

"high"

image_url: optional string

Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

formaturi

text: optional string

The text content (for input_text).

transcript: optional string

Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

type: optional "input_text" or "input_audio" or "input_image"

The content type (input_text, input_audio, or input_image).

One of the following:

"input_text"

"input_audio"

"input_image"

role: "user"

The role of the message sender. Always user.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

An assistant message item in a Realtime conversation.

content: array of object { audio, text, transcript, type }

The content of the message.

audio: optional string

Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

text: optional string

The text content.

transcript: optional string

The transcript of the audio content, this will always be present if the output type is audio.

type: optional "output_text" or "output_audio"

The content type, output_text or output_audio depending on the session output_modalities configuration.

One of the following:

"output_text"

"output_audio"

role: "assistant"

The role of the message sender. Always assistant.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

A function call item in a Realtime conversation.

arguments: string

The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

name: string

The name of the function being called.

type: "function_call"

The type of the item. Always function_call.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

call_id: optional string

The ID of the function call.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

A function call output item in a Realtime conversation.

call_id: string

The ID of the function call this output is for.

output: string

The output of the function call, this is free text and can contain any information or simply be empty.

type: "function_call_output"

The type of the item. Always function_call_output.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }

A Realtime item responding to an MCP approval request.

id: string

The unique ID of the approval response.

approval_request_id: string

The ID of the approval request being answered.

approve: boolean

Whether the request was approved.

type: "mcp_approval_response"

The type of the item. Always mcp_approval_response.

reason: optional string or null

Optional reason for the decision.

RealtimeMcpListTools object { server_label, tools, type, id }

A Realtime item listing tools available on an MCP server.

server_label: string

The label of the MCP server.

tools: array of object { input_schema, name, annotations, description }

The tools available on the server.

input_schema: unknown

The JSON schema describing the tool's input.

name: string

The name of the tool.

annotations: optional unknown or null

Additional annotations about the tool.

description: optional string or null

The description of the tool.

type: "mcp_list_tools"

The type of the item. Always mcp_list_tools.

id: optional string

The unique ID of the list.

RealtimeMcpToolCall object { id, arguments, name, 5 more }

A Realtime item representing an invocation of a tool on an MCP server.

id: string

The unique ID of the tool call.

arguments: string

A JSON string of the arguments passed to the tool.

name: string

The name of the tool that was run.

server_label: string

The label of the MCP server running the tool.

type: "mcp_call"

The type of the item. Always mcp_call.

approval_request_id: optional string or null

The ID of an associated approval request, if any.

error: optional RealtimeMcpProtocolError { code, message, type } or RealtimeMcpToolExecutionError { message, type } or RealtimeMcphttpError { code, message, type } or null

The error from the tool call, if any.

One of the following:

RealtimeMcpProtocolError object { code, message, type }

code: number

message: string

type: "protocol_error"

RealtimeMcpToolExecutionError object { message, type }

message: string

type: "tool_execution_error"

RealtimeMcphttpError object { code, message, type }

code: number

message: string

type: "http_error"

output: optional string or null

The output from the tool call.

RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }

A Realtime item requesting human approval of a tool invocation.

id: string

The unique ID of the approval request.

arguments: string

A JSON string of arguments for the tool.

name: string

The name of the tool to run.

server_label: string

The label of the MCP server making the request.

type: "mcp_approval_request"

The type of the item. Always mcp_approval_request.

type: "conversation.item.retrieved"

The event type, must be conversation.item.retrieved.

OBJECT

conversation.item.retrieved

{}

conversation.item.truncated

Returned when an earlier assistant audio message item is truncated by the client with a conversation.item.truncate event. This event is used to synchronize the server's understanding of the audio with the client's playback.

This action will truncate the audio and remove the server-side text transcript to ensure there is no text in the context that hasn't been heard by the user.

audio_end_ms: number

The duration up to which the audio was truncated, in milliseconds.

content_index: number

The index of the content part that was truncated.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the assistant message item that was truncated.

type: "conversation.item.truncated"

The event type, must be conversation.item.truncated.

OBJECT

conversation.item.truncated

{
    "event_id": "event_2526",
    "type": "conversation.item.truncated",
    "item_id": "msg_004",
    "content_index": 0,
    "audio_end_ms": 1500
}

error

Returned when an error occurs, which could be a client problem or a server problem. Most errors are recoverable and the session will stay open, we recommend to implementors to monitor and log error messages by default.

error: RealtimeError { message, type, code, 2 more }

Details of the error.

message: string

A human-readable error message.

type: string

The type of error (e.g., "invalid_request_error", "server_error").

code: optional string or null

Error code, if any.

event_id: optional string or null

The event_id of the client event that caused the error, if applicable.

param: optional string or null

Parameter related to the error, if any.

event_id: string

The unique ID of the server event.

type: "error"

The event type, must be error.

OBJECT

error

{
    "event_id": "event_890",
    "type": "error",
    "error": {
        "type": "invalid_request_error",
        "code": "invalid_event",
        "message": "The 'type' field is missing.",
        "param": null,
        "event_id": "event_567"
    }
}

input_audio_buffer.cleared

Returned when the input audio buffer is cleared by the client with a input_audio_buffer.clear event.

event_id: string

The unique ID of the server event.

type: "input_audio_buffer.cleared"

The event type, must be input_audio_buffer.cleared.

OBJECT

input_audio_buffer.cleared

{
    "event_id": "event_1314",
    "type": "input_audio_buffer.cleared"
}

input_audio_buffer.committed

Returned when an input audio buffer is committed, either by the client or automatically in server VAD mode. The item_id property is the ID of the user message item that will be created, thus a conversation.item.created event will also be sent to the client.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the user message item that will be created.

type: "input_audio_buffer.committed"

The event type, must be input_audio_buffer.committed.

previous_item_id: optional string or null

The ID of the preceding item after which the new item will be inserted. Can be null if the item has no predecessor.

OBJECT

input_audio_buffer.committed

{
    "event_id": "event_1121",
    "type": "input_audio_buffer.committed",
    "previous_item_id": "msg_001",
    "item_id": "msg_002"
}

input_audio_buffer.dtmf_event_received

SIP Only: Returned when an DTMF event is received. A DTMF event is a message that represents a telephone keypad press (0–9, *, #, A–D). The event property is the keypad that the user press. The received_at is the UTC Unix Timestamp that the server received the event.

event: string

The telephone keypad that was pressed by the user.

received_at: number

UTC Unix Timestamp when DTMF Event was received by server.

type: "input_audio_buffer.dtmf_event_received"

The event type, must be input_audio_buffer.dtmf_event_received.

OBJECT

input_audio_buffer.dtmf_event_received

{
    "type":" input_audio_buffer.dtmf_event_received",
    "event": "9",
    "received_at": 1763605109,
}

input_audio_buffer.speech_started

Sent by the server when in server_vad mode to indicate that speech has been detected in the audio buffer. This can happen any time audio is added to the buffer (unless speech is already detected). The client may want to use this event to interrupt audio playback or provide visual feedback to the user.

The client should expect to receive a input_audio_buffer.speech_stopped event when speech stops. The item_id property is the ID of the user message item that will be created when speech stops and will also be included in the input_audio_buffer.speech_stopped event (unless the client manually commits the audio buffer during VAD activation).

audio_start_ms: number

Milliseconds from the start of all audio written to the buffer during the session when speech was first detected. This will correspond to the beginning of audio sent to the model, and thus includes the prefix_padding_ms configured in the Session.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the user message item that will be created when speech stops.

type: "input_audio_buffer.speech_started"

The event type, must be input_audio_buffer.speech_started.

OBJECT

input_audio_buffer.speech_started

{
    "event_id": "event_1516",
    "type": "input_audio_buffer.speech_started",
    "audio_start_ms": 1000,
    "item_id": "msg_003"
}

input_audio_buffer.speech_stopped

Returned in server_vad mode when the server detects the end of speech in the audio buffer. The server will also send an conversation.item.created event with the user message item that is created from the audio buffer.

audio_end_ms: number

Milliseconds since the session started when speech stopped. This will correspond to the end of audio sent to the model, and thus includes the min_silence_duration_ms configured in the Session.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the user message item that will be created.

type: "input_audio_buffer.speech_stopped"

The event type, must be input_audio_buffer.speech_stopped.

OBJECT

input_audio_buffer.speech_stopped

{
    "event_id": "event_1718",
    "type": "input_audio_buffer.speech_stopped",
    "audio_end_ms": 2000,
    "item_id": "msg_003"
}

rate_limits.updated

Emitted at the beginning of a Response to indicate the updated rate limits. When a Response is created some tokens will be "reserved" for the output tokens, the rate limits shown here reflect that reservation, which is then adjusted accordingly once the Response is completed.

event_id: string

The unique ID of the server event.

rate_limits: array of object { limit, name, remaining, reset_seconds }

List of rate limit information.

limit: optional number

The maximum allowed value for the rate limit.

name: optional "requests" or "tokens"

The name of the rate limit (requests, tokens).

One of the following:

"requests"

"tokens"

remaining: optional number

The remaining value before the limit is reached.

reset_seconds: optional number

Seconds until the rate limit resets.

type: "rate_limits.updated"

The event type, must be rate_limits.updated.

OBJECT

rate_limits.updated

{
    "event_id": "event_5758",
    "type": "rate_limits.updated",
    "rate_limits": [
        {
            "name": "requests",
            "limit": 1000,
            "remaining": 999,
            "reset_seconds": 60
        },
        {
            "name": "tokens",
            "limit": 50000,
            "remaining": 49950,
            "reset_seconds": 60
        }
    ]
}

response.output_audio.delta

Returned when the model-generated audio is updated.

content_index: number

The index of the content part in the item's content array.

delta: string

Base64-encoded audio data delta.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the item.

output_index: number

The index of the output item in the response.

response_id: string

The ID of the response.

type: "response.output_audio.delta"

The event type, must be response.output_audio.delta.

OBJECT

response.output_audio.delta

{
    "event_id": "event_4950",
    "type": "response.output_audio.delta",
    "response_id": "resp_001",
    "item_id": "msg_008",
    "output_index": 0,
    "content_index": 0,
    "delta": "Base64EncodedAudioDelta"
}

response.output_audio.done

Returned when the model-generated audio is done. Also emitted when a Response is interrupted, incomplete, or cancelled.

content_index: number

The index of the content part in the item's content array.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the item.

output_index: number

The index of the output item in the response.

response_id: string

The ID of the response.

type: "response.output_audio.done"

The event type, must be response.output_audio.done.

OBJECT

response.output_audio.done

{
    "event_id": "event_5152",
    "type": "response.output_audio.done",
    "response_id": "resp_001",
    "item_id": "msg_008",
    "output_index": 0,
    "content_index": 0
}

response.output_audio_transcript.delta

Returned when the model-generated transcription of audio output is updated.

content_index: number

The index of the content part in the item's content array.

delta: string

The transcript delta.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the item.

output_index: number

The index of the output item in the response.

response_id: string

The ID of the response.

type: "response.output_audio_transcript.delta"

The event type, must be response.output_audio_transcript.delta.

OBJECT

response.output_audio_transcript.delta

{
    "event_id": "event_4546",
    "type": "response.output_audio_transcript.delta",
    "response_id": "resp_001",
    "item_id": "msg_008",
    "output_index": 0,
    "content_index": 0,
    "delta": "Hello, how can I a"
}

response.output_audio_transcript.done

Returned when the model-generated transcription of audio output is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

content_index: number

The index of the content part in the item's content array.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the item.

output_index: number

The index of the output item in the response.

response_id: string

The ID of the response.

transcript: string

The final transcript of the audio.

type: "response.output_audio_transcript.done"

The event type, must be response.output_audio_transcript.done.

OBJECT

response.output_audio_transcript.done

{
    "event_id": "event_4748",
    "type": "response.output_audio_transcript.done",
    "response_id": "resp_001",
    "item_id": "msg_008",
    "output_index": 0,
    "content_index": 0,
    "transcript": "Hello, how can I assist you today?"
}

response.content_part.added

Returned when a new content part is added to an assistant message item during response generation.

content_index: number

The index of the content part in the item's content array.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the item to which the content part was added.

output_index: number

The index of the output item in the response.

part: object { audio, text, transcript, type }

The content part that was added.

audio: optional string

Base64-encoded audio data (if type is "audio").

text: optional string

The text content (if type is "text").

transcript: optional string

The transcript of the audio (if type is "audio").

type: optional "audio" or "text"

The content type ("text", "audio").

One of the following:

"audio"

"text"

response_id: string

The ID of the response.

type: "response.content_part.added"

The event type, must be response.content_part.added.

OBJECT

response.content_part.added

{
    "event_id": "event_3738",
    "type": "response.content_part.added",
    "response_id": "resp_001",
    "item_id": "msg_007",
    "output_index": 0,
    "content_index": 0,
    "part": {
        "type": "text",
        "text": ""
    }
}

response.content_part.done

Returned when a content part is done streaming in an assistant message item. Also emitted when a Response is interrupted, incomplete, or cancelled.

content_index: number

The index of the content part in the item's content array.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the item.

output_index: number

The index of the output item in the response.

part: object { audio, text, transcript, type }

The content part that is done.

audio: optional string

Base64-encoded audio data (if type is "audio").

text: optional string

The text content (if type is "text").

transcript: optional string

The transcript of the audio (if type is "audio").

type: optional "audio" or "text"

The content type ("text", "audio").

One of the following:

"audio"

"text"

response_id: string

The ID of the response.

type: "response.content_part.done"

The event type, must be response.content_part.done.

OBJECT

response.content_part.done

{
    "event_id": "event_3940",
    "type": "response.content_part.done",
    "response_id": "resp_001",
    "item_id": "msg_007",
    "output_index": 0,
    "content_index": 0,
    "part": {
        "type": "text",
        "text": "Sure, I can help with that."
    }
}

response.created

Returned when a new Response is created. The first event of response creation, where the response is in an initial state of in_progress.

event_id: string

The unique ID of the server event.

response: RealtimeResponse { id, audio, conversation_id, 8 more }

The response resource.

id: optional string

The unique ID of the response, will look like resp_1234.

audio: optional object { output }

Configuration for audio output.

output: optional object { format, voice }

format: optional RealtimeAudioFormats

The format of the output audio.

One of the following:

PCMAudio object { rate, type }

The PCM audio format. Only a 24kHz sample rate is supported.

rate: optional 24000

The sample rate of the audio. Always 24000.

type: optional "audio/pcm"

The audio format. Always audio/pcm.

PCMUAudio object { type }

The G.711 μ-law format.

type: optional "audio/pcmu"

The audio format. Always audio/pcmu.

PCMAAudio object { type }

The G.711 A-law format.

type: optional "audio/pcma"

The audio format. Always audio/pcma.

voice: optional string or "alloy" or "ash" or "ballad" or 7 more

The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

One of the following:

string

"alloy" or "ash" or "ballad" or 7 more

The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

One of the following:

"alloy"

"ash"

"ballad"

"coral"

"echo"

"sage"

"shimmer"

"verse"

"marin"

"cedar"

conversation_id: optional string

Which conversation the response is added to, determined by the conversation field in the response.create event. If auto, the response will be added to the default conversation and the value of conversation_id will be an id like conv_1234. If none, the response will not be added to any conversation and the value of conversation_id will be null. If responses are being triggered automatically by VAD the response will be added to the default conversation

max_output_tokens: optional number or "inf"

Maximum number of output tokens for a single assistant response, inclusive of tool calls, that was used in this response.

One of the following:

number

"inf"

metadata: optional Metadata or null

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

object: optional "realtime.response"

The object type, must be realtime.response.

output: optional array of ConversationItem

The list of output items generated by the response.

One of the following:

RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

content: array of object { text, type }

The content of the message.

text: optional string

The text content.

type: optional "input_text"

The content type. Always input_text for system messages.

role: "system"

The role of the message sender. Always system.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemUserMessage object { content, role, type, 3 more }

A user message item in a Realtime conversation.

content: array of object { audio, detail, image_url, 3 more }

The content of the message.

audio: optional string

Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

detail: optional "auto" or "low" or "high"

The detail level of the image (for input_image). auto will default to high.

One of the following:

"auto"

"low"

"high"

image_url: optional string

Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

formaturi

text: optional string

The text content (for input_text).

transcript: optional string

Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

type: optional "input_text" or "input_audio" or "input_image"

The content type (input_text, input_audio, or input_image).

One of the following:

"input_text"

"input_audio"

"input_image"

role: "user"

The role of the message sender. Always user.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

An assistant message item in a Realtime conversation.

content: array of object { audio, text, transcript, type }

The content of the message.

audio: optional string

Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

text: optional string

The text content.

transcript: optional string

The transcript of the audio content, this will always be present if the output type is audio.

type: optional "output_text" or "output_audio"

The content type, output_text or output_audio depending on the session output_modalities configuration.

One of the following:

"output_text"

"output_audio"

role: "assistant"

The role of the message sender. Always assistant.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

A function call item in a Realtime conversation.

arguments: string

The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

name: string

The name of the function being called.

type: "function_call"

The type of the item. Always function_call.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

call_id: optional string

The ID of the function call.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

A function call output item in a Realtime conversation.

call_id: string

The ID of the function call this output is for.

output: string

The output of the function call, this is free text and can contain any information or simply be empty.

type: "function_call_output"

The type of the item. Always function_call_output.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }

A Realtime item responding to an MCP approval request.

id: string

The unique ID of the approval response.

approval_request_id: string

The ID of the approval request being answered.

approve: boolean

Whether the request was approved.

type: "mcp_approval_response"

The type of the item. Always mcp_approval_response.

reason: optional string or null

Optional reason for the decision.

RealtimeMcpListTools object { server_label, tools, type, id }

A Realtime item listing tools available on an MCP server.

server_label: string

The label of the MCP server.

tools: array of object { input_schema, name, annotations, description }

The tools available on the server.

input_schema: unknown

The JSON schema describing the tool's input.

name: string

The name of the tool.

annotations: optional unknown or null

Additional annotations about the tool.

description: optional string or null

The description of the tool.

type: "mcp_list_tools"

The type of the item. Always mcp_list_tools.

id: optional string

The unique ID of the list.

RealtimeMcpToolCall object { id, arguments, name, 5 more }

A Realtime item representing an invocation of a tool on an MCP server.

id: string

The unique ID of the tool call.

arguments: string

A JSON string of the arguments passed to the tool.

name: string

The name of the tool that was run.

server_label: string

The label of the MCP server running the tool.

type: "mcp_call"

The type of the item. Always mcp_call.

approval_request_id: optional string or null

The ID of an associated approval request, if any.

error: optional RealtimeMcpProtocolError { code, message, type } or RealtimeMcpToolExecutionError { message, type } or RealtimeMcphttpError { code, message, type } or null

The error from the tool call, if any.

One of the following:

RealtimeMcpProtocolError object { code, message, type }

code: number

message: string

type: "protocol_error"

RealtimeMcpToolExecutionError object { message, type }

message: string

type: "tool_execution_error"

RealtimeMcphttpError object { code, message, type }

code: number

message: string

type: "http_error"

output: optional string or null

The output from the tool call.

RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }

A Realtime item requesting human approval of a tool invocation.

id: string

The unique ID of the approval request.

arguments: string

A JSON string of arguments for the tool.

name: string

The name of the tool to run.

server_label: string

The label of the MCP server making the request.

type: "mcp_approval_request"

The type of the item. Always mcp_approval_request.

output_modalities: optional array of "text" or "audio"

The set of modalities the model used to respond, currently the only possible values are [\"audio\"], [\"text\"]. Audio output always include a text transcript. Setting the output to mode text will disable audio output from the model.

One of the following:

"text"

"audio"

status: optional "completed" or "cancelled" or "failed" or 2 more

The final status of the response (completed, cancelled, failed, or incomplete, in_progress).

One of the following:

"completed"

"cancelled"

"failed"

"incomplete"

"in_progress"

status_details: optional RealtimeResponseStatus { error, reason, type }

Additional details about the status.

error: optional object { code, type }

A description of the error that caused the response to fail, populated when the status is failed.

code: optional string

Error code, if any.

type: optional string

The type of error.

reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"

The reason the Response did not complete. For a cancelled Response, one of turn_detected (the server VAD detected a new start of speech) or client_cancelled (the client sent a cancel event). For an incomplete Response, one of max_output_tokens or content_filter (the server-side safety filter activated and cut off the response).

One of the following:

"turn_detected"

"client_cancelled"

"max_output_tokens"

"content_filter"

type: optional "completed" or "cancelled" or "failed" or "incomplete"

The type of error that caused the response to fail, corresponding with the status field (completed, cancelled, incomplete, failed).

One of the following:

"completed"

"cancelled"

"failed"

"incomplete"

usage: optional RealtimeResponseUsage { input_token_details, input_tokens, output_token_details, 2 more }

Usage statistics for the Response, this will correspond to billing. A Realtime API session will maintain a conversation context and append new Items to the Conversation, thus output from previous turns (text and audio tokens) will become the input for later turns.

input_token_details: optional RealtimeResponseUsageInputTokenDetails { audio_tokens, cached_tokens, cached_tokens_details, 2 more }

Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens.

audio_tokens: optional number

The number of audio tokens used as input for the Response.

cached_tokens: optional number

The number of cached tokens used as input for the Response.

cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }

Details about the cached tokens used as input for the Response.

audio_tokens: optional number

The number of cached audio tokens used as input for the Response.

image_tokens: optional number

The number of cached image tokens used as input for the Response.

text_tokens: optional number

The number of cached text tokens used as input for the Response.

image_tokens: optional number

The number of image tokens used as input for the Response.

text_tokens: optional number

The number of text tokens used as input for the Response.

input_tokens: optional number

The number of input tokens used in the Response, including text and audio tokens.

output_token_details: optional RealtimeResponseUsageOutputTokenDetails { audio_tokens, text_tokens }

Details about the output tokens used in the Response.

audio_tokens: optional number

The number of audio tokens used in the Response.

text_tokens: optional number

The number of text tokens used in the Response.

output_tokens: optional number

The number of output tokens sent in the Response, including text and audio tokens.

total_tokens: optional number

The total number of tokens in the Response including input and output text and audio tokens.

type: "response.created"

The event type, must be response.created.

OBJECT

response.created

{
  "type": "response.created",
  "event_id": "event_C9G8pqbTEddBSIxbBN6Os",
  "response": {
    "object": "realtime.response",
    "id": "resp_C9G8p7IH2WxLbkgPNouYL",
    "status": "in_progress",
    "status_details": null,
    "output": [],
    "conversation_id": "conv_C9G8mmBkLhQJwCon3hoJN",
    "output_modalities": [
      "audio"
    ],
    "max_output_tokens": "inf",
    "audio": {
      "output": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "voice": "marin"
      }
    },
    "usage": null,
    "metadata": null
  },
}

response.done

Returned when a Response is done streaming. Always emitted, no matter the final state. The Response object included in the response.done event will include all output Items in the Response but will omit the raw audio data.

Clients should check the status field of the Response to determine if it was successful (completed) or if there was another outcome: cancelled, failed, or incomplete.

A response will contain all output items that were generated during the response, excluding any audio content.

event_id: string

The unique ID of the server event.

response: RealtimeResponse { id, audio, conversation_id, 8 more }

The response resource.

id: optional string

The unique ID of the response, will look like resp_1234.

audio: optional object { output }

Configuration for audio output.

output: optional object { format, voice }

format: optional RealtimeAudioFormats

The format of the output audio.

One of the following:

PCMAudio object { rate, type }

The PCM audio format. Only a 24kHz sample rate is supported.

rate: optional 24000

The sample rate of the audio. Always 24000.

type: optional "audio/pcm"

The audio format. Always audio/pcm.

PCMUAudio object { type }

The G.711 μ-law format.

type: optional "audio/pcmu"

The audio format. Always audio/pcmu.

PCMAAudio object { type }

The G.711 A-law format.

type: optional "audio/pcma"

The audio format. Always audio/pcma.

voice: optional string or "alloy" or "ash" or "ballad" or 7 more

The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

One of the following:

string

"alloy" or "ash" or "ballad" or 7 more

The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar. We recommend marin and cedar for best quality.

One of the following:

"alloy"

"ash"

"ballad"

"coral"

"echo"

"sage"

"shimmer"

"verse"

"marin"

"cedar"

conversation_id: optional string

Which conversation the response is added to, determined by the conversation field in the response.create event. If auto, the response will be added to the default conversation and the value of conversation_id will be an id like conv_1234. If none, the response will not be added to any conversation and the value of conversation_id will be null. If responses are being triggered automatically by VAD the response will be added to the default conversation

max_output_tokens: optional number or "inf"

Maximum number of output tokens for a single assistant response, inclusive of tool calls, that was used in this response.

One of the following:

number

"inf"

metadata: optional Metadata or null

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

object: optional "realtime.response"

The object type, must be realtime.response.

output: optional array of ConversationItem

The list of output items generated by the response.

One of the following:

RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

content: array of object { text, type }

The content of the message.

text: optional string

The text content.

type: optional "input_text"

The content type. Always input_text for system messages.

role: "system"

The role of the message sender. Always system.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemUserMessage object { content, role, type, 3 more }

A user message item in a Realtime conversation.

content: array of object { audio, detail, image_url, 3 more }

The content of the message.

audio: optional string

Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

detail: optional "auto" or "low" or "high"

The detail level of the image (for input_image). auto will default to high.

One of the following:

"auto"

"low"

"high"

image_url: optional string

Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

formaturi

text: optional string

The text content (for input_text).

transcript: optional string

Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

type: optional "input_text" or "input_audio" or "input_image"

The content type (input_text, input_audio, or input_image).

One of the following:

"input_text"

"input_audio"

"input_image"

role: "user"

The role of the message sender. Always user.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

An assistant message item in a Realtime conversation.

content: array of object { audio, text, transcript, type }

The content of the message.

audio: optional string

Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

text: optional string

The text content.

transcript: optional string

The transcript of the audio content, this will always be present if the output type is audio.

type: optional "output_text" or "output_audio"

The content type, output_text or output_audio depending on the session output_modalities configuration.

One of the following:

"output_text"

"output_audio"

role: "assistant"

The role of the message sender. Always assistant.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

A function call item in a Realtime conversation.

arguments: string

The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

name: string

The name of the function being called.

type: "function_call"

The type of the item. Always function_call.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

call_id: optional string

The ID of the function call.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

A function call output item in a Realtime conversation.

call_id: string

The ID of the function call this output is for.

output: string

The output of the function call, this is free text and can contain any information or simply be empty.

type: "function_call_output"

The type of the item. Always function_call_output.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeMcpApprovalResponse object { id, approval_request_id, approve, 2 more }

A Realtime item responding to an MCP approval request.

id: string

The unique ID of the approval response.

approval_request_id: string

The ID of the approval request being answered.

approve: boolean

Whether the request was approved.

type: "mcp_approval_response"

The type of the item. Always mcp_approval_response.

reason: optional string or null

Optional reason for the decision.

RealtimeMcpListTools object { server_label, tools, type, id }

A Realtime item listing tools available on an MCP server.

server_label: string

The label of the MCP server.

tools: array of object { input_schema, name, annotations, description }

The tools available on the server.

input_schema: unknown

The JSON schema describing the tool's input.

name: string

The name of the tool.

annotations: optional unknown or null

Additional annotations about the tool.

description: optional string or null

The description of the tool.

type: "mcp_list_tools"

The type of the item. Always mcp_list_tools.

id: optional string

The unique ID of the list.

RealtimeMcpToolCall object { id, arguments, name, 5 more }

A Realtime item representing an invocation of a tool on an MCP server.

id: string

The unique ID of the tool call.

arguments: string

A JSON string of the arguments passed to the tool.

name: string

The name of the tool that was run.

server_label: string

The label of the MCP server running the tool.

type: "mcp_call"

The type of the item. Always mcp_call.

approval_request_id: optional string or null

The ID of an associated approval request, if any.

error: optional RealtimeMcpProtocolError { code, message, type } or RealtimeMcpToolExecutionError { message, type } or RealtimeMcphttpError { code, message, type } or null

The error from the tool call, if any.

One of the following:

RealtimeMcpProtocolError object { code, message, type }

code: number

message: string

type: "protocol_error"

RealtimeMcpToolExecutionError object { message, type }

message: string

type: "tool_execution_error"

RealtimeMcphttpError object { code, message, type }

code: number

message: string

type: "http_error"

output: optional string or null

The output from the tool call.

RealtimeMcpApprovalRequest object { id, arguments, name, 2 more }

A Realtime item requesting human approval of a tool invocation.

id: string

The unique ID of the approval request.

arguments: string

A JSON string of arguments for the tool.

name: string

The name of the tool to run.

server_label: string

The label of the MCP server making the request.

type: "mcp_approval_request"

The type of the item. Always mcp_approval_request.

output_modalities: optional array of "text" or "audio"

The set of modalities the model used to respond, currently the only possible values are [\"audio\"], [\"text\"]. Audio output always include a text transcript. Setting the output to mode text will disable audio output from the model.

One of the following:

"text"

"audio"

status: optional "completed" or "cancelled" or "failed" or 2 more

The final status of the response (completed, cancelled, failed, or incomplete, in_progress).

One of the following:

"completed"

"cancelled"

"failed"

"incomplete"

"in_progress"

status_details: optional RealtimeResponseStatus { error, reason, type }

Additional details about the status.

error: optional object { code, type }

A description of the error that caused the response to fail, populated when the status is failed.

code: optional string

Error code, if any.

type: optional string

The type of error.

reason: optional "turn_detected" or "client_cancelled" or "max_output_tokens" or "content_filter"

The reason the Response did not complete. For a cancelled Response, one of turn_detected (the server VAD detected a new start of speech) or client_cancelled (the client sent a cancel event). For an incomplete Response, one of max_output_tokens or content_filter (the server-side safety filter activated and cut off the response).

One of the following:

"turn_detected"

"client_cancelled"

"max_output_tokens"

"content_filter"

type: optional "completed" or "cancelled" or "failed" or "incomplete"

The type of error that caused the response to fail, corresponding with the status field (completed, cancelled, incomplete, failed).

One of the following:

"completed"

"cancelled"

"failed"

"incomplete"

usage: optional RealtimeResponseUsage { input_token_details, input_tokens, output_token_details, 2 more }

Usage statistics for the Response, this will correspond to billing. A Realtime API session will maintain a conversation context and append new Items to the Conversation, thus output from previous turns (text and audio tokens) will become the input for later turns.

input_token_details: optional RealtimeResponseUsageInputTokenDetails { audio_tokens, cached_tokens, cached_tokens_details, 2 more }

Details about the input tokens used in the Response. Cached tokens are tokens from previous turns in the conversation that are included as context for the current response. Cached tokens here are counted as a subset of input tokens, meaning input tokens will include cached and uncached tokens.

audio_tokens: optional number

The number of audio tokens used as input for the Response.

cached_tokens: optional number

The number of cached tokens used as input for the Response.

cached_tokens_details: optional object { audio_tokens, image_tokens, text_tokens }

Details about the cached tokens used as input for the Response.

audio_tokens: optional number

The number of cached audio tokens used as input for the Response.

image_tokens: optional number

The number of cached image tokens used as input for the Response.

text_tokens: optional number

The number of cached text tokens used as input for the Response.

image_tokens: optional number

The number of image tokens used as input for the Response.

text_tokens: optional number

The number of text tokens used as input for the Response.

input_tokens: optional number

The number of input tokens used in the Response, including text and audio tokens.

output_token_details: optional RealtimeResponseUsageOutputTokenDetails { audio_tokens, text_tokens }

Details about the output tokens used in the Response.

audio_tokens: optional number

The number of audio tokens used in the Response.

text_tokens: optional number

The number of text tokens used in the Response.

output_tokens: optional number

The number of output tokens sent in the Response, including text and audio tokens.

total_tokens: optional number

The total number of tokens in the Response including input and output text and audio tokens.

type: "response.done"

The event type, must be response.done.

OBJECT

response.done

{
  "type": "response.done",
  "event_id": "event_CCXHxcMy86rrKhBLDdqCh",
  "response": {
    "object": "realtime.response",
    "id": "resp_CCXHw0UJld10EzIUXQCNh",
    "status": "completed",
    "status_details": null,
    "output": [
      {
        "id": "item_CCXHwGjjDUfOXbiySlK7i",
        "type": "message",
        "status": "completed",
        "role": "assistant",
        "content": [
          {
            "type": "output_audio",
            "transcript": "Loud and clear! I can hear you perfectly. How can I help you today?"
          }
        ]
      }
    ],
    "conversation_id": "conv_CCXHsurMKcaVxIZvaCI5m",
    "output_modalities": [
      "audio"
    ],
    "max_output_tokens": "inf",
    "audio": {
      "output": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "voice": "alloy"
      }
    },
    "usage": {
      "total_tokens": 253,
      "input_tokens": 132,
      "output_tokens": 121,
      "input_token_details": {
        "text_tokens": 119,
        "audio_tokens": 13,
        "image_tokens": 0,
        "cached_tokens": 64,
        "cached_tokens_details": {
          "text_tokens": 64,
          "audio_tokens": 0,
          "image_tokens": 0
        }
      },
      "output_token_details": {
        "text_tokens": 30,
        "audio_tokens": 91
      }
    },
    "metadata": null
  }
}

response.function_call_arguments.delta

Returned when the model-generated function call arguments are updated.

call_id: string

The ID of the function call.

delta: string

The arguments delta as a JSON string.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the function call item.

output_index: number

The index of the output item in the response.

response_id: string

The ID of the response.

type: "response.function_call_arguments.delta"

The event type, must be response.function_call_arguments.delta.

OBJECT

response.function_call_arguments.delta

{
    "event_id": "event_5354",
    "type": "response.function_call_arguments.delta",
    "response_id": "resp_002",
    "item_id": "fc_001",
    "output_index": 0,
    "call_id": "call_001",
    "delta": "{\"location\": \"San\""
}

response.function_call_arguments.done

Returned when the model-generated function call arguments are done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

arguments: string

The final arguments as a JSON string.

call_id: string

The ID of the function call.

event_id: string

The unique ID of the server event.

item_id: string

The ID of the function call item.

name: string

The name of the function that was called.

output_index: number

The index of the output item in the response.

response_id: string

The ID of the response.

type: "response.function_call_arguments.done"

The event type, must be response.function_call_arguments.done.

OBJECT

response.function_call_arguments.done

{
    "event_id": "event_5556",
    "type": "response.function_call_arguments.done",
    "response_id": "resp_002",
    "item_id": "fc_001",
    "output_index": 0,
    "call_id": "call_001",
    "name": "get_weather",
    "arguments": "{\"location\": \"San Francisco\"}"
}

response.output_item.added

Returned when a new Item is created during Response generation.

event_id: string

The unique ID of the server event.

item: ConversationItem

A single item within a Realtime conversation.

One of the following:

RealtimeConversationItemSystemMessage object { content, role, type, 3 more }

A system message in a Realtime conversation can be used to provide additional context or instructions to the model. This is similar but distinct from the instruction prompt provided at the start of a conversation, as system messages can be added at any point in the conversation. For major changes to the conversation's behavior, use instructions, but for smaller updates (e.g. "the user is now asking about a different topic"), use system messages.

content: array of object { text, type }

The content of the message.

text: optional string

The text content.

type: optional "input_text"

The content type. Always input_text for system messages.

role: "system"

The role of the message sender. Always system.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemUserMessage object { content, role, type, 3 more }

A user message item in a Realtime conversation.

content: array of object { audio, detail, image_url, 3 more }

The content of the message.

audio: optional string

Base64-encoded audio bytes (for input_audio), these will be parsed as the format specified in the session input audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

detail: optional "auto" or "low" or "high"

The detail level of the image (for input_image). auto will default to high.

One of the following:

"auto"

"low"

"high"

image_url: optional string

Base64-encoded image bytes (for input_image) as a data URI. For example data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA.... Supported formats are PNG and JPEG.

formaturi

text: optional string

The text content (for input_text).

transcript: optional string

Transcript of the audio (for input_audio). This is not sent to the model, but will be attached to the message item for reference.

type: optional "input_text" or "input_audio" or "input_image"

The content type (input_text, input_audio, or input_image).

One of the following:

"input_text"

"input_audio"

"input_image"

role: "user"

The role of the message sender. Always user.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemAssistantMessage object { content, role, type, 3 more }

An assistant message item in a Realtime conversation.

content: array of object { audio, text, transcript, type }

The content of the message.

audio: optional string

Base64-encoded audio bytes, these will be parsed as the format specified in the session output audio type configuration. This defaults to PCM 16-bit 24kHz mono if not specified.

text: optional string

The text content.

transcript: optional string

The transcript of the audio content, this will always be present if the output type is audio.

type: optional "output_text" or "output_audio"

The content type, output_text or output_audio depending on the session output_modalities configuration.

One of the following:

"output_text"

"output_audio"

role: "assistant"

The role of the message sender. Always assistant.

type: "message"

The type of the item. Always message.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemFunctionCall object { arguments, name, type, 4 more }

A function call item in a Realtime conversation.

arguments: string

The arguments of the function call. This is a JSON-encoded string representing the arguments passed to the function, for example {"arg1": "value1", "arg2": 42}.

name: string

The name of the function being called.

type: "function_call"

The type of the item. Always function_call.

id: optional string

The unique ID of the item. This may be provided by the client or generated by the server.

call_id: optional string

The ID of the function call.

object: optional "realtime.item"

Identifier for the API object being returned - always realtime.item. Optional when creating a new item.

status: optional "completed" or "incomplete" or "in_progress"

The status of the item. Has no effect on the conversation.

One of the following:

"completed"

"incomplete"

"in_progress"

RealtimeConversationItemFunctionCallOutput object { call_id, output, type, 3 more }

A function call output item in a Realtime conversation.

call_id: string

The ID of the function call this output is for.

output: string

The output of the function call, this is free text and can contain any information or simply be empty.

type: "function_call_output"

The type of the item. Always function_call_output.