# Establish WebSocket real-time voice connection

> Audio Generation

Establish WebSocket connection for real-time voice interaction.

**Endpoint URL Format:**
```
wss://{endpoint}/v1/realtime?model={model_name}
```

**Example:**
```
wss://api.modelverse.cn/v1/realtime?model=gpt-realtime-2
```

**Authentication:**
```http
Authorization: Bearer {api_key}
```

**Connection Flow:**
1. Establish WebSocket connection with model parameter and Authorization header
2. Wait for server to return `session.created` event
3. Send `session.update` to configure session parameters
4. Start sending audio data

---

## Client Events

Send the following JSON events through WebSocket connection:

### 1. session.update

**Applicable Models:** Conversation, Translation, Transcription (all three models)

Update session configuration. Configuration differs for each model type:

**Conversation Model Configuration:**
- `session.type`: "realtime"
- `session.instructions`: System prompt
- `session.voice`: Voice (alloy/echo/fable/onyx/shimmer)
- `session.output_modalities`: ["audio"] indicates audio output

**Translation Model Configuration:**
- No `session.type` field required
- `session.audio.input.transcription`: Transcription configuration
- `session.audio.input.noise_reduction`: Noise reduction configuration
- `session.audio.output.language`: Target language

**Transcription Model Configuration:**
- `session.type`: "transcription"
- `session.audio.input.transcription`: Transcription configuration

**Request Example:**
```json
{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "You are a helpful assistant",
    "voice": "alloy",
    "output_modalities": ["audio"],
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "turn_detection": null
      },
      "output": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "voice": "alloy"
      }
    }
  }
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "session.update" |
| session | object | Session configuration object |
| session.type | string | Session type: realtime/transcription |
| session.instructions | string | System prompt (conversation model) |
| session.voice | string | Voice: alloy/echo/fable/onyx/shimmer |
| session.output_modalities | array | Output modalities, ["audio"] indicates audio output |
| session.audio | object | Audio configuration |

---

### 2. input_audio_buffer.append

**Applicable Models:** Conversation, Transcription

Send audio data (used by conversation and transcription models).

**Audio Format Requirements:**
- Encoding: PCM 16-bit
- Sample Rate: 24000Hz
- Channels: Mono
- Transmission: Base64 encoded

**Request Example:**
```json
{
  "type": "input_audio_buffer.append",
  "event_id": "event_001",
  "audio": "//uQxAAAAAA..."
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "input_audio_buffer.append" |
| event_id | string | Event unique identifier |
| audio | string | Base64 encoded PCM audio data |

---

### 3. session.input_audio_buffer.append

**Applicable Models:** Translation (dedicated)

Send audio data (**dedicated for translation model**).

**Note:** Translation model uses this event, not `input_audio_buffer.append`.

**Request Example:**
```json
{
  "type": "session.input_audio_buffer.append",
  "audio": "//uQxAAAAAA..."
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "session.input_audio_buffer.append" |
| audio | string | Base64 encoded PCM audio data |

---

### 4. input_audio_buffer.commit

**Applicable Models:** Conversation, Transcription

Commit audio buffer, notify server to start processing sent audio data.

**Usage Scenarios:**
- Conversation model: Submit and wait for AI response
- Transcription model: Submit and wait for transcription result

**Request Example:**
```json
{
  "type": "input_audio_buffer.commit",
  "event_id": "event_002"
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "input_audio_buffer.commit" |
| event_id | string | Event unique identifier |

---

### 5. response.create

**Applicable Models:** Conversation (dedicated)

Request AI to generate response. **Only for conversation model**.

**Trigger Timing:**
- Send after user finishes speaking
- Or automatically triggered after `input_audio_buffer.commit`

**Request Example:**
```json
{
  "type": "response.create"
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "response.create" |

---

## Server Events

Receive the following JSON events through WebSocket connection:

### 1. session.created

**Applicable Models:** Conversation, Translation, Transcription (all three models)

Indicates WebSocket connection successful, session created.

**Trigger Timing:**
- Returned immediately after client successfully establishes WebSocket connection

**Next Steps:**
- Client should wait for this event before sending `session.update` to configure session

**Response Example:**
```json
{
  "type": "session.created",
  "event_id": "evt_001",
  "session": {
    "id": "sess_abc123",
    "model": "gpt-realtime-2",
    "instructions": "You are a helpful assistant"
  }
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "session.created" |
| event_id | string | Event unique identifier |
| session | object | Session information |
| session.id | string | Session ID |
| session.model | string | Model name |
| session.instructions | string | System prompt |

---

### 2. response.output_audio.delta

**Applicable Models:** Conversation

AI generated voice segment (Base64 encoded PCM audio).

**Characteristics:**
- Streamed return, multiple delta segments form complete voice
- Client needs to play in order
- Audio format: PCM 16-bit, 24000Hz, mono

**Response Example:**
```json
{
  "type": "response.output_audio.delta",
  "delta": "//uQxAAAAAA..."
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "response.output_audio.delta" |
| delta | string | Base64 encoded PCM audio data segment |

---

### 3. response.output_audio_transcript.delta

**Applicable Models:** Conversation

Text segment corresponding to AI voice (real-time subtitles).

**Characteristics:**
- Streamed return, synchronized with audio segments
- Can be used to display real-time subtitles

**Response Example:**
```json
{
  "type": "response.output_audio_transcript.delta",
  "delta": "Hello, I am"
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "response.output_audio_transcript.delta" |
| delta | string | Text segment content |

---

### 4. response.done

**Applicable Models:** Conversation

Indicates AI response has been fully generated.

**Trigger Timing:**
- All AI voice has been sent
- Client can start next round of conversation

**Response Example:**
```json
{
  "type": "response.done"
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "response.done" |

---

### 5. session.input_transcript.delta

**Applicable Models:** Translation

Source language transcription text segment returned by translation model.

**Usage Scenarios:**
- Real-time display of source language text spoken by user
- Synchronized display with target language translation

**Response Example:**
```json
{
  "type": "session.input_transcript.delta",
  "delta": "Hello"
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "session.input_transcript.delta" |
| delta | string | Source language transcription text segment |

---

### 6. session.output_transcript.delta

**Applicable Models:** Translation

Target language translation text segment returned by translation model.

**Usage Scenarios:**
- Real-time display of translated target language text

**Response Example:**
```json
{
  "type": "session.output_transcript.delta",
  "delta": "你好"
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "session.output_transcript.delta" |
| delta | string | Target language translation text segment |

---

### 7. session.output_audio.delta

**Applicable Models:** Translation

Translated voice segment returned by translation model (Base64 encoded).

**Characteristics:**
- Streamed return of translated voice
- Audio format: PCM 16-bit, 24000Hz, mono

**Response Example:**
```json
{
  "type": "session.output_audio.delta",
  "delta": "//uQxAAAAAA..."
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "session.output_audio.delta" |
| delta | string | Base64 encoded translated audio data |

---

### 8. conversation.item.input_audio_transcription.completed

**Applicable Models:** Transcription

Final transcription result returned by transcription model.

**Trigger Timing:**
- After user finishes speaking
- Returns complete transcription text

**Response Example:**
```json
{
  "type": "conversation.item.input_audio_transcription.completed",
  "transcript": "Hello, this is a transcription test"
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "conversation.item.input_audio_transcription.completed" |
| transcript | string | Completed transcription text content |

---

### 9. error

**Applicable Models:** Conversation, Translation, Transcription (all three models)

Error information returned by server.

**Common Errors:**
- Invalid model name
- Authentication failed
- Audio format error
- Session configuration error

**Response Example:**
```json
{
  "type": "error",
  "error": {
    "code": "invalid_model",
    "message": "Invalid model name"
  }
}
```

**Field Description:**
| Field | Type | Description |
|------|------|-------------|
| type | string | Fixed value "error" |
| error | object | Error information |
| error.code | string | Error code |
| error.message | string | Error message |

## Endpoint

`GET https://api.modelverse.cn/`

## Parameters

| Name | In | Type | Required | Description |
| --- | --- | --- | --- | --- |
| model | query | string | Yes | Model name, e.g., gpt-realtime-2, gpt-realtime-translate, gpt-realtime-whisper |

## Responses

- **101** — WebSocket connection successfully established
- **400** — Invalid model parameter
- **401** — Authentication failed

## OpenAPI Definition

```json
{
  "openapi": "3.1.0",
  "info": {
    "title": "Modelverse GPT-Realtime Real-time Voice API",
    "version": "1.0.0",
    "description": "Modelverse GPT-Realtime real-time voice model API, supporting real-time voice conversation, translation, and transcription.\n\n**Model List:**\n| Model Type | Model Names |\n|-----------|------------|\n| Conversation | gpt-realtime, gpt-realtime-1.5, gpt-realtime-2, gpt-realtime-2.1, gpt-realtime-2.1-mini |\n| Translation | gpt-realtime-translate |\n| Transcription | gpt-realtime-whisper |\n"
  },
  "servers": [
    {
      "url": "wss://api.modelverse.cn/v1/realtime",
      "description": "China Mainland node, WebSocket real-time voice connection"
    },
    {
      "url": "wss://api-sg.umodelverse.ai/v1/realtime",
      "description": "Singapore node, WebSocket real-time voice connection"
    },
    {
      "url": "wss://api-us-ca.umodelverse.ai/v1/realtime",
      "description": "US Los Angeles node, WebSocket real-time voice connection"
    },
    {
      "url": "wss://api-ge-fra.umodelverse.ai/v1/realtime",
      "description": "Frankfurt node, WebSocket real-time voice connection"
    }
  ],
  "paths": {
    "/": {
      "get": {
        "operationId": "connectRealtime",
        "summary": "Establish WebSocket real-time voice connection",
        "description": "Establish WebSocket connection for real-time voice interaction.\n\n**Endpoint URL Format:**\n```\nwss://{endpoint}/v1/realtime?model={model_name}\n```\n\n**Example:**\n```\nwss://api.modelverse.cn/v1/realtime?model=gpt-realtime-2\n```\n\n**Authentication:**\n```http\nAuthorization: Bearer {api_key}\n```\n\n**Connection Flow:**\n1. Establish WebSocket connection with model parameter and Authorization header\n2. Wait for server to return `session.created` event\n3. Send `session.update` to configure session parameters\n4. Start sending audio data\n\n---\n\n## Client Events\n\nSend the following JSON events through WebSocket connection:\n\n### 1. session.update\n\n**Applicable Models:** Conversation, Translation, Transcription (all three models)\n\nUpdate session configuration. Configuration differs for each model type:\n\n**Conversation Model Configuration:**\n- `session.type`: \"realtime\"\n- `session.instructions`: System prompt\n- `session.voice`: Voice (alloy/echo/fable/onyx/shimmer)\n- `session.output_modalities`: [\"audio\"] indicates audio output\n\n**Translation Model Configuration:**\n- No `session.type` field required\n- `session.audio.input.transcription`: Transcription configuration\n- `session.audio.input.noise_reduction`: Noise reduction configuration\n- `session.audio.output.language`: Target language\n\n**Transcription Model Configuration:**\n- `session.type`: \"transcription\"\n- `session.audio.input.transcription`: Transcription configuration\n\n**Request Example:**\n```json\n{\n  \"type\": \"session.update\",\n  \"session\": {\n    \"type\": \"realtime\",\n    \"instructions\": \"You are a helpful assistant\",\n    \"voice\": \"alloy\",\n    \"output_modalities\": [\"audio\"],\n    \"audio\": {\n      \"input\": {\n        \"format\": { \"type\": \"audio/pcm\", \"rate\": 24000 },\n        \"turn_detection\": null\n      },\n      \"output\": {\n        \"format\": { \"type\": \"audio/pcm\", \"rate\": 24000 },\n        \"voice\": \"alloy\"\n      }\n    }\n  }\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"session.update\" |\n| session | object | Session configuration object |\n| session.type | string | Session type: realtime/transcription |\n| session.instructions | string | System prompt (conversation model) |\n| session.voice | string | Voice: alloy/echo/fable/onyx/shimmer |\n| session.output_modalities | array | Output modalities, [\"audio\"] indicates audio output |\n| session.audio | object | Audio configuration |\n\n---\n\n### 2. input_audio_buffer.append\n\n**Applicable Models:** Conversation, Transcription\n\nSend audio data (used by conversation and transcription models).\n\n**Audio Format Requirements:**\n- Encoding: PCM 16-bit\n- Sample Rate: 24000Hz\n- Channels: Mono\n- Transmission: Base64 encoded\n\n**Request Example:**\n```json\n{\n  \"type\": \"input_audio_buffer.append\",\n  \"event_id\": \"event_001\",\n  \"audio\": \"//uQxAAAAAA...\"\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"input_audio_buffer.append\" |\n| event_id | string | Event unique identifier |\n| audio | string | Base64 encoded PCM audio data |\n\n---\n\n### 3. session.input_audio_buffer.append\n\n**Applicable Models:** Translation (dedicated)\n\nSend audio data (**dedicated for translation model**).\n\n**Note:** Translation model uses this event, not `input_audio_buffer.append`.\n\n**Request Example:**\n```json\n{\n  \"type\": \"session.input_audio_buffer.append\",\n  \"audio\": \"//uQxAAAAAA...\"\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"session.input_audio_buffer.append\" |\n| audio | string | Base64 encoded PCM audio data |\n\n---\n\n### 4. input_audio_buffer.commit\n\n**Applicable Models:** Conversation, Transcription\n\nCommit audio buffer, notify server to start processing sent audio data.\n\n**Usage Scenarios:**\n- Conversation model: Submit and wait for AI response\n- Transcription model: Submit and wait for transcription result\n\n**Request Example:**\n```json\n{\n  \"type\": \"input_audio_buffer.commit\",\n  \"event_id\": \"event_002\"\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"input_audio_buffer.commit\" |\n| event_id | string | Event unique identifier |\n\n---\n\n### 5. response.create\n\n**Applicable Models:** Conversation (dedicated)\n\nRequest AI to generate response. **Only for conversation model**.\n\n**Trigger Timing:**\n- Send after user finishes speaking\n- Or automatically triggered after `input_audio_buffer.commit`\n\n**Request Example:**\n```json\n{\n  \"type\": \"response.create\"\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"response.create\" |\n\n---\n\n## Server Events\n\nReceive the following JSON events through WebSocket connection:\n\n### 1. session.created\n\n**Applicable Models:** Conversation, Translation, Transcription (all three models)\n\nIndicates WebSocket connection successful, session created.\n\n**Trigger Timing:**\n- Returned immediately after client successfully establishes WebSocket connection\n\n**Next Steps:**\n- Client should wait for this event before sending `session.update` to configure session\n\n**Response Example:**\n```json\n{\n  \"type\": \"session.created\",\n  \"event_id\": \"evt_001\",\n  \"session\": {\n    \"id\": \"sess_abc123\",\n    \"model\": \"gpt-realtime-2\",\n    \"instructions\": \"You are a helpful assistant\"\n  }\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"session.created\" |\n| event_id | string | Event unique identifier |\n| session | object | Session information |\n| session.id | string | Session ID |\n| session.model | string | Model name |\n| session.instructions | string | System prompt |\n\n---\n\n### 2. response.output_audio.delta\n\n**Applicable Models:** Conversation\n\nAI generated voice segment (Base64 encoded PCM audio).\n\n**Characteristics:**\n- Streamed return, multiple delta segments form complete voice\n- Client needs to play in order\n- Audio format: PCM 16-bit, 24000Hz, mono\n\n**Response Example:**\n```json\n{\n  \"type\": \"response.output_audio.delta\",\n  \"delta\": \"//uQxAAAAAA...\"\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"response.output_audio.delta\" |\n| delta | string | Base64 encoded PCM audio data segment |\n\n---\n\n### 3. response.output_audio_transcript.delta\n\n**Applicable Models:** Conversation\n\nText segment corresponding to AI voice (real-time subtitles).\n\n**Characteristics:**\n- Streamed return, synchronized with audio segments\n- Can be used to display real-time subtitles\n\n**Response Example:**\n```json\n{\n  \"type\": \"response.output_audio_transcript.delta\",\n  \"delta\": \"Hello, I am\"\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"response.output_audio_transcript.delta\" |\n| delta | string | Text segment content |\n\n---\n\n### 4. response.done\n\n**Applicable Models:** Conversation\n\nIndicates AI response has been fully generated.\n\n**Trigger Timing:**\n- All AI voice has been sent\n- Client can start next round of conversation\n\n**Response Example:**\n```json\n{\n  \"type\": \"response.done\"\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"response.done\" |\n\n---\n\n### 5. session.input_transcript.delta\n\n**Applicable Models:** Translation\n\nSource language transcription text segment returned by translation model.\n\n**Usage Scenarios:**\n- Real-time display of source language text spoken by user\n- Synchronized display with target language translation\n\n**Response Example:**\n```json\n{\n  \"type\": \"session.input_transcript.delta\",\n  \"delta\": \"Hello\"\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"session.input_transcript.delta\" |\n| delta | string | Source language transcription text segment |\n\n---\n\n### 6. session.output_transcript.delta\n\n**Applicable Models:** Translation\n\nTarget language translation text segment returned by translation model.\n\n**Usage Scenarios:**\n- Real-time display of translated target language text\n\n**Response Example:**\n```json\n{\n  \"type\": \"session.output_transcript.delta\",\n  \"delta\": \"你好\"\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"session.output_transcript.delta\" |\n| delta | string | Target language translation text segment |\n\n---\n\n### 7. session.output_audio.delta\n\n**Applicable Models:** Translation\n\nTranslated voice segment returned by translation model (Base64 encoded).\n\n**Characteristics:**\n- Streamed return of translated voice\n- Audio format: PCM 16-bit, 24000Hz, mono\n\n**Response Example:**\n```json\n{\n  \"type\": \"session.output_audio.delta\",\n  \"delta\": \"//uQxAAAAAA...\"\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"session.output_audio.delta\" |\n| delta | string | Base64 encoded translated audio data |\n\n---\n\n### 8. conversation.item.input_audio_transcription.completed\n\n**Applicable Models:** Transcription\n\nFinal transcription result returned by transcription model.\n\n**Trigger Timing:**\n- After user finishes speaking\n- Returns complete transcription text\n\n**Response Example:**\n```json\n{\n  \"type\": \"conversation.item.input_audio_transcription.completed\",\n  \"transcript\": \"Hello, this is a transcription test\"\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"conversation.item.input_audio_transcription.completed\" |\n| transcript | string | Completed transcription text content |\n\n---\n\n### 9. error\n\n**Applicable Models:** Conversation, Translation, Transcription (all three models)\n\nError information returned by server.\n\n**Common Errors:**\n- Invalid model name\n- Authentication failed\n- Audio format error\n- Session configuration error\n\n**Response Example:**\n```json\n{\n  \"type\": \"error\",\n  \"error\": {\n    \"code\": \"invalid_model\",\n    \"message\": \"Invalid model name\"\n  }\n}\n```\n\n**Field Description:**\n| Field | Type | Description |\n|------|------|-------------|\n| type | string | Fixed value \"error\" |\n| error | object | Error information |\n| error.code | string | Error code |\n| error.message | string | Error message |\n",
        "parameters": [
          {
            "name": "model",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Model name, e.g., gpt-realtime-2, gpt-realtime-translate, gpt-realtime-whisper"
          }
        ],
        "responses": {
          "101": {
            "description": "WebSocket connection successfully established"
          },
          "400": {
            "description": "Invalid model parameter",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/ErrorResponse"
                }
              }
            }
          },
          "401": {
            "description": "Authentication failed",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/ErrorResponse"
                }
              }
            }
          }
        }
      }
    }
  },
  "components": {
    "securitySchemes": {
      "bearerAuth": {
        "type": "http",
        "scheme": "bearer",
        "bearerFormat": "API key",
        "description": "Carried in HTTP Header during WebSocket connection"
      }
    },
    "schemas": {
      "ErrorResponse": {
        "type": "object",
        "description": "Error response",
        "properties": {
          "error": {
            "type": "object",
            "properties": {
              "code": {
                "type": "string"
              },
              "message": {
                "type": "string"
              },
              "type": {
                "type": "string"
              }
            }
          }
        }
      }
    }
  }
}
```