# Vidu/LipSync

Lip-sync video generation model, supporting both audio-driven and text-driven modes.

## Submit Async Task

### Endpoint

`https://api-us-ca.umodelverse.ai/v1/tasks/submit`

### Input

| Parameter | Type | Required | Description |
| :-------- | :---- | :-------- | :----------- |
| model | string | Yes | Model name, must be set to: `vidu-lip-sync` |
| input.video_url | string | Yes | Source video URL (must be accessible). The model will match lip movements based on this video.<br><br>**Format requirements**: supports mp4, mov, avi<br>**Duration**: 1–600 seconds (recommended 10–120 seconds)<br>**File size**: ≤ 5GB<br>**Resolution**: each side between 360p–4096p<br>**Encoding**: must be H.264 (otherwise refer to [format conversion](https://shengshu.feishu.cn/docx/FIQHdcreYoCYX6x4kORckwmWnjf))<br>**Content**: must comply with portrait rights regulations<br>**Video requirements**:<br>&nbsp;&nbsp;- **Face**: must be a real person (cartoon allowed only if facial proportions are realistic). Face should face camera, horizontal rotation ≤ 45°, vertical ≤ 15°, minimal occlusion, stable lighting<br>&nbsp;&nbsp;- **Audio**: no restrictions |
| input.audio_url | string | No | Audio file URL (choose one of `audio_url` or `text`). The generated speech content and voice will follow the audio.<br><br>**Formats**: wav, mp3, wma, m4a, aac, ogg<br>**Duration**: 1–600 seconds<br>**File size**: ≤ 100MB |
| input.text | string | No | Text content (choose one of `audio_url` or `text`). Used for lip-sync generation.<br><br>**Length**: 4–2000 characters (2–1000 Chinese or 4–2000 English)<br>**Priority**: if both provided, `audio_url` takes precedence<br>**Paragraphs**: separated by newline<br>**Pause control**: use `<#x#>` for pauses:<br>&nbsp;&nbsp;- x = duration in seconds [0.01–99.99], max 2 decimals<br>&nbsp;&nbsp;- must be placed between valid speech segments<br>&nbsp;&nbsp;- cannot use multiple pause markers consecutively<br>&nbsp;&nbsp;- **Example**: `Hello<#2#>I am Modelverse<#2#>Nice to meet you` |
| input.ref_photo_url | string | No | Face reference image URL, used to specify target when multiple faces exist.<br><br>**Formats**: jpg, jpeg, png, bmp, webp<br>**Resolution**: 192–4096px<br>**File size**: ≤ 10MB<br>**Content**: must contain a clear frontal face from the video<br>**Default behavior**: if not provided, selects the largest visible face in the first frame with faces |
| parameters.vidu_type | string | Yes | Vidu API type, must be `lip-sync` |
| parameters.speed | float | No | Speech speed, default 1.0, range [0.5, 2]. Only effective in text-driven mode |
| parameters.voice_id | string | No | Voice ID, only effective in text-driven mode. See [voice list](https://shengshu.feishu.cn/sheets/EgFvs6DShhiEBStmjzccr5gonOg) |
| parameters.volume | int | No | Volume level, range 0–10, default 0 (normal). Higher = louder. Only effective in text-driven mode |

**Notes:**

- One of `audio_url` or `text` must be provided
- `speed`, `voice_id`, and `volume` only apply in text-driven mode

### Request Example (Audio-driven)

⚠️ If you are using Windows, it is recommended to use Postman or another API client tool.

```shell
curl --location --globoff 'https://api-us-ca.umodelverse.ai/v1/tasks/submit' \
--header 'Authorization: <YOUR_API_KEY>' \
--header 'Content-Type: application/json' \
--data '{
    "model": "vidu-lip-sync",
    "input": {
      "video_url": "https://umodelverse-inference.cn-wlcb.ufileos.com/maxcot-dance.mp4",
      "audio_url": "https://umodelverse-inference.cn-wlcb.ufileos.com/%E6%AC%A2%E8%BF%8E%E4%BD%BF%E7%94%A8Modelverse_API.mp3"
    },
    "parameters": {
      "vidu_type": "lip-sync"
    }
  }'
```

### Request Example (Text-driven)

```shell
curl --location --globoff 'https://api-us-ca.umodelverse.ai/v1/tasks/submit' \
--header 'Authorization: <YOUR_API_KEY>' \
--header 'Content-Type: application/json' \
--data '{
    "model": "vidu-lip-sync",
    "input": {
      "video_url": "https://umodelverse-inference.cn-wlcb.ufileos.com/maxcot-dance.mp4",
      "text": "Hello, welcome to use the lip-sync feature"
    },
    "parameters": {
      "vidu_type": "lip-sync",
      "voice_id": "your_voice_id",
      "speed": 1.0,
      "volume": 4
    }
  }'
```

### Output

| Parameter      | Type   | Description                         |
| :------------- | :----- | :---------------------------------- |
| output.task_id | string | Unique identifier of the async task |
| request_id     | string | Unique request identifier           |

### Response Example

```json
{
  "output": {
    "task_id": "task_id"
  },
  "request_id": "request_id"
}
```

## Query Task Status

### Endpoint

`https://api-us-ca.umodelverse.ai/v1/tasks/status?task_id=<task_id>`

### Request Example

```shell
curl --location 'https://api-us-ca.umodelverse.ai/v1/tasks/status?task_id=<task_id>' \
--header 'Authorization: <YOUR_API_KEY>'
```

### Output

| Parameter            | Type    | Description                                             |
| :------------------- | :------ | :------------------------------------------------------ |
| output.task_id       | string  | Unique identifier of the async task                     |
| output.task_status   | string  | Task status: `Pending`, `Running`, `Success`, `Failure` |
| output.urls          | array   | List of video result URLs                               |
| output.submit_time   | integer | Task submission timestamp                               |
| output.finish_time   | integer | Task completion timestamp                               |
| output.error_message | string  | Error message when failed                               |
| usage.duration       | integer | Video duration (seconds)                                |
| request_id           | string  | Unique request identifier                               |

### Response Example (Success)

```json
{
  "output": {
    "task_id": "task_id",
    "task_status": "Success",
    "urls": ["https://xxxxx/xxxx-lipsync.mp4"],
    "submit_time": 1756959000,
    "finish_time": 1756959050
  },
  "usage": {
    "duration": 30
  },
  "request_id": ""
}
```

### Response Example (Failure)

```json
{
  "output": {
    "task_id": "task_id",
    "task_status": "Failure",
    "submit_time": 1756959000,
    "finish_time": 1756959019,
    "error_message": "error_message"
  },
  "usage": {
    "duration": 5
  },
  "request_id": ""
}
```