# Improving Prompt Cache Hit Rate

This document describes how to improve the prompt cache hit rate on the UModelverse platform through a stable request structure, session identifiers, and model protocol configuration, thereby reducing time to first token (TTFT) and inference costs.

## Using X-Session-ID to Maintain Session Affinity

`X-Session-ID` is a session identifier passed through an HTTP header. The platform does its best to route consecutive requests of the same session to the same inference instance, thereby improving the cache hit rate of that `session` conversation.

### Usage

Add `X-Session-ID` to the request header:

```bash
X-Session-ID: session-abc123
```

Complete request example:

```bash
curl -X POST 'https://api.modelverse.cn/v1/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer your-api-key' \
  -H 'X-Session-ID: session-abc123' \
  -d '{
    "model": "your-model",
    "messages": [
      { "role": "system", "content": "You are an assistant." },
      { "role": "user", "content": "Please help me summarize this content." }
    ]
  }'
```

### Recommendations

- Keep `X-Session-ID` unchanged across the multi-turn conversation of the same user.
- Use different session IDs for different users, different sessions, or unrelated contexts.
- Use a stable identifier generated on the business side as the session ID; avoid regenerating it for every request.
- Avoid sharing a single session ID among many highly concurrent users, as this weakens scheduling affinity and may also risk context crosstalk.

### Boundaries of Affinity

`X-Session-ID` provides scheduling affinity, not a permanent binding to an inference instance. The platform tries to reuse the same inference instance, but it may select a different instance in the following scenarios:

- Concurrency under the same session ID is high and a single inference instance cannot handle all requests.
- The original inference instance is under heavy load, has a long queue, or is temporarily unavailable.
- The session affinity relationship exceeds the TTL configured by the platform, after which subsequent requests select a suitable inference instance again.

Therefore, `X-Session-ID` increases the probability that requests of the same session hit the local cache, but it does not guarantee that every request hits the same inference instance.

## cache_control in Anthropic Messages

When invoking the Anthropic Messages protocol (`/v1/messages`), you must add the `cache_control` field to the request to enable prompt caching. Other models or OpenAI-compatible interfaces usually do not require caching to be enabled manually — simply keeping the request prefix stable is enough.

According to the official Anthropic documentation, prompt caching can be enabled in two ways: a top-level `cache_control`, and explicit cache breakpoints on content blocks. The UModelverse platform currently does not support `cache_control` at the top level of the request. Add `cache_control` to a specific content block to designate a cache breakpoint.

The following example places `cache_control` on a stable `system` content block:

```bash
curl https://api.modelverse.cn/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: $MODELVERSE_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "xxxx",
    "max_tokens": 1024,
    "system": [
      {
        "type": "text",
        "text": "You are an assistant. Please answer questions concisely and accurately.",
        "cache_control": { "type": "ephemeral" }
      }
    ],
    "messages": [
      {
        "role": "user",
        "content": "Please help me summarize this content."
      }
    ]
  }'
```

If you need to cache long contexts, tool definitions, or historical messages, place `cache_control` on the corresponding stable content block. Do not place `cache_control` on the root object of the request, otherwise the UModelverse platform cannot recognize the field. For the specific fields, TTL, and billing rules, see the [Anthropic Prompt caching documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching).

The default TTL of the prompt cache is 5 minutes, and Anthropic also offers a 1-hour cache option. This cache TTL is not the same concept as the platform scheduling affinity TTL of `X-Session-ID`.

## Keeping the System Prompt Stable

Do not write dynamic content such as the current date, current time, random numbers, or request IDs into the system prompt.

Changes in time or random content change the system prompt content, so the prefix no longer matches and the cache is invalidated. For example, when the date changes at midnight, a system prompt containing the date changes as a whole, which may reduce the cache hit rate and increase TTFT.

We recommend placing dynamic content in user messages rather than in the system prompt.

```json
{
  "messages": [
    {
      "role": "system",
      "content": "You are an assistant. Please answer questions concisely and accurately."
    },
    {
      "role": "user",
      "content": "Today is May 9, 2026. Please help me generate a daily report."
    }
  ]
}
```

Not recommended:

```json
{
  "messages": [
    {
      "role": "system",
      "content": "Today is May 9, 2026. You are an assistant."
    },
    {
      "role": "user",
      "content": "Please help me generate a daily report."
    }
  ]
}
```

## Designing a Stable Request Structure

A well-designed request structure is the foundation for improving the cache hit rate. The more stable the request prefix, the easier it is to reuse an existing cache.

### Keeping the messages Structure Stable

- **Keep roles stable**: do not frequently change the `role` of each message in `messages`.
- **Keep the number of messages as stable as possible**: do not insert, delete, or merge historical messages irregularly.
- **Keep the message order stable**: do not rearrange existing messages.

### Appending New Content Only at the End of messages

New conversation turns should be appended to the end of the `messages` array. Avoid inserting into the middle or modifying existing messages.

Round 1 request:

```json
{
  "messages": [
    {
      "role": "system",
      "content": "You are an assistant."
    },
    {
      "role": "user",
      "content": "Question 1"
    }
  ]
}
```

Round 2 request:

```json
{
  "messages": [
    {
      "role": "system",
      "content": "You are an assistant."
    },
    {
      "role": "user",
      "content": "Question 1"
    },
    {
      "role": "assistant",
      "content": "Answer 1"
    },
    {
      "role": "user",
      "content": "Question 2"
    }
  ]
}
```

In this structure, the round 2 request preserves the complete prefix of the round 1 request, so the existing KV cache is easier to reuse.

## Summary of Optimizations

| Optimization | Effect | Recommended practice |
| --- | --- | --- |
| Use `X-Session-ID` | Increases the probability that the same session is routed to the same inference instance | Pass a stable session ID in the HTTP header |
| Understand the boundaries of affinity | Avoids mistaking the session ID for a hard binding to an instance | Note that high concurrency, instance pressure, and affinity TTL can all trigger rescheduling |
| Use `cache_control` | Enables prompt caching under the Anthropic Messages protocol | Add `cache_control` to a stable content block in `/v1/messages` requests; do not place it at the top level of the request |
| Keep the system prompt stable | Prevents prefix changes from invalidating the cache | Do not write dynamic content such as timestamps or random numbers into the system prompt |
| Append messages at the end | Keeps the request prefix consistent | Append new conversation turns only to the end of `messages` |