# Streaming

> Receive chat completions as server-sent events, with token usage in the final chunk.

Source: https://docs.serenityedge.ai/docs/streaming/

Set `stream: true` to receive the response as it is generated. The API answers with `Content-Type: text/event-stream` and sends one `data:` line per chunk, each containing a `chat.completion.chunk` object. The stream ends with `data: [DONE]`.

## Request

```bash
curl https://api.serenityedge.ai/v1/chat/completions \
  -H "Authorization: Bearer $SERENITY_EDGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "orion-plus",
    "stream": true,
    "stream_options": {"include_usage": true},
    "messages": [{"role": "user", "content": "Write a haiku about the Mediterranean."}]
  }'
```

## Response

```text
data: {"id":"chatcmpl-8f0f1e2a","object":"chat.completion.chunk","created":1758542400,"model":"orion-plus","choices":[{"index":0,"delta":{"role":"assistant","content":"Blue"},"finish_reason":null}]}

data: {"id":"chatcmpl-8f0f1e2a","object":"chat.completion.chunk","created":1758542400,"model":"orion-plus","choices":[{"index":0,"delta":{"content":" waves"},"finish_reason":null}]}

data: {"id":"chatcmpl-8f0f1e2a","object":"chat.completion.chunk","created":1758542400,"model":"orion-plus","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-8f0f1e2a","object":"chat.completion.chunk","created":1758542400,"model":"orion-plus","choices":[{"index":0,"delta":{}}],"usage":{"prompt_tokens":14,"completion_tokens":17,"total_tokens":31,"prompt_tokens_details":{"cached_tokens":0},"cost":1.1614e-05}}

data: [DONE]
```

Rebuild the message by concatenating `delta.content` across chunks. The first chunk carries `delta.role`; the last content chunk carries a non-null `finish_reason`.

## Usage in the final chunk

Every stream ends with one extra chunk before `[DONE]` that carries the `usage` object, whether or not you send `stream_options: {"include_usage": true}`: the API asks the model for it on every stream. That chunk is the only place where usage appears in a streamed response, so keep reading until `[DONE]` if you meter consumption on your side.

Two details of that chunk differ from what some OpenAI clients expect:

-   **Its `choices` array is not empty.** It holds one choice with an empty `delta` and no `finish_reason`, as in the example above. Read `usage` from whichever chunk carries it, and do not detect the usage chunk by an empty `choices` array. Code that reads `choices[0].delta.content` gets nothing from it, as from any chunk without text.
-   **Its `usage` carries `cost`** on a Serenity Edge API key: an estimate of the request's price in US dollars at the published prices, with cached tokens at the cached input price. It ignores contract rates and discounts, so it is informational and not the bill, which is computed from the token counts (see [Pricing and limits](https://docs.serenityedge.ai/docs/pricing-and-limits/#prices)). It is not sent on non-streamed answers or on the [Responses API](https://docs.serenityedge.ai/docs/responses-api/).

## SDK examples

### Python

```python
from openai import OpenAI

client = OpenAI(base_url="https://api.serenityedge.ai/v1", api_key="YOUR_SERENITY_EDGE_API_KEY")

stream = client.chat.completions.create(
    model="orion-plus",
    messages=[{"role": "user", "content": "Write a haiku about the Mediterranean."}],
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)
    if chunk.usage:
        print(f"\n\nTokens: {chunk.usage.total_tokens}")
```

### Node.js

```javascript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.serenityedge.ai/v1",
  apiKey: process.env.SERENITY_EDGE_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "orion-plus",
  messages: [{ role: "user", content: "Write a haiku about the Mediterranean." }],
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  if (chunk.usage) console.log(`\n\nTokens: ${chunk.usage.total_tokens}`);
}
```

### .NET

```csharp
using OpenAI.Chat;

ChatClient chat = client.GetChatClient("orion-plus");

await foreach (StreamingChatCompletionUpdate update in chat.CompleteChatStreamingAsync(
    new UserChatMessage("Write a haiku about the Mediterranean.")))
{
    foreach (ChatMessageContentPart part in update.ContentUpdate)
        Console.Write(part.Text);

    if (update.Usage is not null)
        Console.WriteLine($"\n\nTokens: {update.Usage.TotalTokenCount}");
}
```

## Streaming tool calls

When the model decides to call a tool, the chunks carry `delta.tool_calls` fragments instead of `delta.content`. Each fragment has an `index`; the first fragment for an index carries the `id` and the function `name`, later fragments carry pieces of `function.arguments`. Concatenate the argument fragments per index, then parse the JSON once `finish_reason` is `tool_calls`. If it is `length`, the output limit cut the answer, possibly in the middle of a call: do not run it. See [Tool calling](https://docs.serenityedge.ai/docs/tool-calling/#check-finish_reason-and-usage-before-you-run-a-call).

## Errors during a stream

How an error reaches you depends on when it happens:

-   **Before the model is called** (invalid key, bad parameter, the rate limits of your access route): a normal JSON error with the matching HTTP status, not an event stream.
-   **When the model answers within 2 seconds**: an error it returns is also a normal JSON error with its HTTP status.
-   **Otherwise the stream is committed first.** If the model has not answered after 2 seconds (while it reads a long prompt, for example), the API answers HTTP 200 with `Content-Type: text/event-stream` and sends a `: ping` comment line at once, then one every 10 seconds until the model answers, so that the connection stays active. A ping is also sent during the generation when no chunk has arrived for 10 seconds. Comment lines start with `:` and carry no data; the OpenAI SDKs and other SSE parsers skip them.

Once the stream is committed, and for an error in the middle of a generation, the HTTP status stays 200 and the error arrives as one last `data:` event carrying the [error object](https://docs.serenityedge.ai/docs/errors-and-rate-limits/#error-object), with the same `type` and `code` as the HTTP error would have, followed by `[DONE]`:

```text
: ping

data: {"error":{"message":"The service is at capacity for this model. Retry after 5 seconds.","type":"rate_limit_error","code":"rate_limit_exceeded","param":null}}

data: [DONE]
```

So `[DONE]` alone does not mean success: check each `data:` payload for an `error` member before reading `choices`. A capacity error states the seconds to wait in its message, because no `Retry-After` header can be sent once the stream has started. The OpenAI Python SDK raises `openai.APIError` with this error object when it reads such an event. A stream that closes without `[DONE]` was cut off, for example by the network; treat it as failed and retry if appropriate.
