Skip to main content

Streaming

Set stream: true to receive the response as it is generated. The API answers with Content-Type: text/event-stream and sends one data: line per chunk, each containing a chat.completion.chunk object. The stream ends with data: [DONE].

Request​

curl https://api.serenityedge.ai/v1/chat/completions \
-H "Authorization: Bearer $SERENITY_EDGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "orion-plus",
"stream": true,
"stream_options": {"include_usage": true},
"messages": [{"role": "user", "content": "Write a haiku about the Mediterranean."}]
}'

Response​

data: {"id":"chatcmpl-8f0f1e2a","object":"chat.completion.chunk","created":1758542400,"model":"orion-plus","choices":[{"index":0,"delta":{"role":"assistant","content":"Blue"},"finish_reason":null}]}

data: {"id":"chatcmpl-8f0f1e2a","object":"chat.completion.chunk","created":1758542400,"model":"orion-plus","choices":[{"index":0,"delta":{"content":" waves"},"finish_reason":null}]}

data: {"id":"chatcmpl-8f0f1e2a","object":"chat.completion.chunk","created":1758542400,"model":"orion-plus","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-8f0f1e2a","object":"chat.completion.chunk","created":1758542400,"model":"orion-plus","choices":[{"index":0,"delta":{}}],"usage":{"prompt_tokens":14,"completion_tokens":17,"total_tokens":31,"prompt_tokens_details":{"cached_tokens":0},"cost":1.1614e-05}}

data: [DONE]

Rebuild the message by concatenating delta.content across chunks. The first chunk carries delta.role; the last content chunk carries a non-null finish_reason.

Usage in the final chunk​

Every stream ends with one extra chunk before [DONE] that carries the usage object, whether or not you send stream_options: {"include_usage": true}: the API asks the model for it on every stream. That chunk is the only place where usage appears in a streamed response, so keep reading until [DONE] if you meter consumption on your side.

Two details of that chunk differ from what some OpenAI clients expect:

  • Its choices array is not empty. It holds one choice with an empty delta and no finish_reason, as in the example above. Read usage from whichever chunk carries it, and do not detect the usage chunk by an empty choices array. Code that reads choices[0].delta.content gets nothing from it, as from any chunk without text.
  • Its usage carries cost on a Serenity Edge API key: an estimate of the request's price in US dollars at the published prices, with cached tokens at the cached input price. It ignores contract rates and discounts, so it is informational and not the bill, which is computed from the token counts (see Pricing and limits). It is not sent on non-streamed answers or on the Responses API.

SDK examples​

Python​

from openai import OpenAI

client = OpenAI(base_url="https://api.serenityedge.ai/v1", api_key="YOUR_SERENITY_EDGE_API_KEY")

stream = client.chat.completions.create(
model="orion-plus",
messages=[{"role": "user", "content": "Write a haiku about the Mediterranean."}],
stream=True,
stream_options={"include_usage": True},
)

for chunk in stream:
if chunk.choices:
print(chunk.choices[0].delta.content or "", end="", flush=True)
if chunk.usage:
print(f"\n\nTokens: {chunk.usage.total_tokens}")

Node.js​

import OpenAI from "openai";

const client = new OpenAI({
baseURL: "https://api.serenityedge.ai/v1",
apiKey: process.env.SERENITY_EDGE_API_KEY,
});

const stream = await client.chat.completions.create({
model: "orion-plus",
messages: [{ role: "user", content: "Write a haiku about the Mediterranean." }],
stream: true,
stream_options: { include_usage: true },
});

for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
if (chunk.usage) console.log(`\n\nTokens: ${chunk.usage.total_tokens}`);
}

.NET​

using OpenAI.Chat;

ChatClient chat = client.GetChatClient("orion-plus");

await foreach (StreamingChatCompletionUpdate update in chat.CompleteChatStreamingAsync(
new UserChatMessage("Write a haiku about the Mediterranean.")))
{
foreach (ChatMessageContentPart part in update.ContentUpdate)
Console.Write(part.Text);

if (update.Usage is not null)
Console.WriteLine($"\n\nTokens: {update.Usage.TotalTokenCount}");
}

Streaming tool calls​

When the model decides to call a tool, the chunks carry delta.tool_calls fragments instead of delta.content. Each fragment has an index; the first fragment for an index carries the id and the function name, later fragments carry pieces of function.arguments. Concatenate the argument fragments per index, then parse the JSON once finish_reason is tool_calls. If it is length, the output limit cut the answer, possibly in the middle of a call: do not run it. See Tool calling.

Errors during a stream​

How an error reaches you depends on when it happens:

  • Before the model is called (invalid key, bad parameter, the rate limits of your access route): a normal JSON error with the matching HTTP status, not an event stream.
  • When the model answers within 2 seconds: an error it returns is also a normal JSON error with its HTTP status.
  • Otherwise the stream is committed first. If the model has not answered after 2 seconds (while it reads a long prompt, for example), the API answers HTTP 200 with Content-Type: text/event-stream and sends a : ping comment line at once, then one every 10 seconds until the model answers, so that the connection stays active. A ping is also sent during the generation when no chunk has arrived for 10 seconds. Comment lines start with : and carry no data; the OpenAI SDKs and other SSE parsers skip them.

Once the stream is committed, and for an error in the middle of a generation, the HTTP status stays 200 and the error arrives as one last data: event carrying the error object, with the same type and code as the HTTP error would have, followed by [DONE]:

: ping

data: {"error":{"message":"The service is at capacity for this model. Retry after 5 seconds.","type":"rate_limit_error","code":"rate_limit_exceeded","param":null}}

data: [DONE]

So [DONE] alone does not mean success: check each data: payload for an error member before reading choices. A capacity error states the seconds to wait in its message, because no Retry-After header can be sent once the stream has started. The OpenAI Python SDK raises openai.APIError with this error object when it reads such an event. A stream that closes without [DONE] was cut off, for example by the network; treat it as failed and retry if appropriate.