Tools & structured output

Send the OpenAI tools array and response_format as you already do. Stav translates them for each Inference Provider, and when you route, only models that support them are considered.

Function calling

import json

tools = [{
    "type": "function",
    "function": {
        "name": "get_invoice",
        "description": "Look up an invoice by number.",
        "parameters": {
            "type": "object",
            "properties": {"invoice_no": {"type": "string"}},
            "required": ["invoice_no"],
        },
    },
}]

messages = [{"role": "user", "content": "What's the total on INV-4471?"}]
resp = client.chat.completions.create(model="auto", messages=messages, tools=tools)

call = resp.choices[0].message.tool_calls[0]
print(call.function.name, call.function.arguments)

Your application runs the tool. Append the assistant message, append a tool message with the result, and call again:

messages.append(resp.choices[0].message)
messages.append({
    "role": "tool",
    "tool_call_id": call.id,
    "content": json.dumps(lookup_invoice(call.function.arguments)),
})
final = client.chat.completions.create(model="auto", messages=messages, tools=tools)

Tool-call deltas stream normally.

Routing with tools

When model is auto or a Router, sending tools limits the candidates to models that support function calling. Failover candidates meet the same rule.

Structured output

response_format accepts:

{"type": "text"}                                  # default
{"type": "json_object"}                           # any valid JSON
{"type": "json_schema", "json_schema": {...}}     # schema-conformant, strict only

{"type": "grammar"} (regex or EBNF) is a 400 unsupported_parameter. No Inference Provider on Stav offers it. For a fixed set of labels, use json_schema with an enum.

JSON Schema

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": contract_text}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "clause_extraction",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "governing_law"

strict must be true. A missing or false value is rejected with a 400: a schema that is only sometimes enforced gives you no guarantee. If you want loose JSON, use json_object and validate yourself.

Support by dialect

Routing with schemas

json_schema limits routing to models with structured-output support. json_object limits it to models with a JSON mode or structured output.

grammar

The request schema also accepts {"type": "grammar", ...} (regex or EBNF). No Inference Provider in the catalogue supports it today, so it returns 501. Use json_schema or an enum in a schema instead.

Combining tools, schemas and reasoning

resp = client.chat.completions.create(
    model="auto",
    messages=messages,
    tools=tools,
    response_format={"type": "json_schema", "json_schema": schema},
    reasoning_effort="medium",
)

With auto, the router only considers models that support all three. See Reasoning.

When a capability needs another API

Some models take a capability on one provider API and not on another. For example, a model can accept tools on Chat Completions only with reasoning_effort: "none", and with any level on the provider's Responses API.

Stav measures this per Provider Endpoint and per API, and verifies it with a real request before it uses it. When your request needs a capability that the natural API cannot deliver with your parameters, Stav sends it to the API of the same endpoint that can:

• It never changes a parameter you sent. It never moves the request to another endpoint or provider.
• X-Stav-Upstream-Surface names the API that served, and Your workspace → Monitor → Request log shows one line for it.
• The answer comes back in the shape you asked for.
• When the provider serves the request through its Responses API, Stav asks it not to store anything (store: false).

If no API of the endpoint can deliver it with your parameters, the request fails with 400 capability_unreachable. param names the parameter, and the message says which values work. See Errors.

Reasoning across tool turns. When a turn was served through the Responses API, the assistant message carries stav_reasoning_state (on /v1/messages, the signature of a thinking block; on /v1/responses, the encrypted_content of a reasoning item). Send the message back unchanged on the next turn and the model continues its reasoning. The state is opaque and signed, works only on the same Provider Endpoint, and is never stored or logged. If your client drops it, nothing fails; the model starts its reasoning again.

Gemini thought signatures. Gemini 3 models sign each function call and want the signature back on the next turn. On /v1/chat/completions it is on the tool call, at extra_content.google.thought_signature, as Google sends it. On /v1/messages and /v1/responses it is inside the same state. In both cases: send the assistant turn back unchanged. If a signature is lost, the request still works; Google then skips its check for that function call, which can lower the quality of the answer.

Prompt caching

cache_control markers on messages, tools or the system prompt are honoured on Anthropic models, on both /v1/chat/completions and /v1/messages. On every other dialect they have no effect and cause no error. OpenAI, xAI and Mistral cache prefixes on their own.

Cache reads appear as usage.prompt_tokens_details.cached_tokens, a part of prompt_tokens, not an addition to it.

Other surfaces

• /v1/messages — Anthropic tools with input_schema work on every model. Anthropic server tools are left out and named in X-Stav-Dropped-Params unless the Provider Endpoint serves the Messages API itself (Claude models), where the request is proxied unchanged. See Anthropic SDK migration.
• /v1/responses — function tools work on every model. Hosted tools only work where the Provider Endpoint serves the Responses API itself (OpenAI and xAI models), where the request is proxied unchanged. See OpenAI SDK migration.

What Stav does not provide

• No server-side tool execution on the translated paths. Your tools are your functions.
• No vector store or rerank endpoint. /v1/embeddings is available for your own retrieval pipeline. The one file upload is /v1/audio/transcriptions — audio in, text out; it is not an attachment to a chat turn.
• Vision input is supported. Send image content blocks; routing limits the pool to vision-capable models.

Next steps

• Request parameters
• Smart Router
• Reasoning