RedRaven Python SDK

The RedRaven Python SDK is scoped to RedRaven evaluation workflows. It runs test cases against your model code and submits responses to RedRaven for scoring.

Install

uv add redraven

or:

pip install redraven

Environment

export REDRAVEN_API_KEY="rr_..."
export REDRAVEN_ORGANIZATION_ID="00000000-0000-0000-0000-000000000000"
export REDRAVEN_BASE_URL="https://api.redraven.fireraven.ai"

Core Flow

RedRaven runs in three phases:

Create or select a test.
Run your agent against the cases with call_agent.
Wait for evaluation and read the summary.

Do not call wait_for_evaluation_ready until call_agent returns. call_agent means every case response was submitted; only then can server-side scoring finish.

async with redraven.Client() as client:
    test_id = await client.generate_test(
        generate_kwargs={
            "project_id": "11111111-1111-1111-1111-111111111111",
            "test_name": "SDK generated test",
            "business_context": "Healthcare SaaS for clinicians.",
            "use_case": "Symptom triage assistant.",
            "certifications": ["HIPAA"],
            "max_policies": 5,
            "max_prompts_per_policy": 2,
        },
        wait_for_dataset=True,
    )

    run = await client.call_agent(test_id=test_id, llm=my_llm, concurrency=8)
    await client.wait_for_evaluation_ready(test_id=test_id)
    result = await client.get_eval_summary(
        test_id=test_id,
        expected_cases=run.expected_cases,
    )

For a single convenience call, use generate_and_run_test(generate_kwargs, llm, ...) to generate the dataset, run your agent, wait for evaluation, and return the terminal summary.

Method Roles

Method	Role
`generate_test(generate_kwargs, wait_for_dataset=False, ...)`	Creates a test and optionally waits for dataset artifacts.
`wait_for_dataset_ready(test_id)`	Blocks until the generated dataset is ready.
`call_agent(test_id, llm, ...)`	Runs your callable over cases and submits responses.
`wait_for_evaluation_ready(test_id)`	Waits for server-side scoring after responses have been submitted.
`get_eval_summary(test_id, wait_for_completion=False, ...)`	Reads the SDK eval summary, optionally waiting for completion.
`generate_and_run_test(generate_kwargs, llm, ...)`	Runs the full generate, agent, evaluation, and summary flow.

Resuming Runs

call_agent is resumable by default. If a run is interrupted, call it again with the same test_id; cases already submitted with a terminal status are skipped.

run = await client.call_agent(test_id=test_id, llm=my_llm)

Set resume=False only when you intentionally want to re-run every case.

If a resumed run skips all already-submitted responses, wait_for_evaluation_ready still asks the backend to ensure the evaluation job exists before polling.

Multimodal Tests

For image-mode tests, define an LLM callable that accepts messages:

def my_llm(prompt: str, messages: list[dict] | None = None) -> str:
    payload = messages or [{"role": "user", "content": prompt}]
    # call your provider with payload
    return "..."

HTTP Exports

The SDK focuses on test execution. For stable dashboard-style exports, use the RedRaven HTTP API with the same credentials:

GET /tests/{test_id}/results
GET /tests/{test_id}/recommendations
GET /tests/{test_id}/report/download

Do not confuse these exports with the SDK evaluation summary:

get_eval_summary and GET /tests/{test_id}/results/summary?kind=eval describe SDK pipeline materialization.
GET /tests/{test_id}/results returns stable dashboard metrics for integrations.

Example with httpx:

import os
import httpx

base_url = os.environ["REDRAVEN_BASE_URL"].rstrip("/")
headers = {
    "X-API-Key": os.environ["REDRAVEN_API_KEY"],
    "X-Organization-Id": os.environ["REDRAVEN_ORGANIZATION_ID"],
}

response = httpx.get(
    f"{base_url}/api/v1/tests/{test_id}/results",
    headers=headers,
    timeout=30,
)
response.raise_for_status()
overview = response.json()