Python SDK

The Python SDK package name is redraven and supports Python 3.10+.

Install

Use uv:

uv add redraven

Or with pip:

pip install redraven

Configure

Set your Redraven credentials:

export REDRAVEN_API_KEY="rr_..."
export REDRAVEN_BASE_URL="https://app.redraven.fireraven.ai"

Pass only the host URL. The SDK automatically adds the /api/v1 prefix.

Get the API key from your Redraven organization settings.

Core methods

High-level methods on redraven.Client (in order):

1. generate_test(generate_kwargs, wait_for_dataset=False, ...) -> str (returns test_id)
2. wait_for_dataset_ready(test_id, ...) -> None
3. run_test(test_id, llm, ...) -> EvalSummary
4. generate_and_run_test(generate_kwargs, llm, ...) -> EvalSummary

Quickstart

import asyncio
import redraven

async def my_llm(prompt: str) -> str:
    # Call your own LLM provider here.
    return f"echo: {prompt}"

async def main():
    async with redraven.Client() as client:
        result = await client.run_test(
            test_id="<your-existing-test-id>",
            llm=my_llm,
            concurrency=4,
            retries=2,
            allow_partial=True,
        )
        print(f"state={result.state} received={result.received} failed={result.failed}")

asyncio.run(main())

Prepare a test first

Create a test in the Redraven app (or via the SDK), then use its test_id when calling run_test(...).

Generate tests with the SDK

test_id = await client.generate_test(
    generate_kwargs={
        "project_id": "11111111-1111-1111-1111-111111111111",
        "test_name": "SDK generated test",
        "business_context": "Healthcare SaaS for clinicians.",
        "use_case": "Symptom triage assistant.",
        "certifications": ["HIPAA"],
        "max_policies": 5,
        "max_prompts_per_policy": 2,
    },
    wait_for_dataset=True,
)

generate_test(...) returns the created test_id (string).

You can also run an explicit wait step:

await client.wait_for_dataset_ready(test_id)

Run an existing test

result = await client.run_test(
    test_id=test_id,
    llm=my_llm,
    concurrency=8,
)

One-call flow (generate + run)

result = await client.generate_and_run_test(
    generate_kwargs={
        "project_id": "11111111-1111-1111-1111-111111111111",
        "test_name": "SDK generated test",
        "business_context": "Healthcare SaaS for clinicians.",
        "use_case": "Symptom triage assistant.",
        "certifications": ["HIPAA"],
        "max_policies": 5,
        "max_prompts_per_policy": 2,
    },
    llm=my_llm,
    concurrency=8,
)

Result fields

run_test(...) returns an EvalSummary with:

• state, expected_cases, received, failed, failed_case_ids
• summary (aggregated evaluation output)
• manifest (evaluation trace metadata)

Common errors

• RedravenConfigError: missing API key or base URL
• RedravenHTTPError: backend returned non-2xx response
• RedravenTimeoutError: run did not reach terminal state in time
• RedravenPartialRunError: raised when allow_partial=False and some cases fail

Resumability

If a run is interrupted, call run_test(...) again with the same test_id. Previously submitted cases are safely reused by the backend.

Notes

• The LLM call runs in your process, so your model API key stays local.
• my_llm can be sync or async.