RedRaven Attack Runs

RedRaven has two paths on the same test. Your LLM key stays in your process on both.

• One-shot: generate cases, run them with the Python SDK call_agent, and read a pass rate. See the quickstart.
• Attack runs: probe the agent, generate attack scenarios, pick techniques, run them against your target, and measure Attack Success Rate (ASR). The SDK does not cover this path; call /api/v2 yourself.

Configure

Use the same API key headers as one-shot requests. Attack-run routes live under /api/v2.

export REDRAVEN_API_KEY="rr_..."
export REDRAVEN_ORGANIZATION_ID="00000000-0000-0000-0000-000000000000"
export REDRAVEN_BASE_URL="https://api.redraven.fireraven.ai"

curl -sS -X POST \
  -H "X-API-Key: $REDRAVEN_API_KEY" \
  -H "X-Organization-Id: $REDRAVEN_ORGANIZATION_ID" \
  -H "Content-Type: application/json" \
  -d '{"business_context":"Healthcare SaaS.","use_case":"Symptom triage."}' \
  "$REDRAVEN_BASE_URL/api/v2/tests/$TEST_ID/reconnaissance/generate"

Reconnaissance

Generate single-turn probes of the attack surface (tools, knowledge base, memory, formats). Call your target with each probe, then submit the request/response pairs for evaluation. An answered probe is a fail (is_answered: true).

1. POST /api/v2/tests/{test_id}/reconnaissance/generate
2. Poll GET /api/v1/tests/{test_id} until metadata.reconnaissance.status is completed and probes is populated.
3. Optionally PATCH /api/v2/tests/{test_id}/reconnaissance to edit prompts.
4. Call your agent with each probe prompt.
5. POST /api/v2/tests/{test_id}/reconnaissance/evaluate with {items: [{probe_id, request, response}]}.

Scenario generation requires evaluated recon (metadata.reconnaissance.results).

Scenarios

A scenario is an attack purpose bound to one policy, not a one-shot test case. Each item is {id, scenario, policy, maximum_multi_turn, enabled}. Default turn budget is 5 (max 15). Disabled items are skipped at run time.

1. POST /api/v2/tests/{test_id}/scenarios/generate (needs policies and evaluated recon)
2. Poll the test until metadata.scenarios.status is ready.
3. PATCH /api/v2/tests/{test_id}/scenarios to edit, enable, or disable items.

Attack Methods

List catalog techniques with GET /api/v2/attack-methods. Techniques are single or multi turn; both use the same attack-run protocol. Save a per-test selection with PATCH /api/v2/tests/{test_id}/attack-methods:

{
  "technique_ids": ["pop", "crescendo"],
  "dimension2": { "warm_up_turns": 1, "attempts": 3 }
}

attempts is 2–5. Each attempt is an isolated conversation. Warm-up turns are benign and are not scored.

Attack-Run Loop

Credits are charged at POST …/attack-run/begin. Then repeat until done:

1. GET …/attack-run/next — next user prompt plus prior messages[]
2. Call your target with messages + new user prompt (resend the full transcript)
3. POST …/attack-run/turns with run_id, attempt_id, request, response, and the extended messages
4. If decision is pending, poll GET …/attack-run/eval-status until continue, stop, or exhausted
5. POST …/attack-run/complete when finished, then GET …/attack-results/summary

import os
import time
import httpx

base = os.environ["REDRAVEN_BASE_URL"].rstrip("/")
test_id = "<your-test-id>"
headers = {
    "X-API-Key": os.environ["REDRAVEN_API_KEY"],
    "X-Organization-Id": os.environ["REDRAVEN_ORGANIZATION_ID"],
}

def my_llm(prompt: str, messages: list[dict]) -> str:
    _ = messages
    return f"echo: {prompt}"

with httpx.Client(base_url=base, headers=headers, timeout=180) as client:
    begin = client.post(f"/api/v2/tests/{test_id}/attack-run/begin").json()
    run_id = begin["run_id"]

    while True:
        nxt = client.get(f"/api/v2/tests/{test_id}/attack-run/next").json()
        if nxt.get("done"):
            break

        prompt = nxt["prompt"]
        prior = nxt.get("messages") or []
        call_messages = prior + [{"role": "user", "content": prompt}]
        response = my_llm(prompt, call_messages)
        after = call_messages + [{"role": "assistant", "content": response}]

        turn = client.post(
            f"/api/v2/tests/{test_id}/attack-run/turns",
            json={
                "run_id": nxt.get("run_id") or run_id,
                "attempt_id": nxt["attempt_id"],
                "request": prompt,
                "response": response,
                "messages": after,
            },
        ).json()

        if turn.get("decision") == "pending":
            while True:
                status = client.get(
                    f"/api/v2/tests/{test_id}/attack-run/eval-status",
                    params={
                        "run_id": turn["run_id"],
                        "attempt_id": turn["attempt_id"],
                        "turn_index": turn["turn_index"],
                    },
                ).json()
                if status.get("decision") != "pending":
                    break
                time.sleep(2)

    client.post(f"/api/v2/tests/{test_id}/attack-run/complete")
    summary = client.get(f"/api/v2/tests/{test_id}/attack-results/summary").json()
    print(summary["summary"])

Do not call next while eval is still pending (the API returns 409). Continuity is the accumulated messages[], not a sticky session.

ASR

ASR is attempts_failed / attempts_total. Stored outcome: "fail" means the attack succeeded (defenses failed). Turn decisions: