Get Evaluation Answer

GET

https://api.captain.dev/v3/evals/:eval_id/answers/:case_id/:config_name

GET

/v3/evals/:eval_id/answers/:case_id/:config_name

import requests
BASE_URL = "https://api.captain.dev"
API_KEY = "your_api_key"
EVAL_ID = "eval_019b4c2f11d04c8e9a7b3d5c2e1840ff"
CASE_ID = "auth-window"
CONFIG = "rerank"
response = requests.get(
    f"{BASE_URL}/v3/evals/{EVAL_ID}/answers/{CASE_ID}/{CONFIG}",
    headers={"Authorization": f"Bearer {API_KEY}"},
    timeout=30.0
)
if response.status_code == 200:
    answer = response.json()
    print(f"query: {answer['request']['query']}")
    for i, r in enumerate(answer["response"]["results"], start=1):
        print(f"[{i}] {r['document']['filename']}  score={r['score']:.3f}")
else:
    print(f"Error: {response.status_code}")

{
  "config_name": "rerank",
  "eval_id": "eval_019b4c2f11d04c8e9a7b3d5c2e1840ff",
  "id": "auth-window",
  "request": {
    "query": "What language authorizes the merchant to charge the card on file?",
    "limit": 10,
    "rerank": {
      "enabled": true,
      "candidate_limit": 30,
      "model": "voyage-rerank-2.5"
    },
    "semantic_ratio": 0.5,
    "filter": {
      "policy_area": {
        "$eq": "payments"
      }
    }
  },
  "response": {
    "total_results": 3,
    "execution_time_ms": 405,
    "results": [
      {
        "chunk_id": "9f8e7d6c5b4a39281706f5e4d3c2b1a0:2",
        "score": 0.71,
        "rerank_score": 0.94,
        "text": "By signing below the merchant is authorized to charge the card on file for the amounts set out in Schedule A.",
        "document": {
          "id": "9f8e7d6c5b4a39281706f5e4d3c2b1a0",
          "filename": "payment-authorization-form.pdf"
        }
      }
    ]
  },
  "status": "scored",
  "query_id": "019b4c2f-2a71-7d04-9e61-8c3b5a2f1d77"
}

The exact Query v3 request one unit sent and the response it received, kept on the eval for as long as the eval exists. 404 until the unit has been queried.

Path parameters

eval_idstringRequired

case_idstringRequired

config_namestringRequired

Response

Successful Response

config_namestring

eval_idstring

idstring

The case id (items[].id).

requestobject

The exact Query v3 request body the unit sent.

responseobject

The exact Query v3 response body the unit received.

statusenum

query_idstring or nullOptional

Errors

422

Unprocessable Entity Error