OpenAI API (unofficial showcase)
POST/evals/{eval_id}/runs

Create an evaluation run

Creates and starts a run for an evaluation using a specified data source and model configuration. The run's data source is validated against the schema defined by the evaluation; provide data_source in one of the supported formats. Use the returned run identifier to monitor the evaluation run.

  • IdempotentThe SDK sends Idempotency-Key, so a retried request is only applied once.

1 parameter · 3 body fields
eval_idstringrequired
The identifier of the evaluation to create a run for.

Configuration for the evaluation run. data_source is required.

namestringoptional
The name of the run.
metadataobjectoptional
Metadata to attach to the run, provided as an object of string key-value pairs, or null.
data_sourceobjectrequired
Details about the run's data source.

5 status codes
201Returns the created evaluation run, including its identifier, evaluation identifier, status, report URL, result counts, data source, and other run details.
objectstringrequired
The type of the object. Always "eval.run".
Allowed:eval.runDefault:eval.run
idstringrequired
Unique identifier for the evaluation run.
eval_idstringrequired
The identifier of the associated evaluation.
statusstringrequired
The status of the evaluation run.
modelstringrequired
The model that is evaluated, if applicable.
namestringrequired
The name of the evaluation run.
created_atintegerrequired
Unix timestamp (in seconds) when the evaluation run was created.
report_urlstringrequired
The URL to the rendered evaluation run report on the UI dashboard.
result_countsobjectrequired
Counters summarizing the outcomes of the evaluation run.
per_model_usagearrayrequired
Usage statistics for each model during the evaluation run.
per_testing_criteria_resultsarrayrequired
Results per testing criteria applied during the evaluation run.
data_sourceobjectrequired
Information about the run's data source.
metadataobjectrequired
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
errorobjectrequired
An object representing an error response from the Eval API.
400Returned when the request is invalid, for example if the evaluation object is missing.
errorobjectrequired
404Returned when the evaluation or a referenced resource is not found.
errorobjectrequired
429Returned when the request is rejected because a rate limit was exceeded.
errorobjectrequired
500Returned when the evaluation run could not be created.
errorobjectrequired

Error handling

A 400 is returned for an invalid request, such as one that omits the evaluation object. A 404 is returned when the evaluation or a referenced resource is not found. A 429 is returned when the request exceeds a rate limit. A 500 is returned when the evaluation run cannot be created. data_source is required and must include type and source. The eval_id path parameter must identify the evaluation to run.