OpenAI API (unofficial showcase)
POST/evals

Create an evaluation

Creates an evaluation with a data source configuration and testing criteria for assessing model performance. Supply data_source_config to define the evaluation data and testing_criteria to specify the graders; name and metadata are optional. Run the evaluation against models and model configurations by creating an evaluation run.

  • IdempotentThe SDK sends Idempotency-Key, so a retried request is only applied once.

4 body fields

Configuration for the evaluation, including its data source and testing criteria. data_source_config and testing_criteria are required.

namestringoptional
The name of the evaluation.
metadataobjectoptional
A set of up to 16 key-value pairs attached to the evaluation; keys are strings of up to 64 characters and values are strings.
data_source_configobjectrequired
The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
testing_criteriaarray<LabelModelGrader>required
A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).

3 status codes
201Returns the created evaluation, including its identifier, name, data source configuration, testing criteria, creation timestamp, and metadata.
objectstringrequired
The object type.
Allowed:evalDefault:eval
idstringrequired
Unique identifier for the evaluation.
namestringrequired
The name of the evaluation.
data_source_configobjectrequired
Configuration of data sources used in runs of the evaluation.
testing_criteriaarray<LabelModelGrader>required
A list of testing criteria.
created_atintegerrequired
The Unix timestamp (in seconds) for when the eval was created.
metadataobjectrequired
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
400Returned when the evaluation input or grader configuration is invalid.
errorobjectrequired
429Returned when the request is rejected because a rate limit was exceeded.
errorobjectrequired

Error handling

A 400 is returned when the evaluation input or grader configuration is invalid. A 429 is returned when the request exceeds a rate limit. data_source_config and testing_criteria are required; each data source configuration must include its required fields, including type and, where specified, item_schema or source.