model_evaluation_runs

Creates, updates, deletes, gets or lists a model_evaluation_runs resource.

Overview

Name	model_evaluation_runs
Type	Resource
Id	digitalocean.genai.model_evaluation_runs

Fields

The following fields are returned by SELECT queries:

genai_get_model_evaluation_run

A successful response.

Name	Datatype	Description
name	string	Name of the evaluation run. (example: example name)
candidate_model_name	string	(example: example name)
dataset_name	string	(example: example name)
eval_preset_name	string	(example: example name)
judge_model_name	string	(example: example name)
candidate_inference_config	object	Inference configuration for the candidate model during evaluation.
candidate_model_source	string	Whether inference runs against the serverless platform, a dedicated deployment, or a model router. (CANDIDATE_MODEL_SOURCE_SERVERLESS, CANDIDATE_MODEL_SOURCE_DEDICATED, CANDIDATE_MODEL_SOURCE_ROUTER) (default: CANDIDATE_MODEL_SOURCE_SERVERLESS, example: CANDIDATE_MODEL_SOURCE_SERVERLESS)
candidate_model_uuid	string	Candidate model being evaluated. (example: 123e4567-e89b-12d3-a456-426614174000)
completed_at	string (date-time)	(example: 2023-01-01T00:00:00Z)
created_at	string (date-time)	(example: 2023-01-01T00:00:00Z)
dataset_uuid	string	Dataset used for the evaluation. (example: 123e4567-e89b-12d3-a456-426614174000)
error_description	string	Error description if the run failed or partially succeeded. (example: example string)
eval_preset_uuid	string	(example: 123e4567-e89b-12d3-a456-426614174000)
eval_run_uuid	string	UUID of the evaluation run. (example: 123e4567-e89b-12d3-a456-426614174000)
judge_model_uuid	string	Judge model used to score responses. (example: 123e4567-e89b-12d3-a456-426614174000)
metrics	array	Metrics selected for this evaluation.
progress	object	Per-phase progress for a model evaluation run. The candidate phase invokes the candidate model once per dataset row; the judge phase scores each candidate-success row with the configured metrics. Counts grow as the run advances; compare against total_rows to render a progress bar.
result_summary	object	Aggregated result summary for a completed model evaluation run.
star_metric	object
started_at	string (date-time)	(example: 2023-01-01T00:00:00Z)
status	string	Model Evaluation Run Statuses (MODEL_EVALUATION_RUN_STATUS_UNSPECIFIED, MODEL_EVALUATION_RUN_QUEUED, MODEL_EVALUATION_RUN_RUNNING_DATASET, MODEL_EVALUATION_RUN_EVALUATING_RESULTS, MODEL_EVALUATION_RUN_CANCELLING, MODEL_EVALUATION_RUN_CANCELLED, MODEL_EVALUATION_RUN_SUCCESSFUL, MODEL_EVALUATION_RUN_PARTIALLY_SUCCESSFUL, MODEL_EVALUATION_RUN_FAILED) (default: MODEL_EVALUATION_RUN_STATUS_UNSPECIFIED, example: MODEL_EVALUATION_RUN_STATUS_UNSPECIFIED)

genai_list_model_evaluation_runs

A successful response.

Name	Datatype	Description
name	string	Name of the evaluation run. (example: example name)
candidate_model_name	string	Name of the candidate model being evaluated. (example: example name)
dataset_name	string	Name of the dataset used for evaluation. (example: example name)
judge_model_name	string	(example: example name)
candidate_model_source	string	Whether inference runs against the serverless platform, a dedicated deployment, or a model router. (CANDIDATE_MODEL_SOURCE_SERVERLESS, CANDIDATE_MODEL_SOURCE_DEDICATED, CANDIDATE_MODEL_SOURCE_ROUTER) (default: CANDIDATE_MODEL_SOURCE_SERVERLESS, example: CANDIDATE_MODEL_SOURCE_SERVERLESS)
candidate_model_uuid	string	UUID of the candidate model being evaluated. (example: 123e4567-e89b-12d3-a456-426614174000)
created_at	string (date-time)	Timestamp when the run was created. (example: 2023-01-01T00:00:00Z)
dataset_uuid	string	UUID of the dataset used for evaluation. (example: 123e4567-e89b-12d3-a456-426614174000)
eval_run_uuid	string	UUID of the evaluation run. (example: 123e4567-e89b-12d3-a456-426614174000)
judge_model_uuid	string	Judge model used to score responses. (example: 123e4567-e89b-12d3-a456-426614174000)
progress	object	Per-phase progress for a model evaluation run. The candidate phase invokes the candidate model once per dataset row; the judge phase scores each candidate-success row with the configured metrics. Counts grow as the run advances; compare against total_rows to render a progress bar.
status	string	Model Evaluation Run Statuses (MODEL_EVALUATION_RUN_STATUS_UNSPECIFIED, MODEL_EVALUATION_RUN_QUEUED, MODEL_EVALUATION_RUN_RUNNING_DATASET, MODEL_EVALUATION_RUN_EVALUATING_RESULTS, MODEL_EVALUATION_RUN_CANCELLING, MODEL_EVALUATION_RUN_CANCELLED, MODEL_EVALUATION_RUN_SUCCESSFUL, MODEL_EVALUATION_RUN_PARTIALLY_SUCCESSFUL, MODEL_EVALUATION_RUN_FAILED) (default: MODEL_EVALUATION_RUN_STATUS_UNSPECIFIED, example: MODEL_EVALUATION_RUN_STATUS_UNSPECIFIED)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
genai_get_model_evaluation_run	select	eval_run_uuid	page, per_page	To retrieve a model evaluation run, send a GET request to /v2/genai/model_evaluation_runs/{eval_run_uuid}.
genai_list_model_evaluation_runs	select		eval_preset_uuid, status, page, per_page, statuses, candidate_types, search, sort_by, sort_direction	To list model evaluation runs, send a GET request to /v2/genai/model_evaluation_runs.
genai_create_model_evaluation_run	insert			To create a model evaluation run, send a POST request to /v2/genai/model_evaluation_runs.
genai_update_model_evaluation_run	update	eval_run_uuid		To update a model evaluation run's display name, send a PATCH request to /v2/gen-ai/model_evaluation_runs/{eval_run_uuid}.
genai_cancel_model_evaluation_run	replace	eval_run_uuid		To cancel an in-progress model evaluation run, send a PUT request to /v2/gen-ai/model_evaluation_runs/{eval_run_uuid}/cancel.
genai_delete_model_evaluation_run	delete	eval_run_uuid		To delete a model evaluation run, send a DELETE request to /v2/gen-ai/model_evaluation_runs/{eval_run_uuid}. The run must be in a terminal status (successful, partially_successful, failed, or cancelled). For runs still in progress, either wait for the run to finish or cancel it, then retry the delete once the run reaches a terminal status.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
eval_run_uuid	string	UUID of the model evaluation run to delete. The run must be in a terminal status (successful, partially_successful, failed, or cancelled). For runs still in progress, either wait for the run to finish or cancel it, then retry the delete. (example: "123e4567-e89b-12d3-a456-426614174000")
candidate_types	array	Filter by one or more candidate model source types (serverless, dedicated, router). Empty means no candidate-type filter. (example: [CANDIDATE_MODEL_SOURCE_SERVERLESS])
eval_preset_uuid	string	UUID of the evaluation preset to filter by. (example: 123e4567-e89b-12d3-a456-426614174000)
page	integer	Page number. (example: 1)
per_page	integer	Items per page. (example: 1)
search	string	Free-text search across the eval run name, candidate model name and dataset name (case-insensitive substring match). Empty means no search. (example: example string)
sort_by	string	Field to sort by. Defaults to creation date when unspecified. (example: MODEL_EVALUATION_RUN_SORT_FIELD_UNSPECIFIED)
sort_direction	string	Sort direction. Defaults to descending when unspecified. (example: SORT_DIRECTION_UNSPECIFIED)
status	string	Filter by evaluation run status. (example: MODEL_EVALUATION_RUN_STATUS_UNSPECIFIED)
statuses	array	Filter by one or more statuses. Empty means no status filter. (example: [MODEL_EVALUATION_RUN_STATUS_UNSPECIFIED])

SELECT examples

genai_get_model_evaluation_run

To retrieve a model evaluation run, send a GET request to /v2/genai/model_evaluation_runs/{eval_run_uuid}.

SELECT
name,
candidate_model_name,
dataset_name,
eval_preset_name,
judge_model_name,
candidate_inference_config,
candidate_model_source,
candidate_model_uuid,
completed_at,
created_at,
dataset_uuid,
error_description,
eval_preset_uuid,
eval_run_uuid,
judge_model_uuid,
metrics,
progress,
result_summary,
star_metric,
started_at,
status
FROM digitalocean.genai.model_evaluation_runs
WHERE eval_run_uuid = '{{ eval_run_uuid }}' -- required
AND page = '{{ page }}'
AND per_page = '{{ per_page }}'
;

genai_list_model_evaluation_runs

To list model evaluation runs, send a GET request to /v2/genai/model_evaluation_runs.

SELECT
name,
candidate_model_name,
dataset_name,
judge_model_name,
candidate_model_source,
candidate_model_uuid,
created_at,
dataset_uuid,
eval_run_uuid,
judge_model_uuid,
progress,
status
FROM digitalocean.genai.model_evaluation_runs
WHERE eval_preset_uuid = '{{ eval_preset_uuid }}'
AND status = '{{ status }}'
AND page = '{{ page }}'
AND per_page = '{{ per_page }}'
AND statuses = '{{ statuses }}'
AND candidate_types = '{{ candidate_types }}'
AND search = '{{ search }}'
AND sort_by = '{{ sort_by }}'
AND sort_direction = '{{ sort_direction }}'
;

INSERT examples

genai_create_model_evaluation_run

To create a model evaluation run, send a POST request to /v2/genai/model_evaluation_runs.

INSERT INTO digitalocean.genai.model_evaluation_runs (
candidate_inference_config,
candidate_model_name,
candidate_model_source,
candidate_model_uuid,
dataset_uuid,
eval_preset_uuid,
judge_model_uuid,
metric_uuids,
name,
preset_name,
preset_save_sections,
save_as_preset,
source,
star_metric
)
SELECT 
'{{ candidate_inference_config }}',
'{{ candidate_model_name }}',
'{{ candidate_model_source }}',
'{{ candidate_model_uuid }}',
'{{ dataset_uuid }}',
'{{ eval_preset_uuid }}',
'{{ judge_model_uuid }}',
'{{ metric_uuids }}',
'{{ name }}',
'{{ preset_name }}',
'{{ preset_save_sections }}',
{{ save_as_preset }},
'{{ source }}',
'{{ star_metric }}'
RETURNING
eval_run_uuid
;

Manifest

# Description fields are for documentation purposes
- name: model_evaluation_runs
props:
  - name: candidate_inference_config
    description: |
      Inference configuration for the candidate model during evaluation.
    value:
      max_tokens: {{ max_tokens }}
      stop_token: "{{ stop_token }}"
      system_prompt: "{{ system_prompt }}"
      temperature: {{ temperature }}
  - name: candidate_model_name
    value: "{{ candidate_model_name }}"
    description: |
      Model slug used to call the candidate model API.
      For dedicated inference, this is the model slug from the deployment.
      For serverless, this should match the model's internal name.
  - name: candidate_model_source
    value: "{{ candidate_model_source }}"
    description: |
      Whether inference runs against the serverless platform, a dedicated deployment, or a model router.
    valid_values: ['CANDIDATE_MODEL_SOURCE_SERVERLESS', 'CANDIDATE_MODEL_SOURCE_DEDICATED', 'CANDIDATE_MODEL_SOURCE_ROUTER']
    default: CANDIDATE_MODEL_SOURCE_SERVERLESS
  - name: candidate_model_uuid
    value: "{{ candidate_model_uuid }}"
    description: |
      UUID of the candidate model to evaluate.
  - name: dataset_uuid
    value: "{{ dataset_uuid }}"
    description: |
      UUID of the dataset to use for evaluation.
  - name: eval_preset_uuid
    value: "{{ eval_preset_uuid }}"
  - name: judge_model_uuid
    value: "{{ judge_model_uuid }}"
    description: |
      UUID of the judge model used to score responses.
  - name: metric_uuids
    value:
      - "{{ metric_uuids }}"
    description: |
      UUIDs of metrics to evaluate (selected from ListModelEvaluationMetrics).
  - name: name
    value: "{{ name }}"
  - name: preset_name
    value: "{{ preset_name }}"
  - name: preset_save_sections
    value:
      - "{{ preset_save_sections }}"
    description: |
      Which sections of this run's resolved configuration to persist as a
      reusable preset. Each selected section saves only its own fields; the
      remaining sections stay empty on the preset and must be supplied inline
      on future runs that reference it. Empty means do not save a preset
      (unless the deprecated `save_as_preset` boolean is true, in which case
      all sections are saved). Ignored when `eval_preset_uuid` is set. Use
      `preset_name` to label the saved preset.
  - name: save_as_preset
    value: {{ save_as_preset }}
    description: |
      Deprecated: use `preset_save_sections`. When `true` and
      `preset_save_sections` is empty, all five sections of the resolved
      configuration are saved as a reusable preset (legacy behavior). Ignored
      when `eval_preset_uuid` is set.
  - name: source
    value: "{{ source }}"
    description: |
      Source of the run creation (api, sdk, cli).
  - name: star_metric
    value:
      metric_uuid: "{{ metric_uuid }}"
      name: "{{ name }}"
      success_threshold: {{ success_threshold }}
      success_threshold_pct: {{ success_threshold_pct }}

UPDATE examples

genai_update_model_evaluation_run

To update a model evaluation run's display name, send a PATCH request to /v2/gen-ai/model_evaluation_runs/{eval_run_uuid}.

UPDATE digitalocean.genai.model_evaluation_runs
SET 
eval_run_uuid = '{{ eval_run_uuid }}',
name = '{{ name }}'
WHERE 
eval_run_uuid = '{{ eval_run_uuid }}' --required
RETURNING
run;

REPLACE examples

genai_cancel_model_evaluation_run

To cancel an in-progress model evaluation run, send a PUT request to /v2/gen-ai/model_evaluation_runs/{eval_run_uuid}/cancel.

REPLACE digitalocean.genai.model_evaluation_runs
SET 
eval_run_uuid = '{{ eval_run_uuid }}'
WHERE 
eval_run_uuid = '{{ eval_run_uuid }}' --required
RETURNING
run;

DELETE examples

genai_delete_model_evaluation_run

To delete a model evaluation run, send a DELETE request to /v2/gen-ai/model_evaluation_runs/{eval_run_uuid}. The run must be in a terminal status (successful, partially_successful, failed, or cancelled). For runs still in progress, either wait for the run to finish or cancel it, then retry the delete once the run reaches a terminal status.

DELETE FROM digitalocean.genai.model_evaluation_runs
WHERE eval_run_uuid = '{{ eval_run_uuid }}' --required
;