Eval catalog

Checks whether the output directly addresses the user's question or task.

Detects demographic, political, or other unfair bias in model outputs.

Checks whether citations in the output correctly point at source material that supports the cited claim.

Measures how effectively the model used the provided context when producing the output.

Flags traces where the token cost of the output exceeds a configured limit.

Verifies factual claims in the output against widely known ground truth.

Checks whether the output is fully supported by the provided input context (no unsupported claims).

Checks whether the model chose the correct tool/function and supplied valid arguments.

Checks whether the output is grounded in the retrieved context, with no free-floating generation.

Detects fabricated facts, invented citations, or claims not supported by the input.

Detects whether the model was successfully jailbroken — followed instructions to bypass its own safety policies.

Checks whether the output is valid JSON and conforms to the expected schema described in the input.

Verifies the output is written in the language requested by the input.

Flags traces where wall-clock latency exceeds a configured limit.

Checks that multi-turn conversations remain internally consistent across turns (no contradictions with earlier statements).

Validates that the output conforms to a specified schema, structure, or format described in the input.

Detects personally identifiable information leaking into the output (emails, SSNs, phone numbers, addresses).

Detects whether untrusted input content was able to override the system's original instructions.

Detects whether the model successfully resisted prompt injection attempts embedded in the input.

Checks whether refusals are appropriately calibrated — the model refuses truly disallowed requests and does not over-refuse benign ones.

Checks whether the retrieved documents in the input are relevant to the user's query (few irrelevant chunks).

Checks whether the retrieved context contains enough information to fully answer the query.

Checks that output sentiment matches the expected sentiment label provided in the input.

Validates that the model selected the right tool for the task, independent of its argument correctness.

Detects harmful, hateful, harassing, or abusive content in the output.