Data Types & Constraints

Schema Overview

The Supervised AI Testing API utilizes a strictly typed JSON schema for all request and response bodies. To ensure consistency across the Supervised AI platform, all data must adhere to the following primitive types and structured object formats.

Base Data Types

The following primitive types are used throughout the API:

| Type | Description | Example | | :--- | :--- | :--- | | UUID | A universally unique identifier (v4). | "f47ac10b-58cc-4372-a567-0e02b2c3d479" | | String | UTF-8 encoded text. | "Model Evaluation Alpha" | | Float | 64-bit floating-point number. | 0.985 | | Integer | 32-bit signed integer. | 500 | | Boolean | Logical true or false. | true | | ISO-8601 | Date and time in UTC. | "2023-10-27T10:00:00Z" |

Core Objects & Constraints

Test Case Object

The TestCase object represents a single unit of evaluation data.

| Field | Type | Required | Description | Constraints | | :--- | :--- | :--- | :--- | :--- | | id | UUID | Yes | Unique identifier for the test case. | Unique within the dataset. | | input | String / Object | Yes | The data passed to the model. | Max size 2MB. | | expected_output | String / Object | Yes | The ground truth for evaluation. | Must match model output format. | | tags | Array<String> | No | Metadata for filtering test cases. | Max 10 tags per case. | | priority | Integer | No | Execution weight. | Value between 1 and 5. |

Example:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "input": "Summarize the latest quarterly report.",
  "expected_output": "The report shows a 15% increase in revenue...",
  "tags": ["production", "regression"],
  "priority": 3
}

Evaluation Metric Constraint

Used to define the success criteria for a testing run.

| Field | Type | Required | Description | Constraints | | :--- | :--- | :--- | :--- | :--- | | metric_name | Enum | Yes | The evaluation metric to apply. | See Metric Types. | | threshold | Float | Yes | The minimum passing score. | 0.0 to 1.0. | | comparator | String | Yes | Comparison operator. | GT, GTE, LT, LTE. |

Example:

{
  "metric_name": "ROUGE_L",
  "threshold": 0.85,
  "comparator": "GTE"
}

Enumerations

Metric Types

The metric_name field must be one of the following supported values:

• ACCURACY: Direct match between prediction and ground truth.
• F1_SCORE: Harmonic mean of precision and recall.
• BLEU: Evaluates candidate text against reference text.
• ROUGE_L: Measures longest common subsequence for summarization tasks.
• LATENCY: Response time in milliseconds (Constraint: LT or LTE recommended).

Test Status

The status of a testing job or individual test case.

• PENDING: Queued for execution.
• RUNNING: Currently being processed by the evaluation engine.
• PASSED: Met or exceeded all metric constraints.
• FAILED: Failed one or more metric constraints.
• ERROR: System or model failure during execution.

Global Validation Rules

1. Strict Typing: Passing a string representation of a number where a Float or Integer is expected will result in a 400 Bad Request.
2. Character Limits: All String fields, unless otherwise specified, have a default maximum length of 4096 characters.
3. Null Values: Optional fields should be omitted rather than passed as null unless the specific endpoint documentation permits nullable values.
4. Array Limits: Collection objects (like tags or constraints) are limited to a maximum of 50 items per request to ensure performance.