Data Flow & Lifecycle

Data Flow Overview

The testing-api facilitates the systematic evaluation of AI models within the Supervised AI platform. Data flows through a structured lifecycle, moving from raw input ingestion to processed evaluation metrics. This process ensures that test suites are executed consistently and results are stored for historical comparison.

1. Request Initiation (Input Data)

Users interact with the API by submitting a test execution request. This is the entry point where external data (model references and test parameters) enters the system.

Input Payload

To initiate a test run, the user must provide a configuration object containing the model identifiers and the dataset scope.

| Field | Type | Description | | :--- | :--- | :--- | | model_id | String | The unique identifier for the AI model being tested. | | test_suite_id | String | The ID of the predefined set of test cases to execute. | | parameters | Object | (Optional) Key-value pairs for hyperparameter overrides or environment flags. | | callback_url | String | (Optional) A webhook URL to notify upon completion. |

Example Request:

POST /v1/tests/run
{
  "model_id": "model-001-beta",
  "test_suite_id": "regression-suite-v2",
  "parameters": {
    "threshold": 0.85,
    "concurrency": 4
  }
}

2. Processing Lifecycle

Once a request is accepted, the test enters a managed lifecycle. The API tracks the state of the data as it moves through internal processing queues.

Lifecycle States

• PENDING: The request has been received and is queued for execution.
• RUNNING: The model is currently processing the test suite data.
• COMPLETED: Processing is finished, and metrics have been generated.
• FAILED: An error occurred during execution (e.g., model timeout or data mismatch).

State Retrieval

Users can poll the status of a specific test run using the test_id returned during initiation.

Example Usage:

GET /v1/tests/{test_id}/status

# Response
{
  "test_id": "test_7890abc",
  "status": "RUNNING",
  "progress": 0.45,
  "started_at": "2023-10-27T10:00:00Z"
}

3. Data Output & Evaluation

After the "COMPLETED" state is reached, the API transforms raw model outputs into structured evaluation metrics. This is the final stage of the data flow where results are made available for analysis.

Output Schema

The response contains both metadata about the run and the specific performance metrics defined in the test suite.

| Property | Type | Description | | :--- | :--- | :--- | | results | Object | The core metrics (e.g., accuracy, f1-score, latency). | | artifacts | Array | URIs to generated logs or visualization files. | | summary | String | A high-level pass/fail determination based on configured thresholds. |

Example Response:

{
  "test_id": "test_7890abc",
  "status": "COMPLETED",
  "results": {
    "accuracy": 0.942,
    "latency_ms": 120,
    "f1_score": 0.91
  },
  "summary": "PASSED"
}

4. Internal Data Handling

While the user interacts with the public endpoints, the API handles several internal transitions to ensure data integrity:

• Validation Layer: Before queuing, the API validates that the model_id is active and the test_suite_id exists.
• Persistence: All request parameters and resulting metrics are persisted to the platform's database, allowing for audit trails and version comparison.
• Cleanup: Temporary processing data generated during the model execution is purged after the final metrics are extracted and stored.