Supervised AI Ecosystem

Integration Overview

The testing-api serves as the standardized communication layer between the Supervised AI Core Platform and the various evaluation environments. It provides a structured interface that allows the platform to trigger tests, monitor evaluation progress, and ingest performance metrics across diverse AI models.

In the Supervised AI ecosystem, this API acts as the bridge that translates high-level testing requirements (defined in the platform UI or CLI) into executable tasks for specific testing runners.

System Architecture Role

The testing structure is designed to interact with three primary components of the Supervised AI platform:

1. Model Registry: Pulls model metadata and endpoint configurations to ensure the test suite is targeting the correct version.
2. Dataset Management: Interfaces with the platform's data layer to inject ground-truth datasets and test cases into the evaluation pipeline.
3. Analytics Engine: Routes the raw results from the testing-api back to the platform dashboard for visualization, drift analysis, and bias reporting.

Core Interaction Workflow

To integrate a new testing module or utilize the existing structure, the following lifecycle is followed:

1. Test Configuration

The platform sends a configuration object to the testing-api. This defines the scope of the evaluation, including the metrics to be measured (e.g., accuracy, latency, toxicity).

{
  "test_id": "eval_001x_alpha",
  "model_context": {
    "model_id": "llama-3-8b-instruct",
    "version": "1.2.0"
  },
  "suite_config": {
    "metrics": ["hallucination_rate", "response_time"],
    "thresholds": {
      "hallucination_rate": 0.05
    }
  }
}

2. Execution and Monitoring

The testing-api manages the execution state. Developers can poll the status or set up webhooks to receive real-time updates on the evaluation progress.

| Endpoint | Method | Description | | :--- | :--- | :--- | | /v1/tests/start | POST | Initiates a new testing session based on the provided schema. | | /v1/tests/{id}/status | GET | Retrieves the current state (Pending, Running, Completed, Failed). | | /v1/tests/{id}/logs | GET | Streams execution logs for real-time debugging. |

3. Result Reporting

Once execution is complete, the testing-api standardizes the output into a format compatible with the Supervised AI Analytics Engine.

{
  "test_id": "eval_001x_alpha",
  "status": "success",
  "results": {
    "summary": {
      "passed": 95,
      "failed": 5
    },
    "metrics": {
      "latency_p95": "240ms",
      "accuracy_score": 0.92
    }
  }
}

Security and Authentication

All interactions within the ecosystem are secured via the Supervised AI Identity Provider (IdP).

• Service Accounts: Automated runners should use platform-issued Service Account tokens.
• Scoped Access: API keys must have testing:execute and results:write permissions to interact with the testing structure.

To authenticate your requests, include the bearer token in the header:

curl -X POST https://api.supervised.ai/v1/tests/start \
  -H "Authorization: Bearer ${SUPERVISED_AI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{...}'

Internal Components

While the testing-api is primarily consumed as a set of RESTful endpoints, it utilizes internal Adapters to communicate with specific model frameworks (e.g., PyTorch, TensorFlow, or LLM providers). These adapters are managed internally by the platform and are transparent to the end-user, ensuring a consistent API experience regardless of the underlying model architecture.