Automated Testing Workflows

Integration Overview

The testing-api provides a robust interface for validating supervised learning models and datasets. By integrating these tests into your CI/CD pipelines, you ensure that model performance, data integrity, and schema compliance are verified before any deployment to production.

The API is designed to be triggered via standard HTTP requests, making it compatible with any CI tool such as GitHub Actions, GitLab CI/CD, Jenkins, or CircleCI.

Automated CI/CD Workflows

GitHub Actions Integration

The most common way to integrate the testing-api is through a post-training step in your GitHub Actions workflow. This ensures that every time a model is retrained or a dataset is updated, the validation suite runs automatically.

name: Model Validation Pipeline

on:
  push:
    branches: [ main ]
  workflow_dispatch:

jobs:
  validate-model:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger Supervised AI Testing Suite
        run: |
          curl -X POST "https://api.supervised-ai.com/v1/test/run" \
          -H "Authorization: Bearer ${{ secrets.TESTING_API_KEY }}" \
          -H "Content-Type: application/json" \
          -d '{
            "project_id": "supervised-platform-01",
            "model_version": "v2.4.0",
            "dataset_uri": "s3://my-bucket/validation-data/",
            "thresholds": {
              "accuracy": 0.92,
              "f1_score": 0.88
            },
            "environment": "staging"
          }'

CLI Trigger Example

If you prefer using a local runner or a custom shell script within your pipeline, you can use the CLI interface (if installed) or a basic curl command to poll for results.

# Execute tests and wait for result
test-api-cli run --config ./testing-config.json --wait --timeout 300

API Reference: Validation Trigger

To initiate a validation workflow, send a POST request to the /test/run endpoint.

Request Body Schema

Response Structure

The API returns a JSON object containing the test_id which can be used to track the status of the validation.

{
  "status": "initiated",
  "test_id": "val_987654321",
  "estimated_duration_seconds": 45,
  "links": {
    "status_check": "https://api.supervised-ai.com/v1/test/status/val_987654321"
  }
}

Failure Handling and Exit Codes

When integrating with CI/CD, the success or failure of the workflow is determined by the test_result returned by the API.

Success (Exit Code 0): All validation metrics meet or exceed the defined thresholds. The model is safe for deployment.
Failure (Exit Code 1): One or more metrics failed to meet the required thresholds. The pipeline should stop, and the deployment should be blocked.
Error (Exit Code 2): The testing suite could not be completed due to infrastructure issues, missing data, or invalid configuration.

Blocking Deployments

In your CI configuration, you should configure the step to fail if the API returns a fail status. This prevents "bad" models from reaching production:

# Example logic for checking status
STATUS=$(curl -s "https://api.supervised-ai.com/v1/test/status/$TEST_ID" | jq -r '.result')

if [ "$STATUS" == "failed" ]; then
  echo "Validation failed! Blocking deployment."
  exit 1
fi

Security and Authentication

The testing-api requires an API Key for all requests.

Generate an API key from the Supervised AI platform dashboard.
Store this key as a Secret in your CI/CD environment (e.g., TESTING_API_KEY).
Pass the key in the Authorization header as a Bearer token.