Troubleshooting & FAQ

Troubleshooting

This section covers common issues encountered when integrating with the Supervised AI Testing API and provides actionable solutions.

Authentication Failures (401 Unauthorized)

The API requires a valid Bearer Token for all requests.

Symptoms: Receiving a 401 Unauthorized response.
Checklist:
- Ensure the Authorization header is formatted correctly: Authorization: Bearer <YOUR_TOKEN>.
- Verify that your API key has not expired or been revoked in the Supervised AI dashboard.
- Ensure you are hitting the correct environment (e.g., Development vs. Production keys).

Schema Validation Errors (422 Unprocessable Entity)

The testing API enforces a strict JSON schema for input data and test configurations.

Symptoms: Receiving a 422 error with a response body detailing "Validation failed."
Solution: Compare your request body against the expected schema. A common mistake is passing a single object when an array of test cases is expected.
Example Correct Structure:

{
  "test_suite_id": "ts_88234",
  "inputs": [
    {
      "input_text": "Is this model performing well?",
      "expected_output": "positive"
    }
  ]
}

Request Timeouts (504 Gateway Timeout)

Large test batches or complex model evaluations may exceed standard HTTP timeout windows.

Symptoms: The connection closes before a response is received.
Solution:
- Batching: Break large test suites into smaller chunks (e.g., 50 cases per request).
- Asynchronous Mode: If available for your specific endpoint, use the async=true flag to receive a job ID instead of waiting for immediate results.

Rate Limiting (429 Too Many Requests)

To ensure platform stability, we limit the frequency of API calls.

Symptoms: Receiving a 429 status code.
Solution:
- Implement exponential backoff in your integration logic.
- Check the X-RateLimit-Remaining header to monitor your current usage.

Frequently Asked Questions (FAQ)

What data formats are supported for testing?

Currently, the Testing API supports structured JSON for text-based evaluations. If you are testing multimodal models (images/audio), you must provide publicly accessible URLs or Base64 encoded strings within the JSON payload.

Is there a limit to the number of concurrent test runs?

Yes. Default accounts are limited to 5 concurrent test executions. If you trigger a 6th execution while others are running, it will be queued or rejected depending on your subscription tier.

How do I handle internal "Testing API Structure" errors?

While the testing-api handles the interface, it relies on internal orchestration modules. If you receive a 500 Internal Server Error, please check the error_ref field in the response and provide it to our support team.

Can I run tests against a local model?

The Testing API is designed to evaluate models hosted on the Supervised AI platform or accessible via a public/private endpoint. To test a local model, you must use our Tunneling Agent to expose your local port to our testing infrastructure.

What is the difference between `validation` and `evaluation` modes?

Validation (mode: "val"): Checks if the model output matches the schema and basic constraints.
Evaluation (mode: "eval"): Performs deep analysis, including LLM-assisted grading or semantic similarity checks against ground truth data.

Input/Output Type Summary

Troubleshooting & FAQ