feat(azure-ai-projects): add endpoint evaluator samples for API Key auth and entra id auth#47669
feat(azure-ai-projects): add endpoint evaluator samples for API Key auth and entra id auth#47669ahmad-nader wants to merge 1 commit into
Conversation
…nd Entra ID auth Add two E2E samples demonstrating endpoint-based evaluators: - sample_endpoint_evaluator_with_api_key.py: API Key authentication flow - sample_endpoint_evaluator_with_entra_id.py: Entra ID (managed identity) authentication flow Both samples show the full workflow: create connection, register evaluator, create evaluation, run with inline data, and poll for results. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Thank you for your contribution @ahmad-nader! We will review the pull request and get back to you soon. |
There was a problem hiding this comment.
Pull request overview
This PR adds two end-to-end sample scripts to azure-ai-projects demonstrating endpoint-based evaluators — one using API Key authentication and one using Entra ID (managed identity). Each sample walks through creating a workspace connection, registering an evaluator, creating an evaluation, running it with inline data, and polling for results. The samples are intended to live alongside the existing evaluation samples under samples/evaluations/.
The main blocking concern is that both samples depend on a model (EndpointBasedEvaluatorDefinition) that does not exist in the package on the PR's base branch (main), so neither sample can import or run as-is. There is also a Black-formatting issue in the API Key sample, and the new samples are not added to the evaluations README.md index (which the repo convention maintains, though that file is outside this PR's diff).
Changes:
- Adds
sample_endpoint_evaluator_with_entra_id.py(managed-identity auth flow via aCustomKeys/AADconnection). - Adds
sample_endpoint_evaluator_with_api_key.py(API key auth flow via anApiKeyconnection, with anEvaluatorMetricdefinition). - Both rely on
EndpointBasedEvaluatorDefinition, which is not present in the SDK.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| sdk/ai/azure-ai-projects/samples/evaluations/sample_endpoint_evaluator_with_entra_id.py | New Entra ID auth endpoint-evaluator sample; imports nonexistent EndpointBasedEvaluatorDefinition. |
| sdk/ai/azure-ai-projects/samples/evaluations/sample_endpoint_evaluator_with_api_key.py | New API Key auth endpoint-evaluator sample; imports nonexistent EndpointBasedEvaluatorDefinition and has a mis-indented metrics block that fails Black. |
| from azure.identity import DefaultAzureCredential | ||
| from azure.ai.projects import AIProjectClient | ||
| from azure.ai.projects.models import ( | ||
| EndpointBasedEvaluatorDefinition, |
| from azure.identity import DefaultAzureCredential | ||
| from azure.ai.projects import AIProjectClient | ||
| from azure.ai.projects.models import ( | ||
| EndpointBasedEvaluatorDefinition, |
| metrics={ | ||
| "score": EvaluatorMetric( | ||
| type="ordinal", | ||
| min_value=1, | ||
| max_value=5, | ||
| ) | ||
| }, |
Add two E2E samples demonstrating endpoint-based evaluators:
Both samples show the full workflow: create connection, register evaluator, create evaluation, run with inline data, and poll for results.
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines