Skip to content

feat(azure-ai-projects): add endpoint evaluator samples for API Key auth and entra id auth#47669

Open
ahmad-nader wants to merge 1 commit into
Azure:mainfrom
ahmad-nader:ahmadnader/custom-evaluators
Open

feat(azure-ai-projects): add endpoint evaluator samples for API Key auth and entra id auth#47669
ahmad-nader wants to merge 1 commit into
Azure:mainfrom
ahmad-nader:ahmadnader/custom-evaluators

Conversation

@ahmad-nader

@ahmad-nader ahmad-nader commented Jun 25, 2026

Copy link
Copy Markdown

Add two E2E samples demonstrating endpoint-based evaluators:

  • sample_endpoint_evaluator_with_api_key.py: API Key authentication flow
  • sample_endpoint_evaluator_with_entra_id.py: Entra ID (managed identity) authentication flow

Both samples show the full workflow: create connection, register evaluator, create evaluation, run with inline data, and poll for results.

Description

Please add an informative description that covers that changes made by the pull request and link all relevant issues.

If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

…nd Entra ID auth

Add two E2E samples demonstrating endpoint-based evaluators:
- sample_endpoint_evaluator_with_api_key.py: API Key authentication flow
- sample_endpoint_evaluator_with_entra_id.py: Entra ID (managed identity) authentication flow

Both samples show the full workflow: create connection, register evaluator,
create evaluation, run with inline data, and poll for results.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 25, 2026 13:19
@github-actions github-actions Bot added AI Projects Community Contribution Community members are working on the issue customer-reported Issues that are reported by GitHub users external to the Azure organization. labels Jun 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution @ahmad-nader! We will review the pull request and get back to you soon.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds two end-to-end sample scripts to azure-ai-projects demonstrating endpoint-based evaluators — one using API Key authentication and one using Entra ID (managed identity). Each sample walks through creating a workspace connection, registering an evaluator, creating an evaluation, running it with inline data, and polling for results. The samples are intended to live alongside the existing evaluation samples under samples/evaluations/.

The main blocking concern is that both samples depend on a model (EndpointBasedEvaluatorDefinition) that does not exist in the package on the PR's base branch (main), so neither sample can import or run as-is. There is also a Black-formatting issue in the API Key sample, and the new samples are not added to the evaluations README.md index (which the repo convention maintains, though that file is outside this PR's diff).

Changes:

  • Adds sample_endpoint_evaluator_with_entra_id.py (managed-identity auth flow via a CustomKeys/AAD connection).
  • Adds sample_endpoint_evaluator_with_api_key.py (API key auth flow via an ApiKey connection, with an EvaluatorMetric definition).
  • Both rely on EndpointBasedEvaluatorDefinition, which is not present in the SDK.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
sdk/ai/azure-ai-projects/samples/evaluations/sample_endpoint_evaluator_with_entra_id.py New Entra ID auth endpoint-evaluator sample; imports nonexistent EndpointBasedEvaluatorDefinition.
sdk/ai/azure-ai-projects/samples/evaluations/sample_endpoint_evaluator_with_api_key.py New API Key auth endpoint-evaluator sample; imports nonexistent EndpointBasedEvaluatorDefinition and has a mis-indented metrics block that fails Black.

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import (
EndpointBasedEvaluatorDefinition,
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import (
EndpointBasedEvaluatorDefinition,
Comment on lines +154 to +160
metrics={
"score": EvaluatorMetric(
type="ordinal",
min_value=1,
max_value=5,
)
},
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Projects Community Contribution Community members are working on the issue customer-reported Issues that are reported by GitHub users external to the Azure organization.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants