feat(guardrails): add LLMAsJudgeValidator for the @guardrail decorator [AL-470]#1802
Conversation
github-actions
Bot
added
test:uipath-langchain
There was a problem hiding this comment.
Pull request overview
Adds a new decorator-path built-in guardrail validator (LLMAsJudgeValidator) so @guardrail can use the llm_as_judge validator type (bringing the decorator path to feature parity with low-code agents and the middleware path).
Changes:
- Introduces
LLMAsJudgeValidator, which builds aBuiltInValidatorGuardrailwithvalidator_type="llm_as_judge"and the expected parameter IDs/types. - Exports the validator through the guardrails decorator/validator public modules.
- Adds focused unit tests for guardrail construction and decorator behavior (block/log actions + mocked backend).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/uipath-platform/tests/services/test_llm_as_judge_validator.py | New unit tests validating guardrail construction, parameters, defaults, stage support, and input validation. |
| packages/uipath-platform/tests/services/test_guardrails_decorators.py | Adds decorator-level behavior tests covering block/log actions and an end-to-end mocked API path. |
| packages/uipath-platform/src/uipath/platform/guardrails/decorators/validators/llm_as_judge.py | Implements the new LLMAsJudgeValidator and its BuiltInValidatorGuardrail construction. |
| packages/uipath-platform/src/uipath/platform/guardrails/decorators/validators/init.py | Re-exports LLMAsJudgeValidator from the validators package. |
| packages/uipath-platform/src/uipath/platform/guardrails/decorators/init.py | Re-exports LLMAsJudgeValidator from the decorators public API. |
| packages/uipath-platform/src/uipath/platform/guardrails/init.py | Re-exports LLMAsJudgeValidator from the top-level guardrails package API. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Threshold scale matches the backend OOTB catalog (0-6, step 2, default 2): | ||
| # HIGHER = more lenient (only flag clear violations), LOWER = stricter. |
There was a problem hiding this comment.
Fixed in 4f27bee — clarified the wording: threshold accepts any float in [0,6] (the catalog 'step 2' is a UI increment only, not enforced), and supported_stages is the default empty list meaning all stages are allowed. Documentation-only, no behavior change.
| any scope/stage the ``@guardrail`` decorator wraps (scope is implicit in the | ||
| decorated target; ``supported_stages`` is unset so both PRE and POST are allowed). |
There was a problem hiding this comment.
Fixed in 4f27bee — reworded to: supported_stages is left as the default empty list, which means all stages are allowed (both PRE and POST). No behavior change.
apetraru-uipath
force-pushed
the
feat/AL-470-llm-as-judge-decorator
branch
from
4f27bee to
8bcd9f9
Compare
|
❌ The last analysis has failed. |
apetraru-uipath
force-pushed
the
feat/AL-470-llm-as-judge-decorator
branch
2 times, most recently
from
fe9f1fb to
abd92af
Compare
| value=self.threshold, | ||
| ), | ||
| ] | ||
| if self.positive_examples: |
There was a problem hiding this comment.
Should we add some validations to match the constraints we have for low coded agents?
Like max 2 positive and max 2 negative examples, and each example to have a max of 1000 chars?
Also the guardrail text to be at most 4000 chars?
There was a problem hiding this comment.
Added those constraints
…r [AL-470] Adds an llm_as_judge built-in validator to the coded-agent decorator path, closing the last gap (low-code and the guardrail middleware already support it). - LLMAsJudgeValidator(BuiltInGuardrailValidator) builds a BuiltInValidatorGuardrail with validator_type="llm_as_judge" and parameters guardrailText (text), model (enum), threshold (number, 0-6, default 2), plus optional positiveExamples/negativeExamples (text-list). No scope/action on the validator (those are @guardrail args); selector stays None (scope comes from the wrapped target); all stages supported. - Exported through validators/__init__, decorators/__init__, and platform/guardrails/__init__. - Tests: construction/params/validation (test_llm_as_judge_validator.py) and decorator-level block/log behavior including a real-validator, mocked-backend case (test_guardrails_decorators.py). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
apetraru-uipath
force-pushed
the
feat/AL-470-llm-as-judge-decorator
branch
from
f4edfee to
713fe9c
Compare
|
🚨 Heads up:
|
What changed?
Adds
LLMAsJudgeValidator— thellm_as_judgebuilt-in validator for the@guardraildecorator path. This closes the last coded-agent gap: low-code agents and the guardrail
middleware already support
llm_as_judge; the decorator path did not.LLMAsJudgeValidator(BuiltInGuardrailValidator)builds aBuiltInValidatorGuardrailwithvalidator_type="llm_as_judge"and parametersguardrailText(text),model(enum),threshold(number, 0–6, default 2), plus optionalpositiveExamples/negativeExamples(text-list). The wire ids match the low-code catalog exactly.
@guardrailarguments);selectorstaysNone(scope comes from the wrapped target); all stages supported.
validators/__init__.py,decorators/__init__.py, andplatform/guardrails/__init__.py.How has this been tested?
tests/services/test_llm_as_judge_validator.py:validator_type, parameter ids/values,examples-only-when-present,
selector is None, all-stages, and constructor validation.tests/services/test_guardrails_decorators.py: decorator-level behavior —BlockActionraises
GuardrailBlockException,LogActioncontinues, and a real-validator, mocked-backendcase (
patch("uipath.platform.UiPath", ...)).ruff+mypyclean; full guardrails decorator + azure-validator suites pass.joke-agent-decoratorsample (separateuipath-langchainPR,AL-470): the
@guardrailjudge fires at POST and calls the real validate endpoint.Are there any breaking changes?
🤖 Generated with Claude Code