feat(guardrails): add llm_as_judge middleware and @guardrail re-export [AL-470]#976
Open
apetraru-uipath wants to merge 1 commit into
Open
feat(guardrails): add llm_as_judge middleware and @guardrail re-export [AL-470]#976apetraru-uipath wants to merge 1 commit into
apetraru-uipath wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds LLM-as-judge guardrail support to the uipath_langchain guardrails integration via a new middleware and a decorator-path re-export, with accompanying unit tests and updated samples.
Changes:
- Introduces
UiPathLLMAsJudgeMiddlewareto build aBuiltInValidatorGuardrail(validator_type="llm_as_judge")and wire PRE/POST hooks across AGENT/LLM/TOOL scopes. - Re-exports
LLMAsJudgeValidatorviauipath_langchain.guardrailsand adds a guarded test for the re-export. - Updates both “joke-agent” samples (middleware + decorator) to demonstrate usage.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/guardrails/test_llm_as_judge_validator.py | Adds a guarded test for the LLMAsJudgeValidator re-export and expected built-in guardrail shape. |
| tests/guardrails/middlewares/test_llm_as_judge.py | Adds unit tests for middleware construction, hook wiring, actions, and async tool path. |
| src/uipath_langchain/guardrails/middlewares/llm_as_judge.py | Implements the new LLM-as-judge middleware and built-in guardrail construction. |
| src/uipath_langchain/guardrails/middlewares/init.py | Exposes UiPathLLMAsJudgeMiddleware from the middleware package. |
| src/uipath_langchain/guardrails/init.py | Re-exports LLMAsJudgeValidator and exposes the new middleware at the top-level guardrails module. |
| samples/joke-agent/graph.py | Demonstrates middleware-based LLM-as-judge guardrail at AGENT/POST. |
| samples/joke-agent-decorator/graph.py | Demonstrates decorator-based LLM-as-judge guardrail at POST. |
| "GuardrailValidatorBase", | ||
| "HarmfulContentValidator", | ||
| "IntellectualPropertyValidator", | ||
| "LLMAsJudgeValidator", |
6918d8f to
db0334c
Compare
valentinabojan
approved these changes
Jul 8, 2026
25f6bb1 to
56742bd
Compare
…t [AL-470] Adds llm_as_judge support to both coded-agent guardrail paths. - Middleware: UiPathLLMAsJudgeMiddleware (AGENT/LLM/TOOL scopes) builds a BuiltInValidatorGuardrail(validator_type="llm_as_judge") and wires the PRE/POST hooks; demonstrated in the joke-agent (middleware) sample. - Decorator: re-export LLMAsJudgeValidator from uipath.platform.guardrails.decorators; demonstrated in the joke-agent-decorator sample. - Tests: middleware unit tests (construction, hook wiring per scope/stage, block/log/ filter actions, async tool path) and a guarded re-export test that skips until the installed uipath-platform provides the validator. Note: the decorator re-export (and its test) depend on the uipath-python PR (AL-470) publishing LLMAsJudgeValidator; the re-export import resolves once uipath-platform ships it. The middleware works with the currently published uipath-platform. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
b2cae8e to
3c76797
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What changed?
Adds
llm_as_judgesupport to both coded-agent guardrail paths in uipath-langchain.UiPathLLMAsJudgeMiddleware(AGENT/LLM/TOOL scopes) builds aBuiltInValidatorGuardrail(validator_type="llm_as_judge")and wires the PRE/POST hooks;demonstrated in the
joke-agent(middleware) sample. Works with the currently publisheduipath-platform.LLMAsJudgeValidatorfromuipath.platform.guardrails.decorators(added in the paired uipath-python PR, AL-470); demonstrated in the
joke-agent-decoratorsample.
How has this been tested?
async tool path.
uipath-platformshipsLLMAsJudgeValidator).ruff+mypyclean; full guardrails suite + circular-import tests pass locally.forced an off-topic rule to confirm the block/log verdict).
The decorator re-export in
guardrails/__init__.pyis a hard import ofLLMAsJudgeValidator,which is not in a published
uipath-platformyet — it ships in the paired uipath-python PR(AL-470). CI here will fail on that import until uipath-platform is released; the guarded
re-export test skips in the meantime. Merge after the uipath-python PR releases.
Are there any breaking changes?
🤖 Generated with Claude Code