Skip to content

feat(guardrails): add llm_as_judge middleware and @guardrail re-export [AL-470]#976

Open
apetraru-uipath wants to merge 1 commit into
mainfrom
feat/AL-470-llm-as-judge-coded-agents
Open

feat(guardrails): add llm_as_judge middleware and @guardrail re-export [AL-470]#976
apetraru-uipath wants to merge 1 commit into
mainfrom
feat/AL-470-llm-as-judge-coded-agents

Conversation

@apetraru-uipath

Copy link
Copy Markdown
Contributor

What changed?

Adds llm_as_judge support to both coded-agent guardrail paths in uipath-langchain.

  • Middleware: UiPathLLMAsJudgeMiddleware (AGENT/LLM/TOOL scopes) builds a
    BuiltInValidatorGuardrail(validator_type="llm_as_judge") and wires the PRE/POST hooks;
    demonstrated in the joke-agent (middleware) sample. Works with the currently published
    uipath-platform.
  • Decorator: re-export LLMAsJudgeValidator from uipath.platform.guardrails.decorators
    (added in the paired uipath-python PR, AL-470); demonstrated in the joke-agent-decorator
    sample.

How has this been tested?

  • Middleware unit tests: construction, hook wiring per scope/stage, block/log/filter actions,
    async tool path.
  • Guarded decorator re-export test (skips until the installed uipath-platform ships
    LLMAsJudgeValidator).
  • ruff + mypy clean; full guardrails suite + circular-import tests pass locally.
  • Both samples validated end-to-end against the real validate endpoint (judge fires at POST;
    forced an off-topic rule to confirm the block/log verdict).

⚠️ Cross-repo dependency (expected red CI until merged)

The decorator re-export in guardrails/__init__.py is a hard import of LLMAsJudgeValidator,
which is not in a published uipath-platform yet — it ships in the paired uipath-python PR
(AL-470)
. CI here will fail on that import until uipath-platform is released; the guarded
re-export test skips in the meantime. Merge after the uipath-python PR releases.

Are there any breaking changes?

  • None
  • Under Feature Flag
  • UiPath CLI Runtime changes

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings July 7, 2026 20:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds LLM-as-judge guardrail support to the uipath_langchain guardrails integration via a new middleware and a decorator-path re-export, with accompanying unit tests and updated samples.

Changes:

  • Introduces UiPathLLMAsJudgeMiddleware to build a BuiltInValidatorGuardrail(validator_type="llm_as_judge") and wire PRE/POST hooks across AGENT/LLM/TOOL scopes.
  • Re-exports LLMAsJudgeValidator via uipath_langchain.guardrails and adds a guarded test for the re-export.
  • Updates both “joke-agent” samples (middleware + decorator) to demonstrate usage.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/guardrails/test_llm_as_judge_validator.py Adds a guarded test for the LLMAsJudgeValidator re-export and expected built-in guardrail shape.
tests/guardrails/middlewares/test_llm_as_judge.py Adds unit tests for middleware construction, hook wiring, actions, and async tool path.
src/uipath_langchain/guardrails/middlewares/llm_as_judge.py Implements the new LLM-as-judge middleware and built-in guardrail construction.
src/uipath_langchain/guardrails/middlewares/init.py Exposes UiPathLLMAsJudgeMiddleware from the middleware package.
src/uipath_langchain/guardrails/init.py Re-exports LLMAsJudgeValidator and exposes the new middleware at the top-level guardrails module.
samples/joke-agent/graph.py Demonstrates middleware-based LLM-as-judge guardrail at AGENT/POST.
samples/joke-agent-decorator/graph.py Demonstrates decorator-based LLM-as-judge guardrail at POST.

Comment thread src/uipath_langchain/guardrails/__init__.py
"GuardrailValidatorBase",
"HarmfulContentValidator",
"IntellectualPropertyValidator",
"LLMAsJudgeValidator",
@apetraru-uipath apetraru-uipath force-pushed the feat/AL-470-llm-as-judge-coded-agents branch from 6918d8f to db0334c Compare July 8, 2026 07:58
@apetraru-uipath apetraru-uipath force-pushed the feat/AL-470-llm-as-judge-coded-agents branch 3 times, most recently from 25f6bb1 to 56742bd Compare July 8, 2026 10:25
…t [AL-470]

Adds llm_as_judge support to both coded-agent guardrail paths.

- Middleware: UiPathLLMAsJudgeMiddleware (AGENT/LLM/TOOL scopes) builds a
  BuiltInValidatorGuardrail(validator_type="llm_as_judge") and wires the PRE/POST
  hooks; demonstrated in the joke-agent (middleware) sample.
- Decorator: re-export LLMAsJudgeValidator from uipath.platform.guardrails.decorators;
  demonstrated in the joke-agent-decorator sample.
- Tests: middleware unit tests (construction, hook wiring per scope/stage, block/log/
  filter actions, async tool path) and a guarded re-export test that skips until the
  installed uipath-platform provides the validator.

Note: the decorator re-export (and its test) depend on the uipath-python PR (AL-470)
publishing LLMAsJudgeValidator; the re-export import resolves once uipath-platform
ships it. The middleware works with the currently published uipath-platform.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@apetraru-uipath apetraru-uipath force-pushed the feat/AL-470-llm-as-judge-coded-agents branch from b2cae8e to 3c76797 Compare July 8, 2026 12:39
@sonarqubecloud

sonarqubecloud Bot commented Jul 8, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants