feat(eval): add agent memory settings to eval sets and CLI override#1801
Open
mjnovice wants to merge 3 commits into
Open
feat(eval): add agent memory settings to eval sets and CLI override#1801mjnovice wants to merge 3 commits into
mjnovice wants to merge 3 commits into
Conversation
Port of the old-world eval memory support (UiPath/Agents #3560/#3603) to the eval-set schema and CLI: - EvaluationSetAgentMemorySettings model plus agentMemoryEnabled / agentMemorySettings fields on EvaluationSet and LegacyEvaluationSet, carried through the legacy eval-set migration. - New --agent-memory-settings-id option on `uipath eval`; the resolved override is passed to the runtime factory via the agent_memory_settings kwarg. Eval runs disable agent memory unless the eval set explicitly enables it, matching the Agents backend behavior for eval runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
test:uipath-langchain
Contributor
There was a problem hiding this comment.
Pull request overview
Adds evaluation-set support for agent memory configuration and wires it through the uipath eval CLI so eval runs can explicitly disable/enable agent memory and optionally select a specific memory settings profile.
Changes:
- Extend eval-set schema models with
agentMemoryEnabledandagentMemorySettings(plus a dedicatedEvaluationSetAgentMemorySettingsmodel). - Preserve these fields during legacy eval-set migration.
- Add
--agent-memory-settings-idtouipath evaland pass a resolvedagent_memory_settingsoverride intoruntime_factory.new_runtime(...), with new unit tests covering parsing/migration/override resolution.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| packages/uipath/src/uipath/eval/models/evaluation_set.py | Adds agent-memory settings models/fields to EvaluationSet + LegacyEvaluationSet. |
| packages/uipath/src/uipath/eval/helpers.py | Carries agent-memory fields through legacy→v1.0 eval-set migration. |
| packages/uipath/src/uipath/_cli/cli_eval.py | Adds CLI option and resolves/passes agent-memory override into runtime creation. |
| packages/uipath/tests/cli/eval/test_agent_memory_settings.py | New tests for schema defaults/serialization, migration passthrough, and override resolver behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+159
to
+168
| logger.info( | ||
| f"Applying agent memory settings override: searchMode={target.search_mode}, " | ||
| f"resultCount={target.result_count}, threshold={target.threshold}" | ||
| ) | ||
| return { | ||
| "enabled": True, | ||
| "resultCount": target.result_count, | ||
| "searchMode": target.search_mode, | ||
| "threshold": target.threshold, | ||
| } |
The eval-set editor persists a "default" entry (all same-as-agent) alongside user-defined settings, so resolve it by id like any other before falling back to the first configured entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🚨 Heads up:
|
|
JosephMar
approved these changes
Jul 7, 2026
smflorentino
reviewed
Jul 7, 2026
| settings_override = _resolve_model_settings_override( | ||
| model_settings_id, eval_context.evaluation_set | ||
| ) | ||
| agent_memory_settings_override = ( |
Collaborator
There was a problem hiding this comment.
same comment as in other PRs
smflorentino
approved these changes
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports eval-run agent memory support from the old world (UiPath/Agents #3560 eval-set schema + #3603 backend override) to the eval-set models and
uipath evalCLI.Schema (
eval/models/evaluation_set.py):EvaluationSetAgentMemorySettingsmodel —id,resultCount,searchMode,thresholdas strings, with"same-as-agent"defaults preserving the agent's own memory configuration.agentMemoryEnabled(defaultfalse) andagentMemorySettings(default[]) on bothEvaluationSetandLegacyEvaluationSet; the legacy→v1.0 migration ineval/helpers.pycarries them through. Defaults keep existing eval sets fully backwards compatible.CLI (
_cli/cli_eval.py):--agent-memory-settings-idoption (default"default"picks the first configured setting)._resolve_agent_memory_settings_overrideresolves the eval set's memory config into an override dict passed toruntime_factory.new_runtime(..., agent_memory_settings=...), alongside the existingsettings(model override) kwarg.Semantics match the old-world backend: eval runs never consume agent memory unless the eval set explicitly enables it —
agentMemoryEnabled: false(the default) sends{"enabled": false}, which the agents runtime factory uses to strip memorySpace features for the run. Factories that don't understand the kwarg ignore it (same contract as the existing model-settings override), so coded agents are unaffected.Companion PRs: UiPath/uipath-agents-python#605 (applies the override; bumps its uipath dependency to this release) and UiPath/Agents#5758 (backend forwards the selected settings id to serverless eval jobs). Merge sequence: this PR first (releases 2.13.7) → #605 → Agents#5758.
Testing
packages/uipath/tests/cli/eval/test_agent_memory_settings.py: schema parsing, backwards-compatible defaults, camelCase serialization, legacy migration passthrough, and all resolver branches (disabled, default id, explicit id, unknown id fallback, enabled-without-settings).tests/cli/eval+tests/evaluatorssuites: 931 passed.ruff format,ruff check,mypyclean on changed files.🤖 Generated with Claude Code