RubricBasedEvaluator _normalize_text too basic — fails on judge model markdown output

[tottenjordan]

Description

_normalize_text() in rubric_based_evaluator.py only does text.lower().strip(). When LLM judge models return rubric verdicts with markdown formatting (bullets, smart quotes, extra whitespace, non-ASCII characters), the exact-match rubric lookup fails silently, producing incorrect scores.

Reproduction

Judge model returns: "• The response correctly identifies the tool"
Expected rubric: "the response correctly identifies the tool"
_normalize_text() produces: "• the response correctly identifies the tool"
Result: No match → score defaults to 0 or lowest rubric

Common Patterns That Fail

• Leading bullets: •, *, -
• Smart quotes: "...", '...'
• Non-ASCII: accented characters, em-dashes
• Multi-space: "the response" vs "the response"
• Trailing whitespace/newlines

Suggested Fix

Enhanced normalization:

def _normalize_text(text: str) -> str:
    if not isinstance(text, str):
        return ""
    text = re.sub(r'^[\s*•\-]+', '', text)   # Strip leading bullets
    text = re.sub(r'[\s*•\-]+$', '', text)   # Strip trailing
    text = re.sub(r'\s+', ' ', text)          # Collapse whitespace
    text = text.encode('ascii', 'ignore').decode()  # Remove non-ASCII
    return text.lower().strip()

Additionally, a substring fallback when exact match fails would prevent silent scoring failures:

# If exact match fails, try substring match
for rubric_text, score in rubric_map.items():
    if normalized_response in rubric_text or rubric_text in normalized_response:
        return score

Impact

Without this fix, GEPA optimization produces unreliable rubric-based scores (rubric_based_final_response_quality_v1, rubric_based_tool_use_quality_v1), leading to suboptimal prompt evolution.

Environment

• google-adk 2.2.0
• Judge models: gemini-2.5-pro, gemini-3.5-flash

[surajksharma07]

@tottenjordan Thanks for the clear writeup — reproduced this on 2.2.0 with a quick offline test: every pattern you listed (bullets, bold, double spaces, smart quotes) misses the exact-match lookup and the rubric just gets dropped from the average with only a warning log.

Your normalization direction works, though I'd suggest two tweaks: swap the ascii-ignore step for NFKC normalization plus a smart-quote/dash mapping so non-English rubrics don't get mangled, and add a uniqueness guard to the substring fallback so two similar rubrics can't cross-match.

Since _normalize_text is applied to both sides of the comparison, strengthening it is fairly low-risk.

Could you try that variant as a workaround and verify it fixes the scores on your GEPA runs?

If it holds up after thorough testing, it would make a solid PR — and a raw failing judge transcript from your runs would be a great regression fixture to include!

[tottenjordan]

Thanks @surajksharma07 — tried both tweaks, they work well together.

1. NFKC + smart-char mapping instead of ascii-ignore:

Replaced .encode('ascii', 'ignore').decode('ascii') with unicodedata.normalize('NFKC', text) plus an explicit translation table for smart quotes/dashes:

import unicodedata, re

_SMART_CHARS = {
    0x2018: "'", 0x2019: "'",  # smart single quotes
    0x201C: '"', 0x201D: '"',  # smart double quotes
    0x2013: "-", 0x2014: "-",  # en/em dashes
    0x2026: "...",             # ellipsis
}

def _fuzzy_normalize(text):
    text = unicodedata.normalize('NFKC', text)
    text = text.translate(_SMART_CHARS)
    text = re.sub(r'^[\s*•\-"\']+', '', text)
    text = re.sub(r'[\s*•\-"\']+$', '', text)
    text = re.sub(r'\s+', ' ', text)
    return text.lower().strip()

This preserves accented characters (e.g. réponse stays réponse instead of rponse) while still normalizing all the judge garbling patterns we've seen:

Input	Normalized	Match
- The response correctly uses tools	the response correctly uses tools	✅
* **The response correctly uses tools**	the response correctly uses tools	✅
“The response correctly uses tools”	the response correctly uses tools	✅
— The response correctly uses tools	the response correctly uses tools	✅
The response correctly uses tools	the response correctly uses tools	✅

2. Uniqueness guard on substring fallback:

Changed the substring fallback from first-match-wins to only-if-unique:

# Before (fragile — first match wins, could cross-match similar rubrics)
for config_text, r in normalized_map.items():
    if config_text in norm_text or norm_text in config_text:
        rubric = r
        break

# After (only accept if exactly one rubric matches)
candidates = [r for ct, r in normalized_map.items()
              if ct in norm_text or norm_text in ct]
if len(candidates) == 1:
    rubric = candidates[0]

Verified: when two rubrics share a common substring (e.g., "uses tools correctly" vs "uses tools efficiently"), a judge response of just "uses tools" correctly gets rejected instead of arbitrarily matching the first one.

Test results: 107 tests pass on our production codebase with both changes applied. We'll run this through our next GEPA optimization pipeline to validate on real judge transcripts. Happy to include a failing transcript as a regression fixture once we capture one.

[tottenjordan]

Follow-up: End-to-end pipeline validation complete.

Ran a full GEPA optimization pipeline (deploy → eval_before → optimize → redeploy → eval_after → analysis) with the NFKC normalization + uniqueness guard applied:

• Pipeline: Vertex AI Pipeline gepa-run-8fb68a8f52-20260611-115752
• Model: claude-sonnet-4-6 on 5 eval cases, gemini-2.5-pro as judge
• GEPA config: 15 max metric calls, 4 rubric-based criteria (hallucinations_v1, safety_v1, rubric_based_final_response_quality_v1, rubric_based_tool_use_quality_v1)
• Result: Zero "Rubric not matched" warnings across all generations. All rubric scores populated correctly.

Unit tests (29 total, all pass):

We added TestFuzzyNormalize (12 parametrized cases) and TestSubstringUniquenessGuard (2 cases) covering:

• Markdown bullets (- text, * **text**, • text)
• Smart quotes ("text", mid-word ')
• Dashes (— text, – text)
• Whitespace collapse (double spaces, leading whitespace)
• NFKC normalization (ellipsis … → ..., accented chars preserved)
• Non-string/empty input edge cases
• Ambiguous substring rejection (2+ candidates → no match)

We didn't capture a raw failing judge transcript from this run since the normalization resolved everything cleanly — no failures to capture. We can intentionally disable the patch and re-run to generate one for the regression fixture if that would be useful for the PR.

[surajksharma07]

@tottenjordan went through #6080 and it's exactly the right shape: NFKC plus the smart-char map keeps non-English rubrics intact, the uniqueness guard behaves correctly in the ambiguous-substring test, and a zero-warning GEPA run on 4 rubric criteria is solid validation.

One edge case to close before review: if the judge emits a blank Property: line, _normalize_text returns "", which is a substring of every rubric — so with a single configured rubric the fallback matches it as the lone candidate.
Guarding with if not rubric and normalized_rubric_text: (plus a small test) fixes that.

Minor: a logger.debug when the fallback rescues a match would keep judge drift visible in eval logs.

No need to re-run with the patch disabled — the parametrized cases in TestNormalizeText already pin the regression, so the synthetic transcript wouldn't add much!

Team is looking more into it.
Thanks!