Skip to content

fix(isthmus): elide redundant identity projections#1009

Open
nielspardon wants to merge 1 commit into
substrait-io:mainfrom
nielspardon:fix/issue-371-redundant-identity-project
Open

fix(isthmus): elide redundant identity projections#1009
nielspardon wants to merge 1 commit into
substrait-io:mainfrom
nielspardon:fix/issue-371-redundant-identity-project

Conversation

@nielspardon

Copy link
Copy Markdown
Member

Problem

Calcite wraps queries such as SELECT "a", "B" FROM foo GROUP BY a, b in a name-changing identity projection:

LogicalProject(a=[$0], B=[$1])
  LogicalAggregate(group=[{0, 1}])
    LogicalTableScan(table=[[FOO]])

The field references are $0, $1 in order — only the output names differ (bB). SubstraitRelVisitor.visit(Project) turned every such projection into a Substrait Project, but Substrait carries output names on Plan.Root, not on the Project, so the node is pure redundancy. assertFullRoundTrip failed because the reverse conversion (Substrait → Calcite → Substrait) elides the identity projection, so root1 kept the project while root3 did not:

root1: Root{ Project{remap=[2,3], expressions=[$0,$1], input=Aggregate}, names=[a, B] }
root3: Root{ Aggregate, names=[a, B] }   ← project gone → assertEquals(root1, root3) fails

Change

Skip emitting a Project for an identity projection — all expressions are input refs, in order, with matching types, as reported by Calcite's own RexUtil.isIdentity. Because the check lives in visit(Project), it applies to both conversion legs, so the two sides converge to the project-free form and round-trip cleanly. Output names are unaffected: convert(RelRoot, ...) derives them from validatedRowType, independent of the projection.

Scope is limited to identity projections. Non-identity all-field-ref projections (permutations / column pruning) still emit a Project as before; that optimization is noted as future work in the retained TODO.

Tests

  • New regression test NameRoundtripTest.roundTripNameChangingProjectionOverAggregate covering the exact query from the issue.
  • The assertFullRoundTripWithIdentityProjectionWorkaround helper (whose Javadoc noted it should be removed long-term) is deleted; DdlRoundtripTest and DmlRoundtripTest now use the stronger plain assertFullRoundTrip.
  • SubqueryPlanTest.existsNestedCorrelatedSubquery navigation updated: the inner UNIQUE (SELECT * ...) identity projection is now elided, matching how the outer EXISTS subquery was already represented. Downstream field-reference / steps-out assertions are unchanged, confirming the elision is semantically neutral.
  • Full :isthmus:test, :isthmus:check (PMD + javadoc), and spotlessCheck pass.

Closes #371

🤖 Generated with AI

Calcite wraps queries such as `SELECT "a", "B" FROM foo GROUP BY a, b`
in a name-changing identity projection (`LogicalProject(a=[$0], B=[$1])`)
whose field references are `$0, $1` in order — only the output names
differ. `SubstraitRelVisitor.visit(Project)` turned every such projection
into a Substrait `Project`, but Substrait carries output names on
`Plan.Root`, not on the `Project`, so the node is pure redundancy. The
full round trip failed because the reverse conversion elides the identity
projection, leaving the two sides structurally different.

Skip emitting a `Project` for an identity projection (all input refs, in
order, with matching types, as reported by `RexUtil.isIdentity`). The
check runs on both conversion legs, so both sides converge to the
project-free form. Output names are unaffected — `convert(RelRoot, ...)`
takes them from `validatedRowType`.

This lets the identity-projection test workaround be removed: the DDL/DML
round trips now pass the stronger plain `assertFullRoundTrip`.

Closes substrait-io#371

🤖 Generated with AI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[isthmus] investigate redundant name-changing LogicalProject

1 participant