build(deps)!: update Substrait packages to 0.96.0#209
Conversation
Regenerate uv and Pixi locks, remove fields dropped from the spec, and migrate derivation expressions to the refactored ANTLR contexts. Keep native engine round trips opt-in while their bundled Substrait consumers lag the pinned spec and can crash the interpreter. BREAKING CHANGE: removes support for legacy timestamp, time, timestamp_tz, and Grouping.grouping_expressions fields removed from Substrait. Co-authored-by: Niels Pardon <par@zurich.ibm.com>
|
looks good, but I don't want to create additional work for Niels to rebase unless we are in a rush. #204 is probably not too far from mergable state anyway. @nielspardon wdyt? |
Yes, that also was my initial reaction. Rebases are less of a hassle for me these days since I can usually also just delegate that to an AI. I think it is awesome to have you @bvolpato as another committer helping out getting our code base updated to the latest spec though I feel like we should coordinate amongst ourselves. If we manage to coordinate then we could e.g. also tackle the other integrations like Datafusion and DuckDB which then would allow us to enable the integration tests again in substrait-python. There's also quite a long list of spec upgrade PRs waiting for review in e.g. the substrait-java repo where a committer could help us make progress by reviewing those and then tackling some of the other outstanding upgrade issues. |
Summary
substrait-protobuf,substrait-extensions, andsubstrait-antlrfrom 0.86.0 to 0.96.0.Changes
uv.lockandpixi.lock.SUBSTRAIT_ENGINE_TESTS=1; bundled consumers can crash natively when reading 0.96 plans.Repro
PR #181 leaves
pixi.lockstale and fails against removed protobuf fields and renamed ANTLR contexts. This PR regenerates both locks and adapts affected code paths.Testing
pixi install --lockeduv run --python 3.10 --frozen pytest(204 passed, 30 skipped)pixi run lintpixi run format --check./check_substrait_package_versions.shDid this cause problems?
External DuckDB, DataFusion, and PyArrow round-trip tests cannot safely gate this upgrade yet because their bundled Substrait consumers lag 0.96 and can segfault. They remain available through explicit opt-in.
BREAKING CHANGE: removes support for legacy timestamp, time, timestamp_tz, and
Grouping.grouping_expressionsfields removed from Substrait.