Skip to content

Dataframe transforms and Parquet save operators#2031

Merged
rolandwalker merged 1 commit into
mainfrom
RW/dataframe-post-processing
Jul 21, 2026
Merged

Dataframe transforms and Parquet save operators#2031
rolandwalker merged 1 commit into
mainfrom
RW/dataframe-post-processing

Conversation

@rolandwalker

Copy link
Copy Markdown
Contributor

Description

Add a trailing operator .| which allows applying arbitrary dataframe transforms with Polars. This is similar to the $| operator, except that normal tabular output is maintained, so long as the Polars expression also returns a DataFrame or Series.

Add a trailing operator .> which saves output to a Parquet file. This is similar to the $> operator for shell redirection.

Together these features are intended to make mycli more friendly to Data Science practitioners, lowering the boundaries between the CLI/REPL and notebook modes of working.

In many cases, dataframe transformations may be achievable in plain SQL, but for many practitioners, dataframes are the preferred approach. In other cases, there are computations which are achievable in dataframes but not in SQL. Medians are a simple example.

There are performance implications of using the dataframe transform: the entire SQL result must be fetched and loaded into memory.

Counter to the familiarity argument above, the .| operator supplies a Polars dataframe rather than a Pandas dataframe.

While Pandas is the industry standard, Polars is growing, and should be familiar enough to be a workable alternative. Polars is sometimes a bit faster, and usually far more memory-efficient. But the reason it is chosen here over Pandas is that the dependency chain for mycli is simpler.

Polars is also configured as an optional dependency in pyproject.toml.

There is more extensive documentation of the functionality in the PR.

Simple example showing a transform which returns a DataFrame, which is then rendered as usual:

Screenshot 2026-07-21 at 6 00 22 AM

Checklist

  • I added this contribution to the changelog.md file.
  • I added my name to the AUTHORS file (or it's already there).
  • To lint and format the code, I ran
    uv run ruff check && uv run ruff format && uv run mypy --install-types .

@rolandwalker rolandwalker self-assigned this Jul 21, 2026
@rolandwalker
rolandwalker force-pushed the RW/dataframe-post-processing branch from 2592d9e to 2376033 Compare July 21, 2026 10:43
@rolandwalker
rolandwalker requested review from amjith and scottnemes and removed request for amjith and scottnemes July 21, 2026 10:43
@rolandwalker
rolandwalker force-pushed the RW/dataframe-post-processing branch from 2376033 to 1ac8553 Compare July 21, 2026 10:47
Comment thread mycli/packages/polars_transform.py Outdated
try:
import polars as pl
except ImportError as exc:
raise PolarsTransformError("Polars transforms require Polars to be installed.'") from exc

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra single quote at the end here and same thing just below

@scottnemes scottnemes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Left one nit comment for string fixes

Add a trailing operator .| which allows applying arbitrary dataframe
transforms with Polars.  This is similar to the $| operator, except that
normal tabular output is maintained, so long as the Polars expression
also returns a DataFrame or Series.

Add a trailing operator .> which saves output to a Parquet file.  This
is similar to the $> operator for shell redirection.

Together these features are intended to make mycli more friendly to
Data Science practitioners, lowering the boundaries between the CLI/REPL
and notebook modes of working.

In many cases, dataframe transformations may be achievable in plain SQL,
but for many practitioners, dataframes are the preferred approach.  In
other cases, there are computations which are achievable in dataframes
but not in SQL.  Medians are a simple example.

There are performance implications of using the dataframe transform:
the entire SQL result must be fetched and loaded into memory.

Counter to the familiarity argument above, the .| operator supplies a
Polars dataframe rather than a Pandas dataframe.

While Pandas is the industry standard, Polars is growing, and should be
familiar enough to be a workable alternative.  Polars is sometimes a bit
faster, and usually far more memory-efficient.  But the reason it is
chosen here over Pandas is that the dependency chain for mycli is
simpler.

Polars is also configured as an optional dependency in pyproject.toml.
@rolandwalker
rolandwalker force-pushed the RW/dataframe-post-processing branch from 1ac8553 to 9b5e6ff Compare July 21, 2026 18:07
@rolandwalker
rolandwalker merged commit 9035c84 into main Jul 21, 2026
11 checks passed
@rolandwalker
rolandwalker deleted the RW/dataframe-post-processing branch July 21, 2026 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants