I’m a sysadmin who is slowly learning data work, and most of the machines I touch are small. So the Polars 2.1 out-of-core group-by, announced on 27 October 2026, caught my attention: group-bys on high-cardinality data can now write part of their state to disk instead of crashing when memory runs out.
I haven’t tested it yet. This post covers what the release says, what I’m unsure about, and the experiment I’m going to run. I’ll update it with my numbers.
What the release says
From the Polars 2.1 announcement:
- The streaming group-by can run out of core, with shuffles. Previously, a group-by with a very large number of distinct keys could run out of memory because it had nowhere to spill.
- The post’s own example finished a group-by with 30 million groups under a 200 MB memory budget, by writing part of the state to disk and reading it back.
- The release also adds recursive SQL (
WITH RECURSIVE), Iceberg catalogs attached throughSQLContext, and reading and writing of encrypted Parquet files. - The post mentions close to 90 performance pull requests, but it gives no benchmarks beyond the spill example.
Why the Polars 2.1 out-of-core group-by matters on small servers
A group-by has to keep a running result for every distinct key it sees. With a few hundred keys that is trivial. With tens of millions of distinct keys, such as user IDs, device IDs or session IDs, the intermediate state alone can be larger than the memory on a small server. According to the release post, a high-cardinality group-by like that could previously run out of memory because it had nowhere to spill.
That looks like the situation the Polars 2.1 out-of-core group-by is aimed at. Spilling to disk trades speed for the ability to finish at all, and when you run small virtual machines, finishing is often the first problem to solve. Whether the trade is a good one is what I want to measure.
What I’m cautious about
Three things from the same post are worth reading before upgrading anything:
- Eager mixed string and numeric arithmetic now raises an error. The post’s example is
pl.Series([1, 2]) + "4", which used to return strings. - Python 3.10 support is dropped.
- Encryption,
register_catalog,delete_icebergand masked numpy conversion are marked unstable, so their APIs may change.
The spill example also tells me that it works, not how fast it works. Spilling to disk usually costs time, and I don’t know yet how much. That is the thing I most want to measure.
My test plan
The goal is to see whether the 200 MB claim holds up on data I generate myself, and what it costs in runtime.
- Generate a synthetic table with tens of millions of rows and a very high number of distinct keys, saved as Parquet.
- Run the same group-by (a count and a sum per key) in Polars 2.1 using the streaming engine, and in DuckDB for comparison.
- Run each one under a hard memory cap, for example a Docker
--memorylimit, and repeat with the cap raised in steps. - Record whether it finished, peak memory, and wall-clock time.
- Repeat once on a real CSV of my own, to see if synthetic data is flattering the result.
I’ll check the 2.1 changelog for the exact way to switch on the streaming and out-of-core path before I write the script, rather than guess at the syntax.
How I’ll judge the result
I’ll count the Polars 2.1 out-of-core group-by as a success if it finishes under the 200 MB cap with correct output. Correctness comes first: I’ll compare the group counts and sums against a run with plenty of memory, to confirm that spilling doesn’t change the answer. Speed comes second. If it finishes but takes ten times longer than DuckDB, that is still useful to know. It is just a different story.
I’ll also note the memory cap at which each tool first fails. The point where something breaks says more than a single pass or fail.
Results
I’ll fill this table in once the tests have run.
| Tool | Memory cap | Finished? | Peak memory | Time |
|---|---|---|---|---|
| Polars 2.1 (streaming) | 200 MB | to fill in | to fill in | to fill in |
| Polars 2.1 (streaming) | 1 GB | to fill in | to fill in | to fill in |
| DuckDB | 200 MB | to fill in | to fill in | to fill in |
| DuckDB | 1 GB | to fill in | to fill in | to fill in |
What I hope to learn
If the Polars 2.1 out-of-core group-by holds up, it changes what I can reasonably do on a small server without reaching for a bigger one. If it doesn’t, I’d still like to know where it breaks and why. Either way I’d rather have my own numbers than rely on a vendor’s example.
If you’ve already tried Polars 2.1 on your own data, I’d like to hear what you found. Contact me here: Contact Me