Comments (1)
On the latest compiled main
these queries (via the frame API and SQL API) do seem to have identical plans, and both run successfully on the streaming engine - can see the leading 'STREAMING' in the explain
output from both.
Confirmed that we see 'STREAMING' with both the join constraint a INNER JOIN b ON a.col = b.col
and the slightly more ergonomic a INNER JOIN b USING(col)
:
a = pl.LazyFrame({"col": [1, 2, 3, 4, 5]})
b = pl.LazyFrame({"col": [1, 2, 3, 4, 5]})
joined_df = a.join(b, on="col").select("col")
print("FRAME API:\n", joined_df.explain(streaming=True))
# FRAME API:
# STREAMING:
# SELECT [col("col")] FROM
# INNER JOIN:
# LEFT PLAN ON: [col("col")]
# DF ["col"]; PROJECT 1/1 COLUMNS; SELECTION: None
# RIGHT PLAN ON: [col("col")]
# DF ["col"]; PROJECT 1/1 COLUMNS; SELECTION: None
# END INNER JOIN
joined_sql = pl.sql("SELECT col FROM a INNER JOIN b USING(col)")
print("\nSQL API:\n", joined_sql.explain(streaming=True))
# SQL API:
# STREAMING:
# SELECT [col("col")] FROM
# INNER JOIN:
# LEFT PLAN ON: [col("col")]
# DF ["col"]; PROJECT 1/1 COLUMNS; SELECTION: None
# RIGHT PLAN ON: [col("col")]
# DF ["col"]; PROJECT 1/1 COLUMNS; SELECTION: None
# END INNER JOIN
Resulting in...
shape: (5, 1)
┌─────┐
│ col │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 2 │
│ 3 │
│ 4 │
│ 5 │
└─────┘
The SQL engine has had a lot of updates over the last few weeks, and there have been some streaming plan fixes; I think we can close this out as having been fixed on main
🤔
from polars.
Related Issues (20)
- `search_sorted` silently allows comparisons between different types HOT 2
- Incorrect dtype when doing .over(my_col, mapping_strategy="join") on LazyFrames.
- There are still methods with str_as_lit=True thereby treating strings as literal instead of column, is this intentional? HOT 2
- Tweak data frame HTML repr HOT 10
- Do not enclose str items in quotes in HTML repr HOT 21
- Unclear whether one should use `.describe(_optimized)_plan()` or `.explain()`
- Unable to read CSV when setting schema with a fixed tz offset
- Read Parquet Fails on ADLS for File in Subdirectory of Different Container HOT 3
- Left join large memory usage regression HOT 1
- `df.with_columns(pl.col('struct_col').shift(-1).over('group_col')` raises an error HOT 1
- Add null_values to read_json()
- Add example of `df.join(df2, how='cross')`
- [python] Support `db_kwargs` and `conn_kwargs` for ADBC connection HOT 2
- LazyFrame `collect` method, does not work when using GPU HOT 3
- str.to_titlecase() add option for capital after hyphen
- Trying to package Polars for NixOS results in an error E0599 HOT 3
- Nullability of `Struct` series HOT 3
- Unimplemented join on list[i64] succeeds with empty dataframe in lazy mode HOT 2
- More informative error messages for unsupported options in streaming mode HOT 1
- Getting a "ComputeError: cannot evaluate two Series of different lengths" with straight forward Lazy expressions. HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from polars.