Schema validation is deferred until an explicit schema/columns call as opposed to at definition time

[kszlim]

Schema validation is deferred until an explicit schema/columns call as opposed to at definition time

[cmdlineluser]

May be relevant: #16624 (comment)

Polars will resolve queries upon collection, not upon DSL creation. This is important as we want to be able to send the query to another machine. At that point the schema will be resolved not when you write the query.

[kszlim]

Ah should I close this?

[alexander-beedie]

The call to "schema" will trigger the compute that establishes the execution plan, and that's when it finds out the column isn't present (a call to .explain() or .collect() would also raise an error for the same reason). Essentially lazy mode has become even lazier, so that it can better scale across more use-cases (cloud, for instance) 😅

[kszlim]

I wonder what the implications are for libraries like pandera/patito/etc. They'll have to be built to minimize the number of schema/column calls they make.

I wonder if it's possible to make columns relatively cheap vs schema. I believe columns is used fairly often in a programmatic manner (though perhaps selectors are the ideal/idiomatic way), but I'd imagine that it's a pattern that will be hard to steer users away from.

@ritchie46 @alexander-beedie what are your thoughts on that? I definitely think lots of ETL pipelines involve people doing something along the lines of and in potentially many stages of the pipeline:

exprs = []
for col in ldf.columns:
    exprs.append(pl.col(col)...)