Option::unwrap() panic when loading Parquet files with non-matching schemas as a list of files

[alf239]

Option::unwrap() panic when loading Parquet files with non-matching schemas as a list of files

[mcrumiller]

I've got it. FYI you're not using the pyarrow path, so for hte time being try use_pyarrow=True which may be a bit faster and doesn't trigger this error. If you don't have it installed, run pip install pyarrow.

[mcrumiller]

I've opened a more general issue at #13438 that will address this problem.

[mcrumiller]

I'm trying to hit the failing code path and having trouble without a minimal PR. Can you provide some of the expected datatypes of the files that are failing? How many columns do they have?

[alf239]

Ah, good point, let me see...

For the example above, the schemas are,

OrderedDict([('SomeId', Int32)])
OrderedDict([('SomeId', Int32)])

but then probably the problem is due to the files having different schemas, one having 2 extra columns — I hoped that explicitly specifying the columns would work around that problem (and in 0.19.5 that worked when using globs), but apparently not here.

[alf239]

One file has 3 extra String columns scattered, all past the SomeId; let me see if I can get an example isolated.

[mcrumiller]

For the example above, the schemas are,

OrderedDict([('SomeId', Int32)])
OrderedDict([('SomeId', Int32)])

but then probably the problem is due to the files having different schemas

Hang on, those schemas look identical, how are they different?

[alf239]

Yep, here you go:

import polars as pl

df1 = pl.DataFrame({
    'some-id': [1, 2, 3],
    'client': ['Alice Anders', 'Bob Baker', 'Charlie Chaplin'],
})
df2 = pl.DataFrame({
    'some-id': [4],
})

df1.write_parquet("/tmp/df1.parquet")
df2.write_parquet("/tmp/df2.parquet")

pl.read_parquet(["/tmp/df1.parquet", "/tmp/df2.parquet"], columns="some-id")

adjusting the description, too

[alf239]

It might look unreasonable, but then Polars 0.19.5 could do

import polars as pl

df1 = pl.DataFrame({
    'some-id': [1, 2, 3],
    'client': ['Alice Anders', 'Bob Baker', 'Charlie Chaplin'],
})
df2 = pl.DataFrame({
    'some-id': [4],
})

df1.write_parquet("/tmp/df1.parquet")
df2.write_parquet("/tmp/df2.parquet")

pl.read_parquet("/tmp/df*.parquet", columns="some-id")

successfully

(I do remember https://xkcd.com/1172/)

[alf239]

In 0.20.3 the glob example panics

[alf239]

use_pyarrow=True solves my problem, thank you! It doesn't support globs though (again, nothing blocking me here, just filling in the details)