Default to Arrow String type instead of LargeString

[kevinjqliu]

Default to Arrow String type instead of LargeString

[deanm0000]

pl_flavor doesn't refer to the difference between a large_string and a string. It refers to the difference between a large_string and a utf8_view which doesn't seem to be implemented in pyarrow yet.

It seems @ritchie46 intended to close this as not planned so I'll do that now. Sorry if I'm mistaken on that point.

[ritchie46]

I'm still new to Polars. What are some use cases of LargeString?

Our in-memory engine favors large chunks (often single chunked dataframes). It is pretty easy to reach the 2GB string limit on user data that way.

Is it feasible to expose this boolean flag in py-polars as well?

This is to convert to string_view and is only temporary until arrow consumers implement binview.

[ritchie46]

We will not do that. Arrow default string can only hold 2GB of data per column, leading to all kinds of slicing requirements. We deem the default string utterly unusable for our use cases. You can always cast from LargeString to String and implement your own slicing if required.

[kevinjqliu]

Thanks for the quick reply!

We deem the default string utterly unusable for our use cases

I'm still new to Polars. What are some use cases of LargeString?

You can always cast from LargeString to String and implement your own slicing if required.

We will probably do this for pyiceberg. apache/iceberg-python#520

It looks like in Rust, there's a pl_flavor boolean flag that can be set to use the regular Arrow string instead (1, 2) but this is not available in Python.

Is it feasible to expose this boolean flag in py-polars as well?