Hive Partition Schema

[c-peters]

Hive Partition Schema

[ritchie46]

What we require first is schema inference on hive partitions. Otherwise some parts may be strings and/or different date formats. There needs to be something in place for schema inference and communicating that schema result between the partitions first.

[baycoder0]

+1

I had a similar issue open before: #12894

@ritchie46 is this something the core team has plans around by chance? If not, I'm willing to take a stab at it given some guidance on the desired design.

[deanm0000]

@baycoder0 c-peters is part of the core team.

I think you'd want to get started here

polars/crates/polars-plan/src/logical_plan/hive.rs

Lines 71 to 83 in e1a4179

	let s = if INTEGER_RE.is_match(value) {
	let value = value.parse::<i64>().ok()?;
	Series::new(name, &[value])
	} else if BOOLEAN_RE.is_match(value) {
	let value = value.parse::<bool>().ok()?;
	Series::new(name, &[value])
	} else if FLOAT_RE.is_match(value) {
	let value = value.parse::<f64>().ok()?;
	Series::new(name, &[value])
	} else if value == "__HIVE_DEFAULT_PARTITION__" {
	Series::new_null(name, 1)
	} else {
	Series::new(name, &[percent_decode_str(value).decode_utf8().ok()?])

[deanm0000]

The csv datetime parser seems to be over here

[deanm0000]

Additionally, this is highly related to #13892

[baycoder0]

@deanm0000 I added the initial PR here: #14950. I'd like to get the initial checks. Also, I did exhaustive testing myself and would like to add units tests. However, tests in tests/unit/io/test_hive.py seem to be skipped due to PyArrow 15 right now. Should I add it there? And should I add columns to foods1.ipc and foods2.ipc files that are used to test hive partitioning? Or do you prefer me to create new files?

[fcocquemas]

Are you still considering a parameter to pass a hive partitions schema, similar to how you can pass dtypes to override the inference? This is something @deanm0000 suggested in #13892.

There are some cases where it would be nice to override the regexp-based schema inference. For instance:

• It might be hard to detect all date formats correctly (Is this string %d/%m/%Y or %m/%d/%Y?). In some cases (e.g. 8 digit number), it may look like a date, but really be an Integer or a String.
• BOOLEAN_RE would match value=TRUE, even when it might represent a String, not a Boolean.

Having schema inference as a default is convenient, but an option to override would be nice.