replace_time_zone()'s ambigouous argument doesn't work correctly

[Sage0614]

replace_time_zone()'s ambigouous argument doesn't work correctly

[MarcoGorelli]

thanks for the report - this looks expected, I think you're asking for a non_existent argument, see #11579

closing in favour of that one then

[Sage0614]

@MarcoGorelli will you consider the propose of #13758 at the same time? when you say in #11579

whereas ambiguous datetimes are a fact of life, non-existent ones indicate a data error which should probably be cleaned by the user

I agree with that, but there's no simple way to branch part of the data to be problematic and handle it separately, like if else in pthon. I do try to flag the data error with when-then-otherwise logic, and replace_time_zone is only supposed to handle the 'clean' branch, but no, all code path need to work simutaneously. it is a pain to use the when-then-otherwise logic when the code can potentially error or panic.

[MarcoGorelli]

this is the suggestion I'm making there, so to have non_existent take 'null' or 'raise'

[Sage0614]

this is the suggestion I'm making there, so to have non_existent take 'null' or 'raise'

I am aware of that and I am not complaining about not having the option in replace_time_zone here,
I am discussing the general case, what can user do when they have some use case where polars currently gives an error/panic?
some of problem are fixed by polars team in a week, some of the problem are fixed by maybe in a year, but if I need to use polars in production, I cannot wait till tomorrow.

So, I am asking polars team in addition to the parallel version of when-then-otherwise, to also provide us a version of when-then-otherwise, where exactly one branch of the when-then-otherwise will be executed, so, if I encounter some error/panic, I can have a quick workaround by filtering the data and handle the cases that causes the error with a when-then-otherwise clause, and when the problem is fixed by polars team I can just switch back to the offical solution, with minimal code change.

[MarcoGorelli]

I'd suggest explaining this in #13758

[deanm0000]

In the specific case of non-existent time zones there are two work arounds that I've used.

One is at the end of this.

The other is to create a clean range and then join to the unclean.

So suppose you have

unclean = pl.select(a=pl.datetime_range(pl.datetime(2022,3,13), pl.datetime(2022,3,14), '30m'))

Doing the following will produce an error

unclean.with_columns(pl.col('a').dt.replace_time_zone('America/Chicago'))
ComputeError: datetime '2022-03-13 02:00:00' is non-existent in time zone 'America/Chicago'. Non-existent datetimes are not yet supported

To get around that without looping with a try/except block (as above), you can do

clean=pl.select(b=pl.datetime_range(unclean.select(pl.col('a').min().dt.replace_time_zone("America/Chicago")).item(),
                                    unclean.select(pl.col('a').max().dt.replace_time_zone("America/Chicago")).item(),
                                    unclean['a'].diff().min())).with_columns(
                                        a=pl.col('b').dt.replace_time_zone(None)
                                    )
unclean.join(clean, on='a', how='left')
shape: (49, 2)
┌─────────────────────┬───────────────────────────────┐
│ a                   ┆ b                             │
│ ---                 ┆ ---                           │
│ datetime[μs]        ┆ datetime[μs, America/Chicago] │
╞═════════════════════╪═══════════════════════════════╡
│ 2022-03-13 00:00:00 ┆ 2022-03-13 00:00:00 CST       │
│ 2022-03-13 00:30:00 ┆ 2022-03-13 00:30:00 CST       │
│ 2022-03-13 01:00:00 ┆ 2022-03-13 01:00:00 CST       │
│ 2022-03-13 01:30:00 ┆ 2022-03-13 01:30:00 CST       │
│ 2022-03-13 02:00:00 ┆ null                          │