BUG: pandas >= 2.0 parses 'May' in ambiguous way.

[ffyring]

BUG: pandas >= 2.0 parses 'May' in ambiguous way.

[ffyring]

I did some debugging. The below format is also failing and working fine in pandas < 2.0 versions.

csv_error = """ REF_DATE, VAL 20-05-2024, 120 20-May-2024, 100 """

When we pass format='mixed' it works as expected. I guess this would be the solution.

df_error = pd.read_csv(StringIO(csv_error)) df_error['REF_DATE'] = pd.to_datetime(df_error['REF_DATE'],format='mixed') df_error['NEXT_DATE'] = df_error['REF_DATE'] + pd.Timedelta(days=1)

Thanks, passing format='mixed' works fine.

I still think it is a bug or a regression though, possibly breaking old code bases, since the format really isn't mixed in this case.

[santhoshbethi]

It is working fine for me. I am using the installed versions.

[ffyring]

It is working fine for me. I am using the installed versions.

And the installed version is above 2.0?

[santhoshbethi]

Yes

My bad. I can reproduce the error. I am using pandas 2.2.0 Now.

[santhoshbethi]

I did some debugging. The below format is also failing and working fine in pandas < 2.0 versions.

csv_error = """
REF_DATE, VAL
20-05-2024, 120
20-May-2024, 100
"""

When we pass format='mixed' it works as expected. I guess this would be the solution.

df_error = pd.read_csv(StringIO(csv_error)) df_error['REF_DATE'] = pd.to_datetime(df_error['REF_DATE'],format='mixed') df_error['NEXT_DATE'] = df_error['REF_DATE'] + pd.Timedelta(days=1)

[rhshadrach]

Thanks for the report!

pandas should infer the datetime format correctly, using several entries when the format is ambiguous.

How many? And if they are all "May"?

I still think it is a bug or a regression though, possibly breaking old code bases, since the format really isn't mixed in this case.

The breaking change was announced here: https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html#datetimes-are-now-parsed-with-a-consistent-format

Thanks, passing format='mixed' works fine.

I'd recommend format="%d-%b-%Y" instead, if the format is indeed not mixed.

It seems to me that inference is a guess, and sometimes guesses can be wrong. I'd recommend not relying on inference when possible.

cc @MarcoGorelli

[ffyring]

Thanks for the report!

pandas should infer the datetime format correctly, using several entries when the format is ambiguous.

How many? And if they are all "May"?

I still think it is a bug or a regression though, possibly breaking old code bases, since the format really isn't mixed in this case.

The breaking change was announced here:
I'd recommend format="%d-%b-%Y" instead, if the format is indeed not mixed.

It seems to me that inference is a guess, and sometimes guesses can be wrong. I'd recommend not relying on inference when possible.

cc @MarcoGorelli

Thanks, definitely not a bug then. I think your viewpoint as reasonable and this can be closed.

[ffyring]

Not a bug.