automunge/automunge

Tabular feature encoding pipelines for machine learning with options for string parsing, missing data infill, and stochastic perturbations.

automunge's Issues

Possible date-time encoding feature?

AutoMunge is a great idea! I love it!

Did you think about adding some more interesting encoding for time based features?
Something like here

I think it would make sense in time series cases!

What's your take?

Onehot encoding enhancement

In some cases, onehot encoding categorical variables may hurt performance in tree based models. For some alternatives see
here here and here

Maybe it would be worth incorporating them.
What is your opinion?

Thanks!

[Question] Source Organisation

I saw a talk about the Automunge on Data Centric AI workshop in NIPS. Just taking a peek at the code.
I was going through the code and I noticed that the source is just a single file. Any specific reason for this?

pandas depreciating .mad

appears that pandas will be depreciating .mad (median absolute deviation) starting with 1.5.0
pandas.Series.mad

see e.g.
pandas-dev/pandas#28995
pandas-dev/pandas#11787

We currently collect mad in some of our drift stats as well as the transform built off of _process_MADn / _postprocess_MADn / _inverseprocess_MADn

this will need update when pandas rolls out 1.5.0, I think expectewd in September 2022

Non apply numeric standard transform

Any changes that could allow us to apply transformations solely to categorical data while preserving the original format of numeric data upon inputting the data source?

TypeError: invalid type promotion

Hi,

I am getting an error with a column data type timestamp:

_______________
Begin Automunge processing
evaluating column:  index
Traceback (most recent call last):
  File "D:\Anaconda3\envs\tipjar\lib\site-packages\IPython\core\interactiveshell.py", line 3417, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-24482f45187e>", line 18, in <module>
    featureimportance, postprocess_dict = am.automunge(X_train_events)
  File "D:\Anaconda3\envs\tipjar\lib\site-packages\Automunge\Automunger.py", line 30278, in automunge
    df_train = self.convert_inf_to_nan(df_train, column, category, postprocess_dict)
  File "D:\Anaconda3\envs\tipjar\lib\site-packages\Automunge\Automunger.py", line 29484, in convert_inf_to_nan
    df[column] = np.where(df[column] == np.inf, np.nan, df[column])
  File "<__array_function__ internals>", line 6, in where
TypeError: invalid type promotion

The dataframe has datatype:

X_train_events.dtypes
Out[4]: 
index     datetime64[ns]
open             float64
high             float64
low              float64
close            float64
volume             int64
dtype: object

Also, the index column all have proper date/time stamps ie no NANs, or inf etc..

I am new to automunge so still trying to understand the package. Thanks for any help