Taking only 160 samples as input for generating new samples and also gives negative values in some columns of input tabular data

[ParagPatil3]

Taking only 160 samples as input for generating new samples and also gives negative values in some columns of input tabular data

[ParagPatil3]

Install latest version: pip install --upgrade tabgan

After upgrading tabgan, I am getting memory error as:
MemoryError Traceback (most recent call last)
in
6
7 # generate data
----> 8 new_train1, new_target1 = OriginalGenerator().generate_data_pipe(train, target, test, )
9 new_train1, new_target1 = GANGenerator().generate_data_pipe(train, target, test, )
10

~/anaconda3/envs/tab/lib/python3.7/site-packages/tabgan/abc_sampler.py in generate_data_pipe(self, train_df, target, test_df, deep_copy, only_adversarial, use_adversarial)
45 return generator.adversarial_filtering(new_train, new_target, test_df)
46 else:
---> 47 new_train, new_target = generator.generate_data(new_train, new_target, test_df)
48 new_train, new_target = generator.postprocess_data(new_train, new_target, test_df)
49 if use_adversarial:

~/anaconda3/envs/tab/lib/python3.7/site-packages/tabgan/sampler.py in generate_data(self, train_df, target, test_df)
99 train_df[self.TEMP_TARGET] = target
100 generated_df = train_df.sample(frac=(1 + self.pregeneration_frac * self.get_generated_shape(train_df)),
--> 101 replace=True, random_state=42)
102 generated_df = generated_df.reset_index(drop=True)
103 gc.collect()

~/anaconda3/envs/tab/lib/python3.7/site-packages/pandas/core/generic.py in sample(self, n, frac, replace, weights, random_state, axis)
5059 )
5060
-> 5061 locs = rs.choice(axis_length, size=n, replace=replace, p=weights)
5062 return self.take(locs, axis=axis)
5063

mtrand.pyx in numpy.random.mtrand.RandomState.choice()

mtrand.pyx in numpy.random.mtrand.RandomState.randint()

_bounded_integers.pyx in numpy.random._bounded_integers._rand_int64()

MemoryError: Unable to allocate 236. GiB for an array with shape (31680120000,) and data type int64

[ParagPatil3]

I have got the results with positive integers. Also, I could generate 100k samples. Thank you!

[Diyago]

What is your hardware?

[ParagPatil3]

What is your hardware?

I am trying on linux using virtual machine. Also tried with jupyter notebook using anaconda prompt in windows. (CPU)

[Diyago]

@ParagPatil3 I have fixed problems with negative values. Its was caused that heavy filter were skipped.
Secondly I added some gc.collect() - it reduced memory usage. Please take a look

If problem with memory usage still persist please provide full log.

[Diyago]

Install latest version: pip install --upgrade tabgan

[Diyago]

Thanks for full stack error. I had error for original sampling data - now it works well even with millions rows as input)

By the way you may pass use deep_copy=True - it will decrease memory usage by 2 times.

[Diyago]

@ParagPatil3 reopen as new issue with new details if error still persists