Duplicates present in Unique columns

[gretelai]

Duplicates present in Unique columns

[johntmyers]

Hi @Satishkumar44 you can actually preserve certain columns and use them as seeds for generating the rest of the record so you get exactly 1000 records with the same PKs. Here's some modifications you can do:

Just after you create the config_template and before you create the SyntheticDataBundle you can extract out a list of dictionaries that represent "seed" values for each record.

Also here, I replicated the training data 5 times, which is optional, but in this case helped generate results faster:

import json

seed_data = json.loads(training_df[["REF_ID"]].to_json(orient="records"))
training_df = pd.concat([training_df] * 5).sample(frac=1)

When you create the SyntheticDataBundle you specify which columns you want to preserve as seed fields using the header_prefix=["REF_ID"] param

When you go to generate data, you can feed the list of seed fields in. The model will generate one record for each seed in the list:

bundle.generate(num_lines=nrows, max_invalid=nrows, seed_fields=seed_data, num_proc=1)

Please note that num_proc=1 must be set here, as only one CPU can currently be used for seed based generation.

I tested this and attached my results.
synthetic.csv.zip

[Satishkumar44]

while runing the bundle.generate() facing the below error . Can you help me out.

[johntmyers]

Please share more detail. If you can share the full Notebook. The error means you have too many seed fields specified and you should only have one based on what I provided above.

[Satishkumar44]

I have shared my notebook the data is same as shared before
Synthetic Data Generation.zip

[johntmyers]

This looks like a slightly older version of gretel-synthetics is running. Can you re-install that so you are at version 0.15.2?

[Satishkumar44]

Yes after upgrading to 0.15.2 it works fine, But my issue is when iam trying to generate 2000 rows from the same data without preserving the primary key it is generating data duplicates in primary key column(REF_ID).Is there any way that primary key columns will not contain duplicates.

[johntmyers]

There's currently no way to avoid duplicates unless you specifically provide a list of seeds like demonstrated above. Otherwise you can drop the primary key column, generate records, and add the primary key values back in.

[Satishkumar44]

Thankyou for your support