How do I know the integration of a new dataset was successful

[pypsa]

How do I know the integration of a new dataset was successful

[FabianHofmann]

Hey @ChinCity, it looks quiet good. I think you have to pull the convert_alpha2_to_country line to the top before the country filtering.

Or probably the line

        .loc[lambda df: df.Country.isin(config['target_countries'])]

can be deleted as the config_filter should take care of this.

[ChinCity]

Hey @FabianHofmann,

I tried both of those but I'm still getting an empty dataframe :/

[FabianHofmann]

Have you checked if your raw file is properly loaded? If yes, I would go step by step through the lines and check where the entries are dropped (atm it seems only the config filter could do so)

[ChinCity]

You are right that did it!

Final code:

config = get_config() if config is None else config

change_fueltype={'Fossil Gas': 'Natural Gas',
                 'Fossil Coal-derived gas': 'Natural Gas',
                 'Hydro Run-of-river and poundage': 'Hydro',
                 'Hydro Pumped Storage': 'Hydro',
                 'Hydro Water Reservoir': 'Hydro',
                    'Fossil Hard coal': 'Hard Coal',
                    'Fossil Hard Coal': 'Hard Coal',
                    'Fossil Brown coal/Lignite': 'Lignite',
                    'Fossil Oil': 'Oil',
                    'Fossil Oil shale': 'Oil',
                    'Wind Offshore': 'Wind',
                    'Wind Onshore': 'Wind',
                    'Biomass': 'Bioenergy',
                    'Fossil Peat': 'Bioenergy',
                    'Marine': 'Other'}

df = parse_if_not_stored('JRC_OPEN_update', update, config)
if raw:
    return df
df = (df.rename(columns={'eic_p': 'projectID',
                          'name_p': 'Name',
                          'capacity_p': 'Capacity',
                          'country': 'Country',
                          'type_g': 'Fueltype',
                          'lat': 'lat',
                          'lon': 'lon',
                          'year_commissioned':'YearCommissioned'})
   .loc[lambda df: df.Country.isin(config['target_countries'])]
    .replace(change_fueltype)
    .drop(columns = ['eic_g', 'name_g', 'capacity_g', 'NUTS2',
                     'status_g', 'year_decommissioned', 'type_g2'])
     .pipe(set_column_name, 'JRC_OPEN_update'))

return df