energy-modelling-ireland/ibsg

Generate a clean Irish building stock using this Python-based streamlit web application

ibsg's Issues

Whoops - something went wrong with the link to the updated postcode BER filter

I tried to access the updated postcode BER filter list prototype: https://share.streamlit.io/rdmolony/ibsg/main/app.py

Any ideas?

Filter all BERs in 1GB of RAM

We need to find the right tool for the job. The BERs are >1GB uncompressed. pandas reads data into memory & so will crash in 1GB of RAM.

Options

• dask
• fugue
• ibis
• vaex
• sqlite

Link Postcode BERs to the 2016 Census

• Link the 2016 Census Small Areas to Postcodes
• Fill the 2016 Census with Postcode BERs

Remove DEAP Defaults in-app

• Walls
• Roof

See SEAI's Guide here for default values as of 08/07/2021

[BUG] zip files made by ibsg are empty

Log % buildings removed by each BER filter

Flag selected countyname in filename

If user selects just Dublin, this should be reflected in the output filename

Fill BERs with archetypes without exceeding resources

Is your feature request related to a problem? Please describe.
Filling census buildings with unknown BERs exceeded streamlit sharing's resource limit

Error uploading SEAI BER Public Search data

I used the SEAI's BER Public Search tool https://ndber.seai.ie/BERResearchTool/ber/search.aspx and used the download file from there to upload in zip format. Upon doing this I received the error - "Whoops — something went wrong! An error has been logged."

It appears the way around this is to install 7zip and upload the csv.gz file as a zip file and use that as the input dataset.

Map Overview of Postcode Statistics

Describe the solution you'd like
Map Postcode overview statistics as done here - https://renewables.maps.arcgis.com/apps/webappviewer/index.html?id=360f7b3f6f484d7d89b967b41231daef

Suggested in #15

Archetype Unknown Postcode-BER Buildings

• Fill 2016 Census Small Areas with Postcode-BER Archetypes

Remove erroneous Postcodes

Is your feature request related to a problem? Please describe.
A small number of entered Postcodes do not exist

Describe the solution you'd like
These buildings are removed by ibsg

Flag Date & Location Metadata in Filenames

Unlike the Postcode BERs the Small Area BERs currently need to be manually pulled by the SEAI and so are date-dependent. As of 08/07/2021 the codema-dev SA BERs dataset is from the 14/05/2021 - the clean BERs, BERs + Census and/or BERs + Census + Archetypes should reflect this

If user selects just Dublin, this should be reflected in the output filename

• Flag data date
• Flag selected countyname

Apply Codema Archetypes to BER-Public

Calculate postcode | period built averages on the Postcode BERs

Dublin SA BERs is 1,705,109 buildings

Describe the bug
ibsg is producing a dataset that is much larger than it should be & so is likely full of duplicate buildings.

To Reproduce
Steps to reproduce the behavior:

1. Run ibsg
2. Read the output data into a DataFrame:

df = pd.read_parquet("PARQUET-FILENAME")
len(df)
>> 1705109 

df["countyname"].unique()
>> array(['CO. DUBLIN', 'DUBLIN 1', 'DUBLIN 10', 'DUBLIN 11', 'DUBLIN 12',
       'DUBLIN 13', 'DUBLIN 14', 'DUBLIN 15', 'DUBLIN 16', 'DUBLIN 17',
       'DUBLIN 18', 'DUBLIN 2', 'DUBLIN 20', 'DUBLIN 22', 'DUBLIN 24',
       'DUBLIN 3', 'DUBLIN 4', 'DUBLIN 5', 'DUBLIN 6', 'DUBLIN 6W',
       'DUBLIN 7', 'DUBLIN 8', 'DUBLIN 9'], dtype=object)

Expected behavior
Dublin should only contain 500k or so buildings

[Q] purpose of ibsg? what are 'boundaries'? no link to census? map view? project assistance?

From Philippe Lemarchand email correspondance:

• [A] What do you call boundaries? Are they the geographical boundaries we select as "countyname"?

• [B] I can now open the file and noted there is no difference to the SEAI file. I guess the benefit is that SEAI files are limited on the number of dwellings that can be downloaded in a single file. Your system collates them automatically rather than me doing it manually - is this correct?

• [C] I had ticked the link to the 2016 census but I see no link in the csv file. Please could you tell me what information from the 2016 census we are supposed to see?

• [D] Another thing you might help me understanding as Jason mentioned the use of tableau to have a map view. The BER data do not contain any street name so how is it possible to generate a more detail view of a specific area as it is done here: https://renewables.maps.arcgis.com/apps/webappviewer/index.html?id=360f7b3f6f484d7d89b967b41231daef

• [E] I have never touched Tableau and I wonder, since you work in Codema, if you might be able to help us with our SEAI Stoneybatter Sustainable Energy Community project that is mentored by Codema. I have in mind to develop superposed layers of information regarding Stoneybatter area - is that possible? Is it something you would be interested in developing together?

Replace Missing Values in BER dataset

We should also look at the missing values in BER dataset. We could use mean or median to fill these values or advance data mining techniques.

Archetype Unknown Buildings via DBSCAN

See sklearn docs