opennem/opennem

Energy market data access platform

OpenNEM Energy Market Data Access

NOTE: This is the backend project For the client python project see opennem/opennempy

The OpenNEM project aims to make the wealth of public National Electricity Market (NEM) data more accessible to a wider audience.

This toolkit enables downloading, mirroring and accessing energy data from various networks

Project homepage at https://opennem.org.au

Available on Docker at https://hub.docker.com/r/opennem/opennem

Currently supports:

• Australian NEM: https://www.nemweb.com.au/
• West Australia Energy Market: http://data.wa.aemo.com.au/

Documentation

• The OpenNEM Wiki
• Developer Documentation
• API Interface and documentation

Contact

• File an issue
• Start a discussion
• Contact us on Twitter at @opennem

API Access

IMPORTANT Starting in April and with our next release the OpenNEM API will be placed behind authorization and will require an API key to access. We will be providing a free level of access for researchers.

We plan on gradually and completely deprecating the current API that is open to all.

If you are a current API user, or a potential API user, please fill in this form to register for access to the API. We will be in touch with you with an email with the various setup steps and access details.

The new API and registration can be discussed here

opennem's Issues

Releases not matching deployed released versions

https://data.opennem.org.au/v3/stats/au/NEM/energy/2021.json

https://github.com/opennem/opennem/releases

Fueltech Mix API Endpoint Issue

Timeout on :

https://api.opennem.org.au/stats/fueltech_mix/NEM

Morgan Whyalla Pipeline Station Name

AEMO GI has the names as :

Morgan To Whyalla Pipeline No 1 PS And Water   Filtration Plant
Morgan To Whyalla Pipeline No 2 PS
Morgan To Whyalla Pipeline No 3 PS
Morgan To Whyalla Pipeline No 4 PS

Current station_name_cleaner is butchering it into Morganwhyalla Pipeline - should be Morgan-Whyalla Pipeline

Queensland Coal Generation Shortfall

QLD1 coal generation shortfall this morning. seems to coincide with release of v3.9.10 to prod

Trap Connection Errors

Trap connection errors to db and cache so they don't float up to generalized exceptions

Sentry Issue: BACKEND-H6

OperationalError: SSL SYSCALL error: EOF detected

  File "sqlalchemy/engine/base.py", line 1771, in _execute_context
    self.dialect.do_execute(
  File "sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)

OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected

[SQL: 
select
    t.trading_interval,
    t.fueltech_code,
    sum(t.fueltech_power)
from (
    select
        time_bucket_gapfill('30 minutes', fs.trading_interval) AS trading_interval,
        ft.code as fueltech_code,
        coalesce(avg(fs.generated), 0) as fueltech_power
    from facility_scada fs
    join facility f on fs.facility_code = f.code
    join fueltech ft on f.fueltech_id = ft.code
    where
        fs.is_forecast is False and
...
(8 additional frame(s) were not displayed)
...
  File "sqlalchemy/engine/base.py", line 1814, in _execute_context
    self._handle_dbapi_exception(
  File "sqlalchemy/engine/base.py", line 1995, in _handle_dbapi_exception
    util.raise_(
  File "sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "sqlalchemy/engine/base.py", line 1771, in _execute_context
    self.dialect.do_execute(
  File "sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)

Unhandled exception in task 86fbee7a-b819-4377-bc2e-e856ad107faa.

Missing BoM data

For 15th, 16th, 17th october we're missing intervals in the bom data but have max/min

backfill that data

Leading edge of stripe data is null

in every stripes output the latest is null, 0 - likely to do with timing of tasks and task dependency which relates to #26

Facility List Errors

• commissioning / committed facilities not approved
• dupes in facility list
• name filtering on elements like Muja Cd vs Muja CD (the MUJA_M units)

Temp mismatches time

Temp on 1 day SA view shows temp does not reflect correct temp. See photo

Data exports (CSV) naming convention inconsistent

See attached CSV exports for all three at the same point in time.

20210902 All Regions.csv
20210901 Western Australia (SWIS).csv
20210902 OpenNEM.csv

Flow Emissions

Show single energy source with a single click

Right now, if I want to see only one energy source in the generation graph, I have to go through and click on every single one of the others to turn them off.

It would be nice if there was a way to disable all of them except for one with a single click.

Data exports (CSV) do not indicate timezone

See attached CSV exports for all three at the same point in time.

20210902 All Regions.csv
20210901 Western Australia (SWIS).csv
20210902 OpenNEM.csv

Committed and Commissioning Facilities in Facility List

Currently Committed and Commissioning facilities in the fac list are not appearing in the geojson output.

Error getting nemweb to work using "Quick example" instruction

Hi Nik, I briefly discussed with Simon about my trouble getting nemweb set up on my system to import and wrangle data from nemweb.com.au

I am using Python 3.7.4 on Windows 10, and SQLite 3.32.3

I wonder if I have not done the following correctly. I have never used SQLite previously and not a regular Python programmer.

IMPORTANT If you install via pip you must manually enter the sqlite directory in a file named .nemweb_config.ini in your home directly post- install (...couldn't figure out how to make post-install scripts to work with pip).

Live prices API endpoint

API endpoint for live region price data (5 min data)

Import BoM Actuals

BoM review their data after 14 days and release actuals. Should be imported into observations. The temp differences are up to ~2C.

Station endpoint limit param results in internal server error

https://api.opennem.org.au/station <- works fine
https://api.opennem.org.au/station/?name=Lake <- works fine
https://api.opennem.org.au/station/?name=Lake&facilities_include=false <- works fine
https://api.opennem.org.au/station/?name=L&facilities_include=false&only_approved=true <- works fine

https://api.opennem.org.au/station/?name=L&facilities_include=false&only_approved=true&limit=5 <- Internal Server Error

Interconnect summation not netting out

Originally raised here: https://twitter.com/expert_grass/status/1430720649220091912

QLD / NSW should interconnect should sum to zero between import and export.

As you can see QLD is exporting 1186MW but NSW is only importing 766MW.

At 30min resolution a similar story plays out for energy.

Including the other nodes:

QLD export 1186MW
NSW import 766MW
TAS export 441MW
VIC import 509MW
SA import 193MW

Again this doesn't stack up.

Legend for Increase/Decrease fitted line is truncated by chart top

(This is a duplicate of #58 in openem-fe, I wasn't sure which project was appropriate)
Client:
macOS High Sierra 10.13.6 (17G14042)
Safari Version 13.1.2 (13609.3.5.1.5)
OpenNEM:
https://opennem.org.au/emissions/au/ v4.5.2

Minor usability issue.
In some situations the legend that describes a fitted Increase/Decrease line between two years/quarters is truncated by the chart top.

Repeat by:
Initial chart settings: Quarterly, others default.
Add "Land Management" values, increasing proportion of chart occupied.
Select Q1 2007, then Q3 2008.
Note that the legend "Decrease of 9.? Mt" is truncated
(Can't attach an image, it's refusing JPG and PNG as small as 77Kb.

Unclear mass values in right-hand-side legend

Client:
macOS High Sierra 10.13.6 (17G14042)
Safari Version 13.1.2 (13609.3.5.1.5)
OpenNEM:
https://opennem.org.au/emissions/au/ v4.5.2

Background:
The mass figures displayed in the right-hand legend vary, depending on the cursor location. When the cursor is within a specific year (resp. quarter) then the data is for that year (resp. year-to). However when the cursor is outside the graph then the figures displayed appear to be the sum of all the displayed years, in the case of the annual display.

Issue:

1. For the quarterly display the figures are much bigger, approximately (but not exactly) four times larger, which superficially looks as though the figures displayed on the annual graph are actually quarterly figures. Since they're not labelled it's confusing.

2. The Y axis legend doesn’t alter for the quarterly display, whereas you’d expect each quarter’s figures to be (roughly) one quarter of the annual figures. This is confusing.

Usability Recommendations

opennem observations.pdf

ETL Improvements

enhanced ETL in v3.8 with:

• celery workers
• rabbitmq (persistence from data loss)
• airflow for tasks

address issues with tasks being missed, not exec'd in order, gaps in data etc. and monitoring enhancements

Inconsistency in number of datapoints returned

Hi from electricitymap.org. 👋

While refetching some historical data, we observed some inconsistencies in the number of data points returned for some date times.

https://api.opennem.org.au/stats/flow/network/NEM?month=2017-01-01 correctly returns 8928 data points for the first data element ['data'][0]['history']['data'], which matches with 2017-01-01 00:00:00+10:00-2017-01-31 23:55:00+10:00 and 5min buckets.

https://api.opennem.org.au/stats/flow/network/NEM?month=2017-10-01 only returns 8927 data points for the first data element. We observed multiple dates with this issue.

Currently, we discard the dates that contains these inconsistencies.

Fueltech Mix By Network API endpoint not working or unclear in documentation

Hiya

as per docco, I am having trouble with this API.

eg:

https://api.opennem.org.au/stats/fueltech_mix/NEM

results in:

{"detail":[{"loc":["query","network_code"],"msg":"field required","type":"value_error.missing"}]}

The other endpoints seem to be working ok

Paul

Invert flow data method not checking types before inverting

Sentry Issue: BACKEND-27T

TypeError: bad operand type for unary -: 'NoneType'
(18 additional frame(s) were not displayed)
...
  File "starlette/concurrency.py", line 40, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "opennem/api/stats/router.py", line 379, in power_flows_network_week
    ds_inverted = invert_flow_set(ds)
  File "opennem/core/flows.py", line 237, in invert_flow_set
    flow_set_inverted.history.data = [-i for i in flow_set.history.data]
  File "opennem/core/flows.py", line 237, in <listcomp>
    flow_set_inverted.history.data = [-i for i in flow_set.history.data]

Exception in ASGI application

Negative Emissions

Generators sometimes draw power from the grid - emissions calcs are showing these as negative emissions. Emissions queries need to be altered to generated > 0 in any emissions calculations.

OpenNem Western Australia (SWIS)

The energy (gwh) data for November has not populated on the OpenNem SWIS page. This information is very useful used in conjunction with the MW measurements. Is there an issue or will this data be no longer populated? Many thanks

Unmapped DUID for Victorian Big Battery

DUID VBBG1 currently unmapped and belongs to Victoria Big Battery

REL data is:

360MW battery in VIC1, we have VBBL1 mapped as the load

Gap in QLD weather data

Missing data for QLD weather from:

2021-09-22 09:00:00 -> 2021-09-23 15:00:00 (AEST)

station id is 040913

Tasks

• Find the data to fill in
• bucket_gapfill so that the output series matches up
• Alert on these gaps in future
• Related to the BoM actuals issue as we can backfill using that same process (#29 )

Question regarding calculation of emissions intensity

Not a bug - could you please point me to how Carbon Dioxide Emissions Intensity is calculated in OpenNem for each state? Thanks!

Internal server error (flow/network/NEM?month=2021-06-01)

We experience some internal server errors when querying some API resources. Some examples:

• https://api.opennem.org.au/stats/flow/network/NEM?month=2021-05-01
• https://api.opennem.org.au/stats/flow/network/NEM?month=2021-05-10
• https://api.opennem.org.au/stats/flow/network/NEM?month=2021-06-01

Emails to [email protected] bounce

Database deadlocks

Sentry Issue: BACKEND-9K

(psycopg2.errors.DeadlockDetected) deadlock detected
DETAIL:  Process 11517 waits for ShareLock on transaction 12583126; blocked by process 11516.
Process 11516 waits for ShareLock on transaction 12583127; blocked by process 11517.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (143,57) in relation "_hyper_2_11695_chunk"

[SQL: INSERT INTO balancing_summary (created_by, network_id, trading_interval, network_region, price_dispatch, is_forecast) VALUES (%(created_by_m0)s, %(...

Temperature Data is Misaligned to date

I noticed when using the 1 Year View in days that the temperature data is mismatched.
I was looking at data for SA and noticed higher demand one 22 July. Looked at weather and saw the 23 July (Friday) was lower demand but much colder which seemed odd. I checked BoM and confirmed that the temps shown on OpenNem for 23rd were in fact the dat for 22nd.

Summated "All Regions" (NEM + SWIS) values not adding up

From: https://twitter.com/yestiseye/status/1423481441535496193

The following approach was used to generate files for each generator type to compare and contrast, and data output included:

github.com/jufemaiz/opennem-15

Assumptions is that, while different timezones are in use (UTC+10:00 for NEM and UTC+08:00 for SWIS) all other definitions of time (leading edge / falling edge of the period) are consistent between the two regions on Opennem (to ensure consistency).

For Solar (as raised above) please see:

• https://github.com/jufemaiz/opennem-15/blob/main/output/solar_rooftop_mw.csv
• https://github.com/jufemaiz/opennem-15/blob/main/output/solar_utility_mw.csv

Catch bad dates in API queries

ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near "2021"

Sentry Issue: BACKEND-27V

ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near "2021"
LINE 20:         fs.trading_interval <= '2021-08-31 23:59:59+10:00' a...
                                         ^

[SQL: 
select
    t.trading_interval,
    t.fueltech_code,
    sum(t.fueltech_power)
from (
    select
        time_bucket_gapfill('5 minutes', fs.trading_interval) AS trading_interval,
        ft.code as fueltech_code,
        coalesce(avg(fs.generated), 0) as fueltech_power
    from facility_scada fs
    join facility f on fs.f...
(24 additional frame(s) were not displayed)
...
  File "sqlalchemy/engine/base.py", line 1814, in _execute_context
    self._handle_dbapi_exception(
  File "sqlalchemy/engine/base.py", line 1995, in _handle_dbapi_exception
    util.raise_(
  File "sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "sqlalchemy/engine/base.py", line 1771, in _execute_context
    self.dialect.do_execute(
  File "sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)

Exception in ASGI application

ValidationError: 24 validation errors for OpenAPI

Sentry Issue: BACKEND-KE

ValidationError: 24 validation errors for OpenAPI
components -> schemas -> FacilityFeature -> properties -> bbox -> anyOf -> 0 -> items
  value is not a valid dict (type=type_error.dict)
components -> schemas -> FacilityFeature -> properties -> bbox -> anyOf -> 1 -> items
  value is not a valid dict (type=type_error.dict)
components -> schemas -> FacilityFeature -> $ref
  field required (type=value_error.missing)
components -> schemas -> FacilityGeo -> properties -> bbox -> anyOf -> 0 -> items
  value is not a valid dict...
(15 additional frame(s) were not displayed)
...
  File "starlette/routing.py", line 52, in app
    response = await func(request)
  File "fastapi/applications.py", line 161, in openapi
    return JSONResponse(self.openapi())
  File "fastapi/applications.py", line 136, in openapi
    self.openapi_schema = get_openapi(
  File "fastapi/openapi/utils.py", line 410, in get_openapi
    return jsonable_encoder(OpenAPI(**output), by_alias=True, exclude_none=True)  # type: ignore
  File "pydantic/main.py", line 406, in pydantic.main.BaseModel.__init__

Exception in ASGI application

OpenNEM website solar and gas bugs

Hi, Here are a few bugs I found with the OpenNEM website.

1. Rooftop solar drops to zero over the latest hour or so, for the NEM and NEM states.

2. For Western Australia, rooftop solar shows a few MW output during the night.

3. For Tasmania, gas usage is slightly negative.

I'm new to this so hope I'm doing it right.

ElectricityMaps bugs and issues

the endpoint used by electricity-map:

• returns solar_utility for NSW, QLD, VIC, SA, WEM (there is no utility solar in TAS)
• returns solar_rooftop for WEM

but

• omits solar_rooftop for NSW, QLD, VIC, SA & TAS.

please add the solar_rooftop series for the 5 missing states.

this is resulting in rooftop solar being omitted from electricity-map, and making australia look even worse than it is!

error in opennem setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers

I'm attempted to get involved and started with the developer instructions. I'm using python 3.8 on macos.

poetry install

I get the error

    error in opennem setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.

Full error

[EnvCommandError]
Command ['/Users/tourdownunder/Projects/opennem/.venv/bin/pip', 'install', '--no-deps', '-e', '/Users/tourdownunder/Projects/opennem'] errored with the following return code 1, and output:
Looking in indexes: http://localhost:3141/root/pypi/+simple/
Obtaining file:///Users/tourdownunder/Projects/opennem
    ERROR: Command errored out with exit status 1:
     command: /Users/tourdownunder/Projects/opennem/.venv/bin/python3.8 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/tourdownunder/Projects/opennem/setup.py'"'"'; __file__='"'"'/Users/tourdownunder/Projects/opennem/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/f1/0jzpddz54_7fr79d9szr58s80000gn/T/pip-pip-egg-info-hboxp8ri
         cwd: /Users/tourdownunder/Projects/opennem/
    Complete output (1 lines):
    error in opennem setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 20.1.1; however, version 20.2.1 is available.
You should consider upgrading via the '/Users/tourdownunder/Projects/opennem/.venv/bin/python3.8 -m pip install --upgrade pip' command.

Is there anything I'm missing.

Note I used devpi and thats whats running on http://localhost:3141

Clarity on Weather Station Observations endpoint parameters

Hiya

(I will have a few issues like this but easier to post them separately rather than one large one as I work on this sporadically).

Why does the weather station observations endpoint ask for network as a parameter? From my testing it seems to be ignored anyway as I set it to WEM for a Melbourne station.

Context is a user accessing this API via Power Automate will be prompted with this type of UI. I can hide params that are not intended to be user facing

Curtailment Data

Are there any plans to provide curtailment data similar to what nemlog offers through the OpenNEM front end and API?

Possible enhancements

OpenNEM:
https://opennem.org.au/emissions/au/ v4.5.2

1. It would be convenient to have the Net Total figure displayed next to the Net Total legend on the right hand side, in the same way as the component figures and percentages are displayed. I realise it’s also shown at the top of the graph, however having the total at the bottom of the column would be convenient.

2. Although it’s possible to display by Quarter, the data appears to be a year’s worth, not a quarter in the quarterly display. I’m not sure why you’d want it that granular, but if you want to compare (say) summer to winter quarters then currently it’s not possible.

Support NEM 5 minute price data

AEMO has started publishing 5 minute price data for the NEM. Issue to track implementing + integrating the data

Price Value Field used in Exports

Current discrepancy between price value used in exports. It should be using VWP rather than trading/dispatch price (can't recall off the top of my head which one it's using atm)

TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'

Sentry Issue: BACKEND-KF

TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 662, in _runCallbacks
  File "/usr/local/lib/python3.8/site-packages/scrapy/utils/defer.py", line 150, in f
  File "/tmp/opennem-1630501928-xvmadywh.egg/opennem/utils/pipelines.py", line 61, in wrapper
  File "/tmp/opennem-1630501928-xvmadywh.egg/opennem/pipelines/wem/facility_scada.py", line 60, in process_item
  File "/tmp/opennem-1630501928-xvmadywh.egg/opennem/pipelines/nem/opennem.py", line 95, in unit_scada_generate_facility_scada

Error processing {"content":"PERIOD,PARTICIPANT_CODE,FACILITY_CODE,ACTUAL_MW,PCT_ALT_FUEL,PEAK_MW,OUTAGE_MW,PEAK_OUTAGE_MW,POTENTIAL_MWH,INTERVALS_GENERATING,TOTAL_INTERVALS,PCT_GENERATING,AS_AT\n2021-09-07 18:30:00,\"ALBGRAS\",\"ALBANY_WF1\",7.39,0,7.39,,,3.217,1,1,1,\"2021-09-09 18:30:00\"\n2021-09-07 18:30:00,\"ALCOA\",\"ALCOA_WGP\",0,0,0,16,16,8,0,1,0,\n2021-09-07 18:30:00,\"ALINTA\",\"ALINTA_PNJ_U1\",97.341,0,97.341,,,68.5,1,1,1,\n2021-09-07 18:30:00,\"ALINTA\",\"ALINTA_PNJ_U2\",97.2,0,97.2,,,68.5,1,1,1,\n2021-09-07 18:30:00,\"ALINTA\",\"ALINTA_WGP_GT\",...","table_schema":"<class 'opennem.db.models.opennem.FacilityScada'>","update_fields":["generated"]}

API Documentation and Links

We require better documentation for integration with the OpenNEM API and using the Python client library with examples.

Open issue here for tracking tasks / progress on improving documentation and onboarding API integrations

Tasks

• Link the API generated docs from homepage / developer docs
• Link OpenAPI schema
• Better developer landing page (developers.opennem.org.au) with overview
• Link to CC license / terms
• More examples of API integrations and using client library

Schedule on NEM energy for nightlies

On some occasions the scheduler is missing the next day dispatches from nemweb. ex. 12th Sep 2021

Fix is to check more often and run it again

Add a "Date of commission" column (also "Date of refurbishment/decommission") to facilities listing.

Congratulations on the new Facility features. I'm sure you have more planned for the integration of these facilities into the generation visualisations, but in the mean time, it would he quite useful to have the commissioning date for each generation facility added to the list. In the case of financed projects under construction but not yet commissioned (generating).

I note in the CSV data file from facilities page has a column for Status, but all facilities are listed as "Commissioned". That could have an optional date status, if it is locked in as a ?boolean or enumerated list index value already, then and add a new column for commissioning date I suppose.

The case of coal power stations where individual generation units may have been commissioned, refurbished and/or have been mothballed and decommissioned/demolished at various dates provides a problematic edge case I don't have the answer for. (Muja in WA is an example where there are different dates for commissioning A, B, C, & D, refurbishing A & B, subsequent decommissioning of A & B, and closure announcement for the two units at Muja C.) see https://en.wikipedia.org/wiki/Muja_Power_Station