jufemaiz/aemo

AEMO Gem for managing Australian Energy Market data formats

AEMO Gem

Makes working with AEMO data more pleasant.

Documentation

http://www.rubydoc.info/gems/aemo

Installation

Ruby Versions Supported

• ruby-head (failures allowed)
• 3.2
• 3.1

Deprecated

• < 3.1

Manually from RubyGems.org

% gem install aemo

Or if you are using Bundler

# frozen_string_literal: true

# Gemfile
source 'https://rubygems.org' do
  gem 'aemo'
end

Maintenance

Updating loss factors

1. Connect to AEMO via VPN (tip: be a market participant in order to unlock this step)
2. Log in to MSATS
3. Navigate to Reports and Alerts » CATS » C1 - Data Replication Resynchronisation Report - Data Replication Resynchronisation Report (C1)
4. Export both the CATS_DLF_CODES and CATS_TNI_CODES tables (from 1-Jan-1970 to the current date, both with 30,000 max rows)
5. Wait...
6. Wait some more...
7. Navigate to Participant Outbox (you have new messages link)
8. Get the file!
9. Run the following:

   cd lib/data && ruby xml_to_json.rb

10. Add tests for your new FY to spec/lib/aemo/nmi_spec.rb
11. rspec - note that AEMO can retroactively change factors so if there is an old failing test double check the json to make sure it's not AEMO's fault.
12. Commit, push & create a pull request

aemo's Issues

AEMO::Region does not reference which AEMO Node it falls in

ACT is within NSW in terms of the market, but is a separate Region.

Timezone needs to be set to NEMTIME

Time parsing needs to be undertaken throughout in UTC+10:00 (aka NEMTIME)

Update Nokogiri

Ref: https://snyk.io/test/github/jufemaiz/aemo

CVE on rubyzip v1.1.7

https://gemnasium.com/github.com/jufemaiz/aemo
https://gemnasium.com/gems/rubyzip/versions/1.1.7

AEMO::NMI::NETWORKS have no AEMO::Region represenation

AEMO::NMI#tni_values should return Array of Hashes not an Array of Arrays of Hashes

https://github.com/jufemaiz/aemo/blob/master/lib/aemo/nmi.rb#L716

Change:

possible_values.map { |x| x['mlf_data']['loss_factors'] }

to

possible_values.map { |x| x['mlf_data']['loss_factors'] }.flatten

Parse NEM12 raises runtime error if file has a blank meter configuration

Test file:

100,NEM12,201710271425,COMPA,COMPB
200,NEEE000010,,001,E1,N1,7416444,KWH,30

Expected result

raise ArgumentError, 'NMIConfiguration is not valid'

Actual result

undefined method `match' for nil:NilClass

Drop support for Ruby versions prior to 2.3

Lonely operator support was introduced to Ruby in 2.3.

Ruby 2.2.8 is currently at EOL support with stable releases at 2.3.5 and 2.4.2 (with 2.5.0-preview1 available for review). Src: https://www.ruby-lang.org/en/downloads/

If there is no opposition to dropping 2.2.8 support I will be dropping support by EOY.

AEMO::NEM12 » 300 row » IntervalValue1 . . . IntervalValueN with nil value not raising an error

NEM12 file was parsed raising:

NoMethodError: undefined method `match' for nil:NilClass Did you mean? catch

This was caused due to a 300 row with the following representative data:

300,20180529,0.001,0.0,0.0,0.001,0.0,0.0,0.001,0.0,0.001,0.0,0.0,0.001,0.005,0.01,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,,,,,,,,,,,,,,,,,A,,,20180529160239,

The empty values should not have been empty but rather 0, with a flag of N raised for them. This is due to NEM12 export functionality being used on a dataset that was for the present day.

What should happen is that an exception is raised during the parsing of the dataset as the value present is not valid.

A valid dataset would look like:

300,20180529,0.001,0.0,0.0,0.001,0.0,0.0,0.001,0.0,0.001,0.0,0.0,0.001,0.005,0.01,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,A,,,20180529160239,
400,1,32,A,,
400,33,48,N,,

This would signify no value for the last 16 elements of the 48 provided here (30 minute interval data).

Likewise, 0 could be a valid value and be without the subsequent 400 rows (but not for the example above)

Changes to method flags to support 5 min settlement and same day churn

Additional method flags (from 1 May 2022):

Type	EST and/or SUB	Descriptor
Type 20	SUB	Churn Correction (Like Day)
Type 21	SUB	Five-minute No Historical Data
Type 59	SUB, EST	Five-minute No Historical Data
Type 69	SUB	Linear extrapolation

https://aemo.com.au/-/media/files/electricity/nem/retail_and_metering/market_settlement_and_transfer_solutions/2022/metrology-procedure-part-b-73.pdf?la=en

JSON dependency open-ended

WARNING:  open-ended dependency on json (>= 1.7.5) is not recommended
  if json is semantically versioned, use:
    add_runtime_dependency 'json', '~> 1.7', '>= 1.7.5'
WARNING:  See http://guides.rubygems.org/specification-reference/ for help

Investigate the use of csvreader

Ref:

• https://github.com/csv11/csvreader
• https://rubygems.org/gems/csvreader

Further reading:

• https://github.com/csv11/docs/blob/master/why-the-csv-stdlib-is-broken.md