robmor/netzero

A collection of tools to mine data on the efficiency of a house

netzero's Issues

Best practices for configuration files

Standard practices for configuration files would suggest that JSON is not the best option. While it is an excellent transmission schema, industry standard seems to lean towards YAML or other property files.

Reasons:

• Low Signal to Noise ratio
  • There is a lot of unnecessary punctuation, redundant quotations and useless curly brackets for configuration management.
  • JSON also does not support trailing commas
• Long Strings
  • No support for multi-line strings
  • Strings that carries over onto another line of the file are not possible
• Numbers
  • As defined in the JSON spec, numbers are arbitrary precision finite floating point numbers in decimal notation. If you need to use hexadecimal notation or represent values like infinity or NaN, you might be out of luck

Suggestions: YAML, TOML, Dhall or any custom format implementation

Sources:
https://revelry.co/json-configuration-file-format/
https://softwareengineering.stackexchange.com/questions/351126/how-bad-of-an-idea-is-it-to-use-python-files-as-configuration-files
https://arp242.net/json-config.html
https://www.lucidchart.com/techblog/2018/07/16/why-json-isnt-a-good-configuration-language/

Test gshp data source

Pre-existing data is not accounted for

We can skip over some API querying if we first check what data already exists.

• Abstract table creation away from data sources
• Develop mechanism to search for and recognize gaps in the data
• Allow passing of multiple disjoint intervals into data sources
• Change tables in database to prefer newer data (INSERT OR REPLACE)
• Add option for user to force re-collection of all data (-f)

Test solar data source

Test pepco data source

Data formatting tool

Allow users to export the data stored in the database as a csv file

• Add command-line subcommand
• Add data merging functionality
• Add option to select date range for output data
• Add option to select which sources should be output
• Test feature

Improve tests for weather data source