gauthig / greenbutton-influxdb Goto Github PK
View Code? Open in Web Editor NEWParse SCE Green Button download files and creates import file or sends to influxdb. Grafana dashboard is included.
License: GNU General Public License v3.0
Parse SCE Green Button download files and creates import file or sends to influxdb. Grafana dashboard is included.
License: GNU General Public License v3.0
Windows 11
Latest version of python installed Python 3.11.5
Requirements satisfied
C:\Temp\SCEPython>python greenbutton_import.py -f SCE-Nemfile.sample --nodb
Traceback (most recent call last):
File "C:\Temp\SCEPython\greenbutton_import.py", line 165, in
(delivered, generated, metricsout) = parse_data(args.file, args.verbose,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Temp\SCEPython\utilparse.py", line 27, in parse_data
sce_tou_parse(csv_reader, verbose, metricsout)
File "C:\Temp\SCEPython\utilparse.py", line 54, in sce_tou_parse
dt_local = datetime.strptime(util_timestamp[0], '%Y-%m-%d %H:%M:%S')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib_strptime.py", line 352, in _strptime
raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: ร
Datetime being sent split out by segment and was spending influxdb's cpu/prog wait time to ingest.
need to convert to string before loading into dictionary for data import
Add SSL support to influx
After changes to timezone conversion of pytz library, normal datetime functions do not work which are implemented in the fileoutput function.
Get copy of non-solar custom SCE csv and be able to parse.
Setup new parameter
FileType
sce-nem = SCE CSV in Net Metering format for solar, has both generated and delivered sections
sce-res = SCE CSV in Residential non-nem format, only shows delivered energy.
Program sets event time as end of the interval, but the utility daily tottals are derived form the start of each event. Changing to use Start interval for the time event.
SCE data is local time but influxdb is supposed to be UTC. So any reports in a tool like grafana which converts to your local time will be off by -8.
Add time zone conversion
default to Los Angeles time zone since the program is primary for SCE customers.
Dashboard with
Hi This is not an issue, Just trying to reach out to you for help. I am not a programmer but I found your while looking for a way to read from SCE Green button data. Your code provide excellent information and might be useful for a lot of user. But not being a programmer I couldn't understand how to use it. I have a Docker installed on my synology and am also using Grafana to get data from Teslamate. Can you help me with using your tool. I can then write detailed instructions for general community. Thanks.
Since influxdb does not group by real month, need a tag for month.
Create version of dashboard to use FLUX
Add functionality to push to Influxdb v2 with token security
with open("myfile.txt", 'w') as f:
for key, value in details.items():
f.write('%s:%s\n' % (key, value))
Setup first pass to sending influxdb data the parsed results.
Will be non-ssl, no password influxdb - private lab environment only due to no security
Thanks for your efforts on this project! Attempting to set it up with my local utility provider, Pepco, the format of their residential net metering data is notably different than the SCE example provided. I've attached an example in case you might be able to add support in a future release.
example_pep_electric_interval_data_Service 1_2021-12-06_to_2022-12-06.csv
Currently only accepts command line parameters.
Desire is to have a parameter file to pull all options and parameters for database for quick reuse.
As each utility can format data differently, will add am argument to state which format to parse.
Listing as bug as default policy may delete data
Cleanup code to match PEP 8
Create first python script to generate parsed SCE Green Button data into a CSV format to check and validate the parsing puls the file correctly.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.