jertel/vuegraf

Populate metrics from your Emporia Vue energy monitoring devices into an InfluxDB

vuegraf's Issues

Failed to record new usage data: (<class 'TypeError'>, TypeError("argument of type 'NoneType' is not iterable"), <traceback object at 0x7fcd6f6ca1c0>)

There appears to be an issue, I added a new device this week but I don't think that caused this issue? But maybe?

Thank you for the application and any advice to get my system recording again!

2021-05-22 00:39:17.673075 | INFO  | Login completed,
2021-05-22 00:39:18.050222 | INFO  | Discovered new channel: Entire Home (1,2,3),
2021-05-22 00:39:18.232833 | INFO  | Discovered new channel: Living Room (1,2,3),
2021-05-22 00:39:18.405748 | INFO  | Discovered new channel: OfficePC (1,2,3),
2021-05-22 00:39:18.576398 | INFO  | Discovered new channel: OfficeNet (1,2,3),
2021-05-22 00:39:19.097630 | INFO  | Discovered new channel: GarageEGO (1,2,3),
2021-05-22 00:39:18.926201 | INFO  | Discovered new channel: OfficeTable (1,2,3),
2021-05-22 00:39:18.756354 | INFO  | Discovered new channel: BedroomPC (1,2,3),
2021-05-22 00:39:19.282881 | INFO  | Discovered new channel: Freezer (1,2,3),
2021-05-22 00:39:19.653758 | INFO  | Discovered new channel: Bedroom Fan (1,2,3),
2021-05-22 00:39:19.906125 | ERROR | Failed to record new usage data: (<class 'TypeError'>, TypeError("argument of type 'NoneType' is not iterable"), <traceback object at 0x7fcd6f0fcbc0>),
2021-05-22 00:39:19.486876 | INFO  | Discovered new channel: Refrigerator (1,2,3),
2021-05-22 00:40:20.168105 | ERROR | Failed to record new usage data: (<class 'TypeError'>, TypeError("argument of type 'NoneType' is not iterable"), <traceback object at 0x7fcd6f6ca1c0>)

Starting a couple days ago, I am getting a Fatal Error at launch

This script has been working perfectly for me for months. But the other day I noticed it had stopped gathering data and I get the following error when I launch the script:

Fatal error: (<class 'TypeError'>, TypeError('Parser must be a string or character stream, not NoneType'), <traceback object at 0x7f727d23dac0>)

Missing data due to Internet, System or other outage

Based on discussion point 131, need a procedure to daily find missing data and pull it from emporia.

If you approve, please tag as enhancement and assign it to me. ETA 1 week.

Latest container image crashing

Looks like when the latest image got pulled there's a missing module causing a crash:

Traceback (most recent call last):
  File "/opt/vuegraf/vuegraf.py", line 28, in <module>
    from pyemvue import PyEmVue
  File "/usr/local/lib/python3.12/site-packages/pyemvue/__init__.py", line 2, in <module>
    from pyemvue.pyemvue import PyEmVue
  File "/usr/local/lib/python3.12/site-packages/pyemvue/pyemvue.py", line 9, in <module>
    from pyemvue.auth import Auth
  File "/usr/local/lib/python3.12/site-packages/pyemvue/auth.py", line 3, in <module>
    from jose import jwt
ModuleNotFoundError: No module named 'jose'

Syntax Error in vuegraf.py?

Hello, I've jsut followed the instructions for vuegraf but I'm seeing a syntax error on line 36 of vuegraf.py:

pi@raspberrypi:~/vuegraf-master/src $ python vuegraf.py vuegraf.json File "vuegraf.py", line 36 print('{} | {} | {}'.format(now, level.ljust(5), msg), flush=True) ^ SyntaxError: invalid syntax

Given that this appears to be a debug statement I commented it out but that resulted in a subsequent error :

pi@raspberrypi:~/vuegraf-master/src $ python vuegraf.py vuegraf.json /usr/lib/python2.7/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.2) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Traceback (most recent call last): File "vuegraf.py", line 10, in <module> from pyemvue import PyEmVue File "/usr/local/lib/python2.7/dist-packages/pyemvue/__init__.py", line 2, in <module> from pyemvue.pyemvue import PyEmVue File "/usr/local/lib/python2.7/dist-packages/pyemvue/pyemvue.py", line 85 raise ValueError(f'Scale of {scale} is invalid, must be 1S, 1MIN, 15MIN, or 1H.') ^ SyntaxError: invalid syntax

I apologize as I suspect I'm doing something silly.
Best,
Ben

issues with smart plugs after API update 8/31/21

FYI, I am getting the following error parsing devices:
Fatal error: (<class 'TypeError'>, TypeError('Parser must be a string or character stream, not NoneType')

I updated PyEmVue already with their changes and am pretty sure it is related to the 2 smart plugs i have and this issue with PyEmVue - magico13/PyEmVue#30

Just in case anyone else is seeing that from yesterdays changes.

PLEASE READ: Use Discussions when requesting support

If you are looking for support, please open a new discussion topic rather than creating a new issue. Others may find they have a similar question and will be able to review all discussion topics, whereas issues get closed and hidden from view. Open a new discussion topic here: https://github.com/jertel/vuegraf/discussions

If you are certain you have found a problem with the project, you are welcome to submit a pull request with the desired code change. PRs can be submitted here: https://github.com/jertel/vuegraf/pulls

If you have a problem with the project but do not have the ability to make the change needed to correct it, please include enough information in this issue so that a developer can reproduce the problem. This probably will mean a log snippet is required, as well as the Vue device configuration. You may need to take a mobile screenshot for this.

[Feature Request] Pull historical data on first run

As a user of vuegraf, I want to have the option to pull all historical data to be able to analyze past datetime.now()

User friendly names

Hello,
Thank a bunch for the project, I love Grafana and got the Vue2 yesterday, this is AWESOME! Quick question, I seem to be having issues adding the name that the circuit is named.

As an example, my circuit 5 name in the Vue app named "AC Downstairs", the Circuit Type is "Air Conditioner". When the vuegraf starts in docker in the log it outputs "Discovered new channel: AC Downstairs (5)". In the config I've tried setting the "Left" and the "Right" 5th device down to both the "AC Downstairs" and ""Air Conditioner" but with no luck as it always shows up as "Home-5". "Home" being the Name of the Vue monitor.

Thanks for any help you can provide.

Joe

usageDataPoints timestamps from Emporia

Need to pull the timestamps from the returned values from Emporia and not just assume it is the start and stop times.

Detailed Data Issue

First off, thanks for creating this great project. I am running the latest version 1.3.2 and noticed in my logs, that when it tries to pull the detailed data every hour, it gets the following error every time:

today at 6:42:55 AM 2021-08-25 10:42:55.292851 | ERROR | Failed to record new usage data: (<class 'TypeError'>, TypeError("unsupported operand type(s) for *: 'float' and 'NoneType'"), <traceback object at 0x7f8bfe5eecc0>)
today at 6:42:55 AM Traceback (most recent call last):
today at 6:42:55 AM File "/opt/vuegraf/vuegraf.py", line 219, in
today at 6:42:55 AM extractDataPoints(device, usageDataPoints)
today at 6:42:55 AM File "/opt/vuegraf/vuegraf.py", line 133, in extractDataPoints
today at 6:42:55 AM watts = float(secondsInAMinute * minutesInAnHour * wattsInAKw) * kwhUsage
today at 6:42:55 AM TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'

Minimal Configuration doc

In the doc for Minimal Configuration the "password" has a trailing comma that if used in .json, will cause container startup to fail.
{
"influxDb": {
"host": "my.influxdb.hostname",
"port": 8086,
"user": "root",
"pass": "root",
"database": "vue",
"reset": false
},
"accounts": [
{
"name": "Primary Residence",
"email": "[email protected]",
"password": "my-emporia-password",
}
]
}

Is it possible to create a Virtual datapoint for 'Balance'

I'd like to have a Virtual Datapoint 'Unmonitored' or 'Balance' that is equal to '1,2,3' - Sum(all other datapoints) to account for breakers with no monitor on them.

detailedEnabled variable is not used

detailedEnabled variable is not passed to extractDataPoints method

Doesn't work with HTTPS

It appears it trying to use HTTP when I have port 443 used which should be for HTTPS. Thanks.

Feature Request: Make InfluxDB credentials optional

Would like to know if you would be against making the influx DB credentials optional so unauthenticated inserts could be done?

Currently, the application errors when no user or password is giving in the JSON file.

Traceback (most recent call last):

  File "/opt/vuegraf/vuegraf.py", line 30, in <module>

    influx = InfluxDBClient(config['influxDb']['host'], config['influxDb']['port'], config['influxDb']['user'], config['influxDb']['pass'], config['influxDb']['database'])

KeyError: 'user'

Smart Plug data not submitting to Influx

I'm unable to get the service to submit data into the InfluxDB. I do not see any errors other than the Devices appear to be reporting as channels?

The service status reports it found new channels:

Feb 25 12:35:48 zbox python3[171297]: 2021-02-25 20:35:48.206697 | INFO  | Discovered new channel: Home (1,2,3)
Feb 25 12:35:48 zbox python3[171297]: 2021-02-25 20:35:48.519672 | INFO  | Discovered new channel: Living Room (1,2,3)
Feb 25 12:35:48 zbox python3[171297]: 2021-02-25 20:35:48.844572 | INFO  | Discovered new channel: PC (1,2,3)
Feb 25 12:35:49 zbox python3[171297]: 2021-02-25 20:35:49.039635 | INFO  | Discovered new channel: Server (1,2,3)
Feb 25 12:35:49 zbox python3[171297]: 2021-02-25 20:35:49.495013 | INFO  | Submitted datapoints to database; account="Home"; points=16
Feb 25 12:36:49 zbox python3[171297]: 2021-02-25 20:36:49.946429 | INFO  | Submitted datapoints to database; account="Home"; points=40
Feb 25 12:37:50 zbox python3[171297]: 2021-02-25 20:37:50.442709 | INFO  | Submitted datapoints to database; account="Home"; points=59
Feb 25 12:38:50 zbox python3[171297]: 2021-02-25 20:38:50.938024 | INFO  | Submitted datapoints to database; account="Home"; points=61
Feb 25 12:39:51 zbox python3[171297]: 2021-02-25 20:39:51.360788 | INFO  | Submitted datapoints to database; account="Home"; points=58
Feb 25 12:40:51 zbox python3[171297]: 2021-02-25 20:40:51.874593 | INFO  | Submitted datapoints to database; account="Home"; points=61

Config with secrets removed.

{
   "influxDb":{
      "host":"localhost",
      "port":8086,
      "user":"MyUser",
      "pass":"MyPassword",
      "database":"MyDatabase",
      "reset":false
   },
   "accounts":[
      {
         "name":"Home",
         "email":"MyEmail",
         "password":"MyPassword",
         "devices":[
            {
               "name":"PC"
            },
            {
               "name":"Server"
            },
            {
               "name":"Living Room"
            }
         ]
      }
   ]
}

Already Exsisiting Database

Can't use with database that already exists. I commented the createdatabase function and it worked I'm not very good at python so I'm not sure how to do the if/else logic from the json config.

InfluxDB login details don't allow for tokens (required for InfluxDB v2)

Was trying to use this project for the first time, but being gated by how the the influxdb library is authenticating with an influx2 db.
I'm ~80% sure that it now requires a token to connect (could be wrong), and I believe the latest versions of the influx python library have been updated to allow you to pass a token in.

Devices are not named correctly per JSON

Despite using a Devices section in the JSON which includes a name that is copied directly from the Vue app in the order they were added (JSON below) I do not get device names in the graphs, only a number as seen in the screen shot. I am sure this is me, but not sure what I did or how to fix it.

"devices": [
            {
                "name": "Right Panel",
                "channels": [
                    "Dryer Leg 1",
                    "Dryer Leg 2",
                    "Microwave",
                    "Fridge",
                    "Package Unit Leg 1",
                    "Package Unit Leg 1",
                    "2nd Fl Condenser Leg 1",
                    "2nd Fl Condenser Leg 2",
		"Washing Machine",
		"Upstairs Condenser Leg 1",
		"Upstairs Condenser Leg 2"
                ]
            },
            {
                "name": "Left Panel",
                "channels": [
                    "2nd Fl Heat Pump Leg 1",
                    "2nd Fl Heat Pump Leg 2",
                    "Downstairs Hot Water",
                    "Stove Leg 1",
                    "Stove Leg 2",
                    "Oven Leg 1",
                    "Oven Leg 2",
                    "Office Plugs",
		"Dishwasher",
		"Workshop Panel Leg 1",
		"Workshop Panel Leg 2",
		"Garage Fridge Freezer"

checkpointing data extraction + more

Hi,

As mentioned in the discussions, I am interested in a "batch" feature that would allow vuegraf to be run asynchronously without losing data or requiring user logic with --history to catch up. To accomplish this I have created a --batch command line option which does that on a per channel and per time precision basis (for each channel, example GID 123456-1, there is a separate checkpoint for Day, Hour, Minute). It knows the maximum data retention on the API and also the maximum request windows and will massage any batch requests as needed. I would also like to suggest some additional improvements (my opinion) like a "detailedDatapoint" for influxdb which has the channel type identifier (the label of circuit type) to enable more automated graphing of circuits, channel multiplier usage, etc, as well as some smaller modifications like gzipping the influx writes (better network performance and keep from influxdb write failures which i encountered using the --history CLI option) as well as using second precision on writes instead of the default nanosecond as there is no need (this is recommend by influx for better db performance and compression). Other small changes which we could discuss which I believe prevents any duplicate data between listUsage and chartUsage. All these things are done by the batch process I have created so would be easy to apply to the listUsage where desired (see below). I think the --history function could be effectively dropped as --batch is much more powerful while essentially doing the same thing (you can give it a CLI arg to override the checkpoints / initially seed the db by going back X days)

the listUsage normal operation could easily be combined with the checkpointing so that if you stop running vuegraf for sometime and then start it up with the --batch option it will catchup and then drop into listUsage collection again. I think this could be universally useful for data loss prevention and probably solves whatever problems were in the other issue about data loss as I believe the main cause of this is that the pyemvue library / emporia API sometimes only returns devices and not devices and their channels which will cause small or large data loss depending on the precision / function being executed (why I found per device and per precision checkpointing to be necessary).

I'm still testing but wondered how we can best collaborate? As I mentioned I'm no developer so I'm not familiar with practical details of modern development on commits. I guess you need to give me access to do a pull request to a development branch/tag? I just have one giant commit on a new feature branch -- some help on this part would be appreciated if you are interested in pulling the code.

Add Detailed Tag to InfluxDB v2 output

I noticed when I switched to using InfluxDB2 for Vuegraf, that the detailed tag was omitted. I am not sure how to do a PR, but I edited my local file and added: .tag("detailed", detailed) \

That seemed to add the true/false tag to the data points.

def createDataPoint(account, chanName, watts, timestamp, detailed):
dataPoint = None
if influxVersion == 2:
dataPoint = influxdb_client.Point("energy_usage")
.tag("account_name", account['name'])
.tag("device_name", chanName)
.tag("detailed", detailed)
.field("usage", watts)
.time(time=timestamp)

No data after updating to 1.7.2

Hello, I'm currently running Vuegraf via Docker. Today I updated to 1.7.2 and am no longer getting any data to Influx. I've confirmed that live data is in my Emporia account via the app. Looking at Docker logs I see this repeating:

Attempt 1 to get device usage
Bad data for device 204305 channel 1,2,3 at 2024-02-28 19:59:23+00:00
Waiting 2 seconds before retrying device usage request.

Attempt 2 to get device usage
Bad data for device 204305 channel 1,2,3 at 2024-02-28 19:59:23+00:00
Waiting 4 seconds before retrying device usage request.

Attempt 3 to get device usage
Bad data for device 204305 channel 1,2,3 at 2024-02-28 19:59:23+00:00
2024-02-28 19:59:40.276100+00:00 | INFO  | Submitting datapoints to database; account="Home"; points=0

What information can I provide or look at to help troubleshoot? Thanks!

latest image throwing 403 Client Error: Forbidden for url: https://api.emporiaenergy.com/cu...

I am seeing the following error when pulling jertel/vuegraf:latest image (updated 5 months ago) . rolling back to jertel/vuegraf:1.3.4 I don't have the error anymore.

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.emporiaenergy.com/customers/12134/devices?detailed=true&hierarchy=true
2022-07-06 16:00:31.457138 | INFO  | Login completed
Traceback (most recent call last):
  File "/opt/vuegraf/vuegraf.py", line 115, in <module>
    populateDevices(account)
  File "/opt/vuegraf/vuegraf.py", line 55, in populateDevices
    devices = account['vue'].get_devices()
  File "/usr/local/lib/python3.9/site-packages/pyemvue/pyemvue.py", line 39, in get_devices
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

influx requires minimum 8 characters for password, writeup say to use 'vuegraf', but this is < 8 characters

I installed influxdb per the docker instructions provided. Upon setting up the initial user, I entered vuegraf fro user name, but when I entered vuegraf for the password, the screen said that "Password must be at least 8 characters". Please update accordingly.

Emporia connection or local container issue?

Anyone else receiving a similar issue in their vuegraf container? I have been trying to debug if this is an issue with connecting to Emporia or my InfluxDB container. Error viewed in Docker Desktop for Mac. Have been trying to create a docker-compose file to bring up all containers so no need to run the individual commands.

`Traceback (most recent call last):

File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen

httplib_response = self._make_request(

File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request

six.raise_from(e, None)

File "", line 3, in raise_from

File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request

httplib_response = conn.getresponse()

File "/usr/local/lib/python3.9/http/client.py", line 1347, in getresponse

response.begin()

File "/usr/local/lib/python3.9/http/client.py", line 307, in begin

version, status, reason = self._read_status()

File "/usr/local/lib/python3.9/http/client.py", line 276, in _read_status

raise RemoteDisconnected("Remote end closed connection without"

http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send

resp = conn.urlopen(

File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen

retries = retries.increment(

File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 531, in increment

raise six.reraise(type(error), error, _stacktrace)

File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 734, in reraise

raise value.with_traceback(tb)

File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen

httplib_response = self._make_request(

File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request

six.raise_from(e, None)

File "", line 3, in raise_from

File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request

httplib_response = conn.getresponse()

File "/usr/local/lib/python3.9/http/client.py", line 1347, in getresponse

response.begin()

File "/usr/local/lib/python3.9/http/client.py", line 307, in begin

version, status, reason = self._read_status()

File "/usr/local/lib/python3.9/http/client.py", line 276, in _read_status

raise RemoteDisconnected("Remote end closed connection without"

urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/opt/vuegraf/vuegraf.py", line 28, in

influx.create_database(config['influxDb']['database'])

File "/usr/local/lib/python3.9/site-packages/influxdb/client.py", line 746, in create_database

self.query("CREATE DATABASE {0}".format(quote_ident(dbname)),

File "/usr/local/lib/python3.9/site-packages/influxdb/client.py", line 521, in query

response = self.request(

File "/usr/local/lib/python3.9/site-packages/influxdb/client.py", line 332, in request

response = self._session.request(

File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request

resp = self.send(prep, **send_kwargs)

File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send

r = adapter.send(request, **kwargs)

File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 498, in send

raise ConnectionError(err, request=request)

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))`

PyEmVue updated -- get_usage_over_time no longer supported?

get_usage_over_time method seems to be removed in PyEmVue 0.12.2. This seems to be the reason I get a HTTP 403, perhaps?