Filtering out extreme values

[klaasnicolaas]

Filtering out extreme values

[JamesSwift]

I'm running on a D1 mini (esp8266) and also getting crazy values at random times.

I wanted to add some thoughts on the above ideas. The internal_filter is handy, but it's essentially a low pass filter. Any crazy value BELOW the internal_filter will still get through. So you still might see a 24,000w crazy value next to a 100w known good value.

The advantage of the lambda filter though, is that you are looking for the DIFFERENCE between the last known good reading and the current reading. However, you still seem to be configuring it based on the maximum possible load of your board, Surely MAX_DIFFERENCE shouldn't be the maximum power draw of your house, but somewhere near the biggest possible load added by a single appliance. For instance, in my house the electric shower is about 40a (around 10kw). The chances of someone turning the shower and another appliance on in the exact same instant is basically zero. So I think a saner value for MAX_DIFFERENCE would be 11000.0.

[mark-10-5]

I found out by asking on the ESPHome discord that filter values are processed in order of entry in YAML and you can have as many as you like. So i ended up testing it like this last night.

  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 50000.0;  // adjust this!
      static float last_value = NAN;
      if (isnan(last_value) || std::abs(x - last_value) < MAX_DIFFERENCE)
        return last_value = x;
      else
        return {};

I ran the pulse rate calc first as the values i was working with in the graphs in home assistant are after the calculation so I wanted to filter it after the calculation. This worked pretty well, i didn't get any 1 million W readings but i did get 1 96000W reading. So i think your suggestion of the new code is a good one so i am going to test this.

  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 30000.0;  // adjust this!
      static float last_value_t = NAN;
      static int count_missed_t = 0;
      if (count_missed_t == 3) last_value_t = x;
      if (isnan(last_value_t) || std::abs(x - last_value_t) < MAX_DIFFERENCE) {
        count_missed_t = 0;
        return last_value_t = x;
      } else {
        count_missed_t += 1;
        return last_value_t;
      }

I am going with 30000W now as my panel is 125Amp max which is 24000W max draw so anything over 30000W should be well safe as being false.

Thanks for the suggestions and i will report back if this works.

[onkelbeh]

Had the same problem, with rare very high spikes (>8kW), solved it with a filter:

sensor:
  - platform: pulse_meter
    name: '${name} - Power consumption'
    unit_of_measurement: 'W'
    state_class: measurement
    device_class: power
    icon: mdi:flash-outline
    accuracy_decimals: 0
    internal_filter: 50ms

I did not use an ESP32, have a NodeMCU12S.
Comparison with the values shown directly on the meter matches.

[lemppari]

Makes no sense to me. You could try to add some logging or take this up with the ESPHome people.

[Brink3]

Makes no sense to me. You could try to add some logging or take this up with the ESPHome people.

Dude, I'm so friggin' stupid. I never changed value of MAX_DIFFERENCE. Changed it now to 40000.0, no spikes so far.

[Helmasynti]

I had spikes, lots of spikes. Activating the internal filter cut off most insane ones but there was still a lot of noise left. The filter based on difference still improved the situation especially if MAX_DIFFERENCE was set low enough but even this did not solve the problem entirely. I could still see small spikes appearing. The solution was to decrease the sensitivity of the photodiode by turning the potentiometer on the board anti-clockwise. This removed all spikes and I was able to deactivate the filters. The signal is clean now.

If you have noise in the signal from the photodiode, it may be an easy solution to adjust its sensitivity. This can be done especially if you use the given 3D-case, since it blocks very effectively all the interfering light that might come from outside. So the sensitivity can be rather low.

[louisr0608]

Hey, just adding my two cents here with another filtering method. The filter I put together is similar to those suggested above, but inverted in a way so that 'large' (1kw) changes have to be confirmed for some number of readings before being accepted. The code is adjustable for how many frames of confirmation you want to have (num_samples). This does skew the readings very slightly, but generally evens out over time -- i.e., if you go up 1500w, it won't register for a few frames, but then when it goes back down the drop won't register for the same number of frames. Its not perfect but it solved all the wacky readings and everything been consistent and reasonably accurate since I added this.

A quick preliminary note, if your sensor has an adjustable sensitivity, make it as sensitive as you can. This filter is also combined with the following settings:

internal_filter_mode: "EDGE"
internal_filter: 35ms

You'll want to adjust internal_filter to be compatible with your meters blink rate (i.e, divide the value by 4 if your meter blinks 4000 x per kw). 35ms works well for mine which is 1k blinks per kw.

Adjust MAX_DIF and num_samples to a desired value to force consecutive readings to register, as described above.

filters:
      - lambda: |-
          const uint num_samples = 3;
          float val = x * ((60.0 / ${pulse_rate}) * 1000.0);
          float ret = val;
          const float MAX_DIF = 1000.0;

          static float ret_z0 = NAN;
          static float samples[num_samples] = {NAN};
          bool inrange;

          if (samples[0] == NAN)
          {
            ret_z0 = val;
            for (int i = 0; i < num_samples; i++)
            {
              samples[i] = val;
            }
          }

          inrange = true;
          for (int i = 0; i < num_samples; i++)
          {
            if ((std::abs(val - samples[i])) > MAX_DIF) inrange = false;
          }

          if (!inrange) ret = ret_z0;

          ret_z0 = ret;
          for (int i = num_samples - 1; i > 0; i--)
          {
            samples[i] = samples[i-1];
          }
          samples[0] = val;

          return ret;

Hope this helps someone!

Regards, Louis

[Perka82]

Changing filer mode to pulse and lower the internal filer to 10 ms solved all my problems. No spikes during the last 72 h.

internal_filter_mode: PULSE
internal_filter: 10ms

filters:
  # multiply value = (60 / imp value) * 1000
  # - multiply: 60
  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);

[onkelbeh]

Had the similar problems, also solved it with an internal filter,

sensor:
  - platform: pulse_meter
    name: '${name} - Power consumption'
    unit_of_measurement: 'W'
    state_class: measurement
    device_class: power
    icon: mdi:flash-outline
    accuracy_decimals: 0
    #count_mode:
    #  falling_edge: INCREASE
    #  rising_edge: DISABLE
    internal_filter: 50ms
    pin:
      number: D6
      mode: INPUT
      inverted: true

I have seen no spikes anymore since Nov 21. My measurement is very close to the meter.

[martinkantola]

Had the same issue with random extreme spikes. Traced down the issue to the photodiode board itself. The comparator circuit lacks a hysteresis resistor/capacitor. Adding a 33kOhm resistor and 10nF capacitor between pins 5 and 7 of the comparator chip worked for me. Used 1206 size components and soldered them on the adjacent resistors. Also removed the capacitor connected to pin 5 because it was too large for my meter which uses 10.000 pulses/kWh instead of 1000.

[klaasnicolaas]

This weekend I plan to update the repository, if using GPIO26 is really better then I will be happy to adopt that advice in the V4.0.0 release.

Which GPIO should people best use with a Wemos D1 or ESP8266?

[sparkyws]

The Wemos D1 and the ESP8266 shouldn’t be an issue as they don’t have touch pins

[sparkyws]

This weekend I plan to update the repository, if using GPIO26 is really better then I will be happy to adopt that advice in the V4.0.0 release.

I've been running V4.0.0 for 3 months on GPIO26 using an ESP32s with no issues

[klaasnicolaas]

FYI: I'm currently working on a refactor of the project where the default pin for the ESP32 boards will from now on be GPIO26.

[lemppari]

You could try this (I haven't tested this...)

        - lambda: |-
            float new_value = x * ((60.0 / ${pulse_rate}) * 1000.0);
            float MAX_DIFFERENCE = 50000.0.0;  // adjust this!
            static float last_value = NAN;
            if (isnan(last_value) || std::abs(new_value - last_value) < MAX_DIFFERENCE)
              last_value = new_value;
            return last_value;

The problem with this is that if the first value is a noise value then the following sane values will be ignored. I think I found the source of your filter example and there is an updated version in that page which should handle that situation a little better. Here is a modified code for that

        - lambda: |-
            float new_value = x * ((60.0 / ${pulse_rate}) * 1000.0);
            float MAX_DIFFERENCE = 50000.0;  // adjust this!
            static float last_value_t = NAN;
            static int count_missed_t = 0;
            if (count_missed_t == 3) last_value_t = new_value;
            if (isnan(last_value_t) || std::abs(new_value - last_value_t) < MAX_DIFFERENCE) {
              count_missed_t = 0;
              last_value_t = new_value;
            } else {
              count_missed_t += 1;
            }
            return last_value_t;

This will handle the first noise value problem. It will revert to new "range of values" after getting three out of range values in a row. Let us know if either of these work.

EDITED: First version compared the values before applying the pulse calculation. Should be fixed now.

[lemppari]

Sorry, I made the comparing to previous values before doing the pulse rate calculation. I modified my code also. Please report your progres. :)

[mark-10-5]

I still get the odd value with the max difference code. I will add in the internal_filter also.

It has been much better though. I am still trying to pin point where the noise is coming from but not having much luck.

[Brink3]

I also get these spikes, tried your filters but still gets those spkies time to time.
Screenshot from the original code:

.

[lemppari]

That is very strange. Do you get only single spikes of crazy values or are there multiple crazy values in row?

If the code from my last example works, you should get four crazy values in a row. Here is a sample what should happen. First row are the new-values calculated in the first line of the code (s prefix is sane value and c is a crazy value). Second row is the output of the filter. Third row is the value of the count_missed_t variable after each step.

s1, s2, c3, s4, s5, c6, c7, c8, c9, s10, s11, s12, s13, s14
s1, s2, s2, s4, s5, s5, s5, s5, c9,  c9,  c9,  c9, s13, s14
0,   0,  1,  0,  0,  1,  2,  3,  0,   1,   2,   3,   0,   0

As we can see there needs to be four crazy values in a row to produce crazy values. And if that happens, even after sane values start coming right after those, the output will be four crazy values in a row before sanes values are the "new normal" again.

[Brink3]

There is just one single crazy spike. I have attached a log from the esphome logging.
Line 4 in this log shows a spike.

[19:21:06][D][light:035]: 'Red' Setting:
[19:21:06][D][light:046]:   State: ON
[19:21:06][D][sensor:121]: 'House - Total energy': Sending state 8.43400 kWh with 3 decimals of accuracy
[19:21:06][D][sensor:121]: 'House - Power consumption': Sending state 45569.61719 W with 0 decimals of accuracy
[19:21:06][D][light:035]: 'Red' Setting:
[19:21:06][D][sensor:121]: 'House - Total energy': Sending state 8.43500 kWh with 3 decimals of accuracy
[19:21:06][D][light:035]: 'Red' Setting:
[19:21:06][D][light:046]:   State: OFF
[19:21:06][D][light:035]: 'Red' Setting:
[19:21:09][D][sensor:121]: 'House - Power consumption': Sending state 1107.69226 W with 0 decimals of accuracy
[19:21:09][D][light:035]: 'Red' Setting:
[19:21:09][D][light:046]:   State: ON
[19:21:09][D][sensor:121]: 'House - Total energy': Sending state 8.43600 kWh with 3 decimals of accuracy
[19:21:10][D][light:035]: 'Red' Setting:
[19:21:10][D][light:046]:   State: OFF
[19:21:12][D][sensor:121]: 'House - Power consumption': Sending state 1116.97168 W with 0 decimals of accuracy
[19:21:12][D][light:035]: 'Red' Setting:
[19:21:12][D][light:046]:   State: ON
[19:21:12][D][sensor:121]: 'House - Total energy': Sending state 8.43700 kWh with 3 decimals of accuracy
[19:21:13][D][light:035]: 'Red' Setting:
[19:21:13][D][light:046]:   State: OFF

[mark-10-5]

Did you leave it at 50000.0? I think it works better if you calculate the max wattage of your power board and set it to that.

I am finding the internal_filter is working better then the filter code.

[Brink3]

Did you leave it at 50000.0? I think it works better if you calculate the max wattage of your power board and set it to that.

I am finding the internal_filter is working better then the filter code.

Yes, I had it at 50000.0 ;)

Tried the internal_filter @ 50ms and 100ms, but I got same spikes.

[mark-10-5]

Interesting your spikes are always around 40,000, the internal_filter wouldn't catch that as it is not extreme enough. My spikes were much higher, around 100,000+ up to 1,000,000 often and a much wider value range, which the internal filter would catch as it is so extreme so the pulse would be very quick so be filtered out.

40,000 although a high value is a possible value so you will need the max difference calc to filter that one. Glad it seems to be working so far.

[JamesSwift]

I've managed to solve my crazy values problem!

Firstly, the filter as described above with a value of 11,000.0 worked great at removing the truly crazy values. However now I wasn't distracted by the 6Mw 🤣 pulses, I was still getting random spikes around 1.5kw - 6kw over my base load around 200w.

When you do the math that's a rogue pulse of 2.4s in the midst of a pulse rate of around 18s. That blew my theory that it was just noise in the PWM of the meter's LED. That's a distinct pulse unconnected with the meter's LED. My light detector is sealed off from external light, so what could it possibly be...

Then it struck me! The cable connecting the light detector to the esp8266 ran right next to the radio communications module of the smart meter. It must be an RF induced pulse in the cable! So, I rerouted the cable to be maybe 5cm away and overnight I haven't seen any rogue pulses. It's early days in testing, but I think that was the problem all along. If I still get any interference I'll try adding a 100uf capacitor across the ground and digital output pins as suggested in another issue, which should soak it up.

Just thought it was worth documenting this issue with RF interference here as I'm sure others will encounter it with their smart meter in close proximity,

[bictorv]

    internal_filter: 50ms

For me, an internal_filter: 100ms was needed but seems to work very well - no more spikes of hundreds of kilowatt.

[EspeeFunsail]

I found out by asking on the ESPHome discord that filter values are processed in order of entry in YAML and you can have as many as you like. So i ended up testing it like this last night.

  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 50000.0;  // adjust this!
      static float last_value = NAN;
      if (isnan(last_value) || std::abs(x - last_value) < MAX_DIFFERENCE)
        return last_value = x;
      else
        return {};

I ran the pulse rate calc first as the values i was working with in the graphs in home assistant are after the calculation so I wanted to filter it after the calculation. This worked pretty well, i didn't get any 1 million W readings but i did get 1 96000W reading. So i think your suggestion of the new code is a good one so i am going to test this.

  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 30000.0;  // adjust this!
      static float last_value_t = NAN;
      static int count_missed_t = 0;
      if (count_missed_t == 3) last_value_t = x;
      if (isnan(last_value_t) || std::abs(x - last_value_t) < MAX_DIFFERENCE) {
        count_missed_t = 0;
        return last_value_t = x;
      } else {
        count_missed_t += 1;
        return last_value_t;
      }

I am going with 30000W now as my panel is 125Amp max which is 24000W max draw so anything over 30000W should be well safe as being false.

Thanks for the suggestions and i will report back if this works.

I tried using this in my own glow.yaml, but it seems like some syntax is wrong? Getting this error:
ERROR Error while reading config: Invalid YAML syntax:

while scanning a simple key
in "/config/esphome/home-assistant-glow.yaml", line 130, column 13:
float new_value = x * ((60.0 / $ ...
^
could not find expected ':'
in "/config/esphome/home-assistant-glow.yaml", line 131, column 13:
float MAX_DIFFERENCE = 4100.0;
^

Anyone able to assist?

[EspeeFunsail]

I found out by asking on the ESPHome discord that filter values are processed in order of entry in YAML and you can have as many as you like. So i ended up testing it like this last night.

  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 50000.0;  // adjust this!
      static float last_value = NAN;
      if (isnan(last_value) || std::abs(x - last_value) < MAX_DIFFERENCE)
        return last_value = x;
      else
        return {};

I ran the pulse rate calc first as the values i was working with in the graphs in home assistant are after the calculation so I wanted to filter it after the calculation. This worked pretty well, i didn't get any 1 million W readings but i did get 1 96000W reading. So i think your suggestion of the new code is a good one so i am going to test this.

  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 30000.0;  // adjust this!
      static float last_value_t = NAN;
      static int count_missed_t = 0;
      if (count_missed_t == 3) last_value_t = x;
      if (isnan(last_value_t) || std::abs(x - last_value_t) < MAX_DIFFERENCE) {
        count_missed_t = 0;
        return last_value_t = x;
      } else {
        count_missed_t += 1;
        return last_value_t;
      }

I am going with 30000W now as my panel is 125Amp max which is 24000W max draw so anything over 30000W should be well safe as being false.
Thanks for the suggestions and i will report back if this works.

I tried using this in my own glow.yaml, but it seems like some syntax is wrong? Getting this error: ERROR Error while reading config: Invalid YAML syntax:

while scanning a simple key in "/config/esphome/home-assistant-glow.yaml", line 130, column 13: float new_value = x * ((60.0 / $ ... ^ could not find expected ':' in "/config/esphome/home-assistant-glow.yaml", line 131, column 13: float MAX_DIFFERENCE = 4100.0; ^

Anyone able to assist?

I added some spaces to the stuff below -lamda: and it started to compile, but now I get this:

/data/cache/platformio/packages/framework-arduinoespressif8266/cores/esp8266/Arduino.h:135:16: error: expected unqualified-id before '(' token
#define abs(x) ((x)>0?(x):-(x))
^
/config/esphome/home-assistant-glow.yaml:130:177: note: in expansion of macro 'abs'
float MAX_DIFFERENCE = 4100.0;
^
/config/esphome/home-assistant-glow.yaml:131:3: warning: control reaches end of non-void function [-Wreturn-type]
static float last_value_t = NAN;

[EspeeFunsail]

I had spikes, lots of spikes. Activating the internal filter cut off most insane ones but there was still a lot of noise left. The filter based on difference still improved the situation especially if MAX_DIFFERENCE was set low enough but even this did not solve the problem entirely. I could still see small spikes appearing. The solution was to decrease the sensitivity of the photodiode by turning the potentiometer on the board anti-clockwise. This removed all spikes and I was able to deactivate the filters. The signal is clean now.

If you have noise in the signal from the photodiode, it may be an easy solution to adjust its sensitivity. This can be done especially if you use the given 3D-case, since it blocks very effectively all the interfering light that might come from outside. So the sensitivity can be rather low.

I think this is the issue I'm also having. I will adjust the potentiometer when I get home. The software filters only reduce the values of the spikes, but doesn't actually remove the spikes themselves. I think mine is also just too sensitive.

[mcchots]

I added some spaces to the stuff below -lamda: and it started to compile, but now I get this:

/data/cache/platformio/packages/framework-arduinoespressif8266/cores/esp8266/Arduino.h:135:16: error: expected unqualified-id before '(' token #define abs(x) ((x)>0?(x):-(x))
/config/esphome/home-assistant-glow.yaml:130:177: note: in expansion of macro 'abs' float MAX_DIFFERENCE = 4100.0;
/config/esphome/home-assistant-glow.yaml:131:3: warning: control reaches end of non-void function [-Wreturn-type] static float last_value_t = NAN;

I solved this error by changing the functionstd::abs to just abs

[matthewharrington]

Hi all. I was having the same issue and came here for assistance. I have tried all sorts of things including internal filter and the extreme value calculator mentioned above.

I was using GPIO12 and have just moved to GPIO26 and all my extreme values have completely disappeared. This is the only thing that worked for me.

Cheers, Matt

[martinkantola]

[ABredhauer]

I'm also seeing this issue with spikes of 400,000 randomly.

I can see that people are solving this by adding internal filters but I can see different types being used and I'm very new to ESP32 so can someone help me with which is the best approach?

[jpaulickcz]

So by default I was getting a lot of crazy spikes (8 000 000 W), however the basic internal_filter: 100ms seems to have sorted 95 % percent of the obviously incorrect readings (after a week of observing), the remaining spikes are now only up to a 10 000-15 000 W. I might play with the more elaborate filter posted by louisr0608 earlier.

I have 500 imp flashes per kWh. I use Wemos D1 Mini with the sensor soldered onto 10 m CAT5e cable, so that might play a role as well.

[jessereynolds]

I also had a lot of these crazy spikes going way over what's possible on a single phase residential service (eg 80,000 watts). I have removed the impossible readings with internal_filter: 78ms (theoretical max rate of 60 Amp, 14.4 kW @ 240v, 12.8 pulses per second on my 3200 imp/kWh meter, 78 ms between start of each pulse) and adding a median moving window filter which selects the middle value of the last 3 samples:

    filters:
      - lambda: return x * ((60.0 / id(select_pulse_rate).state) * 1000.0);
      # remove crazy outliers (fingers crossed)
      - median:
          window_size: 3
          send_every: 1
          send_first_at: 1

Going well so far, but I might need to increase the window size, we'll see.

[markusg1234]

Hi,
I was having issues if I opened the power switch board and let light shine in as it would cause a huge spike.
Also some times a spike might occur randomly but not very often.
I am trying a clamp filter and a filter_out and so far so good. I set the max value to 10,000 Watts.

platform: pulse_meter
name: '${friendly_name} - Power Consumption'
id: sensor_energy_pulse_meter
unit_of_measurement: 'W'
state_class: measurement
device_class: power
icon: mdi:flash-outline
accuracy_decimals: 0
pin: ${pulse_pin}
internal_filter_mode: PULSE
internal_filter: 10ms

filters:
  - clamp:
     max_value: 10000
  - filter_out: 10000   
  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);

[sparkyws]

Hi all. I was having the same issue and came here for assistance. I have tried all sorts of things including internal filter and the extreme value calculator mentioned above.

I was using GPIO12 and have just moved to GPIO26 and all my extreme values have completely disappeared. This is the only thing that worked for me.

Cheers, Matt

I raised this issue some time ago, the problem with GPIO12/13 is they are touch pins and the wiring will sense this and give incorrect readings, I changed to 26 and it's been working perfectly since

#300

[sparkyws]

Will depend on the board, but most ESP32's, GPIO13 seem to be touch pins https://lastminuteengineers.com/esp32-pinout-reference/

[xsaMNJfkltrwq5r3]

Before using the software filters it is important to try to get your inputs free of external noise. Try twisting your sensor cables between the sensor and the controller. Then use your software filter to step on anything that gets through. I had the same issue and for me twisting the wires and using a relatively large value on my internal filter time (but less than a realistic pulse period) solved my issue. Recently though I made some changes and I now have the spikes back again, which are occuring at the same time my aircons turn on/off, due to the large surges that would occur at these times. I have to go back and get things back to normal again it will be fine.

[bgmn]

Could internal_filter be added to the settings panel just like the Puls rate - imp/kW is now added? That way you could change the internal filter value and still get the updates from github.

[klaasnicolaas]

Please create a new issue for that.

./Klaas