robtillaart/runningaverage

Arduino library to calculate the running average by means of a circular buffer.

RunningAverage

Arduino library to calculate the running average by means of a circular buffer.

Description

The RunningAverage object gives a running average of the last N floating point numbers, giving them all equal weight. This is done by adding new data to an internal circular buffer, removing the oldest and replace it by the newest. The size of the internal buffer can be set in the constructor.

By keeping track of the _sum the runningAverage can be calculated fast (only 1 division) at any time. This is done with getFastAverage(). However the constant adding and subtracting when adding new elements to the RA object possibly introduces an ever increasing error. In tests adding up to 1500000 numbers this error was always small. But that is no proof. In version 0.2.16 a fix was added that uses the calculation of the sum in getAverage() to update the internal _sum.

Related

• https://github.com/RobTillaart/Correlation
• https://github.com/RobTillaart/GST - Golden standard test metrics
• https://github.com/RobTillaart/Histogram
• https://github.com/RobTillaart/RunningAngle
• https://github.com/RobTillaart/RunningAverage
• https://github.com/RobTillaart/RunningMedian
• https://github.com/RobTillaart/statHelpers - combinations & permutations
• https://github.com/RobTillaart/Statistic

Interface

#include "RunningAverage.h"

Constructor

• RunningAverage(uint16_t size) allocates dynamic memory, one float (4 bytes) per element. No default size (yet).
• ~RunningAverage() destructor to free the memory allocated.

Basic

• void clear() empties the internal buffer.
• void add(float value) wrapper for addValue()
• void addValue(float value) adds a new value to the object, if the internal buffer is full, the oldest element is removed.
• void fillValue(float value, uint16_t number) adds number elements of value. Good for initializing the system to a certain starting average.
• float getValue(uint16_t position) returns the value at position from the additions. Position 0 is the first one to disappear.
• float getAverage() iterates over all elements to get the average, slower but accurate. Updates the variables used by getFastAverage() to improve its accuracy again.
• float getFastAverage() reuses previous calculated values, therefore faster. Accuracy can drift.

Extended functions

• float getStandardDeviation() returns the standard deviation of the current content. Needs more than one element to be calculable.
• float getStandardError() returns the standard error of the current content.
• float getMin() returns minimum since last clear, does not need to be in the buffer any more.
• float getMax() returns maximum since last clear, does not need to be in the buffer any more.
• float getMinInBuffer() returns minimum in the internal buffer.
• float getMaxInBuffer() returns maximum in the internal buffer.

Admin functions

• bool bufferIsFull() returns true if buffer is full.
• float getElement(uint16_t index) get element directly from internal buffer at index. (debug)
• uint16_t getSize() returns the size of the internal array.
• uint16_t getCount() returns the number of slots used of the internal array.

Partial functions

• void setPartial(uint16_t partial = 0) use only a part of the internal array. Allows to change the weight and history factor. 0 ==> use all == default.
• uint16_t getPartial() returns the set value for partial.

Last functions

These functions get the basic statistics of the last N added elements. Returns NAN if there are no elements and it will reduce count if there are less than count elements in the buffer.

• float getAverageLast(uint16_t count) get the average of the last count elements.
• float getMinInBufferLast(uint16_t count) get the minimum of the last count elements.
• float getMaxInBufferLast(uint16_t count) get the maximum of the last count elements.

These functions are useful in cases where you might want to calculate and display the statistics of a subset of the added elements. Reason might be to compare this with the numbers of the whole buffer to notice changes earlier. Otherwise one should create multiple RunningAverage objects each with its own length, effectively having multiple copies of the data added.

Note: if called with a value larger or equal to getCount() (including getSize()) as parameter, the functions will return the statistics of the whole buffer.

Subset (experimental)

• float getAverageSubset(uint16_t start, uint16_t count) Get the average of subset - count elements from start.

Operation

See examples

Future

Must

• update documentation, explain better

Should

• check for optimizations.
  • divide by count happens often ...
• clear(bool zero = true) to suppress setting all to 0. ?

Could

• create a double based derived class? Template class?
• add error handling (important?).
• investigate modus() most frequently occurring value.
  • difficult with floats ?
  • what to do when on two or more values are on par?

Wont

• default size for constructor
  • unknown what would be a good choice.
• clear(bool zero = true) to suppress setting all to 0. ?
  • makes addValue() slightly more complex
  • could introduce conflicts due to randomness data?

Support

If you appreciate my libraries, you can support the development and maintenance. Improve the quality of the libraries by providing issues and Pull Requests, or donate through PayPal or GitHub sponsors.

Thank you,

runningaverage's Issues

Last values could be expanded to subsection of the buffer

Instead of only providing statistics on the last n elements of the buffer, these functions could return info on a subset of the buffer.
Like:
float getAverageLast(uint16_t start, uint16_t count) get the average of count elements after start.

Making the functions overflow so that one value gives the last elements and two values gives this subset.
Would be useful to compare various regions in the buffer and also when doing a running average comparison between now and x amount of recordings ago. I now used to store multiple rolling buffers to do this.

I might have a crack at it later and do a pull request. But now have to finish my project and this way i do not forget the idea!

Investigate larger runs (16 bit) samples to overcome 255 limit

Some applications need to handle larger sets of values to define their running average.
Investigate how to handle 16 bit's sizes.

~RunningAverage() deconstructor to free the memory allocated?

A novice question on C++
Is it valid to both construct RunningAverage(size)
and deconstruct RunningAverage in the main loop,
ie to free up ram for another part of the main loop?
or must construct Running Average be set outside the setup and mail loops?

General question

Hi Rob,
I would like to know whether the ringbuffer lnegth is settable (like with the Running Angle).
Also is the code ready for production? I like the max min option a lot and would like to use it. (Along with a settable length)

Is a 'rolling' max possible?

I want to show the max (and min) 'rolling' values recorded from a stream during the previous 7 days. IOW, keep replacing the oldest with the most recent. Is that possible with this library please?

Potential Improvement? Fixed sized stack / global allocation.

The same features could be made to work with a fixed sized allocation, the size could be given with a templated parameter, no dynamic allocation required.

template<size_t _size>
class FixedRunningAverage {
//...

Optimizations to consider

fillValue()

fillValue is rather inefficient performance wise.
It first set all values to zero and then value is added N times, with all overhead of addValue.

optimized fillValue proposal

• program size 8010 - 7876 = 134 bytes more! (quite a bit)
• performance gain 1512 => 64 micros (a lot)

Testing to do

• fill whole array
• fill partial array
• number == 0 ?

void RunningAverage::fillValue(const float value, const uint16_t number)
{
  uint16_t s = number;
  if (s > _size) s = _size;
  for (uint16_t _index = 0; _index < s; _index++)
  {
    _array[_index] = value;
  }
  if (_index == _partial) _index = 0;  // faster than %
  _min = value;
  _max = value;
  _sum = s * value;
  _count = s;
}

stddev() stderr()

stderr depends on stddev so room for gain if both are needed.
Thoughts to consider:

• caching the value adds performance (dirty flag)
• calculating them both in one call is more efficient

add add() function as inline synonym of addValue()

synchronize interface with runningMedian as far as possible.

weird output result get when call StandardDeviation() without Average()

Note: follow up of - RobTillaart/Statistic#20

---

Get weird result when calculate standard deviation without average function in sketch.

ESP_LOGD("Stats", "%f %f", stats.getStandardDeviation(), stats.getCount());
[output]
[01:14:25][D][Stats:072]: nan -5486124068793694774077193376186634490938385057925026367753787985335901674201208705258107428552676834622700901594284127560530828493162262565205269470179449410856558896118393471584152670587577693885047222545890015691339878440394965043791461872188297303737712264679017607178892013979610612203518046997839872.00

ESP_LOGD("Stats", "%.2f %.2f %.2f", stats.getStandardDeviation(), stats.getCount(), stats.getAverage());
[output]
[01:20:26][D][Stats:072]: 0.05 0.00 0.70

improve the readme.md file

• add interface
• add minimal example?
• add known issues

Request for new partial "get" functions

Hi Rob, could you please consider implementing "partial" versions for some of the get functions? e.g. float RunningAverage::getPartialAverage(uint16_t points) where points specifies how many of the most recent points to use in the calculation. I think just the getAverage, getMaxInBuffer, getMinInBuffer for starters. It wouldn't work for the fast average, but I'm not worried about that.

I guess you could even modify the existing functions and if getSize() or getCount() is passed in, it calculates the value based on all points added to the buffer so far.

The reason I ask is I have a large buffer to store sensor history, but due to screen size limitation, I'm only displaying a portion of the data, and want to easily display min/max/average stats which are correct for the partial "displayed" data. I use the full history for some other calculations.

I guess another solution would be for me to have two circular buffers, one large for storing the full history data, and another smaller version which stores a copy of the subset for display purposes as I have plenty of RAM on the ESP32. Not so good for smaller Atmel type microcontrollers.

Thanks

A way to backup content?

I use RunningAverage to store values from a Barometer and a Thermometer. I do not only get min/max/average, but also draw a graph. It works fine until the Arduino restarts I lose all data, and It will take a day to fill the graphs again.

Is there a way to backup all values so they can survive a reboot? I understand that I can iterate through all values and save somewhere, but is there a smarter way? I guess that If I had a blob of bytes it could be saved to a file every 30 minutes or so.

ok to use this libaray for integers

Hi Rob, I've been searching for a simple circular buffer that can calculate averages. My project is for a Sensirion SCD-41 carbon dioxide (CO2) sensor which returns uint16_t values. This library looks ideal. Bonus is all the other min / max functions.

Do you see any problem with me converting these to float so I can use this libary?

Thanks!

replace tests for _array == NULL with _size tests

Would make the lib more portable to a template version.
Test size==0 would be more explicit about the problem?