Implement unstable irradiance filters

[bt-]

Implement unstable irradiance filters

[bt-]

Draft for a version of the ASTM std approach when you already have the std from the historian:

def drop_unstable_irr(data, std_perc_columns, threshold=2):
    """
    Removes cloudy intervals by the statistical technique defined in ASTM E2848.
    
    Filters out periods of unstable irradiance based on the standard deviation of
    samples as a percentage of the mean of samples. Standard deviations that are
    a higher percentage of the mean than the threshold are deemed periods of
    unstable irradiance and dropped.
    
    Parameters
    ----------
    data : DataFrame
        Data to filter which must include the columns specified by `std_perc_columns`
    std_perc_columns : list
        List of column names for the columns that contain the standard deviations of
        the samples in an averaging interval as a percent of the mean of the samples
        for the averaging interval.
    threshold : numeric
        Values greater than the threshold are removed. E.g. 2 is 2 percent.
    all_cols : boolean, default False
        By default any interval where any of the values in the columns of
        `std_perc_columns` is above threshold is removed. Change to True
        to require all columns to be above theshold to remove.
    """
    df = data[std_perc_columns]
    mask = df > threshold
    return data[~(mask.any(axis=1))]

[bt-]

Latest draft version, which improves on the above by calculating the std as a percent of the mean and adds those percentages to the data and data_filtered attributes:

class CapDataStd(CapData):
    @update_summary
    def filter_unstable_irr_std(self, poa_group, std_group, threshold=2):
        """
        Removes cloudy intervals by the statistical technique defined in ASTM E2848.

        Filters out periods of unstable irradiance based on the standard deviation of
        samples as a percentage of the mean of samples. Standard deviations that are
        a higher percentage of the mean than the threshold are deemed periods of
        unstable irradiance and dropped.

        Parameters
        ----------
        poa_group : str
            Id of `column_groups` for the group that contain the mean POA irradiance
            measurements for each interval.
        std_group : str
            Id of `column_groups` for the group that contain the standard deviation of the POA irradiance
            measurements for each interval.
        threshold : numeric
            Values greater than the threshold are removed. E.g. 2 is 2 percent.
        """
        p = self.loc[poa_group]
        s = self.loc[std_group]
        std_perc_columns_list = []
        for poa_col in p.columns:
            p_tag = p[poa_col]
            s_tag = s.filter(regex=poa_col).iloc[:, 0]
            std_perc = ((s_tag / p_tag) * 100)
            std_perc_columns_list.append(std_perc.rename(s_tag.name + '_perc'))
        std_perc_columns = pd.concat(std_perc_columns_list, axis=1)
        self.data = pd.concat([self.data, std_perc_columns], axis=1)
        std_perc_columns_filtered = std_perc_columns.loc[self.data_filtered.index, :]
        self.data_filtered = pd.concat([self.data_filtered, std_perc_columns_filtered], axis=1)
        mask = std_perc_columns_filtered > threshold
        self.data_filtered = self.data_filtered[~(mask.any(axis=1))]