aUtrecht University bUlsan National Institute of Science and Technology
1.a) Processing of an event without label
1.b) Processing of an event with label
First of all, the labels received are only used to train the models in the framework. The event may either be the last of
, i.e., length of
, in which case the label
becomes known, or not. When an event is not the last one of its trace (see Fig. 1.a), it is used to generate a new prefix of
for prefix
. Then, a prediction
for the new prefix can be computed using the model for prefix
. Receiving the last event
, and its label (see Fig. 1.b) enables (i) to evaluate all the predictions
that have been generated for all prefixes
(evaluation before training), (ii) to update the models
owing to the availability of new labelled prefixes. Finally, it is possible (iii) to compute a new set of predicted labels
, with
and for all the prefixes for which a label has not been yet received (train and retest).
2) Updating the moving window of predictions across model updates
“How often the model performance significantly deviated from (below) the moving average?”
The evaluation metric captures the rate at which the performance of an online predictive model drops below a certain level considered acceptable.
"How volatile are the performance values of the prediction framework?"
The evaluation metric captures the stability over time of the performance of an online predictive model.
“How much is the absolute value of deviated performance of the framework?”
The evaluation metric captures how much the perfor- mance of a model normally drops from its expected acceptable level.
“How quickly can the model be recovered from a sudden change in the prediction ability?”
The evaluation metric measures the speed at which an online model normally recovers from a performance drop.