There is a difference in performance between the masked evaluation and non-masked evalaution

[ExpectationMax]

There is a difference in performance between the masked evaluation and non-masked evalaution

[ExpectationMax]

Issue was due to difference in documentation between pytorch 1.4.0 (the version we were using) and 1.6.0 (the most recent version). The most recent docs suggested, that TransformerEncoderLayer also accepts the parameter attn_mask to be a binary indicator mask. This is not supported in 1.4.0 where attn_mask is purely additive. Due to this discrepancy masking of future values was not correctly performed.

[ExpectationMax]

While the fix reduced the discrepancy between masked and non masked evaluation for the attention model, it did not solve the issue completely:

TEST RESULTS
{'val_auroc': 0.7565103687452617,
 'val_average_precision': 0.06728730609706962,
 'val_balanced_accuracy': 0.6707938577708786,
 'val_loss': 2.148216962814331,
 'val_physionet2019_score': tensor(0.2370, dtype=torch.float64)}
--------------------------------------------------------------------------------
MASKED TEST RESULTS
{'auroc': 0.7429846217485517,
 'average_precision': 0.06584131430110093,
 'balanced_accuracy': 0.6691500984472765,
 'physionet2019_score': 0.2239510489510498}

These differences are still larger than the minor differences between masked and non-masked eval for the GRUModel, indicating that there still might be a degree of label leakage.

[ExpectationMax]

In recent results on EICU there still seems to be a discrepancy. Reopening issue

[ExpectationMax]

I cannot reproduce the issue on my installation.
Here the performance after training the model is still within acceptable precision.

Run 1:
{"validation": {"loss": 5.078474044799805, "average_precision": 0.24232479411151514, "auroc": 0.8313460351396962, "balanced_accuracy": 0.7457917630192751, "physionet2019_score": -0.029034758779837334, "epoch": 12, "created_at": "2020-10-07 14:54:11.687115"},
"validation_masked": {"auroc": 0.831346034944019, "average_precision": 0.2423247922257815, "balanced_accuracy": 0.7457917630192751, "physionet2019_score": -0.029034758779837334} }

Run 2:
{"validation": {"loss": 5.28493070602417, "average_precision": 0.2297666345319404, "auroc": 0.8214369492571697, "balanced_accuracy": 0.7283552421586119, "physionet2019_score": -0.0627334679454873, "epoch": 13, "created_at": "2020-10-07 14:36:42.791446"},
"validation_masked": {"auroc": 0.8214369507107713, "average_precision": 0.22976662822275082, "balanced_accuracy": 0.7283552421586119, "physionet2019_score": -0.0627334679454873}}