Detection evaluation question

[koide3]

Detection evaluation question

[koide3]

Hi @agn-7 ,
To plot an ROC curve, you need to test the system with varying sensitivities (i.e., detection threshold). Maybe you can change the threshold at the following line to change the detection sensitivity (I have not tried though...), and measure FPR/TPR under different thresholds to plot the curve.

https://github.com/koide3/hdl_people_tracking/blob/8c0d863e77207e7012217fcd16ce89f29a78ba67/src/kidono_human_classifier.cpp#L41https://docs.opencv.org/2.4/modules/ml/doc/boosting.html#cvboost-predict

[agn-7]

Thanks for the response,

Also, we can change some parameters value in preprocessing level such as downsampling value ...

But I realized that to evaluating pedestrian detection we couldn't use ROC curve, because we cannot count the TN (non-pedestrian) in each frame. So with borrowing from this link, I found out the true evaluating for this project is only precision and recall (and F1-score to integrating them), because it is a retrieve problem and the number of TP (pedestrian) and non-pedestrian is not equal and is an imbalance.

---

TL;DR

When to Use ROC vs. Precision-Recall Curves?

Generally, the use of ROC curves and precision-recall curves are as follows:

• ROC curves should be used when there are roughly equal numbers of observations for each class.
• Precision-Recall curves should be used when there is a moderate to large class imbalance.

The reason for this recommendation is that ROC curves present an optimistic picture of the model on datasets with a class imbalance.

However, ROC curves can present an overly optimistic view of an algorithm’s performance if there is a large skew in the class distribution. […] Precision-Recall (PR) curves, often used in Information Retrieval, have been cited as an alternative to ROC curves for tasks with a large skew in the class distribution.

[akamob]

Hi, @agn-7,

I'm wondering if it is possible that you could share the precision-recall curve and F1-score of human detection with me?
I want to quantify the performance.

Thank you!