rahlk / bellwether Goto Github PK

Simpler Transfer Learning (Using "Bellwethers"). ARXIV link: https://arxiv.org/abs/1703.06218

Python 40.13% Shell 0.10% Roff 30.92% M 0.01% C 25.84% MATLAB 3.00%

defects effort code-smells issue-lifetime prediction bellwethers

bellwether's Introduction

👋 Hi there, I am Rahul.

I work in the intersection of Machine Learning and Programming Languages for modernizing legacy applications and mainitaining cloud-native applications. Prior to this, I was Post doctoral fellow at Columbia University working ARiSE Lab with Dr. Baishakhi Ray (see rayb.info) where my research explored the ways in which Machine Learning can be used to trigger, detect, and repair various problem across the software lifecycle.

I obtained my PhD form RAISE Lab at NC State University under the guidance of Dr. Tim Menzies (see menzies.us). During my PhD, I worked on actionable analytics for software engineering. I developed algorithms that go beyond prediction to generate insights that can assist decision making. I also worked on developing data mining techniques (transfer learning) to generate insights even when sufficient data is not available. For more details, see:

my Resume.
my Publications.

bellwether's People

Contributors

Stargazers

Watchers

Forkers

vivekaxl sagarwhu zeinabeivazpour ying-2016 yuanqingmei

bellwether's Issues

recall vs loc curves... should not they always be increasing or flat?

e.g

PD and PF (with only downsampling)

## ant

name 	 Pd 	 Pf 	 G
PDE 	 1.00 	 0.99 	 0.96
JDT 	 0.96 	 0.93 	 0.86
EQ  	 0.92 	 0.74 	 0.79
ML  	 0.83 	 0.73 	 0.72
LC  	 0.84 	 0.51 	 0.59
## xerces

name 	 Pd 	 Pf 	 G
PDE 	 0.97 	 0.97 	 0.96
ML  	 0.88 	 0.77 	 0.83
JDT* 	 0.62 	 0.44 	 0.51
LC  	 0.53 	 0.46 	 0.49
EQ  	 0.46 	 0.40 	 0.45
## xalan

name 	 Pd 	 Pf 	 G
ML  	 1.00 	 0.99 	 0.99
PDE 	 0.99 	 0.99 	 0.97
EQ  	 0.73 	 0.61 	 0.66
JDT 	 0.60 	 0.40 	 0.51
LC  	 0.32 	 0.20 	 0.25
## camel

name 	 Pd 	 Pf 	 G
PDE 	 0.92 	 0.84 	 0.87
ML  	 0.92 	 0.86 	 0.83
JDT 	 0.86 	 0.80 	 0.79
EQ  	 0.79 	 0.64 	 0.71
LC  	 0.46 	 0.27 	 0.35
## log4j

name 	 Pd 	 Pf 	 G
JDT 	 0.97 	 0.96 	 0.95
PDE 	 0.98 	 0.95 	 0.93
EQ  	 0.91 	 0.86 	 0.87
ML  	 0.90 	 0.81 	 0.84
LC  	 0.79 	 0.63 	 0.65
## lucene

name 	 Pd 	 Pf 	 G
ML  	 0.99 	 0.94 	 0.96
PDE 	 0.96 	 0.92 	 0.94
LC  	 0.76 	 0.54 	 0.62
JDT* 	 0.61 	 0.46 	 0.50
EQ  	 0.60 	 0.38 	 0.48
## jedit

name 	 Pd 	 Pf 	 G
PDE 	 0.95 	 0.95 	 0.93
ML  	 0.97 	 0.88 	 0.92
JDT* 	 0.66 	 0.39 	 0.52
LC  	 0.47 	 0.20 	 0.28
EQ  	 0.34 	 0.09 	 0.18
## poi

name 	 Pd 	 Pf 	 G
PDE 	 1.00 	 0.97 	 0.97
ML  	 0.92 	 0.73 	 0.80
JDT 	 0.90 	 0.70 	 0.79
EQ  	 0.84 	 0.61 	 0.66
LC  	 0.57 	 0.36 	 0.49
## velocity

name 	 Pd 	 Pf 	 G
PDE 	 0.93 	 0.89 	 0.90
JDT 	 0.93 	 0.89 	 0.90
ML  	 0.77 	 0.73 	 0.72
EQ  	 0.76 	 0.72 	 0.71
LC  	 0.27 	 0.24 	 0.34
## ivy

name 	 Pd 	 Pf 	 G
PDE 	 1.00 	 0.99 	 0.99
JDT 	 0.93 	 0.80 	 0.86
ML  	 0.86 	 0.61 	 0.75
EQ * 	 0.74 	 0.45 	 0.57
LC * 	 0.62 	 0.38 	 0.49
## Apache

name 	 Pd 	 Pf 	 G
xerc 	 1.00 	 1.00 	 1.00
xala 	 1.00 	 1.00 	 1.00
velo 	 0.98 	 0.99 	 0.97
came 	 0.96 	 0.88 	 0.92
luce 	 0.98 	 0.85 	 0.91
log4 	 0.97 	 0.86 	 0.85
poi 	 0.92 	 0.70 	 0.80
jedi 	 0.87 	 0.60 	 0.69
ant* 	 0.72 	 0.46 	 0.57
ivy* 	 0.71 	 0.40 	 0.51
## Safe

name 	 Pd 	 Pf 	 G
xala 	 1.00 	 1.00 	 1.00
xerc 	 1.00 	 1.00 	 1.00
poi 	 1.00 	 0.94 	 0.96
luce 	 1.00 	 0.93 	 0.96
velo 	 0.95 	 1.00 	 0.94
came 	 0.95 	 0.85 	 0.90
log4 	 1.00 	 0.82 	 0.90
ant 	 0.91 	 0.78 	 0.77
jedi 	 0.89 	 0.65 	 0.73
ivy 	 0.91 	 0.53 	 0.64
## Zxing

name 	 Pd 	 Pf 	 G
xerc 	 1.00 	 1.00 	 1.00
xala 	 1.00 	 1.00 	 1.00
came 	 1.00 	 1.00 	 1.00
log4 	 1.00 	 1.00 	 0.97
velo 	 0.95 	 0.98 	 0.96
luce 	 0.95 	 0.81 	 0.88
poi 	 0.93 	 0.82 	 0.87
ant 	 0.87 	 0.82 	 0.82
jedi 	 0.86 	 0.75 	 0.77
ivy 	 0.74 	 0.59 	 0.65
## kc

name 	 Pd 	 Pf 	 G
Zxin 	 0.53 	 0.49 	 0.51
Apac 	 0.51 	 0.47 	 0.50
Safe 	 0.31 	 0.10 	 0.16
## jm

name 	 Pd 	 Pf 	 G
Apac 	 0.74 	 0.78 	 0.72
Zxin 	 0.26 	 0.10 	 0.18
Safe 	 0.04 	 0.01 	 0.02
## mw

name 	 Pd 	 Pf 	 G
Apac 	 0.80 	 0.65 	 0.68
Zxin 	 0.65 	 0.62 	 0.60
Safe 	 0.17 	 0.08 	 0.12
## cm

name 	 Pd 	 Pf 	 G
Zxin 	 0.76 	 0.62 	 0.70
Apac 	 0.40 	 0.28 	 0.34
Safe 	 0.17 	 0.14 	 0.16
## mc

name 	 Pd 	 Pf 	 G
Zxin 	 0.90 	 0.73 	 0.79
Apac 	 0.57 	 0.41 	 0.49
Safe 	 0.08 	 0.04 	 0.06
## kc

name 	 Pd 	 Pf 	 G
JDT 	 1.00 	 0.99 	 0.99
PDE 	 1.00 	 1.00 	 0.99
LC  	 0.97 	 0.95 	 0.95
ML  	 0.99 	 0.93 	 0.95
EQ  	 1.00 	 0.95 	 0.93
## jm

name 	 Pd 	 Pf 	 G
PDE 	 0.95 	 0.91 	 0.91
ML  	 0.78 	 0.65 	 0.75
JDT 	 0.24 	 0.10 	 0.22
EQ  	 0.26 	 0.12 	 0.18
LC  	 0.22 	 0.07 	 0.11
## mw

name 	 Pd 	 Pf 	 G
EQ  	 1.00 	 0.92 	 0.94
ML  	 1.00 	 0.88 	 0.93
JDT 	 1.00 	 0.92 	 0.93
LC  	 1.00 	 0.85 	 0.92
PDE 	 0.96 	 0.86 	 0.87
## cm

name 	 Pd 	 Pf 	 G
ML  	 1.00 	 0.95 	 0.97
PDE 	 0.87 	 0.90 	 0.83
JDT 	 0.86 	 0.63 	 0.74
LC  	 0.88 	 0.62 	 0.72
EQ  	 0.79 	 0.66 	 0.69
## mc

name 	 Pd 	 Pf 	 G
PDE 	 1.00 	 0.99 	 0.99
ML  	 1.00 	 1.00 	 0.99
JDT 	 0.98 	 0.95 	 0.92
LC  	 0.89 	 0.58 	 0.69
EQ  	 0.85 	 0.56 	 0.68
## JDT

name 	 Pd 	 Pf 	 G
xerc 	 1.00 	 1.00 	 1.00
xala 	 1.00 	 1.00 	 1.00
velo 	 1.00 	 1.00 	 0.99
luce 	 1.00 	 0.98 	 0.98
came 	 0.98 	 0.95 	 0.97
log4 	 0.97 	 0.96 	 0.91
ant 	 0.92 	 0.74 	 0.79
poi* 	 0.76 	 0.40 	 0.56
jedi* 	 0.67 	 0.42 	 0.55
ivy 	 0.57 	 0.26 	 0.38
## PDE

name 	 Pd 	 Pf 	 G
xala 	 1.00 	 1.00 	 1.00
xerc 	 1.00 	 1.00 	 0.99
velo 	 0.99 	 0.99 	 0.98
luce 	 0.97 	 0.89 	 0.93
log4 	 0.97 	 0.94 	 0.91
poi 	 0.95 	 0.91 	 0.90
jedi 	 0.90 	 0.74 	 0.73
ant* 	 0.68 	 0.42 	 0.53
came 	 0.58 	 0.23 	 0.41
ivy 	 0.34 	 0.21 	 0.31
## EQ

name 	 Pd 	 Pf 	 G
xerc 	 1.00 	 1.00 	 1.00
xala 	 1.00 	 1.00 	 1.00
luce 	 1.00 	 1.00 	 0.96
velo 	 0.97 	 0.98 	 0.96
poi 	 0.99 	 0.95 	 0.92
came 	 0.97 	 0.83 	 0.86
log4 	 0.99 	 0.84 	 0.86
jedi 	 0.68 	 0.64 	 0.64
ivy* 	 0.87 	 0.45 	 0.60
ant 	 0.44 	 0.20 	 0.36
## LC

name 	 Pd 	 Pf 	 G
xala 	 1.00 	 1.00 	 1.00
xerc 	 1.00 	 1.00 	 1.00
velo 	 0.97 	 0.99 	 0.98
came 	 1.00 	 0.96 	 0.97
luce 	 0.98 	 0.96 	 0.95
poi 	 0.93 	 0.85 	 0.86
log4 	 0.93 	 0.85 	 0.85
ant 	 0.84 	 0.63 	 0.67
jedi 	 0.59 	 0.52 	 0.56
ivy 	 0.58 	 0.33 	 0.39
## ML

name 	 Pd 	 Pf 	 G
xala 	 1.00 	 1.00 	 1.00
xerc 	 1.00 	 1.00 	 1.00
velo 	 1.00 	 1.00 	 0.99
luce 	 0.88 	 0.88 	 0.88
log4 	 0.92 	 0.88 	 0.84
came 	 0.86 	 0.81 	 0.84
poi 	 0.87 	 0.81 	 0.79
ivy 	 0.82 	 0.65 	 0.64
ant 	 0.59 	 0.59 	 0.56
jedi 	 0.38 	 0.27 	 0.32
## kc

name 	 Pd 	 Pf 	 G
xala 	 1.00 	 1.00 	 1.00
xerc 	 1.00 	 1.00 	 1.00
poi 	 1.00 	 0.99 	 1.00
came 	 1.00 	 0.99 	 0.99
log4 	 1.00 	 0.99 	 0.98
luce 	 1.00 	 0.96 	 0.98
velo 	 0.97 	 0.98 	 0.98
jedi 	 0.89 	 0.71 	 0.78
ant 	 0.86 	 0.64 	 0.72
ivy 	 0.81 	 0.61 	 0.67
## jm

name 	 Pd 	 Pf 	 G
xala 	 1.00 	 1.00 	 1.00
velo 	 1.00 	 1.00 	 1.00
xerc 	 1.00 	 1.00 	 0.99
poi 	 0.98 	 0.96 	 0.94
log4 	 0.99 	 0.96 	 0.94
luce 	 0.94 	 0.88 	 0.81
came 	 0.88 	 0.73 	 0.78
ant 	 0.66 	 0.55 	 0.60
jedi 	 0.34 	 0.19 	 0.32
ivy 	 0.19 	 0.06 	 0.12
## mw

name 	 Pd 	 Pf 	 G
xerc 	 1.00 	 1.00 	 1.00
xala 	 1.00 	 1.00 	 1.00
log4 	 1.00 	 0.99 	 1.00
luce 	 1.00 	 0.96 	 0.98
poi 	 1.00 	 0.98 	 0.97
velo 	 0.96 	 0.99 	 0.95
ant 	 1.00 	 0.93 	 0.95
jedi 	 0.96 	 0.89 	 0.85
ivy 	 0.93 	 0.71 	 0.80
came 	 0.93 	 0.74 	 0.73
## cm

name 	 Pd 	 Pf 	 G
xala 	 1.00 	 1.00 	 1.00
xerc 	 1.00 	 1.00 	 1.00
velo 	 0.99 	 0.99 	 0.98
luce 	 1.00 	 0.95 	 0.97
poi 	 0.98 	 0.90 	 0.92
log4 	 0.95 	 0.83 	 0.88
jedi 	 0.98 	 0.81 	 0.88
came 	 0.79 	 0.64 	 0.76
ivy 	 0.90 	 0.67 	 0.76
ant 	 0.86 	 0.74 	 0.71
## mc

name 	 Pd 	 Pf 	 G
xala 	 1.00 	 1.00 	 1.00
xerc 	 1.00 	 1.00 	 1.00
luce 	 1.00 	 0.99 	 0.98
velo 	 1.00 	 0.99 	 0.98
log4 	 1.00 	 1.00 	 0.96
came 	 1.00 	 0.98 	 0.96
poi 	 1.00 	 0.91 	 0.95
ant 	 0.95 	 0.93 	 0.90
ivy 	 0.95 	 0.90 	 0.87
jedi 	 0.92 	 0.65 	 0.77
## ant

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 1.00
cm * 	 0.60 	 0.32 	 0.39
kc  	 0.14 	 0.04 	 0.09
mw  	 0.00 	 0.00 	 0.00
mc  	 0.00 	 0.00 	 0.00
## xerces

name 	 Pd 	 Pf 	 G
jm  	 0.99 	 0.99 	 0.97
cm  	 0.32 	 0.28 	 0.30
kc  	 0.15 	 0.08 	 0.10
mc  	 0.09 	 0.05 	 0.06
mw  	 0.01 	 0.00 	 0.00
## xalan

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 1.00
kc  	 0.20 	 0.19 	 0.18
cm  	 0.08 	 0.05 	 0.07
mc  	 0.05 	 0.02 	 0.04
mw  	 0.01 	 0.01 	 0.00
## camel

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 0.99
kc  	 0.58 	 0.48 	 0.49
mc  	 0.12 	 0.06 	 0.10
cm  	 0.06 	 0.02 	 0.05
mw  	 0.02 	 0.01 	 0.01
## log4j

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 0.99
cm  	 0.62 	 0.54 	 0.54
kc  	 0.08 	 0.05 	 0.07
mc  	 0.15 	 0.02 	 0.06
mw  	 0.00 	 0.00 	 0.00
## lucene

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 0.99
kc  	 0.26 	 0.23 	 0.25
mc  	 0.11 	 0.07 	 0.10
mw  	 0.00 	 0.00 	 0.00
cm  	 0.01 	 0.00 	 0.00
## jedit

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 1.00
kc  	 0.45 	 0.19 	 0.25
mc  	 0.07 	 0.01 	 0.03
cm  	 0.02 	 0.01 	 0.01
mw  	 0.01 	 0.00 	 0.00
## poi

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 0.98 	 0.98
mc  	 0.19 	 0.08 	 0.11
kc  	 0.05 	 0.05 	 0.08
cm  	 0.03 	 0.02 	 0.02
mw  	 0.01 	 0.00 	 0.00
## velocity

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 1.00
mc  	 0.07 	 0.05 	 0.06
cm  	 0.06 	 0.03 	 0.04
kc  	 0.01 	 0.01 	 0.01
mw  	 0.00 	 0.00 	 0.00
## ivy

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 0.98
cm  	 0.29 	 0.12 	 0.21
kc  	 0.08 	 0.12 	 0.10
mc  	 0.04 	 0.01 	 0.02
mw  	 0.01 	 0.00 	 0.00
## Apache

name 	 Pd 	 Pf 	 G
ML  	 0.98 	 0.96 	 0.95
PDE 	 0.99 	 0.93 	 0.95
JDT 	 0.98 	 0.90 	 0.92
LC  	 0.85 	 0.51 	 0.63
EQ * 	 0.73 	 0.42 	 0.55
## Safe

name 	 Pd 	 Pf 	 G
ML  	 1.00 	 1.00 	 0.98
JDT 	 1.00 	 0.96 	 0.97
PDE 	 0.93 	 0.91 	 0.90
EQ  	 0.91 	 0.82 	 0.87
LC  	 0.82 	 0.53 	 0.66
## Zxing

name 	 Pd 	 Pf 	 G
PDE 	 1.00 	 0.99 	 0.98
ML  	 0.98 	 0.98 	 0.96
JDT 	 0.83 	 0.70 	 0.77
EQ  	 0.71 	 0.58 	 0.62
LC  	 0.74 	 0.54 	 0.62
## Apache

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 1.00
kc * 	 0.61 	 0.29 	 0.36
mc  	 0.48 	 0.18 	 0.24
cm  	 0.21 	 0.08 	 0.11
mw  	 0.04 	 0.02 	 0.00
## Safe

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 0.99
cm * 	 0.68 	 0.29 	 0.41
mc  	 0.59 	 0.24 	 0.32
kc  	 0.30 	 0.10 	 0.00
mw  	 0.18 	 0.00 	 0.00
## Zxing

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 0.99
kc  	 0.41 	 0.30 	 0.36
mc  	 0.24 	 0.12 	 0.17
cm  	 0.14 	 0.08 	 0.10
mw  	 0.02 	 0.01 	 0.01
## JDT

name 	 Pd 	 Pf 	 G
jm  	 0.98 	 0.95 	 0.93
kc  	 0.28 	 0.05 	 0.10
cm  	 0.11 	 0.00 	 0.01
mc  	 0.04 	 0.00 	 0.00
mw  	 0.00 	 0.00 	 0.00
## PDE

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 0.94
kc  	 0.33 	 0.09 	 0.14
mw  	 0.01 	 0.00 	 0.00
cm  	 0.00 	 0.00 	 0.00
mc  	 0.00 	 0.00 	 0.00
## EQ

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 0.99
kc * 	 0.69 	 0.36 	 0.43
cm  	 0.29 	 0.07 	 0.12
mc  	 0.14 	 0.02 	 0.04
mw  	 0.02 	 0.00 	 0.00
## LC

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 1.00 	 1.00
kc  	 0.80 	 0.59 	 0.64
cm  	 0.08 	 0.03 	 0.04
mc  	 0.11 	 0.01 	 0.03
mw  	 0.03 	 0.00 	 0.00
## ML

name 	 Pd 	 Pf 	 G
jm  	 1.00 	 0.99 	 0.97
kc  	 0.46 	 0.48 	 0.43
cm  	 0.10 	 0.03 	 0.06
mc  	 0.09 	 0.01 	 0.02
mw  	 0.00 	 0.00 	 0.00
## JDT

name 	 Pd 	 Pf 	 G
Zxin 	 0.47 	 0.42 	 0.46
Apac 	 0.50 	 0.10 	 0.17
Safe 	 0.35 	 0.05 	 0.09
## PDE

name 	 Pd 	 Pf 	 G
Zxin 	 0.66 	 0.70 	 0.66
Apac 	 0.52 	 0.33 	 0.40
Safe 	 0.20 	 0.06 	 0.10
## EQ

name 	 Pd 	 Pf 	 G
Zxin* 	 0.74 	 0.36 	 0.47
Apac 	 0.39 	 0.10 	 0.19
Safe 	 0.15 	 0.01 	 0.02
## LC

name 	 Pd 	 Pf 	 G
Zxin 	 0.65 	 0.55 	 0.57
Apac 	 0.52 	 0.41 	 0.48
Safe 	 0.09 	 0.08 	 0.10
## ML

name 	 Pd 	 Pf 	 G
Zxin 	 0.51 	 0.44 	 0.47
Safe 	 0.26 	 0.15 	 0.19
Apac 	 0.19 	 0.11 	 0.17
## ant

name 	 Pd 	 Pf 	 G
Zxin 	 0.91 	 0.82 	 0.80
Apac 	 0.34 	 0.12 	 0.20
Safe 	 0.28 	 0.05 	 0.09
## xerces

name 	 Pd 	 Pf 	 G
Zxin 	 0.53 	 0.43 	 0.48
Apac 	 0.38 	 0.47 	 0.44
Safe 	 0.07 	 0.07 	 0.08
## xalan

name 	 Pd 	 Pf 	 G
Apac 	 0.37 	 0.33 	 0.34
Zxin 	 0.35 	 0.34 	 0.33
Safe 	 0.28 	 0.12 	 0.16
## camel

name 	 Pd 	 Pf 	 G
Zxin 	 0.76 	 0.64 	 0.66
Apac 	 0.24 	 0.16 	 0.19
Safe 	 0.05 	 0.03 	 0.04
## log4j

name 	 Pd 	 Pf 	 G
Apac 	 0.74 	 0.64 	 0.71
Safe 	 0.37 	 0.43 	 0.40
Zxin 	 0.21 	 0.19 	 0.24
## lucene

name 	 Pd 	 Pf 	 G
Zxin 	 0.50 	 0.45 	 0.46
Apac 	 0.11 	 0.04 	 0.09
Safe 	 0.03 	 0.01 	 0.01
## jedit

name 	 Pd 	 Pf 	 G
Apac 	 0.46 	 0.24 	 0.32
Zxin 	 0.36 	 0.15 	 0.21
Safe 	 0.09 	 0.03 	 0.04
## poi

name 	 Pd 	 Pf 	 G
Zxin 	 0.59 	 0.60 	 0.57
Apac 	 0.36 	 0.43 	 0.43
Safe 	 0.28 	 0.12 	 0.17
## velocity

name 	 Pd 	 Pf 	 G
Zxin 	 0.69 	 0.78 	 0.71
Apac 	 0.42 	 0.33 	 0.37
Safe 	 0.02 	 0.02 	 0.02
## ivy

name 	 Pd 	 Pf 	 G
Zxin 	 0.77 	 0.73 	 0.74
Apac 	 0.53 	 0.24 	 0.44
Safe 	 0.19 	 0.13 	 0.16

Raw Results


 ____     ___  _      _      __    __    ___ ______  __ __    ___  ____
|    \   /  _]| |    | |    |  |__|  |  /  _]      ||  |  |  /  _]|    \
|  o  ) /  [_ | |    | |    |  |  |  | /  [_|      ||  |  | /  [_ |  D  )
|     ||    _]| |___ | |___ |  |  |  ||    _]_|  |_||  _  ||    _]|    /
|  O  ||   [_ |     ||     ||  `  '  ||   [_  |  |  |  |  ||   [_ |    \
|     ||     ||     ||     | \      / |     | |  |  |  |  ||     ||  .  \
|_____||_____||_____||_____|  \_/\_/  |_____| |__|  |__|__||_____||__|\_|

            ^
           (\\
         .--,\\__
          `-.    a`-.__
            |         ')
           / \ _.-'-,`;
          /     |   { /
..-"``~"-'      ;    )
               ;'    `
              ;'
            ;'
    ,    /;'|
 ,;';\   |\ |
      \  || |
       | )| )
       | || |
 jgs   | \| \
       `##`##



### ant
+----------+------+------+------+------+
| Dataset  |  G2  |  ED  |  Pd  |  Pf  |
+==========+======+======+======+======+
| ivy      | 0.71 | 0.33 | 0.65 | 0.22 |
+----------+------+------+------+------+
| lucene   | 0.71 | 0.29 | 0.81 | 0.38 |
+----------+------+------+------+------+
| poi      | 0.70 | 0.32 | 0.69 | 0.29 |
+----------+------+------+------+------+
| jedit    | 0.69 | 0.37 | 0.57 | 0.16 |
+----------+------+------+------+------+
| camel    | 0.67 | 0.32 | 0.81 | 0.44 |
+----------+------+------+------+------+
| velocity | 0.67 | 0.33 | 0.83 | 0.46 |
+----------+------+------+------+------+
| xalan    | 0.66 | 0.33 | 0.82 | 0.47 |
+----------+------+------+------+------+
| log4j    | 0.65 | 0.34 | 0.87 | 0.51 |
+----------+------+------+------+------+
| xerces   | 0.62 | 0.37 | 0.86 | 0.55 |
+----------+------+------+------+------+ 

### camel
+----------+------+------+------+------+
| Dataset  |  G2  |  ED  |  Pd  |  Pf  |
+==========+======+======+======+======+
| lucene   | 0.58 | 0.44 | 0.60 | 0.44 |
+----------+------+------+------+------+
| xerces   | 0.57 | 0.44 | 0.62 | 0.48 |
+----------+------+------+------+------+
| log4j    | 0.56 | 0.44 | 0.64 | 0.50 |
+----------+------+------+------+------+
| velocity | 0.55 | 0.48 | 0.51 | 0.40 |
+----------+------+------+------+------+
| xalan    | 0.55 | 0.51 | 0.44 | 0.31 |
+----------+------+------+------+------+
| poi      | 0.53 | 0.56 | 0.35 | 0.21 |
+----------+------+------+------+------+
| ant      | 0.48 | 0.62 | 0.27 | 0.17 |
+----------+------+------+------+------+
| ivy      | 0.48 | 0.61 | 0.28 | 0.15 |
+----------+------+------+------+------+
| jedit    | 0.36 | 0.73 | 0.13 | 0.05 |
+----------+------+------+------+------+ 

### ivy
+----------+------+------+------+------+
| Dataset  |  G2  |  ED  |  Pd  |  Pf  |
+==========+======+======+======+======+
| lucene   | 0.66 | 0.35 | 0.69 | 0.38 |
+----------+------+------+------+------+
| ant      | 0.64 | 0.43 | 0.50 | 0.16 |
+----------+------+------+------+------+
| xalan    | 0.64 | 0.40 | 0.59 | 0.31 |
+----------+------+------+------+------+
| velocity | 0.63 | 0.37 | 0.70 | 0.43 |
+----------+------+------+------+------+
| xerces   | 0.63 | 0.37 | 0.71 | 0.44 |
+----------+------+------+------+------+
| poi      | 0.60 | 0.48 | 0.45 | 0.19 |
+----------+------+------+------+------+
| log4j    | 0.59 | 0.39 | 0.80 | 0.56 |
+----------+------+------+------+------+
| jedit    | 0.56 | 0.54 | 0.36 | 0.12 |
+----------+------+------+------+------+
| camel    | 0.55 | 0.44 | 0.67 | 0.55 |
+----------+------+------+------+------+ 

### jedit
+----------+------+------+------+------+
| Dataset  |  G2  |  ED  |  Pd  |  Pf  |
+==========+======+======+======+======+
| ant      | 0.71 | 0.32 | 0.65 | 0.22 |
+----------+------+------+------+------+
| poi      | 0.70 | 0.34 | 0.64 | 0.24 |
+----------+------+------+------+------+
| xalan    | 0.66 | 0.33 | 0.79 | 0.45 |
+----------+------+------+------+------+
| log4j    | 0.63 | 0.36 | 0.83 | 0.52 |
+----------+------+------+------+------+
| lucene   | 0.62 | 0.37 | 0.81 | 0.52 |
+----------+------+------+------+------+
| ivy      | 0.59 | 0.50 | 0.42 | 0.16 |
+----------+------+------+------+------+
| camel    | 0.58 | 0.42 | 0.65 | 0.47 |
+----------+------+------+------+------+
| xerces   | 0.57 | 0.41 | 0.83 | 0.61 |
+----------+------+------+------+------+
| velocity | 0.51 | 0.51 | 0.52 | 0.49 |
+----------+------+------+------+------+ 

### log4j
+----------+------+------+------+------+
| Dataset  |  G2  |  ED  |  Pd  |  Pf  |
+==========+======+======+======+======+
| xerces   | 0.62 | 0.42 | 0.58 | 0.34 |
+----------+------+------+------+------+
| velocity | 0.61 | 0.41 | 0.63 | 0.42 |
+----------+------+------+------+------+
| lucene   | 0.57 | 0.55 | 0.35 | 0.08 |
+----------+------+------+------+------+
| ant      | 0.55 | 0.56 | 0.33 | 0.11 |
+----------+------+------+------+------+
| camel    | 0.55 | 0.46 | 0.60 | 0.49 |
+----------+------+------+------+------+
| ivy      | 0.55 | 0.58 | 0.31 | 0.04 |
+----------+------+------+------+------+
| poi      | 0.55 | 0.57 | 0.32 | 0.04 |
+----------+------+------+------+------+
| xalan    | 0.53 | 0.50 | 0.52 | 0.45 |
+----------+------+------+------+------+
| jedit    | 0.44 | 0.67 | 0.20 | 0.02 |
+----------+------+------+------+------+ 

### lucene
+----------+------+------+------+------+
| Dataset  |  G2  |  ED  |  Pd  |  Pf  |
+==========+======+======+======+======+
| log4j    | 0.63 | 0.37 | 0.71 | 0.43 |
+----------+------+------+------+------+
| xerces   | 0.62 | 0.41 | 0.59 | 0.35 |
+----------+------+------+------+------+
| velocity | 0.61 | 0.39 | 0.68 | 0.46 |
+----------+------+------+------+------+
| poi      | 0.60 | 0.47 | 0.45 | 0.19 |
+----------+------+------+------+------+
| xalan    | 0.57 | 0.50 | 0.42 | 0.23 |
+----------+------+------+------+------+
| camel    | 0.55 | 0.47 | 0.57 | 0.47 |
+----------+------+------+------+------+
| ivy      | 0.55 | 0.55 | 0.35 | 0.14 |
+----------+------+------+------+------+
| ant      | 0.48 | 0.63 | 0.25 | 0.08 |
+----------+------+------+------+------+
| jedit    | 0.46 | 0.64 | 0.23 | 0.07 |
+----------+------+------+------+------+ 

### poi
+----------+------+------+------+------+
| Dataset  |  G2  |  ED  |  Pd  |  Pf  |
+==========+======+======+======+======+
| lucene   | 0.63 | 0.37 | 0.76 | 0.48 |
+----------+------+------+------+------+
| log4j    | 0.61 | 0.38 | 0.81 | 0.55 |
+----------+------+------+------+------+
| xerces   | 0.58 | 0.40 | 0.86 | 0.60 |
+----------+------+------+------+------+
| xalan    | 0.55 | 0.51 | 0.43 | 0.29 |
+----------+------+------+------+------+
| ivy      | 0.52 | 0.57 | 0.34 | 0.20 |
+----------+------+------+------+------+
| velocity | 0.51 | 0.54 | 0.43 | 0.39 |
+----------+------+------+------+------+
| ant      | 0.48 | 0.62 | 0.26 | 0.11 |
+----------+------+------+------+------+
| camel    | 0.48 | 0.56 | 0.44 | 0.46 |
+----------+------+------+------+------+
| jedit    | 0.40 | 0.69 | 0.17 | 0.07 |
+----------+------+------+------+------+ 

### velocity
+---------+------+------+------+------+
| Dataset |  G2  |  ED  |  Pd  |  Pf  |
+=========+======+======+======+======+
| ant     | 0.57 | 0.51 | 0.42 | 0.22 |
+---------+------+------+------+------+
| camel   | 0.57 | 0.44 | 0.60 | 0.46 |
+---------+------+------+------+------+
| lucene  | 0.57 | 0.46 | 0.54 | 0.40 |
+---------+------+------+------+------+
| log4j   | 0.55 | 0.51 | 0.44 | 0.31 |
+---------+------+------+------+------+
| xalan   | 0.54 | 0.51 | 0.45 | 0.34 |
+---------+------+------+------+------+
| xerces  | 0.51 | 0.48 | 0.64 | 0.59 |
+---------+------+------+------+------+
| ivy     | 0.46 | 0.63 | 0.25 | 0.15 |
+---------+------+------+------+------+
| poi     | 0.36 | 0.71 | 0.16 | 0.19 |
+---------+------+------+------+------+
| jedit   | 0.32 | 0.75 | 0.11 | 0.06 |
+---------+------+------+------+------+ 

### xalan
+----------+------+------+------+------+
| Dataset  |  G2  |  ED  |  Pd  |  Pf  |
+==========+======+======+======+======+
| log4j    | 0.56 | 0.48 | 0.51 | 0.39 |
+----------+------+------+------+------+
| camel    | 0.52 | 0.54 | 0.41 | 0.32 |
+----------+------+------+------+------+
| lucene   | 0.52 | 0.49 | 0.58 | 0.53 |
+----------+------+------+------+------+
| velocity | 0.52 | 0.54 | 0.40 | 0.33 |
+----------+------+------+------+------+
| ant      | 0.50 | 0.59 | 0.31 | 0.18 |
+----------+------+------+------+------+
| poi      | 0.48 | 0.61 | 0.28 | 0.17 |
+----------+------+------+------+------+
| xerces   | 0.48 | 0.51 | 0.63 | 0.63 |
+----------+------+------+------+------+
| ivy      | 0.47 | 0.62 | 0.28 | 0.19 |
+----------+------+------+------+------+
| jedit    | 0.44 | 0.65 | 0.23 | 0.13 |
+----------+------+------+------+------+ 

### xerces
+----------+------+------+------+------+
| Dataset  |  G2  |  ED  |  Pd  |  Pf  |
+==========+======+======+======+======+
| log4j    | 0.55 | 0.52 | 0.41 | 0.27 |
+----------+------+------+------+------+
| lucene   | 0.55 | 0.52 | 0.41 | 0.26 |
+----------+------+------+------+------+
| camel    | 0.52 | 0.53 | 0.42 | 0.36 |
+----------+------+------+------+------+
| xalan    | 0.52 | 0.56 | 0.35 | 0.23 |
+----------+------+------+------+------+
| poi      | 0.48 | 0.61 | 0.28 | 0.19 |
+----------+------+------+------+------+
| velocity | 0.48 | 0.59 | 0.34 | 0.30 |
+----------+------+------+------+------+
| ivy      | 0.43 | 0.66 | 0.22 | 0.14 |
+----------+------+------+------+------+
| ant      | 0.36 | 0.73 | 0.13 | 0.04 |
+----------+------+------+------+------+
| jedit    | 0.36 | 0.72 | 0.13 | 0.05 |
+----------+------+------+------+------+

Inconsistent Columns NASA dataset

Transferring knowledge in datasets with different columns
Do datasets have different Col IDs?
Yes! Although, within, projects they are the same.
Who has how many?
⁃◆ AR is totally different (Let's ignore that...)
⁃◆ ./Data/mccabe/cm/cm1.csv 39
⁃◆ ./Data/mccabe/jm/jm1.csv 23
⁃◆ ./Data/mccabe/kc/kc3.csv 41
⁃◆ ./Data/mccabe/mc/mc1.csv 40
⁃◆ ./Data/mccabe/mw/mw1.csv 39
⁃◆ ./Data/mccabe/pc/pc1.csv 39
⁃◆ ./Data/mccabe/pc2/pc2.csv 38
What is common?
A few things. Everything's fixed, up-and-running.

False Alarms too high!

Raw results

Remarks

Sorted by G-Score. (Median Values over 10 repeats)
Note very high PFs
No bellwether Individual Metrics.
Now changing to AUC instead.

JDT

name,Pd,Pf,G
lucene, 0.88, 0.71, 0.39
xerces, 0.75, 0.57, 0.32
log4j, 0.83, 0.44, 0.29
xalan, 0.59, 0.34, 0.22
poi, 0.49, 0.18, 0.13
camel, 0.50, 0.10, 0.09
velocity, 0.38, 0.11, 0.08
ant, 0.09, 0.02, 0.02
ivy, 0.09, 0.01, 0.01
jedit, 0.03, 0.00, 0.00

PDE

name,Pd,Pf,G
velocity, 0.93, 0.99, 0.48
lucene, 0.60, 0.36, 0.23
log4j, 0.62, 0.29, 0.20
xerces, 0.30, 0.45, 0.18
poi, 0.32, 0.21, 0.12
ivy, 0.27, 0.09, 0.07
ant, 0.14, 0.09, 0.06
camel, 0.10, 0.10, 0.05
xalan, 0.11, 0.08, 0.05
jedit, 0.00, 0.00, 0.00

EQ

name,Pd,Pf,G
velocity, 0.85, 0.64, 0.37
log4j, 0.85, 0.54, 0.33
lucene, 0.88, 0.48, 0.31
poi, 0.87, 0.39, 0.27
xerces, 0.60, 0.43, 0.25
xalan, 0.80, 0.34, 0.24
camel, 0.39, 0.11, 0.09
ant, 0.09, 0.04, 0.03
jedit, 0.08, 0.01, 0.00
ivy, 0.05, 0.00, 0.00

LC

name,Pd,Pf,G
lucene, 1.00, 0.94, 0.49
log4j, 0.95, 0.89, 0.46
velocity, 0.59, 0.53, 0.28
xerces, 0.55, 0.28, 0.18
xalan, 0.50, 0.29, 0.18
poi, 0.52, 0.16, 0.12
camel, 0.19, 0.07, 0.05
ivy, 0.17, 0.03, 0.03
ant, 0.03, 0.00, 0.00
jedit, 0.00, 0.00, 0.00

ML

name,Pd,Pf,G
log4j, 0.71, 0.53, 0.30
lucene, 0.70, 0.44, 0.27
velocity, 0.69, 0.40, 0.25
xalan, 0.56, 0.32, 0.20
xerces, 0.30, 0.35, 0.16
ivy, 0.13, 0.07, 0.05
poi, 0.13, 0.05, 0.03
camel, 0.04, 0.01, 0.00
ant, 0.03, 0.00, 0.00
jedit, 0.00, 0.00, 0.00

Apache

name,Pd,Pf,G
JDT, 0.70, 0.23, 0.17
PDE, 0.40, 0.19, 0.13
EQ, 0.52, 0.15, 0.11
LC, 0.41, 0.16, 0.11
ML, 0.27, 0.09, 0.07

Safe

name,Pd,Pf,G
EQ, 0.86, 0.76, 0.41
PDE, 0.68, 0.41, 0.26
LC, 0.73, 0.29, 0.21
JDT, 0.68, 0.29, 0.21
ML, 0.55, 0.21, 0.15

Zxing

name,Pd,Pf,G
EQ, 0.69, 0.60, 0.32
JDT, 0.58, 0.39, 0.23
PDE, 0.32, 0.35, 0.17
LC, 0.42, 0.22, 0.15
ML, 0.24, 0.11, 0.08

Apache

name,Pd,Pf,G
lucene, 0.96, 0.77, 0.43
log4j, 0.95, 0.75, 0.42
poi, 0.93, 0.65, 0.38
xalan, 0.78, 0.42, 0.27
xerces, 0.70, 0.44, 0.27
velocity, 0.54, 0.43, 0.24
camel, 0.27, 0.08, 0.06
ivy, 0.10, 0.02, 0.02
ant, 0.05, 0.02, 0.01
jedit, 0.02, 0.00, 0.00

Safe

name,Pd,Pf,G
log4j, 0.91, 0.82, 0.43
xalan, 0.91, 0.68, 0.39
lucene, 0.86, 0.56, 0.34
velocity, 0.59, 0.76, 0.33
xerces, 0.73, 0.53, 0.31
camel, 0.82, 0.44, 0.29
poi, 0.68, 0.38, 0.24
ant, 0.59, 0.26, 0.18
ivy, 0.55, 0.12, 0.10
jedit, 0.27, 0.15, 0.10

Zxing

name,Pd,Pf,G
velocity, 0.86, 0.95, 0.45
lucene, 0.89, 0.72, 0.40
xerces, 0.78, 0.70, 0.37
poi, 0.81, 0.65, 0.36
xalan, 0.79, 0.57, 0.33
log4j, 0.72, 0.48, 0.29
camel, 0.33, 0.16, 0.11
ivy, 0.13, 0.10, 0.06
jedit, 0.14, 0.05, 0.03
ant, 0.10, 0.05, 0.03

Apache

name,Pd,Pf,G
jm, 0.80, 0.53, 0.32
kc, 0.70, 0.32, 0.22
cm, 0.31, 0.05, 0.04
mc, 0.02, 0.01, 0.01
mw, 0.00, 0.00, 0.00

Safe

name,Pd,Pf,G
jm, 0.86, 0.41, 0.28
kc, 0.32, 0.56, 0.20
cm, 0.45, 0.18, 0.13
mw, 0.14, 0.00, 0.00
mc, 0.14, 0.00, 0.00

Zxing

name,Pd,Pf,G
jm, 0.64, 0.44, 0.26
kc, 0.41, 0.33, 0.18
cm, 0.08, 0.03, 0.02
mc, 0.03, 0.01, 0.01
mw, 0.00, 0.00, 0.00

JDT

name,Pd,Pf,G
jm, 0.54, 0.26, 0.17
kc, 0.05, 0.02, 0.01
cm, 0.01, 0.01, 0.00
mw, 0.00, 0.00, 0.00
mc, 0.00, 0.00, 0.00

PDE

name,Pd,Pf,G
jm, 0.45, 0.16, 0.12
kc, 0.07, 0.05, 0.03
cm, 0.05, 0.02, 0.01
mw, 0.00, 0.00, 0.00
mc, 0.00, 0.00, 0.00

EQ

name,Pd,Pf,G
jm, 0.67, 0.28, 0.20
cm, 0.13, 0.03, 0.02
kc, 0.10, 0.01, 0.00
mw, 0.00, 0.00, 0.00
mc, 0.05, 0.00, 0.00

LC

name,Pd,Pf,G
kc, 0.80, 0.98, 0.44
jm, 0.69, 0.45, 0.27
cm, 0.16, 0.03, 0.03
mw, 0.02, 0.00, 0.00
mc, 0.02, 0.00, 0.00

ML

name,Pd,Pf,G
jm, 0.26, 0.18, 0.11
kc, 0.26, 0.08, 0.06
cm, 0.04, 0.04, 0.02
mw, 0.00, 0.00, 0.00
mc, 0.00, 0.00, 0.00

JDT

name,Pd,Pf,G
Apache, 0.23, 0.05, 0.04
Safe, 0.23, 0.05, 0.04
Zxing, 0.07, 0.06, 0.03

PDE

name,Pd,Pf,G
Apache, 0.06, 0.07, 0.03
Safe, 0.05, 0.06, 0.03
Zxing, 0.00, 0.00, 0.00

EQ

name,Pd,Pf,G
Safe, 0.48, 0.24, 0.16
Apache, 0.35, 0.17, 0.11
Zxing, 0.04, 0.01, 0.01

LC

name,Pd,Pf,G
Zxing, 0.22, 0.10, 0.07
Apache, 0.05, 0.09, 0.03
Safe, 0.19, 0.02, 0.02

ML

name,Pd,Pf,G
Apache, 0.13, 0.19, 0.08
Safe, 0.23, 0.09, 0.06
Zxing, 0.07, 0.07, 0.03

ant

name,Pd,Pf,G
JDT, 0.81, 0.49, 0.31
EQ, 0.54, 0.24, 0.17
LC, 0.26, 0.07, 0.05
PDE, 0.08, 0.05, 0.03
ML, 0.14, 0.03, 0.02

xerces

name,Pd,Pf,G
EQ, 0.37, 0.21, 0.14
PDE, 0.22, 0.21, 0.11
JDT, 0.19, 0.19, 0.10
LC, 0.22, 0.10, 0.07
ML, 0.03, 0.01, 0.00

xalan

name,Pd,Pf,G
PDE, 0.34, 0.26, 0.15
EQ, 0.32, 0.26, 0.14
JDT, 0.11, 0.07, 0.04
LC, 0.12, 0.05, 0.04
ML, 0.04, 0.02, 0.01

camel

name,Pd,Pf,G
PDE, 0.27, 0.20, 0.12
LC, 0.28, 0.13, 0.09
EQ, 0.18, 0.15, 0.08
ML, 0.10, 0.06, 0.04
JDT, 0.09, 0.03, 0.02

log4j

name,Pd,Pf,G
JDT, 0.75, 0.66, 0.35
PDE, 0.54, 0.69, 0.30
EQ, 0.57, 0.48, 0.26
ML, 0.48, 0.33, 0.20
LC, 0.25, 0.03, 0.02

lucene

name,Pd,Pf,G
PDE, 0.52, 0.51, 0.26
EQ, 0.36, 0.16, 0.11
LC, 0.23, 0.10, 0.07
JDT, 0.09, 0.03, 0.02
ML, 0.05, 0.00, 0.00

jedit

name,Pd,Pf,G
PDE, 0.60, 0.44, 0.25
EQ, 0.59, 0.34, 0.22
LC, 0.20, 0.06, 0.05
JDT, 0.20, 0.06, 0.04
ML, 0.01, 0.01, 0.01

poi

name,Pd,Pf,G
EQ, 0.91, 0.76, 0.42
PDE, 0.17, 0.18, 0.09
JDT, 0.09, 0.03, 0.02
LC, 0.03, 0.00, 0.00
ML, 0.01, 0.00, 0.00

velocity

name,Pd,Pf,G
LC, 0.28, 0.15, 0.10
JDT, 0.23, 0.15, 0.09
EQ, 0.19, 0.10, 0.06
PDE, 0.12, 0.07, 0.05
ML, 0.07, 0.03, 0.02

ivy

name,Pd,Pf,G
ML, 0.40, 0.25, 0.16
PDE, 0.28, 0.20, 0.12
JDT, 0.40, 0.16, 0.11
EQ, 0.39, 0.14, 0.11
LC, 0.22, 0.05, 0.04

ant

name,Pd,Pf,G
kc, 0.10, 0.05, 0.03
jm, 0.02, 0.00, 0.00
cm, 0.01, 0.00, 0.00
mw, 0.00, 0.00, 0.00
mc, 0.00, 0.00, 0.00

xerces

name,Pd,Pf,G
jm, 0.39, 0.29, 0.17
kc, 0.30, 0.18, 0.11
cm, 0.05, 0.01, 0.00
mw, 0.01, 0.01, 0.00
mc, 0.01, 0.00, 0.00

xalan

name,Pd,Pf,G
jm, 0.55, 0.34, 0.21
kc, 0.07, 0.03, 0.02
cm, 0.02, 0.01, 0.01
mc, 0.01, 0.01, 0.00
mw, 0.00, 0.00, 0.00

camel

name,Pd,Pf,G
kc, 0.56, 0.46, 0.25
jm, 0.40, 0.30, 0.17
cm, 0.00, 0.00, 0.00
mw, 0.00, 0.00, 0.00
mc, 0.00, 0.00, 0.00

log4j

name,Pd,Pf,G
jm, 0.78, 0.69, 0.37
kc, 0.35, 0.31, 0.17
cm, 0.31, 0.22, 0.13
mw, 0.00, 0.00, 0.00
mc, 0.00, 0.00, 0.00

lucene

name,Pd,Pf,G
jm, 0.77, 0.70, 0.37
kc, 0.04, 0.01, 0.01
cm, 0.02, 0.00, 0.00
mw, 0.00, 0.00, 0.00
mc, 0.00, 0.00, 0.00

jedit

name,Pd,Pf,G
jm, 0.38, 0.37, 0.19
kc, 0.05, 0.01, 0.01
mc, 0.04, 0.00, 0.00
cm, 0.00, 0.00, 0.00
mw, 0.00, 0.00, 0.00

poi

name,Pd,Pf,G
kc, 0.76, 0.53, 0.31
jm, 0.51, 0.27, 0.18
mw, 0.00, 0.00, 0.00
cm, 0.00, 0.00, 0.00
mc, 0.00, 0.00, 0.00

velocity

name,Pd,Pf,G
kc, 0.34, 0.31, 0.16
jm, 0.10, 0.09, 0.05
cm, 0.05, 0.06, 0.03
mw, 0.01, 0.00, 0.00
mc, 0.00, 0.00, 0.00

ivy

name,Pd,Pf,G
jm, 0.22, 0.07, 0.05
cm, 0.02, 0.00, 0.00
kc, 0.00, 0.00, 0.00
mw, 0.00, 0.00, 0.00
mc, 0.01, 0.00, 0.00

ant

name,Pd,Pf,G
Safe, 0.21, 0.21, 0.11
Zxing, 0.03, 0.11, 0.02
Apache, 0.01, 0.01, 0.00

xerces

name,Pd,Pf,G
Zxing, 0.41, 0.54, 0.23
Apache, 0.24, 0.16, 0.10
Safe, 0.17, 0.11, 0.07

xalan

name,Pd,Pf,G
Safe, 0.14, 0.15, 0.07
Zxing, 0.15, 0.12, 0.07
Apache, 0.05, 0.03, 0.02

camel

name,Pd,Pf,G
Zxing, 0.28, 0.25, 0.13
Safe, 0.25, 0.22, 0.12
Apache, 0.29, 0.17, 0.11

log4j

name,Pd,Pf,G
Zxing, 0.14, 0.10, 0.06
Apache, 0.27, 0.03, 0.03
Safe, 0.01, 0.00, 0.00

lucene

name,Pd,Pf,G
Zxing, 0.32, 0.21, 0.13
Safe, 0.32, 0.20, 0.12
Apache, 0.30, 0.14, 0.10

jedit

name,Pd,Pf,G
Zxing, 0.19, 0.18, 0.09
Apache, 0.04, 0.04, 0.02
Safe, 0.06, 0.01, 0.01

poi

name,Pd,Pf,G
Apache, 0.03, 0.02, 0.01
Safe, 0.02, 0.02, 0.01
Zxing, 0.00, 0.02, 0.00

velocity

name,Pd,Pf,G
Zxing, 0.25, 0.29, 0.13
Safe, 0.17, 0.13, 0.07
Apache, 0.06, 0.06, 0.03

ivy

name,Pd,Pf,G
Zxing, 0.05, 0.11, 0.04
Apache, 0.06, 0.02, 0.01
Safe, 0.06, 0.01, 0.01

kc

name,Pd,Pf,G
PDE, 0.67, 0.53, 0.29
JDT, 0.56, 0.38, 0.23
EQ, 0.53, 0.34, 0.21
LC, 0.56, 0.32, 0.21
ML, 0.25, 0.30, 0.14

jm

name,Pd,Pf,G
EQ, 0.16, 0.06, 0.05
PDE, 0.08, 0.06, 0.04
ML, 0.02, 0.00, 0.00
LC, 0.02, 0.00, 0.00
JDT, 0.02, 0.00, 0.00

mw

name,Pd,Pf,G
EQ, 0.89, 0.78, 0.41
PDE, 0.85, 0.75, 0.40
JDT, 0.74, 0.50, 0.30
LC, 0.74, 0.43, 0.27
ML, 0.44, 0.21, 0.14

cm

name,Pd,Pf,G
JDT, 0.52, 0.35, 0.21
EQ, 0.62, 0.30, 0.20
LC, 0.48, 0.22, 0.15
PDE, 0.33, 0.21, 0.13
ML, 0.24, 0.08, 0.06

mc

name,Pd,Pf,G
JDT, 0.93, 0.84, 0.44
LC, 0.89, 0.68, 0.39
ML, 0.70, 0.32, 0.22
EQ, 0.63, 0.32, 0.21
PDE, 0.33, 0.24, 0.14

kc

name,Pd,Pf,G
log4j, 0.78, 0.88, 0.41
lucene, 0.86, 0.61, 0.36
poi, 0.78, 0.64, 0.35
camel, 0.78, 0.59, 0.33
xalan, 0.72, 0.61, 0.33
xerces, 0.58, 0.54, 0.28
velocity, 0.39, 0.44, 0.21
ant, 0.22, 0.07, 0.06
ivy, 0.14, 0.04, 0.03
jedit, 0.08, 0.03, 0.02

jm

name,Pd,Pf,G
velocity, 0.91, 0.85, 0.44
xerces, 0.50, 0.61, 0.28
log4j, 0.66, 0.45, 0.27
xalan, 0.42, 0.41, 0.21
lucene, 0.24, 0.10, 0.07
camel, 0.13, 0.03, 0.02
poi, 0.08, 0.02, 0.02
ivy, 0.03, 0.01, 0.01
ant, 0.01, 0.00, 0.00
jedit, 0.00, 0.00, 0.00

mw

name,Pd,Pf,G
xalan, 1.00, 0.98, 0.49
poi, 0.96, 0.87, 0.46
log4j, 1.00, 0.84, 0.46
velocity, 1.00, 0.83, 0.45
lucene, 0.85, 0.68, 0.38
camel, 0.81, 0.59, 0.34
ant, 0.59, 0.64, 0.31
xerces, 0.78, 0.47, 0.29
ivy, 0.67, 0.35, 0.23
jedit, 0.63, 0.28, 0.19

cm

name,Pd,Pf,G
xalan, 0.98, 0.91, 0.47
velocity, 0.93, 0.89, 0.45
poi, 0.98, 0.84, 0.45
lucene, 0.95, 0.78, 0.43
log4j, 0.90, 0.69, 0.39
xerces, 0.60, 0.71, 0.32
camel, 0.50, 0.29, 0.18
ivy, 0.45, 0.21, 0.14
ant, 0.24, 0.10, 0.07
jedit, 0.14, 0.08, 0.05

mc

name,Pd,Pf,G
log4j, 1.00, 0.99, 0.50
lucene, 1.00, 0.96, 0.49
poi, 1.00, 0.92, 0.48
xalan, 0.96, 0.81, 0.44
xerces, 0.93, 0.79, 0.43
velocity, 0.63, 0.60, 0.31
camel, 0.56, 0.38, 0.23
jedit, 0.56, 0.33, 0.21
ivy, 0.56, 0.31, 0.20
ant, 0.48, 0.12, 0.09

kc

name,Pd,Pf,G
Apache, 0.72, 0.57, 0.32
Zxing, 0.22, 0.37, 0.14
Safe, 0.17, 0.06, 0.05

jm

name,Pd,Pf,G
Apache, 0.06, 0.03, 0.02
Zxing, 0.06, 0.02, 0.01
Safe, 0.02, 0.01, 0.01

mw

name,Pd,Pf,G
Safe, 0.59, 0.36, 0.22
Zxing, 0.33, 0.32, 0.16
Apache, 0.22, 0.04, 0.04

cm

name,Pd,Pf,G
Apache, 0.52, 0.36, 0.21
Safe, 0.17, 0.09, 0.06
Zxing, 0.00, 0.00, 0.00

mc

name,Pd,Pf,G
Safe, 0.67, 0.30, 0.21
Apache, 0.41, 0.18, 0.13
Zxing, 0.15, 0.12, 0.07

Why this program both has Python2 and Python3 statements?

In old/ folder, the syntax seems to be Python2 but in other folder, code seems like Python3.
The question is, when run the run_all.py script, both python 2 and python 3 will be called, so there are so much syntax errors reported.
I'm a student from NJUS who want to reproduce your code. Thanks!
Email: [email protected]

PD and PF (without SMOTE)

FALSE ALARMS

Still too high without SMOTE
This was done using Scikit Learn's Random Forest
Next: Same with Logistic Regression from Weka (as per HDP paper)

/Users/rkrsn/git/rahlk/Bellwether/src/SEER/pickles/pretty_print.py

## JDT

name 	 Pd 	 Pf 	 G
log4 	 0.94 	 0.77 	 0.85
xala 	 0.81 	 0.86 	 0.83
xerc 	 0.82 	 0.69 	 0.75
luce 	 0.82 	 0.50 	 0.62
velo* 	 0.72 	 0.45 	 0.56
poi 	 0.56 	 0.10 	 0.17
came 	 0.47 	 0.09 	 0.16
ivy 	 0.08 	 0.01 	 0.02
ant 	 0.12 	 0.00 	 0.01
jedi 	 0.01 	 0.00 	 0.00
## PDE

name 	 Pd 	 Pf 	 G
luce 	 0.93 	 0.89 	 0.91
velo 	 0.82 	 0.91 	 0.86
log4 	 0.87 	 0.64 	 0.74
poi 	 0.78 	 0.63 	 0.70
xala 	 0.80 	 0.58 	 0.67
xerc 	 0.38 	 0.16 	 0.22
ivy 	 0.06 	 0.03 	 0.04
came 	 0.03 	 0.01 	 0.01
ant 	 0.00 	 0.00 	 0.00
jedi 	 0.00 	 0.00 	 0.00
## EQ

name 	 Pd 	 Pf 	 G
velo 	 0.88 	 0.95 	 0.92
xerc 	 0.56 	 0.67 	 0.61
xala* 	 0.74 	 0.45 	 0.56
log4* 	 0.84 	 0.32 	 0.46
luce* 	 0.69 	 0.28 	 0.40
poi* 	 0.63 	 0.22 	 0.32
came 	 0.42 	 0.15 	 0.22
ivy 	 0.16 	 0.06 	 0.09
jedi 	 0.12 	 0.01 	 0.02
ant 	 0.07 	 0.01 	 0.01
## LC

name 	 Pd 	 Pf 	 G
luce 	 1.00 	 0.87 	 0.93
log4 	 0.92 	 0.84 	 0.88
velo 	 0.88 	 0.77 	 0.82
xerc 	 0.84 	 0.79 	 0.81
xala 	 0.89 	 0.50 	 0.64
came 	 0.53 	 0.49 	 0.51
poi 	 0.42 	 0.09 	 0.15
ivy 	 0.03 	 0.03 	 0.03
jedi 	 0.03 	 0.01 	 0.01
ant 	 0.03 	 0.00 	 0.00
## ML

name 	 Pd 	 Pf 	 G
xala 	 0.77 	 0.52 	 0.62
log4 	 0.63 	 0.55 	 0.59
velo 	 0.47 	 0.31 	 0.38
luce 	 0.45 	 0.20 	 0.28
xerc 	 0.18 	 0.34 	 0.23
poi 	 0.18 	 0.10 	 0.13
ivy 	 0.11 	 0.04 	 0.05
jedi 	 0.13 	 0.02 	 0.04
ant 	 0.07 	 0.02 	 0.03
came 	 0.04 	 0.02 	 0.03
## Apache

name 	 Pd 	 Pf 	 G
LC  	 0.41 	 0.19 	 0.26
EQ  	 0.33 	 0.20 	 0.25
JDT 	 0.44 	 0.17 	 0.24
ML  	 0.51 	 0.15 	 0.23
PDE 	 0.06 	 0.04 	 0.05
## Safe

name 	 Pd 	 Pf 	 G
EQ  	 0.86 	 0.68 	 0.76
PDE* 	 0.64 	 0.44 	 0.52
LC * 	 0.73 	 0.35 	 0.48
JDT* 	 0.77 	 0.32 	 0.46
ML  	 0.55 	 0.12 	 0.19
## Zxing

name 	 Pd 	 Pf 	 G
JDT* 	 0.62 	 0.39 	 0.48
EQ * 	 0.60 	 0.39 	 0.47
PDE 	 0.50 	 0.38 	 0.43
ML  	 0.31 	 0.23 	 0.27
LC  	 0.36 	 0.20 	 0.26
## Apache

name 	 Pd 	 Pf 	 G
luce 	 0.91 	 0.65 	 0.75
log4 	 0.83 	 0.55 	 0.66
xala 	 0.68 	 0.53 	 0.60
xerc* 	 0.78 	 0.46 	 0.58
poi* 	 0.80 	 0.38 	 0.51
came* 	 0.81 	 0.34 	 0.48
velo 	 0.36 	 0.51 	 0.42
ivy 	 0.57 	 0.24 	 0.34
jedi 	 0.13 	 0.02 	 0.04
ant 	 0.04 	 0.01 	 0.02
## Safe

name 	 Pd 	 Pf 	 G
xala 	 0.91 	 0.82 	 0.86
luce 	 0.86 	 0.62 	 0.72
xerc 	 0.86 	 0.56 	 0.68
poi 	 0.82 	 0.56 	 0.66
velo 	 0.59 	 0.68 	 0.63
came* 	 0.77 	 0.41 	 0.54
log4* 	 0.82 	 0.38 	 0.52
jedi* 	 0.64 	 0.35 	 0.45
ivy* 	 0.68 	 0.24 	 0.35
ant* 	 0.64 	 0.24 	 0.34
## Zxing

name 	 Pd 	 Pf 	 G
velo 	 0.99 	 0.90 	 0.94
log4 	 0.81 	 0.79 	 0.80
xala 	 0.86 	 0.72 	 0.78
poi 	 0.81 	 0.66 	 0.73
luce 	 0.77 	 0.51 	 0.62
xerc 	 0.64 	 0.59 	 0.61
came* 	 0.67 	 0.41 	 0.51
ivy 	 0.34 	 0.16 	 0.22
ant 	 0.11 	 0.04 	 0.06
jedi 	 0.02 	 0.01 	 0.01
## Apache

name 	 Pd 	 Pf 	 G
jm * 	 0.67 	 0.39 	 0.49
mc  	 0.14 	 0.24 	 0.18
cm  	 0.42 	 0.10 	 0.17
kc  	 0.07 	 0.05 	 0.06
mw  	 0.00 	 0.00 	 0.00
## Safe

name 	 Pd 	 Pf 	 G
jm * 	 0.86 	 0.35 	 0.50
mc  	 0.32 	 0.15 	 0.20
kc  	 0.00 	 0.00 	 0.00
mw  	 0.14 	 0.00 	 0.00
cm  	 0.05 	 0.00 	 0.00
## Zxing

name 	 Pd 	 Pf 	 G
jm  	 0.58 	 0.38 	 0.46
cm  	 0.08 	 0.07 	 0.08
mc  	 0.02 	 0.08 	 0.03
kc  	 0.02 	 0.06 	 0.03
mw  	 0.01 	 0.00 	 0.01
## JDT

name 	 Pd 	 Pf 	 G
jm  	 0.75 	 0.66 	 0.70
kc  	 0.20 	 0.04 	 0.07
mc  	 0.21 	 0.02 	 0.04
cm  	 0.02 	 0.01 	 0.01
mw  	 0.00 	 0.00 	 0.00
## PDE

name 	 Pd 	 Pf 	 G
jm  	 0.43 	 0.32 	 0.37
kc  	 0.06 	 0.03 	 0.04
mc  	 0.10 	 0.01 	 0.02
cm  	 0.00 	 0.01 	 0.01
mw  	 0.00 	 0.00 	 0.00
## EQ

name 	 Pd 	 Pf 	 G
jm * 	 0.69 	 0.37 	 0.48
kc  	 0.48 	 0.15 	 0.23
cm  	 0.04 	 0.01 	 0.01
mc  	 0.01 	 0.01 	 0.01
mw  	 0.02 	 0.00 	 0.00
## LC

name 	 Pd 	 Pf 	 G
jm * 	 0.73 	 0.39 	 0.51
mc  	 0.03 	 0.03 	 0.03
kc  	 0.05 	 0.01 	 0.01
mw  	 0.02 	 0.00 	 0.00
cm  	 0.00 	 0.00 	 0.00
## ML

name 	 Pd 	 Pf 	 G
jm  	 0.20 	 0.10 	 0.13
cm  	 0.02 	 0.01 	 0.01
kc  	 0.01 	 0.00 	 0.00
mc  	 0.02 	 0.00 	 0.00
mw  	 0.00 	 0.00 	 0.00
## JDT

name 	 Pd 	 Pf 	 G
Safe 	 0.36 	 0.20 	 0.26
Apac 	 0.07 	 0.02 	 0.03
Zxin 	 0.10 	 0.01 	 0.01
## PDE

name 	 Pd 	 Pf 	 G
Safe 	 0.47 	 0.30 	 0.37
Apac 	 0.29 	 0.14 	 0.19
Zxin 	 0.03 	 0.01 	 0.01
## EQ

name 	 Pd 	 Pf 	 G
Apac 	 0.16 	 0.18 	 0.17
Safe 	 0.21 	 0.04 	 0.06
Zxin 	 0.07 	 0.04 	 0.05
## LC

name 	 Pd 	 Pf 	 G
Apac 	 0.34 	 0.16 	 0.22
Zxin 	 0.08 	 0.02 	 0.04
Safe 	 0.08 	 0.02 	 0.03
## ML

name 	 Pd 	 Pf 	 G
Safe 	 0.25 	 0.18 	 0.21
Zxin 	 0.27 	 0.17 	 0.21
Apac 	 0.06 	 0.04 	 0.05
## ant

name 	 Pd 	 Pf 	 G
LC * 	 0.67 	 0.22 	 0.33
EQ  	 0.54 	 0.22 	 0.32
PDE 	 0.21 	 0.37 	 0.27
JDT 	 0.32 	 0.08 	 0.13
ML  	 0.09 	 0.01 	 0.02
## xerces

name 	 Pd 	 Pf 	 G
PDE 	 0.67 	 0.74 	 0.71
LC  	 0.22 	 0.10 	 0.14
JDT 	 0.13 	 0.10 	 0.11
EQ  	 0.12 	 0.07 	 0.09
ML  	 0.04 	 0.03 	 0.04
## xalan

name 	 Pd 	 Pf 	 G
PDE 	 0.70 	 0.67 	 0.69
JDT 	 0.73 	 0.57 	 0.64
EQ  	 0.55 	 0.49 	 0.52
LC  	 0.25 	 0.09 	 0.13
ML  	 0.04 	 0.01 	 0.02
## camel

name 	 Pd 	 Pf 	 G
EQ  	 0.57 	 0.42 	 0.48
PDE 	 0.27 	 0.20 	 0.23
ML  	 0.10 	 0.06 	 0.07
LC  	 0.13 	 0.04 	 0.06
JDT 	 0.05 	 0.01 	 0.01
## log4j

name 	 Pd 	 Pf 	 G
EQ * 	 0.62 	 0.48 	 0.54
LC  	 0.50 	 0.22 	 0.31
PDE 	 0.13 	 0.15 	 0.14
JDT 	 0.25 	 0.10 	 0.14
ML  	 0.02 	 0.00 	 0.00
## lucene

name 	 Pd 	 Pf 	 G
JDT 	 0.40 	 0.14 	 0.20
PDE 	 0.13 	 0.20 	 0.16
LC  	 0.18 	 0.05 	 0.08
EQ  	 0.17 	 0.05 	 0.08
ML  	 0.08 	 0.03 	 0.05
## jedit

name 	 Pd 	 Pf 	 G
EQ  	 0.60 	 0.34 	 0.43
JDT 	 0.53 	 0.17 	 0.25
PDE 	 0.24 	 0.13 	 0.17
ML  	 0.17 	 0.11 	 0.13
LC  	 0.33 	 0.08 	 0.13
## poi

name 	 Pd 	 Pf 	 G
PDE 	 0.21 	 0.25 	 0.23
EQ  	 0.14 	 0.15 	 0.15
JDT 	 0.06 	 0.03 	 0.04
ML  	 0.06 	 0.02 	 0.03
LC  	 0.02 	 0.01 	 0.01
## velocity

name 	 Pd 	 Pf 	 G
EQ  	 0.49 	 0.41 	 0.45
PDE 	 0.14 	 0.16 	 0.15
ML  	 0.11 	 0.09 	 0.10
JDT 	 0.14 	 0.07 	 0.09
LC  	 0.07 	 0.05 	 0.06
## ivy

name 	 Pd 	 Pf 	 G
EQ  	 0.93 	 0.73 	 0.82
PDE 	 0.29 	 0.43 	 0.34
LC  	 0.42 	 0.15 	 0.23
ML  	 0.11 	 0.04 	 0.06
JDT 	 0.18 	 0.03 	 0.05
## ant

name 	 Pd 	 Pf 	 G
jm * 	 0.66 	 0.30 	 0.41
kc  	 0.44 	 0.18 	 0.26
mc  	 0.09 	 0.02 	 0.04
mw  	 0.02 	 0.00 	 0.00
cm  	 0.00 	 0.00 	 0.00
## xerces

name 	 Pd 	 Pf 	 G
jm  	 0.62 	 0.52 	 0.57
kc  	 0.18 	 0.15 	 0.16
mc  	 0.08 	 0.04 	 0.05
cm  	 0.02 	 0.00 	 0.01
mw  	 0.00 	 0.00 	 0.00
## xalan

name 	 Pd 	 Pf 	 G
jm  	 0.33 	 0.24 	 0.28
mc  	 0.02 	 0.00 	 0.00
mw  	 0.01 	 0.00 	 0.00
kc  	 0.00 	 0.00 	 0.00
cm  	 0.00 	 0.00 	 0.00
## camel

name 	 Pd 	 Pf 	 G
kc  	 0.31 	 0.23 	 0.27
jm  	 0.30 	 0.19 	 0.23
cm  	 0.02 	 0.01 	 0.01
mc  	 0.01 	 0.00 	 0.01
mw  	 0.00 	 0.00 	 0.00
## log4j

name 	 Pd 	 Pf 	 G
jm  	 0.50 	 0.33 	 0.40
mc  	 0.11 	 0.04 	 0.06
cm  	 0.09 	 0.03 	 0.05
mw  	 0.05 	 0.04 	 0.04
kc  	 0.07 	 0.01 	 0.02
## lucene

name 	 Pd 	 Pf 	 G
mc  	 0.19 	 0.15 	 0.17
jm  	 0.21 	 0.07 	 0.11
kc  	 0.02 	 0.01 	 0.01
cm  	 0.01 	 0.00 	 0.00
mw  	 0.00 	 0.00 	 0.00
## jedit

name 	 Pd 	 Pf 	 G
jm  	 0.47 	 0.14 	 0.22
kc  	 0.02 	 0.01 	 0.01
cm  	 0.04 	 0.01 	 0.01
mc  	 0.02 	 0.00 	 0.00
mw  	 0.00 	 0.00 	 0.00
## poi

name 	 Pd 	 Pf 	 G
jm  	 0.37 	 0.20 	 0.26
mc  	 0.12 	 0.06 	 0.08
kc  	 0.08 	 0.02 	 0.03
mw  	 0.01 	 0.00 	 0.00
cm  	 0.01 	 0.00 	 0.00
## velocity

name 	 Pd 	 Pf 	 G
jm  	 0.49 	 0.35 	 0.41
mc  	 0.45 	 0.36 	 0.40
kc  	 0.32 	 0.19 	 0.24
mw  	 0.01 	 0.00 	 0.00
cm  	 0.00 	 0.00 	 0.00
## ivy

name 	 Pd 	 Pf 	 G
kc  	 0.42 	 0.34 	 0.38
mc  	 0.42 	 0.28 	 0.33
jm  	 0.39 	 0.17 	 0.24
cm  	 0.03 	 0.00 	 0.00
mw  	 0.01 	 0.00 	 0.00
## ant

name 	 Pd 	 Pf 	 G
Safe 	 0.49 	 0.26 	 0.34
Zxin 	 0.24 	 0.37 	 0.29
Apac 	 0.01 	 0.01 	 0.01
## xerces

name 	 Pd 	 Pf 	 G
Safe 	 0.38 	 0.38 	 0.38
Zxin 	 0.33 	 0.29 	 0.31
Apac 	 0.17 	 0.11 	 0.14
## xalan

name 	 Pd 	 Pf 	 G
Apac 	 0.22 	 0.20 	 0.21
Zxin 	 0.21 	 0.20 	 0.21
Safe 	 0.13 	 0.06 	 0.08
## camel

name 	 Pd 	 Pf 	 G
Apac 	 0.30 	 0.29 	 0.29
Zxin 	 0.06 	 0.07 	 0.07
Safe 	 0.02 	 0.01 	 0.01
## log4j

name 	 Pd 	 Pf 	 G
Zxin 	 0.08 	 0.07 	 0.08
Apac 	 0.04 	 0.10 	 0.06
Safe 	 0.02 	 0.01 	 0.01
## lucene

name 	 Pd 	 Pf 	 G
Zxin 	 0.22 	 0.09 	 0.13
Apac 	 0.03 	 0.01 	 0.02
Safe 	 0.04 	 0.00 	 0.00
## jedit

name 	 Pd 	 Pf 	 G
Apac 	 0.36 	 0.15 	 0.21
Safe 	 0.02 	 0.01 	 0.01
Zxin 	 0.00 	 0.00 	 0.00
## poi

name 	 Pd 	 Pf 	 G
Zxin 	 0.12 	 0.12 	 0.12
Apac 	 0.11 	 0.04 	 0.06
Safe 	 0.04 	 0.07 	 0.05
## velocity

name 	 Pd 	 Pf 	 G
Safe 	 0.23 	 0.17 	 0.19
Apac 	 0.13 	 0.08 	 0.10
Zxin 	 0.01 	 0.01 	 0.01
## ivy

name 	 Pd 	 Pf 	 G
Apac 	 0.13 	 0.09 	 0.10
Safe 	 0.13 	 0.07 	 0.10
Zxin 	 0.09 	 0.06 	 0.08
## kc

name 	 Pd 	 Pf 	 G
EQ  	 0.78 	 0.66 	 0.72
JDT 	 0.56 	 0.39 	 0.46
LC  	 0.56 	 0.24 	 0.33
PDE 	 0.42 	 0.18 	 0.25
ML  	 0.06 	 0.01 	 0.02
## jm

name 	 Pd 	 Pf 	 G
PDE 	 0.04 	 0.02 	 0.03
EQ  	 0.03 	 0.00 	 0.01
ML  	 0.01 	 0.00 	 0.01
LC  	 0.01 	 0.00 	 0.00
JDT 	 0.01 	 0.00 	 0.00
## mw

name 	 Pd 	 Pf 	 G
EQ  	 0.96 	 0.89 	 0.92
JDT 	 0.89 	 0.68 	 0.77
PDE* 	 0.67 	 0.44 	 0.53
LC * 	 0.70 	 0.37 	 0.49
ML  	 0.59 	 0.24 	 0.35
## cm

name 	 Pd 	 Pf 	 G
PDE 	 0.79 	 0.65 	 0.71
JDT 	 0.36 	 0.18 	 0.24
LC  	 0.31 	 0.13 	 0.19
EQ  	 0.33 	 0.10 	 0.15
ML  	 0.21 	 0.07 	 0.10
## mc

name 	 Pd 	 Pf 	 G
PDE 	 0.75 	 0.75 	 0.75
JDT 	 0.78 	 0.60 	 0.68
EQ  	 0.58 	 0.41 	 0.48
LC * 	 0.61 	 0.29 	 0.40
ML  	 0.53 	 0.29 	 0.38
## kc

name 	 Pd 	 Pf 	 G
velo 	 0.97 	 0.97 	 0.97
came 	 1.00 	 0.84 	 0.91
log4 	 0.97 	 0.84 	 0.90
xala 	 0.89 	 0.73 	 0.80
luce 	 0.86 	 0.69 	 0.77
xerc 	 0.78 	 0.71 	 0.74
poi 	 0.67 	 0.58 	 0.62
jedi 	 0.31 	 0.14 	 0.20
ivy 	 0.33 	 0.11 	 0.17
ant 	 0.19 	 0.14 	 0.16
## jm

name 	 Pd 	 Pf 	 G
velo 	 0.84 	 0.68 	 0.75
came* 	 0.71 	 0.48 	 0.57
xerc 	 0.43 	 0.62 	 0.51
log4 	 0.36 	 0.32 	 0.34
poi 	 0.47 	 0.25 	 0.32
luce 	 0.37 	 0.16 	 0.22
xala 	 0.23 	 0.17 	 0.19
ivy 	 0.00 	 0.00 	 0.00
ant 	 0.00 	 0.00 	 0.00
jedi 	 0.00 	 0.00 	 0.00
## mw

name 	 Pd 	 Pf 	 G
log4 	 1.00 	 0.97 	 0.98
velo 	 0.96 	 0.94 	 0.95
poi 	 1.00 	 0.90 	 0.94
luce 	 0.89 	 0.85 	 0.87
came 	 0.89 	 0.81 	 0.85
xerc 	 0.89 	 0.70 	 0.79
xala 	 0.85 	 0.67 	 0.75
ant* 	 0.74 	 0.34 	 0.46
jedi 	 0.41 	 0.41 	 0.41
ivy 	 0.44 	 0.19 	 0.27
## cm

name 	 Pd 	 Pf 	 G
log4 	 0.95 	 0.87 	 0.91
xala 	 0.93 	 0.81 	 0.86
velo 	 0.93 	 0.73 	 0.82
poi 	 0.86 	 0.63 	 0.73
luce* 	 0.81 	 0.42 	 0.55
xerc* 	 0.64 	 0.42 	 0.51
came 	 0.45 	 0.16 	 0.23
jedi 	 0.14 	 0.05 	 0.08
ant 	 0.07 	 0.03 	 0.05
ivy 	 0.10 	 0.03 	 0.04
## mc

name 	 Pd 	 Pf 	 G
log4 	 0.97 	 0.91 	 0.94
poi 	 0.97 	 0.90 	 0.93
luce 	 0.89 	 0.74 	 0.81
velo 	 0.78 	 0.83 	 0.80
xala 	 0.89 	 0.66 	 0.76
xerc 	 0.78 	 0.60 	 0.67
came 	 0.56 	 0.37 	 0.44
ivy 	 0.39 	 0.15 	 0.22
ant 	 0.11 	 0.04 	 0.06
jedi 	 0.17 	 0.03 	 0.04
## kc

name 	 Pd 	 Pf 	 G
Apac 	 0.47 	 0.31 	 0.38
Zxin 	 0.36 	 0.37 	 0.37
Safe 	 0.42 	 0.22 	 0.29
## jm

name 	 Pd 	 Pf 	 G
Apac 	 0.02 	 0.00 	 0.00
Zxin 	 0.00 	 0.00 	 0.00
Safe 	 0.01 	 0.00 	 0.00
## mw

name 	 Pd 	 Pf 	 G
Safe 	 0.52 	 0.30 	 0.38
Apac 	 0.04 	 0.09 	 0.05
Zxin 	 0.00 	 0.00 	 0.00
## cm

name 	 Pd 	 Pf 	 G
Zxin 	 0.29 	 0.19 	 0.23
Safe 	 0.07 	 0.06 	 0.06
Apac 	 0.10 	 0.04 	 0.05
## mc

name 	 Pd 	 Pf 	 G
Safe 	 0.36 	 0.16 	 0.22
Zxin 	 0.03 	 0.11 	 0.04
Apac 	 0.08 	 0.01 	 0.01

TCA+ vs Bellwether

[~]$ python -B ~/git/rahlk/Bellwether/src/TCA+/par_exec.py

Summary (Now compiling....)

Looked at Pd, Pf, G (where g = sqrt(Pd*(1-Pf)); the harmonic mean between Pd and Pf used for sorting)
No clear Bellwether on any of the metrics. We were able to find on a metric called Balance (which was the euclidean distance from ideal score (Pd, Pf) = (1, 0)). Our results
here.
Now, note that out round robin evaluation strategy of finding the best dataset significantly outperforms TCA+
The authors of TCA+ do not report results on this data set. Moreover, they conveniently breeze through the most important part, the TCA algorithm. See sections 3.1 and 3.2 (this was a beast to implement! But offered no major benefits).
Their primary contribution of TCA+ is a proposal on choosing the best normalization strategy.

Comparisons

(I'm sorry, this is a formatted raw dump. I'll create a summary table and paste it here.)

Target Project: ANT

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.60 | 0.60 | 0.49 |
+----------+------+------+------+
| poi      | 0.59 | 0.60 | 0.48 |
+----------+------+------+------+
| velocity | 0.74 | 0.69 | 0.47 |
+----------+------+------+------+
| log4j    | 0.69 | 0.69 | 0.46 |
+----------+------+------+------+
| xerces   | 0.86 | 0.83 | 0.38 |
+----------+------+------+------+
| camel    | 0.08 | 0.06 | 0.27 |
+----------+------+------+------+
| jedit    | 0.06 | 0.09 | 0.23 |
+----------+------+------+------+
| ivy      | 0.05 | 0.07 | 0.22 |
+----------+------+------+------+
| xalan    | 0.03 | 0.03 | 0.17 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| ivy      | 0.67 | 0.25 | 0.71 |
+----------+------+------+------+
| jedit    | 0.61 | 0.20 | 0.70 |
+----------+------+------+------+
| xalan    | 0.72 | 0.32 | 0.70 |
+----------+------+------+------+
| lucene   | 0.77 | 0.39 | 0.69 |
+----------+------+------+------+
| poi      | 0.78 | 0.40 | 0.68 |
+----------+------+------+------+
| camel    | 0.79 | 0.45 | 0.66 |
+----------+------+------+------+
| velocity | 0.65 | 0.34 | 0.66 |
+----------+------+------+------+
| log4j    | 0.89 | 0.54 | 0.64 |
+----------+------+------+------+
| xerces   | 0.83 | 0.53 | 0.63 |
+----------+------+------+------+

Target Project: LUCENE

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| velocity | 0.76 | 0.78 | 0.41 |
+----------+------+------+------+
| poi      | 0.28 | 0.47 | 0.39 |
+----------+------+------+------+
| log4j    | 0.92 | 0.90 | 0.30 |
+----------+------+------+------+
| xerces   | 0.09 | 0.07 | 0.29 |
+----------+------+------+------+
| ant      | 0.05 | 0.08 | 0.22 |
+----------+------+------+------+
| camel    | 0.05 | 0.03 | 0.22 |
+----------+------+------+------+
| jedit    | 0.03 | 0.11 | 0.17 |
+----------+------+------+------+
| ivy      | 0.03 | 0.03 | 0.17 |
+----------+------+------+------+
| xalan    | 0.03 | 0.05 | 0.15 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| camel    | 0.58 | 0.27 | 0.65 |
+----------+------+------+------+
| log4j    | 0.75 | 0.46 | 0.63 |
+----------+------+------+------+
| poi      | 0.41 | 0.17 | 0.58 |
+----------+------+------+------+
| ivy      | 0.39 | 0.17 | 0.57 |
+----------+------+------+------+
| xerces   | 0.62 | 0.47 | 0.57 |
+----------+------+------+------+
| velocity | 0.42 | 0.26 | 0.56 |
+----------+------+------+------+
| xalan    | 0.42 | 0.29 | 0.55 |
+----------+------+------+------+
| ant      | 0.32 | 0.12 | 0.53 |
+----------+------+------+------+
| jedit    | 0.22 | 0.06 | 0.45 |
+----------+------+------+------+

Target Project: JEDIT

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| velocity | 0.48 | 0.46 | 0.51 |
+----------+------+------+------+
| poi      | 0.53 | 0.55 | 0.49 |
+----------+------+------+------+
| lucene   | 0.42 | 0.44 | 0.48 |
+----------+------+------+------+
| xerces   | 0.81 | 0.80 | 0.41 |
+----------+------+------+------+
| xalan    | 0.12 | 0.14 | 0.32 |
+----------+------+------+------+
| ivy      | 0.10 | 0.12 | 0.30 |
+----------+------+------+------+
| log4j    | 0.92 | 0.93 | 0.26 |
+----------+------+------+------+
| ant      | 0.04 | 0.03 | 0.19 |
+----------+------+------+------+
| camel    | 0.00 | 0.03 | 0.06 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| ant      | 0.68 | 0.26 | 0.71 |
+----------+------+------+------+
| xalan    | 0.65 | 0.33 | 0.66 |
+----------+------+------+------+
| ivy      | 0.51 | 0.17 | 0.65 |
+----------+------+------+------+
| lucene   | 0.67 | 0.36 | 0.65 |
+----------+------+------+------+
| camel    | 0.76 | 0.47 | 0.63 |
+----------+------+------+------+
| poi      | 0.48 | 0.23 | 0.61 |
+----------+------+------+------+
| velocity | 0.62 | 0.44 | 0.59 |
+----------+------+------+------+
| log4j    | 0.91 | 0.64 | 0.57 |
+----------+------+------+------+
| xerces   | 0.80 | 0.61 | 0.56 |
+----------+------+------+------+

Target Project: XERCES

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| velocity | 0.52 | 0.56 | 0.48 |
+----------+------+------+------+
| lucene   | 0.37 | 0.41 | 0.47 |
+----------+------+------+------+
| poi      | 0.30 | 0.31 | 0.46 |
+----------+------+------+------+
| xalan    | 0.24 | 0.27 | 0.42 |
+----------+------+------+------+
| log4j    | 0.74 | 0.85 | 0.34 |
+----------+------+------+------+
| jedit    | 0.03 | 0.02 | 0.17 |
+----------+------+------+------+
| ivy      | 0.02 | 0.02 | 0.15 |
+----------+------+------+------+
| camel    | 0.02 | 0.01 | 0.13 |
+----------+------+------+------+
| ant      | 0.02 | 0.01 | 0.12 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| log4j    | 0.47 | 0.27 | 0.59 |
+----------+------+------+------+
| camel    | 0.44 | 0.32 | 0.55 |
+----------+------+------+------+
| velocity | 0.41 | 0.26 | 0.55 |
+----------+------+------+------+
| xalan    | 0.41 | 0.31 | 0.53 |
+----------+------+------+------+
| lucene   | 0.38 | 0.33 | 0.50 |
+----------+------+------+------+
| poi      | 0.31 | 0.21 | 0.50 |
+----------+------+------+------+
| ant      | 0.17 | 0.09 | 0.39 |
+----------+------+------+------+
| jedit    | 0.16 | 0.07 | 0.38 |
+----------+------+------+------+
| ivy      | 0.14 | 0.07 | 0.36 |
+----------+------+------+------+

Target Project: XALAN

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| log4j    | 0.47 | 0.37 | 0.55 |
+----------+------+------+------+
| lucene   | 0.35 | 0.40 | 0.46 |
+----------+------+------+------+
| xerces   | 0.70 | 0.70 | 0.46 |
+----------+------+------+------+
| poi      | 0.33 | 0.35 | 0.46 |
+----------+------+------+------+
| velocity | 0.26 | 0.26 | 0.44 |
+----------+------+------+------+
| camel    | 0.11 | 0.10 | 0.32 |
+----------+------+------+------+
| ivy      | 0.10 | 0.13 | 0.30 |
+----------+------+------+------+
| jedit    | 0.07 | 0.03 | 0.26 |
+----------+------+------+------+
| ant      | 0.06 | 0.05 | 0.24 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.55 | 0.42 | 0.57 |
+----------+------+------+------+
| camel    | 0.49 | 0.37 | 0.56 |
+----------+------+------+------+
| log4j    | 0.55 | 0.43 | 0.56 |
+----------+------+------+------+
| xerces   | 0.62 | 0.55 | 0.53 |
+----------+------+------+------+
| velocity | 0.34 | 0.26 | 0.50 |
+----------+------+------+------+
| ant      | 0.27 | 0.14 | 0.49 |
+----------+------+------+------+
| poi      | 0.29 | 0.19 | 0.48 |
+----------+------+------+------+
| ivy      | 0.25 | 0.15 | 0.46 |
+----------+------+------+------+
| jedit    | 0.23 | 0.13 | 0.45 |
+----------+------+------+------+

Target Project: CAMEL

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| velocity | 0.74 | 0.62 | 0.53 |
+----------+------+------+------+
| log4j    | 0.55 | 0.55 | 0.50 |
+----------+------+------+------+
| xalan    | 0.59 | 0.61 | 0.48 |
+----------+------+------+------+
| lucene   | 0.31 | 0.35 | 0.45 |
+----------+------+------+------+
| poi      | 0.25 | 0.24 | 0.44 |
+----------+------+------+------+
| xerces   | 0.86 | 0.85 | 0.36 |
+----------+------+------+------+
| jedit    | 0.01 | 0.01 | 0.12 |
+----------+------+------+------+
| ant      | 0.01 | 0.01 | 0.08 |
+----------+------+------+------+
| ivy      | 0.01 | 0.02 | 0.08 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.51 | 0.33 | 0.59 |
+----------+------+------+------+
| xalan    | 0.44 | 0.28 | 0.57 |
+----------+------+------+------+
| log4j    | 0.67 | 0.53 | 0.56 |
+----------+------+------+------+
| velocity | 0.56 | 0.44 | 0.56 |
+----------+------+------+------+
| xerces   | 0.60 | 0.54 | 0.53 |
+----------+------+------+------+
| ant      | 0.28 | 0.14 | 0.49 |
+----------+------+------+------+
| poi      | 0.29 | 0.20 | 0.49 |
+----------+------+------+------+
| ivy      | 0.26 | 0.17 | 0.46 |
+----------+------+------+------+
| jedit    | 0.18 | 0.07 | 0.41 |
+----------+------+------+------+

Target Project: VELOCITY

TCA+

+--------+------+------+------+
|  Name  |  Pd  |  Pf  |  G   |
+========+======+======+======+
| lucene | 0.54 | 0.56 | 0.49 |
+--------+------+------+------+
| poi    | 0.29 | 0.25 | 0.46 |
+--------+------+------+------+
| camel  | 0.23 | 0.24 | 0.41 |
+--------+------+------+------+
| log4j  | 0.77 | 0.80 | 0.39 |
+--------+------+------+------+
| xerces | 0.04 | 0.08 | 0.20 |
+--------+------+------+------+
| ivy    | 0.03 | 0.02 | 0.18 |
+--------+------+------+------+
| ant    | 0.02 | 0.07 | 0.13 |
+--------+------+------+------+
| jedit  | 0.01 | 0.01 | 0.12 |
+--------+------+------+------+
| xalan  | 0.01 | 0.03 | 0.10 |
+--------+------+------+------+

Bellwether Method

+--------+------+------+------+
|  Name  |  Pd  |  Pf  |  G   |
+========+======+======+======+
| lucene | 0.52 | 0.35 | 0.58 |
+--------+------+------+------+
| xalan  | 0.47 | 0.29 | 0.58 |
+--------+------+------+------+
| log4j  | 0.77 | 0.57 | 0.57 |
+--------+------+------+------+
| camel  | 0.43 | 0.39 | 0.51 |
+--------+------+------+------+
| xerces | 0.47 | 0.44 | 0.51 |
+--------+------+------+------+
| ant    | 0.29 | 0.15 | 0.49 |
+--------+------+------+------+
| poi    | 0.31 | 0.24 | 0.49 |
+--------+------+------+------+
| ivy    | 0.19 | 0.09 | 0.42 |
+--------+------+------+------+
| jedit  | 0.11 | 0.07 | 0.31 |
+--------+------+------+------+

Target Project: POI

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| velocity | 0.56 | 0.55 | 0.50 |
+----------+------+------+------+
| lucene   | 0.82 | 0.78 | 0.43 |
+----------+------+------+------+
| log4j    | 0.89 | 0.86 | 0.35 |
+----------+------+------+------+
| xerces   | 0.09 | 0.12 | 0.29 |
+----------+------+------+------+
| xalan    | 0.07 | 0.14 | 0.25 |
+----------+------+------+------+
| ivy      | 0.04 | 0.03 | 0.21 |
+----------+------+------+------+
| ant      | 0.02 | 0.01 | 0.14 |
+----------+------+------+------+
| jedit    | 0.02 | 0.02 | 0.12 |
+----------+------+------+------+
| camel    | 0.01 | 0.00 | 0.08 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.73 | 0.40 | 0.66 |
+----------+------+------+------+
| camel    | 0.75 | 0.51 | 0.61 |
+----------+------+------+------+
| ivy      | 0.44 | 0.19 | 0.60 |
+----------+------+------+------+
| log4j    | 0.85 | 0.59 | 0.59 |
+----------+------+------+------+
| xalan    | 0.50 | 0.33 | 0.58 |
+----------+------+------+------+
| xerces   | 0.42 | 0.34 | 0.53 |
+----------+------+------+------+
| ant      | 0.27 | 0.13 | 0.48 |
+----------+------+------+------+
| velocity | 0.22 | 0.27 | 0.40 |
+----------+------+------+------+
| jedit    | 0.14 | 0.05 | 0.36 |
+----------+------+------+------+

Target Project: LOG4J

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.46 | 0.32 | 0.56 |
+----------+------+------+------+
| xerces   | 0.31 | 0.32 | 0.46 |
+----------+------+------+------+
| poi      | 0.67 | 0.69 | 0.45 |
+----------+------+------+------+
| velocity | 0.86 | 0.89 | 0.30 |
+----------+------+------+------+
| ivy      | 0.11 | 0.18 | 0.30 |
+----------+------+------+------+
| jedit    | 0.07 | 0.04 | 0.25 |
+----------+------+------+------+
| ant      | 0.03 | 0.03 | 0.18 |
+----------+------+------+------+
| xalan    | 0.02 | 0.03 | 0.15 |
+----------+------+------+------+
| camel    | 0.02 | 0.01 | 0.15 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| camel    | 0.52 | 0.21 | 0.64 |
+----------+------+------+------+
| poi      | 0.38 | 0.10 | 0.59 |
+----------+------+------+------+
| velocity | 0.45 | 0.21 | 0.59 |
+----------+------+------+------+
| lucene   | 0.36 | 0.08 | 0.57 |
+----------+------+------+------+
| xalan    | 0.50 | 0.34 | 0.57 |
+----------+------+------+------+
| ant      | 0.28 | 0.03 | 0.52 |
+----------+------+------+------+
| xerces   | 0.66 | 0.59 | 0.52 |
+----------+------+------+------+
| ivy      | 0.27 | 0.03 | 0.51 |
+----------+------+------+------+
| jedit    | 0.19 | 0.02 | 0.43 |
+----------+------+------+------+

Target Project: IVY

TCA+

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| jedit    | 0.39 | 0.22 | 0.55 |
+----------+------+------+------+
| poi      | 0.30 | 0.19 | 0.50 |
+----------+------+------+------+
| lucene   | 0.85 | 0.76 | 0.45 |
+----------+------+------+------+
| velocity | 0.72 | 0.73 | 0.44 |
+----------+------+------+------+
| log4j    | 0.62 | 0.70 | 0.43 |
+----------+------+------+------+
| ant      | 0.10 | 0.07 | 0.31 |
+----------+------+------+------+
| camel    | 0.08 | 0.09 | 0.26 |
+----------+------+------+------+
| xerces   | 0.06 | 0.07 | 0.23 |
+----------+------+------+------+
| xalan    | 0.02 | 0.06 | 0.13 |
+----------+------+------+------+

Bellwether Method

+----------+------+------+------+
|   Name   |  Pd  |  Pf  |  G   |
+==========+======+======+======+
| lucene   | 0.71 | 0.38 | 0.66 |
+----------+------+------+------+
| ant      | 0.49 | 0.16 | 0.64 |
+----------+------+------+------+
| camel    | 0.78 | 0.47 | 0.64 |
+----------+------+------+------+
| log4j    | 0.84 | 0.51 | 0.64 |
+----------+------+------+------+
| poi      | 0.53 | 0.24 | 0.64 |
+----------+------+------+------+
| xalan    | 0.57 | 0.29 | 0.64 |
+----------+------+------+------+
| velocity | 0.69 | 0.46 | 0.61 |
+----------+------+------+------+
| jedit    | 0.39 | 0.19 | 0.57 |
+----------+------+------+------+
| xerces   | 0.75 | 0.78 | 0.40 |
+----------+------+------+------+

There seems no holdout project selected in detect predition module?

I use pycharm search for "holdout" in whole project, but it seems no holdout dataset that claimed in your paper in detect prediction module.

TCA+ vs. SEER

TCA+ vs. SEER: PDs, PFs, and Gs.

TCA+ is much better than the current version of SEER.
Honestly, this baffles me, if anything SEER should be similar to Bellwether. Not sure why this is the case. Looking further into this.

Results

As a pdf file: results_tca_seer.pdf