ai-se/e-dom

Epsilon domination

e-dom

epsilon domination

e-dom's Issues

AUC of Popt20 (Higher the better)

• 20 repeats
• higher the better

camel

Jedit

Poi

log4j

synapse

velocity

xalan

xerces

ivy

lucene

parameter settings

• 20 repeats only for d2h measure
• I can only see quantile transformation is picked up quite good amount of times.

Learner

Preprocess

AUC of D2h (lower the better)

• 20 repeats
• lower the better

camel

Jedit

Poi

log4j

synapse

velocity

xalan

xerces

ivy

lucene

AUC of Popt (Higher the better)

• 20 repeats
• higher the better

camel

Jedit

Poi

log4j

synapse

velocity

xalan

xerces

ivy

lucene

Result - popt - Higher the better

Summary

• Y-axis shows the maximum Popt values achieved at every iteration
• X-axis, ran for 1000 different possible subtrees by choosing the least counter value. Total possible subtrees were 85,000.

Conclusion:

• epsilon dom exists, it flatlines for 0.025, 0.05 sooner than 0.1 and 0.2
• This means we do not get much improvement after any possible combination (any subtrees).

camel

Jedit

Poi

log4j

synapse

velocity

xalan

xerces

ivy

lucene

Result - D2h - Lower the better

UCI datasets, basic

Chirp weka: 10 way cross val.
Hyperband paper python

https://www.cs.uic.edu/~aanand/publications/KDD2011-CHIRP.pdf

Explaining classifiers wrt Variance

CSV File

Generated Tree:
https://github.com/ai-se/e-dom/blob/master/src/variance

Samples Needed

• Based on formula, these are the samples needed across different confidence and epsilon value

What we are achieving:

• size of epsilon larger in popt20 than d2h

Popt20

• When does popt20 starts to achieve plateaus for the maximum score achieved.
  

d2h

• When does d2h starts to achieve plateaus for the minimum score achieved.
  

OIL Effort Estimation

Mainly Flash and then goals.

Results

Summary

• Y-axis shows the maximum AUC values achieved at every iteration
• X-axis, ran for 1000 different possible subtrees by choosing the least counter value. Total possible subtrees were 85,000.

Conclusion:

• epsilon dom exists, it flatlines for 0.025, 0.05 sooner than 0.1 and 0.2
• This means we do not get much improvement after any possible combination (any subtrees).

Ivy

Log4j

Synapse

Velocity

Options being explored

Transformations

• StandardScaler
• MinMaxScaler
• MaxAbsScaler
• RobustScaler(quantile_range=(a, b))
  • a,b=_randint(0,50),_randint(51,100)
• KernelCenterer
• QuantileTransformer(n_quantiles=a, output_distribution=c, subsample=b)
  • a, b = _randint(100, 1000), _randint(1000, 1e5)
  • c=_randchoice(['normal','uniform'])
• Normalizer(norm=a)
  • a = _randchoice(['l1', 'l2','max'])
• Binarizer(threshold=a)
  • a=_randuniform(0,100)

Learners

• DecisionTreeClassifier(criterion=b, splitter=c, min_samples_split=a)
  • a=_randuniform(0.0,1.0)
  • b=_randchoice(['gini','entropy'])
  • c=_randchoice(['best','random'])
• RandomForestClassifier(n_estimators=a,criterion=b,min_samples_split=c)
  • a = _randint(50, 150)
  • b = _randchoice(['gini', 'entropy'])
  • c = _randuniform(0.0, 1.0)
• LogisticRegression(penalty=a, tol=b, C=float(c), solver='liblinear')
  • a=_randchoice(['l1','l2'])
  • b=_randuniform(0.0,0.1)
  • c=_randint(1,500)
• MultinomialNB(alpha=a)
  • a=_randuniform(0.0,0.1)
• KNeighborsClassifier(n_neighbors=a, weights=b, p=d, metric=c)
  • a = _randint(2, 25)
  • b = _randchoice(['uniform', 'distance'])
  • c = _randchoice(['minkowski','chebyshev'])
  • if c=='minkowski': d=_randint(1,15) else: d=2

Issue Close Time

• RF or check Mitch's Paper, FFT
  Goal - D2h

FFT against Tabu 100 evaluations:

D2h:

• Same numbers as seen for upto 1000 evaluations:

popt20:

• Same numbers as seen for upto 1000 evaluations:

Result (Text Mining) - D2h - Lower the better

Median and IQR for 20 repeats in each iteration

PitsA

PitsB

PitsC

PitsD

PitsE

PitsF

Update TSE Tex for bad smell, issue close time

Result - popt20 - Higher the better

FFT against Tabu

• Tabu search is showing median of 20 repeats achieved maximum on popt20
• Tabu search is showing median of 20 repeats achieved minimum on d2h

Max Popt

Min d2h

Multi-Goal

3 goals, c-dom:
Defect prediction - IFA, or david lo's measures
Text Mining - Runtimes

Flash Defect Prediction D2h Results

• Ran only with Decision tree tuning, with an initial population of 12, budget 30, total population size 10000
• Cannot run with the search space what dodge was exploring, the attributes to run with cart/regressor will change depending on which learner or preprocessor to use.
• Results:
  • Flash never wins against dodge but performs as good as dodge in 5 out of 10 datasets.

Bad Smell

• Baseline Random Forest, FFT

Goal: D2h