ai-se/magic101

O.I.L: (Optimized Inductive Learning)

Submission

Submitted to Empirical Software Engineering (EMSE) 2018.

ARXIV Link: https://arxiv.org/pdf/1805.00336.pdf

Cite As

@article{xia2018hyperparameter,
  title={Hyperparameter Optimization for Effort Estimation},
  author={Xia, Tianpei and Krishna, Rahul and Chen, Jianfeng and Mathew, George and Shen, Xipeng and Menzies, Tim},
  journal={arXiv preprint arXiv:1805.00336},
  year={2018}
}

Authors

• Tianpei Xia
  • North Carolina State University, USA
  • [email protected]
• Tim Menzies
  • North Carolina State University, USA
  • [email protected]

Data

• Effort Data

Latex Source

• EMSE Submission

License

This is free and unencumbered software released into the public domain.

Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means.

(BTW, it would be great to hear from you if you are using this material. But that is optional.)

In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain. We make this dedication for the benefit of the public at large and to the detriment of our heirs and successors. We intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

For more information, please refer to http://unlicense.org

magic101's Issues

Updates 17-04-2018

Results with NSGA-II, Add SA results and runtimes

MRE:

SA:

Runtimes:

### Generations Stats:

Remarks:

• Tuned learners perform better than the state-of-the-art (CoGEE)

sharelatex

ASE2018

order cong using xml knowledge

Progress Report 06/21

DE Methods:

Learner (CART) Hyperparameters Explaination:

MRE Results (the smaller the better):

SA Results (the larger the better):

Running Time for each dataset:

Running time for each method:

Total running time for all datasets:

DE methods list for experiments:

Simplified COCOMOw effort estimates from a project:

Some initial results (incompleted):

read how other people do this

Code ABEN, generating some random configs, feature extraction code.

Updates 23-04-2018

MRE and SA results in latex:
https://www.sharelatex.com/project/5ad4a7c2acb70f42fe0a5479
"temp.tex" in sharelatex

Runtime:
https://goo.gl/zv5ZmG
"runtime.tex" in sharelatex

Learner * Optimizer results:
https://goo.gl/xeH3km

Table for CART trees hyperparameters is under constructing.

Progress Report 06/13

DE Methods:

Learner (CART) Hyperparameters Explaination:

Outputs Results (MRE):

Outputs Results (SA):

Running Time for each dataset:

Total running time for each method:

A hardness ranking list of DE methods:

Updates 16-04-2018

More results

Remarks:

• Tuned learners perform better than the state-of-the-art (CoGEE)

Others

• MOEA/D and NSGA-II still running
• Working on sharelatex project
• Graphics in progress ...

Updates 20-04-2018

MRE results for ABEN tuning, also add default CART0/ATLM/GoGEE for comparision.

code aben

Test data result from https://zenodo.org/communities/seacraft/search?page=1&size=20&q=&keywords=effort

Comparing DE/GA/NSGA-II/MOEAD

9 datasets, 3-fold cross validation, pop=50, gen=100, repeats=20

Experiments

For DE: (NP = 50, F = 1, CR = 0.5, life = 5)

1. Separate the data into train-part and test-part.

2. (Gen 0) Randomly generate 50 config (after constraints check), for each config[i] (i=1~50), calculate its mMRE (median MRE) on train-part.

3. (Gen 1~N) Use DE to generate 50 new config from precious Gen, and calculate their mMRE on train-part. For each config[i], if new config[i]'s mMRE is less than old config[i], use new config[i] to replace old config[i].

   Stop rules:

   1. reach Gen 100;
   2. reach Count = 5 (life); (initailly count=0, for each time that later gen's median mMRE >= former gen's least mMRE, Count += 1).

4. Use config with least mMRE in Gen N, calculate its mMRE on test-part.

5. Since 20 repeats and 3-fold, we got 20*3 = 60 mMRE values for each dataset.

For GA: (NP = 50, CX = 0.6, MUT = 0.1, life = 5)

1. Separate the data into train-part and test-part.

2. (Gen 0) Randomly generate 50 config (after constraints check), for each config[i] (i=1~50), calculate its mMRE (median MRE) on train-part.

3. (Gen 1~N) Use GA to generate 50 new config from precious Gen, and calculate their mMRE on train-part.

   Stop rules:

   1. reach Gen 100;
   2. reach Count = 5 (life); (initailly count=0, for each time that later gen's median mMRE >= former gen's least mMRE, Count += 1).

4. Use config with least mMRE in Gen N, calculate its mMRE on test-part.

5. Since 20 repeats and 3-fold, we got 20*3 = 60 mMRE values for each dataset.

Current Results (between ATLM, DE and GA):

A sorted graph between DE250 and GA250 in isbg10 dataset:

Runtime GA vs DE:

Number of Gen Comparison (between DE and GA):

Next Task

1. Add MOEA/D

2. Try NSGA-II with adjusted modification

3. Use DE/GA to tune CART

4. More literature review for potential paths

5. Update current OIL with uniform frameworks (DEAP/PyGMO..)

To Do

1. Re-construct OIL architecture (sklearn/utils/model/optimizer)

2. pip install package

3. Tutorial Materials (workshop to REU students)

4. Reverse negative results (Negative Results for Software Effort Estimation, 2016)

abe0 implementation in scikit learn

Updates 24-04-2018

A table of hyperparameters distribution for CART_DE2 has completed.
It named "parameter_distribution.tex" in sharelatex.

Progress Report 07/04

Experimental DE Methods (2):

---

COCOMO-STYLE Datasets pre-processing:

---

COCOMO-3C Range Reduction (From 6 columns to 3 columns):
note: vlow and vhigh values are used, mean is another option.

---

Effort Estimate Function for COCOMO-II:

"a" and "b" are used for DE tuning (DE/COCOMO-II).

---

Default COCOMO-II implementation:

---

Experiment Results:
(Performance Metric: SA, Repeat Times: 20, 3-Way Cross Validation)

---

COCOMO-STYLE Dataset vs Other Effort Dataset:

---

Using these 2 applied new DE methods to former experiments (submitted to ASE'18):
note: all slower methods are ruled out (CART-NSGA2/ABEN*/CoGEE)
(Performance Metric: SA, Repeat Times: 20, 3-Way Cross Validation)

---