kerschke/flacco

Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems

flacco's Issues

Error due to infinite values (L-BFGS-B)

Check, how it is possible that L-BFGS-B wants to use infinite values:

Error in optim(as.numeric(par), fn, method = opt.algo, control = opt.algo.control,  :    L-BFGS-B needs finite values of "fn"

Extend the tutorial

Function Evaluations

We need a counter for the function evaluations of each feature group.

Re-iterate over InfoContent Features

As a follow up to Marios e-mail, I should re-iterate over the info content features.

Improve performance of feature groups

Check, whether nn-computation within nbf-features can be improved.

Running Time

Measure running time (and eventually also memory usage) for the computation of a feature group.

Testing

We should try to include even more tests.
@Dagefoerde could you have a look at it?

Latin Hyper-Cube malfunction?

I am trying to use FLACCO to characterise neural network error surfaces. I got the CRAN version of the library, and I get the following error when attempting to create an initial sample with a latin hypercube:

X = createInitialSample(n.obs = 1000, control=list(init_sample.type = "lhs", init_sample.lower = -5, init_sample.upper = 5), dim = calcDim(irisArch))

The result is:

Error in loadNamespace(name) : there is no package called ‘lhs’

The issue arises only with lhs. The random option works fine. However, if I understand Mersmann's paper correctly, latin hypercube is not really optional.

Installing as R package

Hi,
First i really appreciate your work.
When installing as a R package as mentioned in the READ_ME, the functions for computing features are not found. In the NAMESPACE file, i checked that they are not exported.
Do we need extra packages that are not mentioned?

I would appreciate if you could tell me exactly how i can make it work

Thank you

required package

In the required package, it should be added the packages:

• expm => for the gcm features
• mda => for the ela_level features
• e1071
• numDeriv

When installing flacco as a package, they are not mention as required. But during the run it may crash if these packages are not installed.

provide an init design generator

..

Improve usage of feature computation

Function names should be more intuitive and useful, e.g. calculateConvexityFeatures instead of calculateConvexity. I plan to do the following changes.

Cell Mapping:
(1) calculateAngle --> calculateAngleFeatures
(2) calculateCellConvexity --> calculateCellConvexityFeatures
(3) calculateGradientHomogeneity --> calculateGradientHomogeneityFeatures

ELA:
(1) calculateConvexity --> calculateConvexityFeatures
(2) calculateCurvature --> calculateCurvatureFeatures
(3) calculateLevelset --> calculateLevelsetFeatures
(4) calculateLocalSearch --> calculateLocalSearchFeatures
(5) calculateMetaModel --> calculateMetaModelFeatures
(6) calculateDistribution --> calculateDistributionFeatures

Misc:
(1) calculateBasics --> calculateBasicFeatures
(2) calculateDispersion --> calculateDispersionFeatures
(3) calculateLinModCoefficients --> calculateLinearModelFeatures
(4) calculateNearestBetter --> calculateNearestBetterFeatures
(5) calculatePCA --> calculatePrincipalComponentFeatures

Re-write GCM

So far, the GCM and barrier tree functions are just a 1-to-1 translation from matlab to R.
However, the source code is difficult to read/understand and needs to be modularized.

Re-initiate mda in levelset feats

In the current implementation of the levelset features, mda was replaced by rpart due to a bug in mda, which resulted in frequent crashes of mda.
Since the author(s) of mda apparently fixed that bug (with version 0.4-7), I can re-initiate the original implementation of the levelset features.

Re-write Barrier Tree Features

Need to fix the 1-to-1 translation of the barrier tree features.

coverall skips majority of tests

@mllg: I know that this is not related to any of your packages, but do you have any idea, why coverall skips the majority of my tests? Actually, it appears as if it is only running test_check("flacco", filter = "^base") although it should also run tests for other filters.

library(testthat)

# test general stuff:
test_check("flacco", filter = "^base")

# if (identical(Sys.getenv("TRAVIS"), "true") || identical(Sys.getenv("R_EXPENSIVE_TEST_OK"), "true")) {
  test_check("flacco", filter = "^plot")
# }

and

library(testthat)

# test features:
# if (identical(Sys.getenv("TRAVIS"), "true") || identical(Sys.getenv("R_EXPENSIVE_TEST_OK"), "true")) {
  test_check("flacco", filter = "^cm")
  test_check("flacco", filter = "^ela")
  test_check("flacco", filter = "^gcm")
  test_check("flacco", filter = "^ic")
  test_check("flacco", filter = "^misc")
  test_check("flacco", filter = "calculateFeatures")
# }

I was already guessing that it might be caused by the if-check prior to these tests (although it used to work until the beginning of February) - but comments around that if-clause do not help either :-/

parallelize feature computation

Could be useful for computation of very slow features, such as ela_level, gcm and (if possible) ic.

numDeriv does not respect constraints

Within the "ela_curv" feature set, we make use of the numDeriv package to compute the gradient and hessian of a function. Unfortunately, the function numDeriv:::genD, which is called within numDeriv::hessian, does not care about a function's constraints.
Thus, the estimation of the gradient and hessian need to be re-implemented.

ic.eps.max scaling

Unlike ic.eps.s and ic.eps.ratio, ic.eps.max is not log10-ed. This results in poor feature scaling.
I'd suggest to add a logarithm here to make them more consistent.

Compare:

• ic.eps.s

  flacco/R/feature_ic_infocontent.R

  Line 131 in 3d74dd1

  eps.S = ifelse(length(eps.S) > 0, log10(min(eps.S)), NA_real_)

• ic.eps.ratio aka eps05

  flacco/R/feature_ic_infocontent.R

  Line 141 in 3d74dd1

  eps05 = ifelse(length(eps05) > 0, log10(max(epsilon[eps05])), NA_real_)

• ic.eps.max

  flacco/R/feature_ic_infocontent.R

  Line 144 in 3d74dd1

  M0 = M0, Hmax = max(H), eps.max = median(epsilon[H == max(H)])))

Problems with coveralls and codecov

Trying to run coveralls or codecov results in an error:

> coveralls()
Error in eval(expr, envir, enclos) : Object 'boundary' not found

> codecov()
Error in eval(expr, envir, enclos) : Object 'boundary' not found

As long as this error occurs, we can't really test anymore :-/