yunshengtian/autooed

AutoOED: Automated Optimal Experimental Design Platform

autooed's Issues

Difficulty trying to apply more than one constraint to optimization

Hi !

I am having troubles trying to apply multiple constraints to my design variables. To reproduce my case :

1. create a problem with 4 variables, 2 objectives and 2 constraints
2. create a constraint.py file with constraints x1 - x2 and x1 - x3
3. try to generate samples (5 in my example), an error pop :

I think the problem comes from the evaluate_feasible() in problem.py, more specifically this call to pymoo when G contains multiple constraint evaluations :

Problem.calc_constraint_violation(np.column_stack(np.atleast_2d(G))).

If you need more informations please tell me.

Adding folder for experiment config

Currently, the problem configuration is saved in the "problem/custom/yaml" folder, while the experiment configuration can only be accessed through SQL. I would like to suggest adding a folder in the "system/experiment/custom/yaml" directory.

This addition would allow for easy viewing of previous experiment configurations and enables corrections within the YAML file itself.

Difficulty running a manual experiment

Hi there! Stumbled on AutoOED recently and it looks to be a great fit for what I need. I'm attempting to run a manual experiment and use AutoOED to help guide tuning of parameters. Unfortunately I haven't been able to progress past the initial randomly generated samples. Optimization seems to hang no matter what I do.

1. I created my problem with three continuous variables and two objectives (min(f1), max(f2)).
2. Created the experiment, generated six random samples, manually performed the experiment and input the values with "Enter Performance" on the database tab
3. Pressed "Optimize" with a batch size of six to get the next set of parameters to test
4. Optimization hangs

I'm using the DGEMO algorithm with default values as recommended in the docs.

If you run it via the terminal (master branch) instead of the downloadable executable, I can see that one of the workers dies with ValueError: Buffer has wrong number of dimensions (expected 2, got 1):

(autooed) C:\Users\polyf\Downloads\AutoOED-master>python run_gui.py
Process Process-2:
Traceback (most recent call last):
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\system\agent.py", line 712, in optimize
    X_next, (Y_pred_mean, Y_pred_std) = optimize_predict(config, X, Y, X_busy, batch_size=batch_size)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\core.py", line 160, in optimize_predict
    X_next = optimizer.optimize(X, Y, X_busy, batch_size)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\mobo.py", line 94, in optimize
    return self._optimize(X, Y, batch_size)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\mobo.py", line 109, in _optimize
    X_candidate, Y_candidate = self.solver.solve(X, Y, batch_size, self.acquisition)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\solver\base.py", line 47, in solve
    X_candidate, Y_candidate = self._solve(X, Y, batch_size)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\solver\pareto_discovery\pareto_discovery.py", line 632, in _solve
    res = minimize_ea(self.problem, self.algo, ('n_gen', self.n_gen))
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\site-packages\pymoo\optimize.py", line 76, in minimize
    res = algorithm.solve()
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\site-packages\pymoo\model\algorithm.py", line 208, in solve
    self._solve(self.problem)
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\site-packages\pymoo\model\algorithm.py", line 295, in _solve
    self.finalize()
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\site-packages\pymoo\model\algorithm.py", line 269, in finalize
    return self._finalize()
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\solver\pareto_discovery\pareto_discovery.py", line 613, in _finalize
    self.fam_lbls, self.approx_set, self.approx_front = self.buffer.sparse_approximation()
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\mobo\solver\pareto_discovery\buffer.py", line 249, in sparse_approximation
    labels_opt = cut_from_graph(edges, unary_cost, pairwise_cost, label_cost)
  File "gco_python.pyx", line 139, in pygco.cut_from_graph
ValueError: Buffer has wrong number of dimensions (expected 2, got 1)
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\system\scheduler.py", line 359, in _refresh_optimize
    rowids = self.opt_queue.get(block=False)
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\multiprocessing\queues.py", line 107, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\tkinter\__init__.py", line 1705, in __call__
    return self.func(*args)
  File "C:\Users\polyf\miniconda3\envs\autooed\lib\tkinter\__init__.py", line 749, in callit
    func(*args)
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\system\gui\gui.py", line 302, in refresh
    self.scheduler.refresh()
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\system\scheduler.py", line 454, in refresh
    opt_rowids = self._refresh_optimize()
  File "C:\Users\polyf\Downloads\AutoOED-master\autooed\system\scheduler.py", line 361, in _refresh_optimize
    raise Exception('optimization worker finished without returning rowids of the design to evaluate')
Exception: optimization worker finished without returning rowids of the design to evaluate

I thought maybe an evaluation function was mandatory (although the docs say it isn't), so I added an eval function that just returns the experimental results:

def evaluate_objective(x):
    print(x)
    x1, x2, x3 = x

    if x1 == 20143:
        return 233628.0, 47.66
    if x1 == 24897:
        return 316294.0, 4.416
    if x1 == 15235:
        return 366988.0,23.02
    if x1 == 11547:
        return 224685.0,16.32
    if x1 == 29999:
        return 289703.0,25.38
    if x1 == 17973:
        return 191488.0,32.72

    return 10000000,0

But that didn't seem to help, and I didn't see any print statements in the terminal so I don't think it was being invoked.

Let me know what extra debugging information might be needed! Looking forward to getting this working, would really help my experiment :)

Tuning strategy

Many thanks for this promising work.

Currently, I am applying AutoOED to explore the performance space in my project (3 variables to 2 objectives, and we don't know the objective function). However, the model prediction error increases as the evaluation add, please refer to the figure for more details. In this case, the ranges of 2 objectives are supposed to be limited in [-30, 30], can I constrain these ranges in the code?

Moreover, I wonder how to select the tuning strategy for Gaussian Process, I have tried to change the hyperparameter nu but it didn't work. Could you please share some tunning experiences?

I will appreciate your help.

Best

Mixed variables

Hi there! Great to see a useful tool for MOBO, thanks for sharing the code!

I am quite interested in mixed variables (on my way learning how to implement this), I saw in your original code you did the transformation to change the variables to one-hot encoding. Wonder if there is any changes regarding the acquisition function to deal with mixed variables, could you share some insights on this?

Any advice would be much appreciated!!

There are some problems in program testing, I hope you have a look.

Your project is very valuable. When I experience the relevant function, run_team_scientist.py, the basic method described in the documentation is that the whole program stops responding after the manual optimization is started and before the optimization is finished. I hope you can check it out.

Python version not compatible with matlabengine

Hello,
It appears that you are utilizing Python version 3.7.7 as specified in [environment.yml], which seems to be no longer compatible with the Matlab Engine of newer versions. As a result, optimization using Matlab scripts (e.g. the simulation example) is not possible.