Tuning strategy

[yunshengtian]

Tuning strategy

[yunshengtian]

Thanks for bringing up this issue. I'd like to help with you but I have a couple of questions in this case:

• Would you let me know which algorithm or algorithmic module combination that you are using? E.g. DGEMO or TSEMO, etc.?
• What's your problem like? Is it evaluated in a kind of real-world lab setup? Is the evaluation noisy?
• What happened after the 200th evaluation? It seems like there's an obvious turning point. Did you change any settings?

It's hard to tell without more information, unfortunately. Since people use BO for drastically different problems.

In terms of parameter tuning, actually, we don't tune a lot. For nu we mainly use 1 or 5. But it's known that Gaussian Process will sometimes predict very extreme values if the underlying dataset is somewhat noisy.

Currently, the bounds on the objectives are not supported, but I think it should be relatively easy to add that in a naive way, perhaps like post-processing on the surrogate model's evaluation result.

[KangLiao929]

Thanks for the quick reply. Following your comments, I list more details as follows.

• I used the DGEMO algorithm. For other settings, the Surrogate model: Gaussian Process, Acquisition function: Identity, Solver: ParetoFrontDiscovery (n_gen-10, pop_size-100), Selection: Direct.
• Yes, our problem aims to the laser marking task, i.e., marking the picture on the steel using laser, we need to find the relationship between design space (laser parameters) and performance space (colors) while investigating the diversity of colors. Its evaluations are somehow noisy.
• On the evaluation, we first generated the 100 parameters using AutoOED, and then we used these parameters to mark the steel and obtained the corresponding performance space. Subsequently, the measured colors are loaded into AutoOED and we set the batch size to 100 to generate the next 100 parameters. It seems the model predicts well in the first generation (100~200 as shown in the Figure). For the second generation, I remembered I didn't do anything. Before the marking of this generation, I saw the predicted values are very large.

Thanks for your tuning and other suggestions. It's not sure if I apply AutoOED to our problem correctly and I am looking forward to your further comments.

[KangLiao929]

PS: I found the 0.2.0 version of AutoOED may induce an inverse exploration issue. In my case, I want to maximize two objectives. However, the predicted values seem to exhibit in the minimum region as shown in the following figures (the first iteration and second iteration). Could you please give some suggestions? I have tried different initializations but occurred the same inverse exploration. For the 0.1.0 version of AutoOED, the direction of the exploration is correct.

[yunshengtian]

Thanks for the feedback! This looks like a bug to me. I'll take a look very soon and keep you updated.

[KangLiao929]

Thanks for the feedback! This looks like a bug to me. I'll take a look very soon and keep you updated.

Thanks a lot!

[KangLiao929]

Thanks for the feedback! This looks like a bug to me. I'll take a look very soon and keep you updated.

Maybe the issue exists in the pareto.py. In this script, you call the function convert_minimization (Convert maximization to minimization). I don't understand the reason for this operation. Could you please explain it?

[yunshengtian]

I've fixed that bug in the latest commit. Please pull if you're working on the source code, otherwise, if you're installing through the executable files, I will upload a new version later today.

Thanks again for pointing out this important issue! Your intuition is right, something is wrong with the minimization conversion. That is needed because our underlying optimization algorithm essentially is a minimizer, so we need to convert the maximization problem to the equivalent minimization problem. But the bug was that I converted it twice somewhere in the code. But I've fixed that now and tested on both a maximization problem and a minimization problem. It should work fine now. Let me know if you still have any other problems?

[KangLiao929]

I've fixed that bug in the latest commit. Please pull if you're working on the source code, otherwise, if you're installing through the executable files, I will upload a new version later today.

Thanks again for pointing out this important issue! Your intuition is right, something is wrong with the minimization conversion. That is needed because our underlying optimization algorithm essentially is a minimizer, so we need to convert the maximization problem to the equivalent minimization problem. But the bug was that I converted it twice somewhere in the code. But I've fixed that now and tested on both a maximization problem and a minimization problem. It should work fine now. Let me know if you still have any other problems?

Thanks a lot for addressing this problem! I am working on the source code and I also tried to revise the part using the convert_minimization function. Now, it works for me and I have no further questions.

[yunshengtian]

Great to hear that. Please feel free to reach out to us once you have any other issues.