Add new hyperparameter optimization

[RandomDefaultUser]

Add new hyperparameter optimization

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Jan 27, 2021, 11:56

I've notices the NaN errors as well. This seems to come from the correlation calculation when some values of the jacobian are very small. Might be helpful to introduce some offset there, moving all values away a bit from zero.

Also the size of the correlation matrix scales square with the batch size (unsurprisingly), so expect seriously degraded performance with larger batch sizes - For the example data Batch sizes >>1000 quickly meant that actually training the network was faster than calculating the surrogate metric.

By Nils-Hoffmann on 2021-01-27T11:56:55 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Jan 26, 2021, 15:49

https://arxiv.org/abs/2006.04647: This is the paper Nils' implementation is based upon.

By Fiedler, Lenz (FWU) - 146409 on 2021-01-26T15:49:34 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Dec 21, 2020, 12:18

Asignees: Parvez, Nils

By Fiedler, Lenz (FWU) - 146409 on 2020-12-21T12:18:12 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Jan 26, 2021, 16:18

Some notes for the merge:

• there's a spurious NaN error that sometimes occurs when running Nils' implementation after applying the validation loss fix. Investigate that
• Parvez' implementation needs to install the oapackage and for that needs SWIG, add that to the requirements.

By Fiedler, Lenz (FWU) - 146409 on 2021-01-27T11:56:55 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Jan 28, 2021, 10:21

Yes I have also tracked it down to the correlation calculation and small values. It would definitely be interesting to test some sort of offset there, because values close to zero will be kind of common I think.

The batch size dependence is very interesting! That is sort of inverse to regular hyperparameter optimization on GPUs for which larger batch sizes are generally very beneficial.

By Fiedler, Lenz (FWU) - 146409 on 2021-01-28T10:21:57 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Feb 3, 2021, 18:06

I refactored the @Nils-Hoffmann and @Zevrap-81 code a bit do be more consistent in use with the rest of the code. I am doing a final test on hemera and should that be successful I am closing this issue and addin new, more fine grained issues for the other things we want to investigate/fix in the context of Hyperparameter optimization.

By Fiedler, Lenz (FWU) - 146409 on 2021-02-03T18:19:44 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Feb 3, 2021, 18:19

Sounds great! The investigation of both "NAS without training" and "orthogonal arrays" on a real-world dataset will be very interesting.

By Cangi, Dr. Attila (FWU) - 139621 on 2021-02-03T18:19:44 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Feb 4, 2021, 11:41

Final tests were successful, I am running the first bigger test for the NAS without training and will create an application issue for further hyperparameter calculations.

By Fiedler, Lenz (FWU) - 146409 on 2021-02-04T11:41:45 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Dec 21, 2020, 09:31

changed the description

By Fiedler, Lenz (FWU) - 146409 on 2020-12-21T09:31:13 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Dec 21, 2020, 09:50

changed the description

By Fiedler, Lenz (FWU) - 146409 on 2020-12-21T09:50:21 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Dec 21, 2020, 11:32

changed the description

By Fiedler, Lenz (FWU) - 146409 on 2020-12-21T11:32:09 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Dec 21, 2020, 11:51

assigned to @Nils-Hoffmann

By Fiedler, Lenz (FWU) - 146409 on 2020-12-21T11:51:35 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Dec 21, 2020, 12:14

assigned to @Nils-Hoffmann and unassigned @Zevrap-81

By Fiedler, Lenz (FWU) - 146409 on 2020-12-21T12:14:51 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Dec 21, 2020, 12:14

assigned to @Nils-Hoffmann and unassigned @fiedle09

By Fiedler, Lenz (FWU) - 146409 on 2020-12-21T12:14:32 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Dec 21, 2020, 12:13

assigned to @Zevrap-81 and unassigned @Nils-Hoffmann

By Fiedler, Lenz (FWU) - 146409 on 2020-12-21T12:13:37 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Dec 21, 2020, 12:14

assigned to @Nils-Hoffmann and unassigned @Zevrap-81

By Fiedler, Lenz (FWU) - 146409 on 2020-12-21T12:14:06 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Dec 21, 2020, 12:14

assigned to @fiedle09 and unassigned @Nils-Hoffmann

By Fiedler, Lenz (FWU) - 146409 on 2020-12-21T12:14:26 (imported from GitLab)

[RandomDefaultUser]

In GitLab by @RandomDefaultUser on Dec 21, 2020, 12:14

assigned to @Zevrap-81 and unassigned @Nils-Hoffmann

By Fiedler, Lenz (FWU) - 146409 on 2020-12-21T12:14:42 (imported from GitLab)