Comments (4)
Hi @ThePauliPrinciple ,
Thanks for your nice question and work with our code!
Re
checking if I get close to a walltime on a compute cluster
we do have support for a fixed walltime bound: https://github.com/mir-group/nequip/blob/develop/configs/full.yaml#L210-L211. But if you want to query the job scheduler for example that will have to be custom of course.
I've just added support for custom early stopping conditions on branch: https://github.com/mir-group/nequip/tree/feature-custom-early-stop with an example at https://github.com/mir-group/nequip-example-extension/tree/earlystop. Please give this a try and let me know if it works for you, and I'll merge it down.
If this doesn't fully solve the issue (or even if it does), it might be a more complicated workflow than I'm anticipating, and maybe we should have a quick call to discuss---please feel free to send me an email at the address listed in my profile.
from nequip.
This looks good to me.
Passing the trainer object to the stopper might be useful to some, although for my use case I am only interested in "external" information.
I'm not exactly certain what the comment about restarting means, in particular, when is a stopper considered "stateful"?
The original early stopper also returned values to immediately debug/print, maybe that's also nice to add.
from nequip.
Great!
A stopper is "stateful" when it maintains a state like, say, how many epochs the validation loss hasn't improved (like the patience setting) or what the minimum observed value was (see https://github.com/mir-group/nequip/blob/feature-custom-early-stop/nequip/train/early_stopping.py#L120-L121). If it only depends on the current arguments to the object, and not any state stored in your custom object, then it's not stateful. (State of the trainer, if that was passed in, will be correctly preserved across restarts.)
The original early stopper also returned values to immediately debug/print, maybe that's also nice to add.
What do you mean, exactly?
from nequip.
nequip/nequip/train/early_stopping.py
Line 98 in c56f48f
Here
debug_args
is returned, which is printed to the log:nequip/nequip/train/trainer.py
Lines 874 to 882 in c56f48f
from nequip.
Related Issues (20)
- ❓ [QUESTION] Colab tutorial HOT 1
- ❓ [QUESTION] Restart run HOT 1
- issue when using nequip-deploy 🐛 [BUG] HOT 9
- 🐛 [BUG] Cannot restart run with different dataset HOT 4
- Is it possible to train on xyz format data with multiple molecules❓ [QUESTION] HOT 2
- run works on colab but fails on spyder❓ [QUESTION] HOT 4
- ❓ [QUESTION] Custom layer with control structure not supported? HOT 1
- The use of nequip command HOT 1
- 🐛 [BUG] Cannot run nequip-train with provided example HOT 4
- 🌟 [FEATURE]How to Train and Validate on Separate Datasets HOT 2
- ❓ [QUESTION]
- ❓ [QUESTION] About the data class AtomicData HOT 3
- bugs with "initialize_from_state"🐛 [BUG] HOT 5
- how to choose nosehoover the value of nvt_q❓ [QUESTION]
- ❓ [QUESTION]Finetuning Validation Error Higher than Pre-training Error in Nequip
- Confusion about `num_frames` attribute in `HDF5Dataset` HOT 1
- ❓ [QUESTION]how can I use the ase calculator for testing ?
- What is the unit of virials❓ [QUESTION] HOT 3
- MLFF for Silicon
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nequip.