Comments (2)
Here is a suggestion about caching the conda-pack environment.
- Add a "--cache" option
- When a new conda env is first specified, e.g.
galyleo options --conda-env <env_name> --conda-yml <env_yml> --cache
, a packed conda environment is stored, e.g in ${HOME}/.galyleo-cache/<env_name>/<env_name>.tar.gz. Also, the<env_yml>
file is stored in this directory. - When the same command line is run again, the specified
env_yml
file is checked against the cachedenv_yml
file. If they are identical, then the cached conda environment is used, otherwise, a new packed conda environment is created. - We may also want to cache the /Miniconda3-latest-Linux-x86_64.sh file.
The cache directory may look like this:
${HOME}/.galyleo-cache
Miniconda3-latest-Linux-x86_64.sh
notebooks-sharing
notebooks-sharing.tar.gz
notebooks-sharing.yml
next-environment-here
I've tried to create a simple bash script to show the logic. It isn't working, but it gives you an idea. On Expanse see pwrose/galyleo-cache.sb.
from galyleo.
@pwrose - Done. But I have not yet deployed the update to Expanse yet. I will do so sometime next week and let you know, if you're interested in testing this new feature. I'd like to make a good pass through and clean up the README.md documentation before pushing the new version into production.
from galyleo.
Related Issues (20)
- Running in debug queue needs specifying amount of memory HOT 2
- Allow users to explicitly set conda distribution version HOT 1
- Enable multi-node deployment of conda environment.yml to local /scratch disks
- Implement error check to make sure environment.yml file exists? And path problem? HOT 1
- Change `module purge` to a `module reset`? HOT 1
- Should we cache links some time?
- Create documentation in README.md regarding deployments and updates of galyleo (on SDSC systems)
- Fix --conda-env and *.yaml file name: variable matching issue
- Potential IFS-related bug in galyleo launch command's checklist section HOT 7
- Remove support for PBS/Torque schedulers HOT 1
- Create simple Makefile with a PREFIX option to install and set GALYLEO_INSTALL_DIR
- Create a Spack package for galyleo
- Allow paths to Singularity container images to be either absolute or relative
- Allow users to specify the use of containers stored remotely in container registries
- Conda-yaml should not be required if conda environment specified HOT 1
- Make command-line parsing of launch options more robust
- Potential conda-pack issue? HOT 2
- Add user-specified --bind mounts when checking jupyter --version HOT 1
- Add user-specified module paths by a new option --modulepathadd HOT 1
- Remove the use of miniconda from galyleo; restrict use of default channels
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from galyleo.