Comments (3)
Much appreciated. I will incorporate in the next update. Is there a 2.11 compatible way for caching an optimizer with its internal state?
NB the main reason I've added this is because I am struggling with AWS SageMaker's propensity to die on me with 504 Gateway timeouts. The machine dies; I have to stop/start/and re-run the whole lot again. Caching at at least allows to continue training at the last caching point.
Do you happen to have an idea how to address the issue of AWS dying?
from deephedging.
I haven't dug into the details to be frank, but I found that SciKeras had a similar problem. See discussion here. They seem to since have implemented another approach, with new util functions found here.
As for SageMaker, are you talking about notebook instances? Sounds like it could a RAM issue. Have you tried running it on a larger instance?
from deephedging.
from deephedging.
Related Issues (2)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deephedging.