Comments (3)
working here, have trained a resnet50 model using checkpoints, thanks again for the notes!
from swift-models.
This is due to a shortcoming in the current checkpoint reader's iteration implementation. When I first put this together, there wasn't a way to use reflection to determine if a property was writeable, so it was possible to have the reflection part of the iteration find a path that shouldn't be part of the checkpoint. As a hacky way around this, I let you specify properties that you can ignore for reading and writing via ignoredTensorPaths
in the Checkpointable protocol.
In the case of ResNet (and anything else that uses BatchNorm), inside of BatchNorm are the properties runningMean
and runningVariance
that are not to be used inside a checkpoint. You need to manually ignore these by adjusting the conformance to
extension ResNet: Checkpointable {
public var ignoredTensorPaths: Set<String> {
return ["BatchNorm<Float>.runningMean", "BatchNorm<Float>.runningVariance"]
}
}
The good news is that the new reflection enhancements upstream will allow for immutable properties to be determined, and this workaround can go away. However, because that's not supported in all of our toolchains, I held off on reworking this to use that interface.
Additionally, there's also a current flaw in the checkpoint reading where you'll need to change your code above to read
try model.readCheckpoint(from: temporaryDirectory.appendingPathComponent("ResNet"), name: "ResNet")
for it to find the written checkpoint. I need to align the directory naming between reading and writing checkpoints to do away with this inconsistency. Sorry about the implementation flaws here.
from swift-models.
@BradLarson thanks for the notes, that works here!
from swift-models.
Related Issues (20)
- Find a more descriptive word for `numericalized` HOT 2
- [WordSeg] Alphabet can only contain characters from training
- [WordSeg] Use `Character` instead of `String` in `Alphabet`
- [WordSeg] Make backend configurable
- Use default word separator in CLI arguments
- Transformer model requires more parameters than supported on TPU HOT 2
- Implementing graph convolutional neural networks HOT 1
- Add documentation to the Gym models to specify additional dependencies for DQN
- Documentation for loading custom dataset HOT 5
- Add logging callbacks for training statistics HOT 1
- GPT-2 currently exhausts all available GPU memory on an 8 GB GPU HOT 2
- Better error message on unzip failure HOT 2
- early program exit with xla mode under linux HOT 12
- No dilations in DepthwiseConv2D HOT 2
- macOS S4TF 0.11 Build Failure: Function is not Differentiable HOT 2
- What is TF-427? HOT 1
- Complex Loss Functions Inside TrainingLoop HOT 3
- Checkpointing should be extended to optimizer and training loop state
- Enable Discussions?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from swift-models.