brain-bzh / efficient-deep-learning Goto Github PK
View Code? Open in Web Editor NEWMaterial for course "Efficient Deep Learning" at IMT Atlantique
License: Other
Material for course "Efficient Deep Learning" at IMT Atlantique
License: Other
In the readme, the links for courses 1, 2, 3 and 5 are broken. The hyperlink redirects to ".../cours#_withnotes.pdf" instead of cours#.pdf". The old files were replaced by the new versions in previous commits, but the hyperlinks not. :)
We used the DenseNet model cloned from Kuangliu. In this code the pooling layer and ReLu layer are defined by:
out = F.avg_pool2d(F.relu(self.bn(out)), 4)
Instead of using:
def __init__():
...
self.avg_pool2d = nn.AvgPool2d(4)
self.Relu = nn.ReLU()
def forward(self, x):
...
out = self.Relu(out)
out = self.avg_pool2d(out)
...
In the profile.py
, as the code:
def add_hooks(m):
if isinstance(m, nn.Conv2d):
m.register_forward_hook(count_conv2d)
elif isinstance(m, nn.BatchNorm2d):
m.register_forward_hook(count_bn2d)
elif isinstance(m, nn.ReLU):
m.register_forward_hook(count_relu)
elif isinstance(m, (nn.AvgPool2d)):
m.register_forward_hook(count_avgpool)
elif isinstance(m, nn.Linear):
m.register_forward_hook(count_linear)
elif isinstance(m, nn.Sequential):
m.register_forward_hook(count_sequential)
else:
print("Not implemented for ", m)
So the scoring system will not count the pooling layers and ReLu layers and then shows a fake score slightly better than the real one.
In practise, I here use our best model as an example. By changing replace this line of original code by a nn.module
version:
# out = F.avg_pool2d(F.relu(self.bn(out)), 4)
out = self.bn(out)
out = self.relu(out)
out = self.avg_pool2d(out)
I got a different number of operations as:
# before
Flops: 22675830.0, Params: 59573.0
Score flops: 0.027177419494021596 Score Params: 0.010662824878051312
Final score: 0.03784024437207291
#after
Flops: 22678390.0, Params: 59573.0
Score flops: 0.027180487703383927 Score Params: 0.010662824878051312
Final score: 0.03784331258143524
We can also notice that before the change, pooling and relu layers are not printed.
If a "competitor" didn't notice that, it would be (slightly, if not intented) unfair to others. So I think maybe it would be better to optimize profile.py
to take this issue into account or give a standard implementation of models to everyone.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.