huyvnphan / pytorch_cifar10 Goto Github PK
View Code? Open in Web Editor NEWPretrained TorchVision models on CIFAR10 dataset (with weights)
License: MIT License
Pretrained TorchVision models on CIFAR10 dataset (with weights)
License: MIT License
Hi there, I'm thinking of using this model for some school work (I'll cite of course).
I'm trying to save the trained weights from the vgg16 model into an identical vgg16 Net(nn.Module) thing (all the same layers and sizes and I've changed the layer names to match) because I need access to only one of the convolutional layers.
However, when I do this, the new model with the weights from vgg16_bn is acting like it's never been trained.
my_model = vgg16_bn(pretrained=True)
my_model.eval()
torch.save(my_model.state_dict(), 'vgg16_cifar_weights')
and then later in my "from scratch" vgg16
vgg.load_state_dict(torch.load('PytorchCifarModel/vgg16_cifar_weights_scratch3'))
Can you see what I'm doing wrong? This trained model would be so perfect for my project, I'm hoping that it'll work.
I'll put my from scratch vgg16 here as well in case there's an issue there. I know it's really long winded but it needs to be for my proejct. Thank you!!
class Net(nn.Module):
def __init__(self):
super().__init__()
self.maxpool = nn.MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
self.conv1_1 = nn.Conv2d(3, 64, 3, 1, padding=1)
self.batch1_1 = nn.BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv1_2 = nn.Conv2d(64, 64, 3, 1, 1)
self.batch1_2 = nn.BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv2_1 = nn.Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
self.batch2_1 = nn.BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv2_2 = nn.Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
self.batch2_2 = nn.BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv3_1 = nn.Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
self.batch3_1 = nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv3_2 = nn.Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
self.batch3_2 = nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv3_3 = nn.Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
self.batch3_3 = nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv4_1 = nn.Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
self.batch4_1 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv4_2 = nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
self.batch4_2 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv4_3 = nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
self.batch4_3 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv5_1 = nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
self.batch5_1 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv5_2 = nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
self.batch5_2 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.conv5_3 = nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
self.batch5_3 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
self.classifier = nn.Sequential(
nn.Linear(512 * 1 * 1, 4096),
# nn.Linear(512 * 7 * 7, 4096),
nn.ReLU(True),
nn.Dropout(),
nn.Linear(4096, 4096),
nn.ReLU(True),
nn.Dropout(),
nn.Linear(4096, 10),
)
self.relu_f = nn.ReLU(True)
def forward(self, x):
x = self.conv1_1(x)
x = self.relu_f(self.batch1_1(x))
x = self.conv1_2(x)
x = self.relu_f(self.batch1_2(x))
x = self.maxpool(x)
x = self.conv2_1(x)
x = self.relu_f(self.batch2_1(x))
x = self.conv2_2(x)
x = self.relu_f(self.batch2_2(x))
x = self.maxpool(x)
x = self.conv3_1(x)
x = self.relu_f(self.batch3_1(x))
x = self.conv3_2(x)
x = self.relu_f(self.batch3_2(x))
x = self.conv3_2(x)
x = self.relu_f(self.batch3_3(x))
x = self.maxpool(x)
x = self.conv4_1(x)
x = self.relu_f(self.batch4_1(x))
x = self.conv4_2(x)
x = self.relu_f(self.batch4_2(x))
x = self.conv4_2(x)
x = self.relu_f(self.batch4_3(x))
x = self.maxpool(x)
x = self.conv5_1(x)
x = self.relu_f(self.batch5_1(x))
x = self.conv5_2(x)
x = self.relu_f(self.batch5_2(x))
x = self.conv5_3(x)
x = self.relu_f(self.batch5_3(x))
x = self.maxpool(x)
x = self.avgpool(x)
x = x.view(x.size(0), -1)
x = self.classifier(x)
return x
Hey, I loaded your pretrained model for CIFAR-10 using the code given in the readme, and then simply tried performing inference on my own with the model. For some reason, I'm getting very poor accuracy metrics (11%) on the CIFAR 10 test set.
from torchvision.datasets import CIFAR10
dataset_dir = "/content/"
mean = [0.4914, 0.4822, 0.4465]
std = [0.2023, 0.1994, 0.2010]
transform_val = transforms.Compose([transforms.ToTensor(), transforms.Normalize(mean, std)])
dataset = CIFAR10(root= dataset_dir, train=False, transform=transform_val, download = True)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=128, num_workers=4, pin_memory=True)
Then I perform simple inference with y=model(x)
format to get the embedding, and I argmax the 10-dim values I get to obtain my predicted class values.
Also when I normalise, most of the values just turn out to be class 1 after I argmax. Very weird.
Am I doing something wrong here? I get 11% accuracy doing this.
I test resnet18 on the CIFAR10 test set, the accuracy is 81.24%, but the reported acc is over 93.07%
Is my test code wrong?
The code is below:
import torch
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
from model import *
if __name__ == '__main__':
net = resnet18(pretrained=True, device='cuda').cuda()
net.load_state_dict(torch.load('state_dicts/resnet18.pt'))
mean = (0.4914, 0.4822, 0.4465)
std = (0.2471, 0.2435, 0.2616)
t = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(mean, std),
])
dataset = datasets.CIFAR10(root='.', train=False, transform=t, download=True)
dataloader = DataLoader(dataset, batch_size=10, num_workers=0, shuffle=False)
acc = 0.
preds = []
labels = []
for i, (img, label) in enumerate(dataloader):
# if i < 10:
if 1:
img = img.cuda()
pred = net(img).detach().cpu()
pred_indices = pred.argmax(1)
preds.append(pred_indices)
labels.append(label.cpu())
preds = torch.cat(preds)
labels = torch.cat(labels)
print(sum(preds==labels)/len(preds))
Did you use a Softmax in the very last layer? I see that last layer is a fully connected (nn.Linear).
your only show params of every model in the table,but i need FLOPs of every model,can you show me.thanks
I don't see any softmax used on the output, so I get this error when testing (and training) the pretrained network.
python train.py --test_phase 1 --pretrained 1 --classifier resnet18
The `preds` should be probabilities, but values were detected outside of [0,1] range.
which is a reasonable error. :)
this is the part of code that is problematic:
Line 44 in 7a6e1f9
what function should be used on the output of the model ?, softmax?
why is this not in your codes ?
Originally posted by @kfarivar in #12 (comment)
Hi,
The parameters are 32 or16-bit floating-point?
My Lab was looking for a quick set of pretrained networks for use in numerical experiments for CIFAR10. We found this to be a convenient source. You do not have attribution information within the readme, how would you like to be cited?
Hi,
The std in readme file and data.py file is different. Any inputs?
Hi there,
How to solve this AttributeError?
The full Traceback are as follows:
Traceback (most recent call last):
File "train.py", line 82, in
main(args)
File "train.py", line 40, in main
model = CIFAR10Module(args)
File "/home/amax/PyTorch_CIFAR10/module.py", line 34, in init
self.hparams = hparams
File "/home/amax/anaconda3/envs/cifar/lib/python3.6/site-packages/torch/nn/modules/module.py", line 826, in setattr
object.setattr(self, name, value)
AttributeError: can't set attribute
There is no resnet_orig.pt state dict in the state_dict.zip file
When loading the pretrained ResNets, I am getting very low training set accuracy (around 22% on a subset of 1000 images).
This is a sample from my code:
model = cifar10_models.resnet50(pretrained=True)
mean = np.array([0.4914, 0.4822, 0.4465])
std = np.array([0.2023, 0.1994, 0.2010])
self.normalize = transforms.Normalize(mean=mean, std=std)
self.train_loader = DataLoader(
datasets.ImageFolder(training_dir, transform=transforms.Compose([
transforms.RandomCrop(32, padding=4),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
self.normalize,
])),
batch_size=1, shuffle=True,
num_workers=0, pin_memory=True)
Can you please confirm that these weights are the right ones?
I have this problem when I tried to test the "pre-trained models" from the master branch:
RuntimeError: Dataset not found or corrupted. You can use download=True to download it
I tried to use the version from the 2022 branch, but this version does not have a --download-weights option. I have downloaded manually the weights and extracted them into the folder, and the "corruption" message continues.
I had installed the pytorch 1.7.1 and torchvision 0.8.2, because the version 1.7.0/0.7.0 it's not currently available in pytorch repo.
Any solution for this?
In the implementation of VGG you keep the original AdaptiveAvgPool2d layer with output_size = (7, 7).
This layer creates 49 copies of the same value in the case of CIFAR10 images whose size is 32x32.
In the original VGG for ImageNet this layer has no effect because input images have size 224x224 and the size of the output of vgg.features
is a tensor with shape (N, 512, 7, 7).
It is added to allow the system to be used with inputs of arbitrary sizes.
Using 32x32 images from CIFAR10, the output size of vgg.features
is a tensor with shape (N, 512, 1, 1).
In this case the pooling layer repeat the single value 49 times to create a tensor with the desired shape.
The following linear layer has an enormous number of parameters because of this choice.
I think the pooling should be removed.
Hi and thanks for sharing the weights and the models. Correct me if I'm wrong but I beleive what you report as test accuracy is just the validation accuracy and not the final test accuracy of model evaluated on the untouched test set. You should have train/validt/test
--> train_loader/valid_loader/test_loader
but I can only see train and validation loaders and you report test accuracies based on the validation loader if I'm not mistaken?
Hi,
I verified the noted results of 94.07% percent of the vgg16_bn pretrained net. However, when trying to train the vgg16_bn net from scratch using the noted training parameters (minibatch 256, sgd nesterov momentum 0.9, weight decay 0.001, step learning with reduction factor 0.1 every 200 epochs, 600 epochs total) and the given normalization values, I only achieved best at epoch 332 with validation set acc 0.896300 and training set acc 1.000000. Gap of 0.103700.
How come?
Hey there!
First of all thanks for publishing this repo and the weights, it's really helpful!
I was wondering whether you used the default hyperparameters for training that are specified in the argparser of train.py, or whether you used different ones?
I would be particularly interested in the ones for mobilenet_v2 and VGG16, if you can share.
Thanks in advance!
Hi,
I'm wondering how are the pre-trained models obtained? For example, when I'm loading the pre-trained model for resnet18, is this pre-trained model trained using imagenet or still cifar? Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.