torralba-lab/im2recipe-pytorch

im2recipe Pytorch implementation

im2recipe-pytorch's Issues

Could we also define pytorch version?

Many packages deprecated due to the new version of PyTorch. Could you specify the version of pytorch in this repository?

test.py didn't run

I ran this command, then the below error occurred.
I suppose a reason why this error occurred,
input_var array is initialized, but not appended value in test function.
-- Command

python test.py --model_path=snapshots/model_e220_v-4.700.pth.tar

-- Error

Traceback (most recent call last):
  File "test.py", line 176, in <module>
    main()
  File "test.py", line 80, in main
...
    test(test_loader, model, criterion)
  File "test.py", line 104, in test
    output = model(input_var[0],input_var[1], input_var[2], input_var[3], input_var[4])

mk_dataset.py hasnot taken the images into lmdb

erialized_sample = pickle.dumps( {'ingrs':ingr_vec, 'intrs':st_vecs[partition]['encs'][beg:end],
    'classes':class_dict[entry['id']]+1, 'imgs':imgs[:maxNumImgs]} ) 

with env[partition].begin(write=True) as txn:
    txn.put('{}'.format(entry['id']), serialized_sample)

expected scalar type Float but found Long

I get the following error when training the model

$ python train.py --batch_size 80                                                                                                                                                    [900/1419]
There are 2 parameter groups
Initial base params lr: 0.000100
Initial vision params lr: 0.000000
/home/foodvision/anaconda3/envs/py27/lib/python2.7/site-packages/torchvision-0.2.1-py2.7.egg/torchvision/transforms/transforms.py:188: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize inst$
ad.
Training loader prepared.
Validation loader prepared.
Traceback (most recent call last):
  File "train.py", line 372, in <module>
    main()
  File "train.py", line 115, in main
    train(train_loader, model, criterion, optimizer, epoch)
  File "train.py", line 181, in train
    cos_loss = criterion[0](output[0], output[1], target_var[0])
  File "/home/foodvision/anaconda3/envs/py27/lib/python2.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/foodvision/anaconda3/envs/py27/lib/python2.7/site-packages/torch/nn/modules/loss.py", line 950, in forward
    return F.cosine_embedding_loss(input1, input2, target, margin=self.margin, reduction=self.reduction)
  File "/home/foodvision/anaconda3/envs/py27/lib/python2.7/site-packages/torch/nn/functional.py", line 1794, in cosine_embedding_loss
    return torch.cosine_embedding_loss(input1, input2, target, margin, reduction)
RuntimeError: expected scalar type Float but found Long

Check this if its related: #11

Index error when running test

Hi. I'm trying to run test with the pretrained model but I'm getting this error:

Traceback (most recent call last):
  File "test.py", line 173, in <module>
    main()
  File "test.py", line 34, in main
    model = im2recipe()
  File "/home/lea/im2recipe/trijoint.py", line 120, in __init__
    self.ingRNN_    = ingRNN()
  File "/home/lea/im2recipe/trijoint.py", line 65, in __init__
    self.embs = nn.Embedding(vec.size(0), opts.ingrW2VDim, padding_idx=0) # not sure about the padding idx 
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 98, in __init__
    self.reset_parameters()
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 109, in reset_parameters
    self.weight[self.padding_idx].fill_(0)
IndexError: index 0 is out of bounds for dimension 0 with size 0

How can I test pretrained model?

Hey, I downloaded pretrained model model_e500_v-8.950.pth.tar and test with command python test.py --model_path=model_e500_v-8.950.pth.tar but I got following error:

File "test.py", line 173, in
main()
File "test.py", line 34, in main
model = im2recipe()
File "/trijoint.py", line 120, in init
self.ingRNN_ = ingRNN()
File "/trijoint.py", line 65, in init
self.embs = nn.Embedding(vec.size(0), opts.ingrW2VDim, padding_idx=0) # not sure about the padding idx
File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 110, in init
self.reset_parameters()
File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 121, in reset_parameters
self.weight[self.padding_idx].fill_(0)
IndexError: index 0 is out of bounds for dimension 0 with size 0

can you provide me solution for this.
Thanks.

training on skip-instructions/only ingredients

Please help me training the model on skip-instructions/ingredients only. I think the there are no clear instructions to train the model only to detect ingredients.
thanks

Index Error while training

Hello

I am getting the following error while training :
File "/data/home/ameen.ali/food/im2recipe-Pytorch/data_loader.py", line 103, in __getitem__ path = os.path.join(self.imgPath, loader_path, imgs[imgIdx]['id']) IndexError: list index out of range


ANy idea what is the problem ?

Memory requirements

Can I know the memory requirements for running this, I keep running into an unexpected bus error even when I use (8GB RAM + 24GB Swap file), also the swapfile does not even get fully occupied when the error occurs. I am using a cpu version of torch to run this.

=> loading checkpoint 'model_e220_v-4.700.pth.tar'
=> loaded checkpoint 'model_e220_v-4.700.pth.tar' (epoch 220)
/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torchvision-0.2.1-py2.7.egg/torchvision/transforms/transforms.py:188: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize instead.
Test loader prepared.
321
i
0
321
test.py:110: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.
v = torch.autograd.Variable(input[j], volatile=True)
test.py:116: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.
v = torch.autograd.Variable(target[j], volatile=True)
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
Exception NameError: "global name 'FileNotFoundError' is not defined" in <bound method _DataLoaderIter.del of <torch.utils.data.dataloader._DataLoaderIter object at 0x7f97abe148d0>> ignored
Traceback (most recent call last):
File "test.py", line 199, in
main()
File "test.py", line 90, in main
test(test_loader, model, criterion)
File "test.py", line 127, in test
output = model(input_var[0],input_var[1], input_var[2], input_var[3], input_var[4])
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/akhil/Documents/im2recipe/im2recipe-Pytorch/trijoint.py", line 134, in forward
visual_emb = self.visionMLP(x)
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 109, in forward
return self.module(*inputs, **kwargs)
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torchvision-0.2.1-py2.7.egg/torchvision/models/resnet.py", line 76, in forward
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
File "/home/akhil/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 178, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 12487) is killed by signal: Bus error.

test error

Hi, what do they mean from line 93~ 104 in test.py? I got an error:
File "test.py", line 104, in test
output = model(input_var[0],input_var[1], input_var[2], input_var[3], input_var[4])
IndexError: list index out of range

the imput_var seams empty.

Licence for using the Code (incl. model)

Your project is of great interest to me and i am very excited to try Your models!

You list Your code under the MIT licence and link Your parameter set (the model) in the read.me.

To actually being able to use Your model/code for inference, one would also need the vocabulary: vocab.bin .

Am i correct in the assumption, that You do not allow inference of Your model in commercial use (as part of a back-end service) ?
Otherwise, leave me a comment (which of course would make me very happy).

Recipe Visulization and new Prediction

I am using Ubuntu 18.04,Pytorch 0.4.1, Py27, no cuda
In test.py, i have deleted all line which format is ".cuda()" since my local computer is not suit the GPU way. Moreover , i haved modified the batch_size in agr.py = 1 instead of 160.
Then, i meet the problem in :
File "test.py", line 107, in test
output = model(input_var[0],input_var[1], input_var[2], input_var[3], input_var[4])
IndexError: list index out of range

This the whole process with the command : python test.py --model_path=snapshots/model_e500_v-8.950.pth.tar
=> loading checkpoint 'snapshots/model_e500_v-8.950.pth.tar'
=> loaded checkpoint 'snapshots/model_e500_v-8.950.pth.tar' (epoch 500)
/home/minh/anaconda2/envs/im2recipe/lib/python2.7/site-packages/torchvision/transforms/transforms.py:188: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize instead.
"please use transforms.Resize instead.")
Test loader prepared.
Exception OSError: OSError(104, 'Connection reset by peer') in <bound method _DataLoaderIter.del of <torch.utils.data.dataloader._DataLoaderIter object at 0x7f1a29958810>> ignored
Traceback (most recent call last):
File "test.py", line 179, in
main()
File "test.py", line 81, in main
test(test_loader, model, criterion)
File "test.py", line 107, in test
output = model(input_var[0],input_var[1], input_var[2], input_var[3], input_var[4])
IndexError: list index out of range

Please give me some advices , Thanks anyway !

Asking the way to get recipe by querying img

Hi,
I want to test imge2recipe(querying image and response recipe).
But, I can't find imge2recipe script in git hub
How can I test imge2recipe in script??
Please guide us
Thank

"recipe1M" dataset

Hello,
How can I download the "recipe1M" dataset or just its two files "layer1.json" and "layer2.json".
Thanks in advance.

Dataset Inaccessible, error mentions missing email?

Files needed to run the pretrained model (vocab.bin if I'm not mistaken) that are in the dataset behind the .edu gate can't be accessed. I dug up my college email just to get an error saying that the im2recipe@csail whatever email couldn't be found or it was inaccessible, something along those lines. I can't register again to get the error to copy it exactly as it believes I've already registered, despite the registration info apparently having gone nowhere.

IndexError on attempt to run pertained model

Running

python test.py --model_path=snapshots/model_e500_v-8.950.pth.tar

results in

File "test.py", line 173, in
main()
File "test.py", line 34, in main
model = im2recipe()
File "/home/sws/im2recipe-Pytorch/trijoint.py", line 120, in init
self.ingRNN_ = ingRNN()
File "/home/sws/im2recipe-Pytorch/trijoint.py", line 65, in init
self.embs = nn.Embedding(vec.size(0), opts.ingrW2VDim, padding_idx=0) # not sure about the padding idx
File "/home/sws/.local/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 98, in init
self.reset_parameters()
File "/home/sws/.local/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 109, in reset_parameters
self.weight[self.padding_idx].fill_(0)
IndexError: index 0 is out of bounds for dimension 0 with size 0

I am in trouble to test it with this error.
What can you advise to do? Fix padding ?

does the repository contains instructions to train the model on skip-instructions.

Pre-trained model can't not find test_lmdb

Description

I'm trying to run the pre trained model using the following command within docker:
(I already configure the vocab.bin file)

python test.py --model_path=snapshots/model_e500_v-8.950.pth.tar

I got this error:

lmdb.Error: data/test_lmdb: No such file or directory

I can't find this file either in the repository or in this page http://im2recipe.csail.mit.edu/dataset/download/

train error

@nhynes hi , I've executed the code python train.py --batch_size 32 with the batch size 32，
but I got an error:

Can you please advise? thank you very much!

Registration to access dataset results in error

Not really an issue with this repository, but I'm unable to register for access to the dataset. Upon submitting the registration form at

returns the following error:

Django Material
Server error

There's been an error. It's been reported to the site administrators via email and should be fixed shortly. Thanks for your patience.

Following the error, I'm unable to login with the account I tried to setup.

Error installing torchwordemb

I'm trying to test the pretrained model but got the following error:

Traceback (most recent call last): File "test.py", line 13, in <module> from trijoint import im2recipe File "/project/recipe1m/im2recipe-Pytorch/trijoint.py", line 9, in <module> import torchwordemb ModuleNotFoundError: No module named 'torchwordemb'

but then I get the error when doing pip install torchwordemb

/project/conda_env/recipe1m/lib/python3.7/site-packages/torch/include/ATen/core/dispatch/Dispatcher.h: In member function ‘Return c10::Dispatcher::doCallUnboxed(const c10::Dispatc hTable&, const c10::LeftRight<ska::flat_hash_map<c10::TensorTypeId, c10::KernelFunction> >&, Args ...) const [with Return = bool; Args = {}]’: /project/conda_env/recipe1m/lib/python3.7/site-packages/torch/include/ATen/core/dispatch/Dispatcher.h:191:1: warning: control reaches end of non-void function [-Wreturn-type] } ^
error: command 'gcc' failed with exit status 1

Has anyone had success running test.py and/or installing torchwordemb? 

test.py on existing model gives same results as random ranking

So, I initially got the same error as #12 and solved it using the solution there and also went through all the other issues to solve existing problems. After this, I run test.py and I get MedR and recall results comparable to random ranking results mentioned in the CVPR paper. So, maybe #12 is not solved after all. Also, maybe the model that I am loading (~220 epochs one) has random weights, but that is unlikely.

Another reason could be that I had to replace volatile=True in torch.autograd.Variable with torch.no_grad() wrapper. These are the changes I made to the code:

for i, (input, target) in enumerate(test_loader):
        input_var = list() 
        for j in range(len(input)):
            with torch.no_grad():
	    	v = torch.autograd.Variable(input[j])
	    input_var.append(v.cuda() if not opts.no_cuda else v)
        target_var = list()
        for j in range(len(target)-2): # we do not consider the last two objects of the list
            with torch.no_grad():
            	v = torch.autograd.Variable(target[j])
            target_var.append(v.cuda() if not opts.no_cuda else v)

If the @nhynes or anybody who was able to recreate the results could chip in, it would be extremely helpful. I am using a batch size of 60 for testing since that is what my GPU can handle.

cos_loss ~ 0.9
img_loss ~ 9.2
rec_loss ~ 2.2

DB Download link not working

Dear authors,

I am trying to download your dataset "Recipe1M+ dataset" from the provided link:
http://im2recipe.csail.mit.edu/dataset/login/

When I tried to log in it mentioned the This account is inactive.

I have not received any email for confirming the registration.

I tried to create a new user with different emails but some errors raised, and now it says the account has been taken.

I would really appreciate it if you can kindly follow up and send me the dataset link.

Best,
Allen

IndexError on attempt to run pertained model

Running

python test.py --model_path=snapshots/model_e500_v-8.950.pth.tar

results in

  File "/home/q/Projects/im2recipe-Pytorch/trijoint.py", line 65, in __init__
    self.embs = nn.Embedding(vec.size(0), opts.ingrW2VDim, padding_idx=0) # not sure about the padding idx
  File "/home/q/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 98, in __init__
    self.reset_parameters()
  File "/home/q/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 109, in reset_parameters
    self.weight[self.padding_idx].fill_(0)
IndexError: index 0 is out of bounds for dimension 0 with size 0

I'm using python3.7 fresh conda environment. From other issues I get that I should not have encountered such error. What can you advise to do? Fix padding ?

Training data size

Hi I'm trying to process the data from scratch. But I don't know why the training data size is 238459 instead of 720639 as claimed in the CVPR paper. (I also ran "python tokenize_instructions.py train" and got the output file "tokenized_instructions_train.txt" of 720639 lines.)
I'd appreciate it if you could tell me why and how you chose the 238459 samples. Thank you.

food101_classes_renamed.txt

Where can I download this file?

Problem of running test.py

It seems test.py will not extract recipe embedding features and image embedding features. The code is running for more than 4 hours, but nothing has been saved (still running). I have checked the test.py, it seems that no data fed into the model. I think there may be something wrong with the test function.
%----------------------------------------------------------------------
for i, (input, target) in enumerate(test_loader):
input_var = list()
for j in range(len(input)):
v = torch.autograd.Variable(input[j], volatile=True)
target_var = list()
for j in range(len(target)-2): # we do not consider the last two objects of the list
target[j] = target[j]
v = torch.autograd.Variable(target[j], volatile=True)
target_var.append(v.cuda() if not opts.no_cuda else v)

    # compute output

output = model(input_var[0],input_var[1], input_var[2], input_var[3], input_var[4])
%---------------------------------------------------------------------------------------------------
From the code, input_var is not updated and keeps empty. Could you help me address this?

Running a Single Image file To predict output

Can anyboy help how to provide a single image file to the model and get the output ?

Training strategy doesn't lead to a decreasing loss

The training strategy, specifically the random creation of negative samples at each epoch just doesn't seem to work. The loss doesn't decrease even after 100+ epochs and the performance on validation set remains poor even after this much training. It would be such a huge help if you could post a guideline on how the loss function should behave or how the loss curves should look like for this implementation.

Thanks,
Avijit

a question about the test.py

what's the meaning of the 'target[j] = target[j]' in the line 99 in test.py ???

Invalid device id

I'm facing the following issue, when running the pretrained model

$ python test.py
Traceback (most recent call last):
  File "test.py", line 176, in <module>
    main()
  File "test.py", line 31, in main
    model.visionMLP = torch.nn.DataParallel(model.visionMLP, device_ids=[0,1,2,3])
  File "/home/foodvision/anaconda3/envs/py27/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 111, in __init__
    _check_balance(self.device_ids)
  File "/home/foodvision/anaconda3/envs/py27/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 17, in _check_balance
    dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]
  File "/home/foodvision/anaconda3/envs/py27/lib/python2.7/site-packages/torch/cuda/__init__.py", line 292, in get_device_properties
    raise AssertionError("Invalid device id")
AssertionError: Invalid device id

CUDA device_count output is

$ torch.cuda.device_count()
1L

Output of nvidia-smi

$ nvidia-smi
Tue Nov 20 12:19:10 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44                 Driver Version: 396.44                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 000023D3:00:00.0 Off |                    0 |
| N/A   35C    P0    73W / 149W |      0MiB / 11441MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Any help would be appreciated. Thanks

Visualization

In the lua version, you have a section to see the visualization of the trained model, how this will be achieved with Pytorch once the model is trained?

Selfie in 3/9/3/e/393efa5963.jpg (test set)

I was playing with the test set and found someone's selfie mixed into the data.

`TypeError: file must have 'read' and 'readline' attributes` when loading training data

First of all, thanks a lot for the great public code!

I am currently trying to reproduce the paper using the recipe1M dataset. I want to first reproduce it here, as downloading the other recipe1M+ dataset takes considerable time.

When I run the training algorithm, I get the following error:

Getitem from  195360 train
Transaction is:  <Transaction object at 0x7fb85a311810>
Reading from : b'c7d39222ec'
Serialized samples are:  25150
    data = self._next_data()
  File "/home/david/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
    return self._process_data(data)
  File "/home/david/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
    data.reraise()
  File "/home/david/venv/lib/python3.7/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/david/venv/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/david/venv/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/david/venv/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/david/src/importer/image_loader.py", line 165, in __getitem__
    sample = pickle.load(serialized_sample, encoding='latin1')
TypeError: file must have 'read' and 'readline' attributes

Specifically, the image loaders __getitem__ function does not seem to work properly with the provided lmdb files. Notice that I renamed some variables names to make the code easier to understand. I even ran mk_dataset.py again to re-create the lmdb files, but this doesn't seem to help. Am I missing something?

    def __getitem__(self, index):

        print("Getitem from ", index, self.partition_name)

        assert self.ids, ("Self ids is not defined!", self.ids)

        if self.partition_name == "train":
            match_bool = np.random.uniform() > self.mismatch_ratio
        elif self.partition_name in ("val", "test"):
            match_bool = True
        else:
            assert False, ("Partition name not well defined", self.partition_name)

        target = match_bool and 1 or -1

        with self.env.begin(write=False) as transaction:
            print("Transaction is: ", transaction)
            print("Reading from :", self.ids[index].encode('latin1'))
            serialized_sample = transaction.get(self.ids[index].encode('latin1'))
        print("Serialized samples are: ", len(serialized_sample))

        sample = pickle.loads(serialized_sample, encoding='latin1')

Am I missing something? Any help is greately appreciated!

Extract Word Embedding for a single recipe

Hi,

Can anyone help with extracting only word embedding using pre-trained model for given input recipe instructions.

Thanks

Question about the CCA protocol

Hi, guys. I am also using CCA algorithm to perform dimension reduction, i.e. using the api from sklearn. However, this operation may require more than a whole day to achieve that goal, from 2048-dims to 1048-dims. So I wonder if there is another solution to accelerate the procedure of this operation?

setup.py

Where is the setup.py file?

ONNX Version?

First, thank you for this model and this research. It's been a big help in my process for learning ML and PyTorch.

I was wondering if it would be possible to provide your pre-trained model in an ONNX format? My end goal here would be to use it on iOS and Android, and both platforms have hardware accelerated inference engines (CoreML/Tensorflow Lite respectively.) My initial reading has lead me to believe it's more compact and generally faster to use the native solutions rather PyTorch itself on mobile. Thoughts? It appears PyTorch can output to the ONNX format, but I'm not 100% sure I understand enough of the inputs and outputs of this model to do it correctly myself.

Very confused about the test.py

I tried the following code:
python test.py --model_path=model/model_e220_v-4.700.pth.tar --img_path=./data/Tomato-and-egg-stir-fry-ingredients.jpg --data_path=./data/, but it turns out the following error:

    print(..., file=sys.stderr)
          ^
SyntaxError: invalid syntax

What is the right way to load data?
What if I would like to test a single image or a group of images?

It seems that test.py has --cuda option, but...

Can I use test.py and pretrained data without cuda?
I look forward to hearing from you.

vocab size

I have tried the testing code with the pre-trained model, however, it reports the error "runtimeError: While copying the parameter named ingRNN_.embs.weight, whose dimensions in the model are torch.Size([30566, 300]) and whose dimensions in the checkpoint are torch.Size([30167, 300])."

im2recipe-Pytorch issue

I wanted to try to train your model using a new dataset. Following your instructions, I would like to launch the script "tokenize_instructions.py", which however requires the "det_ingrs.json" file. I can't figure out how you got that file, through a particular script or some other way? and if so, how can I get it for the new recipe dataset I would like to use.

The paper talks about a model " The initial ingredient name extraction task is solved by a bi-directional LSTM that performs logistic regression on each word in the ingredient text", but I couldn't find the code associated with this model, which I think could be the one used to then obtain the "det_ingrs.json" file, but i'm not sure.

If you could help me with these doubts I would be very grateful.

I got Cuda Error when I run test.py ,And My soft environment is ubuntu16.04 python3.7 cpu

/home/pc/anaconda3/envs/py3.7/bin/python /home/pc/work/yq_work/im2recipe-Pytorch/test.py
test begin!
Traceback (most recent call last):
File "/home/pc/work/yq_work/im2recipe-Pytorch/test.py", line 176, in
main()
File "/home/pc/work/yq_work/im2recipe-Pytorch/test.py", line 36, in main
model = im2recipe()
File "/home/pc/work/yq_work/im2recipe-Pytorch/trijoint.py", line 102, in init
resnet = models.resnet50(pretrained=True)
File "/home/pc/anaconda3/envs/py3.7/lib/python3.7/site-packages/torchvision/models/resnet.py", line 261, in resnet50
**kwargs)
File "/home/pc/anaconda3/envs/py3.7/lib/python3.7/site-packages/torchvision/models/resnet.py", line 223, in _resnet
progress=progress)
File "/home/pc/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/hub.py", line 506, in load_state_dict_from_url
return torch.load(cached_file, map_location=map_location)
File "/home/pc/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/serialization.py", line 529, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/pc/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/serialization.py", line 702, in _legacy_load
result = unpickler.load()
File "/home/pc/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/serialization.py", line 665, in persistent_load
deserialized_objects[root_key] = restore_location(obj, location)
File "/home/pc/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/serialization.py", line 156, in default_restore_location
result = fn(storage, location)
File "/home/pc/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/serialization.py", line 132, in _cuda_deserialize
device = validate_cuda_device(location)
File "/home/pc/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/serialization.py", line 116, in validate_cuda_device
raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

Process finished with exit code 1

how does the remove1m.txt be generated

everything is in the title

Training model with batch size 40

I've executed the code python train.py --batch_size 40 with the batch size 40 (since my GPU was failing on 160 and 80).

I estimated 12 days since I have lowered the batch size about 3x time and again it's like 14 days now. Can you please advise, when will it be done?

Pretrained model link broken

The link "http://data.csail.mit.edu/im2recipe/model_e500_v-8.950.pth.tar" leads to a 502 Bad Gateway.