fangyh09/image2lmdb

Convert image folder to lmdb, adapted from Efficient-PyTorch

Image2LMDB

Convert image folder to lmdb, adapted from https://github.com/Lyken17/Efficient-PyTorch

.
├── folder2lmdb.py
├── img
│   ├── train
│   │   ├── bar_dir
│   │   │   ├── 100000.jpg
│   │   │   ├── 100001.jpg
│   │   │   ├── 100002.jpg
│   │   │   ├── 100003.jpg
│   │   │   ├── 100004.jpg
│   │   │   ├── 100005.jpg
│   │   │   ├── 100006.jpg
│   │   │   ├── 100007.jpg
│   │   │   ├── 100008.jpg
│   │   │   └── 100009.jpg
│   │   └── foo_dir
│   │       ├── 100000.jpg
│   │       ├── 100001.jpg
│   │       ├── 100002.jpg
│   │       ├── 100003.jpg
│   │       ├── 100004.jpg
│   │       ├── 100005.jpg
│   │       ├── 100006.jpg
│   │       ├── 100007.jpg
│   │       ├── 100008.jpg
│   │       └── 100009.jpg
│   
│   
├── main.py
├── README.md
└── requirements.txt

Convert image folder to lmdb

python folder2lmdb.py img

Test it

python main.py img/train.lmdb

key 0
key 1
torch.Size([2, 224, 224, 3])
key 2
key 3
torch.Size([2, 224, 224, 3])
key 4
key 5
torch.Size([2, 224, 224, 3])
key 6
key 7
torch.Size([2, 224, 224, 3])
key 8
key 9
torch.Size([2, 224, 224, 3])
key 10
key 11
torch.Size([2, 224, 224, 3])
key 12
key 13
torch.Size([2, 224, 224, 3])
key 14
key 15
torch.Size([2, 224, 224, 3])
key 16
key 17
torch.Size([2, 224, 224, 3])
key 18
key 19
torch.Size([2, 224, 224, 3])

Original Repo:

https://github.com/Lyken17/Efficient-PyTorch

image2lmdb's Issues

what is "train_images_idx.txt" for ImageNet?

what is "train_images_idx.txt" for ImageNet?

Can the generated dataset in lmdb format be used in FuDanOCR?

Thank you very much for the code you provided. I recently wanted to use my own dataset for FuDanOCR, but the question about the format of the dataset is very troublesome. Can the dataset in lmdb format generated by your method be used in FuDanOCR? I would be very grateful if you could tell me.

Training speed

Hi, thanks for this script!
Will converting the ImageFolder to the LMDB file make training faster?

MemoryError

In my dataset (datasize:500000 images), I get MemoryError when dealing with 85000 images. What could be the reasons? Thanks in advance.

[80000/500000]
[85000/500000]
File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 804, in next
idx, data = self._get_data()
File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 771, in _get_data
success, data = self._try_get_data()
File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 724, in _try_get_data
data = self.data_queue.get(timeout=timeout)
File "/usr/lib64/python3.6/multiprocessing/queues.py", line 113, in get
return _ForkingPickler.loads(res)
File "/usr/local/lib64/python3.6/site-packages/torch/multiprocessing/reductions.py", line 290, in rebuild_storage_fd
shared_cache[fd_id(fd)] = StorageWeakRef(storage)
File "/usr/local/lib64/python3.6/site-packages/torch/multiprocessing/reductions.py", line 49, in setitem
dict.setitem(self, key, storage_ref)
MemoryError

My dataloader keeps the same as the github.