How long time you guys trained the model with a dic including 6031 characters?

[mingtzge]

How long time you guys trained the model with a dic including 6031 characters?

[iredescentone]

How many train data do u use for crnn?

[Ryansanity]

How many train data do u use for crnn?

it generates a new dataset(25000 data) randomly each 3000 epoch

[Wenqiang-Miao]

Hello, in this project, we have 6032 characters, and i generate 6 million train data，with batch_size=128，each epoch cost about 3 hours(I trained about 15 epoch，then stopped，so the total train time is about 2 days....2 weeks seems too long...)， use an TITAN Xp.

I guess the train time depends on the number of characters, but the train time is too long...Maybe you should check your code. You can use command nvidia-smi to check GPU-Util.

By the way，is it convenient to reveal what your task is？ I'm curious why there are so many characters to recognize...

[Ryansanity]

Hello, in this project, we have 6032 characters, and i generate 6 million train data，with batch_size=128，each epoch cost about 3 hours(I trained about 15 epoch，then stopped，so the total train time is about 2 days....2 weeks seems too long...)， use an TITAN Xp.

I guess the train time depends on the number of characters, but the train time is too long...Maybe you should check your code. You can use command nvidia-smi to check GPU-Util.

By the way，is it convenient to reveal what your task is？ I'm curious why there are so many characters to recognize...

I saw your config set: __C.TRAIN.EPOCHS = 580000,so this is your iterations numbers or epoch numbers?
AND i am working on passport OCR.

[iredescentone]

Hello, in this project, we have 6032 characters, and i generate 6 million train data，with batch_size=128，each epoch cost about 3 hours(I trained about 15 epoch，then stopped，so the total train time is about 2 days....2 weeks seems too long...)， use an TITAN Xp.
I guess the train time depends on the number of characters, but the train time is too long...Maybe you should check your code. You can use command nvidia-smi to check GPU-Util.
By the way，is it convenient to reveal what your task is？ I'm curious why there are so many characters to recognize...

In what i m training ,there are 5032 characters, 13million train data,batch_size=128,each epoch cost 107 hours, really 107 hours. use an 2080ti. Our task seems pretty simmilar, and i wonder why mine is so slow, Memory-Usage | GPU-Util=10956MiB / 10989MiB | 56%. GPU-Util seems lower ,i am not using a tfrecord for last train version with 8 million train data, the speed using tfrecord didn't speed up much.
is there any clue? appreciate lot...

[iredescentone]

How many train data do u use for crnn?

it generates a new dataset(25000 data) randomly each 3000 epoch

__C.TRAIN.EPOCHS = 580000, seems to be iterations number .
25000train data? it 's to small to accomplish a recognition task with 20838 characters. I wonder you are not using a tfrecord like me?
as I know, the number classification as 20838 is too big ,i wonder whether the model could do it(convergent).

[Ryansanity]

How many train data do u use for crnn?
it generates a new dataset(25000 data) randomly each 3000 epoch
__C.TRAIN.EPOCHS = 580000, seems to be iterations number .
25000train data? it 's to small to accomplish a recognition task with 20838 characters. I wonder you are not using a tfrecord like me?
as I know, the number classification as 20838 is too big ,i wonder whether the model could do it(convergent).

no, the number of total training data is far more than 25000,i just change another dataset every 3000 epoch(i think it should be iteration, but epoch in author's source code), and change another dataset means generating another 25000 data randomly.

[iredescentone]

How many train data do u use for crnn?
it generates a new dataset(25000 data) randomly each 3000 epoch
__C.TRAIN.EPOCHS = 580000, seems to be iterations number .
25000train data? it 's to small to accomplish a recognition task with 20838 characters. I wonder you are not using a tfrecord like me?
as I know, the number classification as 20838 is too big ,i wonder whether the model could do it(convergent).

no, the number of total training data is far more than 25000,i just change another dataset every 3000 epoch(i think it should be iteration, but epoch in author's source code), and change another dataset means generating another 25000 data randomly.

i see. then you train data should be 75 million. a new way of training to me ,thank you for explain.
then our training speed seems differ not much^_^

[Ryansanity]

How many train data do u use for crnn?
it generates a new dataset(25000 data) randomly each 3000 epoch
__C.TRAIN.EPOCHS = 580000, seems to be iterations number .
25000train data? it 's to small to accomplish a recognition task with 20838 characters. I wonder you are not using a tfrecord like me?
as I know, the number classification as 20838 is too big ,i wonder whether the model could do it(convergent).

no, the number of total training data is far more than 25000,i just change another dataset every 3000 epoch(i think it should be iteration, but epoch in author's source code), and change another dataset means generating another 25000 data randomly.

How many train data do u use for crnn?
it generates a new dataset(25000 data) randomly each 3000 epoch
__C.TRAIN.EPOCHS = 580000, seems to be iterations number .
25000train data? it 's to small to accomplish a recognition task with 20838 characters. I wonder you are not using a tfrecord like me?
as I know, the number classification as 20838 is too big ,i wonder whether the model could do it(convergent).

How many train data do u use for crnn?
it generates a new dataset(25000 data) randomly each 3000 epoch
__C.TRAIN.EPOCHS = 580000, seems to be iterations number .
25000train data? it 's to small to accomplish a recognition task with 20838 characters. I wonder you are not using a tfrecord like me?
as I know, the number classification as 20838 is too big ,i wonder whether the model could do it(convergent).
no, the number of total training data is far more than 25000,i just change another dataset every 3000 epoch(i think it should be iteration, but epoch in author's source code), and change another dataset means generating another 25000 data randomly.

i see. then you train data should be 75 million. a new way of training to me ,thank you for explain.
then our training speed seems differ not much^_^

may i ask you how high your val set accuracy can reach?

[iredescentone]

How many train data do u use for crnn?
it generates a new dataset(25000 data) randomly each 3000 epoch
__C.TRAIN.EPOCHS = 580000, seems to be iterations number .
25000train data? it 's to small to accomplish a recognition task with 20838 characters. I wonder you are not using a tfrecord like me?
as I know, the number classification as 20838 is too big ,i wonder whether the model could do it(convergent).

no, the number of total training data is far more than 25000,i just change another dataset every 3000 epoch(i think it should be iteration, but epoch in author's source code), and change another dataset means generating another 25000 data randomly.

How many train data do u use for crnn?
it generates a new dataset(25000 data) randomly each 3000 epoch
__C.TRAIN.EPOCHS = 580000, seems to be iterations number .
25000train data? it 's to small to accomplish a recognition task with 20838 characters. I wonder you are not using a tfrecord like me?
as I know, the number classification as 20838 is too big ,i wonder whether the model could do it(convergent).

How many train data do u use for crnn?
it generates a new dataset(25000 data) randomly each 3000 epoch
__C.TRAIN.EPOCHS = 580000, seems to be iterations number .
25000train data? it 's to small to accomplish a recognition task with 20838 characters. I wonder you are not using a tfrecord like me?
as I know, the number classification as 20838 is too big ,i wonder whether the model could do it(convergent).
no, the number of total training data is far more than 25000,i just change another dataset every 3000 epoch(i think it should be iteration, but epoch in author's source code), and change another dataset means generating another 25000 data randomly.

i see. then you train data should be 75 million. a new way of training to me ,thank you for explain.
then our training speed seems differ not much^_^

may i ask you how high your val set accuracy can reach?

i use LER(label error rate), last version my loss reach [0.9,1.8], LER is 1%. i think this is not acurrate to Measure my model. cause there is no real scence data in my val set.

[Ryansanity]

label error rate

My method is really hard to converge....

[Wenqiang-Miao]

Hello, in this project, we have 6032 characters, and i generate 6 million train data，with batch_size=128，each epoch cost about 3 hours(I trained about 15 epoch，then stopped，so the total train time is about 2 days....2 weeks seems too long...)， use an TITAN Xp.
I guess the train time depends on the number of characters, but the train time is too long...Maybe you should check your code. You can use command nvidia-smi to check GPU-Util.
By the way，is it convenient to reveal what your task is？ I'm curious why there are so many characters to recognize...

I saw your config set: __C.TRAIN.EPOCHS = 580000,so this is your iterations numbers or epoch numbers?
AND i am working on passport OCR.

hello, 580000 is iterations; My guess is that data processing takes up too much time，gpu is waiting for data.