Increasing the number of training images. Error Too many index array

[sullychen]

Increasing the number of training images. Error Too many index array

[alexdominguez09]

This is the exact error message:

Epoch: 0, Step: 60, Loss: 6.16457
Traceback (most recent call last):
File "train.py", line 40, in
xs, ys = driving_data.LoadTrainBatch(batch_size)
File "/media/alex/61170219-f8e4-49b1-bb7b-32dd8ba2b6f6/cnn/steering_wheel/Autopilot-TensorFlow/driving_data.py", line 44, in LoadTrainBatch
x_out.append(scipy.misc.imresize(scipy.misc.imread(train_xs[(train_batch_pointer + i) % num_train_images])[-150:], [66, 200]) / 255.0)
IndexError: too many indices for array

[SullyChen]

How are you appending the data? Are you making sure to update the .txt file with the new images you added?

[alexdominguez09]

Yes. I updated the files data.txt where the driving_data.py are referencing it.

In fact, I trained it and tested it ok, when the data set is samll (22000 images and angles).

Is there anything to take into account if I want to train with many more images, lets say 500000 or one million. Modifying epochs, batch sizes or any other constant in any script?.

Thanks

[SullyChen]

There shouldn't be anything you need to do. For debugging, you can try the following: print(driving_data.num_images) and see what it outputs.

[alexdominguez09]

It is solved now. It turns that some data within my dataset was corrupt, a special character probably. I could not get the exact line failing, as I am talking tens of thousands of lines, so I remove some chunks of data with the corrupted lines and it worked.

So I take then that no manipulation or changes to the model needs to be done to train one million of lines or thousands.

thanks

[coral-agc]

It is solved now. It turns that some data within my dataset was corrupt, a special character probably. I could not get the exact line failing, as I am talking tens of thousands of lines, so I remove some chunks of data with the corrupted lines and it worked.

So I take then that no manipulation or changes to the model needs to be done to train one million of lines or thousands.

thanks

Hey Alex,

I am having the same problem and wanted to ask for some clarification- are you saying that some of your image files were corrupt, or that with your .txt file some lines were corrupt?

I've run train.py four separate times and it has failed with this error after 120, 220, 180, then 40 in epoch 0, so I'm trying to think what would cause the inconsistency.

Thanks!

[alexdominguez09]

Dear coral-agc,

I meant to say the corruption was in my .txt files, where I discover lines with no values, or strange values and characters (result of how I created those .txt files with a serial to usb connection) via an arduino. My image files were ok.

I hope that help.

Regards.