sai19872000/behavioral-cloning

Behavioral-Cloning

The goals of this project are the following:

• Build a Convolutional neural network model to simululate the human driving behaviour
• The final model should be able to complete one lap on the simulator. The simulator has two modes

1. Training mode and
2. autonomous mode The training mode is used to collected the training data for neural network. The autonomous mode is used to test the model. There are two tracks available. I used both the tracks to collect the data. The first track is plain with some curvy edges where as the second track is more complex with hills and very sharp edges which is hard to keep the car on the track even on the training mode.

Files

This project has the following files

1. model.py (this containes the python code which inclues the CNN implementation in keras)
2. model.h5 (final trained model architecture along with weights)
3. drive.py (provided by udacity to run the model in automonous mode on simulator)
4. video.mp4 (video of the simulator in automonous mode)

to run the model on the autonomous mode use the following commad

python drive.py model.h5	

Trainining images

I initially used the training data set provided by Udacity and produced good results both on tracks and then later used data set generated by myself by driving the car on the simulator on both the tracks. The training on the tracks involved

1. 3 laps of driving clock wise on track1
2. 3 laps of driving counter clock wise on track1
3. smooth driving on curves clock wise and anti clock wise
4. recovery from crubs and edges
5. 1 lap of driving on clock wise on track 2
6. 1 lap of driving on counter clock wise on track 2

The training data includes the camera images from the dash board of the car, left and right of the car along with steering angle, throttle, brake and speed

Image augmentation

The center image along with the steering angle looks as follows

Using left and right images

The left and right images can also be used as additional data to train the network when a correction is added/subtracted to the left/right images.

correction = 0.25
steering_c = 0.0
steering_l = steering_c+correction
steering_r = steering_c-correction

steering = (steering_c,steering_l,steering_r)

The center images need no correction and hence set to zero and correction value is added the left image and the subtracted from the right image. I have tested 4 different values of correction

| correction | 0.15 | 0.2 | 0.25 | 0.3 |

and found that "0.25" works the best. The center image along with the left and right image looks as follows, the steering angle is indicated on the top of the image

Flipping the images

The image can flipped and the corresponding steering angle is multiplied by "-1.0" to generate additional data.

image = cv2.flip(image,1)
measurement = measurement*-1.0

The following images shows the effect of flipping

Jitter

Random brightness, shadow and shear can be applied to the images and the sheering angle is adjusted accordingly to generate more training data. After jittering the images looks as follows

Data clipping

The initial distribution of the prediction (i.e. the steering angle) of the training data is as follows

This data shows that most of the data is has steering angle of zero this is not ideal for training the network in a generic track. So I have randomly droped some data in the range to close to zero and this made the distribution as follows

Using this training set I performed the image augmentation described above and got the following

Cropping and normalizing the image

Finally the image is resized to the size of (66,200) to fit the NVIDIA self driving car architecture after clipping the top 2.8th of the image to remove all the unnecessary information such as the sky, trees etc.. and the bottom 25 pixes to remove the hood images.

img_shape = image.shape
image = image[math.floor(img_shape[0]/2.8):img_shape[0]-25, 0:img_shape[1]]
image = cv2.resize(image,(new_col,new_row), interpolation=cv2.INTER_AREA)

The final images is looks as

The images are finally normalized by keras lambda layer

model.add(Lambda(lambda x: x/255.0 - 0.5, input_shape=(66,200,3)))

Keras generator

I initially used a keras generator but later dropped it as it was not necessary for the data set size I am using with my computer. However here is implementation of the generator I was using before

def train_generator(lines_df):
    # training generator
    while 1:
        for i_line in range(len(lines_df)):
            line_data = lines_df.iloc[[i_line]].reset_index()
            #print(line_data)
            img = image_aug(lines_df)
            img = img.reshape(1, img.shape[0], img.shape[1], img.shape[2])
            measurement = line_data['measure'][0]
            measurement = np.array([[y]])
            yield img, measurement

Spitting the data to training and testing

I have used sklearn library train_test_split funtion to split the data to training and validation set. In the split 80% is used for training and 20% for validation

from sklearn.model_selection import train_test_split
train_lines, validation_lines = train_test_split(lines, test_size=0.2)

Model Architecture

I have used the Nvidia's self driving car architecture as a starting point and slightly modified to get better results. Here is the architecture

In this model the normalized image goes through inital convolutional layer with 24 feature maps and 5 by 5 filter followed by 2 convolutional layers each with 5 by 5 filter with 36 and 48 feature maps. After this 2 more convolutional layers each with 64 feature maps and 3 by 3 filter follow. The feature maps are now flattened followed by 4 fully connected layers with 100, 50, 10 and 1 hidden units respectively constitute the architecture.

The keras implementation is as follows

from keras.layers.core import Dropout
model = Sequential()
model.add(Lambda(lambda x: x/255.0 - 0.5, input_shape=(66,200,3)))
model.add(Convolution2D(24,5,5,subsample=(2,2),activation="elu",W_regularizer=l2(0.001)))
model.add(Convolution2D(36,5,5,subsample=(2,2),activation="elu",W_regularizer=l2(0.001)))
model.add(Convolution2D(48,5,5,subsample=(2,2),activation="elu",W_regularizer=l2(0.001)))
model.add(Convolution2D(64,3,3,activation="elu",W_regularizer=l2(0.001)))
model.add(Convolution2D(64,3,3,activation="elu",W_regularizer=l2(0.001)))
model.add(Flatten())
model.add(Dense(100,W_regularizer=l2(0.001)))

model.add(ELU())
model.add(Dense(50,W_regularizer=l2(0.001)))

model.add(ELU())
model.add(Dense(10,W_regularizer=l2(0.001)))
model.add(ELU())
model.add(Dense(1))

model.compile(loss = 'mse', optimizer = Adam(lr=1e-4))

• I have added l2 regularizer to the weight along with 2,2 subsampling to initial 3 convolutional layers to orginal architecture to prevent overfitting and for the model to able to drive in multiple tracks

• I have used ELU() activation vs relu for smoother steering angles vs relu

• The loss funtion is choosen to 'mse' since our steering angle is a continuous variable

• I have used Adam optimizer with a learning rate of 1e-4. I tried several learning rates ranging between 1e-2 to 1e-5 and found that the 1e-4 suits the best for this problem

• I trained the model for 20 epochs; the snapshot below shows the model after 10 epochs

As seen the model validation error was decreasing along with the training error at the end of 20 epochs the training error and validation error seem to low and close.

Model performance on tracks

This model performs really good on tranck one and performs pretty decent on track 2. I have attached the performance video of the model on track one "video.mp4"

Possible improvements

More training data if collected from track 2 can improve the performance on track2. Better image augmentation techniques can be implimented for faster car performance on both tracks