fvarnals/linear-regression

Exercise 1 from Coursera Machine Learning - Andrew Ng

Machine Learning-Linear Regression Exercise

Implement linear regression with one variable to predict profits for a food truck. Given data of profits for each food truck and the population of the city they are in, we seek to predict whether a new city will return a good profit, based on its population.

ex1data1.txt contains our dataset, where the first column gives the population of a city and the second column gives the profit of a food truck in that city. A negative value for profit indicates a loss.

1. plotData.m - Plot the data in our dataset:
   figure; % open a new figure window
plot(x,y,'rx','MarkerSize',10); %Plot the data
ylabel('Profit in $10,000s'); %Set the y-axis label
xlabel('Population of City in 10,000s'); %Set the x-axis label
   

2. computeCost.m - Compute the cost of our linear regression model:
   function J = computeCost(X, y, theta)

• We seek to calculate the cost (accuracy) of our model using the mean squared error, following the below function:
  

• Where the hypothesis is given by the linear model:
  
  

• Using the values of population size in 10,000s (X), and profit in $10,000s (y), we calculate the cost using initial theta values of -1 and 2.

3. gradientDescent.m - Function to run gradient descent:
   

• Having computed the cost function, we can now run gradient descent to find the Theta values which minimise the cost/error of our function, thereby giving better accuracy for out predictions of profit for a given population size.

• We calculate the errors_vector which gives us the error for each prediciton, i.e. for each datapoint we calculate the difference between predicted and actual profit:
  h = X * theta; %predicted profit
errors_vector = h - y;
  

• We then use this to calculate the gradient of the cost with respect to our Theta values:
  (X' * errors_vector)
  

• And repeatedly take small steps in that direction, using the learning rate alpha to insure we do not overshoot and miss the minimum; moving down the gradient, to find the values of theta which give the minimum cost:
  for iter = 1:num_iters
h = X * theta; %predicted profit
errors_vector = h - y;
theta_change = alpha * (1/m) * (X' * errors_vector);
theta = theta - theta_change;
  

• As we perform gradient descent to minimize the cost function J(θ), we monitor the convergence by computing the cost at each step:
  J_history(iter) = computeCost(X, y, theta)
  (We expect the value of J(θ) to converge to a steady value by the end of the algorithm.)

4. featureNormalize.m - Function for normalisation when using multiple features:
   

• We scale the features by taking each value and subtracting the mean value of that feature, and dividing by the standard deviation. We use feature scaling to help gradient descent converge more quickly.

5. normalEqn.m - Function to compute the normal equations:
   

• We also use the normal equation (shown below) to find the optimum values of theta. This method does not require feature scaling and avoids having to run gradient descent, making the process faster. It's a good alternative method as long as the number of features is not very large (~10,000+). This is because the process involves the inversion of the matrix X which is very computationally expensive (i.e. slow) when dealing with very high dimension matrices.