cs231n / cs231n.github.io Goto Github PK
View Code? Open in Web Editor NEWPublic facing notes page
License: MIT License
Public facing notes page
License: MIT License
Here it is linked to this wiki page which is not working anymore, and does not appear to have a cached version online as verified with this service. The wiki of the linked project seems to no longer work -- is there a mirror available anywhere?
Since you do not have a LICENSE file, the copyright is unclear. Am I or a company I may work for allowed to use this material and under what conditions?
There's a tricky point in the paragraph on intuition of convolutional layers (ConvNet Layers > Convolutional Layer > Overview and Intuition). The paragraph reads as follows:
The CONV layer's parameters consist of a set of learnable filters. Every filter is small spatially (along width and height), but extends through the full depth of the input volume. During the forward pass, we slide (more precisely, convolve) each filter across the width and height of the input volume, producing a 2-dimensional activation map of that filter. As we slide the filter, across the input, we are computing the dot product between the entries of the filter and the input.
If I'm not mistaken, the dot product is only used if you stretched the local regions of the image out into columns, as described much later in the part about "Implementation as Matrix Multiplication". This threw me off a bit when studying the chapter. In the simple/intuitive case you'd do element-wise matrix multiplication, correct? I think that'd make more sense in the first paragraph, and I'd be happy to fix this and create a pull request if the above is accurate!
this issue is not an really an issue with the course or its associated material in general (infact I think they are very well written!). In the past weeks I've spent my spare time on digesting symbolic calculus on computational graphs. I've created three notebooks from my findings that I'll happily share with the community. Currently these notebooks cover:
You can find the notebooks here:
https://github.com/cheind/py-cgraph
I'd hope someone finds them interesting. In case you find them helpful it would be very welcomed if linked from a proper place.
Thanks,
Christoph
Under the Interpreting a linear classifier
section, the ship score
in the included graphic should be 60.75
instead of 61.95
.
Shown below are the simplification steps (it appears that the bias was not subtracted in the original maths, see step 5.)
1. <0, 0.25, 0.2, -0.3> * <56, 231, 24, 2> + (-1.2)
2. 0*56 + 0.25*231 + 0.2*24 + (-0.3)*2 - 1.2
3. 0 + 57.75 + 4.8 - 0.6 - 1.2
4. 62.55 - 0.6 - 1.2
5. 61.95 - 1.2
6. 60.75
In the code example given in http://cs231n.github.io/classification/ in the the section "Validation sets for Hyperparameter tuning" , it would be more efficient to train using the Nearest Neighbor class outside the for loop for "k". In the present state of the example, there's no point in training for every iteration of k, which is just going to be used for predicting the k-nearest neighbors.
In the Data Preprocessing section, PCA is described in the following way:
# Assume input data matrix X of size [N x D]
X -= np.mean(X, axis = 0) # zero-center the data (important)
cov = np.dot(X.T, X) / X.shape[0] # get the data covariance matrix
U,S,V = np.linalg.svd(cov)
Why do you calculate the SVD of the covariance matrix? As I understand it, there are two equivalent (mutually exclusive) alternatives in order to perform PCA:
X.T*X
U*S*V.T
If you want to use the eigenvalues of the covariance matrix to project your data X, then X*V
(with V as the eigenvectors of the covariance matrix) is consistent with the explanation given in the next line, described as 'projecting into the eigenbasis'. However, if you want to use SVD, then the projection is already performed by calculating U*S
. It is also possible to compute the projection via X*V
where V are the right singular vectors from the SVD (this is due to the fact that the eigenvectors of the covariance matrix are equal to the right singular vectors V)
I think there was a confusion in describing this part of the course. Or did I miss something?
cs231n_caffe_torch7_keras_lasagne_v2 doesn't exist in AWS Community AMIs anymore
Hi,
in linear-classify.md you first introduce the linear classifier as
$$f(x_i, W, b) = W x_i + b $$
In the above equation, we are assuming that the image (x_i) has all of its pixels flattened out to a single column vector of shape [D x 1]. The matrix W (of size [K x D]), and the vector b (of size [K x 1]) are the parameters of the function
Later, in SVM loss you have
$$ L_i = \sum_{j\neq y_i} \max(0, w_j^T x_i - w_{y_i}^T x_i + \Delta) $$
where (w_j) is the j-th row of (W) reshaped as a column.
This sounds overly complex to me:
(w_j) is the j-th row of (W) reshaped as a column
is the justification to have the transpose in w_j^T x_i. But all it means, referring to your initial definition of W, is that you have the dot-product of the w_j's row with x_i. So, given your initial definition the loss could be defined as
$$ L_i = \sum_{j\neq y_i} \max(0, w_j x_i - w_{y_i} x_i + \Delta) $$
where w_j is the j-th row of (W).
Am I overlooking something?
In q1 Dropout section it sets the dropout parameters to the following to numerically check the backward pass of dropout:
dropout_param = {'p': 0.8, 'mode': 'test', 'seed': 123}
This doesn't make much sense - shouldn't mode be set to 'train'?
On linear_classifier.py line 82, the input X is described as having shape (D, N) when the notebook passes it with dimension (N, D).
In the Layer Sizing Patterns
section, one sentence reads:
Another sligthly less common setting is to use 3x3 receptive fields with a stride of 2, but this makes.
I suspect you wanted to say something about information loss with a larger receptive field!
(Also, many thanks for the great tutorial with links to recent literature.)
I tried to work on the assignment3's Image Generation part, for the 'feature inversion' cell, I get a lot of 'nan' through testing the model with 'kitten.jpg'. However, I can solve this by changing the data type of the pretrained model to 'np.float64'. It killed a lot of my time and is it an issue that needs to be pointed out?
In the 'VGGNet in detail' section the amount of memory used by the network is shown as follows:
TOTAL memory: 24M * 4 bytes ~= 93MB
however adding up all the memory
values for the listed layers produces a total of only ~15MB * 4 bytes. Is the memory usage calculation correct? If the calculation is correct then a quick explanation of how the figure of 24M was derived would be useful.
For example in http://cs231n.github.io/optimization-2:
MathJax 2.5.3 in Google Chrome 46.0.2490.80 (64-bit) in Arch Linux.
Update MathJax please?
Have you taken the files off or did the path change to http://cs231n.stanford.edu/coco_captioning.zip ?
Would be great if you could put them back up.
Hi Andrej,
Thanks for putting such a great resource about convnets and Deep Learning online. I believe this would be greatly helpful for many people including me.
I was reading the convnets page and found this minor issue that you might want to fix. The page says:
The Krizhevsky et al. architecture ... Real-world example: Since (227 - 11)/4 + 1 = 55, we can see that ...
However, they actually use crops of 224x224. So the formula would be (224 - 11)/4 + 1 = 54.25
, which is not an integer but I guess still valid configurations, right?
For the case of Zeiler et al. (Visualizing and Understanding convnets), this formula also gives a non-integer result: (224 - 7)/2 + 1 = 109.5
..
Best,
Anh
When discussing pool layers it looks like this sentence was stopped short.
Another slightly less common setting is to use 3x3 receptive fields with a stride of 2, but this makes. It is very uncommon ...
Hi Andrej, Fei-Fei!
First, thanks you all for great course!
Second, I`d like to argue with slide
I am doing caffenet benchmark and one of the sections are lr_policy.
The linear decay policy gives the best test accuracy, given same start_lr, amond different "step", sqrt and sqr decay. And if you compare curve to your slide, "step" policy looks much more "good", while linear one looks like "too low". :)
https://github.com/ducha-aiki/caffenet-benchmark/blob/master/Lr_policy.md
In section "Compute the loss":
corect_logprobs = -np.log(probs[range(num_examples),y])
and
# compute the loss: average cross-entropy loss and regularization
data_loss = np.sum(corect_logprobs)/num_examples
reg_loss = 0.5*reg*np.sum(W*W)
loss = data_loss + reg_loss
corect_logprobs
=> correct_logprobs
This tutorial (http://cs231n.github.io/aws-tutorial/) says that Caffe, Torch7, Theano, Keras and Lasagne are pre-installed. But when I tried /anaconda2/bin/python -> import keras, ImportError: No module named keras occurs. I assume keras is not installed. Am I missing something?
Hi. I think the statement that you make here: http://cs231n.github.io/neural-networks-3/#gradcheck , namely that the first gradient check has an error of O(h), while the second one has an error of O(h^2) is false. It makes no sense that the two expressions have different errors, since f(x+h) can be Taylor expanded around f(x-h) and will give the same error rate as if it would have been expanded around f(x).
But formally: f(x+ h) - f(x) = h * grad_f(x) + O(h^2) , f(x+ h) - f(x-h) = f(x) + h * grad_f(x) + O(h^2) - (f(x) - h * grad_f(x) + O(h^2)) = 2h * grad_f(x) + O(h^2). This shows that the two expressions have the exact same error.
To be honest, I'm not sure that I understood correctly or not. So I wrote it in here as an issue, not as a pull request.
In the very last paragraph of "Converting FC layers to CONV layers" section in ConvNet note( http://cs231n.github.io/convolutional-networks/#convert ), there is an example about efficiently applying the original ConvNet using stride of 16 (rather than 32). The document says that it could be solved by combining result of the original one and the shifted, in both width and height by 16, one. But, I think it needs to be applied 4 times to fill the matrix, as far as I understood. (if not, only black or white region of the chess board would be filled, once you consider the result of the final matrix as a chess board)
(not 100% sure.. but maybe..)
In the paragraph Coarse model it says "the output spikes in many systems in known to be important", the last in should be a is.
In the second cell of the softmax exercise, the test mask overlaps the training mask (get_CIFAR10_data function)
href in linear-classify.md
to interacitve web demo is broken
See screenshot https://www.dropbox.com/s/ecvwru5sf81xlfd/Screenshot%202016-07-05%2014.09.26.png?dl=0
Python Numpy Tutorial has the following typo in Array indexing section:
# Create the following rank 2 array with shape (3, 4)
# [[ 1 2 3 4]
# [ 5 6 7 8]
# [ 9 10 11 12]]
a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
I'm pretty sure it should be
# Create the following rank **3** array with shape (3, 4)
In the file 'neural-networks-3.md' there is a sentence under the heading 'Hyperparameter ranges' reading:
'That is, we are generating a random random with a uniform distribution, but then raising it to the power of 10. '
I have a feeling that 'random random' is the case, perhaps should be random variable?
In the markdown cell explaining the LSTM line 8,
I think
We then compute the input gate g∈ℝH,
should have been
We then compute the input gate i∈ℝH,
Could you generate ePub or PDF of this docs ?
It seems that the address for image is always "file:///C:/assets/...." so that .md file cannot load the image (for example, file:///C:/Users/users/Documents/cs231n.github.io/...). I temporarily fixed this by copying the assets folder into the root of my C disc directly. I am not sure if this is a bug with the file itself or chrome extension.
Also a minor typo "noone" is in the sentence "In one dimension, the "sum of indicator bumps" function (g(x) = \sum_i c_i \mathbb{1}(a_i < x < b_i)) where (a,b,c) are parameter vectors is also a universal approximator, but noone would suggest that we use this functional form in Machine Learning."
Following requirements are mentioned twice in reuirements.txt that comes with the zip file for assignemnt 1.
This gives the error "Double requirement given" when running pip install -r requirements.txt
Every time the travis checks are ran after my commits, I have the follwing mistake and I saw a lot of pull requests having the same:
Flake8 test *
+flake8 --exit-zero '--exclude=test_*,six.py' skimage doc/examples viewer_examples
Traceback (most recent call last):
File "/home/travis/venv/bin/flake8", line 7, in
from flake8.main import main
File "/home/travis/venv/local/lib/python2.7/site-packages/flake8/main.py", line 5, in
import setuptools
File "/home/travis/venv/local/lib/python2.7/site-packages/setuptools/init.py", line 11, in
from setuptools.extension import Extension
File "/home/travis/venv/local/lib/python2.7/site-packages/setuptools/extension.py", line 8, in
from .dist import _get_unpatched
File "/home/travis/venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 21, in
packaging = pkg_resources.packaging
AttributeError: 'module' object has no attribute 'packaging'`
Is there a way to avoid it?
Thank you
From this tutorial,
After searching for cs231n_caffe_torch7_keras_lasagne_v2 in the Community AMIs search bar there are no results. The page says there are No AMIs found matching your filter criteria:
I tried searching for ami-125b2c72 as well, but AWS still said that no AMIs matched the criteria. I haven't selected any filters on the side bar, and am searching in the Community AMIs tab, not the AWS Marketplace.
Is anyone else having this issue?
As @karpathy said in the video, when drag and drop the circle point in the demo, it disappeared.
That's because you use e.clientX
and e.clientY
in handleMouseDown
and handleMouseMove
function.
Try to use e.pageX
and e.pageY
instead.
I am sorry if I seem like I'm spamming issues.. I'm not sure if this is the best way to report typos
If you want me to report through other channel, please let me know.
in
def lstm_step_backward()
I think
dx, dh, dc, dWx, dWh, db = None, None, None, None, None, None
is supposed to be
dx, dprev_h, dprev_c, dWx, dWh, db = None, None, None, None, None, None
The loss formulae for the 3 1-dimensional points imply that point i belongs to class i, while this hypothesis has not been stated explicitly before.
NetworkVisualization-TensorFlow - Pretrained Model
After downloading Tensorflow checkpoint, the script throws exception because 'cs231n/datasets/squeezenet.ckpt'
cannot be detected.
These are valid checkpoint files in the directory:
Tensorflow, however, can still detect .ckpt.*
in a directory.
Proposed fix:
Either detect wildcards if not glob.glob(SAVE_PATH + "*"):
or meta file if not os.path.exists(SAVE_PATH + ".meta"):
From: https://github.com/cs231n/cs231n.github.io/blob/master/classification.md
In the classification notes there is a link to http://planetmath.org/vectorpnorm this page is currently (at the time of writing) throwing an error.
Suggest replacement with a different resource until the site is fixed.
This is a very basic issue:
6 import matplotlib.pyplot as plt
7
/Volumes/theory/Experiments/tensorflow/assignment1/cs231n/data_utils.py in <module>()
4 import numpy as np
5 import os
----> 6 from scipy.misc import imread
I've tried all sort of possible solutions online, but the main problem is that this module is not round. imread
. I've tried to restart from scratch on macOS system w/out getting past this. Does anyone have any suggestions?
Regularization defined in the function svm_loss_naive
in cs231n/classifiers/linear_svm.py
is inconsistent with the formulas in lectures and notes.
Regularization is defined as 0.5 * reg * np.sum(W * W)
with an extra factor of 0.5.
This causes erroneous results and headaches when the effect of regularization (based on formulas from the notes) is added in gradient calculation.
Hello! I was reading through https://cs231n.github.io/neural-networks-2/#init recently, and was confused by these two sentences (discussing the initialization variance for ReLU layer weights):
[...] the variance of neurons in the network should be 2.0/n.
This gives the initialization w = np.random.randn(n) * sqrt(2.0/n)
I might be misunderstanding this, but it seems to me that the second line and the referenced paper both create weights with a variance of 2.0/n
, which would make the neurons have a variance of 2.0
(before any kind of activation function).
Is that a correct interpretation? And if so, shouldn't the first line read that the neurons should have a variance of 2.0
instead of 2.0/n
(or that the weights should have a variance of 2.0/n
)?
http://cs231n.github.io/optimization-2/
# set some inputs
x = -2; y = 5; z = -4
# perform the forward pass
q = x + y # q becomes 3
f = q * z # f becomes -12
# perform the backward pass (backpropagation) in reverse order:
# first backprop through f = q * z
dfdz = q # df/dz = q, so gradient on z becomes 3
dfdq = z # df/dq = z, so gradient on q becomes -4
# now backprop through q = x + y
dfdx = 1.0 * dfdq # dq/dx = 1. And the multiplication here is the chain rule!
dfdy = 1.0 * dfdq # dq/dy = 1
dfdx = 1.0 * dfdq # dq/dx = 1. And the multiplication here is the chain rule!
dfdy = 1.0 * dfdq
Shouldn't the above be -4 ?
In the circuit diagram it's not immediately clear where the final 1 comes from?
Also isn't the partial derivative of f(x,y,z) with respect to x = z? also that of y.
Just to make sure I am not crazy I punched into wolfram.
http://www.wolframalpha.com/widgets/view.jsp?id=d052b64db910c143ed1f1a05298ba14c
so I get
fx = z
fy = z
fz = x + y
meaning the gradient is
fxyz = [z,z,x+y]
so for the inputs
gradient is fxyz = [-4,-4,3] ?
I am sorry I am not sure if the mistake is on my end.
Thanks and let me know ? =)
If you can explain to me where the 1 comes from as well that would be great =)
would it be the length of the final vector?
in line 55 I think
- dWx: Gradients of input-to-hidden weights, of shape (N, H)
should have been
- dWx: Gradients of input-to-hidden weights, of shape (D, H)
Could you please include the script or documentation how this project is converted into HTML?
In 'solver.py' line 30
shouldn't
data = {
'X_train': # training data
'y_train': # training labels
'X_val': # validation data
'X_train': # validation labels <--
}
have to be as the following??
data = {
'X_train': # training data
'y_train': # training labels
'X_val': # validation data
'y_val': # validation labels <--
}
This could be trivial..
We can use CNN to classify more than 10 thousands of images of the ImageNet.
I find that CNN could only classify 10-20 text classes as this paper write.
So what is the high limit of short text classification's classes' number?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.