parasdahal/deepnet

Educational deep learning library in plain Numpy.

Hi there 👋

deepnet's Issues

does your code actually work?

cifar10 overfitting badly.

cifar10's test accuracy does not go above 33%. What can be done?

question in batchnorm

Hi, thanks for your share.
I found some wrong in the implementation of BatchNorm。you compute mean and var for every batch, rather than for every channels

About optimization

Hello. Are you quite sure that history of optimizer (e.g. moments) should be zeroed at the beginning of each epoch?

question in cnn back propagation

Hi, thanks for your work. I learned a lot from your blog and code!

In your gradient test code, I found that there may be something wrong in the back propagation
for dX of CNN, test code is as blow and data used in the code are here:npy.zip

w = np.load('w.npy')
b = np.load('b.npy')
dout = np.load('dout.npy')
x = np.load('x.npy')

c_layer = Conv((1, 28, 28),n_filter=32,h_filter=3,w_filter=3,stride=1,padding=1)
c_layer.W = w
c_layer.b = b
dx_num = numerical_gradient_array(lambda x: c_layer.forward(x), x, dout)
dw_num = numerical_gradient_array(lambda w: c_layer.forward(x), w, dout)
db_num = numerical_gradient_array(lambda b: c_layer.forward(x), b, dout)

out = c_layer.forward(x)
dx,grads = c_layer.backward(dout)
dw,db = grads
print("Testing backward pass of Conv Layer")
print("dX error: ",rel_error(dx,dx_num))
print("dW error: ",rel_error(dw,dw_num))
print("db error: ",rel_error(db,db_num))

the results is as blow:

Testing backward pass of Conv Layer
dX error: 1.0
dW error: 4.938012368517188e-11
db error: 2.0764855776951717e-07

some questions about dropout

Hi,
I first used a simple three-tier CNN structure without dropout. The accuracy rate was over 90%. However, after I used dropout, the accuracy rate dropped to 40%. Is there any problem with my data set or is it another problem?

thank you
I

Derivative of ReLU

deepnet/deepnet/layers.py

Lines 213 to 216 in 51a9e61

	def backward(self, dout):
	dX = dout.copy()
	dX[self.X <= 0] = 0
	return dX, []

Derivative of ReLU equal to 0 if x<0 and equal 1 if x>0, isn't it?

Which version of Python should be used?

Python2 or Python3?

'bool' object is not callable in solver.py

I put the dataset manually in ./data/mnist.pkl.gz and run the python run_cnn.py mnist, but it gives an error:

Traceback (most recent call last):
File "run_cnn.py", line 46, in
learning_rate=0.01, X_test=X_test, y_test=y_test)
File "/Users/JianGuo/PycharmProjects/deepnet/deepnet/solver.py", line 83, in sgd_momentum
minibatches = get_minibatches(X_train, y_train, minibatch_size)
File "/Users/JianGuo/PycharmProjects/deepnet/deepnet/solver.py", line 12, in get_minibatches
X, y = shuffle(X, y)
TypeError: 'bool' object is not callable

I doubt that this would be a compatible issue with the sklearn, but even I had installed the old version of scikit-learn(0.15), it turns out the same result.

Python version:

Python 3.6.3 (default, Oct 4 2017, 06:09:15)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

pip version:

pip 9.0.1 from /Users/JianGuo/PycharmProjects/deepnet/.env/lib/python3.6/site-packages (python 3.6)

requirements.txt:

numpy==1.11.3
scipy==0.16.1
matplotlib==1.5.0
ipykernel==4.2.2
ipython==4.0.1
ipython-genutils==0.1.0
ipywidgets==4.1.1
scikit-learn==0.15

Did I miss something here?