leriomaggio/deep-learning-keras-tensorflow

Introduction to Deep Neural Networks with Keras and Tensorflow

deep-learning-keras-tensorflow's Issues

Fine tuning VGG16

Hi,

Just a quick question, I'm trying to fine tune the VGG16 as suggested in the notebook 2.4. However, I got an error: The shape of the input to "Flatten" is not fully defined (got (None, None, 512). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.. Not sure if anyone can give me a hint on this. My code is shown below:

from keras.applications import VGG16

model = VGG16(include_top=False)
for layer in model.layers:
    layer.trainable = False

top_model = Sequential()
top_model.add(Flatten(input_shape=model.output_shape[1:]))
top_model.add(Dense(768, activation='sigmoid'))
top_model.add(Dropout(0.0))
top_model.add(Dense(768, activation='sigmoid'))
top_model.add(Dropout(0.0))
top_model.add(Dense(n_labels, activation='softmax'))

model.add(top_model)

train_model(model,
            (X_train_gte5, y_train_gte5),
            (X_test_gte5, y_test_gte5), nb_classes)

Why is the 'change' to the weights impacted by the last activation?

In 1.1 Introduction - Deep Learning and ANN.ipynb there's this line:

change = output_deltas[k] * self.ah[j]

and change is then being used to the update the output weights.

self.wo[j][k] += N*change + M*self.co[j][k]

Why are the activations from the hidden layer being used to tweak the weights? shouldn't the weight changes be only based on the weights and the relative change of the loss?

Image inpainting using vgg19

is it possible to make the vgg19 model complete missing parts of an image according to its category ?
thanks

Deep Learning for Edge detection

Hello!
I have a dataset of images in which the edges are manually labeled with different colors. I am planning to use deep learning on these training images. Then, I am thinking to test the model so that in the new input images, the model will predict those edges automatically.

Whether the model can include the Embedding layer?

My model is defined as follows：
def create_model():
#input
input_layer=Input(shape=(max_squence_len,),name="x_seq")
embedding_layer=Embedding(input_dim=max_token_num,output_dim=embedding_dim) (input_layer)
#conv layers
convs=[]
filter_sizes=[2,3,4]
for fs in filter_sizes:
l_conv=Conv1D(filters=50,kernel_size=fs,activation="relu")(embedding_layer)
l_pool=MaxPooling1D(pool_size=2)(l_conv)
l_pool=Flatten()(l_pool)
convs.append(l_pool)
merge=concatenate(convs,axis=1)
out=Dropout(0.5)(merge)
output=Dense(32, activation='relu')(out)
output=Dense(units=31, activation='softmax')(output)
model=Model([input_layer],output)
adam = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
model.compile(loss="categorical_crossentropy",optimizer=adam,metrics=['accuracy'])
model=KerasClassifier(build_fn=self.create_model,batch_size=64,epochs=25)
batch_size = [10, 20, 40, 60, 80, 100]
epochs = [5,10,15,20,25,30]
param_grid = dict(batch_size=batch_size, epochs=epochs)
grid = GridSearchCV(estimator=model, param_grid=param_grid,cv=5)
grid.fit(x_train,y_train)
print("best_score:",grid.best_score_,"best_param:",grid.best_params_)
But the program throws an error:TypeError: cannot deepcopy this pattern object
can you help me solve this problem?

Theano does not recognise this flag:

UserWarning: Theano does not recognise this flag: lib.cnmem
warnings.warn('Theano does not recognise this flag: {0}'.format(key))

what is sentence_matrix

what is the value of sentence_matrix and i am unable to import data_helpers.. can you please help me

Configuration issues with keras import

Hi All,

I put this up on theano-users google group but I wanted to put it up here too:

Using the deep-learning.yml install, I was able to install anaconda, tensorflow, cuda, keras and theano and others. I already had a cuda 7.5 working but I noticed that the deep-learning yml installs cuda 7.0 which is fine. I followed the instructions and I can import standard python packages like numpy, scipy pandas, etc. However, I am having trouble importing theano, via keras. When I import keras, it gives this error:

import keras
Using Theano backend.
Traceback (most recent call last):
File "", line 1, in
File "/home/nkiruka_ubuntu/anaconda3/envs/deep-learning/lib/python3.5/site-packages/keras/init.py", line 2, in
from . import backend
File "/home/nkiruka_ubuntu/anaconda3/envs/deep-learning/lib/python3.5/site-packages/keras/backend/init.py", line 58, in
from .theano_backend import *
File "/home/nkiruka_ubuntu/anaconda3/envs/deep-learning/lib/python3.5/site-packages/keras/backend/theano_backend.py", line 1, in
import theano
File "/home/nkiruka_ubuntu/anaconda3/envs/deep-learning/lib/python3.5/site-packages/theano/init.py", line 103, in
import theano.sandbox.cuda
File "/home/nkiruka_ubuntu/anaconda3/envs/deep-learning/lib/python3.5/site-packages/theano/sandbox/cuda/init.py", line 697, in
use(device=config.device, force=config.force_device, test_driver=False)
File "/home/nkiruka_ubuntu/anaconda3/envs/deep-learning/lib/python3.5/site-packages/theano/sandbox/cuda/init.py", line 516, in use
device = int(device[3:])
ValueError: invalid literal for int() with base 10: ' # switch to cpu if no GPU is available on your machine'

Now, I know my GPU works because I can run the samples. My theano.rc looks like this:
[global]
floatX = float32
device = gpu # switch to cpu if no GPU is available on your machine

[nvcc]
fastmath = True

[lib]
cnmem=.90

[cuda]
root = /usr/local/cuda-7.5/

If I remove the line device=gpu by commenting it out, then I get this error:
import keras
Using Theano backend.
Traceback (most recent call last):
File "", line 1, in
File "/home/nkiruka_ubuntu/anaconda3/envs/deep-learning/lib/python3.5/site-packages/keras/init.py", line 2, in
from . import backend
File "/home/nkiruka_ubuntu/anaconda3/envs/deep-learning/lib/python3.5/site-packages/keras/backend/init.py", line 58, in
from .theano_backend import *
File "/home/nkiruka_ubuntu/anaconda3/envs/deep-learning/lib/python3.5/site-packages/keras/backend/theano_backend.py", line 1, in
import theano
File "/home/nkiruka_ubuntu/anaconda3/envs/deep-learning/lib/python3.5/site-packages/theano/init.py", line 95, in
if hasattr(theano.tests, "TheanoNoseTester"):
AttributeError: module 'theano' has no attribute 'tests'

Can anyone direct me as to where I am going wrong?

Inline images don't show up

In the current version inline images in Markdown parts of Jupyter Notebook do not work (on least in GitHub), vide:

sequence classification using LSTM

I do sequence classification using LSTM and The length of sequences are variable.
So I need to pad it with max-length, and I also set mask_zero flag = true.

My Network model like below:

http://deeplearning.net/tutorial/lstm.html

My code as below:

def LSTM_Classification():

max_features = 19694+1
batch_size = 32
embedding_dims = 32
nb_epoch = 3
path = "./dataset/SemEval_task3.pkl"
max_words = max_features

(X_train, y_train), (X_test, y_test) = guan_SemEval_task3.load_data(path)
x_new_train = padding_all(X_train)
y_new_train = np.array([(1 if y is 0 else 0, 1 if y is 1 else 0, 1 if y is 2 else 0) for y in y_train])

x_new_test = padding_all(X_test)    
y_new_test = np.array([(1 if y is 0 else 0, 1 if y is 1 else 0, 1 if y is 2 else 0) for y in y_test])
print (x_new_train.shape)
print (y_new_train.shape)
print (x_new_test.shape)
print (y_new_test.shape)

print('Build model...')
model = Sequential()
model.add(Embedding(max_features, embedding_dims, dropout=0.2,mask_zero=True))
model.add(LSTM(32, dropout_W=0.2, dropout_U=0.2,return_sequences=True))  
print(model.output_shape) #(None, None, 32)


model.add(TimeDistributed(Dense(1)))
print(model.output_shape) #(None, None, 1)

avgpool = Lambda(lambda x: K.mean(x, axis=1, keepdims=False), output_shape=lambda x: (x[0], x[2]))
avgpool.supports_masking = True
model.add(avgpool)    
print(model.output_shape) #(None, 1)


#model.add(Dense(3))
dense3 = Dense(3)
dense3.supports_masking = True
model.add(dense3)
print(model.output_shape) #(None, 1)
model.add(Activation('softmax'))
print(model.output_shape) #(None, 1)

model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])
model.summary()
early_stop = EarlyStopping(monitor='val_loss', patience=20, verbose=1)
model.fit(x_new_train, y_new_train,nb_epoch=nb_epoch,batch_size=batch_size,
          validation_data=(x_new_test, y_new_test),callbacks=[early_stop])    


print('Testing...')
score, acc = model.evaluate(x_new_test, y_new_test,batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)

the output_shape of the model seem to be right. but when I run this code,
the error message as below:

(16541L, 849L)
(16541L, 3L)
(1976L, 849L)
(1976L, 3L)
Build model...
(None, None, 32)
(None, None, 1)
(None, 1)
(None, 3)
(None, 3)

---

Layer (type) Output Shape Param # Connected to

embedding_5 (Embedding) (None, None, 32) 630240 embedding_input_5[0][0]

---

lstm_5 (LSTM) (None, None, 32) 8320 embedding_5[0][0]

---

timedistributed_5 (TimeDistribute(None, None, 1) 33 lstm_5[0][0]

---

lambda_5 (Lambda) (None, 1) 0 timedistributed_5[0][0]

---

dense_10 (Dense) (None, 3) 6 lambda_5[0][0]

---

activation_3 (Activation) (None, 3) 0 dense_10[0][0]

Total params: 638599

---

Train...
Train on 16541 samples, validate on 1976 samples
Epoch 1/3
Traceback (most recent call last):

File "", line 1, in
runfile('D:/prj_python_spyder/test1/test1/programs/test_fasttext.py', wdir='D:/prj_python_spyder/test1/test1/programs')

File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 714, in runfile
execfile(filename, namespace)

File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile
exec(compile(scripttext, filename, 'exec'), glob, loc)

File "D:/prj_python_spyder/test1/test1/programs/test_fasttext.py", line 508, in
main();

File "D:/prj_python_spyder/test1/test1/programs/test_fasttext.py", line 498, in main
doLSTM_AVGPooling()

File "D:/prj_python_spyder/test1/test1/programs/test_fasttext.py", line 305, in doLSTM_AVGPooling
validation_data=(x_new_test, y_new_test),callbacks=[early_stop])

File "C:\Anaconda2\lib\site-packages\keras\models.py", line 597, in fit
sample_weight=sample_weight)

File "C:\Anaconda2\lib\site-packages\keras\engine\training.py", line 1107, in fit
callback_metrics=callback_metrics)

File "C:\Anaconda2\lib\site-packages\keras\engine\training.py", line 825, in _fit_loop
outs = f(ins_batch)

File "C:\Anaconda2\lib\site-packages\keras\backend\theano_backend.py", line 657, in call
return self.function(*inputs)

File "C:\Anaconda2\lib\site-packages\theano\compile\function_module.py", line 879, in call
storage_map=getattr(self.fn, 'storage_map', None))

File "C:\Anaconda2\lib\site-packages\theano\gof\link.py", line 325, in raise_with_op
reraise(exc_type, exc_value, exc_trace)

File "C:\Anaconda2\lib\site-packages\theano\compile\function_module.py", line 866, in call
self.fn() if output_subset is None else\

ValueError: Input dimension mis-match. (input[0].shape[1] = 32, input[1].shape[1] = 849)
Apply node that caused the error: Elemwise{mul,no_inplace}(InplaceDimShuffle{x,0}.0, Elemwise{Cast{float32}}.0)
Toposort index: 350
Inputs types: [TensorType(float32, row), TensorType(float32, matrix)]
Inputs shapes: [(1L, 32L), (32L, 849L)]
Inputs strides: [(128L, 4L), (3396L, 4L)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[Sum{axis=[1], acc_dtype=float64}(Elemwise{mul,no_inplace}.0)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

The 849 of the max length I pad in sequences.

Could someone please help to fix this error ? I have got stuck several days.
Thanks a lot.

Import matplotlib RuntimeError

Another issue I ran into while import matplotlib at 1.1 Introduction

import matplotlib
---> 24 from matplotlib.backends import _macosx
     25 
     26 

RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are Working with Matplotlib in a virtual enviroment see 'Working with Matplotlib in Virtual environments' in the Matplotlib FAQ

Got it to work using brew install python --framework (https://github.com/gldnspud/virtualenv-pythonw-osx). Thank you for your attention.

Couldn't build conda env

Hi, Valerio,
I am still new to anaconda and was trying to build the conda env on Mac terminal by
conda env create -f deep-learning-osx.yml

And got the error
NoPackagesFoundError: Package missing in current osx-64 channels:
- backports.shutil_get_terminal_size 1.0.0 py35_0.

After reading this post (conda/conda#4994), I deleted some of the build numbers and/or version numbers, and also deleted backports.shutil_get_terminal_size. The environment was then created (I can share my version if needed). Check import/installed version passed. I am still at the beginning of your tutorial, and not sure if this will lead to future problems. Any help with this issue would be much appreciated!

Also, thank you very much for sharing this consice tutorial!

setup error with pydatait branch

When I use the deep-learning.yml file in pydatait branch, I get the error:

Could not find a version that satisfies the requirement pygpu==0.2.1 (from versions: )
No matching distribution found for pygpu==0.2.1

I noticed pygpu is not required in setup recipe under branch master for example, should I exclude it?
Is it only required by Theano?

EDIT

I succeeded in setting it all up, installing things by hands incrementally (keras first).
I paste below my .yml file, just in case (ah, I switched to tensorflow as keras backend, since Theano still gives problems).

Content of deep_learning_keras.yml:

name: deep_learning_keras
channels:
- conda-forge
- defaults
dependencies:
- blas=1.1=openblas
- bleach=1.5.0=py36_0
- ca-certificates=2017.1.23=0
- certifi=2017.1.23=py36_0
- cycler=0.10.0=py36_0
- dbus=1.10.10=3
- decorator=4.0.11=py36_0
- entrypoints=0.2.2=py36_1
- expat=2.1.0=2
- fontconfig=2.12.1=4
- freetype=2.7=1
- gettext=0.19.7=1
- glib=2.51.4=0
- gmp=6.1.2=0
- gst-plugins-base=1.8.0=0
- gstreamer=1.8.0=1
- h5py=2.7.0=np112py36_0
- hdf5=1.8.17=10
- html5lib=0.999=py36_0
- icu=58.1=1
- ipykernel=4.6.0=py36_0
- ipython=5.3.0=py36_0
- ipython_genutils=0.2.0=py36_0
- ipywidgets=6.0.0=py36_0
- jinja2=2.9.5=py36_0
- jpeg=9b=0
- jsonschema=2.5.1=py36_0
- jupyter=1.0.0=py36_0
- jupyter_client=5.0.1=py36_0
- jupyter_console=5.1.0=py36_0
- jupyter_core=4.3.0=py36_0
- keras=2.0.2=py36_1
- libffi=3.2.1=3
- libgpuarray=0.6.2=np112py36_0
- libiconv=1.14=4
- libpng=1.6.28=0
- libsodium=1.0.10=0
- libxcb=1.12=1
- libxml2=2.9.4=4
- line_profiler=2.0=py36_0
- mako=1.0.6=py36_0
- markupsafe=0.23=py36_1
- matplotlib=2.0.0=np112py36_3
- mistune=0.7.4=py36_0
- nbconvert=5.1.1=py36_1
- nbformat=4.2.0=py36_0
- ncurses=5.9=10
- nose=1.3.7=py36_2
- notebook=5.0.0=py36_0
- numpy=1.12.1=py36_blas_openblas_200
- openblas=0.2.19=1
- openssl=1.0.2k=0
- pandas=0.19.2=np112py36_1
- pandoc=1.19.2=0
- pandocfilters=1.4.1=py36_0
- pcre=8.39=0
- pexpect=4.2.1=py36_0
- pickleshare=0.7.3=py36_0
- pip=9.0.1=py36_0
- prompt_toolkit=1.0.14=py36_0
- ptyprocess=0.5.1=py36_0
- pygments=2.2.0=py36_0
- pyparsing=2.2.0=py36_0
- pyqt=5.6.0=py36_0
- python=3.6.1=0
- python-dateutil=2.6.0=py36_0
- pytz=2017.2=py36_0
- pyyaml=3.12=py36_1
- pyzmq=16.0.2=py36_1
- qt=5.6.2=1
- qtconsole=4.3.0=py36_0
- readline=6.2=0
- scikit-learn=0.18.1=np112py36_blas_openblas_200
- scipy=0.19.0=np112py36_blas_openblas_200
- setuptools=33.1.1=py36_0
- simplegeneric=0.8.1=py36_0
- sip=4.18=py36_1
- six=1.10.0=py36_1
- sqlite=3.13.0=1
- terminado=0.6=py36_0
- testpath=0.3=py36_0
- theano=0.9.0=py36_0
- tk=8.5.19=1
- tornado=4.4.3=py36_0
- traitlets=4.3.2=py36_0
- wcwidth=0.1.7=py36_0
- webencodings=0.5=py36_0
- wheel=0.29.0=py36_0
- widgetsnbextension=2.0.0=py36_0
- xorg-libxau=1.0.8=3
- xorg-libxdmcp=1.1.2=3
- xz=5.2.2=0
- yaml=0.1.6=0
- zeromq=4.2.1=1
- zlib=1.2.11=0
- libgfortran=3.0.0=1
- mkl=2017.0.1=0
- pip:
  - appdirs==1.4.3
  - ipython-genutils==0.2.0
  - jupyter-client==5.0.1
  - jupyter-console==5.1.0
  - jupyter-core==4.3.0
  - line-profiler==2.0
  - packaging==16.8
  - prompt-toolkit==1.0.14
  - protobuf==3.2.0
  - pygpu==0.6.2
  - tensorflow-gpu==1.0.1

theano configuration error

I want to configure the "device" in theanorc to "cpu“ but it raised a value error.
And I removed the theanorc file, then it works.

is video available pls?

I guess you have video recorded?
may you share this video?

typo in environment creation

Great looking tutorial. Just a small typo in the environment setup of README.md.

conda create env ... should be condo env create ...

typo in readme

The readme says

conda create env -f deep-learning.yml

when it should be

conda env create -f deep-learning.yml