jych/cle

cle's Issues

Data

Hi,
Thank you for releasing cle library on github.
I want to exactly repeat your experiment described in "GFRNN" paper.
It seems EnWiki object loads from a numpy dataformat, which is not included in library.
If it's OK, would you please share your enwiki_char_and_word.npz file?

Thank you.
Best,

Potential issue with global normalization on sequences

cle/cle/data/prep.py

Line 105 in eb728ea

X_len = np.array([len(x) for x in X]).sum()

Here X_len corresponds to the number of sequences and not consider feature dimensionality. In the following lines, mean is calculated by first summing all entries and then dividing by the X_len, assuming that it is total number of entries.

RMSProp,Adam fails on GPU

Hi,
I just started w/ theano so be warned:). And I am facing a problem with RMS/Adam optimizers when running on GPU. Here I instrumented the version of RMS optimizer to print content of parameters and updates

class RMS_nan(RMSProp):
    def get_updates(self, grads):
        updates = OrderedDict()
        for p, g in grads.items():
            lr_scaler = self.lr_scalers.get(str(p), 1.)
            u = sharedX(p.get_value() * 0.)
            avg_grad = sharedX(p.get_value() * 0.)
            sqr_grad = sharedX(p.get_value() * 0.)
            g = theano.printing.Print('GPDATES0')(g)
            avg_grad_t = self.sec_mom * avg_grad + (1 - self.sec_mom) * g
            avg_grad_t = theano.printing.Print('GPDATES1')(avg_grad_t)
            sqr_grad_t = g**2 # self.sec_mom * sqr_grad + (1 - self.sec_mom) * g**2
            sqr_grad_t = theano.printing.Print('GPDATES2')(sqr_grad_t)
            g_t = g / T.sqrt(sqr_grad_t - avg_grad_t**2 + self.e)
            g_t = theano.printing.Print('GPDATES')(g_t)
            u_t = self.mom * u - lr_scaler * self.lr * g_t
            u_t = theano.printing.Print('UPDATES')(u_t)
            p_t = p + u_t
            updates[avg_grad] = avg_grad_t
            updates[sqr_grad] = sqr_grad_t
            updates[u] = u_t
            updates[p] = p_t
        return updates

the output of first run looks like that:

GPDATES0  = <CudaNdarray object at 0x7fc3978a84f0>
**GPDATES2  = [[ nan   0.   0. ...,   0.  nan   0.] comes from g^2 **
 [ nan   0.   0. ...,   0.  nan   0.]
 [ nan   0.   0. ...,   0.  nan   0.]
 ..., 
 [ nan   0.   0. ...,   0.  nan   0.]
 [ nan   0.   0. ...,   0.  nan   0.]
 [ nan   0.   0. ...,   0.  nan   0.]]
GPDATES1  = [[ -4.11258770e-05   0.00000000e+00   0.00000000e+00 ...,   0.00000000e+00
   -1.58047169e-05   0.00000000e+00]
 [ -4.12098307e-05   0.00000000e+00   0.00000000e+00 ...,   0.00000000e+00
   -1.58372259e-05   0.00000000e+00]
 [ -4.11814217e-05   0.00000000e+00   0.00000000e+00 ...,   0.00000000e+00
   -1.58262919e-05   0.00000000e+00]
 ..., 
 [ -4.12097797e-05   0.00000000e+00   0.00000000e+00 ...,   0.00000000e+00
   -1.58373150e-05   0.00000000e+00]
 [ -4.12381414e-05   0.00000000e+00   0.00000000e+00 ...,   0.00000000e+00
   -1.58481744e-05   0.00000000e+00]
 [ -4.11252622e-05   0.00000000e+00   0.00000000e+00 ...,   0.00000000e+00
   -1.58048151e-05   0.00000000e+00]]
GPDATES __str__ = [[ nan   0.   0. ...,   0.  nan   0.]
 [ nan   0.   0. ...,   0.  nan   0.]
 [ nan   0.   0. ...,   0.  nan   0.]

To me it seems like operation square(g) is giving nans for the reason unkown to me atm. Replacing square(g) with 2.*g gets rid of nans in GPDATES2 print statement.

Running on GeForce GTX 560 Ti, Python 2.7.10 |Anaconda 2.4.0 (64-bit), Theano version 0.7.0

mutlivariate numerical prediction.

Hi!
great library!
was wondering if there is an example of just a generic multivariate time series prediciton.
Many thanks,
Andrew

Example how to run/test a trained network

Great library! Could you give an example how to run/test a trained a network? Thanks. - kj