masa-su / tars Goto Github PK
View Code? Open in Web Editor NEWA deep generative model library in Theano and Lasagne
A deep generative model library in Theano and Lasagne
Currently temperature is cast to numpy.float32 in the attempt to fix #59:
class ConcreteSample(GumbelSample):
...
def __init__(self, temp=0.1, seed=1):
super(ConcreteSample, self).__init__(seed=seed)
self.temp = np.float32(temp)
It would be more preferable if we could use floatX such as lasagne.utils.floatX(temp)
while making sure all test cases pass on different machines.
かなりごちゃごちゃしてきたので,思い切って整理したい
すべてのメソッドにdocstringを書く
VAEやGANの親クラスを作成した方がいいかどうか検討中(現在はobjectを継承).
distributionのBernoulliやGaussianはDistributionを継承しているので,そちらに合わせてModelクラスを作ってもいいかもしれない.
KL距離をアニーリングできるようにしたい.
今の実装だと,KLに重み付けできないので,theano変数でパラメータを設定する必要がある.
スケジューリングをVAEクラス外部で行うか内部で行うかは要検討.
Hi,
Thank you for sharing.
I have a problem. I want to add convolutional operation to the model. However, it performs poorly. Below is my generator part code:
z = InputLayer((None,z_dim))
network = DenseLayer(z,num_units=1024,nonlinearity=activation)
network = DenseLayer(network,num_units=128 * 7 * 7,nonlinearity=activation)
network = lasagne.layers.ReshapeLayer(network, ((-1,128,7,7)))
network=lasagne.layers.TransposedConv2DLayer(network,num_filters=64,\ filter_size=(3,3),stride=(2,2),output_size=(14,14),crop='same',nonlinearity=rectify)
p_mean=lasagne.layers.TransposedConv2DLayer(network,1,\ filter_size=(3,3),stride=(2,2),output_size=(28,28),crop='same',nonlinearity=sigmoid)
p = Deterministic(p_mean,given=[z]) #p(x|z)
Other codes are not changed except for some minor changes to avoid error.
However, the generation is poor.
So, how can I add conv operation, is there something wrong?
Thank you!
I've implemented the JMVAE models on v0.0.2.
https://arxiv.org/abs/1611.01891
I'm going to implement them on v0.0.3. It is necessary to be aware that some specifications have changed from v0.0.2 to v.0.0.3.
To make sure that the sampling/log_likelihood methods return proper values.
In the current implementation, we implement prior distributions as the Distribution_sample class but I believe that it is better to implement them as the Distribution_model class.
This issue is quite related to #54 .
今のVAEクラスで問題ないが,入力が3次元なので,次の問題がある.
・t_repeatが2次元or4次元しか対応していない.
・distributionのmean_sum_samplesが2次元or4次元しか対応していない.
Such as Distribution
, Bernoulli
, and Gaussian
.
公開日をいつにしようか問題.
・個人的には来月頭,早ければ今月末には公開したい.
・公開に向けてやることリストとスケジュールを考える必要がある.
・何人かに手伝ってもらう予定
I've implemented the distribution class "Multilayer" which enable to model multi-latent layers.
But it can't be used in the current version, so we should fix it.
We would like to implement generative adversarial nets as "implicit distribution" as follows:
z = InputLayer((None, z_dim))
g = batch_norm(DenseLayer(z,num_units=512,nonlinearity=activation))
g = batch_norm(DenseLayer(g,num_units=512,nonlinearity=activation))
g_mean = DenseLayer(g,num_units=x_dim,nonlinearity=sigmoid)
g = Deterministic(g_mean,given=[z]) #p(x|z)
x = InputLayer((None, x_dim))
d_0 = DenseLayer(x,num_units=512,nonlinearity=leaky_rectify)
d_1 = DenseLayer(d_0,num_units=512,nonlinearity=leaky_rectify)
d_mean = DenseLayer(d_1,num_units=1,nonlinearity=sigmoid)
d = Bernoulli(d_mean,given=[x])
p = ImplicitDistribution(mean=g, loss=d)
VRNNを実装したが,現在うまく動作しない.
おそらく,学習部分か推論部分にバグがあると思われる.
DRAW [Gregor+ 2015]を実装したい.
・https://github.com/skaae/lasagne-draw を参考にすれば実装できそう.
def test_consistency(self):
# Ensure that returned values stay the same when setting a fixed seed.
mu, beta = 0, 1
gumbel_sample = Gumbel_sample(temp=0.01, seed=1234567890)
actual = TestGumbelSample.get_sample(mu, beta, gumbel_sample, 5)
desired = [
1.7462246417999268,
0.2874840497970581,
0.9974965453147888,
0.3290553689002991,
-1.0215276479721069
]
assert_array_almost_equal(actual, desired, decimal=15)
Added test cases ensuring the consistency of sampling methods, and they all passed on the local environment.
However, if we set the decimals more than 6 or 7, test cases other than Gaussian and Gumbel distributions will fail when run on other machines including servers.
Example log on travis ci:
_____________________ TestBernoulliSample.test_consistency _____________________
self = <Tars.tests.test_distribution_samples.TestBernoulliSample testMethod=test_consistency>
def test_consistency(self):
# Ensure that returned values stay the same when setting a fixed seed.
mean = 0.5
bernoulli_sample = Bernoulli_sample(temp=0.1, seed=1234567890)
actual = TestBernoulliSample.get_sample(mean, bernoulli_sample, 5)
desired = [
0.9999971508356551,
0.9101269246280252,
0.0248156363467501,
0.3538078165724645,
0.1615775890919983
]
> assert_array_almost_equal(actual, desired, decimal=15)
E AssertionError:
E Arrays are not almost equal to 15 decimals
E
E (mismatch 20.0%)
E x: array([ 0.999997150839051, 0.910126924628025, 0.02481563634675 ,
E 0.353807816572465, 0.161577589091998])
E y: array([ 0.999997150835655, 0.910126924628025, 0.02481563634675 ,
E 0.353807816572465, 0.161577589091998])
Tars/tests/test_distribution_samples.py:97: AssertionError
Currently abstract methods in Distribution
are not implemented in its child classes, but in the derived classes of Distribution_sample
even though it calls them in other methods such as sample_given_x
and log_likelihood_given_x
, assuming its child classes inherit Distribution_sample
.
class Distribution(object):
__metaclass__ = ABCMeta
...
def sample_given_x(self, x, repeat=1, **kwargs):
...
return [x, self.sample(*tolist(mean))]
def log_likelihood_given_x(self, samples, **kwargs):
...
return self.log_likelihood(sample, *tolist(mean))
...
@abstractmethod
def sample(self):
pass
@abstractmethod
def log_likelihood(self):
pass
class Gaussian_sample(Distribution_sample):
"""
Gaussian distribution
p(x) = \frac{1}{\sqrt{2*\pi*var}} * exp{-\frac{{x-mean}^2}{2*var}}
"""
def sample(self, mean, var):
"""
Paramaters
----------
mean : Theano variable, the output of a fully connected layer (Linear)
var : Theano variable, the output of a fully connected layer (Softplus)
"""
eps = self.srng.normal(mean.shape, dtype=mean.dtype)
return mean + T.sqrt(var) * eps
def log_likelihood(self, samples, mean, var):
"""
Paramaters
--------
sample : Theano variable
mean : Theano variable, the output of a fully connected layer (Linear)
var : Theano variable, the output of a fully connected layer (Softplus)
"""
loglike = self._gaussian_like(samples, mean, var)
return mean_sum_samples(loglike)
def _gaussian_like(self, x, mean, var):
c = - 0.5 * math.log(2 * math.pi)
_var = var + epsilon() # avoid NaN
return c - T.log(_var) / 2 - (x - mean)**2 / (2 * _var)
class Gaussian(Gaussian_sample, Distribution_double):
def __init__(self, mean_network, var_network, given, seed=1):
Distribution_double.__init__(self, mean_network, var_network, given)
super(Gaussian, self).__init__(seed=seed)
self._set_theano_func()
This is a bit confusing, and it might be better to change the class inheritance structure to resolve this. For example, we could let Distribution
have a variable of Gaussian_sample
in which sample
and log_likelihood
methods are defined, and call them in sample_given_x
and log_likelihood_given_x
methods. Also, we should use camelcase for class names.
https://github.com/jych/cle/blob/master/cle/cost/__init__.py
これに書かれている分布は全て実装したい.
具体的には
・GMM
・BiGMM
・BiGauss(実装済みだが,より汎用的にしたい)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.