turinglang / turing.jl Goto Github PK
View Code? Open in Web Editor NEWBayesian inference with probabilistic programming.
Home Page: https://turinglang.org
License: MIT License
Bayesian inference with probabilistic programming.
Home Page: https://turinglang.org
License: MIT License
On Ubuntu 16.04, starting with a fresh Julia install, I'm getting the following error. Could there be an undocumented dependency?
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.4.5 (2016-03-18 00:58 UTC)
_/ |\__'_|_|_|\__'_| |
|__/ | x86_64-linux-gnu
julia> Pkg.update
update (generic function with 1 method)
julia> Pkg.update()
INFO: Initializing package repository /home/chad/.julia/v0.4
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
INFO: Updating METADATA...
INFO: Computing changes...
INFO: No packages to install, update or remove
julia> Pkg.add("Turing")
INFO: Cloning cache of ArrayViews from git://github.com/JuliaLang/ArrayViews.jl.git
INFO: Cloning cache of Compat from git://github.com/JuliaLang/Compat.jl.git
INFO: Cloning cache of ConjugatePriors from git://github.com/JuliaStats/ConjugatePriors.jl.git
INFO: Cloning cache of Distributions from git://github.com/JuliaStats/Distributions.jl.git
INFO: Cloning cache of PDMats from git://github.com/JuliaStats/PDMats.jl.git
INFO: Cloning cache of StatsBase from git://github.com/JuliaStats/StatsBase.jl.git
INFO: Cloning cache of StatsFuns from git://github.com/JuliaStats/StatsFuns.jl.git
INFO: Cloning cache of Turing from git://github.com/yebai/Turing.jl.git
INFO: Installing ArrayViews v0.6.4
INFO: Installing Compat v0.7.14
INFO: Installing ConjugatePriors v0.1.2
INFO: Installing Distributions v0.8.9
INFO: Installing PDMats v0.3.6
INFO: Installing StatsBase v0.8.0
INFO: Installing StatsFuns v0.2.0
INFO: Installing Turing v0.0.1
INFO: Building Turing
ERROR: could not open file /usr/share/julia/julia-config.jl
ERROR: could not open file /usr/share/julia/julia-config.jl
ERROR: could not open file /usr/share/julia/julia-config.jl
gcc -O2 -shared -fPIC task.c -o libtask.so
task.c:6:19: fatal error: julia.h: No such file or directory
compilation terminated.
Makefile:9: recipe for target 'task' failed
make: *** [task] Error 1
===============================[ ERROR: Turing ]================================
LoadError: failed process: Process(`make`, ProcessExited(2)) [2]
while loading /home/chad/.julia/v0.4/Turing/deps/build.jl, in expression starting on line 1
================================================================================
================================[ BUILD ERRORS ]================================
WARNING: Turing had build errors.
- packages with build errors remain installed in /home/chad/.julia/v0.4
- build the package(s) and all dependencies with `Pkg.build("Turing")`
- build a single package by running its `deps/build.jl` script
================================================================================
INFO: Package database updated
Tests do not pass for v0.5.
I looked into the errors and also did the test on my machine, finding that the SMC
and PG
engines seem to be broken. I guess it may be resulted from the update of task.c
?
~
: macro => function #173randoc
, randc
? #156We can remove isdefined()
using the fact that users are only allowed to use TArray
in @model
.
We need to remove the current doc
and use the automatic generate doc API to generate it.
We need to introduce an interval for outputting sampler status, and keep it consistent across all samplers.
I think it is better to keep examples in a single repository as a submodule in Turing.jl
. How do you think?
We have some undecided interface issues since the introduction of the Hamiltonian Monte Carlo (HMC) sampler. More specifically, we need an interface to tell the HMC sampler which variable it should handle. In addition, we also need a way to compose samplers (e.g. Gibbs sampling subsets of variables with different samplers). To solve these issues altogether, I propose the following new interface.
## Model definition
gaussdemo() = begin
s ~ InverseGamma(2,3)
m ~ Normal(0,sqrt(s))
for i=1:length(x)
x[i] ~ Normal(m, sqrt(s))
end
return(s, m, x)
end
# Standard sampler interface
res = sample(gaussdemo, data = Dict(:x=>[1 2]), HMC(..., :s, :m))
# Compositional inference interface
res = sample(gaussdemo, data = Dict(:x=>[1 2]), HMC(..., :m), PG(..., :s))
# Sampler's default behavior is to sample all random variables
res = sample(gaussdemo, data = Dict(:x=>[1 2]), HMC(...))
# Tagging parameters for samplers like PMC, SMC2
res = sample(gaussdemo, data = Dict(:x=>[1 2]), PMC(..., params=[:m]))
# Running program without a sampler produces a draw from the prior
s, m = sample(gaussdemo)
s, m = gaussdemo()
In summary, first, this design removed macros @assume
, @observe
and @predict
. Instead, it introduces additional parameters to each sampler to distinguish different types of variables. Second, this design introduces an interface to compose samplers (e.g. HMC and PG).
I met the following error when running the gdemo example in the README.md file.
ERROR: KeyError: key :modelarglist not found
in getindex; at ./dict.jl:688 [inlined]
in macro expansion; at /Users/yebai/.julia/v0.5/Turing/src/core/compiler.jl:290 [inlined]
in anonymous at ./<missing>:?
Chen, T., Fox, E. B., & Guestrin, C. (2014, February 17). Stochastic Gradient Hamiltonian Monte Carlo. arXiv.org.
The proposal below is naive.
The Stochastic HMC can use the current code with only making some changes to observe()
:
1. Count the total number of observations in the first run of the model
2. For later iterations
i. Choose a subset of observations for each iteration (according to obs_frac
(observation fraction) set by user)
ii. Only accumulate logpdf of those chosen observations (Note: the subset is fixed for each HMC iteration)
This two changes are done in this branch: https://github.com/yebai/Turing.jl/tree/stochhmc and some simple tests are done, which initially looks fine.
Followed our discussion on correlation last time, I think of using the following way to generate correlated observation sets:
1. User set a correlation ratio corr_rate
2. When generating new observation subset, fix a fraction of observations according to corr_rate
and sample the remaining
Expected problem: when obs_frac
is large, we would not able to do so. E.g.
We have 10 obs and obs_frac
is 0.8. Say our first subset is [1,2,3,4,5,6,7,8].
User pick corr_rate
to be 0.125. Then we need to fix one obs, say [1] and sample 7 obs from [9, 10], which is impossible.
The current form for reporting results is a bit arbitrary. It makes sense to standardise output and report results in the form of joint posterior.
FYI. The Mamba.jl package has a format for reporting Monte Carlo results. It is also the default format for Stan.jl. But we might need to adapt this format to store sample weights.
I wonder if we should create a better hierarchy for samplers. The current was seems a bit arbitrary: we use ParticleSampler
in the default sample function and overload some in their own place (IS and HMC). Also, when I split AD from HMC, I put the type of sampler it can accept as GradientSampler
, which could be meaningful for future work.
Shall we try to design something like this:
abstract Sampler{T<:InferenceAlgorithm}
abstract ParticleSampler{T} <: Sampler{T}
abstract GradientSampler{T} <: Sampler{T}
type ImportanceSampler{IS} <: ParticleSampler{IS}
...
end
type SMCSampler{SMC} <: ParticleSampler{SMC}
...
end
type PGSampler{PG} <: ParticleSampler{PG}
...
end
type HMCSampler{HMC} <: GradientSampler{T}
...
end
It's helpful to implement some utility functions for the Chain
type for inspecting samples from various MCMC samplers, e.g.:
At the moment, we have various data structures that stores intermediate (MCMC) sampling states. It probably makes sense to unify these data structures.
(I don't have a complete idea how to do this yet, but let's start thinking about this issue and post ideas here.)
Hello,
Great package. I have some questions if anyone has a second.
Thanks!
I guess it's due to my new way to run tests. Need to fix it ASAP.
Maybe we should implement some models from the Stan examples repo and compare Stan's result with Turing's result. This would allow us more systematically test our Gibbs sampler and provide some insights to the efficiency of current code.
For a start, we could implement some models in the basic_estimator
folder and misc
folder?
The HMC branch introduces a bug to the compiler, basically the assume
macro converts all distributions to ddistribution type. This causes problems for distributions without corresponding distribution type.
Create a readthedoc website, see e.g. Distributions
@xukai92 The current compiler does not handle models without explicit return
statement. Could we return all variables by default when return
is absent?
Keeps getting this deprecated WARNING msg.
This is a suggestion from Brooks, which I support, to remove @predict
in favour of standard return
. It accomplishes the same goal without introducing new syntax. Also the experience of Anglican team indicates that people often associate @predict
with the posterior predictive distribution, which is clearly not what it is.
There are some changes in the master branch that prepares Turing for Julia 0.5. Since we are shifting to Julia 0.5 gradually, it makes sense to merge changes in the master branch back to development branch.
The current form for reporting MCMC output is a bit arbitrary. It makes sense to standardise the output and report results in the form of joint posterior.
It's also helpful to implement some utility functions for the Chain type for inspecting samples from various MCMC samplers, e.g.:
IAT: integrated autocorrelation time: http://search.r-project.org/library/LaplacesDemon/html/IAT.html
ESS: effective sample size: http://search.r-project.org/library/LaplacesDemon/html/ESS.html
MCSE: Monte Carlo standard error: http://search.r-project.org/library/LaplacesDemon/html/MCSE.html
PS. The Mamba.jl package has a format for reporting Monte Carlo results. It is also the default format for Stan.jl. But we might need to adapt this format to store sample weights.
Hi @yebai , our notebooks actually uses the development version of Turing. Do you think we need to add codes to switch from master to development in the 01-Introduction.ipynb
?
@xukai92 could you have a look at https://travis-ci.org/yebai/Turing.jl/builds/153336076 and see what breaks the development branch? Seems like merging HMC introduces some incompatibility issues.
This is a remainder that 076844d contains some temp fix in
in order to fix package conflicts when using the newest version of Distributions.jl.
We need to remove these stuff when there are new releases of relevant packages.
@yebai When do you plan to merge the two branches?
@assume z ~ Binomial()
gives LoadError: BoundsError: attempt to access 1-element Array{Any,1} at index [2]
The PR for StatsFun.jl
is accepted. We can remove the hacking code now.
I'm gonna put issues about implementing the HMC sampler here.
One of them is about the compiler. I wonder why we convert the @observe x ~ Distribution(params)
to the logpdf()
statement in the compiler level instead of doing this inside the sampler?
It makes me hard to pass Dual
type to my custom distribution wrapper in order to get the gradient information.
Do you think I need the amend the compiler or there is any other approach? @yebai
Here is my plan for the improvement of the HMC sampler (with priorities from high to low)
Progress:
dDistribution
with Distribution
I've already looked into how to do the first one. I've check the Distribution
package and see most of the distributions are supporting Dual
now, expect some Kolmogorov-related distributions which I'm not even familiar with. I believe this one can be done soon.
For discussion of error caused by large step size here: #66 (comment)
It would be nice if we can have a user-friendly interface for automatic differentiation, e.g.
@model gdemo(x) = begin
s ~ InverseGamma(2,3)
m ~ Normal(0,sqrt(s))
for i=1:length(x)
x[i] ~ Normal(m, sqrt(s))
end
return(s, m, x)
end
g = @gradient(gdemo([2.0, 3.0]), varInfo = nothing)
By default, the user does not need to pass in varInfo
(i.e. differentiate through all parameters).
This would be very helpful for users who want to contribute new gradient-based inference methods.
Out test coverage is 68% for the development branch at this issues is post, and I am aiming to improve it to over 90%. There are fews things need to be known for all.
Keep runtest.jl
structured
Please check the new runtest.jl
file. When adding new tests in the future, keep it structured using the current way so that we can tell which source file we are testing for each test case.
Review of old test cases
I noticed some comments in them are out-of-date. @yebai Can you also have a check and remove unnecessary ones.
Add new tests
Clearly there are some source files not having specific test files. I guess we can add more along with adding more commands ( #29 ), which is not actively in process.
The current interface pass in variables through a dictionary. This design requires us implicitly copy all items in the data dictionary to model's scope, which is not the most consistent design with Julia's function interface. A possible better design would be:
@model gdemo(x) = begin
s ~ InverseGamma(2,3)
m ~ Normal(0,sqrt(s))
for i=1:length(x)
x[i] ~ Normal(m, sqrt(s))
end
return(s, m, x)
end
x = [2.0, 3.0]
sampler = Gibbs(HMC(..., :m), PG(..., :s))
chn = @sample(gdemo(x), sampler)
The most important changes are:
=
in model declaration, e.g. gdemo(x) = statement_block
. We might want to depreciate the syntax without =
in the future (i.e. gdemo(x) statement_block
) to stay in line with Julia's assignment form of function definitions.EDIT (28 FEB 2017): removed obsolete comment.
We need a plan for better organisation of the source code. We can collect issues regarding this here.
For example, code in /src/core/intrinsic.jl
seems to be only relevant to samplers (and a bit messy). It would be easy to maintain if we move it to /src/samplers/
(or sub-folder).
We've enabled automated test for Unix and Mac. It makes sense to also enable automated test for Windows. An example configuration file can be found at link.
We should create a test for passing dual types to Distribution constructors, e.g.
d1 = Dual(1.1)
Normal(d1, d1)
The test can include most common distributions (gamma, beta, normal, mvnormal etc).
This parallels with the test for passing dual types to logpdf
function.
The current HMC sampler does not work with truncated distribution. For example, the following example would throw an error
negbindata = [0, 1, 4, 0, 2, 2, 5, 0, 1]
@model negbinmodel(y) begin
local α, β
α ~ Truncated(Cauchy(0,10), 0, +Inf)
β ~ Truncated(Cauchy(0,10), 0, +Inf)
for i = 1:length(y)
y[i] ~ NegativeBinomial(α, β) # α > 0, 0 < β < 1
end
return(α, β)
end
@sample(negbinmodel(negbindata), HMC(1000, 0.02, 1))
ERROR MESSAGE:
MethodError: no method matching pdf(::Distributions.Truncated{Distributions.Cauchy{Float64},Distributions.Continuous}, ::ForwardDiff.Dual{0,Float64})
Closest candidates are:
pdf(::Distributions.Distribution{Distributions.Univariate,Distributions.Continuous}, ::Real) at /Users/yebai/.julia/v0.5/Distributions/src/univariates.jl:102
pdf(!Matched::Distributions.Kolmogorov, ::Real) at /Users/yebai/.julia/v0.5/Distributions/src/univariate/continuous/kolmogorov.jl:74
pdf(!Matched::Distributions.Arcsine{T<:Real}, ::Real) at /Users/yebai/.julia/v0.5/Distributions/src/univariate/continuous/arcsine.jl:74
...
in logpdf(::Distributions.Truncated{Distributions.Cauchy{Float64},Distributions.Continuous}, ::ForwardDiff.Dual{0,Float64}, ::Bool) at transform.jl:39
in assume(::Turing.HMCSampler{Turing.HMC}, ::Distributions.Truncated{Distributions.Cauchy{Float64},Distributions.Continuous}, ::Turing.Var, ::Turing.VarInfo) at hmc.jl:154
in macro expansion at compiler.jl:10 [inlined]
in negbinmodel(::Dict{Any,Any}, ::Turing.VarInfo, ::Turing.HMCSampler{Turing.HMC}) at compiler.jl:21
in step(::#negbinmodel, ::Dict{Any,Any}, ::Turing.HMCSampler{Turing.HMC}, ::Turing.VarInfo, ::Bool) at hmc.jl:59
in run(::Function, ::Dict{Any,Any}, ::Turing.HMCSampler{Turing.HMC}) at hmc.jl:109
in sample(::Function, ::Dict{Any,Any}, ::Turing.HMC) at hmc.jl:184
in macro expansion; at compiler.jl:295 [inlined]
in anonymous at <missing>:?
in include_string(::String, ::String) at loading.jl:441
in include_string(::String, ::String, ::Int64) at eval.jl:28
in include_string(::Module, ::String, ::String, ::Int64, ::Vararg{Int64,N}) at eval.jl:32
in (::Atom.##53#56{String,Int64,String})() at eval.jl:40
in withpath(::Atom.##53#56{String,Int64,String}, ::String) at utils.jl:30
in withpath(::Function, ::String) at eval.jl:46
in macro expansion at eval.jl:57 [inlined]
in (::Atom.##52#55{Dict{String,Any}})() at task.jl:60
We really need better comment practice for our source code. I guess we can start to improve it by assigning those source code to who wrote it.
Using """
to add comments before functions so that they can be captured by Base.Docs.binding
(more practice see documentation)
This style of documentation can be also easily toggled in Juno/Atom, which would potentially help future development a lot.
We may also refer to other good open source projects, e.g. normal.jl, in which the comments contains three parts 1) description 2) example code 3) external link.
Please tick once the file is well commented (file list is from the development branch).
/
Turing.jl
core/
compiler.jl
io.jl
samplers/
hmc.jl
is.jl
pgibbs.jl
smc.jl
trace/
tarray.jl
We should have a list of tests for the Gibbs constructor. The current implementation is a bit fragile and incomplete. For example, we should at least test the following cases:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.