carljohanhoel/mcts.jl

Monte Carlo Tree Search for Markov decision processes using the POMDPs.jl framework

MCTS

This package implements the Monte-Carlo Tree Search algorithm in Julia for solving Markov decision processes (MDPs). The user should define the problem according to the generative interface in POMDPs.jl. Examples of problem definitions can be found in POMDPModels.jl. For an extensive tutorial, see this notebook.

There is also a BeliefMCTSSolver that solves a POMDP by converting it to an MDP in the belief space.

Special thanks to Jon Cox for writing the original version of this code.

Installation

After installing POMDPs.jl, start Julia and run the following command:

using POMDPs
POMDPs.add("MCTS")

Documentation

Documentation can be found on the following site: juliapomdp.github.io/MCTS.jl/latest/

Usage Example

If mdp is an MDP defined with the POMDPs.jl interface, the MCTS solver can be used to find an optimized action, a, for the MDP in state s as follows:

using POMDPModels # for the GridWorld problem
using MCTS
mdp = GridWorld()
solver = MCTSSolver(n_iterations=50, depth=20, exploration_constant=5.0)
policy = solve(solver, mdp)
a = action(policy, s)

See this notebook for an example of how to visualize the search tree.

See this notebook for examples of customizing solver behavior, specifically the Rollouts section for using heuristic rollout policies.