sarmueller/gibo

This repository contains the code for the paper "Local policy search with Bayesian optimization".

gibo's Issues

ValueError: Expected value argument to be within the support

Hi,
I would like to reproduce the result of GIBO on Swimmer-v1.
I have use the command as your suggestions python run_rl_experiment.py -c ./configs/rl_experiment/gibo_default.yaml. However, I got a value error because these values are outside the range.
May I change the value a and b of lengthscale_hyperprior? Or is there more general solution for the problem?
Thanks,
Gia-Lac Tran

Follows are the error I got:
--- Iteration 1 (0 objective calls so far) ---
Reward of parameters theta_(t-1): -0.04.
theta_t: [[ 0.01080808 0.0026861 0.02242399 0.05099441 0.00220321 -0.02808427
-0.00069419 -0.02702736 0.02220907 0.00552518 0.01492226 0.00722155
-0.01049817 0.01392695 -0.00154528 0.01221718]] predicted mean 0.32 and variance 0.07 of f(theta_i).
lengthscale: [[0.15499999 0.15499999 0.15499999 0.15499999 0.15499999 0.15499999
0.15499999 0.15499999 0.15499999 0.15499999 0.15499999 0.15499999
0.15499999 0.15499999 0.15499999 0.15499999]], outputscale: 2.0, noise [0.01]
--- Iteration 2 (17 objective calls so far) ---
Reward of parameters theta_(t-1): -0.04.
theta_t: [[ 0.01629168 0.03781871 0.03882242 0.02736505 -0.01680078 -0.0423358
-0.0222224 -0.00730637 0.04065228 -0.03006371 0.00688156 -0.00962662
0.00798267 0.00671405 -0.00651452 0.02337452]] predicted mean 0.25 and variance 0.08 of f(theta_i).
lengthscale: [[0.15499999 0.15499999 0.15499999 0.15499999 0.15499999 0.15499999
0.15499999 0.15499999 0.15499999 0.15499999 0.15499999 0.15499999
0.15499999 0.15499999 0.15499999 0.15499999]], outputscale: 2.0, noise [0.01]
--- Iteration 3 (34 objective calls so far) ---
Reward of parameters theta_(t-1): 0.05.
Traceback (most recent call last):
File "run_rl_experiment.py", line 72, in
params, calls_in_iteration = loop(
File "/home/gialac/gibo_origin/gibo/src/loop.py", line 76, in loop
optimizer()
File "/home/gialac/gibo_origin/gibo/src/optimizers.py", line 47, in call
self.step()
File "/home/gialac/gibo_origin/gibo/src/optimizers.py", line 619, in step
botorch.fit.fit_gpytorch_model(mll)
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/botorch/fit.py", line 125, in fit_gpytorch_model
mll, _ = optimizer(mll, track_iterations=False, **kwargs)
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/botorch/optim/fit.py", line 239, in fit_gpytorch_scipy
res = minimize(
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/scipy/optimize/_minimize.py", line 623, in minimize return _minimize_lbfgsb(fun, x0, args, jac, bounds,
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/scipy/optimize/lbfgsb.py", line 360, in _minimize_lbfgsb
f, g = func_and_grad(x)
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py", line 267, in fun_and_grad
self._update_fun()
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py", line 233, in _update_fun
self._update_fun_impl()
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py", line 137, in update_fun
self.f = fun_wrapped(self.x)
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py", line 134, in fun_wrapped
return fun(np.copy(x), *args)
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/scipy/optimize/optimize.py", line 74, in call
self._compute_if_needed(x, *args)
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/scipy/optimize/optimize.py", line 68, in _compute_if_needed
fg = self.fun(x, *args)
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/botorch/optim/utils.py", line 215, in _scipy_objective_and_grad
loss = -mll(*args).sum()
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/gpytorch/module.py", line 30, in call
outputs = self.forward(*inputs, **kwargs)
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/gpytorch/mlls/exact_marginal_log_likelihood.py", line 63, in forward
res = self._add_other_terms(res, params)
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/gpytorch/mlls/exact_marginal_log_likelihood.py", line 43, in add_other_terms
res.add(prior.log_prob(closure(module)).sum())
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/gpytorch/priors/prior.py", line 27, in log_prob
return super(Prior, self).log_prob(self.transform(x))
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/torch/distributions/uniform.py", line 73, in log_prob
self._validate_sample(value)
File "/home/gialac/anaconda3/envs/gibo/lib/python3.8/site-packages/torch/distributions/distribution.py", line 288, in _validate_sample
raise ValueError(
ValueError: Expected value argument (Tensor of shape (1, 16)) to be within the support (Interval(lower_bound=0.009999999776482582, upper_bound=0.30000001192092896)) of the distribution UniformPrior(low: 0.009999999776482582, high: 0.30000001192092896), but found invalid values:
tensor([[0.9817, 0.5958, 1.1015, 0.5158, 0.8297, 1.1924, 0.8811, 0.8303, 0.9897,
0.6611, 0.9598, 0.8612, 0.8075, 0.9080, 0.9310, 0.9350]],
grad_fn=)

Kernel grad in acquisition function

Hello,

I noticed the the kernel grad w.r.t. the input x has two separate implementations:

• One in the model (https://github.com/sarmueller/gibo/blob/main/src/model.py#L225-L245) where X is the training set.
• The other in the acquisition function (https://github.com/sarmueller/gibo/blob/main/src/acquisition_function.py#L41-L61) where X could be anything.

However, the first one squares the length scale on line 240 while the other doesn't on line 56. As far as I know, the length scale should indeed be squared in the kernel grad, so is this a bug in the acquisition function?

Thanks in advance,
Quan