Julia packages build errors while following the install_env_gail_full.md

[sisl]

Julia packages build errors while following the install_env_gail_full.md

[Wang4758]

@sblinz I had the same error before. The error was because the agent _infos dictionary contains the 'latent' key. You can fix it by replacing the get_action function in gaussian_latent_var_gru_policy.py by following codes:

def get_action(self, observation):
    actions, agent_infos = self.get_actions([observation]) 
    infos = dict()
    for k, v in agent_infos.items():
        if k is 'latent_info':
            infos[k] = dict()
            infos[k]['latent'] = v['latent'][0]
        else:
            infos[k] = v[0]
    # return actions[0], {k: v[0] for k, v in agent_infos.items()}
    return actions[0], infos

[sblinz-zz]

I also saw the BayesNets warning and had to install v0.7.0. After that things started working but there were a lot other changes I had to make to install_env_gail_full.md to finish.

I'm on Mac OS 10.14 and I've never used Julia before today so very possible not all of this is necessary or is specific to my system but it worked for me.

Note that v0.7.0 doesn't use juliarc.jl for setup but looks in ~/.julia/config/startup.jl

Here is what I did differently than install_env_gail_full.md:

1. Get AutomotiveDrivingModels

• cd ~/.julia/dev/AutomotiveDrivingModels
• git checkout 74050e9ae44bda72a485c2573ac4f0df2bc3e767
• remove semi-colon in src/2d/roadway/roadways.jl:591
• remove comment block in src/2d/roadway/roadway_generation.jl:111-123
• change Base.parse() function signature in src/2d/roadway/roadways.jl:35 to line::AbstractString

2. Run julia interpreter and run Pkg.add("Iterators"); don't add Iterators to startup.jl as recommended by the v0.6.0 instructions, that caused a lot of problems for me until I removed it

3. On Mac libz.so is libz.dylib so find it and add to LD_PRELOAD

4. Vec

• cd ~/.julia/dev/Vec
• git checkout ab4f652aa12a2d06433e62bb39e0a17f06ca9cbf (v0.1.0 release)
• I'm new to julia and couldn't figure out how to get the interpreter to use dev/Vec instead of packages/Vec so I just copied the dev/Vec files to packages/Vec. I'm sure there is a more correct way using julia dev and free commands.

5. AutoViz

• cd ~/.julia/dev/AutoViz
• git checkout f04a667b54de3d4b0252b958898577e6243c6f57 (v0.6.0 release)
• add Reexport to REQUIRE file

6. I had a lot of weird issues with Pkg.build("HDF5"), don't remember them all but follow the errors.

7. Pkg.add("***") followed by using *** for each of the following packages (I'm sure there is proper dependencies way of doing this but I'm very new to Julia and couldn't figure out how. Adding them or their git repo's to REQUIRE or build.jl didn't work)

• DataStructures
• Parameters
• StatsBase

[djp42]

We have also been encountering related errors this Fall - it seems ever since julia v1.0 came out, so its not too surprising - but we are not necessarily sure about the best way to fix them besides just updating everything to v1.0 which we frankly do not have the bandwidth for at the moment. Another alternative would be to host a docker image or something like that somewhere, with the versions we use.

It seems like, from that error, some of the packages dropped support for julia0.6, which is the one we first were developing on. So we might be forced to just suck it up and update to julia v1.0, which will hopefully be supported for longer than the incremental versions... but who knows. Any help is appreciated :)

[sblinz-zz]

I've managed to get through all of the ngsim_env package installations and the last step using AutoEnvs now works.

Now I'm stuck on the NGSIM data part. When I run convert_raw_ngsim_to_trajdatas() it takes a really long time and seems to hang in various places in NGSIM/src/trajdata.jl.

Sometimes it hangs at the for loop for carid in carid_set(tdraw) inside load_ngsim_trajdata(). Other times it gets through that for loop but hangs somewhere inside Base.convert()

[djp42]

Hmm, never come across that issue before. Have you checked your memory usage while running it? We had other issues with out of memory errors at some points in the past.

Glad to see you got the packages working though, and thank you for recording the steps!

[sblinz-zz]

It's iterating through the for loop in Base.convert() now, just taking very long running locally on my machine. Seems to have been related to package loading in julia (I still don't understand it well). Once I restarted the interpreter it got to the convert() call for the first file at least.

I jumped ahead to the python tests and am getting segfaults. I don't expect them to pass but pretty sure the segfaults are not related to needing the trajectory data? I tried removing the PyCall.jl file as recommended by the instructions but the only one I have is in ~/.julia/packages/PyCall/.../src and when I removed that I just get PyCall not found errors. I also tried removing ~/.julia/compiled/v0.7/PyCall/***.jl but that didn't help. Maybe I need to remove something else? Or the entire PyCall package? Keep in mind I'm using julia v0.7.0 so slightly different directory structure than v0.6.2 that the instructions assume.

Thanks very much for responses, I get you guys don't have a lot of bandwidth for this right now. We are planning a Docker container once we get can get to the end and train and validate a policy. Any help is greatly appreciated though!

[djp42]

Hmm, I do remember it taking a good bit of time but not an absurd amount. And the good news is you only have to do it once.

For the segfaults, do you have the log for what the error looked like? I remember once encountering non-pycall segfaults, but it wasn’t too complicated to fix.

Also regarding pycall, if the cache doesn’t exist or whatever, just remove that PyCall.jl and try to reinstall it, i believe through another Pkg.add() command. Alternatively, just remove and reinstall the actual PyCall package.

This may help with clearing the cache, although as a disclaimer I have never run the code there so I do not know that it works. https://discourse.julialang.org/t/how-do-you-clear-cache-from-a-precompile/7821/2

Let me know if that helps, and I will do my best to keep up-to-date with this thread, I appreciate your patience, and look forward to the docker image :). I am hopeful we will get a v1.0 up and running in the near future as well..

[sblinz-zz]

I deleted the directories ~/.julia/compiled/v0.7/PyCall and ~/.julia/packages/PyCall and then ran Pkg.add("PyCall"); it re-created and ran build on it but still getting the segfaults. Here's the first one (there is one for each test, all similar):

(rllab3) (anaconda2-5.2.0) reba:tests sb$ python runtests.py 
test_julia_env (test_julia_env.TestJuliaEnv) ... WARNING: `JULIA_HOME` environment variable is renamed to `JULIA_BINDIR`

signal (11): Segmentation fault: 11
in expression starting at no file:0
jl_apply_tuple_type_v_ at /Users/osx/buildbot/slave/package_osx64/build/src/jltypes.c:1326 [inlined]
jl_apply_tuple_type_v at /Users/osx/buildbot/slave/package_osx64/build/src/jltypes.c:1341
arg_type_tuple at /Users/osx/buildbot/slave/package_osx64/build/src/gf.c:1654
jl_lookup_generic_ at /Users/osx/buildbot/slave/package_osx64/build/src/gf.c:2149
jl_apply_generic at /Users/osx/buildbot/slave/package_osx64/build/src/gf.c:2179
do_call at /Users/osx/buildbot/slave/package_osx64/build/src/interpreter.c:324
eval_stmt_value at /Users/osx/buildbot/slave/package_osx64/build/src/interpreter.c:363 [inlined]
eval_body at /Users/osx/buildbot/slave/package_osx64/build/src/interpreter.c:686
jl_interpret_toplevel_thunk_callback at /Users/osx/buildbot/slave/package_osx64/build/src/interpreter.c:799
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x1149ed0ef)
unknown function (ip: 0xffffffffffffffff)
jl_interpret_toplevel_thunk at /Users/osx/buildbot/slave/package_osx64/build/src/interpreter.c:808
jl_toplevel_eval_flex at /Users/osx/buildbot/slave/package_osx64/build/src/toplevel.c:831
jl_toplevel_eval_flex at /Users/osx/buildbot/slave/package_osx64/build/src/toplevel.c:778
jl_toplevel_eval_in at /Users/osx/buildbot/slave/package_osx64/build/src/builtins.c:633
eval at ./boot.jl:319
exec_options at ./logging.jl:317
_start at ./client.jl:432
true_main at /usr/local/bin/julia (unknown line)
main at /usr/local/bin/julia (unknown line)
Allocations: 4642 (Pool: 4632; Big: 10); GC: 0
ERROR
test_multiagent_ngsim_env (test_multiagent_ngsim_env.TestMultiagentNGSIMEnv) ... WARNING: `JULIA_HOME` environment variable is renamed to `JULIA_BINDIR`

[djp42]

Never seen that segfault before. It might be something related to macOS, we have done all the development exclusively in Ubuntu.

My suggestion would be to look for others with issues of julia segfaulting on macOS.
This thread was one of the few I saw that included jlytpes in the segfault: https://groups.google.com/forum/#!topic/julia-users/UxaeV1YhshQ

It might not be related to OS, just a thought.

[sblinz-zz]

@djp42 Thanks for keeping along. I'm running into some serious version conflicts between v0.7.0 julia which I started along and some non SISL packages (currently wit JLD). Lot's of packages ngsim depends on have moved along in development and broke compatibility with v0.6 julia at some point. It's kind of whack0-a-mol.

Is there any chance you can share the Manifest and Project toml files for your ngsim installation that works? Doesn't even matter what julia version - with the Manifest I could get all the exact git commits that worked then and exactly re-create the installation.

[djp42]

@sblinz no problem, I want to get this to work too :).

This is the only toml file I have:
Project.toml.zip. I will update tomorrow with if someone else's repo has others.

What Manifest files do you want? These are what shows up for a manifest search:

./miniconda3/pkgs/qt-5.6.2-5/mkspecs/common/winrt_winphone/manifests/10.0/AppxManifest.xml.in
./miniconda3/pkgs/qt-5.6.2-5/mkspecs/common/winrt_winphone/manifests/8.1/AppxManifest.xml.in
./miniconda3/pkgs/qt-5.6.2-5/mkspecs/common/winrt_winphone/manifests/8.1_wp/AppxManifest.xml.in
./miniconda3/pkgs/qt-5.6.2-h974d657_12/mkspecs/common/winrt_winphone/manifests/10.0/AppxManifest.xml.in
./miniconda3/pkgs/qt-5.6.2-h974d657_12/mkspecs/common/winrt_winphone/manifests/8.1/AppxManifest.xml.in
./miniconda3/pkgs/qt-5.6.2-h974d657_12/mkspecs/common/winrt_winphone/manifests/8.1_wp/AppxManifest.xml.in
./miniconda3/envs/rllab3/mkspecs/common/winrt_winphone/manifests/10.0/AppxManifest.xml.in
./miniconda3/envs/rllab3/mkspecs/common/winrt_winphone/manifests/8.1/AppxManifest.xml.in
./miniconda3/envs/rllab3/mkspecs/common/winrt_winphone/manifests/8.1_wp/AppxManifest.xml.in
./.julia/v0.6/Conda/deps/usr/pkgs/qt-5.6.2-h974d657_12/mkspecs/common/winrt_winphone/manifests/10.0/AppxManifest.xml.in
./.julia/v0.6/Conda/deps/usr/pkgs/qt-5.6.2-h974d657_12/mkspecs/common/winrt_winphone/manifests/8.1/AppxManifest.xml.in
./.julia/v0.6/Conda/deps/usr/pkgs/qt-5.6.2-h974d657_12/mkspecs/common/winrt_winphone/manifests/8.1_wp/AppxManifest.xml.in
./.julia/v0.6/Conda/deps/usr/mkspecs/common/winrt_winphone/manifests/10.0/AppxManifest.xml.in
./.julia/v0.6/Conda/deps/usr/mkspecs/common/winrt_winphone/manifests/8.1/AppxManifest.xml.in
./.julia/v0.6/Conda/deps/usr/mkspecs/common/winrt_winphone/manifests/8.1_wp/AppxManifest.xml.in

[sblinz-zz]

@djp42 The Manifest I mean is part of the Julia installation, it's a manifest for the different Julia packages that are installed. For v0.6 it should be in something like ~/.julia/environments/v0.6/Manifest.toml (mine is in this path with v0.7). Your manifest files are for specific packages.

Here is a snippet of my ~/Users/.../.julia/enviroments/v0.7/Manifest.toml

[[AssetRegistry]]
deps = ["Distributed", "JSON", "Pidfile", "Pkg", "SHA", "Test"]
git-tree-sha1 = "b25e88db7944f98789130d7b503276bc34bc098e"
uuid = "bf4720bc-e11a-5d0c-854e-bdca1663c893"
version = "0.1.0"

[[AutoRisk]]
deps = ["Compat", "DataArrays", "DataFrames", "Discretizers", "Distributions", "HDF5", "Reexport"]
path = "/Users/.../.julia/dev/AutoRisk"
uuid = "e0b28f5a-3ea4-5d34-bf3f-495f78123ff5"
version = "0.0.0"

The key thing is that ordinary installed packages (e.g., the first one here) list the git commit that that package is checked out at (for ordinary packages this would have been the most recent commit at the time it was added; by now it would be an old commit). For development packages (like AutoRisk above), there is no git commit listed but if you go to the path (e.g., ~/Users/.../.julia/dev/AutoRisk) and run git status you can get the commit. These are packages that you essentially move to the dev/ folder so you can change them (e.g., check out other commits or just modify the code). The ngsim_env installation has a few like this.

If I can get my hands on such a manifest file from a working installation of ngsim_dev I can at least get all the non-dev package commits that work together (e.g., BayesNets, JLD, etc). For the dev one's I can probably figure them out myself as the main one's are already listed in the instructions.

Thanks

[djp42]

I checked with my other teammate and he also does not have the Manifest file. It might be something only for Julia 0.7+ (I saw something similar when Googling it). I'm beginning to think we may want to try another approach... We will potentially be able to spend some more time updating and/or docker filing after Nov 12 or so, but no promises there. Currently we are trying to copy the .julia of a working install to other computers and see if it works (we have had issues doing so over the past week or two). But even if it does, it might not for you because of MacOS vs Linux.

[sblinz-zz]

Thanks for checking. Yeah, the Mac + v0.7 julia approach is not going to happen, too many conflicting parts, so I'm going to try it from scratch in a Docker Linux container with v0.6. I'll let you know of my progress, hopefully I can get it working before mid November.

[djp42]

Development: I have successfully gotten past the first few phases of issues on a blank Ubuntu 16.04 LTS.
Steps: download and extract the zip files for .julia (very large. Has all of the packages).
Follow the install steps, with some exceptions:

1. Before the ngsim_env steps, there is a super hacky step, to avoid path issues. Basically, we create a symbolic link in /home/ that has my username on the working install dphillips and links it to your home directory. I didn't have enough time to find where to actually fix the paths, but on using PyCall and using Conda there were path issues. Fixing the Conda issue was just to run this command: export CONDA_JL_HOME="/<path/to>/rllab3", but didnt have time to figure out the other one. The hacky solution is: sudo ln -s /home/<your_username>/ /home/dphillips
2. I don't think we need to revert the previous versions.
3. I didn't get further than the using PyCall call, but I think that's pretty good for now.

Not sure if that is applicable since you're on Mac, but I wanted to share with you in case you wanted to try your hand at it.
Thanks!

[sblinz-zz]

@djp42 Thanks for the notes. I started from scratch inside a Docker container running Ubuntu 16.04. I was able to get through all the setup instructions through the julia tests julia ngsim_env/julia/test/runtests.jl.

Note: I had to manually check out some older versions of several julia packages to get through all the steps inside julia ngsim_env/julia/deps/build.jl but I was able to get to the end of it and manually build AutomotiveDrivingModels, AutoViz, and BayesNets. I also got through using PyCall, using PyPlot, using AutoEnvs and already had the trajectory data converted and the Julia tests ran without error (though a few warnings).

But, I'm getting some weird issues trying to run the Python tests (this is where I got stuck on mac btw). When I run python runtests.py I get this error which I cannot figure out. It happens from the call self.j = julia.Julia() inside ngsim_env/python/julia_env/julia_env.py. In fact, I get the same error if I just go into the ordinary python interpreter and execute import julia and then julia.Julia().

From the error it looks like maybe some path issue with the Python julia module trying to use the python inside the julia Conda package instead of the rllab3 python. I tried your path changes above and also tried going into the julia interpreter first and setting ENV["PYTHON"] = <path to my conda rllab3 python> (the same python I call the test with since I have the conda rllab3 environment activated when running the test).

Any thoughts?

Fatal Python error: initsite: Failed to import the site module
Traceback (most recent call last):
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/site.py", line 168, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/importlib/util.py", line 14, in <module>
    from contextlib import contextmanager
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/contextlib.py", line 5, in <module>
    from collections import deque
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/collections/__init__.py", line 24, in <module>
    import heapq as _heapq
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/heapq.py", line 587, in <module>
    from _heapq import *
SystemError: initialization of _heapq did not return an extension module

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/site.py", line 570, in <module>
    main()
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/site.py", line 557, in main
    known_paths = addsitepackages(known_paths)
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/site.py", line 349, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/site.py", line 207, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/site.py", line 178, in addpackage
    import traceback
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/traceback.py", line 3, in <module>
    import collections
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/collections/__init__.py", line 24, in <module>
    import heapq as _heapq
  File "/root/.julia/v0.6/Conda/deps/usr/lib/python3.7/heapq.py", line 587, in <module>
    from _heapq import *
SystemError: initialization of _heapq did not return an extension module

[sblinz-zz]

Turns out that error is solved by removing ~/.julia/lib/v0.6/PyCall.jl - missed that at first because it wasn't a segfault. Now all Python tests are passing except test_vectorized_ngsim_env because I haven't installed hgail yet.

[sblinz-zz]

I made it through to the end of install_env_gail_full.md!

Oddly, the hgail tests hgail/tests/runtests.py have different outcomes each time I run them. I had 1 failure when I ran it the 1st time but when I reran they all passed. I was curious so reran a few more times and it kept alternating passing and failing 1 test. The failed runs only reported 1 failed test but strangely it wasn't always the same test.

Here's one test that failed though the first run it was a different test that failed.

======================================================================
FAIL: test_hgail_two_round_stochastic_env (algos.test_hgail_impl.TestHGAIL)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hgail/tests/algos/test_hgail_impl.py", line 206, in test_hgail_two_round_stochastic_env
    np.testing.assert_array_almost_equal(l0_dist_2, [[1,0]], 1)
  File "/root/miniconda3/envs/rllab3/lib/python3.5/site-packages/numpy/testing/utils.py", line 962, in assert_array_almost_equal
    precision=decimal)
  File "/root/miniconda3/envs/rllab3/lib/python3.5/site-packages/numpy/testing/utils.py", line 778, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 1 decimals

(mismatch 100.0%)
 x: array([[ 0.8,  0.2]], dtype=float32)
 y: array([[1, 0]])

----------------------------------------------------------------------

[djp42]

AWESOME!! We also have made it to the end on our new install! That is interesting about the non-deterministic hgail tests. I will ask someone that is more familiar with those than I am. It might just be stochasticity, or some package having a different precision, etc. Flaky tests are generally bad anyway...

TL;DR, let me get back to you.

[sblinz-zz]

That's great. I'm getting a Dockerfile ready so hopefully will share that soon. I seem to be stuck on the very last part of the installation:

julia extract_ngsim_demonstrations.jl runs for a long time and keeps showing counts like frame **** / 9500.

But, it gets killed for some reason each time. For example it'll show frame 5737 / 9452Killed and return me back to my terminal prompt. It may be a Docker thing but I can't figure it out so far.

Since this step looks like it's supposed to create a file ngsim_env/data/trajectories/ngsim_all.h5 but that folder is still empty for me I'm guessing I can't go on to "Train and run a single agent GAIL policy".

Would it be enough if you just gave me a copy of ngsim_all.h5 or does this script do more than that?

I'm anxious to get on to training and running the gail algorithm.

[djp42]

That is interesting that it dies. It is fine if it takes a long time to run, that is expected.

I put the .tar.gz in this google drive folder. See if that helps, and let me know of any more problems.

[sblinz-zz]

Thanks, that's really helpful. Nice to have the julia packages as well. One question: trajectories folder has ngsim_all.h5 (14.19GB) and ngsim.h5 (1.15GB) but last line of instructions has cp ngsim_all.h5 ngsim.h5...are they supposed to be the same or different?

Update on Dockerfile progress: I was able to get julia packages working by manually cloning and checking out different versions through an interactive terminal to a running docker container. As I went I wrote out corresponding Dockerfile commands but when I tried to build that Dockerfile there were package conflicts. The order of julia package cloning and building is really sensitive. So, either I mixed up steps when writing the Dockerfile or the way Docker builds is a little different than manually calling the commands (I know there are some things around shells and sub-processes and layers when you execute RUN commands in Dockerfile, maybe related to that).

[djp42]

Regarding the ngsim_all vs ngsim, I believe the ngsim_all contains trajectories from different ngsim datasets that we do not really care about. When I was checking how they were used some time back, it seemed that there were limited differences. I suggest just using them as they appear, but if you are interested in testing the differences, copy the small one and rename it to ngsim_all. But we have done this before and saw no difference. The ngsim_all file is a relic of an older setup process.

Regarding Docker, I have never used it successfully, and I understand it has a very steep learning curve. Sorry I can't be more help there.

[sblinz-zz]

Great, thank you. I'm now running the example training python imitate.py --exp_name NGSIM-gail --n_itr 1000 --policy_recurrent True from the README. It's running inside a Docker container which is nice, averaging about 30s / iteration so looking like ~8 hours.

I'm planning to put together a repo that will have some of the data that takes a really long time to produce (NGSIM trajdata, ngsim_env trajectories) and a Dockerfile so that others can go straight to training and running policies. Will keep you posted.

[djp42]

Great, definitely keep me updated. I think I'll keep this issue open until I have time to go through and update the README with new instructions / link to files or the docker.

[sblinz-zz]

@djp42 I was able to successfully run python imitate.py --exp_name NGSIM-gail --n_itr 1000 --policy_recurrent True and ngsim_env/data/experiments/NGSIM-gail/imitate/log has the last 100 itr_***.npz files as well as args.npz and log.txt.

I was also able to generate a simulation video from this trained policy based on these instructions.

But, when I went to run python validate.py --n_proc 5 --exp_dir ../../data/experiments/NGSIM-gail/ --params_filename itr_1000.npz --random_seed 42 I get this odd error. It happens even if I include --debug true (which calls validate.py::single_process_collect_trajectories() instead of parallel_collect_trajectories()).

Any thoughts?

pid: 1 traj: 0 / 2062Traceback (most recent call last):
  File "validate.py", line 381, in <module>
    random_seed=run_args.random_seed
  File "validate.py", line 266, in collect
    random_seed=random_seed
  File "validate.py", line 226, in single_process_collect_trajectories
    random_seed
  File "validate.py", line 135, in collect_trajectories
    env_kwargs=dict(egoid=egoid, start=starts[egoid])
  File "validate.py", line 31, in simulate
    a, a_info = policy.get_action(x)
  File "/home/hgail/hgail/policies/gaussian_latent_var_gru_policy.py", line 193, in get_action
    return actions[0], {k: v[0] for k, v in agent_infos.items()}
  File "/home/hgail/hgail/policies/gaussian_latent_var_gru_policy.py", line 193, in <dictcomp>
    return actions[0], {k: v[0] for k, v in agent_infos.items()}
KeyError: 0

[binderwang]

I have tried various methods but still can't install ngsim_env, from Julia 0.6 to 1.0.3, is there any plan to update the install guidance?

[HanyangLiu]

I'm definitely encountering the same problem about the version conflict of Julia. Is the Dockerfiles for ngsim ready?

[raunakbh92]

Code is now working with julia 1.1. Has not been merged into master yet as some tests are pending. Please see this issue to make the code work with Julia1.1.