probcomp/developer

Developer environment for probcomp repos

developer's Issues

Add a section on trcrpm to the introduction tutorial notebook

A section on trcrpm similar to those that already exist for BayesDB, Crosscat, Venture, and Gen.jl should be added to /tutorials/introduction.ipynb.

Depends on #18.

Suggestions to improve step-by-step getting started guide for documentation

What we need is a step-by-step guide that holds the user's hand, assuming they are arriving with a completely clean slate: install docker/docker-compose, clone the Github repos, pull the notebook image from dockerhub, etc, to be written clearly and detailed systematically

A list of items to cover might be something like:

Step 1: Ask your system maintainer to install docker (>1.13) and docker-compose (>1.19).
Step 2: Ask your system maintainer to add you to the docker group.
Step 3: Run docker pull probcomp/notebook:latest to get the base image.
Step 4: Run git clone on bayeslite/cgpm/crosscat/iventure/developer ...
Step 5: Run cd developer, then run docker-compose up -p Step 6: Runmake bootstrap`
Step 7: ...

URL displayed when starting Docker image should lead to introduction.ipynb

Document the internal architecture of probcomp/developer

The internal architecture of the developer Docker image should be documented such that a new contributor with a basic familiarity with Docker and devops could come up to speed and become productive making contributions to it with minimal assistance.

Plotting Error with Matplotlib in notebooks

When I try to plot in matplotlib, I get the following. I have to update the matplotlib version and then restart the kernel to get around it.

AttributeErrorTraceback (most recent call last)
/opt/conda/envs/python2/lib/python2.7/site-packages/ipykernel/pylab/backend_inline.pyc in show(close, block)
37 display(
38 figure_manager.canvas.figure,
---> 39 metadata=_fetch_figure_metadata(figure_manager.canvas.figure)
40 )
41 finally:

/opt/conda/envs/python2/lib/python2.7/site-packages/ipykernel/pylab/backend_inline.pyc in _fetch_figure_metadata(fig)
172 """Get some metadata to help with displaying a figure."""
173 # determine if a background is needed for legibility
--> 174 if _is_transparent(fig.get_facecolor()):
175 # the background is transparent
176 ticksLight = _is_light([label.get_color()

/opt/conda/envs/python2/lib/python2.7/site-packages/ipykernel/pylab/backend_inline.pyc in _is_transparent(color)
193 def _is_transparent(color):
194 """Determine transparency from alpha."""
--> 195 rgba = colors.to_rgba(color)
196 return rgba[3] < .5

AttributeError: 'module' object has no attribute 'to_rgba'

tutorials/ directory needs to be in work/ since the former is not writeable

We need tutorials/ be in work/tutorials since the user joyvan does not have write permissions to the tutorials directory, leading the notebooks to fail to create .bdb files, etc.

probcomp-1:~% docker exec -it fsaad_notebook_1 /bin/bash
(python2) jovyan@610d8ffacd20:~$ cd tutorials/
(python2) jovyan@610d8ffacd20:~/tutorials$ ls
extrapolation-with-gp.ipynb  gapminder-missing-data.ipynb  goal-inference-part2.ipynb  hyperparameter-inference-with-gp.ipynb  population-assembly-tutorial.ipynb  satellites-predictive.ipynb
gapminder-exploratory.ipynb  goal-inference-part1.ipynb    goal-inference-part3.ipynb  introduction.ipynb                      resources                           test
(python2) jovyan@610d8ffacd20:~/tutorials$ touch foo
touch: cannot touch 'foo': Permission denied

Add robustness to developer docker image when multiple developers on the same machine/server

Dockerized development on a shared server, where multiple users have their own running containers against a shared `probcomp/notebook:latest' image, is likely to face issues with isolation.

Add some explicit guidelines to help ensure that their images/containers/ports remain isolated. Below are some failure modes I anticipate, and possible solutions:

1. The probcomp/notebook:latest dockerfile is not version controlled. If user X runs docker pull probcomp/notebook:latest, then this will also result in user Y obtaining the update, without them knowing.

• Solution 1: Tell developers to create docker tags to pin their images.
• Solution 2: Version the probcomp/notebook dockerfiles.

2. The docker containers live in a shared namespace. Therefore, if two users run docker-compose up separately, they may wipe one another's containers.

• Tell developers to use docker-compose -p <username> up.
• Update `docker-compose exec' command in bootstrap accordingly.

3. The docker-compose.yml hard-codes port 8888. Therefore, users will not be able to run containers simultaneously.

• Document to users that they may need to change 8888:8888 in `docker-compose.yml' to something else like 8889:8888 (until it works).

Verify all tutorial notebooks are in a working state

Starting with /tutorials/introduction.ipynb make sure that all the notebooks linked to or referred to by /tutorials/introduction.ipynb can be run without issues. For any that do have issues report them as bugs and send them to @vkmvkmvkmvkm for review.

Feature request: allow to add volumes to the set of mounted volumes in a file that is not tracked by git

Every collaborator I have worked with and I myself ended up editing the volumes section in the docker-compose.yml file to mount project directories.

Edits to docker-compose.yml are tracked by git. It would be great if we could read additional volumes that a user wants to mount from a separate file that is not tracked by git.

Review all the documention for the stack

Once #19 and #20 are complete, @vkmvkmvkmvkm should spend 3 hours revising the entire stack documentation, to match the high-level story we want to tell.

Add trcrpm as a top-level package to the stack

trcrpm is a new Python package that is under development by @fsaad. It should be included in the stack by default and should be usable from the Python notebooks that ship with it.

cannot run tests in local source directory due to non-standard directory structure

Because the various probcomp repos utilize a src/ directory instead of a <MODULE_NAME>/ directory, tests cannot be run from within the repo directories themselves. Adding a symlink (e.g. ln -s src crosscat) fixes this issue but this workaround may have other side-effects. It would be preferable to update the directory structures. cc: @fsaad @zane @Schaechtle

$ make crosscat-test
====================================================================== test session starts =======================================================================
platform linux2 -- Python 2.7.11, pytest-2.8.5, py-1.5.2, pluggy-0.3.1
rootdir: /home/jovyan/crosscat, inifile:
collected 106 items

. ..........................................................................................................

=================================================================== 106 passed in 6.99 seconds ===================================================================
$ make crosscat-dev


Package plan for package removal in environment /opt/conda/envs/python2:

The following packages will be REMOVED:

    crosscat: 2017.12.05-0 probcomp/label/edge

$ make crosscat-test
====================================================================== test session starts =======================================================================
platform linux2 -- Python 2.7.11, pytest-2.8.5, py-1.5.2, pluggy-0.3.1
rootdir: /home/jovyan/crosscat, inifile:

================================================================== no tests ran in 0.05 seconds ==================================================================
ERROR: file or package not found: crosscat
make: *** [crosscat-test] Error 4

add symlink:

 2018-02-19 23:55:35 ⌚  avinson-notebook in ~/crosscat
○ → ln -s src crosscat

now fixed:

 $ make crosscat-test
====================================================================== test session starts =======================================================================
platform linux2 -- Python 2.7.11, pytest-2.8.5, py-1.5.2, pluggy-0.3.1
rootdir: /home/jovyan/crosscat, inifile:
collected 106 items

crosscat/tests/unit_tests/test_col_ensure.py .....................
crosscat/tests/unit_tests/test_continuous_component_model.py .....................
crosscat/tests/unit_tests/test_general_utils.py ...
crosscat/tests/unit_tests/test_multinomial_component_model.py .......................
crosscat/tests/unit_tests/test_pred_prob.py ....
crosscat/tests/unit_tests/test_quality_test_utils.py ...............
crosscat/tests/unit_tests/test_synthetic_data_generator.py ...................

=================================================================== 106 passed in 8.32 seconds ===================================================================

Unable to make docker

I ran the following:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
fifizhang@Fionas-MacBook-Air ~/D/U/developer> make up
Starting fifizhang_notebook_1 ... done
Attaching to fifizhang_notebook_1
notebook_1  | Enabling develop on: bayeslite
notebook_1  |   Uninstalling bayeslite conda package
fifizhang_notebook_1 exited with code 1
fifizhang@Fionas-MacBook-Air ~/D/U/developer> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
probcomp/notebook   edge                27fd4c60b978        8 hours ago         7GB
fifizhang@Fionas-MacBook-Air ~/D/U/developer> git pull
remote: Enumerating objects: 2, done.
remote: Counting objects: 100% (2/2), done.
remote: Total 3 (delta 2), reused 2 (delta 2), pack-reused 1
Unpacking objects: 100% (3/3), done.
From https://github.com/probcomp/developer
   f9d98e2..a6ef99c  master     -> origin/master
Updating f9d98e2..a6ef99c
Fast-forward
 docker-compose.yml | 1 +
 1 file changed, 1 insertion(+)
Current branch master is up to date.
fifizhang@Fionas-MacBook-Air ~/D/U/developer> make down
Removing fifizhang_notebook_1 ... done
Removing network fifizhang_default
fifizhang@Fionas-MacBook-Air ~/D/U/developer> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
probcomp/notebook   edge                27fd4c60b978        8 hours ago         7GB
fifizhang@Fionas-MacBook-Air ~/D/U/developer> make up
Creating network "fifizhang_default" with the default driver
Creating fifizhang_notebook_1 ... done
Attaching to fifizhang_notebook_1
notebook_1  | Enabling develop on: bayeslite
notebook_1  |   Uninstalling bayeslite conda package
notebook_1  |   Creating bayeslite.egg-link
notebook_1  |   Building bayeslite
fifizhang_notebook_1 exited with code 2
fifizhang@Fionas-MacBook-Air ~/D/U/developer> make pull
Pulling notebook ... done
fifizhang@Fionas-MacBook-Air ~/D/U/developer> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
probcomp/notebook   edge                27fd4c60b978        8 hours ago         7GB
fifizhang@Fionas-MacBook-Air ~/D/U/developer> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
probcomp/notebook   edge                27fd4c60b978        9 hours ago         7GB
fifizhang@Fionas-MacBook-Air ~/D/U/developer> docker rmi 27fd4c60b978
Error response from daemon: conflict: unable to delete 27fd4c60b978 (must be forced) - image is being used by stopped container cdd23777479e
fifizhang@Fionas-MacBook-Air ~/D/U/developer> docker rmi -f 27fd4c60b978
Untagged: probcomp/notebook:edge
Untagged: probcomp/notebook@sha256:56c63c62c9f684c7a77b41b12d2f84ea37aca35870c78b3a4851296a589752da
Deleted: sha256:27fd4c60b9785b7cc0cca79df1080940b36ea3a6bdb5ce5b09d050de023a22fc
fifizhang@Fionas-MacBook-Air ~/D/U/developer> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
fifizhang@Fionas-MacBook-Air ~/D/U/developer> make up
Pulling notebook (probcomp/notebook:edge)...
edge: Pulling from probcomp/notebook
22dc81ace0ea: Already exists
1a8b3c87dba3: Already exists
91390a1c435a: Already exists
07844b14977e: Already exists
b78396653dae: Already exists
f6ed4d15e19b: Already exists
4b867dd0bba7: Already exists
2bc99ec6f992: Already exists
7037e70e2c06: Already exists
33568953b30a: Already exists
037d62f84cd0: Already exists
bdae9b6f5195: Already exists
54c52394a1ce: Already exists
3757aa1597fa: Already exists
d5e58821a64b: Already exists
7db0be35025a: Already exists
476d99934d31: Already exists
cd1b2a2f2e8d: Already exists
940b699cefc3: Already exists
e51e94aac825: Already exists
5da6d5d627a3: Already exists
8065e746338a: Already exists
d296f434cbea: Already exists
cf663e4b1db5: Already exists
dbdd546d03f8: Already exists
5e78f0d7b4fe: Already exists
dcd1f5746f10: Already exists
f71df5c1a0bb: Already exists
d99d0a25810a: Already exists
1e6c3305a960: Already exists
badfc5cc8995: Already exists
5809434196c0: Already exists
fe36bdef5ba7: Already exists
975bdaecf08d: Already exists
59dbbb6a0b60: Already exists
ba7c386b12db: Already exists
Digest: sha256:56c63c62c9f684c7a77b41b12d2f84ea37aca35870c78b3a4851296a589752da
Status: Downloaded newer image for probcomp/notebook:edge
Starting fifizhang_notebook_1 ... done
Attaching to fifizhang_notebook_1
notebook_1  | Enabling develop on: bayeslite
notebook_1  |   Uninstalling bayeslite conda package
fifizhang_notebook_1 exited with code 1
fifizhang@Fionas-MacBook-Air ~/D/U/developer> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
probcomp/notebook   edge                27fd4c60b978        9 hours ago         7GB
fifizhang@Fionas-MacBook-Air ~/D/U/developer> make down
Removing fifizhang_notebook_1 ... done
Removing network fifizhang_default
fifizhang@Fionas-MacBook-Air ~/D/U/developer> make up
Creating network "fifizhang_default" with the default driver
Creating fifizhang_notebook_1 ... done
Attaching to fifizhang_notebook_1
notebook_1  | Enabling develop on: bayeslite
notebook_1  |   Uninstalling bayeslite conda package
notebook_1  |   Creating bayeslite.egg-link
notebook_1  |   Building bayeslite
fifizhang_notebook_1 exited with code 2````

make bayeslite fails due to user permissions

probcomp-1:/tmp/developer% make bayeslite
running install
running bdist_egg
running egg_info
creating bayeslite.egg-info
error: could not create 'bayeslite.egg-info': Permission denied
Makefile:47: recipe for target 'bayeslite' failed
make: *** [bayeslite] Error 1