yelp/venv-update

Synchronize your virtualenv quickly and exactly.

venv-update

Quickly and exactly synchronize a large python project's virtualenv with its requirements.

Please see http://venv-update.readthedocs.org/en/master/ for the complete documentation.

How to Contribute

1. Fork this repository on github: https://help.github.com/articles/fork-a-repo/

2. Clone it: https://help.github.com/articles/cloning-a-repository/

3. Make a feature branch for your changes:

    git remote add upstream https://github.com/Yelp/venv-update.git
    git fetch upstream
    git checkout upstream/master -b my-feature-branch
   

4. Make sure the test suite works before you start:

    source .activate.sh
    make test
   

5. Commit patches: http://gitref.org/basic/

6. Push to github: git pull && git push origin

7. Send a pull request: https://help.github.com/articles/creating-a-pull-request/

Running tests:

Run a particular test:

py.test tests/functional/simple_test.py::test_downgrade

See all output from a test:

py.test -s -k downgrade

Check coverage of a single test:

./test tests/functional/simple_test.py::test_downgrade

Yelpers

To develop and run tests suites on a devbox, make sure to:

1. Python 3.6.0 on a xenial devbox breaks coverage. Use a bionic devbox instead.

2. Override pip.conf to use public pypi. Don't forget to delete it after you're done!

$ cat ~/.pip/pip.conf
[global]
index-url = https://pypi.org/simple/

3. sudo apt-get install pypy-dev so TOXENV=pypy doesn't fail spectacularly

venv-update's Issues

Switching drop-in package removes overlapping files

The reason for this is both packages provide the same file. We install first and then uninstall after, removing the shared files from the new package.

This should reproduce:

set -ex

function dotest() {
    local req=$1
    echo "$req" > reqs.txt
    python venv_update.py venv reqs.txt
    ./venv/bin/python -c 'import MySQLdb'
}

# Demonstrate it works by itself
rm -rf venv
dotest 'mysql-python'
rm -rf venv
dotest 'mysqlclient'
rm -rf venv

# Now break it
dotest 'mysql-python'
dotest 'mysqlclient'

$ bash test.sh
+ rm -rf venv
+ dotest mysql-python
+ local req=mysql-python
+ echo mysql-python
+ python venv_update.py venv reqs.txt
> /usr/bin/python -m virtualenv venv
New python executable in /home/asottile/workspace/venv-update/venv/bin/python
Installing setuptools, pip, wheel...done.
> pip --version
pip 7.0.3 from /usr/local/lib/python2.7/dist-packages (python 2.7)
> venv/bin/python -m pip.__main__ install 'pip>=1.5.0,<6.0.0'
You are using pip version 7.1.0, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
DEPRECATION: --download-cache has been deprecated and will be removed in the future. Pip now automatically uses and configures its cache.
Collecting pip<6.0.0,>=1.5.0
  Using cached pip-1.5.6-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 7.1.0
    Uninstalling pip-7.1.0:
      Successfully uninstalled pip-7.1.0
Successfully installed pip-1.5.6
> venv/bin/python venv_update.py --stage2 venv reqs.txt
> pip install --upgrade --use-wheel --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse wheel==0.24.0
Requirement already up-to-date: wheel==0.24.0 in ./venv/lib/python2.7/site-packages
Cleaning up...
> pip wheel --wheel-dir=/home/asottile/.pip/wheelhouse wheel==0.24.0 --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse --requirement=reqs.txt
Downloading/unpacking wheel==0.24.0
  File was already downloaded /home/asottile/.pip/wheelhouse/wheel-0.24.0-py2.py3-none-any.whl
Downloading/unpacking mysql-python (from -r reqs.txt (line 1))
  File was already downloaded /home/asottile/.pip/wheelhouse/MySQL_python-1.2.5-cp27-none-linux_x86_64.whl
Cleaning up...
> pip install --upgrade --use-wheel --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse --no-index --requirement=reqs.txt
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking mysql-python (from -r reqs.txt (line 1))
Installing collected packages: mysql-python
Successfully installed mysql-python
Cleaning up...
+ ./venv/bin/python -c 'import MySQLdb'
+ rm -rf venv
+ dotest mysqlclient
+ local req=mysqlclient
+ echo mysqlclient
+ python venv_update.py venv reqs.txt
> /usr/bin/python -m virtualenv venv
New python executable in /home/asottile/workspace/venv-update/venv/bin/python
Installing setuptools, pip, wheel...done.
> pip --version
pip 7.0.3 from /usr/local/lib/python2.7/dist-packages (python 2.7)
> venv/bin/python -m pip.__main__ install 'pip>=1.5.0,<6.0.0'
You are using pip version 7.1.0, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
DEPRECATION: --download-cache has been deprecated and will be removed in the future. Pip now automatically uses and configures its cache.
Collecting pip<6.0.0,>=1.5.0
  Using cached pip-1.5.6-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 7.1.0
    Uninstalling pip-7.1.0:
      Successfully uninstalled pip-7.1.0
Successfully installed pip-1.5.6
> venv/bin/python venv_update.py --stage2 venv reqs.txt
> pip install --upgrade --use-wheel --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse wheel==0.24.0
Requirement already up-to-date: wheel==0.24.0 in ./venv/lib/python2.7/site-packages
Cleaning up...
> pip wheel --wheel-dir=/home/asottile/.pip/wheelhouse wheel==0.24.0 --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse --requirement=reqs.txt
Downloading/unpacking wheel==0.24.0
  File was already downloaded /home/asottile/.pip/wheelhouse/wheel-0.24.0-py2.py3-none-any.whl
Downloading/unpacking mysqlclient (from -r reqs.txt (line 1))
  File was already downloaded /home/asottile/.pip/wheelhouse/mysqlclient-1.3.6-cp27-none-linux_x86_64.whl
Cleaning up...
> pip install --upgrade --use-wheel --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse --no-index --requirement=reqs.txt
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking mysqlclient (from -r reqs.txt (line 1))
Installing collected packages: mysqlclient
Successfully installed mysqlclient
Cleaning up...
+ ./venv/bin/python -c 'import MySQLdb'
+ rm -rf venv
+ dotest mysql-python
+ local req=mysql-python
+ echo mysql-python
+ python venv_update.py venv reqs.txt
> /usr/bin/python -m virtualenv venv
New python executable in /home/asottile/workspace/venv-update/venv/bin/python
Installing setuptools, pip, wheel...done.
> pip --version
pip 7.0.3 from /usr/local/lib/python2.7/dist-packages (python 2.7)
> venv/bin/python -m pip.__main__ install 'pip>=1.5.0,<6.0.0'
You are using pip version 7.1.0, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
DEPRECATION: --download-cache has been deprecated and will be removed in the future. Pip now automatically uses and configures its cache.
Collecting pip<6.0.0,>=1.5.0
  Using cached pip-1.5.6-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 7.1.0
    Uninstalling pip-7.1.0:
      Successfully uninstalled pip-7.1.0
Successfully installed pip-1.5.6
> venv/bin/python venv_update.py --stage2 venv reqs.txt
> pip install --upgrade --use-wheel --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse wheel==0.24.0
Requirement already up-to-date: wheel==0.24.0 in ./venv/lib/python2.7/site-packages
Cleaning up...
> pip wheel --wheel-dir=/home/asottile/.pip/wheelhouse wheel==0.24.0 --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse --requirement=reqs.txt
Downloading/unpacking wheel==0.24.0
  File was already downloaded /home/asottile/.pip/wheelhouse/wheel-0.24.0-py2.py3-none-any.whl
Downloading/unpacking mysql-python (from -r reqs.txt (line 1))
  File was already downloaded /home/asottile/.pip/wheelhouse/MySQL_python-1.2.5-cp27-none-linux_x86_64.whl
Cleaning up...
> pip install --upgrade --use-wheel --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse --no-index --requirement=reqs.txt
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking mysql-python (from -r reqs.txt (line 1))
Installing collected packages: mysql-python
Successfully installed mysql-python
Cleaning up...
+ ./venv/bin/python -c 'import MySQLdb'
+ dotest mysqlclient
+ local req=mysqlclient
+ echo mysqlclient
+ python venv_update.py venv reqs.txt
Keeping virtualenv from previous run.
> pip --version
pip 7.0.3 from /usr/local/lib/python2.7/dist-packages (python 2.7)
> venv/bin/python -m pip.__main__ install 'pip>=1.5.0,<6.0.0'
Requirement already satisfied (use --upgrade to upgrade): pip>=1.5.0,<6.0.0 in ./venv/lib/python2.7/site-packages
Cleaning up...
> venv/bin/python venv_update.py --stage2 venv reqs.txt
> pip install --upgrade --use-wheel --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse wheel==0.24.0
Requirement already up-to-date: wheel==0.24.0 in ./venv/lib/python2.7/site-packages
Cleaning up...
> pip wheel --wheel-dir=/home/asottile/.pip/wheelhouse wheel==0.24.0 --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse --requirement=reqs.txt
Downloading/unpacking wheel==0.24.0
  File was already downloaded /home/asottile/.pip/wheelhouse/wheel-0.24.0-py2.py3-none-any.whl
Downloading/unpacking mysqlclient (from -r reqs.txt (line 1))
  File was already downloaded /home/asottile/.pip/wheelhouse/mysqlclient-1.3.6-cp27-none-linux_x86_64.whl
Cleaning up...
> pip install --upgrade --use-wheel --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse --no-index --requirement=reqs.txt
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking mysqlclient (from -r reqs.txt (line 1))
Installing collected packages: mysqlclient
Successfully installed mysqlclient
Cleaning up...
> pip uninstall --yes mysql-python
Uninstalling MySQL-python:
  Successfully uninstalled MySQL-python
+ ./venv/bin/python -c 'import MySQLdb'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named MySQLdb

Can we use the same wheel store as pip?

pip >= 7 builds and caches wheels, similar to what we do. Can we store wheels we build in the same location to share the work?

On my machine it prints lines like:

Running setup.py bdist_wheel for numpy
Stored in directory: /home/c/ck/ckuehl/.cache/pip/wheels/b3/d0/9c/c860a697e3ea6929e396a5c55fc5aa2ed7413d24332a9c5299

for reference, here's what that directory looks like: https://gist.github.com/anonymous/9e746e749735f7d49d76

update docs s/venv-update/pip-faster/ as appropriate

Docs and help texts, mainly.
The most important instances in setup.py.

Convert virtualenv to no-site-packages while doing intelligent edit-distance upgrade

I think the text file is the only difference between the two. With this you could factor out the --ignore-installed wackiness

support older and newer versions of virtualenv/pip

This doesn't work with new versions of pip or virtualenv. I got it to work by

• Making a bootstrap virtualenv, then installing virtualenv 1.11.6 in that, and
• Adding pip<2 to the list of dependencies in the beginning of venv_update.py

It also didn't work with the super-old virtualenv on my system (1.9.1), or the pip version that it installed (1.4.1).

Absolute venv-update symlink makes it hard to mount ~/.cache/venv-update into Docker containers

If you have multiple processes using venv-update at the same time (e.g. Jenkins) the symlink will flip frequently:

ckuehl@jenkins10-uswest2atestopia:/ephemeral/jenkins_slave/workspace/yelp-main-minified-compile-without-caches$ ls -l /nail/home/jenkins/.cache/venv-update/1.0rc4/
drwxr-xr-x 5 jenkins jenkins 4096 2016-01-14 13:34 venv/
lrwxrwxrwx 1 jenkins jenkins   92 2016-01-14 13:34 venv-update -> /nail/home/jenkins/.cache/venv-update/1.0rc4/venv/lib/python2.6/site-packages/venv_update.py
ckuehl@jenkins10-uswest2atestopia:/ephemeral/jenkins_slave/workspace/yelp-main-minified-compile-without-caches$ ls -la /nail/home/jenkins/.cache/venv-update/1.0rc4/
drwxr-xr-x 5 jenkins jenkins 4096 2016-01-14 13:38 venv/
lrwxrwxrwx 1 jenkins jenkins   91 2016-01-14 13:38 venv-update -> /nail/home/ymuser/.cache/venv-update/1.0rc4/venv/lib/python2.6/site-packages/venv_update.py

(the first time, it's a path on the local Jenkins slave; the second time, it's a path inside a Docker container running on the slave)

This isn't safe. @asottile suggests we can probably fix it by using a relative symlink.

Wheels should not be cached if requirement is editable

add a --conflict={ignore|warn|error|fail} option

Any large python application can very easily have irreconcilable dependencies, especially if one of the downstream dependencies puts a very tight bound (or pins exactly to deal with e.g. 0.X packages) on their dependencies. Normal pip will happily ignore these conflicts, and while I understand that in general the default of "ignore bad things" is a bad default and I agree the default should be "fail if bad things happen", it would be really practical to be able to ignore version conflicts that the author thinks are ok.

Presumably this would either be a global option of "ignore all conflicts" or the ability to ignore conflicts on certain packages. I slightly prefer the second, what do you guys think?

pip-faster should not be vendored with venv-update

venv-update finds and installs pip-faster from pypi, like other packages

coverage is flaky

all tests passed but we only got 89% coverage (python2.7 job)
https://travis-ci.org/Yelp/pip-faster/jobs/101654030

sometimes we get 91% coverage and sometimes 93% coverage, e.g.

93%: https://travis-ci.org/Yelp/pip-faster/jobs/101654026
91%: https://travis-ci.org/Yelp/pip-faster/jobs/101654031

no-input mode or "assume yes mode" required for some pip commands

this showed up in my ci logs:

Obtaining gutter from git+https://github.com/disqus/gutter.git@eb8285ea2fa072d11699fac6c55a202591e14b6d#egg=gutter (from -r requirements/global.txt (line 9))
  git clone in ./env/src/gutter exists with URL https://github.com/dpetzold/gutter.git
  The plan is to install the git repository https://github.com/disqus/gutter.git
What to do?  (s)witch, (i)gnore, (w)ipe, (b)ackup Cleaning up…

This command needs to be run with some way to force wipe or something for headless servers.

make venv_update updatable

My leading idea is "why not use pip?". It mentions in the readme that this was not done intentionally, could someone add more info as to why?

The setup.py supports entry_points so that says to me the project is intended to be installed, why not make it network updatable too?

OSX support

I currently don't have access to python 3.4 on my OSX box, but here is the repro with python 3.5:

http://paste.pound-python.org/show/ypTUsScHfGagXeZ2NM9f/

change host-python requirement to simply virtualenv, any version

venv-update will install wheels that don't match the python version

For instance this satisfied pyscss under python2.7:

{'file_tags': set([(u'cp26', u'none', u'linux_x86_64')]), 'name': u'pyScss', 'abis': [u'none'], 'pyversions': [u'cp26'], 'filename': u'pyScss-1.2.0.post3-cp26-none-linux_x86_64.whl', 'version': u'1.2.0.post3', 'plats': [u'linux_x86_64']}

I'd test it like this:

1. Make a cp26 wheel under ~/.wheelhouse
2. Run venv update -ppython2.7 with that as a requirement
3. Validate that the wheel being built is in the output
4. Make sure the wheel ends up in ~/.wheelhouse after running

Calling convention for venv-update is not reflected in help

venv-update: remove requirement on virtualenv in $PATH

See: http://eli.thegreenplace.net/2013/04/20/bootstrapping-virtualenv

import sys
import subprocess

VENV_VERSION = '1.9.1'
PYPI_VENV_BASE = 'http://pypi.python.org/packages/source/v/virtualenv'
PYTHON = 'python2'
INITIAL_ENV = 'py-env0'

def shellcmd(cmd, echo=True):
    """ Run 'cmd' in the shell and return its standard out.
    """
    if echo: print '[cmd] {0}'.format(cmd)
    out = subprocess.check_output(cmd, stderr=sys.stderr, shell=True)
    if echo: print out
    return out

dirname = 'virtualenv-' + VENV_VERSION
tgz_file = dirname + '.tar.gz'

# Fetch virtualenv from PyPI
venv_url = PYPI_VENV_BASE + '/' + tgz_file
shellcmd('curl -O {0}'.format(venv_url))

# Untar
shellcmd('tar xzf {0}'.format(tgz_file))

# Create the initial env
shellcmd('{0} {1}/virtualenv.py {2}'.format(PYTHON, dirname, INITIAL_ENV))

# Install the virtualenv package itself into the initial env
shellcmd('{0}/bin/pip install {1}'.format(INITIAL_ENV, tgz_file))

# Cleanup
shellcmd('rm -rf {0} {1}'.format(dirname, tgz_file))

can't install "-pre-1" packages

This might be a py35 issue. The crucial bit is UserWarning: Normalizing '1.4.0-pre-1' to '1.4.0rc1'.

$ cat requirements.txt
docker-py==1.4.0-pre-1

$ PIP_INDEX_URL=https://pypi.yelpcorp.com/simple python ../venv_update.py
Keeping virtualenv from previous run.
> pip --version
pip 7.0.1 from /home/buck/venv/mypy/lib/python3.5/site-packages (python 3.5)
> virtualenv_run/bin/python -m pip.__main__ install 'pip>=1.5.0,<6.0.0'
Requirement already satisfied (use --upgrade to upgrade): pip>=1.5.0,<6.0.0 in ./virtualenv_run/lib/python3.5/site-packages
Cleaning up...
> virtualenv_run/bin/python ../venv_update.py --stage2 virtualenv_run requirements.txt
> pip install --upgrade --use-wheel --download-cache=/home/buck/.pip/cache --find-links=file:///home/buck/.pip/wheelhouse wheel==0.24.0
Requirement already up-to-date: wheel==0.24.0 in ./virtualenv_run/lib/python3.5/site-packages
Cleaning up...
> pip wheel --wheel-dir=/home/buck/.pip/wheelhouse wheel==0.24.0 --download-cache=/home/buck/.pip/cache --find-links=file:///home/buck/.pip/wheelhouse --requirement=requirements.txt
Downloading/unpacking wheel==0.24.0
  File was already downloaded /home/buck/.pip/wheelhouse/wheel-0.24.0-py2.py3-none-any.whl
Downloading/unpacking docker-py==1.4.0-pre-1 (from -r requirements.txt (line 1))
  Using download cache from /home/buck/.pip/cache/https%3A%2F%2Fpypi.yelpcorp.com%2Fpackages%2Fdocker-py-1.4.0-pre-1.tar.gz
  Running setup.py (path:/home/buck/trees/yelp/venv-update/tmp/virtualenv_run/build/docker-py/setup.py) egg_info for package docker-py
    /home/buck/trees/yelp/venv-update/tmp/virtualenv_run/lib/python3.5/site-packages/setuptools/dist.py:285: UserWarning: Normalizing '1.4.0-pre-1' to '1.4.0rc1'
      normalized_version,

    warning: no files found matching 'requirements3.txt'
  Requested docker-py==1.4.0-pre-1 (from -r requirements.txt (line 1)), but installing version 1.4.0rc1
Downloading/unpacking requests>=2.5.2 (from docker-py==1.4.0-pre-1->-r requirements.txt (line 1))
  Downloading requests-2.7.0-py2.py3-none-any.whl (470kB): 470kB downloaded
  Saved /home/buck/.pip/wheelhouse/requests-2.7.0-py2.py3-none-any.whl
  Storing download in cache at /home/buck/.pip/cache/https%3A%2F%2Fpypi.yelpcorp.com%2Fpackages%2Frequests-2.7.0-py2.py3-none-any.whl
Downloading/unpacking six>=1.3.0 (from docker-py==1.4.0-pre-1->-r requirements.txt (line 1))
  File was already downloaded /home/buck/.pip/wheelhouse/six-1.9.0-py2.py3-none-any.whl
Downloading/unpacking websocket-client>=0.32.0 (from docker-py==1.4.0-pre-1->-r requirements.txt (line 1))
  Downloading websocket_client-0.32.0-py3-none-any.whl (196kB): 196kB downloaded
  Saved /home/buck/.pip/wheelhouse/websocket_client-0.32.0-py3-none-any.whl
  Storing download in cache at /home/buck/.pip/cache/https%3A%2F%2Fpypi.yelpcorp.com%2Fpackages%2Fwebsocket_client-0.32.0-py3-none-any.whl
Building wheels for collected packages: docker-py
  Running setup.py bdist_wheel for docker-py
  Destination directory: /home/buck/.pip/wheelhouse
Successfully built docker-py
Cleaning up...
> pip install --upgrade --use-wheel --download-cache=/home/buck/.pip/cache --find-links=file:///home/buck/.pip/wheelhouse --no-index --requirement=requirements.txt
Ignoring indexes: https://pypi.yelpcorp.com/simple
Downloading/unpacking docker-py==1.4.0-pre-1 (from -r requirements.txt (line 1))
  Could not find a version that satisfies the requirement docker-py==1.4.0-pre-1 (from -r requirements.txt (line 1)) (from versions: 1.4.0rc1)
Cleaning up...
No distributions matching the version for docker-py==1.4.0-pre-1 (from -r requirements.txt (line 1))
Storing debug log for failure in /home/buck/.pip/pip.log

Something went wrong! Sending 'virtualenv_run' back in time, so make knows it's invalid.
Waiting for all subprocesses to finish...
DONE
> touch virtualenv_run --reference requirements.txt --date '1 day ago'

why not use optparse or argparse instead of the custom parseargs?

If there is no compelling reason I'll refactor parseargs away in favor of a builtin solution.

pytnon2.6 was mentioned so probably optparse.

Installing into virtualenv which doesn't have virtualenv while it is sourced causes trace

$ cat test.sh
rm -rf venv
virtualenv venv
. venv/bin/activate
echo mccabe > reqs.txt
curl https://raw.githubusercontent.com/Yelp/venv-update/master/venv_update.py > venv_update.py
python venv_update.py venv reqs.txt
$ bash !$
bash test.sh
New python executable in venv/bin/python
Installing setuptools, pip...done.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 17803  100 17803    0     0   114k      0 --:--:-- --:--:-- --:--:--  241k

Something went wrong! Sending 'venv' back in time, so make knows it's invalid.
Waiting for all subprocesses to finish...
DONE
> touch venv --reference reqs.txt --date '1 day ago'

Traceback (most recent call last):
  File "venv_update.py", line 584, in <module>
    exit(main())
  File "venv_update.py", line 566, in main
    return venv_update(stage, venv_path, reqs, venv_args)
  File "venv_update.py", line 551, in venv_update
    validate_venv(venv_path, venv_args)
  File "venv_update.py", line 353, in validate_venv
    from virtualenv import __version__ as virtualenv_version
ImportError: No module named virtualenv

Sphinx causes an infinite loop in trace_requirements

requirements.txt

sphinx

$ python venv_update.py test_venv_run requirements.txt

will hang in the while loop in trace_requirements as it seems sphinx has a requirement loop.

specifically because sphinx depends on sphinx-rtd-them and vice versa.

sphinx-rtd-theme>=0.1,<0.2 (from sphinx (from -r requirements.txt (line 1)))
sphinx>=1.1 (from sphinx-rtd-theme>=0.1,<0.2 (from sphinx (from -r requirements.txt (line 1))))

test suite flakes out with pytest-xdist INTERNALERROR too much

Example:

INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 413, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 424, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 315, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "<remote exec>", line 56, in pytest_runtestloop
INTERNALERROR>   File "<remote exec>", line 72, in run_tests
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 413, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 424, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 315, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/pytest_timeout.py", line 70, in pytest_runtest_protocol
INTERNALERROR>     timeout_setup(item)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/pytest_timeout.py", line 95, in timeout_setup
INTERNALERROR>     signal.signal(signal.SIGALRM, handler)
INTERNALERROR> ValueError: signal only works in main thread
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 413, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 424, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 315, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/xdist/dsession.py", line 504, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/xdist/dsession.py", line 522, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/xdist/dsession.py", line 560, in slave_slavefinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('../../../tests/testing/capture_subprocess_test.py::test_capture_subprocess', <SlaveController gw5>)
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 413, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 424, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 315, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/execnet/gateway_base.py", line 701, in receive
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR> EOFError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 413, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 424, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/core.py", line 315, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/execnet/gateway_base.py", line 701, in receive
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR> EOFError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)```

messaging is too loud on wheel cache miss

If we're able to find a wheel from the index server, we shouldn't output "SLOW!! no wheel found for pinned requirement". A wheel was found just not locally.

venv_update doesn't notice the venv has moved

You asked me to open this issue as a bug. Here's a reproduction:

#!/bin/bash
set -eu                                                                                                                                                                                                                                                                           

mkdir example                                                                                                                                                                                                                                                                     
pushd example                                                                                                                                                                                                                                                                     

echo "flake8==2.4.0" > requirements.txt                                                                                                                                                                                                                                           
wget https://raw.githubusercontent.com/Yelp/venv-update/master/venv_update.py                                                                                                                                                                                                     
python venv_update.py                                                                                                                                                                                                                                                             

popd                                                                                                                                                                                                                                                                              
mv example example2                                                                                                                                                                                                                                                               

pushd example2                                                                                                                                                                                                                                                                    
python venv_update.py                                                                                                                                                                                                                                                             
virtualenv_run/bin/flake8    # This fails                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
popd

The failure from flake8:

repro.sh: virtualenv_run/bin/flake8: /path/example/virtualenv_run/bin/python: bad interpreter: No such file or directory

After the virtualenv is moved, venv_update runs, but isn't aware of the change, so it reports "Keeping virtualenv from previous run." instead of recreating it.

Stage 1 python shouldn't be part of venv-update's invalidation state

We really only care about the stage2 python:

$ git diff yelp/vendor/
diff --git a/yelp/vendor/venv_update.py b/yelp/vendor/venv_update.py
index a720f2b..1df53f0 100644
--- a/yelp/vendor/venv_update.py
+++ b/yelp/vendor/venv_update.py
@@ -373,10 +373,17 @@ def validate_venv(venv_path, venv_args):
         except IOError:
             previous_state = {}

+
         if previous_state.get('validation') == validation:
             info('Keeping virtualenv from previous run.')
             return
         else:
+            import pprint
+            print(' OLD '.center(79, '*'))
+            pprint.pprint(previous_state.get('validation'))
+            print(' NEW '.center(79, '*'))
+            pprint.pprint(validation)
+            print('*' * 79)
             info('Removing invalidated virtualenv.')
             run(('rm', '-rf', venv_path))

$ make virtualenv_run
time ./bin/venv-update "virtualenv_run" requirements.txt requirements-dev.txt
************************************* OLD *************************************
[u'2.7.6 (python2.7_2.7.6-2+lucid1:0e6ac340ab29+, Jul 27 2015, 20:36:29) \n[GCC 4.4.3]',
 u'1.11.5',
 [u'--python=/usr/bin/python2.7'],
 u'/nail/home/asottile/pg/yelp-main/virtualenv_run']
************************************* NEW *************************************
[u'2.6.7 (r267:88850, Dec  2 2011, 20:27:26) \n[GCC 4.4.3]',
 u'1.11.5',
 [u'--python=/usr/bin/python2.7'],
 u'/nail/home/asottile/pg/yelp-main/virtualenv_run']
*******************************************************************************
Removing invalidated virtualenv.
> rm -rf virtualenv_run
...

erronious files touched

run(('touch', venv_path, '--reference', reqs[0], '--date', '1 day ago')) creates a --date, --reference, and 1 day ago file in my path.

`run(('rm', '-rf', venv_path))` is worrisome

run(('rm', '-rf', venv_path))

I was considering putting some checks around this line to ensure that we don't delete something important, like /, or ., or ...

thoughts?

does not install packages marked with -e editable

installs: git+ssh://whatever#egg=whatever
doesn't install: -e git+ssh://whatever#egg=whatever

spurious version conflict (?)

demo:

$ cat req.txt
-e [email protected]:Yelp/yelp_uri.git@66645c234c11b28457d24fecf38ac511fdf9267c#egg=yelp_uri
# yelp-uri<1.1.0


$ pip-faster install --upgrade --prune -r req.txt
Error when trying to get requirement for VCS system Command /usr/bin/git config remote.origin.url failed with error code 1 in /nail/home/buck/trees/yelp/pip-faster, falling back to uneditable format
Could not determine repository location of /nail/home/buck/trees/yelp/pip-faster
Obtaining yelp-uri from [email protected]:Yelp/yelp_uri.git@66645c234c11b28457d24fecf38ac511fdf9267c#egg=yelp_uri (from -r req.txt (line 1))
  Updating ./tmpenv/src/yelp-uri clone (to 66645c234c11b28457d24fecf38ac511fdf9267c)
  Could not find a tag or branch '66645c234c11b28457d24fecf38ac511fdf9267c', assuming commit.
  Running setup.py (path:/nail/home/buck/trees/yelp/pip-faster/tmp/tmpenv/src/yelp-uri/setup.py) egg_info for package yelp-uri

  Installing extra requirements: 'egg'
Downloading/unpacking six (from yelp-uri->-r req.txt (line 1))
  File was already downloaded /nail/home/buck/.pip/wheelhouse/six-1.10.0-py2.py3-none-any.whl
Requirement already up-to-date: yelp-encodings in ./tmpenv/lib/python2.7/site-packages (from yelp-uri->-r req.txt (line 1))
Requirement already up-to-date: yelp-bytes in ./tmpenv/lib/python2.7/site-packages (from yelp-uri->-r req.txt (line 1))
Building wheels for collected packages: yelp-uri
  Running setup.py bdist_wheel for yelp-uri
  Destination directory: /nail/home/buck/.pip/wheelhouse
Successfully built yelp-uri
Installing collected packages: yelp-uri, six
  Running setup.py develop for yelp-uri

    Creating /nail/home/buck/trees/yelp/pip-faster/tmp/tmpenv/lib/python2.7/site-packages/yelp-uri.egg-link (link to .)
    Adding yelp-uri 1.1.0 to easy-install.pth file

    Installed /nail/home/buck/trees/yelp/pip-faster/tmp/tmpenv/src/yelp-uri
Successfully installed yelp-uri six
Cleaning up...


$ cat req.txt
# -e [email protected]:Yelp/yelp_uri.git@66645c234c11b28457d24fecf38ac511fdf9267c#egg=yelp_uri
yelp-uri<1.1.0


$ pip-faster install --upgrade --prune -r req.txt
Error when trying to get requirement for VCS system Command /usr/bin/git config remote.origin.url failed with error code 1 in /nail/home/buck/trees/yelp/pip-faster, falling back to uneditable format
Could not determine repository location of /nail/home/buck/trees/yelp/pip-faster
Downloading/unpacking yelp-uri<1.1.0 (from -r req.txt (line 2))
  File was already downloaded /nail/home/buck/.pip/wheelhouse/yelp_uri-1.0.1-py2-none-any.whl
Requirement already up-to-date: yelp-bytes in ./tmpenv/lib/python2.7/site-packages (from yelp-uri<1.1.0->-r req.txt (line 2))
Requirement already up-to-date: yelp-encodings in ./tmpenv/lib/python2.7/site-packages (from yelp-uri<1.1.0->-r req.txt (line 2))
Installing collected packages: yelp-uri
  Found existing installation: yelp-uri 1.1.0
    Uninstalling yelp-uri:
      Successfully uninstalled yelp-uri
Successfully installed yelp-uri
Cleaning up...
Error: version conflict: yelp-uri 1.1.0 (tmpenv/src/yelp-uri) <-> yelp-uri<1.1.0 from file:///nail/home/buck/.pip/wheelhouse/yelp_uri-1.0.1-py2-none-any.whl (from -r req.txt (line 2))
Storing debug log for failure in /nail/home/buck/.pip/pip.log

-e <pkg> fails to install sub-requirements

cat requirements.txt
-e ../foo

venv-update
...
can't find any downloads that satisfy <thing foo requires>

Use system cache directories for cache and wheelhouse

There is a standard under Linux for where to store per user caches of data http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html. It would be good if venv-update used these directories instead of ~/.pip

There are similar standards also for Mac OS X which would also be good to use.

Formatting typo in README.md

Looks like:

pip-faster [![Build
Status](https://travis-ci.org/Yelp/pip-faster.svg?branch=master)](https://travis-ci.org/Yelp/pip-faster) Coverage
Status

to me. Looks like that's supposed to be a hyperlink.

Make pip_faster arguments match regular pip

venv_update doesn't work with Debian pip

ckuehl@raptors:~$ docker run -ti debian:jessie bash
root@d728bcf544e2:/# apt-get update && \
    apt-get -y install python python-virtualenv python-pip wget
[snip]
root@d728bcf544e2:/# wget https://raw.githubusercontent.com/Yelp/pip-faster/master/venv_update.py
[snip]
root@d728bcf544e2:/# echo mock > requirements.txt
root@d728bcf544e2:/# python venv_update.py 
> usr/bin/python -m virtualenv virtualenv_run
Running virtualenv with interpreter /usr/bin/python2
New python executable in /virtualenv_run/bin/python2
Also creating executable in /virtualenv_run/bin/python
Installing setuptools, pip...done.
> pip --version
pip 1.5.6 from /usr/lib/python2.7/dist-packages (python 2.7)
> virtualenv_run/bin/python -m pip.__main__ install 'pip>=1.5.0,<6.0.0'
Requirement already satisfied (use --upgrade to upgrade): pip>=1.5.0,<6.0.0 in /virtualenv_run/li
ges
Cleaning up...
> virtualenv_run/bin/python venv_update.py --stage2 virtualenv_run requirements.txt
Traceback (most recent call last):
  File "venv_update.py", line 595, in <module>
    exit(main())
  File "venv_update.py", line 577, in main
    return venv_update(stage, venv_path, reqs, venv_args)
  File "venv_update.py", line 565, in venv_update
    return stage2(venv_path, reqs)
  File "venv_update.py", line 555, in stage2
    return do_install(reqs)
  File "venv_update.py", line 408, in do_install
    previously_installed = pip_get_installed()
  File "venv_update.py", line 218, in pip_get_installed
    for dist in fresh_working_set()
  File "venv_update.py", line 280, in fresh_working_set
    from pip._vendor import pkg_resources
ImportError: No module named _vendor

I suspect it's because Debian devendorizes pip to comply with Policy.

This seems to fix it:

--- venv_update.py.old      2015-09-19 20:59:52.214736883 +0000
+++ venv_update.py   2015-09-19 21:00:59.326736246 +0000
@@ -277,7 +277,11 @@

 def fresh_working_set():
     """return a pkg_resources "working set", representing the *currently* installed pacakges"""
-    from pip._vendor import pkg_resources
+    try:
+        from pip._vendor import pkg_resources
+    except ImportError:
+        # Debian de-vendorizes vendored pip dependencies
+        import pkg_resources

     class WorkingSetPlusEditableInstalls(pkg_resources.WorkingSet):

@@ -298,7 +302,11 @@
     from collections import deque
     from pip import logger
     from pip.req import InstallRequirement
-    from pip._vendor import pkg_resources
+    try:
+        from pip._vendor import pkg_resources
+    except ImportError:
+        # Debian de-vendorizes vendored pip dependencies
+        import pkg_resources

     working_set = fresh_working_set()

venv_update fails if pip._vendor is not available (i.e. minimum pip version requirement)

I'm seeing if there is a reasonable way to add in stage1 something that will detect the pip version and update it if necessary.

Does anyone know the minimum version required?

double requirement: argparse==1.2.1

argparse is required in my requirements.txt, and pip wheel gets mad and stops.

Double requirement given: argparse==1.2.1 (from -r requirements.txt (line 5)) (already in argparse==1.2.1, name='argparse')

Of course, I can take it out for the moment

A warning should be printed when installing an unpinned package

This is the slowest of all but we don't print any warnings about it.

"remove from the cache and wheelhouse that have not been accessed in a week" destroys wheels we still want

I think the intent of this change was to remove old packages which are no longer needed (e.g. packages which are updated every day).

However, it's making some of our runs much slower. If I build a virtualenv with venv_update, then run venv_update again a week later, it will be a no-op, but wipe out my wheel cache since I haven't used those wheels (since the packages were already installed). Then if I run venv_update again, it will rebuild all the wheels.

ckuehl@supernova:~/ocf-proj/ocfweb$ cd $(mktemp -d)
ckuehl@supernova:/tmp/tmp.T2CbjVYkZK$ echo ocflib > requirements.txt
ckuehl@supernova:/tmp/tmp.T2CbjVYkZK$ cp ~/proj/pip-faster/{pip_faster.py,venv_update.py} .
ckuehl@supernova:/tmp/tmp.T2CbjVYkZK$ python3.4 venv_update.py -ppython3.4
> /usr/bin/python3.4 -m virtualenv virtualenv_run -ppython3.4
[...]
Building wheels for collected packages: ocflib,colorama,cracklib,dnspython3,pexpect,pycrypto,pyyaml,redis,sqlalchemy,ply
  Running setup.py bdist_wheel for ocflib
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for colorama
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for cracklib
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for dnspython3
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for pexpect
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for pycrypto
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for pyyaml
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for redis
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for sqlalchemy
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for ply
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
[...]
Successfully installed ocflib pysnmp requests dnspython3 pyyaml cracklib redis colorama pycrypto pexpect ldap3 paramiko sqlalchemy pymysql pyasn1 pysmi ecdsa ply
ckuehl@supernova:/tmp/tmp.T2CbjVYkZK$ du -hs ~/.pip/wheelhouse
3.7M    /home/c/ck/ckuehl/.pip/wheelhouse
ckuehl@supernova:/tmp/tmp.T2CbjVYkZK$ find ~/.pip/wheelhouse -type f -exec touch -a --date='10 days ago' {} \;
ckuehl@supernova:/tmp/tmp.T2CbjVYkZK$ python3.4 venv_update.py -ppython3.4
Keeping virtualenv from previous run.
[...]
Requirement already up-to-date: ocflib in ./virtualenv_run/lib/python3.4/site-packages (from -r requirements.txt (line 1))
Cleaning up...
> find /home/c/ck/ckuehl/.pip/cache /home/c/ck/ckuehl/.pip/wheelhouse -not -atime -7 -delete
ckuehl@supernova:/tmp/tmp.T2CbjVYkZK$ du -hs ~/.pip/wheelhouse
4.0K    /home/c/ck/ckuehl/.pip/wheelhouse
ckuehl@supernova:/tmp/tmp.T2CbjVYkZK$ python3.4 venv_update.py -ppython3.4
> /usr/bin/python3.4 -m virtualenv virtualenv_run -ppython3.4
[...]
Building wheels for collected packages: ocflib,colorama,cracklib,dnspython3,pexpect,pycrypto,pyyaml,redis,sqlalchemy,ply
  Running setup.py bdist_wheel for ocflib
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for colorama
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for cracklib
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for dnspython3
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for pexpect
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for pycrypto
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for pyyaml
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for redis
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for sqlalchemy
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
  Running setup.py bdist_wheel for ply
  Destination directory: /home/c/ck/ckuehl/.pip/wheelhouse
[...]
Successfully installed ocflib pysnmp requests dnspython3 pyyaml cracklib redis colorama pycrypto pexpect ldap3 paramiko sqlalchemy pymysql pyasn1 pysmi ecdsa ply
ckuehl@supernova:/tmp/tmp.T2CbjVYkZK$ du -hs ~/.pip/wheelhouse
3.7M    /home/c/ck/ckuehl/.pip/wheelhouse

proper test fixtures

Currently the test suite uses real, live packages from real, live pypi.python.org.
Let's stop doing this.

Most of the work is complete at https://github.com/Yelp/pip-faster/commits/add_dummy_packages

This issue represents the work of finalizing and merging that branch.

If `venv_update` symlink in `~/.cache` is wrong, we'll never fix it

The symlink is wrong:

/nail/home/ymuser/.cache/venv-update/1.0rc3:
total 16
drwxr-xr-x 3 ymuser ymuser 4096 2016-01-12 23:39 .
drwxr-xr-x 3 ymuser ymuser 4096 2016-01-12 23:39 ..
drwxr-xr-x 5 ymuser ymuser 4096 2016-01-12 23:39 venv
lrwxrwxrwx 1 ymuser ymuser   92 2016-01-12 23:39 venv-update -> /nail/home/jenkins/.cache/venv-update/1.0rc3/venv/lib/python2.6/site-packages/venv_update.py

but because ~/.cache/venv-update/1.0rc3/venv/bin/python exists, we assume the virtualenv is good-to-go and then we fail on the assertion below (assert exists(venv_update)).

Move stage 2 to pip_faster python package

Missing package error from venv-update is confusing

When I run venv-update red error messages show up about packages not existing. @bukzor Feel free to provide more detail as to why.

support pip6

Full disclosure: While we do intend to start using a modern pip in v2, it's
not currently a high priority.
This means probably a 6-12 month period before this happens.
However, if anyone in the community takes it upon themselves and can make
the test suite pass, we'll gladly merge.

venv_update should probably refuse/warn when installing another version of pip-faster inside the virtualenv

If your requirements.txt contains a version of pip-faster which doesn't match with the vendored venv_update, every time you run venv_update it will reinstall the version matching the vendored venv_update and then reinstall the user-requested version.

Since this almost always represents an error on the user's part, we should consider either refusing or complaining loudly.

git url without #egg= fails to install

$ python venv_update.py venv reqs2.txt 
> virtualenv venv
New python executable in venv/bin/python
Installing setuptools, pip...done.
> venv/bin/python venv_update.py --stage2 venv reqs2.txt
> pip install --upgrade --use-wheel --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse argparse==1.2.1 wheel==0.24.0
Requirement already up-to-date: argparse==1.2.1 in /usr/lib/python2.7
Requirement already up-to-date: wheel==0.24.0 in ./venv/lib/python2.7/site-packages
Cleaning up...
> pip wheel --wheel-dir=/home/asottile/.pip/wheelhouse argparse==1.2.1 wheel==0.24.0 --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse --requirement=reqs2.txt
Downloading/unpacking git+git://github.com/Yelp/yelp_cheetah@no_more_KWS (from -r reqs2.txt (line 1))
  Cloning git://github.com/Yelp/yelp_cheetah (to no_more_KWS) to /tmp/pip-LsizwD-build
  Running setup.py (path:/tmp/pip-LsizwD-build/setup.py) egg_info for package from git+git://github.com/Yelp/yelp_cheetah@no_more_KWS

Downloading/unpacking argparse==1.2.1
  File was already downloaded /home/asottile/.pip/wheelhouse/argparse-1.2.1-py2-none-any.whl
Downloading/unpacking wheel==0.24.0
  File was already downloaded /home/asottile/.pip/wheelhouse/wheel-0.24.0-py2.py3-none-any.whl
Downloading/unpacking markupsafe (from yelp-cheetah==0.5.0alpha-no-kws->-r reqs2.txt (line 1))
  File was already downloaded /home/asottile/.pip/wheelhouse/MarkupSafe-0.23-cp27-none-linux_x86_64.whl
Building wheels for collected packages: yelp-cheetah
  Running setup.py bdist_wheel for yelp-cheetah
  Destination directory: /home/asottile/.pip/wheelhouse
Successfully built yelp-cheetah
Cleaning up...
> pip install --upgrade --use-wheel --download-cache=/home/asottile/.pip/cache --find-links=file:///home/asottile/.pip/wheelhouse --no-index --requirement=reqs2.txt
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking git+git://github.com/Yelp/yelp_cheetah@no_more_KWS (from -r reqs2.txt (line 1))
  Cloning git://github.com/Yelp/yelp_cheetah (to no_more_KWS) to /tmp/pip-VDdgy5-build
  Running setup.py (path:/tmp/pip-VDdgy5-build/setup.py) egg_info for package from git+git://github.com/Yelp/yelp_cheetah@no_more_KWS

Requirement already up-to-date: argparse in /usr/lib/python2.7 (from yelp-cheetah==0.5.0alpha-no-kws->-r reqs2.txt (line 1))
Requirement already up-to-date: markupsafe in ./venv/lib/python2.7/site-packages (from yelp-cheetah==0.5.0alpha-no-kws->-r reqs2.txt (line 1))
Installing collected packages: yelp-cheetah
  Found existing installation: yelp-cheetah 0.5.0alpha-no-kws
    Uninstalling yelp-cheetah:
      Successfully uninstalled yelp-cheetah
  Running setup.py install for yelp-cheetah
    building 'Cheetah._namemapper' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c Cheetah/c/_namemapper.c -o build/temp.linux-x86_64-2.7/Cheetah/c/_namemapper.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/Cheetah/c/_namemapper.o -o build/lib.linux-x86_64-2.7/Cheetah/_namemapper.so

    Installing cheetah-compile script to /home/asottile/workspace/venv-update/venv/bin
Successfully installed yelp-cheetah
Cleaning up...

Something went wrong! Sending 'venv' back in time, so make knows it's invalid.
Waiting for all subprocesses to finish... DONE
> touch venv --reference reqs2.txt --date '1 day ago'

Traceback (most recent call last):
  File "venv_update.py", line 477, in <module>
    exit(main())
  File "venv_update.py", line 459, in main
    return venv_update(stage, venv_path, reqs, venv_args)
  File "venv_update.py", line 448, in venv_update
    return do_install(reqs)
  File "venv_update.py", line 388, in do_install
    required_with_deps = trace_requirements(required)
  File "venv_update.py", line 284, in trace_requirements
    dist = working_set.find(req.req)
  File "/home/asottile/workspace/venv-update/venv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 501, in find
    dist = self.by_key.get(req.key)
AttributeError: 'NoneType' object has no attribute 'key'

Something went wrong! Sending 'venv' back in time, so make knows it's invalid.
Waiting for all subprocesses to finish... DONE
> touch venv --reference reqs2.txt --date '1 day ago'

$ cat reqs2.txt 
git+git://github.com/Yelp/yelp_cheetah@no_more_KWS

dogfood venv-update during travis, tox

pip-faster may install un-available package versions due to wheel cache

Demo:

• a project with a custom PyPI server which has only flake8 version 2.3.0
• a project with a requirements.txt of unpinned flake8
• a flake8 2.5.1 wheel already in my wheelhouse (from another project which uses public PyPI) /nail/home/ckuehl/.cache/pip-faster/wheelhouse/flake8-2.5.1-py2.py3-none-any.whl

pip-faster install -i https://my.pypi.server/simple flake8 will mistakenly install version 2.5.1 instead of 2.3.0 which means I'll be using a different version than the rest of my team. And if I freeze requirements and ship them, it will pin a version that nobody else can install.

Can't create Python 3 virtualenvs without Python 2 pip installed

As far as I can tell, venv_update is able to make and manage Python 3 virtualenvs, but it fails due to running pip --version if Python 2's pip is not installed.

ckuehl@raptors:~$ docker run -ti debian:jessie bash
root@964a0ecca6d4:/# apt-get update && \
    apt-get install -y python3 python3-virtualenv python3-pip wget
[snip]

root@964a0ecca6d4:/# wget https://raw.githubusercontent.com/Yelp/pip-faster/master/venv_update.py
[snip]

# see https://github.com/Yelp/pip-faster/issues/65
root@964a0ecca6d4:/# patch -p0
--- venv_update.py.old      2015-09-19 20:59:52.214736883 +0000
+++ venv_update.py   2015-09-19 21:00:59.326736246 +0000
@@ -277,7 +277,11 @@

 def fresh_working_set():
     """return a pkg_resources "working set", representing the *currently* installed pacakges"""
-    from pip._vendor import pkg_resources
+    try:
+        from pip._vendor import pkg_resources
+    except ImportError:
+        # Debian de-vendorizes vendored pip dependencies
+        import pkg_resources

     class WorkingSetPlusEditableInstalls(pkg_resources.WorkingSet):

@@ -298,7 +302,11 @@
     from collections import deque
     from pip import logger
     from pip.req import InstallRequirement
-    from pip._vendor import pkg_resources
+    try:
+        from pip._vendor import pkg_resources
+    except ImportError:
+        # Debian de-vendorizes vendored pip dependencies
+        import pkg_resources

     working_set = fresh_working_set()
patching file venv_update.py

root@964a0ecca6d4:/# echo mock > requirements.txt

root@964a0ecca6d4:/# python3.4 venv_update.py -ppython3.4
> usr/bin/python3.4 -m virtualenv virtualenv_run -ppython3.4
Already using interpreter /usr/bin/python3.4
Using base prefix '/usr'
New python executable in /virtualenv_run/bin/python3.4
Also creating executable in /virtualenv_run/bin/python
Installing setuptools, pip...done.
> pip --version

Something went wrong! Sending 'virtualenv_run' back in time, so make knows it's invalid.
Waiting for all subprocesses to finish...
DONE
> touch virtualenv_run --reference requirements.txt --date '1 day ago'

Traceback (most recent call last):
  File "venv_update.py", line 603, in <module>
    exit(main())
  File "venv_update.py", line 585, in main
    return venv_update(stage, venv_path, reqs, venv_args)
  File "venv_update.py", line 571, in venv_update
    return stage1(venv_path, reqs)
  File "venv_update.py", line 551, in stage1
    run(('pip', '--version'))
  File "venv_update.py", line 97, in run
    check_call(cmd)
  File "/usr/lib/python3.4/subprocess.py", line 556, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.4/subprocess.py", line 537, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.4/subprocess.py", line 859, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.4/subprocess.py", line 1457, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'pip'

I don't know the best approach to fix it. It should be using pip3 instead:

root@964a0ecca6d4:/# which pip
root@964a0ecca6d4:/# which pip3
/usr/bin/pip3

Installing python-pip makes venv_update succeed, but it's not actually checking the right pip version.

Another workaround is ln -s /usr/bin/pip3 /usr/local/bin/pip but this isn't a very good solution.

Maybe it could do something like {sys.executable} -m pip --version instead?

don't fail when we can't find an already-installed package

I wrote this, but now can't find any reasonable use case for it.
I'll let it simmer here in case I do find one.

TODO: close 2016-03-01

$ git diff pip_faster.py
diff --git a/pip_faster.py b/pip_faster.py
index 5c14c33..02493bf 100644
--- a/pip_faster.py
+++ b/pip_faster.py
@@ -30,6 +30,7 @@ from pip.commands.install import InstallCommand
 from pip.commands.install import RequirementSet
 from pip.exceptions import InstallationError
 from pip.index import BestVersionAlreadyInstalled
+from pip.index import DistributionNotFound
 from pip.index import PackageFinder
 from pip.wheel import WheelBuilder

@@ -129,8 +130,17 @@ class FasterPackageFinder(PackageFinder):
             logger.info('slow: full search for unpinned requirement %s', req)

         # otherwise, do the full network search, per usual
-        print('FULL SEARCH FOR', req)
-        return super(FasterPackageFinder, self).find_requirement(req, upgrade)
+        try:
+            return super(FasterPackageFinder, self).find_requirement(req, upgrade)
+        except DistributionNotFound:
+            # when we can't find anything else, the installed version is the best version
+            if req.satisfied_by:
+                assert False, 'notfound, but installed coverage'
+                logger.notify("Faster! couldn't find any candidates, but it's already installed.")
+                raise BestVersionAlreadyInstalled  # TODO: test coverage
+            else:
+                assert False, 'notfound coverage'
+                raise
         # TODO: optimization -- do optimisitic wheel search even for unpinned reqs

bin/venv-update venv requirements.txt -p python2.7 doesn't work

./bin/venv-update venv requirements.txt -p python2.7
> rm -rf virtualenv_run/
> virtualenv virtualenv_run/ -p
Usage: virtualenv [OPTIONS] DEST_DIR

virtualenv: error: -p option requires an argument