cjolowicz/hypermodern-python

Hypermodern Python

hypermodern-python

Companion repository for the Hypermodern Python article series
https://medium.com/@cjolowicz/hypermodern-python-d44485d9d769

👀 Looking for a reusable project template? Check out the Hypermodern Python Cookiecutter:
https://github.com/cjolowicz/cookiecutter-hypermodern-python

hypermodern-python's Issues

Link to Mocking with Pytest section broken

Page: https://cjolowicz.github.io/posts/hypermodern-python-02-testing/

Clicking on the link titled Mocking with pytest-mock does not do anything.

It looks like there is a typo in the ID for the link ("#mocking-with-pytestmock") with the one in the corresponding section of the article ("mocking-with-pytest-mock").

delete

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Part 4 - something not right

Hi!

Just a heads up. There is passage in part 4 when talking about flake8-annotations:

Configure Flake8 to switch on the warnings generated by the plugin (TYP like type):

.flake8

[flake8]
select = ANN,B,B9,BLK,C,E,F,I,S,W

But the warning prefix for the plugin is ANN not TYP.

Congratulations and thanks again for the amazing work. Hope to read more from you in the future!

Transitive dependencies aren't pinned by install_with_constraints in Ch 3

After completing the Managing dependencies with Nox sessions in Poetry section in Chapter 3, I encounter an error about dependencies not being pinned.

It looks like the dependencies of my dev dependencies (which are now installed via install_with_constraints) are not being pinned correctly

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    bandit from https://files.pythonhosted.org/packages/5a/50/ff2f2c8f1f0ca1569f678eeb608c0f973b835985410985594fbee96be820/bandit-1.6.2-py2.py3-none-any.whl#sha256=336620e220cf2d3115877685e264477ff9d9abaeb0afe3dc7264f55fa17a3952 (from flake8-bandit==2.1.2->-c requirements-constrained.txt (line 47))
    flake8-polyfill from https://files.pythonhosted.org/packages/86/b5/a43fed6fd0193585d17d6faa7b85317d4461f694aaed546098c69f856579/flake8_polyfill-1.0.2-py2.py3-none-any.whl#sha256=12be6a34ee3ab795b19ca73505e7b55826d5f6ad7230d31b18e106400169b9e9 (from flake8-bandit==2.1.2->-c requirements-constrained.txt (line 47))
    pycodestyle from https://files.pythonhosted.org/packages/10/5b/88879fb861ab79aef45c7e199cae3ef7af487b5603dcb363517a50602dd7/pycodestyle-2.6.0-py2.py3-none-any.whl#sha256=2295e7b2f6b5bd100585ebcb1f616591b652db8a741695b3d8f5d28bdc934367 (from flake8-bandit==2.1.2->-c requirements-constrained.txt (line 47))
    attrs>=19.2.0 from https://files.pythonhosted.org/packages/14/df/479736ae1ef59842f512548bacefad1abed705e400212acba43f9b0fa556/attrs-20.2.0-py2.py3-none-any.whl#sha256=fce7fc47dfc976152e82d53ff92fa0407700c21acd20886a13777a0d20e655dc (from flake8-bugbear==20.1.4->-c requirements-constrained.txt (line 51))
    importlib-metadata; python_version < "3.8" from https://files.pythonhosted.org/packages/6d/6d/f4bb28424bc677bce1210bc19f69a43efe823e294325606ead595211f93e/importlib_metadata-2.0.0-py2.py3-none-any.whl#sha256=cefa1a2f919b866c5beb7c9f7b0ebb4061f30a8a9bf16d609b000e2dfaceb9c3 (from flake8==3.8.4->-c requirements-constrained.txt (line 57))
    pyflakes<2.3.0,>=2.2.0 from https://files.pythonhosted.org/packages/69/5b/fd01b0c696f2f9a6d2c839883b642493b431f28fa32b29abc465ef675473/pyflakes-2.2.0-py2.py3-none-any.whl#sha256=0d94e0e05a19e57a99444b6ddcf9a6eb2e5c68d3ca1e98e90707af8152c90a92 (from flake8==3.8.4->-c requirements-constrained.txt (line 57))
    mccabe<0.7.0,>=0.6.0 from https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl#sha256=ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42 (from flake8==3.8.4->-c requirements-constrained.txt (line 57))
    toml>=0.10.1 from https://files.pythonhosted.org/packages/9f/e1/1b40b80f2e1663a6b9f497123c11d7d988c0919abbf3c3f2688e448c5363/toml-0.10.1-py2.py3-none-any.whl#sha256=bda89d5935c2eac546d648028b9901107a595863cb36bae0c73ac804a9b4ce88 (from black==20.8b1->-c requirements-constrained.txt (line 1))
    pathspec<1,>=0.6 from https://files.pythonhosted.org/packages/5d/d0/887c58853bd4b6ffc7aa9cdba4fc57d7b979b45888a6bd47e4568e1cf868/pathspec-0.8.0-py2.py3-none-any.whl#sha256=7d91249d21749788d07a2d0f94147accd8f845507400749ea19c1ec9054a12b0 (from black==20.8b1->-c requirements-constrained.txt (line 1))
    mypy-extensions>=0.4.3 from https://files.pythonhosted.org/packages/5c/eb/975c7c080f3223a5cdaff09612f3a5221e4ba534f7039db34c35d95fa6a5/mypy_extensions-0.4.3-py2.py3-none-any.whl#sha256=090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d (from black==20.8b1->-c requirements-constrained.txt (line 1))
    appdirs from https://files.pythonhosted.org/packages/3b/00/2344469e2084fb287c2e0b57b72910309874c3245463acd6cf5e3db69324/appdirs-1.4.4-py2.py3-none-any.whl#sha256=a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128 (from black==20.8b1->-c requirements-constrained.txt (line 1))
    typing-extensions>=3.7.4 from https://files.pythonhosted.org/packages/60/7a/e881b5abb54db0e6e671ab088d079c57ce54e8a01a3ca443f561ccadb37e/typing_extensions-3.7.4.3-py3-none-any.whl#sha256=7cb407020f00f7bfc3cb3e7881628838e69d8f3fcab2f64742a5e76b2f841918 (from black==20.8b1->-c requirements-constrained.txt (line 1))
    regex>=2020.1.8 from https://files.pythonhosted.org/packages/93/8c/17f45cdfb39b13d4b5f909e4b4c2917abcbdef9c0036919a0399769148cf/regex-2020.9.27.tar.gz#sha256=a6f32aea4260dfe0e55dc9733ea162ea38f0ea86aa7d0f77b15beac5bf7b369d (from black==20.8b1->-c requirements-constrained.txt (line 1))
    typed-ast>=1.4.0 from https://files.pythonhosted.org/packages/0c/f0/c351d6aeb7f5ba00da2e5aac43ef09b71be95aed45bfca943a5016854269/typed_ast-1.4.1-cp37-cp37m-macosx_10_9_x86_64.whl#sha256=8ce678dbaf790dbdb3eba24056d5364fb45944f33553dd5869b7580cdbb83614 (from black==20.8b1->-c requirements-constrained.txt (line 1))
nox > Session lint-3.7 failed.

Some info

poetry --version
Poetry version 1.1.0

pyenv --version
pyenv 1.2.20

# pyproject.toml
[tool.poetry]
name = "cheap-chef"
version = "0.1.0"
description = "Recipe recommendations based on coupons at your local grocery store"
authors = ["Joseph Haaga <[email protected]>"]
license = "MIT"
readme = "README.md"
homepage = "https://github.com/josephhaaga/cheap-chef"
repository = "https://github.com/josephhaaga/cheap-chef"
keywords = ["cooking", "food"]

[tool.poetry.dependencies]
python = "^3.7"
click = "^7.1.2"
requests = "^2.24.0"

[tool.poetry.dev-dependencies]
pytest = "^6.1.0"
coverage = {version = "^5.3", extras = ["toml"]}
pytest-cov = "^2.10.1"
pytest-mock = "^3.3.1"
flake8 = "^3.8.3"
black = "^20.8b1"
flake8-black = "^0.2.1"
flake8-import-order = "^0.18.1"
flake8-bugbear = "^20.1.4"
flake8-bandit = "^2.1.2"
safety = "^1.9.0"

[tool.poetry.scripts]
cheap-chef = "cheap_chef.console:main"

[tool.coverage.paths]
source = ["src", "*/site-packages"]

[tool.coverage.run]
branch = true
source = ["cheap_chef"]

[tool.coverage.report]
show_missing = true
fail_under = 100

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Something like this for node?

Hi Claudio,
I really enjoyed this comprehensive read. You don't find something like this often on the web. Are you aware of a similar introduction for node or even typescript?
Thank in advance,
Patrick

test session remove and install dependencies at each call

In the chapter Managing dependencies in Nox sessions with Poetry you explain how we can use poetry to pin the development dependencies and avoid installing some according to the type of test session.

The result is each time I launch now -rs test first poetry uninstall dependencies that pip will re-install just after. It doesn't seem optimal.

Log:

nox > Running session tests-3.7
nox > Re-using existing virtual environment at .nox/tests-3-7.
nox > poetry install --no-dev
Installing dependencies from lock file


Package operations: 0 installs, 0 updates, 14 removals

  - Removing coverage (5.1)
  - Removing importlib-metadata (1.6.1)
  - Removing more-itertools (8.4.0)
  - Removing packaging (20.4)
  - Removing pluggy (0.13.1)
  - Removing py (1.8.2)
  - Removing pyparsing (2.4.7)
  - Removing pytest (5.4.3)
  - Removing pytest-cov (2.10.0)
  - Removing pytest-mock (3.1.1)
  - Removing six (1.15.0)
  - Removing toml (0.10.1)
  - Removing wcwidth (0.2.4)
  - Removing zipp (3.1.0)
  - Installing hypermodern-python (0.1.0)
nox > poetry export --dev --format=requirements.txt --output=/var/folders/v5/wtydb2rx72b74xds_1hc2xr80000gq/T/tmp2tb8qu3a
nox > pip install --constraint=/var/folders/v5/wtydb2rx72b74xds_1hc2xr80000gq/T/tmp2tb8qu3a coverage[toml] pytest pytest-cov pytest-mock
nox > pytest --cov -m not e2e
...

Add pytest's best-practice config to `pyproject.toml`

The pyproject.toml file can be used to store pytest's config (otherwise declared in a separate pytest.ini file), see:
https://docs.pytest.org/en/6.2.x/customize.html#pyproject-toml

It could be valuable to add the recommended options for pytest:

• Treat warnings as errors by default: https://docs.pytest.org/en/7.2.x/reference/reference.html#confval-filterwarnings
• Use importlib import mode: https://docs.pytest.org/en/7.2.x/explanation/goodpractices.html#tests-outside-application-code

# pyproject.toml

[tool.pytest.ini_options]
filterwarnings = ["error"]
addopts = ["--import-mode=importlib"]

Note that the importlib option probably unnecessitates placing the hypermodern_python package in the src folder (src/hypermodern_python):
src/hypermodern_python -> hypermodern_python

Migrate mypy's config from `mypy.ini` into `pyproject.toml`

Instead of a separate file for mypy's config mypy.ini, use the already existing pyproject.toml file to store mypy's config.
See:
https://mypy.readthedocs.io/en/stable/config_file.html#using-a-pyproject-toml-file

Specifically,

hypermodern-python/mypy.ini

Lines 1 to 4 in ed593b8

	[mypy]
	[mypy-desert,marshmallow,nox.*,pytest,pytest_mock,_pytest.*]
	ignore_missing_imports = True

⬇️

# pyproject.toml

[[tool.mypy.overrides]]
module = [
    "desert",
    "marshmallow",
    "nox.*",
    "pytest",
    "pytest_mock",
    "_pytest.*",
]
ignore_missing_imports = true

Dependency errors when using python 3.9 and 3.8

Issue

When using python version 3.9: For me, all nox sessions where install_with_constrains is used with python version 3.9 fail.

For instance running the tests session works fine, till chapter 4. After adding more nox sessions and installing more dependencies I started to see multiple dependencies for different python versions (I am following the tutorial using python 3.9 and 3.8 respectively).

After some research, it seems this problem is related to Duplicate dependency export #1970

Console Output

CLICK ME

nox > Running session tests-3.9
nox > Re-using existing virtual environment at .nox/tests-3-9.
nox > poetry install --no-dev
Installing dependencies from lock file

Package operations: 2 installs, 0 updates, 11 removals

  • Removing coverage (5.3)
  • Removing iniconfig (1.1.1)
  • Removing packaging (20.4)
  • Removing pluggy (0.13.1)
  • Removing py (1.9.0)
  • Removing pyparsing (2.4.7)
  • Removing pytest (6.1.1)
  • Removing pytest-cov (2.10.1)
  • Removing pytest-mock (3.3.1)
  • Removing six (1.15.0)
  • Removing toml (0.10.1)
  • Installing mypy-extensions (0.4.3)
  • Installing typing-extensions (3.7.4.3)

Installing the current project: hypermodern-bersten (0.1.0)
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmppd82bei6
nox > pip install --constraint=/tmp/tmppd82bei6 coverage[toml] pytest pytest-cov pytest-mock
nox > Command pip install --constraint=/tmp/tmppd82bei6 coverage[toml] pytest pytest-cov pytest-mock failed with exit code 1:
Ignoring atomicwrites: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and sys_platform == "win32" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") or sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.4.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring attrs: markers 'python_version == "3.8" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version == "3.8"' don't match your environment
Ignoring colorama: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and sys_platform == "win32" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") and platform_system == "Windows" or sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.5.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") and platform_system == "Windows"' don't match your environment
Ignoring decorator: markers 'python_version == "3.8" and python_full_version < "3.0.0" and python_full_version >= "2.7.0" or python_version == "3.8" and python_full_version >= "3.2.0"' don't match your environment
Ignoring importlab: markers 'python_version == "3.8" and python_full_version >= "2.7.0"' don't match your environment
Ignoring networkx: markers 'python_version == "3.8" and python_full_version >= "2.7.0"' don't match your environment
Ignoring ninja: markers 'python_version == "3.8"' don't match your environment
Ignoring pytype: markers 'python_version == "3.8"' don't match your environment
Ignoring pyyaml: markers 'python_version == "3.8" and python_full_version < "3.0.0" or python_version == "3.8" and python_full_version >= "3.5.0"' don't match your environment
Ignoring six: markers 'python_version == "3.8" and python_full_version < "3.0.0" and python_full_version >= "2.7.0" or python_version == "3.8" and python_full_version >= "3.4.0"' don't match your environment
Ignoring typed-ast: markers 'python_version == "3.8"' don't match your environment
Collecting coverage==5.3
  Using cached coverage-5.3-cp39-cp39-manylinux1_x86_64.whl (228 kB)
Collecting pytest-cov==2.10.1
  Using cached pytest_cov-2.10.1-py2.py3-none-any.whl (19 kB)
Collecting pytest-mock==3.3.1
  Using cached pytest_mock-3.3.1-py3-none-any.whl (11 kB)
Collecting pytest==6.1.1
  Using cached pytest-6.1.1-py3-none-any.whl (272 kB)
Collecting toml==0.10.1
  Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
Requirement already satisfied: attrs>=17.4.0 in ./.nox/tests-3-9/lib/python3.9/site-packages (from pytest==6.1.1->-c /tmp/tmppd82bei6 (line 176)) (20.2.0)
Collecting pluggy==0.13.1
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting iniconfig==1.1.1
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting packaging==20.4
  Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting py==1.9.0
  Using cached py-1.9.0-py2.py3-none-any.whl (99 kB)
Collecting pyparsing==2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting six
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    six from https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl#sha256=8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced (from packaging==20.4->-c /tmp/tmppd82bei6 (line 146))
nox > Session tests-3.9 failed.

IDE question

Hi thanks for the guide, very interesting so far,
What would be the reason IDEs where left out of the discussion?

Upgrade to poetry 1.2 and separate the dev-dependencies into groups

Instead of installing all the dev-dependencies on each nox session:

hypermodern-python/pyproject.toml

Lines 21 to 42 in ed593b8

	[tool.poetry.dev-dependencies]
	pytest = "^5.3.2"
	coverage = {extras = ["toml"], version = "^5.0.1"}
	pytest-cov = "^2.8.1"
	pytest-mock = "^2.0.0"
	flake8 = "^3.7.9"
	black = "^19.10b0"
	flake8-black = "^0.1.1"
	flake8-import-order = "^0.18.1"
	flake8-bugbear = "^20.1.2"
	flake8-bandit = "^2.1.2"
	safety = "^1.8.5"
	mypy = "^0.761"
	pytype = {version = "^2020.1.8", python = "3.7"}
	flake8-annotations = "^2.0.0"
	typeguard = "^2.7.1"
	flake8-docstrings = "^1.5.0"
	darglint = "^1.1.2"
	xdoctest = "^0.11.0"
	sphinx = "^2.3.1"
	sphinx-autodoc-typehints = "^1.10.3"
	codecov = "^2.0.22"

Split the dev-deps into groups - a feature provided by poetry 1.2:
https://python-poetry.org/docs/managing-dependencies/#dependency-groups
Install only the relevant dependencies.

This requires upgrading poetry to 1.2 (e.g., 1.2.2):

hypermodern-python/.github/workflows/tests.yml

Line 17 in ed593b8

- run: pip install poetry==1.0.5

Suggested groups:

• test
• lint
• typing
• docs

Issue with pyenv shell initialization as documented in blog post

While reviewing Part 1 of the corresponding blog series it appears that one of the shell commands needs updated for more recent versions of pyenv. After running the following shell commands and later doing pyenv local <version #>, I found that things were not working as expected where python was mapped to the appropriate local versions. This was all done on macOS with Homebrew installed pyenv/poetry.

export PATH="~/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

After browsing around I found several posts that suggested that the eval "$(pyenv init -)" needed to be changed to eval "$(pyenv init --path)" for more recent pyenv than was available when the posts were done. After doing this things seemed to work as advertised.

References:

• pyenv/pyenv#2014
• https://stackoverflow.com/questions/68462156/upgraded-pyenv-by-brew-now-only-system-python-activated-on-new-terminal

MyPy Cannot Find Library Stubs for Specific Python Version

Disclaimer: This is not an issue per se, but a general question. I posted it to StackOverflow first, but there was no response, so I'm reposting it here in the hopes of finding answers.

I'm following the tutorial on type checking with mypy here.

When I run a nox session to run mypy against the code, the Python 3.7 check complains about mypy not being able to find stubs for some libraries (e.g. click) whereas the checks for Python 3.8 finish successfully. And until I ignored requests, that was part of errors in 3.7 too. Why does this happen?

(my-hypermodern-python-VDazB31k-py3.8) ➜ nox -rs mypy
nox > Running session mypy-3.8
nox > Re-using existing virtual environment at .nox/mypy-3-8.
nox > poetry export --dev --format=requirements.txt --without-hashes --output=/tmp/tmpyiz1np1u
nox > python -m pip install --constraint=/tmp/tmpyiz1np1u mypy
nox > mypy src tests noxfile.py
Success: no issues found in 8 source files
nox > Session mypy-3.8 was successful.
nox > Running session mypy-3.7
nox > Re-using existing virtual environment at .nox/mypy-3-7.
nox > poetry export --dev --format=requirements.txt --without-hashes --output=/tmp/tmp7_a7k3nt
nox > python -m pip install --constraint=/tmp/tmp7_a7k3nt mypy
nox > mypy src tests noxfile.py
src/my_hypermodern_python/wikipedia.py:1: error: Cannot find implementation or library stub for module named "click"
src/my_hypermodern_python/console.py:3: error: Cannot find implementation or library stub for module named "click"
src/my_hypermodern_python/console.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
tests/test_console.py:1: error: Cannot find implementation or library stub for module named "click.testing"
tests/test_console.py:1: error: Cannot find implementation or library stub for module named "click"
Found 4 errors in 3 files (checked 8 source files)
nox > Command mypy src tests noxfile.py failed with exit code 1
nox > Session mypy-3.7 failed.
nox > Ran multiple sessions:
nox > * mypy-3.8: success
nox > * mypy-3.7: failed

WSL2 - Ubuntu 20.04
Python 3.7 and 3.8 versions from pyenv
nox 2020.12.31
mypy 0.910
click 7.1.2

# noxfile.py

def install_with_constraints(session, *args, **kwargs):
    with tempfile.NamedTemporaryFile() as requirements:
        session.run(
            "poetry",
            "export",
            "--dev",
            "--format=requirements.txt",
            "--without-hashes",
            f"--output={requirements.name}",
            external=True,
        )
        session.install(f"--constraint={requirements.name}", *args, **kwargs)

# Nox session for mypy
@nox.session(python=["3.8", "3.7"])
def mypy(session):
    args = session.posargs or locations
    install_with_constraints(session, "mypy")
    session.run("mypy", *args)

# mypy.ini

[mypy]

[mypy-nox.*,pytest,requests]
ignore_missing_imports = True

`poetry install` fails with TypeError when run from `nox`

REPRO

• clone this repo
• check out the chapter07 branch
• run nox

ERROR

[TypeError]
expected string or bytes-like object

LOG

06:24 $ git log | head
commit 71dab2c92819ba95a25fa4e4127d9ea62a580c93
Author: Claudio Jolowicz <[email protected]>
Date:   Thu Nov 21 09:20:04 2019 +0100

    Add Docker Hub badge

commit d288141b13628db345453923a63cbaa32d801819
Author: Claudio Jolowicz <[email protected]>
Date:   Wed Nov 20 19:25:17 2019 +0100

✔ ~/repos/ext/hypermodern-python [remotes/origin/chapter07 L|✔]
06:24 $ nox
nox > Running session lint-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/lint-3-8
nox > pip install flake8 flake8-annotations flake8-bandit flake8-black flake8-bugbear flake8-docstrings flake8-import-order darglint
nox > flake8 src tests noxfile.py docs/conf.py
nox > Session lint-3.8 was successful.
nox > Running session lint-3.7
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/lint-3-7
nox > pip install flake8 flake8-annotations flake8-bandit flake8-black flake8-bugbear flake8-docstrings flake8-import-order darglint
nox > flake8 src tests noxfile.py docs/conf.py
nox > Session lint-3.7 was successful.
nox > Running session mypy-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/mypy-3-8
nox > pip install mypy
nox > mypy src tests noxfile.py docs/conf.py
Success: no issues found in 9 source files
nox > Session mypy-3.8 was successful.
nox > Running session mypy-3.7
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/mypy-3-7
nox > pip install mypy
nox > mypy src tests noxfile.py docs/conf.py
Success: no issues found in 9 source files
nox > Session mypy-3.7 was successful.
nox > Running session pytype
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/pytype
nox > pip install pytype
nox > pytype --config=pytype.cfg src tests noxfile.py docs/conf.py
Computing dependencies
Analyzing 9 sources with 0 local dependencies
ninja: Entering directory `/Users/tdgreenw/repos/ext/hypermodern-python/.pytype'
[9/9] check tests.conftest
Success: no errors found
nox > Session pytype was successful.
nox > Running session tests-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/tests-3-8
nox > poetry install

[TypeError]
expected string or bytes-like object
nox > Command poetry install failed with exit code 1
nox > Session tests-3.8 failed.
nox > Running session tests-3.7
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/tests-3-7
nox > poetry install

[TypeError]
expected string or bytes-like object
nox > Command poetry install failed with exit code 1
nox > Session tests-3.7 failed.
nox > Ran multiple sessions:
nox > * lint-3.8: success
nox > * lint-3.7: success
nox > * mypy-3.8: success
nox > * mypy-3.7: success
nox > * pytype: success
nox > * tests-3.8: failed
nox > * tests-3.7: failed

Reference page doesn't work on ReadTheDocs

The reference page relies on the automodule directive. My guess is that readthedocs needs to install the module to be able to use this. Because readthedocs doesn't use poetry, the only way I see to do this is to upload the package to PyPI and include it in the .readthedocs.yml file.

hypermodern_python==1.0.0

Can the Hypermodern Python essays be published elsewhere (maybe here)?

Medium.com is prompting me for a commercial relationship I don't care to have. Your first essay was pretty cool, and I would read more if more was actually available.

information about binary wheel builds

As I understand you plan to update your series, may be it will be good if you'll mention about the auditwheel project. Many interesting details can be found here

Developer installation

Hello and thank you for your great work on this setup! ❤️ 🤘

With your nox + poetry setup, it is currently not supported to perform a developer installation in conjunction with your install_with_constraints function.

I am adding something like this to my pyproject.toml:

my-dependency = { path = "../my-dependency" }

And if I then run one of my nox sessions, I get this:

ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/fredrik/code/repos/my-dependency
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

This I get because pip is being used to conduct the installation and I guess that there is a limitation to what pip can currently do. The entry in the temporary constraints text file is: -e ../my-dependency

I'm wondering if this is a road worth pursuing, so I created this issue.

Perhaps I could remove the entries in the constraints file starting with -e. For any dependency removed, I could perhaps build a wheel using the pep517 module and then finally install this wheel via pip and the constraints file.

python -m pep517.build --source ../my-dependency --binary

Do you see any better alternative to this?
It would be nice to avoid the build and just do the install straight away, but I don't know if any tool supports this right now.

Problem on part2 - running nox

Hi, I am following along your articles (great work BTW) and I don't know if here is the most appropriate place to reach you, but I couldn't run nox following the instructions.

I think that there is some conflict between pyenv and previously installed anaconda. I get permission errors:

$ nox
nox > Running session tests-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/tests-3-8
nox > Command /home/user/anaconda3/bin/python3 -m virtualenv /media/user/7A28-2BBB/Hypermodern Python/.nox/tests-3-8 -p python3.8 failed with exit code 1:
PermissionError: [Errno 1] Operation not permitted: '/home/user/.pyenv/versions/3.8.2/bin/python3.8' -> '/media/user/7A28-2BBB/Hypermodern Python/.nox/tests-3-8/bin/python'
nox > Session tests-3.8 failed.
nox > Running session tests-3.7
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/tests-3-7
nox > Command /home/user/anaconda3/bin/python3 -m virtualenv /media/user/7A28-2BBB/Hypermodern Python/.nox/tests-3-7 -p python3.7 failed with exit code 1:
PermissionError: [Errno 1] Operation not permitted: '/home/user/anaconda3/bin/python3' -> '/media/user/7A28-2BBB/Hypermodern Python/.nox/tests-3-7/bin/python'
nox > Session tests-3.7 failed.
nox > Ran multiple sessions:
nox > * tests-3.8: failed
nox > * tests-3.7: failed

Do you have any idea how can I can overcome that?

I've also tried:

source ~/.poetry/env
poetry run nox

And I get the same output.

Consider replacing requests

Requests has been the default suggestion for a python http client library, however it is starting to show its age.

I suggest replacing it in this guide with one of the alternatives, httpx

It is a drop-in replacement for requests, and it also adds async support among other things.

Links to Pytest Documentation Broken

The second post of this series on testing includes a few helpful links to Pytest documentation, but unfortunately the pages seem to have moved elsewhere. (I get a "Page not found".) Perhaps the links should be amended.

A couple of examples:

The one on test fixtures:

Click’s testing.CliRunner can invoke the command-line interface from within a test case. Since this is likely to be needed by most test cases in this module, let’s turn it into a test fixture.

conftest.py

The mock_requests_get fixture is now used by two test modules. You could move it to a separate module and import from there, but Pytest offers a more convenient way

Default branch on Github now is "main"; this conflicts with some links in the repo

Since October 2020 the default branch for Github repositories is main instead of master. Since in the repo and articles there are a few links that rely on "master" this could cause confusion for new users since some things won't work as expected:

• release-drafter.yml
• test-pypi.yml
• The codecov badge in README.md

Perhaps these should be updated to main instead? However on gitlab.com the default branch still remains master so if someone creates the repo there they face the same issues. In any case, a comment in the article would be helpful too.

Redundant matrix strategy in `.github/workflows/tests.yml`

Thanks for a great blog-series and repo!

I noticed that the matrix strategy python-version: ['3.7', '3.8'] in .github/workflows/tests.yml is redundant, since nox will take care of testing both versions anyway.

nox session doesn't find pytest installed by poetry

I'm following along the Hypermodern Python series and it seems in Chapter 2, Test automation with Nox there is a conflict between nox and poetry. Specifically when I use the example noxfile.py:

import nox

@nox.session(python=["3.8", "3.7"])
def tests(session):
    session.run("poetry", "install", external=True)
    session.run("pytest", "--cov")

I get the following error:

nox > Running session tests-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/tests-3-8
nox > poetry install
Installing dependencies from lock file

No dependencies to install or update

  - Installing testpkg (0.1.0)
nox > Program pytest not found.
nox > Session tests-3.8 failed.

(and the same for tests-3.7)

It seems that installing pytest via poetry as a dev dependency doesn't put it on the path. In the previous section of Chapter 2, pytest was invoked through poetry and this does work in the nox session as well:

session.run('poetry', 'run', 'pytest', '--cov')

Is this an issue with my setup or is it expected that pytest can't be run as a command when installed by poetry?

Nox complains about not finding pip

Hello,
I'm on Chapter 3, and I've added most of the stuff in that article (without comitting atomic - sorry!) and I can't seem to commit and push the changes so far. When running nox -rs lint, it fails;

$ nox -rs lint
nox > Running session lint-3.8
nox > Re-using existing virtual environment at .nox/lint-3-8.
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmpl2c9lseu
nox > pip install -c /tmp/tmpl2c9lseu flake8 flake8-annotations flake8-bandit flake8-black flake8-bugbear flake8-import-order
nox > Error: pip is not installed into the virtualenv, it is located at /home/per/.pyenv/shims/pip. Pass external=True into run() to explicitly allow this.
nox > Session lint-3.8 failed.

All other nox sessions complete successfully, all other being typeguard, mypy, safety, tests and black. Yet lint doesn't play ball. This is the code for it, think it's identical to yours.

def install_with_constraints(session, *args, **kwargs):
    with tempfile.NamedTemporaryFile() as requirements:
        session.run(
            "poetry",
            "export",
            "--dev",
            "--format=requirements.txt",
            f"--output={requirements.name}",
            external=True,
        )
        session.install(f"--constraint={requirements.name}", *args, **kwargs)


@nox.session(python=["3.8"])
def lint(session):
    args = session.posargs or locations
    install_with_constraints(
        session,
        "flake8",
        "flake8-annotations",
        "flake8-bandit",
        "flake8-black",
        "flake8-bugbear",
        "flake8-import-order",
    )
    session.run("flake8", *args)

Any ideas? Having a hard time debugging on my own. Thanks for an awesome guide.

small typo in hypermodern-python-01-setup

At the bottom, underneath the "Thanks for reading!" section, there is a typo in the following date:

" ... published on January 8, 20202"

Super tiny issue!

Thanks for the guide, it's super helpful.

Questions!

I was going to answer here: pyupio/safety#201 but it's a bit off-topic, so here I come!

---

Very interesting, thanks. A few questions to make sure I understand correctly:

• for each "session", nox uses a specific virtualenv (like tox)
• this is why safety is isolated here
• and it uses the constraints feature of pip to make sure to check the right versions, without actually installing them
• do you list nox as a dev dep in pyproject.toml?

I like the approach for the following reasons:

• replacement of Makefile in pure Python
• install only the packages required for a specific task, not the whole dev dep list (great in CI)

But I have doubts about the possible redundancies:

• poetry-managed venv AND one venv for each nox session?
• nox sessions (+venvs) AND pre-commit hooks?

Could it be possible to use nox in pre-commit, or pre-commit in nox to avoid these?

Anyway, thanks a lot, this is really interesting 🙂

Debugging

Just a quick question: why didn't you put on console.py a:

if __name__ == "__main__":
    main()

Isn't it easier to debug code with that? I don't remember if you mention your IDE in the series but how do you usually start debugging with the code as is?

Kind regards..

install_with_constraints fails with the new poetry 1.1.0

First of all big thanks for an excellent example for Python project tooling!

Have you also noticed this problem with latest poetry update?
poetry 1.1.0
nox 2020.8.22

❯ nox -rs black
nox > Running session black
nox > Re-using existing virtual environment at .nox/black.
nox > poetry export --dev --format=requirements.txt --output=/var/folders/qj/sjb9y3s56sj1s60cwhmhln3cy765d7/T/tmp57tr87b4

  IndexError

  list index out of range

  at ~/.pyenv/versions/3.8.1/lib/python3.8/site-packages/poetry/utils/exporter.py:76 in _export_requirements_txt
       72│             if not dev
       73│             else self._poetry.package.all_requires,
       74│             with_nested=True,
       75│         ):
    →  76│             package = repository.find_packages(dependency=dependency)[0]
       77│
       78│             # If a package is optional and we haven't opted in to it, continue
       79│             if package.optional and package.name not in extra_package_names:
       80│                 continue
nox > Command poetry export --dev --format=requirements.txt --output=/var/folders/qj/sjb9y3s56sj1s60cwhmhln3cy765d7/T/tmp57tr87b4 failed with exit code 1
nox > Session black failed.

It works fine if downgrading poetry to 1.0.5.
Strangely running below command directly i.e. not in nox session works with poetry 1.1.0.

❯ poetry export --dev --format=requirements.txt --output=/var/folders/qj/sjb9y3s56sj1s60cwhmhln3cy765d7/T/tmp57tr87b4

Nox session install not working

I have been following along, and I realise the articles are a few years old and poetry, pip etc. have moved along.

But when the install_with_constraints function does the install it seems as though pip doesn't like the constaints file.
Just wondering if this is something anyone else has seen and/or knows a way around?

$ nox -s black
nox > Running session black
nox > Creating virtual environment (virtualenv) using python3.10 in .nox/black
nox > poetry export --with dev --format=requirements.txt --output=/tmp/tmplk132qu3
nox > python -m pip install --constraint=/tmp/tmplk132qu3 black
nox > Command python -m pip install --constraint=/tmp/tmplk132qu3 black failed with exit code 1:
DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
ERROR: Constraints cannot have extras
nox > Session black failed.

Pytest default src layout tweak

Claudio,

Thank you very much for this comprehensive article, which I discovered from a recent Talk Python to Me podcast (362).

I was struck by the use of src layout, which I have seen before but did not understand. I immediately tried to adopt it for a
small project, but was struck that if I created a blank project with src layout, pytest was not working by default.

For example:

$ poetry new --name rnickle_template --src rnickle_template
$ ls
README.rst  poetry.lock  pyproject.toml  src  tests
$ pytest | grep ^E
E   ModuleNotFoundError: No module named 'rnickle_template'
ERROR tests/test_rnickle_template.py

There's an easy fix:

$ touch src/conftest.py
$ pytest
[ redacted ]
= 1 passed in 0.39s =

Or in pytest >= 7:

[tool.pytest.ini_options]
pythonpath = [
  "src"
]

Perhaps we can add a comment about that to the setup page?

Thanks,

Rick

[tool.poetry.scripts] hypermodern-python script not found

REPRO

• checkout chapter07 branch

08:51 $ git log | head
commit 71dab2c92819ba95a25fa4e4127d9ea62a580c93
Author: Claudio Jolowicz <[email protected]>
Date:   Thu Nov 21 09:20:04 2019 +0100

    Add Docker Hub badge

08:55 $ grep scripts pyproject.toml -A 2
[tool.poetry.scripts]
hypermodern-python = "hypermodern_python.console:main"

08:56 $ poetry install
Installing dependencies from lock file

No dependencies to install or update

  - Installing hypermodern-python (0.1.0)

08:56 $ hypermodern-python
-bash: hypermodern-python: command not found

I suspect that placing the code in the src directory is causing this.

Where nox should be installed

In the second chapter of Hypermodern Python you use pip install --user --upgrade nox to install nox did you do this inside the virtualenvironment created by poetry? OR you deactivate the virtualenvironment and install globally nox?

If you did it inside poetry virtualenvironment you use pip instead of poetry so that it doesn't stay in the pyptoject.toml packages?

readthedoc from pyproject.toml

Hi. Thank you for this great " Hypermodern Python" series.
With setup like this:

# pyproject.toml
...
[tool.poetry.dependencies]
# project deps
...
python = "<3.8,>=3.7"
# dev deps
black = { version = "^19.10b0", optional = true }
coverage = {version = "^5.1", optional = true }
...
sphinx-rtd-theme = { version = "^0.4.3", optional = true }
toml = { version = "^0.10.0", optional = true }
typeguard = { version = "^2.7.1", optional = true }

[tool.poetry.extras]
dev = ["black", "coverage", ... "toml", "typeguard"]
docs = ["m2r", "sphinx", "sphinx-autodoc-typehints", "sphinx-rtd-theme", "toml"]

# readthedocs.yml
version: 2

sphinx:
  configuration: docs/conf.py

python:
  version: 3.7
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

we can

• read pyproject.toml from docs/conf.py for metadata
• use pyproject.toml "as is" without additional docs/requirements.txt file

Thankyou

This is fantastic :)

It's certainly interesting to see what these tools can do for a project.

Appreciate minimizing the entry-points by using nox.