treebeardtech/nbmake

📝 Pytest plugin for testing notebooks

nbmake's Issues

Error reporting still seems incorrect

1. duplication?
2. The wrong cell src contents are shown?

https://github.com/jrbourbeau/dask-examples/runs/2557562505

Better Python API

related to ukaea/PROCESS#3091 (comment) and discord discussions.

Pytest 8.1.0 compatibility

Hi,

I think an update is required in to be compatible with Pytest 8.1.0.
Source (CI logs of a project): https://github.com/BlueBrain/BlueCelluLab/actions/runs/8138274377/job/22239014458?pr=142
The packages installed as well as the environment can be found on the CI logs.

     File "/home/runner/work/BlueCelluLab/BlueCelluLab/.tox/examples/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1490, in parse
      self._preparse(args, addopts=addopts)
    File "/home/runner/work/BlueCelluLab/BlueCelluLab/.tox/examples/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1377, in _preparse
      self.pluginmanager.load_setuptools_entrypoints("pytest11")
    File "/home/runner/work/BlueCelluLab/BlueCelluLab/.tox/examples/lib/python3.11/site-packages/pluggy/_manager.py", line 415, in load_setuptools_entrypoints
      self.register(plugin, name=ep.name)
    File "/home/runner/work/BlueCelluLab/BlueCelluLab/.tox/examples/lib/python3.11/site-packages/_pytest/config/__init__.py", line 497, in register
      plugin_name = super().register(plugin, name)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/runner/work/BlueCelluLab/BlueCelluLab/.tox/examples/lib/python3.11/site-packages/pluggy/_manager.py", line 167, in register
      self._verify_hook(hook, hookimpl)
    File "/home/runner/work/BlueCelluLab/BlueCelluLab/.tox/examples/lib/python3.11/site-packages/pluggy/_manager.py", line 342, in _verify_hook
      raise PluginValidationError(
  pluggy._manager.PluginValidationError: Plugin 'nbmake' for hook 'pytest_collect_file'
  hookimpl definition: pytest_collect_file(path: str, parent: Any) -> Optional[Any]
  Argument(s) {'path'} are declared in the hookimpl but can not be found in the hookspec

Related discussion on pytest: pytest-dev/pytest#12065 (comment)

Internal error when kernel is not found for notebook

Describe the bug
Internal error when kernel is not found for notebook

To Reproduce

Set a notebook kernel to something non-existent and pytest --nbmake.

Expected behavior

Error message, report has unexecuted notebook.

Support for running arbitrary hidden code after a cell runs

Is your feature request related to a problem? Please describe.

When running in CI, I'd like to run arbitrary code after a cell has run, to configure the notebook for the CI environment.

eg: Suppose I have a notebook cell with:

access_token = "EXAMPLE" 

When running the notebook via nbmake, after this cell runs I'd like to be able to fetch and set the token from a secret store, eg:

import os, boto3
if os.getenv("CI"):
    client = boto3.client("secretsmanager")
    access_token = client.get_secret_value(SecretId="the_token_arn")["SecretString"]

Describe the solution you'd like

Potentially this could work similar to the mock functionality, ie: specified via cell metadata.

Describe alternatives you've considered

Without these I'd have to include the secret fetching in the notebook cell. But I'd rather not expose that detail to end users reading the notebook, because its only really relevant for CI, and not when they use the notebook themselves.

Error when importing 'secure_write' from 'jupyter_core.paths'

Describe the bug
As suggested in yaml2sbml-dev/yaml2sbml#34, I tried to use nbmake to add the notebooks in the repo, to the unit tests. I received the error message:

NBMAKE` INTERNAL ERROR
cannot import name 'secure_write' from 'jupyter_core.paths' (/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jupyter_core/paths.py)

To Reproduce
Run pytest --nbmake in yaml2sbml. (I guess)

Desktop (please complete the following information):

• OS: Mac iOS
• Editor: Pycharm

Additional context
If you need any further information or anything, that I can help to identify the problem, please let me know :)

Internal error when notebook is invalid json

This is handled fine but not surfaced correctly.

Add a --recursive flag?

The * wildcard search does not work in Windows terminals. For example,
pytest --nbmake */*.ipynb finds all ipynbs one level below in a bash terminal, but not in powershell or cmd. Maybe there could be a pytest --nbmake --recursive command that recursively searches all lower levels for ipynbs?

Add an option to specify the pattern of notebook files to include in tests (eg "test-*.ipynb")

It is possible to run pytest --nbmake in a project folder to collect tests both as .py files and .ipynb files. This however will collect all notebooks in the tree. In my projects there are notebooks for different purposes and some are for testing only. It would be nice to have an option to specify the pattern of notebooks to include in the tests (say "test-*.ipynb"). This option ideally could be set in the pyproject.toml file. So that one can run pytest --nbmake in any project and be sure to pick all relevant tests be it python or notebook.

Prevent `FileNotFoundError:` when reading an external file.

Is your feature request related to a problem? Please describe.
When running pytest --nbmake on a .ipynb file that has a path to an external file within the directory causes a FileNotFoundError: to occur. However, if the file is from a URL the test is completed.

Describe the solution you'd like
Be able to import an external file and read it with pd.read_csv() for example so the test can be completed.

Describe alternatives you've considered
This is not an issue using in nbconvert with the Python API interface, but nbmake is better in the command line than nbconvert.

Additional context

Test coverage

see aiidalab/aiidalab-widgets-base#444

Broken link in README

The following link in the README is broken:

It is also possible to parallelise at a CI-level using strategies, see example

Should probably link to a specific commit of what the file used to look like before it was removed.

NBMAKE INTERNAL ERROR: Exception ignored in socket

Describe the bug

I'm seeing the following error message in CI:

NBMAKE INTERNAL ERROR
Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/jsonschema/exceptions.py", line 134, in _set
    for k, v in kwargs.items():
ResourceWarning: unclosed <socket.socket fd=27, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>

  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/_pytest/runner.py", line 338, in from_call
    result: Optional[TResult] = func()
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/_pytest/runner.py", line [25](https://github.com/microsoft/torchgeo/runs/8172122269?check_suite_focus=true#step:5:26)9, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line [26](https://github.com/microsoft/torchgeo/runs/8172122269?check_suite_focus=true#step:5:27)5, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 88, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 78, in unraisable_exception_runtest_hook
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

To Reproduce

1. See https://github.com/microsoft/torchgeo/runs/8172122269?check_suite_focus=true
2. Failing notebook is https://github.com/microsoft/torchgeo/blob/releases/v0.3/docs/tutorials/transforms.ipynb

Expected behavior

I would expect nbmake to either pass all tests, or inform me of what tests may have failed.

Desktop

• OS: Ubuntu 20.04
• Browser: Chrome
• Version: 1.3.4

Additional context

Seems similar to #71 but the error message is slightly different.

Logs printing duplicate stack traces

https://github.com/yaml2sbml-dev/yaml2sbml/runs/1983797297?check_suite_focus=true

Running nbmake with specific kernel from CLI

Hi,

I've noticed that when running nbmake on a notebook, it'll use the kernelspec that has been saved in the metadata of the notebook. However, I have two kernels for a project that I'm working on - one with Spark installed, one without - so I'd like to be able to specify the kernel I want to use when running nbmake, without having to manually change the metadata of the notebook each time. Is this already possible? If not, can this feature be added?

Does the cell "raises-exception" tag actually work?

Describe the bug
I have a notebook I am testing with nbmake, and I have a cell that throws an exception, and I have added the documented metadata tag, but the test fails.

Additional context
I also took a look in the code, and while I am not familiar with the codebase, I'm not seeing code that is looking for that metadata tag. Is this feature actually enabled in the current branch? Thank you for any help you can provide.

Inaccurate err message when report fails to build

Describe the bug

When the final report fails to build, you still receive a link to view the output.

Expected behavior
Notification of a non-fatal error

Treat exceptions raised outside of notebook cells as errors

Is your feature request related to a problem? Please describe.
When executing a notebook cell, exceptions may be raised after the cell executes. This can happen, for example, when plotting a matplotlib figure using %matplotlib inline and an invalid label such as r"\\tau" (the double-slash is an incorrect escape). These exceptions raised after the cell completes are not picked up by nbmake (and also don't halt notebook execution in the Jupyter kernel).

Describe the solution you'd like
It would be nice to report these errors as failures. I don't know enough about how nbmake is implemented to know how difficult this would be.

Describe alternatives you've considered

• Detect and fail notebooks with these errors by default.
• Add a command-line option that toggles this behaviour on and off.
• Add a notebook metadata option that toggles this behaviour on and off.

Additional context
Thank you for maintaining nbmake -- it is an excellent tool.

Ignore Specific Cells when Running nbmake with PyTest

Is your feature request related to a problem? Please describe.
It is possible to configure a notebook so that some cells are ignored during the automated test? For example at the bottom of my notebook I have some code that will train a deep learning model and I want to skip that during the testing. The feature is discussed hear but I don't see how to do it treebeardtech/nbmake-action#58

Describe the solution you'd like
Ability to decorate a notebook cell so that it is note executed by nbmake+pytest, but otherwise looks normal when run interactively.

Describe alternatives you've considered
Creating 2 versions of the notebook, one with and without the troublesome code. But that will be more difficult to maintain.

Additional context
Add any other context or screenshots about the feature request here.

Passing `-Werror` to `pytest --nbmake` has no effect

Describe the bug
Passing -Werror to pytest --nbmake has no effect. It should, instead, cause the tests to fail if the code in a notebook causes a warning.

To Reproduce
Steps to reproduce the behavior:

1. Create a notebook which emits a warning.
2. Run it using pytest --nbmake --Werror **/*ipynb

Expected behavior
I expect tests to fail.

Useful feedback on identifying failing cells (nbmake vs nbval)

1. nbmake is not clearly able to run all cells to see which pass (maybe allow errors should run all AND fail if cells fail if an option is set). (@jhlegarreta chime in if you have any thoughts here)
2. nbmake requires reading the stack trace to get the failing cell, whereas nbval shows all failing cells in the report

carpentries-incubator/SDC-BIDS-dMRI#207 (comment)

NBMAKE INTERNAL ERROR: There is no current event loop

Describe the bug

Starting with nbmake 1.2, we're seeing the following error message in our tests:

NBMAKE INTERNAL ERROR
There is no current event loop
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/_pytest/runner.py", line 338, in from_call
    result: Optional[TResult] = func()
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/_pytest/runner.py", line 259, in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
returnself._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
returnself._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
raise e
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/_pytest/runner.py", line 166, in pytest_runtest_call
    item.runtest()
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/nbmake/pytest_items.py", line 52, in runtest
    res: NotebookResult = run.execute()
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/nbmake/nb_run.py", line 79, in execute
raise err
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/nbmake/nb_run.py", line 63, in execute
    c.execute(cwd=self.filename.parent)
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/nbclient/util.py", line 84, in wrapped
return just_run(coro(*args, **kwargs))
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/nbclient/util.py", line 47, in just_run
    loop = asyncio.get_event_loop()
...
sys:1: RuntimeWarning: coroutine 'NotebookClient.async_execute' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

To Reproduce

Error log can be seen at https://github.com/microsoft/torchgeo/runs/7272949937. Click on "Install pip dependencies" to see the output of pip list and the versions of all packages we are using.

Expected behavior

These tests work fine with nbmake 1.1: https://github.com/microsoft/torchgeo/runs/7273001912. There seems to have been some change in nbmake 1.2 that broke things.

Desktop (please complete the following information):

• OS: Linux
• Browser: N/A
• Version: N/A

Support for hidden cells executed only at test time

Is your feature request related to a problem? Please describe.

Many notebooks are extremely time intensive to run. In machine learning, notebooks may involve downloading very large datasets or training a model over hundreds of epochs. A single notebook may take hours to run, but we would still like to test these notebooks quickly in CI.

Describe the solution you'd like

We already have nbmake support for skipping cells, it would be interesting to be able to add hidden cells that are executed only by nbmake. With this, I could have a normal cell like:

download = True
num_epochs = 100

that is executed by all users, then a hidden cell containing:

download = False
num_epochs = 1

that is only executed by nbmake. This would keep testing time down while keeping the notebook as simple as possible.

Hidden cells definitely aren't the only way to implement this. For my use case, I really just need to change the values of a couple of variables. If nbmake offered a way to do this in a configuration file where it automatically replaced variables with these values that would also suffice. Instead of a configuration file, I would be fine with including this metadata in the notebook itself.

My goal here is that the notebook appears as simple as possible and doesn't include any visible code that is specific to nbmake testing. Any solution that offers this is equally valid to me, I definitely welcome alternative proposals.

Describe alternatives you've considered

Some alternatives I've come across:

• Use sed to replace the variable value during CI (difficult on Windows, doesn't work locally)
• Use environment variables like num_epochs = os.environ.get('NUM_EPOCHS', 100) (ugly, doesn't work locally)

Additional context

Some examples of these alternatives in the wild:

• TorchGeo
• GPyTorch

Test session with `nbmake` doesn't terminate on keyboard interrupt

Describe the bug
A running test session with e.g.pytest --nbmake *.ipynb cannot be terminated with Ctrl-C, the nbmake . The only way to terminate the test session was to kill the pytest process with SIGKILL.

To Reproduce
Run test session on multiple notebooks with a glob:

pytest --nbmake *.ipynb

Then try to terminate it with Ctrl+C.

Lack of support for missing kernels

Should link to docs/have docs for handling this:

https://github.com/billbrod/foveated-metamers/runs/2221053480

Poetry SolveProblemError due to impossible version constraints on importlib_metadata

Describe the bug

The poetry.lock file is not up to date with pyproject.toml, I noticed this when making a nix flake for nbmake (using poetry2nix).

Starting the devcontainer gives a warning about this as well:

Running the postCreateCommand from devcontainer.json...

[157415 ms] Start: Run in container: /bin/sh -c poetry install
Creating virtualenv nbmake in /workspaces/nbmake/.venv
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml.
 You may be getting outdated dependencies. Run update to update them.

However, running poetry update or poetry lock results in:

vscode ➜ /workspaces/nbmake (main) $ poetry lock
Updating dependencies
Resolving dependencies... (60.2s)

  SolverProblemError

  The current project's Python requirement (>=3.7.0,<4.0.0) is not compatible with some of the required packages Python requirement:
    - importlib-metadata requires Python >=3.8, so it will not be satisfied for Python >=3.7.0,<3.8
    - importlib-metadata requires Python >=3.8, so it will not be satisfied for Python >=3.7.0,<3.8
    - importlib-metadata requires Python >=3.8, so it will not be satisfied for Python >=3.7.0,<3.8
    - importlib-metadata requires Python >=3.8, so it will not be satisfied for Python >=3.7.0,<3.8
  
  Because no versions of importlib-metadata match >6.8.0,<6.9.0 || >6.9.0,<6.10.0 || >6.10.0,<6.11.0 || >6.11.0,<7.0.0
   and importlib-metadata (6.8.0) requires Python >=3.8, importlib-metadata is forbidden.
  And because importlib-metadata (6.9.0) requires Python >=3.8
   and importlib-metadata (6.10.0) requires Python >=3.8, importlib-metadata is forbidden.
  So, because importlib-metadata (6.11.0) requires Python >=3.8
   and nbmake depends on importlib-metadata (^6.8.0), version solving failed.

  at ~/.local/lib/python3.10/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties
    
    For importlib-metadata, a possible solution would be to set the `python` property to ">=3.8,<4.0.0"
    For importlib-metadata, a possible solution would be to set the `python` property to ">=3.8,<4.0.0"
    For importlib-metadata, a possible solution would be to set the `python` property to ">=3.8,<4.0.0"
    For importlib-metadata, a possible solution would be to set the `python` property to ">=3.8,<4.0.0"

    https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies,
    https://python-poetry.org/docs/dependency-specification/#using-environment-markers
vscode ➜ /workspaces/nbmake (main) $ 

The issue is this dependency specification:

importlib-metadata =  { version = "^6.8.0", python = "<3.8" }

The python version requirement was bumped to >=3.8 in importlib-metadata v6.8.0.

This can be fixed either by changing the dependency to ^6.7.0 (or maybe <6.8.0) or dropping support for Python <3.8. I've implemented the later suggestion in a PR.

To Reproduce
Steps to reproduce the behavior:

1. Run devcontainer
2. Run poetry lock

Expected behavior

Up to date lock file that installs without error.

Desktop (please complete the following information):
devcontainer

Support for nbclient>0.4

Hey everyone! Is there any plan for supporting nbclient>0.4.

Is there any known problem related to that? thanks!

Bump to pydantic 2.0

Is your feature request related to a problem? Please describe.
We currently have a compatibility issue when upgrading our stack to pydantic 2.0 due to nbmake incompatibility.

Describe the solution you'd like
Pydantic 2.0 support.

Additional context
Pydantic provide bump-pydantic to help dev realize the migration.

Design Test Virtualisation

Ensuring !pip install commands work without contaminating the test environment in the general case requires some thought -- requirements:

1. When running against system packages e.g. in a github actions runner, you want the notebook's virtualenv to fall back to --system-site-packages
2. When running locally in a venv, you may prefer to clone the virtualenv from which pytest is running.
3. When running in a conda environment, you either want to clone the conda environment, or create a virtualenv that falls back to it.
4. How do we test this behaviour? Some integration test running in a docker container?

UnicodeEncodeError on Windows raised in pytest_terminal_summary()

Describe the bug

• Running pytest with nbmake 1.5.1 on Windows raises an exception in pytest_terminal_summary()
• Comparing 1.5.0 (where the error doesn't happen) and 1.5.1, it looks like the following line might be related:

  nbmake/src/nbmake/pytest_plugin.py

  Line 77 in ea6d22a

  print(f"\n📝 nbmake support is available in discord: {discord}\n")

Complete stack trace

 C:\Miniconda\envs\watertap-dev\lib\site-packages\coverage\control.py:887: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")
Traceback (most recent call last):
  File "C:\Miniconda\envs\watertap-dev\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Miniconda\envs\watertap-dev\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Miniconda\envs\watertap-dev\Scripts\pytest.exe\__main__.py", line 7, in <module>
    sys.exit(console_main())
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\_pytest\config\__init__.py", line 198, in console_main
    code = main()
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\_pytest\config\__init__.py", line 175, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_callers.py", line 102, in _multicall
    res = hook_impl.function(*args)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_callers.py", line 102, in _multicall
    res = hook_impl.function(*args)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\_pytest\main.py", line 320, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\_pytest\main.py", line 273, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_callers.py", line 138, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_callers.py", line 121, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\_pytest\logging.py", line 861, in pytest_sessionfinish
    return (yield)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_callers.py", line 123, in _multicall
    teardown.send(result)  # type: ignore[union-attr]
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\_pytest\terminal.py", line 864, in pytest_sessionfinish
    self.config.hook.pytest_terminal_summary(
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_callers.py", line 138, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_callers.py", line 121, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\_pytest\terminal.py", line 886, in pytest_terminal_summary
    return (yield)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_callers.py", line 121, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\_pytest\warnings.py", line 129, in pytest_terminal_summary
    return (yield)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\pluggy\_callers.py", line 102, in _multicall
    res = hook_impl.function(*args)
  File "C:\Miniconda\envs\watertap-dev\lib\site-packages\nbmake\pytest_plugin.py", line 77, in pytest_terminal_summary
    print(f"\n\U0001f4dd nbmake support is available in discord: {discord}\n")
  File "C:\Miniconda\envs\watertap-dev\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f4dd' in position 2: character maps to <undefined>

To Reproduce

• Update nbmake to 1.5.1
• Run pytest --nbmake *.ipynb (i.e. default options)

Expected behavior

• The pytest command completes without errors

Desktop (please complete the following information):

• Version: 1.5.1
• OS: windows-2022 (GitHub Actions hosted runner)
• Python versions: 3.8, 3.11
• Python installed through Conda
• nbmake installed through pip
• Output of pip list:

  Package                       Version         Editable project location
  ----------------------------- --------------- -------------------------
  alabaster                     0.7.13
  anyio                         4.3.0
  argon2-cffi                   23.1.0
  argon2-cffi-bindings          21.2.0
  arrow                         1.3.0
  astroid                       2.15.8
  asttokens                     2.4.1
  async-lru                     2.0.4
  attrs                         23.2.0
  Babel                         2.14.0
  backcall                      0.2.0
  beautifulsoup4                4.12.3
  black                         22.3.0
  bleach                        6.1.0
  certifi                       2024.2.2
  cffi                          1.16.0
  cfgv                          3.4.0
  charset-normalizer            3.3.2
  click                         8.1.7
  colorama                      0.4.6
  comm                          0.2.1
  contourpy                     1.1.1
  coverage                      7.4.3
  cryptography                  42.0.5
  cycler                        0.12.1
  dataclasses-json              0.5.14
  debugpy                       1.8.1
  decorator                     5.1.1
  defusedxml                    0.7.1
  dill                          0.3.8
  distlib                       0.3.8
  dnspython                     2.6.1
  docutils                      0.20.1
  exceptiongroup                1.2.0
  executing                     2.0.1
  fastjsonschema                2.19.1
  filelock                      3.13.1
  fonttools                     4.49.0
  fqdn                          1.5.1
  h11                           0.14.0
  h5py                          3.10.0
  htmlmin                       0.1.12
  httpcore                      1.0.4
  httpx                         0.27.0
  idaes-pse                     2.3.0
  identify                      2.5.35
  idna                          3.6
  imagesize                     1.4.1
  importlib-metadata            7.0.1
  importlib_resources           6.1.2
  iniconfig                     2.0.0
  ipykernel                     6.29.3
  ipython                       8.12.3
  ipywidgets                    8.1.2
  isoduration                   20.11.0
  isort                         5.13.2
  jedi                          0.19.1
  Jinja2                        3.0.3
  json-schema-for-humans        0.47
  json5                         0.9.20
  jsonpointer                   2.4
  jsonschema                    4.21.1
  jsonschema-specifications     2023.12.1
  jupyter                       1.0.0
  jupyter_client                8.6.0
  jupyter-console               6.6.3
  jupyter_core                  5.7.1
  jupyter-events                0.9.0
  jupyter-lsp                   2.2.3
  jupyter_server                2.13.0
  jupyter_server_terminals      0.5.2
  jupyterlab                    4.1.2
  jupyterlab_pygments           0.3.0
  jupyterlab_server             2.25.3
  jupyterlab_widgets            3.0.10
  kiwisolver                    1.4.5
  lazy-object-proxy             1.10.0
  linkify-it-py                 2.0.3
  markdown2                     2.4.13
  MarkupSafe                    2.1.5
  marshmallow                   3.21.0
  matplotlib                    3.7.5
  matplotlib-inline             0.1.6
  mccabe                        0.7.0
  mistune                       3.0.2
  mongomock                     4.1.2
  mpmath                        1.3.0
  mypy-extensions               1.0.0
  nbclient                      0.6.8
  nbconvert                     7.16.1
  nbformat                      5.9.2
  nbmake                        1.5.1
  nbsphinx                      0.9.3
  nest-asyncio                  1.6.0
  networkx                      3.1
  nodeenv                       1.8.0
  notebook                      7.1.1
  notebook_shim                 0.2.4
  numpy                         1.24.4
  overrides                     7.7.0
  packaging                     23.2
  pandas                        2.0.3
  pandocfilters                 1.5.1
  parso                         0.8.3
  pathspec                      0.12.1
  pickleshare                   0.7.5
  pillow                        10.2.0
  Pint                          0.21.1
  pip                           23.3.1
  pkgutil_resolve_name          1.3.10
  platformdirs                  4.2.0
  pluggy                        1.4.0
  ply                           3.11
  pre-commit                    3.5.0
  prometheus_client             0.20.0
  prompt-toolkit                3.0.43
  psutil                        5.9.8
  pure-eval                     0.2.2
  pycparser                     2.21
  pydantic                      1.10.14
  Pygments                      2.17.2
  pylint                        2.17.7
  pymongo                       4.6.2
  Pyomo                         6.7.1
  pyparsing                     3.1.1
  pytest                        8.0.2
  pytest-cov                    4.1.0
  python-dateutil               2.9.0.post0
  python-json-logger            2.0.7
  pytz                          2024.1
  pywin32                       306
  pywinpty                      2.0.13
  PyYAML                        6.0.1
  pyzmq                         25.1.2
  qtconsole                     5.5.1
  QtPy                          2.4.1
  referencing                   0.33.0
  requests                      2.31.0
  rfc3339-validator             0.1.4
  rfc3986-validator             0.1.1
  rpds-py                       0.18.0
  scipy                         1.10.1
  Send2Trash                    1.8.2
  sentinels                     1.0.0
  setuptools                    68.2.2
  six                           1.16.0
  sniffio                       1.3.1
  snowballstemmer               2.2.0
  soupsieve                     2.5
  Sphinx                        7.1.2
  sphinx-rtd-theme              2.0.0
  sphinxcontrib-applehelp       1.0.4
  sphinxcontrib-devhelp         1.0.2
  sphinxcontrib-htmlhelp        2.0.1
  sphinxcontrib-jquery          4.1
  sphinxcontrib-jsmath          1.0.1
  sphinxcontrib-qthelp          1.0.3
  sphinxcontrib-serializinghtml 1.1.5
  stack-data                    0.6.3
  sympy                         1.12
  terminado                     0.18.0
  tinycss2                      1.2.1
  tomli                         2.0.1
  tomlkit                       0.12.4
  tornado                       6.4
  traitlets                     5.14.1
  types-python-dateutil         2.8.19.20240106
  typing_extensions             4.10.0
  typing-inspect                0.9.0
  tzdata                        2024.1
  uc-micro-py                   1.0.3
  uri-template                  1.3.0
  urllib3                       1.26.18
  virtualenv                    20.25.1
  watertap                      0.12.dev0       D:\a\watertap\watertap
  wcwidth                       0.2.13
  webcolors                     1.13
  webencodings                  0.5.1
  websocket-client              1.7.0
  wheel                         0.41.2
  widgetsnbextension            4.0.10
  wrapt                         1.16.0
  zipp                          3.17.0

Support for latest version of ipykernel

Currently nbmake requires ipykernel < 6.0.0 but there is a newer 6.2.0 version we would like to try out.

Thanks,
Paul

nbmake without execution

I want to iterate over tons of notebooks but without executing them beyond the point
of importing their dependencies (e.g. and raise an error if they author is importing a package not in the environment).

Currently I have:

import importlib
from pathlib import Path


def verify(path):
    for file in path.glob("**/*.ipynb"):
        file = file.relative_to(path.parent).with_suffix('')
        importlib.import_module(f'ipynb.fs.defs.{file}')


if __name__ == '__main__':
    home = Path(__file__).parent / "notebooks"
    verify(home)

Please find the ipynb package here: https://pypi.org/project/ipynb/

pytest 7

currently the pyproject.toml specifies ^6.0.1, will it work with pytest 7+?

test publish to pypi fails for external contributions

We should disable it for external PRs.

No such kernel named python3

Describe the bug
pytest can't seem to find my basic python3 kernel. when running pytest --nbmake -n=auto ./docs/projects/illusion.ipynb the test fails and I receive No such kernel named python3

To Reproduce
I am not sure what steps would be required to reproduce; I simply installed nbmake and pytest-xdist and run the above command within the local clone of my repo https://github.com/AllenInstitute/openscope_databook.git

Expected behavior
I expect the test to pass

Traceback

=================================================================================================================================================================================== test session starts ====================================================================================================================================================================================
platform win32 -- Python 3.9.10, pytest-7.2.1, pluggy-1.0.0
rootdir: C:\Users\carter.peene\Desktop\Projects\openscope_databook
plugins: anyio-3.6.1, nbmake-1.5.3, cov-4.0.0, xdist-3.5.0
8 workers [1 item]      
F                                                                                                                                                                                                                                                                                                                                                                                     [100%]
========================================================================================================================================================================================= FAILURES =========================================================================================================================================================================================
__________________________________________________________________________________________________________________________________________________ C:\Users\carter.peene\Desktop\Projects\openscope_databook\docs\projects\illusion.ipynb __________________________________________________________________________________________________________________________________________________ 
[gw0] win32 -- Python 3.9.10 C:\Users\carter.peene\AppData\Local\Programs\Python\Python39\python.exe
Error - No such kernel: 'python3'
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 
WARNING  traitlets:kernelspec.py:284 Kernelspec name python3 cannot be found!
ERROR    traitlets:manager.py:95 No such kernel named python3
Traceback (most recent call last):
  File "C:\Users\carter.peene\AppData\Local\Programs\Python\Python39\lib\site-packages\jupyter_client\manager.py", line 87, in wrapper
    out = await method(self, *args, **kwargs)
  File "C:\Users\carter.peene\AppData\Local\Programs\Python\Python39\lib\site-packages\jupyter_client\manager.py", line 435, in _async_start_kernel
    kernel_cmd, kw = await self._async_pre_start_kernel(**kw)
  File "C:\Users\carter.peene\AppData\Local\Programs\Python\Python39\lib\site-packages\jupyter_client\manager.py", line 397, in _async_pre_start_kernel
    self.kernel_spec,
  File "C:\Users\carter.peene\AppData\Local\Programs\Python\Python39\lib\site-packages\jupyter_client\manager.py", line 195, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "C:\Users\carter.peene\AppData\Local\Programs\Python\Python39\lib\site-packages\jupyter_client\kernelspec.py", line 285, in get_kernel_spec
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named python3

Learn more about nbmake at https://github.com/treebeardtech/nbmake

================================================================================================================================================================================= short test summary info ================================================================================================================================================================================== 
FAILED docs/projects/illusion.ipynb::

Environment

• OS: Windows 10
• Python 3.9.10

glob operators do not always work

Describe the bug
Hi I think there is an issue with the glob.

Here is the directory
❯ tree examples/1-singlecell -L 1
examples/1-singlecell
├── hoc
├── morphology
├── singlecell.ipynb
└── x86_64

pytest --nbmake examples//1-singlecell/singlecell.ipynb works
pytest --nbmake examples/**/*ipynb does not work
pytest --nbmake **/*ipynb does not work

When they don't work the error is the following:

It says ERROR: file or directory not found: **/*ipynb

To Reproduce

1. Use this versions

• nbmake==1.4.1
• pytest==7.3.2

2. Create a directory like the one I created and repeat the 3 commands provided above.

Expected behavior

The wildcard character to detect the .ipynb files in the inner directories.

`ModuleNotFoundError: No module named 'ipython_genutils'`

Describe the bug
In a rare case, an error might happen due to ipython_genutils not properly installed. That is because nbmake did not require ipython_genutils nor jupyter explicitly. Others also get this behavior while using IPython ipython/ipython_genutils#3.

On my use case, the error happen on matpower-pip workflow and fixed with a commit yasirroni/matpower-pip@ae73edf

To Reproduce
I don't know exactly how to reproduce, but install nbmake seems not installing ipython_genutils

Expected behavior
Whenever someone using nbmake, we can assume that they are going to test IPython file. Then, isn't logical to make nbmake require at least jupyter?

Screenshots

Desktop (please complete the following information):

• OS: Linux
• GitHub action

---

If not installing jupyter by default is the expected behavior, please ignore and close this issue.

Fails to build html report if any notebook doesn't start with a markdown header

Describe the bug
Fails to build html report if any notebook doesn't start with a markdown header

Expected behavior

Autogenerates a page title

nbmake: the jupyter-book command failed.

Running Jupyter-Book v0.8.3
Source Folder: /Users/alexremedios/Desktop/t/_build/nbmake
Config Path: _build/report_config.yml
Output Path: _build/html
/Users/alexremedios/Desktop/t/_build/nbmake/fails.ipynb:30003: WARNING: toctree contains reference to document 'passes' that doesn't have a title: no link will be generated

Theme error:
An error happened in rendering the page fails.
Reason: ValueError('Landing page missing a title: fails')
Traceback (most recent call last):
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx/builders/html/__init__.py", line 1022, in handle_page
    output = self.templates.render(templatename, ctx)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx/jinja2glue.py", line 192, in render
    return self.environment.get_template(template).render(context)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx/themes/basic/page.html", line 10, in top-level template code
    {%- extends "layout.html" %}
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx_book_theme/layout.html", line 7, in top-level template code
    {%- set sidebar_width_class = "col-md-3" %}
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx_book_theme/../pydata_sphinx_theme/layout.html", line 10, in top-level template code
    <a class='right-next' id="next-link" href="{{ next.link|e }}" title="{{ _('next page')}}">{{ next_title or next.title }}</a>
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx/themes/basic/../basic/layout.html", line 174, in top-level template code
    {%- block content %}
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx_book_theme/../pydata_sphinx_theme/layout.html", line 47, in block "content"
    {% block docs_sidebar %}
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx_book_theme/layout.html", line 30, in block "docs_sidebar"
    {% include "sidebar.html" %}
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx_book_theme/sidebar.html", line 14, in top-level template code
    {%- include sidebartemplate %}
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx_book_theme/_templates/sbt-sidebar-nav.html", line 2, in top-level template code
    {{ generate_nav_html(include_item_names=True, with_home_page=theme_home_page_in_toc) }}
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/jinja2/sandbox.py", line 462, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx_book_theme/__init__.py", line 130, in generate_nav_html
    raise ValueError(f"Landing page missing a title: {master_doc}")
ValueError: Landing page missing a title: fails

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/jupyter_book/sphinx.py", line 141, in build_sphinx
    app.build(force_all, filenames)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx/application.py", line 352, in build
    self.builder.build_update()
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 299, in build_update
    len(to_build))
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 361, in build
    self.write(docnames, list(updated_docnames), method)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 535, in write
    self._write_serial(sorted(docnames))
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 545, in _write_serial
    self.write_doc(docname, doctree)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx/builders/html/__init__.py", line 603, in write_doc
    self.handle_page(docname, ctx, event_arg=doctree)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/sphinx/builders/html/__init__.py", line 1030, in handle_page
    (pagename, exc)) from exc
sphinx.errors.ThemeError: An error happened in rendering the page fails.
Reason: ValueError('Landing page missing a title: fails')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alexremedios/Desktop/t/venv/bin/jb", line 8, in <module>
    sys.exit(main())
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/jupyter_book/commands/__init__.py", line 260, in build
    result, builder, OUTPUT_PATH, build_type, PAGE_NAME, click.echo
  File "/Users/alexremedios/Desktop/t/venv/lib/python3.7/site-packages/jupyter_book/commands/__init__.py", line 500, in builder_specific_actions
    raise RuntimeError(_message_box(msg, color="red", doprint=False)) from result
RuntimeError: 
===============================================================================

There was an error in building your book. Look above for the cause.

===============================================================================

[infra] Move CI to devcontainer cli

this will simplify operations by standardising the CI environment and testing the devcontainer implicitly

Widget calls requiring browser context hang during testing

Describe the bug
The ITKWidgets project cannot test a notebooks that calls a widget which is only initialised when a browser is present

To Reproduce
See InsightSoftwareConsortium/itkwidgets#587

Expected behavior
Can call the initialised widget API successfully

Additional context
This issue occurs because nbclient is an alternative to running a full jupyter web app + server (architecture diagram) and doesn't run a browser context.

Problem cell example

image = itk.image_from_array(array)
print('Dimension', image.GetImageDimension())
print('Components', image.GetNumberOfComponentsPerPixel())

from ipynb.

• Widget docs

`pip install nbmake` breaks pytest with `ModuleNotFoundError: No module named 'ipython_genutils'`

Describe the bug
Installing nbmake with pip appears to break the command line invocation of pytest with ModuleNotFoundError: No module named 'ipython_genutils'.

To Reproduce
Steps to reproduce the behavior:

1. Create and activate a new virtual environment

$ python3 -m venv ./nbmake
$ source ./nbmake/bin/activate
(nbmake) $

2. Install pytest and nbmake

(nbmake) $ python3 -m pip install pytest nbmake -q
(nbmake) $ pip list

output of pip list

Package             Version
------------------- -------
appnope             0.1.2
asttokens           2.0.5
attrs               21.4.0
backcall            0.2.0
debugpy             1.5.1
decorator           5.1.1
entrypoints         0.4
executing           0.8.3
importlib-resources 5.4.0
iniconfig           1.1.1
ipykernel           6.9.1
ipython             8.1.1
jedi                0.18.1
jsonschema          4.4.0
jupyter-client      7.1.2
jupyter-core        4.9.2
matplotlib-inline   0.1.3
nbclient            0.5.12
nbformat            5.2.0
nbmake              1.2
nest-asyncio        1.5.4
packaging           21.3
parso               0.8.3
pexpect             4.8.0
pickleshare         0.7.5
pip                 20.2.3
pluggy              1.0.0
prompt-toolkit      3.0.28
ptyprocess          0.7.0
pure-eval           0.2.2
py                  1.11.0
pydantic            1.9.0
Pygments            2.11.2
pyparsing           3.0.7
pyrsistent          0.18.1
pytest              7.0.1
python-dateutil     2.8.2
pyzmq               22.3.0
setuptools          49.2.1
six                 1.16.0
stack-data          0.2.0
tomli               2.0.1
tornado             6.1
traitlets           5.1.1
typing-extensions   4.1.1
wcwidth             0.2.5
zipp                3.7.0

3. Call pytest which raises

(nbmake) $ pytest

Traceback (most recent call last):
  File "/nbmake/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/nbmake/lib/python3.8/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/nbmake/lib/python3.8/site-packages/_pytest/config/__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "/nbmake/lib/python3.8/site-packages/_pytest/config/__init__.py", line 318, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/nbmake/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/nbmake/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/nbmake/lib/python3.8/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/nbmake/lib/python3.8/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/nbmake/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/nbmake/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/nbmake/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "/nbmake/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1283, in parse
    self._preparse(args, addopts=addopts)
  File "/nbmake/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1172, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/nbmake/lib/python3.8/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/nbmake/lib/python3.8/site-packages/nbmake/pytest_plugin.py", line 6, in <module>
    from .pytest_items import NotebookFile
  File "/nbmake/lib/python3.8/site-packages/nbmake/pytest_items.py", line 14, in <module>
    from .nb_run import NotebookRun
  File "/nbmake/lib/python3.8/site-packages/nbmake/nb_run.py", line 5, in <module>
    from nbclient.client import (
  File "/nbmake/lib/python3.8/site-packages/nbclient/__init__.py", line 6, in <module>
    from .client import NotebookClient, execute  # noqa: F401
  File "/nbmake/lib/python3.8/site-packages/nbclient/client.py", line 39, in <module>
    from .output_widget import OutputWidget
  File "/nbmake/lib/python3.8/site-packages/nbclient/output_widget.py", line 6, in <module>
    from .jsonutil import json_clean
  File "/nbmake/lib/python3.8/site-packages/nbclient/jsonutil.py", line 16, in <module>
    from ipython_genutils import py3compat
ModuleNotFoundError: No module named 'ipython_genutils'

Expected behavior
I would expect installing nbmake via pip to not break pytest.

Screenshots
n/a

Desktop (please complete the following information):

• OS: macOS Monterey 12.0 (Apple Silicon / M1 MacBook)
• Browser [e.g. chrome, safari]: n/a
• Version [e.g. 22]: n/a (assuming this means browser version?)

Smartphone (please complete the following information):
n/a

Additional context
None I can think of. Happy to answer follow up questions!

Glob in Windows Powershell

Describe the bug
Run pytest --nbmake -n=auto **/*ipynb get no file in Powershell

To Reproduce
Steps to reproduce the behavior:

1. Run pytest --nbmake -n=auto **/*ipynb

Expected behavior
Works like in the doc

Desktop (please complete the following information):

• OS: Windows

Issues using nbmake with Windows

pykale/pykale#221

add command line option for timeout

Thanks for making this very useful plugin! 🙏

Is your feature request related to a problem? Please describe.

I have some very slow notebooks that need a long timeout (longer than the default of 300s). I'm finding it inconvenient to edit the notebook json metadata manually, i.e. adding

  "execution": {
    "timeout": 3000
  },

Describe the solution you'd like

It would be great to be able to instead specify this option on the command line from pytest, e.g.

pytest --nbmake-timeout=3000 --nbmake notebook.ipynb

More flexible error handling

Is your feature request related to a problem? Please describe.
An issue stated that marking expected errors in metadata could be less cumbersome
Describe the solution you'd like
Potentially a python api for catching certain error types (similar to nbclient)
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
langchain-ai/langchain#94

Is telemetry really needed?

Hi @alex-treebeard,

I've been looking for a Pytest plugin to run notebooks and found your project to be an interesting alternative to e.g nbval. Looking at the repo I found out that you recently added some telemetry in #113. Even if it seems it's possible to deactivate it, I found that very surprising for an open-source project of this dimension. I usually see bigger projects implementing telemetry, they're usually backed by some start-up/company and need data to report progress to their investors. I don't know if this is the case for nbmake. To me at least, this is definitely a red flag and discards nbmake as a tool I would add to my testing framework.

pytest deprecation warning

Describe the bug
Deprecation warning appears when starting pytest using platform: darwin, Python 3.10.4, pytest 7.1.2, pytest-sugar 0.9.4 and nbmake-1.3.0:

../../xxxx/python3.10/site-packages/nbmake/pytest_plugin.py:48
  /Users/xxxx/python3.10/site-packages/nbmake/pytest_plugin.py:48: PytestRemovedIn8Warning: The (fspath: py.path.local) argument to NotebookFile is deprecated. Please use the (path: pathlib.Path) argument instead.
  See https://docs.pytest.org/en/latest/deprecations.html#fspath-argument-for-node-constructors-replaced-with-pathlib-path
    return NotebookFile.from_parent(parent, fspath=path)

Desktop (please complete the following information):

• OS: iOS
• Version 10.15.7.