Tests failing on FreeBSD

[yurivict]

Tests failing on FreeBSD

[arogozhnikov]

that's kinda very interesting repro steps.

testing in repo is performed by test.py script, not pytest.

Your setup seems to lack einops (4 out of 5 errors).
Torch.jit error is maybe intersting for torch, but they can't be caused by einops itself - only if smth is wrong with torch+BSD.

Can you explain your test setup and what are you trying to check/confirm?
Einops is pure-python and OS-agnostic. If it works on linux, it would work in BSD too unless underlying library (numpy/torch/etc) has issues with either of these.

[yurivict]

It runs the standard PyTest-based test command provided by the FreeBSD ports framework. It is used by thousands of other Python-based ports.

This approach uses the same dependency packages that will be used when this package will be run by users.

There is another target that is defined in the port for testing: python test.py numpy pytorch.

It downloads a lot of dependencies for some reason, even though they are supposed to be already installed, and still has 2 failures:

Successfully installed einops-0.7.0
==================================================================================== test session starts ====================================================================================
platform freebsd14 -- Python 3.9.18, pytest-7.4.4, pluggy-1.4.0
Using --randomly-seed=3005111187
rootdir: /usr/ports/misc/py-einops/work-py39/einops-0.7.0
configfile: pyproject.toml
plugins: anyio-4.3.0, hypothesis-6.98.18, cov-4.1.0, datadir-1.5.0, randomly-3.12.0, timeout-2.1.0, time-machine-2.11.0, rerunfailures-11.1.2, flaky-3.7.0, forked-1.6.0, aspectlib-2.0.0, xdist-3.5.0, env-0.6.2, mock-3.10.0, typeguard-4.1.5
collected 105 items                                                                                                                                                                         

tests/test_examples.py ..                                                                                                                                                             [  1%]
tests/test_ops.py ........................                                                                                                                                            [ 24%]
tests/test_parsing.py ....                                                                                                                                                            [ 28%]
tests/test_notebooks.py .F..                                                                                                                                                          [ 32%]
tests/test_other.py ......................................................                                                                                                            [ 83%]
tests/test_packing.py ....                                                                                                                                                            [ 87%]
tests/test_einsum.py ....                                                                                                                                                             [ 91%]
tests/test_layers.py .F.s..s..                                                                                                                                                        [100%]

========================================================================================= FAILURES ==========================================================================================
______________________________________________________________________________________ test_notebook_3 ______________________________________________________________________________________

self = <nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x2f1de08838b0>, msg_id = '81dab144-2fafa360cbc487d82513b3d4_49971_9'
cell = {'cell_type': 'code', 'execution_count': 6, 'id': '3987bbbc-ccc2-4374-8979-fd3fb29c4910', 'metadata': {'execution': {'...it=1.)),\n    torch.jit.script(ResMLP_Blocks3(128, dim=128, layerscale_init=1.)),\n]:\n    %timeit -n 10 y = layer(x)'}
timeout = 60
task_poll_output_msg = <Task pending name='Task-256' coro=<NotebookClient._async_poll_output_msg() running at /usr/local/lib/python3.9/site-packages/nbclient/client.py:813> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x2f1ddef25e20>()]>>
task_poll_kernel_alive = <Task cancelled name='Task-255' coro=<NotebookClient._async_poll_kernel_alive() done, defined at /usr/local/lib/python3.9/site-packages/nbclient/client.py:821>>

    async def _async_poll_for_reply(
        self,
        msg_id: str,
        cell: NotebookNode,
        timeout: int | None,
        task_poll_output_msg: asyncio.Future[t.Any],
        task_poll_kernel_alive: asyncio.Future[t.Any],
    ) -> dict[str, t.Any]:
        msg: dict[str, t.Any]
        assert self.kc is not None
        new_timeout: float | None = None
        if timeout is not None:
            deadline = monotonic() + timeout
            new_timeout = float(timeout)
        error_on_timeout_execute_reply = None
        while True:
            try:
                if error_on_timeout_execute_reply:
                    msg = error_on_timeout_execute_reply  # type:ignore[unreachable]
                    msg["parent_header"] = {"msg_id": msg_id}
                else:
>                   msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))

/usr/local/lib/python3.9/site-packages/nbclient/client.py:782: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.9/site-packages/jupyter_core/utils/__init__.py:198: in ensure_async
    result = await obj
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <jupyter_client.channels.AsyncZMQSocketChannel object at 0x2f1de088b0d0>, timeout = 60000.0

    async def get_msg(  # type:ignore[override]
        self, timeout: t.Optional[float] = None
    ) -> t.Dict[str, t.Any]:
        """Gets a message if there is one that is ready."""
        assert self.socket is not None
        if timeout is not None:
            timeout *= 1000  # seconds to ms
        ready = await self.socket.poll(timeout)
        if ready:
            res = await self._recv()
            return res
        else:
>           raise Empty
E           _queue.Empty

/usr/local/lib/python3.9/site-packages/jupyter_client/channels.py:315: Empty

During handling of the above exception, another exception occurred:

    def test_notebook_3():
        [notebook] = Path(__file__).parent.with_name("docs").glob("3-*.ipynb")
        if not is_backend_tested("torch"):
            pytest.skip()
>       render_notebook(notebook, replacements={})

tests/test_notebooks.py:66: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_notebooks.py:28: in render_notebook
    ep.preprocess(nb, {"metadata": {"path": str(filename.parent.absolute())}})
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:102: in preprocess
    self.preprocess_cell(cell, resources, index)
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:123: in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
/usr/local/lib/python3.9/site-packages/jupyter_core/utils/__init__.py:165: in wrapped
    return loop.run_until_complete(inner)
/usr/local/lib/python3.9/asyncio/base_events.py:647: in run_until_complete
    return future.result()
/usr/local/lib/python3.9/site-packages/nbclient/client.py:1005: in async_execute_cell
    exec_reply = await self.task_poll_for_reply
/usr/local/lib/python3.9/site-packages/nbclient/client.py:806: in _async_poll_for_reply
    error_on_timeout_execute_reply = await self._async_handle_timeout(timeout, cell)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x2f1de08838b0>, timeout = 60
cell = {'cell_type': 'code', 'execution_count': 6, 'id': '3987bbbc-ccc2-4374-8979-fd3fb29c4910', 'metadata': {'execution': {'...it=1.)),\n    torch.jit.script(ResMLP_Blocks3(128, dim=128, layerscale_init=1.)),\n]:\n    %timeit -n 10 y = layer(x)'}

    async def _async_handle_timeout(
        self, timeout: int, cell: NotebookNode | None = None
    ) -> None | dict[str, t.Any]:
        self.log.error("Timeout waiting for execute reply (%is)." % timeout)
        if self.interrupt_on_timeout:
            self.log.error("Interrupting kernel")
            assert self.km is not None
            await ensure_async(self.km.interrupt_kernel())
            if self.error_on_timeout:
                execute_reply = {"content": {**self.error_on_timeout, "status": "error"}}
                return execute_reply
            return None
        else:
            assert cell is not None
>           raise CellTimeoutError.error_from_timeout_and_cell(
                "Cell execution timed out", timeout, cell
            )
E           nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 60 seconds.
E           The message was: Cell execution timed out.
E           Here is a preview of the cell contents:
E           -------------------
E           ['x = torch.zeros([32, 128, 128])', 'for layer in [', '    ResMLP_Blocks(128, dim=128, layerscale_init=1.),', '    ResMLP_Blocks2(128, dim=128, layerscale_init=1.),', '    ResMLP_Blocks3(128, dim=128, layerscale_init=1.),']
E           ...
E           ['    torch.jit.script(ResMLP_Blocks(128, dim=128, layerscale_init=1.)),', '    torch.jit.script(ResMLP_Blocks2(128, dim=128, layerscale_init=1.)),', '    torch.jit.script(ResMLP_Blocks3(128, dim=128, layerscale_init=1.)),', ']:', '    %timeit -n 10 y = layer(x)']
E           -------------------

/usr/local/lib/python3.9/site-packages/nbclient/client.py:856: CellTimeoutError
------------------------------------------------------------------------------------- Captured log call -------------------------------------------------------------------------------------
ERROR    traitlets:client.py:845 Timeout waiting for execute reply (60s).
_____________________________________________________________________________________ test_torch_layer ______________________________________________________________________________________

    def test_torch_layer():
        if not is_backend_tested("torch"):
            pytest.skip()
        else:
            # checked that torch present
            import torch
            import torch.jit
    
            model1 = create_torch_model(use_reduce=True)
            model2 = create_torch_model(use_reduce=False)
            input = torch.randn([10, 3, 32, 32])
            # random models have different predictions
            assert not torch.allclose(model1(input), model2(input))
            model2.load_state_dict(pickle.loads(pickle.dumps(model1.state_dict())))
            assert torch.allclose(model1(input), model2(input))
    
            # tracing (freezing)
>           model3 = torch.jit.trace(model2, example_inputs=input)

tests/test_layers.py:219: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py:489: in _fn
    return fn(*args, **kwargs)
/usr/local/lib/python3.9/site-packages/torch/_dynamo/external_utils.py:17: in inner
    return fn(*args, **kwargs)
/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:806: in trace
    return trace_module(
/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:1102: in trace_module
    _check_trace(
/usr/local/lib/python3.9/site-packages/torch/utils/_contextlib.py:115: in decorate_context
    return func(*args, **kwargs)
/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:575: in _check_trace
    diag_info = graph_diagnostic_info()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def graph_diagnostic_info():
        mod_canonicalized = torch._C._jit_pass_canonicalize(traced_func.graph)
        torch._C._jit_pass_inline(mod_canonicalized)
        torch._C._jit_pass_erase_shape_information(mod_canonicalized)
        mod_str = str(mod_canonicalized)
        mod_str = re.sub(r"___torch_mangle_[0-9]+\.", "", mod_str)
        check_canonicalized = torch._C._jit_pass_canonicalize(check_mod_func.graph)
        torch._C._jit_pass_inline(check_canonicalized)
        torch._C._jit_pass_erase_shape_information(check_canonicalized)
        check_str = str(check_canonicalized)
        check_str = re.sub(r"___torch_mangle_[0-9]+\.", "", check_str)
    
        graph_diff_errors = None
        if mod_str != check_str:
            import difflib
    
            graph_diff = difflib.ndiff(
                mod_str.splitlines(True), check_str.splitlines(True)
            )
            graph_diff_errors = "Graph diff:\n" + indent("".join(graph_diff)) + "\n"
    
            for n_mod, n_check in zip(
                mod_canonicalized.nodes(), check_canonicalized.nodes()
            ):
                if str(n_mod) != str(n_check):
                    graph_diff_errors += "First diverging operator:\n"
                    node_diff = difflib.ndiff(
                        str(n_mod).splitlines(True), str(n_check).splitlines(True)
                    )
                    source_printout = (
                        "Node diff:\n" + indent("".join(node_diff)) + "\n"
                    )
                    mod_stack = n_mod.sourceRange()
                    if mod_stack:
                        source_printout += (
                            "Trace source location:\n" + indent(mod_stack) + "\n"
                        )
                    check_stack = n_check.sourceRange()
                    if check_stack:
                        source_printout += (
                            "Check source location:\n" + indent(check_stack) + "\n"
                        )
                    graph_diff_errors += source_printout
    
                    break  # For now, only print out the first pair of nodes that diverges
    
        tensor_compare_errors = None
        # Check Tensor-valued constant nodes
        for n_mod, n_check in zip(
            mod_canonicalized.nodes(), check_canonicalized.nodes()
        ):
            if n_mod.kind() != n_check.kind():
                break  # Graphs have already diverged
    
            if n_mod.kind() == "prim::Constant" and not (
                n_mod.mustBeNone() or n_check.mustBeNone()
            ):
                if not n_mod.hasAttribute("value"):
                    continue
                if n_mod.kindOf("value") != "t" or n_check.kindOf("value") != "t":
                    continue
    
>               mod_tensor_val = n_mod.t("value")
E               RuntimeError: required keyword attribute 'value' has the wrong type

/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:444: RuntimeError
===================================================================================== warnings summary ======================================================================================
../../../../../local/lib/python3.9/site-packages/jupyter_client/connect.py:22
  /usr/local/lib/python3.9/site-packages/jupyter_client/connect.py:22: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
  given by the platformdirs library.  To remove this warning and
  see the appropriate new directories, set the environment variable
  `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`.
  The use of platformdirs will be the default in `jupyter_core` v6
    from jupyter_core.paths import jupyter_data_dir, jupyter_runtime_dir, secure_write

tests/test_ops.py::test_repeat_array_api
  /usr/ports/misc/py-einops/work-py39/einops-0.7.0/tests/test_ops.py:540: UserWarning: The numpy.array_api submodule is still experimental. See NEP 47.
    import numpy.array_api as xp

tests/test_other.py::testmod
  /usr/local/lib/python3.9/site-packages/_pytest/python.py:198: PytestReturnNotNoneWarning: Expected None, but tests/test_other.py::testmod returned TestResults(failed=0, attempted=0), which will be an error in a future version of pytest.  Did you mean to use `assert` instead of `return`?
    warnings.warn(

tests/test_einsum.py::test_layer
  /usr/ports/misc/py-einops/work-py39/einops-0.7.0/einops/layers/_einmix.py:112: UserWarning: EinMix: weight has no dimensions (means multiplication by a number)
    warnings.warn('EinMix: weight has no dimensions (means multiplication by a number)')

tests/test_layers.py::test_reduce_imperative
  /usr/local/lib/python3.9/site-packages/numpy/core/_methods.py:53: RuntimeWarning: overflow encountered in reduce
    return umr_prod(a, axis, dtype, out, keepdims, initial, where)

tests/test_layers.py::test_torch_layers_scripting
  <unknown>:827: DeprecationWarning: invalid escape sequence \s

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== short test summary info ==================================================================================
FAILED tests/test_notebooks.py::test_notebook_3 - nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 60 seconds.
FAILED tests/test_layers.py::test_torch_layer - RuntimeError: required keyword attribute 'value' has the wrong type
============================================================= 2 failed, 101 passed, 2 skipped, 6 warnings in 178.36s (0:02:58) ==============================================================
Traceback (most recent call last):
  File "/usr/ports/misc/py-einops/work-py39/einops-0.7.0/test.py", line 100, in <module>
    main()
  File "/usr/ports/misc/py-einops/work-py39/einops-0.7.0/test.py", line 96, in main
    assert return_code == 0
AssertionError
*** Error code 1

Difference probably means that test dependency version specifications aren't accurate or not set in pyproject.toml

[arogozhnikov]

Difference probably means that test dependency version specifications aren't accurate or not set in pyproject.toml

einops has no dependencies. It works atop whatever framework user chose, but does not require either of those, so they will not appear in requirements/pyproject.

Einops' tests are meant to be run in CI: they confirm documentation is not outdated/broken (hence notebook testing) and that interaction with all frameworks is still correct.
These tests may demonstrate some errors in user setup or upstream frameworks - but unlikely to find any errors in einops - as einops is very platform-agnostic.

Exact issues you post above were already opened: #268 and #269 and those are not related to einops.