Switch to "pyproject.toml" and src layout: Installing from source is not easy on WSL Linux and MacOS

[omid-b]

Switch to "pyproject.toml" and src layout: Installing from source is not easy on WSL Linux and MacOS

[jcapriot]

Thanks for the detailed information here! I see two problems here, the first will be reliable fixed by moving to a pyproject.toml file.

1. cython, numpy, and setuptools_scm are all required for building discretize from source and is not installed in a fresh environment by using python setup.py install. (But they should be when doing pip install . in a fresh environment, which doesn’t appear to be something that was listed here). We should be more clear that using python setup.py is not a recommended path to installation, and pip install would be better.

2. as you hinted setuptools_scm is unable to determine the package version. It uses info from got tags to determine the version automatically instead of it being hardcoded in. It has a hint suggestion in its error report about what is possibly causing this issue:

LookupError: setuptools-scm was unable to detect version for /mnt/c/Users/baghom/Desktop/discretize-main.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

what method did you use to obtain the source for your installation?

[omid-b]

Thank you very much for your speedy reply!

For the example above, I downloaded the package from main repository by clicking on CODE -> Download Zip , as this is the method most people would go for...

Since I have already forked discretize, I'm gonna try a similar demonstration using the cloning method below:

$ git clone [email protected]:omid-b/discretize
Cloning into 'discretize'...
remote: Enumerating objects: 34285, done.
remote: Counting objects: 100% (3976/3976), done.
remote: Compressing objects: 100% (1077/1077), done.
remote: Total 34285 (delta 2964), reused 3835 (delta 2879), pack-reused 30309
Receiving objects: 100% (34285/34285), 54.92 MiB | 514.00 KiB/s, done.
Resolving deltas: 100% (24698/24698), done.

$ cd discretize/

Create a fresh env ...

$ virtualenv env --python=3.10 && source env/bin/activate
created virtual environment CPython3.10.12.final.0-64 in 3058ms
  creator CPython3Posix(dest=/mnt/c/Users/baghom/Desktop/Tests/env, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/omid-b/.local/share/virtualenv)
    added seed packages: pip==22.0.2, setuptools==59.6.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Now install ...

$ pip install .
Processing /mnt/c/Users/baghom/Desktop/Tests/discretize
  Preparing metadata (setup.py) ... done
Collecting cython>=0.29.31
  Using cached Cython-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
Collecting numpy>=1.20
  Using cached numpy-1.25.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
Collecting scipy>=1.8
  Using cached scipy-1.11.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
Collecting setuptools_scm
  Using cached setuptools_scm-7.1.0-py3-none-any.whl (43 kB)
Collecting tomli>=1.0.0
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting packaging>=20.0
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting typing-extensions
  Using cached typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Requirement already satisfied: setuptools in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from setuptools_scm->discretize==0.1.dev5527+g58043b9) (59.6.0)
Building wheels for collected packages: discretize
  Building wheel for discretize (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/mnt/c/Users/baghom/Desktop/Tests/discretize/setup.py", line 80, in <module>
          from Cython.Build import cythonize
      ModuleNotFoundError: No module named 'Cython'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for discretize
  Running setup.py clean for discretize
Failed to build discretize
Installing collected packages: typing-extensions, tomli, packaging, numpy, cython, setuptools_scm, scipy, discretize

[omid-b]

$ pip install cython
Requirement already satisfied: cython in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (3.0.2)

This is weird!

[omid-b]

Following #332 , I downgraded my cython

$ pip install cython==0.29.35
Collecting cython==0.29.35
  Using cached Cython-0.29.35-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
Installing collected packages: cython
  Attempting uninstall: cython
    Found existing installation: Cython 3.0.2
    Uninstalling Cython-3.0.2:
      Successfully uninstalled Cython-3.0.2
Successfully installed cython-0.29.35

try again...

$ pip install .
Processing /mnt/c/Users/baghom/Desktop/Tests/discretize
  Preparing metadata (setup.py) ... done
Requirement already satisfied: cython>=0.29.31 in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from discretize==0.1.dev5527+g58043b9) (0.29.35)
Requirement already satisfied: numpy>=1.20 in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from discretize==0.1.dev5527+g58043b9) (1.25.2)
Requirement already satisfied: scipy>=1.8 in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from discretize==0.1.dev5527+g58043b9) (1.11.2)
Requirement already satisfied: setuptools_scm in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from discretize==0.1.dev5527+g58043b9) (7.1.0)
Requirement already satisfied: packaging>=20.0 in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from setuptools_scm->discretize==0.1.dev5527+g58043b9) (23.1)
Requirement already satisfied: typing-extensions in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from setuptools_scm->discretize==0.1.dev5527+g58043b9) (4.7.1)
Requirement already satisfied: setuptools in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from setuptools_scm->discretize==0.1.dev5527+g58043b9) (59.6.0)
Requirement already satisfied: tomli>=1.0.0 in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from setuptools_scm->discretize==0.1.dev5527+g58043b9) (2.0.1)
Building wheels for collected packages: discretize
  Building wheel for discretize (setup.py) ... done
  Created wheel for discretize: filename=discretize-0.1.dev5527+g58043b9-cp310-cp310-linux_x86_64.whl size=4691875 sha256=60e3e11dfbcc8d6bc25777bf9eda2704321ede39e456811fdcb553218c87acd3
  Stored in directory: /tmp/pip-ephem-wheel-cache-0yplpgrx/wheels/fe/72/26/d10ab46336913418a55201c5aadc78a51d73715b4e0c9e47ca
Successfully built discretize
Installing collected packages: discretize
  Attempting uninstall: discretize
    Found existing installation: discretize 0.1.dev5527+g58043b9
    Uninstalling discretize-0.1.dev5527+g58043b9:
      Successfully uninstalled discretize-0.1.dev5527+g58043b9
Successfully installed discretize-0.1.dev5527+g58043b9

Nice! Let's test it now!

$ python -c "import discretize"
cannot import name 'interputils_cython' from 'discretize._extensions' (/mnt/c/Users/baghom/Desktop/Tests/discretize/discretize/_extensions/__init__.py)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/mnt/c/Users/baghom/Desktop/Tests/discretize/discretize/__init__.py", line 42, in <module>
    from discretize.unstructured_mesh import SimplexMesh
  File "/mnt/c/Users/baghom/Desktop/Tests/discretize/discretize/unstructured_mesh.py", line 7, in <module>
    from discretize._extensions.simplex_helpers import (
ModuleNotFoundError: No module named 'discretize._extensions.simplex_helpers'

Trouble! it seemed like it was installed properly, but it did not build the extensions apparently!

[jcapriot]

And if you change directories out of the source folders are you able to?
It's clearly trying to import the local version in your folder structure and not the one from site-packages.

[omid-b]

Yes as you mentioned, it tries to read from the source folder rather than my env.

Here is the hotfix (but I still insist installation of discretize should not involve this many troubles!)

cd into source code dir and then copying the build files here...

$ cd /mnt/c/Users/baghom/Desktop/Tests/discretize/discretize/_extensions/
$ cp ../../../env/lib/python3.10/site-packages/discretize/_extensions/*.so ./
$ python -c "import discretize"
$ python -c "import discretize._extensions"
$ python -c "import discretize.tests"

No errors now...

[jcapriot]

But if you're doing that, why not just do an editable install? pip install -e .? That would be a better practice than copying things back from site-packages if what you want to do is to do a local install?

[omid-b]

Yes I could... but how about others who are not python experts (not that I am an expert; not at all!), and just want to install and use SimPEG/Discretize? I can make it fixed the if owners/main-developers accept my contribution...

[jcapriot]

I'm just trying to make sure we understand the root cause of the issue that you are encountering, which is as I said,

Cython, numpy, and setuptools_scm are required for building the package, but only numpy is a run-time requirement. The current setup.py configuration does not successfully install the needed files when this occurs.

[omid-b]

I understand your point, but still believe that it could be easily fixed to make the process more hassle-free...

An easy fix perhaps would be to modify requirements.txt as follows:

cython==0.29.35
numpy
setuptools_scm
-e .

But this hotfix is very temporary and things will go wrong again soon. Moving toward a src-layout and pyproject.toml is way to go...