Comments (26)
pip 19.0 supports PEP 517 now: https://pip.pypa.io/en/stable/news/.
from maturin.
I've finished sdist support in 78c516f and published a beta version with the (as far as I tested) complete PEP 517 support. Testing and experience reports with the new beta are welcome! The usage is documented in the readme, even though you need to specify pyo3-pack==0.7.0-beta.1
instead of just pyo3-pack
or use pip install --pre
.
from maturin.
I've tried implementing PEP 517 and also written a bit about that in the blog post, but it failed because pip hasn't implemented that interface yet. The current status is that you can build source distributions with disabled sdist feature, but can't install them. Writing and testing the PEP 517 interface is also blocked on pip, especially since I expect the implementation to differ from the PEP.
from maturin.
Yes I stumbled on your blog post after opening the issue 😉
Setuptools already has an implementation although I'm not sure if it's really used anywhere.
from maturin.
@konstin do you have any thoughts on how this should go? I see the pyo3-pack wheel just distributes a script right now, and this would need a library as well.
from maturin.
In the first step, we need to build an sdist. I had some support for this but dropped it. You can see the (likely reusable parts) at 96b0187#diff-3e8b6c785f497d88e8869ef9110c91ea and 96b0187#diff-b507bbbbb0351464abee9a55d2f759ed (and 96b0187 in general).
For the bindings actually implementing pep 517 we have afaik three options:
- Python calling pyo3-pack with subprocess.
Pro: Easy to implement. I'd say we can do everything in less than a hundred lines of python. We can even implementprepare_metadata_for_build_wheel
with a new subcommand and some minor refactoring.
Con: We need to ensure on all platforms and environments that we can find the pyo3-pack binary, that stdout/stderr work and that we can return the final sdist/wheel name. - cffi
Pro: None of the frictions of the python script
Con: cffi itself can't handle default arguments (because c can't), so we'd need to add a python script besides the cffi bindings, which pyo3-pack currently can't do. Worse is that there is afaik no proper error handling support - pyo3
Pro: Pure rust solution (as in a single shared lib / no .py files) and proper error handling
Con: We're stuck on nightly only pyo3 compiles on stable and need to publish a lot of wheels
cffi seems to be the worst option, but I don't have preference for pyo3 or subprocess. For subprocess, we can either add the script to the pyo3-pack package or make a miniature standalone package that depends on pyo3-pack. For the pyo3 version I think the best is to add an additional crate to this repo.
For all three solutions, we need to check what pip does when an exception is raised in one of the hooks. We also need to make check in pyo3-pack that cargo is installed and print an explanation otherwise (this can actually be done independent of pep 517).
from maturin.
I've implemented the wheel part of PEP 517 in 671fcdb, which also contains an untested sdist writer. The implementation uses the subprocess-solution.
I've filed tox-dev/tox#1344 for isolated builds with tox.
from maturin.
Hi @konstin!
Thanks for your work on this :)
I've been playing with https://github.com/indygreg/PyOxidizer, and I wanted to bundle an application that uses a pyo3-packed rust library in it's deps.
PyOxidezer doesn't support wheels, so it was a good time try out sdist.
Unfortunely pip is really bad when it comes to unicode (pypa/pip#6566)
I've created a pyproject.toml
file like:
[build-system]
requires = ["pyo3-pack>=0.7.0-beta.1", "toml"]
build-backend = "pyo3_pack"
and tried to pip install .
It seems to be this line:
https://github.com/PyO3/pyo3-pack/blob/master/pyo3_pack/__init__.py#L114
Which blows up because of the emojis in the output (windows doesn't like emojis yet).
WDYT?
from maturin.
Thank you for testing @omerbenamram! Seems I've got bitten by windows default encoding not being utf8 once again. That should be fixed in e7a6a09, and I've published 0.7.0-beta.2 with the fix.
from maturin.
I see building the .tar.gz
requires pyproject.toml
be packaged (via include
). If it's not specified in the manifest, pip complains setup.py can't be found. Perhaps you can either error on build, or include it automatically, to avoid publishing broken artifacts?
from maturin.
I notice that if target/wheels
doesn't exist, then pyo3-pack build
fails with:
🔗 Found pyo3 bindings
🐍 Found CPython 3.7m at python3.7
💥 pyo3-pack failed
No such file or directory (os error 2)
I need to first create it via build --no-sdist
.
from maturin.
@ijl Could you explain how you got a .tar.gz
? I thought I checked for a pyproject.toml
before building a source distribution.
Re the second issue: Good catch, I'll fix that.
from maturin.
@konstin pyo3-pack build --release --strip
, then pip install target/wheels/....tar.gz
. It would be nice to not need to build the wheel, too. Maybe --format bdist,sdist
(by default) or something instead of --no-sdist
?
from maturin.
target/wheels
missing should be fixed with 027c33e
from maturin.
Thanks for fixing that. Oh, and I mean there was a pyproject.toml
present, but my error was not having it include
in Cargo.toml
.
from maturin.
Oh, I see now. I've add a check in dd712c7. Thanks for reporting that!
from maturin.
@konstin It's now a UnicodeEncodeError
🤔
from maturin.
@omerbenamram I've tried another workaround for the encoding errors in #153, which makes it pass on machine (German / cp1252). It still prints a warning however about the output not being cp1252 and print surrogates instead of emoji, even though the terminal supports utf8.
It would be nice to not need to build the wheel, too. Maybe --format bdist,sdist (by default) or something instead of --no-sdist?
@ijl Could you explain the use case for only building an sdist?
from maturin.
@konstin I want to have a test in which I build an sdist and install it via pip. Building the project twice would be unnecessarily slow.
from maturin.
I see this as working well.
I think it would be prudent to make the documented usage build a non-manylinux release wheel with pinned versions:
[build-system]
build-backend = "pyo3_pack"
requires = ["pyo3-pack>=0.7<0.8", "toml>=0.10,<0.11"]
[tool.pyo3-pack]
cargo-extra-args = "--release"
manylinux = "off"
from maturin.
@konstin it works! sweet :)
from maturin.
@konstin I want to have a test in which I build an sdist and install it via pip. Building the project twice would be unnecessarily slow.
Is there a reason not to use tox with build isolation, which builds an sdist before building a wheel, or pip install .
? Or could you reuse the sdist from the initial project build? The functionality for just building a wheel is already implemented as part of the PEP 517 backend, but apparently there's no way that pip will give you an sdist. Maybe a pyo3-pack sdist
command would be the best solution for now.
I see this as working well.
I think it would be prudent to make the documented usage build a non-manylinux release wheel with pinned versions:
[build-system] build-backend = "pyo3_pack" requires = ["pyo3-pack>=0.7<0.8", "toml>=0.10,<0.11"] [tool.pyo3-pack] cargo-extra-args = "--release" manylinux = "off"
I've changed the implementation to always use release mode and opened a thread on the python discourse on the topic. I've also exposed the strip option. Having manylinux = "off"
is definitely a good idea; I've also added that.
from maturin.
I have it right now as pyo3-pack build
and pip install target/wheels/*.tar.gz
, which reuses the initial build's sdist artifact. I'm not using tox, and to run pip install .
would verify the repository has all the files it needs but not that the sdist does. pyo3-pack sdist
would be useful, but for me this is also good enough now.
from maturin.
I've created pypa/pip#6669 for solving this properly in pip.
from maturin.
beta.5 works perfectly for my use case, including pyo3-pack sdist
. Thank you for adding that.
from maturin.
Thank you all for the beta testing and the feedback!
The only thing missing is the tox integration, which fails due to a bug in tox; I've opened #176 to track that. For everything else that comes up, please open a new issue.
from maturin.
Related Issues (20)
- Add JSON schema generation for pyproject.toml with schemars for SchemaStore HOT 1
- maturin build & upload fails due to "malformed summary" HOT 20
- [Feature request] Add llvm tools in maturin image HOT 1
- Unable to release with crate meta description HOT 4
- Add pypy and maybe graalpy support to `generate-ci`
- Add gitlab ci support HOT 1
- Move chat to discord HOT 2
- Skip uv tests on FreeBSD
- AIX still not supported HOT 2
- Remove rust-cpython support
- Feature: Don't copy dependencies' DLLs by default HOT 16
- Better logging when copying DLLs
- Refactor `maturin develop` tests to only install necessary dependencies
- Building an x86 extension module against an x86 Python on an ARM macOS machine
- Extend `--skip-auditwheel` to support check mode
- Unpackaged `default-members` lead to build failures when using source distributions
- Scripts in python/project have to run `__init__.py` for --help, support including them outside directory
- `python -m maturin develop` doesn't have non-zero exit code on Windows HOT 6
- Unable to compile windows wheel with zstd as dependency
- "Failed to normalize readme path `README.md`" when it exists HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from maturin.