Comments (6)
I would personally not be in favor of making a split manifest functionality without additional features(like workspaces) that can benefit from this as it would add additional complexity to both the pixi code base and when misusing this, also to the projects them self.
from pixi.
Although we don't have a design for this laying around the question would be if that fixes the issue for you? If you think it will I would like to do a full write-up.
Yeah, workspaces very much sounds like a direction that would work well for us. This one definitely isn't urgent at all but came up when I was talking to Tim and Bas last week so I opened an issue to keep from forgetting about the idea.
from pixi.
Interesting. I really appreciate the possibility to keep all my env-related stuff in one file instead of having multiple environment.yml's like I did with conda before. Moreover, possibility of bundling Pixi into pyproject.yml is also nice for python-only projects.
Would be interesting to see work on such a large project
from pixi.
Yes, lovely stuff, and could potentially help reduce duplication in matrix use cases, and enable monorepos and skeleton-based tools to get spun up on pixi
more quickly.
As long as there is a way to get a single inspectable file back out, fully realized, things are pretty ok. Otherwise, behold the nightmare that is GitHub actions uses
or, worse still, the GitLab pipeline spaghetti: can't know until you try, and have already spun up 20 containers!
concatenation
Yep, TOML is quite resilient to cat
. I've explored not actually checking in pixi.toml
and having a wrapper script which doesn't need pixi
... cleaned up a bit, my "run"-focused pxr
:
#!/usr/bin/env bash
cat pixi.base.toml src/*/pixi.partial.toml > pixi.toml
set -eux
for task in "$@"; do
pixi run $task
done
Writing those files can be a bit hard to reason about, however, as all those paths need to be pixi.toml
-relative.
include
As with the inputs
/outputs
feature, task
could provide some inspiration with its top-level includes
which can be a simple path string or a rich object.
Typographically, this could be adaptable here, assuming a reserved namespace delimiter, e.g. :
:
[project]
includes.docs = "docs/pixi.toml"
includes.atest = { manifest = "atest/pixi.toml", features = ["deps-atest"] }
[project.includes.ff]
manifest = "atest/pixi.toml"
features = ["tasks-atest"]
env = { BROWSER = "firefox" }
[project.includes.cr]
manifest = "atest/pixi.toml"
features = ["tasks-atest"]
env = { BROWSER="chromium" }
[tasks.all]
depends_on = ["docs:build", "ff:robot", "cr:robot"]
inputs = [
"build/docs/.buildinfo",
"build/reports/robot/chromium",
"build/reports/robot/firefox",
]
[environments]
docs = ["docs:deps-docs", "docs:tasks-docs"]
atest = ["atest:deps-atest", "ff:tasks-atest", "cr:tasks-atest"]
Dangerously close to going down the spaghetti route of "post-install hooks": if an include
could be declared as coming from an environment, all of a sudden partial pixi
workflows could be packaged like proper software:
[project.includes.ex]
manifest = { environment = "default", path = "share/pixi/tasks/example.com/pixi.toml" }
env = { EX_PROJECT_NAME="my-project" }
[feature.baseline.dependencies]
example-com-pixi-tasks = "2024.04.*"
Having been down this road, it gets pretty hairy if a child wants to reference its parent files, features, tasks, etc. This could either be banned (which would limit the overall functionality) or would require some more syntax to make an includable partial contract more explicit and self-documenting, e.g.
[partial]
parent = "root"
# just use JSON Schema: don't get fancy
[partial.env.EX_PROJECT_REPO]
description = "the location of repo"
[partial.env.EX_PROJECT_NAME]
description = "a lovely project name according to your friendly example.com company policies"
pattern = "[a-z][a-z/d\\-]+"
from pixi.
Looking at Rerun's pixi.toml
, this would definitely benefit from a split.
Since the pixi project started we've thought about a "Workspace" feature. Which would allow you to have multiple manifest files which can be included in the main manifest file. Taken huge inspiration from https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html and https://docs.npmjs.com/cli/v7/using-npm/workspaces
Although we don't have a design for this laying around the question would be if that fixes the issue for you? If you think it will I would like to do a full write-up.
Quick summary on the workspace feature:
- Having member packages in subfolders
- Having a member environments specifically for the member but shared dependencies.
- Running tasks from the members from the root entrypoint.
- Depending on members as actual packages which can be shared and cached during development. (requires
pixi build
)
I know we're being credited for not splitting our configuration into multiple files which is something I take seriously as well.
from pixi.
With a SAT hammer sitting around, it's nice if everything is a package nail, but needing build a
as an extra, hidden depends_on
of install:b
to be used by run c
would feel pretty heavy for something that is a another aspect of a project (a la docs, heavyweight tests). Further, it seems that anything touching an env (and presumably the pixi.lock
) has the potential to degrade CI cacheability until something can generate bit-for-bit fully-reproducible .conda
artifacts (which i have yet to see).
from pixi.
Related Issues (20)
- Pixi does not freeze version number for PyPI package installs HOT 2
- `pixi install` fails when using `detached-environments = true` and `.pixi` doesn't exist yet
- `pixi run` warning about project manifest should only show up if `PIXI_PROJECT_MANIFEST` and local `pixi.toml` are different HOT 1
- `[activation.env]` variables are not expanded HOT 1
- `pixi install` / `pixi run` doesn't handle multiple concurrent invocations very well HOT 1
- CUDA virtual package is not detected when using brew version of pixi HOT 2
- Pixi successfull install message missing a newline
- tasks not found when running from osx-64 environment on osx-arm64 machine HOT 1
- Cannot resolve `__osx >=11.0` requirement on `osx-64` HOT 2
- Possibitlity to prioritize environment when running tasks locally
- Quoting not passed to subcommand HOT 2
- `pixi project version` doesn't work correctly in `pyproject.toml`
- `--tls-no-verify` doesn't work HOT 1
- pixi shell not activated due to read-only variable HOT 2
- pypi source distribution chosen over binary wheel HOT 3
- pixi shell broken on Windows if Env vars contains quotes HOT 1
- `pixi add` is really slow in some scenarios HOT 2
- More sized-efficient pixi.lock (with care to be optimized for git packfile). HOT 2
- After pixi update, cannot launch pixi shell anymore HOT 1
- `pixi update` not updating dependency but `pixi add ` does HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pixi.