Comments (3)
Make sure that all of the Open edX packages get all of the appropriate wheels pushed to PyPI with each release. This will generally be a single universal wheel unless the package contains native code extensions.
For ~95% of Open edX repos, would this be as simple as setting {'bdist_wheel':{'universal':'1'}}
?
If so, would that on its own be worth doing, whether or not we go for the rest of the issue?
Create a repository and Docker image(s) with all of the dependencies needed to build the missing dependency wheels. This could also be used in development to try out new packages with missing wheel variants on PyPI, building them here and then supplying them to the dev environment's devpi instance.
I think I would want to look at a list packages that are missing dependency wheels so we can get an idea of how much build time & image size we'd save. If it'd be significant, then I'm in favor of this.
Some of the benefits could be obtained by instead/also using a builder pattern for Dockerfiles
FWIW, Tutor and most of its plugins use the builder pattern, yet I still want to bring the build time and image size down.
from edx-platform.
I think we'd want to start with a repo health check that checks PyPI for wheel availability, that would probably go a long way towards answering your questions. I suspect it's only around 2% of our package dependencies that have any native code extensions, which is why it feels silly to me that we're bloating our build process because of those.
from edx-platform.
For ~95% of Open edX repos, would this be as simple as setting
{'bdist_wheel':{'universal':'1'}}
?
We should already be doing this for most of them, we just need to be consistent about it. The need is for both configuration like https://github.com/openedx/django-user-tasks/blob/master/setup.cfg#L1-L2 and working automated release uploads like https://github.com/openedx/django-user-tasks/blob/master/.github/workflows/pypi-publish.yml . (And actually doing releases instead of linking to commits on GitHub.)
If so, would that on its own be worth doing, whether or not we go for the rest of the issue?
Yes, every package made available as a wheel that wasn't before should speed things up just a little bit more. A lot more in the case of the handful of (mostly upstream) packages with nontrivial native code that needs to be compiled.
I think we'd want to start with a repo health check that checks PyPI for wheel availability, that would probably go a long way towards answering your questions. I suspect it's only around 2% of our package dependencies that have any native code extensions, which is why it feels silly to me that we're bloating our build process because of those.
A couple of repo health checks could definitely help us be more consistent about this, especially if combined with edx/edx-arch-experiments#231 . And the main thing that's been forcing us to keep those build dependencies has been the lack of an official process for building those 3rd-party wheels and a public place to store them. I wanted to set that up for a long time, but it just never rose to the top of 2U's priority list.
Another reason to do this and extend it even to upstream packages that do have wheels on PyPI is protection against the sudden removal of dependencies from PyPI. This actually bit us a few times, and usually resulted in at least a half-day fire drill where a few people had to drop everything, fork the repo, get the fork released on PyPI, and untangle enough dependencies to make everything depend on the fork's name instead. (And deployments are impossible until all that gets sorted out.) That problem goes away if we maintain a package server that nobody else has permission to delete releases from.
from edx-platform.
Related Issues (20)
- Course assets should be served by a view rather than a middleware HOT 3
- Support static assets in the Learning Core runtime
- Update GitHub Actions
- Remove Coursegraph from edx-platform HOT 5
- edx-platform XBlock Extraction HOT 2
- Build out simple e2e Testing in CI HOT 1
- Build and Test both the Dev and Production Webpack Asset Process HOT 1
- Proctoring Technical Debt
- PollBlock Extraction HOT 2
- WordCloudBlock Extraction HOT 2
- AnnotatableBlock Extraction R&D
- Replace Paver quality and js_test commands HOT 2
- Discover courses Page has no "My Courses" and "Programs" option in header
- Sort out Problem component vs problem XBlock
- Dependency Dashboard HOT 4
- Backport Library listing performance fix into Redwood
- Group Community TA is added even though there are no divided discussions configured
- Certificates tab javascript constructor always fails in Instructor page when it is enabled
- Action Required: Fix Renovate Configuration HOT 2
- Assignment's Due Date translation - Only part of the string is being translated
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from edx-platform.