Comments (40)
from maintainer-tools.
Hello
@max3903
check if the versions has changed in the manifest of all the modules (better detection could be implemented later)
Can you help to understand this point?
Do you want to check all oca projects to identify if the version value is agree with guideline?
If is yes, then we can add in MQT lint checks. Example MQT/234
generate a setup.py and an egg file for each modified module, using the dependencies and extra-dependencies from the module manifest
Is this similar to OCA/maintainer-quality-tools#188 (comment)?
from maintainer-tools.
Hello @max3903
I'm not sure I understand the philosophy of what you want to achieve: so it seems you want a new egg for each git commit right? But in my opinion, the solution is rather to have the package manager to download from the git repo directly than generating fake release at every commit with no meaning at all. If the commit is the module version (why not), then the git branch should be used directly. This is achieved a bit with Buildout, but probably the biggest drains are the download of the dependencies recursively and from git too eventually. This is what I mean when I say we are lagging far behind stuff like Bundler. For me, having the buildout recipe use pip or a superior package manager seems like smarter thing worth spending efforts.
For me publishing a python module at every commit is not something worth dedicating resources. It could be at each git tag eventually but at each commit I think it's better to use the git repo directly. At every commit we just add an extra caching and level of indirection for the module, adding risk of errors in the chain with no benefit I can see.
Could you elaborate on you idea please? Is that for the module discovery by the end user? If yes I'm not convinced either as Pypi is not really much more user friendly than just a search on Github.
from maintainer-tools.
@max3903 ok so you sent a mail too. I m reading your email then...
from maintainer-tools.
@max3903 OK. So correct me if I m right: you don t want to publish the module at every commit, you only want to publish it if version changed based on what is written in openerp.py ? Then it makes sense indeed. I would be surprised if there is no such tools already to help doing that. Then we would just adapt them for the Odoo module meta information. So I suggest 1st looking for existing tools to automate the publishing of python modules on pypi.
from maintainer-tools.
Generally speaking, when I say we lack behind Bundler and this block from having 1 module = 1 repo, I'm mainly talking about:
- being able to specify released module version (pypi) or development version (git). You will typically want peace of mind for stable features of your deployment and choose released modules while you will want the most direct workflow possible for features you or the OCA are actively developping for the project, and then you will want to set the sources to git. So today Buildout can use the two but then yes publishing released modules as eggs would help closing the gap.
- being able to do this recursively for the dependencies of the modules without having to resolve everything manually. Here I think we have nothing. By getting stable modules published as egg, we could have the dependencies resolution for the stable modules, is this your idea?
As for having this to work with development versions, may be having pip to fetch modules in Buildout could get this to work?
from maintainer-tools.
@moylop260 I want to check if the version of a module is different between the source and destination branches. If it is then we generate a new package, otherwise nothing happen.
Yes it is similar. I understood that @dreispt has already done that part.
@rvalyi The script will be triggered by a merge, not a commit. I updated the description accordingly.
Regarding the Bundler comparison,
- let's package released versions first. Development ones can come later.
- pip, apt and yum can manage dependencies recursively. I want to be able to do:
# apt-get install odoo8-hr-commission
and it would download, install, configure and start PostgreSQL, Python, Python libraries for Odoo8, Odoo8, Python libraries for my modules and its dependencies, Odoo modules that hr-commission depends on and hr-commission module.
from maintainer-tools.
2015-09-02 16:47 GMT-05:00 max3903 [email protected]:
apt-get install odoo8-hr-commission
in favor of multi platform environments
pip install odoo8-hr-commission
Saludos Cordiales
CEO at Vauxoo https://www.vauxoo.com Odoo's Gold Partner.
[image: --]
Nhomar Hernandez
[image: http://]about.me/nhomar
http://about.me/nhomar?promo=email_sig
from maintainer-tools.
Hello, yes, I think like @nhomar, we should focus on pip packaging, not Debian. Distro packaging will be assuming too many debatable choice and won't be usefull for many of us. Once we have pip packaging it's really easy to have a full blown distro install if we need to.
@max3903 I know that pip install stuff recursively of course. My point is that at the moment when we specify a branch to use in Buildout, these dependencies are not managed recursively and this has to be done manually. Having pip modules will help somewhat in the sense that we will then be able to specify the wanted leaf modules as eggs to install in the buidout.cfg.
Now what you can do with a Gemfile and that would be really nice to have is to specify released packaged to download the dependency tree of modules but still be able to pin a determined module to a dev version with a git repo and get the dependency resolution automatically and pinnable. This is really something very usefull in Ruby projects that I'm lacking a lot in Odoo.
At some point it will be interesting to investigate this Bundler clone in Python https://github.com/Deepwalker/pundler But not to be taken lightly as a bunch of very smart people spent a lot of efforts to bring Bundler where it is today...
from maintainer-tools.
@nhomar bdist supports egg, rpm, deb and even msi and wininst.
https://docs.python.org/2/distutils/builtdist.html
so you can use what you want.
from maintainer-tools.
@rvalyi: download of the dependencies recursively
+1
I commented a current issue that we have by use N
platforms:
OCA/runbot-addons#38 (comment)
If our have centralize the installation of packages recursively this should be a good approach to create instance in all paltforms: runbot, docker, buildout, documentation, travis, manifest external_dependencies, manifest depends, erp_peek with a simple:
* install oca-odoo80-hr-commission
@max3903: ... and it would download, install, configure and start PostgreSQL, Python, Python libraries for Odoo8, Odoo8, Python libraries for my modules and its dependencies, Odoo modules that hr-commission depends on and hr-commission module.
+1
A little comment: database environment should be splitted because exists environment with an (or multiple) odoo server and a (or multiple) postgresql server.
from maintainer-tools.
I'm adapting my previous work on pipo
to fit this design.
I'm assuming that new packages will only be built on a version bump, and I can fit in version number not yet adapted to the OCA guideline format.
I think we can use odoo-hr-commission
instead of odoo8-hr-commission
.
Full version numbers will include the Odoo version, so if you want v7 you will require < 8.0
.
from maintainer-tools.
@max3903 Nice initiative!
For sure, packaging and releasing our addons as simple eggs with the dependency properly defined will ease the deployment.
@rvalyi I understand and share your concern on how to deal with module we want to get the dev version.
The problem mainly comes from that we don't have 1 repo = 1 module. With a 1 repo = 1 module it would be possible to easily get the dev version with pip
pip install -e git+https://github.com/acsone/anybox.recipe.odoo@master-add-release-recipe#egg=anybox.recipe.odoo
A lot of extension exists to specify in buildout that you want some 'dev' version of specific eggs in place of the official one. gp.vcsdevelop, mr.developer, ... Here it is my buildout, when I want to use the dev version of pycoda with l10n-belgium
[buildout]
parts = openerp
find-links = http://download.gna.org/pychart/
versions = versions
socket-timeout = 60
extensions = gp.vcsdevelop
vcs-extend-develop = git+https://github.com/acsone/pycoda@master#egg=pycoda
develop-dir=src
vcs-update = False
[openerp]
recipe = anybox.recipe.odoo:server
version = git https://github.com/odoo/odoo.git openerp 8.0
addons = git https://github.com/OCA/l10n-belgium.git addons-l10n-belgium 8.0
git https://github.com/acsone/l10n-belgium.git addons-l10n-belgium 8.0-add-account_bank_statement_import_coda
git https://github.com/OCA/bank-statement-import bank-statement-import 8.0
git https://github.com/OCA/account-financial-reporting account-financial-reporting 8.0
eggs = xlrd
pyPdf
pycoda
A solution (not tested) could be to create one egg-module-name
repository by addon in github only used to publish the addons as a 1 module = 1 repo. The structure could be structured as
egg_hr_timesheet_task
setup.py
setup.cfg
hr-timesheet #Git submodule -> https://github.com/OCA/hr-timesheet
hr-timesheet/hr_timesheet_task
hr-timesheet/..
....
hr_timesheet_task # symlink -> hr-timesheet/hr_timesheet_task
The goal is to automatically provide a repository structure compliant with setuptools/pip that could be used to get the 'dev' version of an addon.
generate a setup.py and an egg file for each modified module, using the dependencies and extra-dependencies from the module manifest
A mapping file (or ???) will be needed since extra-dependencies doesn't always refer to the name of a python package but the name of an 'importable' python module. For exemple, [account_bank_statement_import_coda](git https://github.com/acsone/l10n-belgium.git addons-l10n-belgium 8.0-add-account_bank_statement_import_coda) depends of pycoda but the extra-dependencies refers 'coda'
https://github.com/acsone/l10n-belgium/blob/8.0-add-account_bank_statement_import_coda/account_bank_statement_import_coda/__openerp__.py#L41
My 2 cents...
lmi
from maintainer-tools.
@lmignon I read it is possible to install a subdirectory of a git repo with pip although I never tried myself:
pip install -e git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path # install a python package from a repo subdirectory
If that works, no need for 1 module one repo
from maintainer-tools.
@max3903 @dreispt how about generating setup.py
from __openerp__.py
in the addon directory with a github webhook on each push? Then having a bot that publishes the eggs on pypi for those where the version has changed on the main branch. I see more flexibility if we decouple the generation of setup.py from the publishing on pypi.
Further, I'm wondering if it is possible to have the setup.py
entirely autogenerated in all cases. So having some mechanism to let authors hand-tune their setup.py
would be even more interesting and could cover cases such as requiring specific addons versions in dependencies or specific python dependencies that cannot be expressed correctly in __openerp__.py
(cfr the last § of #116 (comment)) .
from maintainer-tools.
@sbidoul A new thing learned. This kind of functionality if available makes the 1 module one repo 'obsolete'. The only things we would need are the setup files alongside openerp.py
from maintainer-tools.
@sbidoul Setup.py and actual building are implemented as separate functions.
from maintainer-tools.
@sbidoul @max3903 I did some tests to see if the subdirectory option works with pip. With a setup.py file alongside openerp.py, (see https://github.com/lmignon/l10n-belgium/tree/8.0-add-account_bank_statement_import_coda/account_bank_statement_import_coda), I'm able to run the following command.
./bin/pip install -e 'git+https://github.com/lmignon/[email protected]_bank_statement_import_coda#egg=openerp-account-bank-statement-import-coda&subdirectory=account_bank_statement_import_coda'
Nevertheless, at this stage one issue remains. I'm not able to import the package in my python interpreter.
from maintainer-tools.
@dreispt Do you have a branch we can follow and contribute ?
@ALL What do you think of using the packaging to differentiate the maturity of a module ?
For example with Pypi, we can set the development status to '4 - Beta' by default and if the module is not modified for one month, then we update the development status to '5 - Production/Stable'.
With deb package, we could push to the testing repository by default and after one month, push to the stable one.
Some of you wanted to create an incubator, this could be a great solution. I proposed one month, but this is open to discussion.
from maintainer-tools.
I'm not sure where the code should end up in the end, but in the meanwhile I'm working here: https://github.com/dreispt/pipo/tree/odoo2egg
from maintainer-tools.
@dreispt in maintainer-quality-tools. The repo is cloned in travis configuration, so we can trigger the script during a merge.
from maintainer-tools.
@dreispt If we pack our addons as eggs we must declare these eggs as namespace package under 'openerp.addons' In your script I don't find any trace of such declaration and when I 'pip' install the generated egg, the addon is not available in the python path with from openerp.addons. ....
Do I miss something?
from maintainer-tools.
Maybe that's because my scrip is using "odoo.addons". I guess that's a bad idea, and it should be "openerp.addons"...
The setup.py is defined here.
from maintainer-tools.
@dreispt Same result with 'openerp.addons'. But at this stage it seems that if we want to support addons packaged as eggs in odoo we need to monkey_patch Odoo itself to search for addons in the python path in addition to the addons_path
option. (https://github.com/odoo/odoo/blob/8.0/openerp/modules/module.py#L84)
from maintainer-tools.
FYI sys.path with addons_path param:
https://gist.github.com/moylop260/c18883487be31724f62a
2015-09-14 7:34 GMT-05:00 Laurent Mignon (ACSONE) [email protected]
:
@dreispt https://github.com/dreispt Same result with 'openerp.addons'.
But at this stage it seems that if we want to support addons packaged as
eggs in odoo we need to monkey_patch Odoo itself to search for addons in
the python path in addition to the addons_path option. (
https://github.com/odoo/odoo/blob/8.0/openerp/modules/module.py#L84)—
Reply to this email directly or view it on GitHub
#116 (comment)
.
Moisés López Calderón
Vauxoo - OpenERP's Gold Partner
Mobile: (+521) 477-752-22-30
Office: (+52) 477-773-33-46
web: http://www.vauxoo.com
twitter: @Vauxoo
@moylop260
hangout: [email protected]
from maintainer-tools.
Hi,
[updated to use the setuptools-odoo package]
I made my own try at packaging Odoo modules with setuptools. The general idea is to have a lean setup.py in each addon, next to __openerp_py, and do the rest (release, packaging and distribution) from that foundation.
Here is my prototype: https://gist.github.com/sbidoul/fb3d75de4ac0d44c04ed#file-setup-py
To test:
- make sure you have odoo/odoo#8758
- create and activate a virtualenv
- install Odoo: from the odoo main directory, run
pip install -r requirements.txt && python setup.py develop
- install
setuptools-odoo
withpip install git+https://github.com/acsone/setuptools-odoo.git --upgrade
- copy the
setup.py
from the link above to your addon directory, next to__openerp__.py
, and runpython setup.py develop
- start
odoo.py
as usual: your addon should be available without fiddling with addons-path
The other setup.py commands should work as expected (install, sdist, bdist...).
I'm looking forward to additional tests and feedback on this approach.
from maintainer-tools.
@sbidoul I've successfully tested your script and it's really promising. For the first time I'm able to install Odoo with an OCA addons (for my test I've used https://github.com/OCA/web/tree/8.0/help_online) by using only standard python command. Thank you for this great contribution that opens a lot of new opportunities.
how to handle dependencies (and in particular dependencies to other addons
that are not packaged with setup.py such as standard addons)?
If you package the prepare() method, you can include the list of names of addons provided by odoo. For these addons, the setup.py should only declare a dependency on 'odoo'.
how to handle python dependencies (wrong names in openerp.py)
A solution could be to rely on a requirement.txt file put next to the __openerp__.py
file.
from maintainer-tools.
@ALL do we really want to maintain additional files (requirements.txt, setup.py, spec file for rpm, control file for deb) when we could generate those files on the fly from the information in __openerp__.py
?
from maintainer-tools.
@max3903 I'm not sure we can generate the setup.py
from __openerp__.py
in all cases. That's why I propose an approach where the canonical setup.py
is trivial but can be manually hand-tuned if needed.
It would look like this https://github.com/acsone/setuptools-odoo#setuptools-odoo
Moreover, having setup.py in the addon directory enables classical python development workflow (eg deploying with python setup.py develop
).
from maintainer-tools.
I made good progress with my packaging attempt.
Together with @lmignon we reached the conclusion that making openerp.addons
a true namespace package is not possible without a relatively heavy patch which consists in removing all code from openerp/__init__.py
. This cannot be done by monkey patching. The good news is that odoo/odoo#8758 has been assigned so Odoo SA is looking into it. It will probably never make it in 8.0 nor 9.0, though.
However, following a suggestion from @lmignon I explored another approach which is giving very encouraging results and can be a very acceptable interim solution. In short it consists in packaging addons in another namespace package named odoo_addons
, which is never imported but only used to autodiscover addons. A simple monkey patch in the Odoo startup script is sufficient to make this work at runtime, so it is completely viable. It should work with 9 and probably 7 too with minor changes.
Here is how to test it [update 2015-10-05: updated to use setup.py develop instead of install for setuptools-odoo]:
- create and activate a virtualenv
- install Odoo: from the odoo source directory, run
pip install -r requirements.txt && python setup.py develop
- install
setuptools-odoo
withgit clone https://github.com/acsone/setuptools-odoo.git && cd setuptools-odoo && python setup.py develop
- create your
setup.py
in your addon directory with the following command:wget https://gist.github.com/sbidoul/b5049a5e12a824318091/raw/a875a7073cd1f5250d20b9356199777f9faabd8f/setup-odoo_addons.py -O setup.py
- run
python setup.py develop
in your addon directory - start Odoo with
odoo-server-autodiscover
and notice the addon is automatically discovered without using --addons-path
You can develop normally in the source directories of Odoo and your addon.
Then binary packaging and distribution can be done using the usual setup.py commands.
[update 2015-10-05] While this is already good for testing, note it is still work in progress. In particular, setuptools_odoo.prepare()
does not yet generate install_requires
nor a complete list of the static
folder content in package_data
.
@dreispt I had a look at pipo2eggs and started to re-use some good ideas from yours in setuptools-odoo (eg looking at several manifest names __odoo__.py, __openerp__.py, __terp__.py
). I plan to continue with package_data and dependencies.
Testers and feedback welcome!
from maintainer-tools.
@sbidoul Please feel free to take everything you like and leave behind everything you dislike. I'm sorry I didn't find the time to work more on this, but I believe it is in very good hands now.
from maintainer-tools.
@dreispt thanks. I hope you can find some time to test and provide ideas, because you have a lot of valuable experience in this matter.
from maintainer-tools.
@sbidoul I get an error message at step 3:
$ pip install git+https://github.com/acsone/setuptools-odoo.git --upgrade
Downloading/unpacking git+https://github.com/acsone/setuptools-odoo.git
Cloning https://github.com/acsone/setuptools-odoo.git to /tmp/pip-SUHrCu-build
Running setup.py (path:/tmp/pip-SUHrCu-build/setup.py) egg_info for package from git+https://github.com/acsone/setuptools-odoo.git
Could not find any downloads that satisfy the requirement odoo>=8 in /home/max/odoo/8.0/git/odoo (from setuptools-odoo==0.2.0)
Downloading/unpacking odoo>=8 (from setuptools-odoo==0.2.0)
Cleaning up...
No distributions at all found for odoo>=8 in /home/max/odoo/8.0/git/odoo (from setuptools-odoo==0.2.0)
from maintainer-tools.
@max3903 thanks for testing. I think the error comes from the fact that odoo is installed in develop mode and setuptools-odoo in install mode. I updated the instructions to install setuptools-odoo in develop mode too.
from maintainer-tools.
@sbidoul I've successfully tested your new proposal and it works like a charm... Although I am a little disappointed not to be able to package the modules in their real namespace, your solution seems to be the best compromise to move towards more standard in packaging and the distribution and of Odoo addons.
from maintainer-tools.
@sbidoul I have made it through all the steps. Works like a charm 👍
from maintainer-tools.
Thanks for the test @max3903. I'm making slow but steady progress. I wrote an update there: #156 (comment).
from maintainer-tools.
@sbidoul We are in need to package the modules using DEB, instead of pip. At the stage of maturity we're in now with pip, do you think it is feasible to convert to deb packages using? https://pypi.python.org/pypi/stdeb#bdist-deb-distutils-command, and perhaps publish to deb.odoo-community.org?
from maintainer-tools.
@jbeficent I'm curious about the use case...
in theory it should work, as the setup.py we have are spec compliant. I've never attempted that tough.
odoo-autodiscover might be tricky to get right in a .deb. It's not necessary in 11 tough. And it may work without in 10 in some situations.
Anyway, it's an interesting rabbit hole to explore.
from maintainer-tools.
from maintainer-tools.
Related Issues (20)
- pre-commit crashes in CI HOT 1
- Documentation for the best way to manage a high number of OCA repositories HOT 11
- I faced with error while using this module in pre-commit. HOT 1
- About squashing translation commits HOT 2
- Update our tooling to support Odoo 17 HOT 20
- Mention `odoo-module-migrator` in migration procedure wiki page HOT 1
- Mention `oca-port` in migration procedure wiki page HOT 6
- oca-gen-addon-readme: exits with no error if `description.rst` is not present HOT 14
- Update migration v17 Wiki HOT 2
- Clarify CONTRIBUTOR.md formatting HOT 20
- Wiki migration 17.0: New Settings format HOT 1
- oca-gen-addon-readme: stop emitting an XML declaration HOT 1
- Fail to run pre-commit on OCA project: ERROR No matching distribution found for setuptools_scm HOT 2
- Update wiki "Migration to version 16.0" with `fields_view_get` becomes `get_view` HOT 2
- Mention data-hotkey in the 15.0 migration guide
- delete
- WIKI for documentation HOT 6
- module 'github3' has no attribute 'authorize'
- copy_maintainers script fails syncing the big teams HOT 7
- [17.0][mig] partner_statement: Migration to 17.0 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from maintainer-tools.