fedora-infra/bodhi

Bodhi is a web-system that facilitates the process of publishing updates for a Fedora-based software distribution.

Bodhi

Welcome to Bodhi, Fedora's update gating system.

Bodhi is designed to democratize the package update testing and release process for RPM based Linux distributions. It provides an interface for developers to propose updates to a distribution, and an interface for testers to leave feedback about updates through a +1/-1 karma system.

Bodhi’s main features are:

• Provides an interface for developers and release engineers to manage pushing out package updates for multiple distribution versions.
• Generates pre-release test repositories for end users and testers to install proposed updates.
• Gives testers an interface to leave feedback about package updates, leading to higher quality package updates.
• Announces the arrival of new packages entering the collection.
• Publishes end-user release notes known as errata.
• Generates yum repositories.
• Queries ResultsDB for automated test results and displays them on updates.

Documentation

You can read Bodhi's release notes and documentation online.

If you are interested in contributing to Bodhi, you can read the developer documentation.

IRC

Come join us on Libera! We've got two channels:

• #bodhi - We use this channel to discuss upstream bodhi development
• #fedora-apps - We use this channel to discuss Fedora's Bodhi deployment (it is more generally about all of Fedora's infrastructure applications.)

bodhi's Issues

BodhiClient 2.0

• Port to the new python-fedora OpenIdClient
• Get the current BodhiClient API workin with new Pyramid/Cornice APIs
• Port (and improve?) the old bodhi-client's fedmsg update functionality

Allow provenpackagers to add additional builds to an existing update

14:48  sgallagh> lmacken: RFE for bodhi: if an update for Package X exists, and I as a provenpackager want to add another package to that update (because they need to go stable together), I should be either able to edit the existing 
                 update or create a new one that consumes the existing one. Neither of these is currently the case :(

/cc @sgallagher

Make sure bodhi fedmsg publishes the update 'alias' for all relevant messages.

It is nice for constructing links that don't change.. so it would be nice if we could pass it along with all message types in addition to the update title.

Check if a user was removed from a group

In bodhi.security:remember_me, we check if the user is a member of 'important groups'. We should also check if they were removed from one of them since the last time they logged in.

Obsolete/eject older testing updates before mashing.

We currently hit race conditions when an update is on it's way to testing, and the maintainer submits a newer version during that time. This can lead to the older update getting ignored after it is pushed, and when the newer one goes to stable, the older one is left in testing with garbage-collected signatures.

So, post-tagging pre-mash we should add some additional logic to obsolete the stale testing updates.

Implement improved update creation form

Some new features that I would like to see:

• Auto-completing package name field
• Auto-populating list of candidate builds for the package
• Auto-populating list of potential bugs
• Flexible karma requirement configuration

Port style & templates

Bodhi's current theme is written using Kid, and uses a very old fedora style that no other apps use anymore.

I'm leaning towards a fresh start with https://github.com/ralphbean/bootstrap-fedora

Disable Karma Automatism on AutoQA Test Failures

There has been a recent proposal to disable Bodhi's autopush feature when the update in question has failed AutoQA tests.

This proposal was approved by FESCo (https://fedorahosted.org/fesco/ticket/1242) and now needs implementation.

Please add functionality to bodhi such that automatic pushing is disabled when AutoQA tests have failed.

Port from yum to dnf

We should port from the yum module to dnf.

It's currently used in a couple of different places (along with the unit tests):
https://github.com/fedora-infra/bodhi/blob/pyramid/bodhi/util.py#L231
https://github.com/fedora-infra/bodhi/blob/pyramid/bodhi/metadata.py

Changes to koji multicalls during the push process

We need to ensure that bodhi does not let koji tag multiple builds for the same package out of order during a push.

RIght now if a push contains 2 different builds for the same package, bodhi will order them and call the koji.addTag in the correct order, however the multicall does not have any guarantee in terms of execution ordering. So, we'll need to do multiple passes, tagging the old ones first, waiting for completion, then moving on.

Integrate resultsdb with the backend

This comes in two places I think:

• Check resultsdb at the time a user requests an update be pushed to stable
• Again at mash time when things are being pushed to stable.

We should check resultsdb for the update and have a list of resultsdb tasks that we keep in the database per update. If any of them are not PASSED or INFO, then the request should be denied and the user should be informed. For instance, we could not deny requests if rpmlint fails but we could deny requests if depcheck fails.

We need to keep the configurable list of required testcases per update so that we can handle the future where certain packages have automated test cases that no other packages have. Package X might want to require that their package-x-foobar testcase passes before anything goes to stable.

• Keep a configurable list of required testcases per update in the database (depcheck by default and other future ones).
• Eventually we'll need a way for package maintainers to add their own special test case requirements for their own packages. (Do we let them disable depcheck or other "site-required" ones?)

If depcheck has not even been run yet, perhaps we should deny as well.

For the check during the request-for-stable, we can just

• tell the user the request was denied in the web UI
• tell the user the request was denied via the bodhi client

For the check during mash time, it would need to do something more complicated like:

• remove the request
• take it out of the mash
• unlock the update
• and whatever other stuff needs to happen to keep things in a good state.

Do we stop the mash if the resultsdb is unavailable or giving 500s?

• We need a way to override it for all updates in the case where resultsdb has gone pear-shaped so we can still get updates out.
• We also need a way to override it per-update in the case where say we have a critical update like bash that needs to go out and it is failing some resultsdb check but we humans know better or something.

datagrepper feed improvements

Some suggestions from mizmo via @ralphbean

• deemphasize timestamp (smaller text)
• clickable updates
• make the update appear first and the user's name last (this will have to be done slowly over time in fedmsg_meta).
• use libravatar icons
• call to action "help test this"
• group messages into single items
  • packagedb-cli update (v 2.3.1-1) submitted to testing (fc19, fc20, el6) by pingou
  • maybe under this slug have a snippet of the change log?
• live update the datagrepper widget
• showcase people earning tester badges

[b2] stacktrace when loading bodhi1 database

$ PYTHONPATH=$(pwd) python tools/pickledb.py migrate bodhi-pickledb*

Loading pickled database bodhi-pickledb-130801.1918-F17-EOL
WARNING:fedmsg.crypto.x509:Crypto disabled ImportError('No module named M2Crypto',)
Traceback (most recent call last):
  File "tools/pickledb.py", line 446, in <module>
    main()
  File "tools/pickledb.py", line 441, in main
    load_sqlalchemy_db()
  File "tools/pickledb.py", line 214, in load_sqlalchemy_db
    r = Release(**release)
  File "<string>", line 4, in __init__
  File "/home/pingou/.virtualenvs/bodhi-python2.7/lib/python2.7/site-packages/SQLAlchemy-0.9.7-py2.7-linux-x86_64.egg/sqlalchemy/orm/state.py", line 260, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/home/pingou/.virtualenvs/bodhi-python2.7/lib/python2.7/site-packages/SQLAlchemy-0.9.7-py2.7-linux-x86_64.egg/sqlalchemy/ext/declarative/base.py", line 525, in _declarative_constructor
    (k, cls_.__name__))
TypeError: 'locked' is an invalid keyword argument for Release

Query API

We need to port & improve the query API to ensure that it allows users to query updates & buildroot overrides appropriately.

An example of one of the issues we're having with the current API is that certain parameters, like 'bugs' and 'date_modified', are mutually exclusive.

Traceback when loading pickled db

I hit this error when trying to import the currently posted pickledb. This could pose a problem when we try to migrate staging and ultimately production, so marking as 'critical'.

$ PYTHONPATH=$(pwd) python tools/pickledb.py migrate bodhi-pickledb*

No handlers could be found for logger "bodhi.buildsys"

Loading pickled database bodhi-pickledb-130801.1918-F17-EOL
WARNING:fedmsg.crypto.x509:Crypto disabled ImportError('No module named M2Crypto',)
Traceback (most recent call last):
  File "tools/pickledb.py", line 446, in <module>
    main()
  File "tools/pickledb.py", line 441, in main
    load_sqlalchemy_db()
  File "tools/pickledb.py", line 214, in load_sqlalchemy_db
    r = Release(**release)
  File "<string>", line 4, in __init__
  File "/home/threebean/.virtualenvs/bodhi-python2.7/lib/python2.7/site-packages/SQLAlchemy-0.9.8-py2.7-linux-x86_64.egg/sqlalchemy/orm/state.py", line 260, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/home/threebean/.virtualenvs/bodhi-python2.7/lib/python2.7/site-packages/SQLAlchemy-0.9.8-py2.7-linux-x86_64.egg/sqlalchemy/ext/declarative/base.py", line 525, in _declarative_constructor
    (k, cls_.__name__))
TypeError: 'locked' is an invalid keyword argument for Release

Assign the update id at submission time

...instead of when the update is first pushed to testing.

Gracefully handle attempts to resume a push when one doesn't exist

[bodhi.masher] ERROR 2014-03-11 20:12:20,694 Trying to resume a push, yet /mnt/koji/mash/updates/MASHING-FEDORA doesn't exist!
[cherrypy.msg] INFO 2014-03-11 20:12:20,766 HTTP: Page handler: <function _wrapper at 0x7f1c07078320>
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/CherryPy-2.3.0-py2.6.egg/cherrypy/_cphttptools.py", line 121, in _run
    self.main()
  File "/usr/lib/python2.6/site-packages/CherryPy-2.3.0-py2.6.egg/cherrypy/_cphttptools.py", line 264, in main
    body = page_handler(*virtual_path, **self.params)
  File "/usr/lib/python2.6/site-packages/turbogears/identity/conditions.py", line 262, in _wrapper
    return fn(*args, **kw)
  File "<string>", line 3, in mash
  File "/usr/lib/python2.6/site-packages/turbogears/controllers.py", line 361, in expose
    *args, **kw)
  File "<generated code>", line 0, in run_with_transaction
  File "/usr/lib/python2.6/site-packages/peak/rules/core.py", line 153, in __call__
    return self.body(*args, **kw)
  File "/usr/lib/python2.6/site-packages/turbogears/database.py", line 410, in so_rwt
    retval = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/turbogears/controllers.py", line 244, in _expose
    @abstract()
  File "<generated code>", line 0, in _expose
  File "/usr/lib/python2.6/site-packages/peak/rules/core.py", line 153, in __call__
    return self.body(*args, **kw)
  File "/usr/lib/python2.6/site-packages/turbogears/controllers.py", line 255, in <lambda>
    False, {}, args, kw))
  File "/usr/lib/python2.6/site-packages/turbogears/controllers.py", line 425, in _execute_func
    output = errorhandling.try_call(func, *args, **kw)
  File "/usr/lib/python2.6/site-packages/turbogears/errorhandling.py", line 77, in try_call
    return func(self, *args, **kw)
  File "<string>", line 3, in mash
  File "/usr/lib/python2.6/site-packages/turbogears/controllers.py", line 207, in validate
    return errorhandling.run_with_errors(errors, func, *args, **kw)
  File "/usr/lib/python2.6/site-packages/turbogears/errorhandling.py", line 118, in run_with_errors
    return func(self, *args, **kw)
  File "/usr/lib/python2.6/site-packages/bodhi/admin.py", line 165, in mash
    resume=resume)
  File "/usr/lib/python2.6/site-packages/bodhi/util.py", line 154, in new
    return f(*args, **kw)
  File "/usr/lib/python2.6/site-packages/bodhi/masher.py", line 88, in queue
    self._mash(self._queue.pop())
  File "/usr/lib/python2.6/site-packages/bodhi/masher.py", line 110, in _mash
    thread = MashTask(task[0], task[1], task[2], task[3])
  File "/usr/lib/python2.6/site-packages/bodhi/masher.py", line 216, in __init__
    self._lock()
  File "/usr/lib/python2.6/site-packages/bodhi/masher.py", line 271, in _lock
    raise MashTaskException(msg)
MashTaskException: Trying to resume a push, yet /mnt/koji/mash/updates/MASHING-FEDORA doesn't exist!

[turbogears.access] INFO 2014-03-11 20:12:20,768 ::1 - parasense [11/Mar/2014:20:12:20 +0000] "POST /updates/admin/mash HTTP/1.1" 500 791 "" "Fedora Bodhi Client/0.5.1"

package for epel7

I can build bodhi-server for Fedora 21 now 🎉

.. but epel7 needs a handful more deps to be branched. Namely:

• pcaro-hermit-fonts
• python-colander
• python-cornice rhbz ticket
• python-cryptography (stuck on this rhbz ticket)
• python-nose-cov
• python-simplemediawiki (requested epel7 branch here)
• python-waitress (requested epel7 branch here)

Buildroot override functionality

• Port /overrides JSON api
• Write new grid widgets
• Port bodhi.jobs automatic expiration task

Don't break existing links

When archiving an EOL release, it is like it suddenly disappears.

For example, Fedora 18 was just EOL-ed, so it is not visible any more at https://admin.fedoraproject.org/updates

However, direct links to updates should not become invalid as they now do.

I was looking at RHBZ #1049061 comment 4, whic points to https://admin.fedoraproject.org/updates/openssl-1.0.1e-37.fc18

That link doesn't work any more, Bodhi tells me :

The path /updates/openssl-1.0.1e-37.fc18 cannot be found

Now, it is certainly a good thing that we archive old releases, so they don't appear in the navigation sidebar, or in the research.

But direct links should really keep working forever.

Monthly Bundle support

get_updates_from_builds returns extra 'tg_flash' key

There's an extra tg_flash=None key returned from all get_updates_from_builds calls. Nothing serious, but it would be great if I didn't have to work around it :)

>>> from fedora.client import BodhiClient
>>> import bunch
>>> result = bodhi.send_request(method='get_updates_from_builds', req_params={'builds':'rpy-2.6.1-1.fc21'})
>>> print result.keys()
[u'tg_flash', u'rpy-2.6.1-1.fc21']

python-fedora-0.4.0-2.fc22.noarch

Bug Tracker abstraction

We should abstract our notion of a bug tracker in order to make it easy for non-Fedora users of bodhi to hook up their own !bugzilla tracker.

Provide an updated pickledb

I'm setting up my dev environment from scratch and it would be nice to have a more modern dump of the bodhi db if possible.

Port the Masher

• Make it a stand-alone daemon instead of running it from apache
• multithreading support
• Make the mashing lock release-specific, so we can push multiple Fedora releases at the same time
• Fix the queueing of pushes
• Verify that the message was sent by someone in releng
• Determine which updates to push
• Lock updates
• Lock repo, keep track of state (builds in the push, completed repos, etc)
• Perform gating based on Taskotron depcheck results
• Make sure things are safe to move? (ideally we should trust our own state)
• Update security bug titles
• Move build tags
• Expire buildroot overrides
• Remove pending tags
• Update comps.xml
• mash
• request_complete
• Add testing updates to updates-testing digest
• Generate/update updateinfo.xml
• Inject the updateinfo it into the repodata
• Sanity check the repo
• Flip the symlinks to the new repo
• Cache the new repodata
• Generate stable update notices
• Wait for the repo to hit the master mirror
• Send fedmsgs
• Update bugzillas
• Add comments to updates
• Email stable update announcements
• Email updates-testing digest
• Unlock repo
• Unlock updates
• See if any updates now meet the stable criteria, and set the request

Abstract the ACL system

We should abstract our use of the Fedora PackageDB for ACLs in order to make it easier for others to use bodhi without it. Maybe add a 'passive' acl_system that allows all authenticated users to submit/edit updates.

Support the concept of a 'stack'

It would be great to allow maintainers to define a list of packages as a 'stack' and let them create a new update with them.

When creating a new update for a stack, bodhi should pull in all of the builds tagged as candidates, and complain or fail if any are missing.

Limit buildroot override expiration dates

dgilmore> lmacken: I would like to limit the time that people can get overrides for 
dgilmore> lmacken: I have seen people do them for a year                            
dgilmore> | I think 4 weeks should be enough                                    
dgilmore> | that should let you build your updates and get them stable          

Fix bodhi1 for EPEL-7

There are a bunch of places in the master branch that look to see if the Release.name.startswith('EL'), and also assumes the tag will be el%d.

Lock updates / queue changes during a push

We need to make sure that during a push maintainers can't edit updates and cause all sorts of race conditions.

Port bodhi.mail to pyramid_mailer

We need to port our bodhi.mail module and have it utilize pyramid_mailer. It would also be nice to make the mail messages more configurable (maybe put them in the config file?).

Review UI for any blockers

We're getting close to when we said we would deploy bodhi2. We should step through the UI and make sure its all good to go. i.e., try all the forms, navigate between all the pages. Try resizing everything to test responsiveness.

Warn if maintainers are about to step on eachothers toes

I discussed this idea with Richard Hughes at Flock today.

If a package gets submitted as an update when there is already a different build of it in testing and the submitters are different, then we should display a warning.

Update/build comps for mash

Since we don't know when koji will be able to mash, we should be fully capable of doing it ourselves in bodhi2, and that entails dealing with the comps repo.

Support emoji

This would 🤘

Prioritize stable repos when pushing

Currently in bodhi2,1 thread per tag gets kicked off, but we should ensure that stable repos go out first, then testing.

Bodhi comments now display extraneous \n characters

Taskotron comments now display extraneous \n characters:

taskotron - 2015-06-03 20:17:35
Taskotron: depcheck test PASSED on i386. Result log:\nhttps://taskotron.fedoraproject.org/taskmaster//builders/x86_64/builds/76687/steps/runtask/logs/stdio\n(results are informative only)

https://admin.fedoraproject.org/updates/xorg-x11-drv-intel-2.99.917-11.20150603.fc22

@lmacken looked at this and says it is caused by commit b28f95f .

I have tried to traverse old comments in Bodhi and it seems that it broke around Dec 2014 to Jan 2015. See this update, the old comments are fine, the newer one is not:
https://admin.fedoraproject.org/updates/FEDORA-2014-16953/perl-YAML-Syck-1.28-1.fc20

Ensure notifications get sent out upon taskotron failure

Right now bodhi does not send emails for taskotron comments. We should check to see if it is a failure, and send them out (and emit fedmsgs?)

Feeds!

We need RSS/Atom feeds for all of the things.

Fix mirror polling bug with recent EPEL push

The EPEL masher is currently spinning, unable to find the repodata with the proper hash.

Ensure auto-obsoletion works as expected in bodhi2

There was a PR (#33) to improve the auto-obsoletion behavior in bodhi1 which got buried under other merges and code changes.

This ticket is just a reminder to go back, read that changeset, and make sure that bodhi2 behaves the way we wanted bodhi1 to in that respect.

Add a link to the project, or a link to report a bug

When I look at https://bodhi.stg.fedoraproject.org/ , I see no link to the project page (github page), or at least a link where to report a bug. Please add it, so that we increase the chance that bugs are reported, thanks.

Get python-fedora working with it new API

The fedora.client.BodhiClient needs to continue to work with our API.

Implement improved karma types

https://lists.fedoraproject.org/pipermail/devel/2010-March/134087.html
http://bochecha.fedorapeople.org/bodhi_karma_revamped/

Sign packages before a push

At the moment, signing packages and pushing updates are two, completely disjoint processes.

As a result, it is quite common to have a push failing because of unsigned packages, so you go and sign them, but then when you try to push there are new updates you need to sign, etc...

Bodhi could make signing a part of its push process, to avoid these problems.

However, signing requires user input (the passphrase to unlock the signing key), so that makes it a bit trickier.

Revoke stable request upon AutoQA failure if the stable karma has already been reached

Example: https://admin.fedoraproject.org/updates/FEDORA-2014-17194/wget-1.16.1-2.fc20

Upon AutoQA/Taskotron failure, we should revoke the stable request if the stablekarma threshold has already been reached.

Test suite errors when building on koji

https://kojipkgs.fedoraproject.org//work/tasks/8507/9488507/build.log

fedmsg: ERROR: Exception in MasherThread(f17-updates-testing)
Traceback (most recent call last):
  File "/builddir/build/BUILD/bodhi-2.0/bodhi/masher.py", line 265, in work
    self.send_notifications()
  File "/builddir/build/BUILD/bodhi-2.0/bodhi/masher.py", line 635, in send_notifications
    agent=os.getlogin(),  # Should almost always be "masher"
OSError: [Errno 25] Inappropriate ioctl for device

Assorted UI improvements

• Some tables look weird when squashed horizontally
• linkify #bugs and @people (this is done with "Fedora Flavored Markdown"
• user view comments list should say which update each comment is on but when viewing an update, the comments list doesn't need to redundantly state which update they are on. (the new comment card fragments make this unnecessary...)

Push status page

It would be cool to have a page that displays the status of the updates push process.

don't allow updates to be stuck "between" repos while in "freeze"

Currently it is possible while in freeze to have update/s which are "stuck" between repos.

• Maintainer submits an update.
• Update receives enough karma to get to stable before it get pushed to 'updates-testing'
• Update don't appear in any repo untill freeze is over and it gets pushed to stable repos ('fedora'/'updates')

I think this will improve security (all updates are available in freeze) without manually downloading them from 'koji', also this helps with testing because not everyone downloads packages manually and more people testing things is better.