mika / jenkins-debian-glue Goto Github PK

Scripts for Debian package/repository handling inside Jenkins

License: MIT License

Makefile 0.64% Shell 74.65% Ruby 11.27% Puppet 13.44%

jenkins-debian-glue's Introduction

jenkins-debian-glue

Continuous Integration for Debian and Ubuntu made easy. jenkins-debian-glue allows you to build Debian and Ubuntu packages directly from the Jenkins Continuous Integration system.

Please head over to jenkins-debian-glue.org for setup instructions and documentation.

jenkins-debian-glue's People

Contributors

Stargazers

Watchers

Forkers

tclavier managedit formorer zeha df7cb xtaran credativ joshuatobin adeptg nexeck robin-wittler kartikm kasimon ximion dwink kepstin hopebaytech sylvestre simonhoenscheid mrud vorlock ydubreuil bmiklautz patrickshan hashar azatoth jvtm lurdan drucko jeekl sathieu rahedges padm sinfomicien bionix mkgg brain461 ballock laeti-tia zoobab linuxmaniac pombreda n054 xneelo cclausen webtrekk-kane tfriesner centuriondan disco-stu mhaberler sipseb rgrsoucy 2maz linuxmail mark0n atoz-chevara grapsus willdeberry paladox otgo wuerba ana mortenhartvig alitalia mariogrip sipwise rtloeffler guillemj rezib jgoerzen dpniel aldyl kienanstewart resharma mingshuang wjkennedy i-mscp-deb-packages darkmind ogsts spencerx cyrilbrulebois qweqweqwe45w35 peat-psuwit andrewharle leandron tubbz-alt tryweirder sergiik gorbyo

jenkins-debian-glue's Issues

Set gzip compression and use pigz instead of gzip

I would like to increase the compression used to pack the .deb file. How would I go about doing this? Also it would probably be interesting to use pigz instead of gzip for better performance.

+ in file names makes them un-apt'able

So I know how to override this behavior, but I'm looking for guidance. The automatic versioning in the scripts adds a '+' symbol to the file names, which is then inserted into the reprepro and all is well and fine, until I try apt-get the packages after an update. The '+' symbol gets translated by apt-get into unicode '%7e'. The only work around I've found that works is to provide my own options:

export DCH_EXTRA_OPTS="--new-version=1.0.${CODENAME}${BUILD_NUMBER}"

So why is the default '+' if apt-get translates the '+' on http get into %7e which means the webserver can't find the file any more.

if it helps I'm using hiawatha web server. I haven't tried to confirm this behavior on any other webserver.

Multi architecture build now fails because arch independent packages are built twice

Hello,
I got an issue which, I think, is caused by this commit: 649c708

I use jenkins-debian-glue to backport packages for i386 & amd64 architecture.
Basically, with -b option, architecture independant packages (arch: all) are built twice. But arch-all packages can only be imported once.

I've no problem with source package build since I use a separate job.

What solution do I have now ?

lintian-junit-report not shipped with the debian package

Extract from a failed task log I had after following the instructions here:

http://jenkins-debian-glue.org/getting_started/manual/

+ /usr/bin/lintian-junit-report koumbit-scripts_0.7+0~1356734459.16~1.gbpadb373.dsc
/tmp/hudson3398521647075764978.sh: 1: /usr/bin/lintian-junit-report: not found
Build step 'Exécuter un script shell' marked build as failure

Indeed, the script is not shipped with the Debian package. It is, however, in jenkins-debian-glue-buildenv-lintian, but the documentation doesn't reflect that.

At the very least documentation needs to be updated. Thanks for this great tool!

upload jenkins-debian-glue to the Debian project

The glue scripts are so useful they should be uploaded to the Debian project. The Debian Developers at Wikimedia foundation were wondering why it was not already available. We ended up rebuilding it and putting it on our own apt.

Let the world easily benefit from the very nice scripts!

"release" functionality is unclear

This documentation is quite unclear to me. I have tried setting up things as recommended, and when I set (say) release=unstable, I get the following tree structure:

/srv/repository/release/unstable/dists/unstable/...

I was expecting this to change only the latter part, ie the part after dists/. Now I have discovered that it is much simpler and cleaner to explicitely set REPOS in my jenkins job, and it does what I expect. However, I still wonder what release is for: does it build packages differently?

build for different distributions based on the same git-tree fails

From the console output:

File "pool/main/s/salt/salt-syndic_0.10.4-1+0~1351867589.12~1.gbp3e4f0a_all.deb" is already registered with different checksums!

Maybe i'm doing something wrong. But if i have a matrix build all resulting packages will have the same name, won't they?

Then a file is already present in the repository from the build before with different distribution.

cowdancer workaround bug

Cowdancer workaround should check for cowbuilder base distribution, not a host one. When building debian package on ubuntu using pbuilderrc COMPONENTS variables and not exporting COMPONENTS cowbuilder is unable to build base as universe component is unavailable.

Setting COMPONENTS variable fixes the issue but it makes pbuilderrc COMPONENTS variable almost useless.

more logical default for DEBEMAIL

example.org is designed for documentation, we can have a better default. I fixed this in our repo here:

git://git.koumbit.net/jenkins-debian-glue.git

.. with commit 002b65f7:

commit 002b65f76ccfda6298bbbe54e713972020faef6d
Author: Antoine Beaupré <[email protected]>
Date:   Tue Feb 19 17:00:51 2013 -0500

    use a more sensible default for DEBEMAIL

diff --git a/scripts/generate-git-snapshot b/scripts/generate-git-snapshot
index ec2e699..daaedae 100755
--- a/scripts/generate-git-snapshot
+++ b/scripts/generate-git-snapshot
@@ -8,7 +8,7 @@ if [ -r /etc/jenkins/debian_glue ] ; then
   . /etc/jenkins/debian_glue
 fi

-[ -n "${DEBEMAIL:-}" ] || DEBEMAIL="jenkins-debian-glue Autobuilder <[email protected]>"
+[ -n "${DEBEMAIL:-}" ] || DEBEMAIL="jenkins-debian-glue Autobuilder <jenkins@`hostname -f`>"
 export DEBEMAIL

 [ -n "${SOURCE_DIRECTORY:-}" ] || SOURCE_DIRECTORY='source' # where checkout of sources resides
diff --git a/scripts/generate-svn-snapshot b/scripts/generate-svn-snapshot
index ae8e43a..765c42c 100755
--- a/scripts/generate-svn-snapshot
+++ b/scripts/generate-svn-snapshot
@@ -11,7 +11,7 @@ if [ -r /etc/jenkins/debian_glue ] ; then
   . /etc/jenkins/debian_glue
 fi

-[ -n "${DEBEMAIL:-}" ] || DEBEMAIL="jenkins-debian-glue Autobuilder <[email protected]>"
+[ -n "${DEBEMAIL:-}" ] || DEBEMAIL="jenkins-debian-glue Autobuilder <jenkins@`hostname -f`>"
 export DEBEMAIL

 if [ ! -d source ] ; then

build-and-provide-package: unify $sources and $BASE_PATH

I'm not happy about the current state of $sources vs $BASE_PATH, they are not fully compatible as they are and there's no reason to keep support for both. I tend to drop $sources and just provide backwards compatbility for it through $BASE_PATH to avoid breaking existing installations.

Please install a sample /etc/jenkins/debian_glue

It would be nice to have a sample of /etc/jenkins/debian_glue
With the various possibilities commented.

Wrong XML export on the lintian test results

Switching for .dsc to .changes [1], /usr/bin/lintian-junit-report is not generating a correct xml file.

The test result page gives:

Error Message

postinst-must-call-ldconfig usr/lib/x86_64-linux-gnu/liblldb.so.1

Stacktrace

lldb-3.3: postinst-must-call-ldconfig usr/lib/x86_64-linux-gnu/liblldb.so.1

Standard Output

W: lldb-3.3: package-name-doesnt-match-sonames liblldb
N:
N: The package name of a library package should usually reflect the soname
N: of the included library. The package name can determined from the
N: library file name with the following code snippet:
N:
N: $ objdump -p /path/to/libfoo-bar.so.1.2.3 | sed -n -e's/^[[:space:]]SONAME[[:space:]]//p' | sed -e's/([0-9]).so./\1-/; s/.so.//'
N:
N: Severity: normal, Certainty: possible
N:
N: Check: binaries, Type: binary, udeb
N:
W: lldb-3.3: binary-without-manpage usr/bin/lldb-3.3
N:
N: Each binary in /usr/bin, /usr/sbin, /bin, /sbin or /usr/games should
N: have a manual page
N:
N: Note that though the man program has the capability to check for several
N: program names in the NAMES section, each of these programs should have
N: its own manual page (a symbolic link to the appropriate manual page is
N: sufficient) because other manual page viewers such as xman or tkman
N: don't support this.
...

[1] /usr/bin/lintian-junit-report *.changes > report/lintian.xml

Automated Git plugin installation broke recently

Quoting from jenkins IRC channel:

22:00 < micah> mika: oddly, i dont have the git configuration under /configure - even though I see the plugin is installed 22:02 < micah> hmm hudson.model.FreeStyleProjectjenkins-debian-glue-sourceCannotResolveClassException: hudson.plugins.git.GitSCM 22:03 < micah> weird, the git plugin doesn't exist in the jenkins plugin administration 22:10 < micah> mika: aha, after updating some plugins, the git one shows up now...

Another user also reported via PM that the automated jenkins-debian-glue installation failed regarding the Git plugin on Ubuntu server 12.04, so this needs further investigation what has been changed/broke with recent Git plugin.

provide a way to build "official releases"

The way this work is actually quite nice for jenkins and continuous integration: every time there's a change, a new package gets built and uploaded, and then we can go around and test it. Good. However, in a proper workflow, we would like to stabilise those builds at some point, say: lay down a git tag and then have the jenkins glue automatically build a properly named numbered package in a separate archive.

The way we are setup right now is that we have REPOS setup as a build parameter, so that people can upload packages to unstable, testing or stable. Unfortunately, it still means they have garbage-looking version numbers instead of the real thing.

What I would like instead would be to lock down the stable repo to a certain set of branches, and have the version number of those packages built directly from whatever is on debian/changelog and/or what git describe thinks of the repository.

Is that something that was considered?

possible chroot issues

My current dir looks like this:

--- debian/*
|
|
--- /*
|
|
--- /*
|
|
-- README.md
|
|
-- pom.xml

I have the source building per your instructions. ( I'm not sure if it's building successfully though.)
After that, the binaries job is triggered. That fails with the following: It's not completely clear what the error is from this output. Also, this output is not visible within the Jenkins console output. I piped the content of build-and-provide-package to a text file.

Do you have any idea what might be going wrong here?

Thanks!

when I set DEB_KEEP_BUILD_ENV=true, I get a bit more information.

I: Running cd tmp/buildd// && env PATH="/usr/lib/ccache:/usr/sbin:/usr/bin:/sbin:/bin" dpkg-buildpackage -us -uc -sa -sa -rfakeroot
dpkg-buildpackage: source package heston
dpkg-buildpackage: source version 1.0-1~20140203061331.51
dpkg-buildpackage: source changed by jenkins-debian-glue Autobuilder [email protected]
dpkg-buildpackage: host architecture amd64
I: user script /var/cache/pbuilder/build/cow.19331/tmp/hooks/C10shell starting
┌──────────────────────────────────────────────────────────────────────────────┐
│ FTBFS - problem with building Debian package │
└──────────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────┐
│ DEB_KEEP_BUILD_ENV is set to 'true', keeping build environment │
└──────────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────┐
│ FTBFS - copying build environment, this might take a while │
└──────────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────┐
│ FTBFS - execute the following commands to debug this issue: │
└──────────────────────────────────────────────────────────────────────────────┘
│
│ chroot /var/cache/pbuilder/build/debug.heston-1.0
│ su - pbuilder
│ bash
│ cd /tmp/buildd//debian/..
│ env PATH="/usr/sbin:/usr/bin:/sbin:/bin" dpkg-buildpackage -us -uc -nc -rfakeroot
│
┌──────────────────────────────────────────────────────────────────────────────┐
│ To invoke any hook use /tmp/hooks/NAMEOFHOOK │
└──────────────────────────────────────────────────────────────────────────────┘
I: user script /var/cache/pbuilder/build/cow.19331/tmp/hooks/C10shell finished
I: unmounting /var/cache/pbuilder/ccache filesystem
I: unmounting /var/cache/pbuilder/build filesystem
I: unmounting /tmp/apt-18881 filesystem
I: unmounting /tmp/adt-18881 filesystem
I: unmounting dev/pts filesystem
I: unmounting proc filesystem
-> Cleaning COW directory

At the command line, I "sudo su - jenkins" to become user jenkins and try running
chroot /var/cache/pbuilder/build/debug.heston-1.0
I get the error
chroot: cannot change root directory to /var/cache/pbuilder/build/debug.heston-1.0: Operation not permitted

I have set up everything correctly, per the debian-jenkins-glue documentation (with the jenkins file in /etc/sudoers.d/.

Any ideas?

Add a way to set Distribution in changes files

While I thought setting distribution variable would suffice, it didn't change the Distribution key in the changes file for me (nor in changelog). Trying to set distribution=staging, I 'm ending up with following: "bubba-backend (2.6~~rc10+0~~1364487385.9+staging~1.gbp483755) precise; urgency=low"

Cannot build Debian packages on Ubuntu hosts by default

I'm trying to build packages in Debian chroots on an ubuntu host. Unfortunately, build-and-provide-package, L181 breaks this by default.

A workaround is to set the COMPONENTS variable to "main" in the build step.

Unfortunately, this is also not documented

generate-git-snapshot: Cannot delete the branch which you are currently on

Hi,

I wanted to build Debian packages from the repository URL git://anonscm.debian.org/pkg-e/libs/eio.git. I am already building packages from svn repositories successfully, but this was my first try in building a package from a git repository.

I don´t know if I did not configure my job correct or this is an error in generate-git-snapshot. The source-job fails with the following error message:

...
+ git-buildpackage -nc --git-force-create --git-ignore-new -S -us -uc --git-verbose --git-builder=/bin/true --git-cleaner=/bin/true
gbp:debug: ['git', 'rev-parse', '--show-cdup']
gbp:debug: ['git', 'rev-parse', '--is-bare-repository']
gbp:debug: /bin/true [] []
gbp:debug: ['git', 'symbolic-ref', 'HEAD']
gbp:debug: ['git', 'show-ref', 'refs/heads/jenkins-debian-glue-buildbranch24042']
gbp:debug: ['git', 'show-ref', 'refs/heads/pristine-tar']
gbp:debug: ['git', 'ls-tree', 'HEAD']
gbp:info: eio_1.7.4.orig.tar.gz does not exist, creating from 'HEAD'
gbp:debug: Building upstream tarball with compression 'gzip -9'
gbp:debug: /bin/true ['-nc', '-S', '-us', '-uc'] []
+ cd ..
+ dpkg-source -I -b source
dpkg-source: info: using source format `3.0 (quilt)'
dpkg-source: info: building eio using existing ./eio_1.7.4.orig.tar.gz
dpkg-source: info: building eio in eio_1.7.4-1+0~1362644875.13+experimental~1.gbp68038d.debian.tar.gz
dpkg-source: info: building eio in eio_1.7.4-1+0~1362644875.13+experimental~1.gbp68038d.dsc
++ readlink -f debian/changelog
+ git checkout -- /var/lib/jenkins/jobs/eio-source/workspace/source/debian/changelog
+ git checkout HEAD
+ git branch -D jenkins-debian-glue-buildbranch24042
error: Cannot delete the branch 'jenkins-debian-glue-buildbranch24042' which you are currently on.
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording fingerprints
Recording test results
Finished: FAILURE

After looking in the generate-git-snapshot script I figured out that the last command 'git branch -D "$random_branch' caused the error. All other 'git branch' commands in the script do have a '|| true' behind them which prevents the job to fail if the command fails.

If I put a '|| true' behind the last 'git branch -D "$random_branch' the job succeeds and the packages get build.

Is this an error in my job or inside the script?

Thanks in advance!

using pbuilder (instead of cowbuilder)

Please provide options to switch between builders.

Though tt seems that j-d-g assumes build tool is cowbuilder only, I want to use pbuilder on slave kfreebsd node... (cowbuilder doesn't support that architecture)

Please add a target "make uninstall"

The title says all ;)

How to preserve tests/testsuite.log?

At Wikimedia, we are using jenkins-debian-glue to test Debian packaging.

The labs/toollabs repository creates two packages and uses autoconf as a testsuite. On failures, this leaves tests/testsuite.log for further debugging.

However, as the console log shows, jenkins-debian-glue removes the build environment before artifacts can be gathered as DEB_KEEP_BUILD_ENV is not set.

What is the best way to preserve tests/testsuite.log in this case? Setting DEB_KEEP_BUILD_ENV to true would accumulate unnecessary data.

On Travis CI, I used make check || cat tests/testsuite.log as a test; to achieve the same here, I would have to use override_dh_auto_test which I am hesitant to do.

specify for each job distribution to build

How can i define in job distirbution for building package?
Source code contains:

if [ -z "${distribution:-}" ]; then

but where i can specify needed distribution? (by default this is sid or current distribution... but if i want to build packages to squeeze or wheeze... ?

"sid" is hard coded in to /usr/bin/build-and-provide-package

In /usr/bin/build-and-provide-package there is the following:

if [ -z "${distribution:-}" ]; then
  echo "*** No distribution set, using sid for base.cow if it does not exist yet. ***"
  COWBUILDER_DIST="sid"
else
  echo "*** Using cowbuilder base for distribution ${distribution} ***"
  DIST="-${distribution}"
  COWBUILDER_DIST="${distribution}"
fi

This means that build fails on ubuntu if distribution is not set and the mirror you are using is ubuntu specific (or at least you are specifying an ubuntu subdirectory of it).
My hacky solution was to wrap that in:

if [ -z "${COWBUILDER_DIST:-}" ]; then
  if [ -z "${distribution:-}" ]; then
    echo "*** No distribution set, using sid for base.cow if it does not exist yet. ***"
    COWBUILDER_DIST="sid"
  else
    echo "*** Using cowbuilder base for distribution ${distribution} ***"
    DIST="-${distribution}"
    COWBUILDER_DIST="${distribution}"
  fi
fi

and specify COWBUILDER_DIST=precise in /etc/jenkins/debian_glue. However a better solution would probably be to not hard code the word "sid" there at all but specify the default through some configuration option.

Usage of branches doesn't increase package version

Commit 1418a32 causes debian/changelog to not be modified any longer and therefore not adjusting version number. This might not be what people would expect as you get "stable version" numbers like 1.0.1-1 instead of 1.0.1-1+0~~1346392967.44~~1.gbp015e05 when building a branch version.

Needs further discussion, recording the issue to not release current master as it is.

Use separate name for repo & distribution codename/suite

Currently, same variable $REPOS is used in /usr/bin/generate-reprepro-codename for both repo name and distribution codename in conf/distribution.
We should use $distribution if provided or the one used to generate package.

This would allow to have one repository per software packaged with all flavor in it, which should make repository management easier.

DEB_BUILD_OPTIONS is not forwarded to cowbuilder

In the "execute shell section", the two following declarations are failing:
export DEB_BUILD_OPTIONS="parallel=7"
DEB_BUILD_OPTIONS="parallel=7" /usr/bin/build-and-provide-package

Please add a way to ignore some packages from the removal

In the function "remove_missing_binary_packages" in the script "build-and-provide-package", it would be nice to be able to specify some packages which could be skipped from the removal.

For now, SKIP_REMOVAL disables all removals.
I would like a SKIP_PACKAGE_FROM_REMOVAL which would skip the package X from removal in the repository.
Thanks

dpkg-source can't find upstream tarball generated in gbp 'export-dir'

I have a repository with a debian/gbp.conf as:

[DEFAULT]
upstream-tree=branch
upstream-branch=upstream
debian-branch=master
tarball-dir = ../tarballs
export-dir = ../build-area

The repo is checked out under /source/

When running the generate-git-snapshot command a tarball is properly generated in /build-area/ :

+ git-buildpackage -nc --git-force-create --git-ignore-new --git-ignore-branch -S -us -uc --git-verbose --git-builder=/bin/true --git-cleaner=/bin/true
gbp:debug: ['git', 'branch', '--no-color']
gbp:debug: /bin/true [] []
gbp:debug: ['git', 'branch', '--no-color']
gbp:debug: Looking for orig tarball 'vips_7.32.3.orig.tar.gz' at '../tarballs'
gbp:info: Orig tarball 'vips_7.32.3.orig.tar.gz' not found at '../tarballs'
gbp:info: vips_7.32.3.orig.tar.gz does not exist, creating from 'upstream'
gbp:debug: ['git', 'ls-tree', 'upstream']
gbp:debug: Building upstream tarball with compression 'gzip -9'
gbp:debug: ['git', 'ls-tree', 'HEAD']
gbp:info: Exporting 'HEAD' to '/mnt/jenkins-workspace/workspace/operations-debs-vips-debian-glue/build-area/vips-tmp'
gbp:info: Moving '/mnt/jenkins-workspace/workspace/operations-debs-vips-debian-glue/build-area/vips-tmp' to '/mnt/jenkins-workspace/workspace/operations-debs-vips-debian-glue/build-area/vips-7.32.3'
gbp:debug: /bin/true ['-nc', '-S', '-us', '-uc'] []
gbp:debug: rm ['-rf', '/mnt/jenkins-workspace/workspace/operations-debs-vips-debian-glue/build-area/vips-7.32.3'] []

That generates the orig tarball in /build-area/vips_7.32.3.orig.tar.gz

Then quilt cleanup is run and then:

dpkg-buildpackage -uc -us -nc -d -S -i -I --source-option=--unapply-patches

That fails to find the original tarball which has been generated in /build-area

+ dpkg-buildpackage -uc -us -nc -d -S -i -I --source-option=--unapply-patches
dpkg-buildpackage: source package vips
dpkg-buildpackage: source version 7.32.3-1wmf3+0~20140213131304.3~1.gbpf0d6f3
dpkg-buildpackage: source changed by jenkins-debian-glue Autobuilder <[email protected]>
 dpkg-source -i -I --unapply-patches --before-build source
dpkg-buildpackage: warning: it is a bad idea to generate a source package without cleaning up first, it might contain undesired files.
 dpkg-source -i -I --unapply-patches -b source
dpkg-source: error: can't build with source format '3.0 (quilt)': no upstream tarball found at ../vips_7.32.3.orig.tar.{bz2,gz,lzma,xz}
dpkg-buildpackage: error: dpkg-source -i -I --unapply-patches -b source gave error exit status 255

I have:

git-buildpackage 0.5.32 (from Ubuntu Precise)
jenkins-debian-glue 0.8.0+0~~20140204210212.11~~1.gbpfb322f

dpkg-source includes git files

During generate-git-snapshot:

Lintian reports the git files were getting included. So I looked into the script and noticed that it used -i.git but some reason this doesn't exclude the git files, making the tarball potentially very large.

Debian dpkg-source version 1.16.8.

I worked around it by specifying a debian/source/local-options file which contained

tar-ignore=.git

which was mentioned in lintian.

chown is failing on sshfs or samba

generate-reprepro-codename is failing on:
${SUDO_CMD:-} chown $(id -un) "${REPOSITORY}"/conf

when "${REPOSITORY}"/conf is a remote FS (sshfs or samba for example), the chown will fail.
Since most of the time, it is not necessary, could you make sure it does not disable the build ?

How to use local (or remote) dependencies.

Hello

I am trying to build a package that depends on another package that is not in the vanilla distribution. Right now builds fail due to unmet dependencies.

The dependency is available in an exeternal apt repository, and I have set up jobs to build that dependency locally.

I am new to pbuilder and jenkins-debian-glue so i haven't worked out how to include this dependency in the chrooted environment.

Could you share some pointers about how to do this - is it a case of editing .pbuilderrc, or copying artifacts into the workspace?

Thanks

Ben

do not hardcode default reprepro parameter

I noticed the following parameters in the reprepro config:

DebIndices: Packages Release . .gz
DscIndices: Sources Release .gz

Those are the default settings, it seems to me that stating them explicitely is not required and only adds complexity to the configuration. The following configuration is also redundant, I believe:

Codename: unstable
AlsoAcceptFor: unstable

Always include $distribution in cow base name

My first jobs didn't have distributions configured, so I got two chroots base-i386.cow and base-amd64.cow, based on wheezy.
Now I have jobs with distributions, and I'm getting a new set of chroots: base-wheezy-i386.cow etc. with the same content.

The long name should be used even if the default distribution is assumed.

build-and-provide-package breaks build if distribution is not set

The changes in issue #89 cause the script to break if the distribution or architecture env vars are missing, which is the case in my (previously working) builds. This is because the script is run with set -u. I've provided a pull request that fixes this issue.

IRC channel #jenkins-debian-glue should be registered

Not really an issue with the project itself, but you are advertising #jenkins-debian-glue on freenode somewhere. It would be nice to register the channel with their ChanServ service so people dont ends up in an empty / topic less channel.

Should be as simple as:

/msg ChanServ REGISTER #jenkins-debian-glue Some description there

Make it clear how to build a different architecture on a separate slave

Currently the only supported multi-architecture build options are amd64 and i386, which are probably the most easiest architectures to target directly.

As I've not been able to setup and find information how to setup a system where we have jenkins on a amd64 build machine, and a separate armel builder machine (NAS type machine) which are to work as a slave building all non-arch-all packages to armel arch, I would like to ask if it's possible to make the instructions to include above scenario.

provide Debian packages of latest stable release

Thanks for suggestion to Stefan Schlesinger, it would be useful to provide a Debian repository containing just the stable releases of jenkins-debian-glue so people can use it without breaking stuff due to changes in HEAD.

svn2cl is no longer available in the unstable version of subversion-tools

Unlike the testing version, the unstable version of the subversion-tools no longer contains svn2cl:
http://packages.debian.org/sid/all/subversion-tools/filelist
This is causing jenkins-debian-glue to fail.

Workaround: downgrade to svn 1.6

let reprepro handle incoming itself

We are getting into a sensitive security issue here. We would like to allow people to create their own build jobs, but not be able to upload packages to all parts of a repository.

Basically, we want Jenkins to behave like a regular uploader: sign its packages and put them in incoming/, then reprepro can pick them up and decide, according to repo-wide policy, whether the upload is allowed.

Our workflow is that we like Jenkins to create automatic packages in testing and unstable, but we want a manual peer review before the package is pushed to stable.

The only way I can think of doing this is making the repository owned by a different user than the jenkins user and allow jenkins to upload only in ./incoming.

It also needs to sign its keys.

How does that idea sound to people here?

sign packages or --ignore=uploaders

If Jenkins tries to upload in an existing repository that have an Uploaders: configuration, it will fail because the keys are not signed. The simple fix is to pass --ignore=uploaders to reprepro include or to sign packages when building them.

Do not override PATH in /usr/bin/build-and-provide-package

In /usr/bin/build-and-provide-package, the first thing which is done is overriding the PATH variable. In some cases (using not packaged tools for example), the PATH might be overrided.

processincoming one package with multiple architectures, build on different nodes

At the moment i have the problem, that i am building a package for i386/amd64 (Same node) and armhf (different node).
I cannot include the armhf package, because the .dsc File was included with the i386/amd64 package, but the checksums differe, because the armhf package has been build on a different host. (All packages have the same version number).

Is there any solution for this?
At the moment i think that the only solution is to use includedeb and do not include the dsc/source files.

Any suggestions?

support debc inside j-d-g

The jdg-debc script I've at $customer should be shipped as part of j-d-g so it can be used as POST_BUILD_HOOK.

Clarify that Copy Artifact Plugin is the preferred choice

The "Copy Artifact Plugin" is the way to go, the documentation should clarify that and be adjusted accordingly.

upload packages to external debian repository

I wonder if it's possible to instead of creating a local reprepro repository, to upload the final packages to a external one, using dupload for example.

/usr/bin/generate-git-snapshot doesn't want to generate a snapshot... :(

Hi,

This looks like an excellent project!

I've gone down the manual install route because I'm trying to apply this to a jenkins server managed via Chef and I didn't want to install Puppet on there as well...

I'm running in to an issue having followed the instructions with my first job where it can't see the debian/changelog file when I run /usr/bin/generate-git-snapshot.

The relevant part of the console output is as follows:

D   README.md
- version_information
  ++ dpkg-parsechangelog --count 1
  ++ awk '/^Version/ {print $2}'
  tail: cannot open `debian/changelog' for reading: No such file or directory
  dpkg-parsechangelog: error: tail of debian/changelog gave error exit status 1
- ORIG_VERSION=
  ++ increase-version-number
  Usage: /usr/bin/increase-version-number <version_number>
- INCREASED_VERSION=
  Build step 'Execute shell' marked build as failure
  Archiving artifacts```

The strange thing is that debian/changelog does exist and if I SSH on to the jenkins server, I can move to the relevant workspace and cat debian/changelog without issues, however running the command direct (with the correct variables exported!) fails at exactly the same point!

Can anyone help with this?

Thanks,

Matt

Quilt change causes repo issues

The addition of the quilt processing now causes a ${PACKAGE}_source.changes file to be generated in addition to the usual ${PACKAGE}.changes file. Both changes files refer to the same diff.gz file.

In my existing build configurations that copy *.changes artifacts to be deployed to a remote apt repo, this causes a problem because first the ${PACKAGE}_source.changes file gets processed, which moves the diff.gz file. When the ${PACKAGE}.changes file is processed, the diff.gz (which was just deleted) no longer exists, so the repo package manager is unhappy because of the "missing file" and the package does not get put in the repo.

I'm not sure how to fix this, but the issue is being raised because the change that was made with quilt now breaks deployment jobs that previously worked. It's also not immediately apparent how to tell Jenkins to copy everything except *_source.changes.

0.7.0 broke builds

+ reprepro -v -A amd64 -b /srv/repository --waitforlock 1000 remove blah foo
/usr/bin/build-and-provide-package: line 435: reprepro: command not found

This happens in multiple builds that did work in 0.6.x. I'm not sure why. reprepro is not installed on the base build system, but it shouldn't have to be, right?

clarify 'Local subdirectory for repo (optional)' option for recent Jenkins versions

The "Source Code Management -> Branches to build -> Advanced" option is no longer valid for recent versions of Jenkins/its git plugin, instead it's available under "Additional Behaviours" nowadays, the docs and the scripts (where it's mentioned in comments) should clarify that.

build-and-provide-package: drop hardcoded "binaries"

For historical reasons build-and-provide-package has "some "$WORKSPACE"/binaries/ magic hardcoded, this should be cleaned up.