pgbuildfarm/client-code

PostgreSQL Build Farm Client

client-code's Issues

Improve handling if https://buildfarm.postgresql.org/branches_of_interest.txt can't be fetched

Occasionally https://buildfarm.postgresql.org/branches_of_interest.txt can't be fetched (say because the machine is rebooted), and then everyone gets a bunch of spam from cron because

        die "getting branches of interest ($url)" unless $branches_of_interest;

At the very least the error message should be better, but it'd be even better if we could use a stale file, or retry after a delay, or ...

libsepol INFO messages

Sometime in the past year the cleanup for TestSepgsql.pm started emitting a message to stdout for every run.

This seems to be because of a change in libsepol which affects this command: "semodule -r sepgsql-regtest". I looked at the underlying source and there seems to be no way to silence the INFO message. See:
libsemanage/src/direct_api.c:semanage_direct_remove_key
and
libsepol/src/debug.c

I have locally applied the following patch:

*** PGBuild/Modules/TestSepgsql.pm.orig 2018-08-23 12:50:15.303314923 -0700
--- PGBuild/Modules/TestSepgsql.pm 2018-08-23 12:53:14.608369668 -0700

---

*** 250,256 ****

    print time_str(), "cleaning up ", __PACKAGE__, "\n" if $verbose > 1;

! system("sudo semodule -r sepgsql-regtest");
return;
}

--- 250,256 ----

    print time_str(), "cleaning up ", __PACKAGE__, "\n" if $verbose > 1;

! system("sudo semodule -r sepgsql-regtest > /dev/null");
return;
}

Please consider doing this, or whatever you consider to be the proper fix.

Idea: Compiler warnings

Let the status turn yellow if compiler warnings are encountered, ala

nodeMerge.c:488:8: warning: variable ‘relkind’ set but not used [-Wunused-but-set-variable]

Incorrect check for wait_timeout watchdog in the END block

In the END block of run_build.pl first line seems to be intended to check whether this block is run from man process (and have to delete working directory) or from wait_timeout watchdog (and have to do nothing).

return if (defined($waiter_pid) && $waiter_pid == $$);

Unfortunately, $waiter_pid variable is never defined in waiter process. It is assigned only in main process to the pid of child (waiter) process after the fork. So END block is unable to tell waiter process from main process without defined wait_timeout.

So, if job is terminated via wait_timeout, two processes attempt to recursive remove working directory and prroduce a lot of error messages about suddenly disappearted files they wamted to remove.

This line should read

retiurn if $main_pid == $$;

Variable $main_pid is always defined (just above END block) so it is of no use to check its existense.

Environment filtering in meson builds causes problems with sanitizers

Hi,

Currently one needs to include at least detect_leaks=0 and detect_stack_use_after_return=0 in ASAN_OPTIONS to have a build succeed. I just updated my buildfarm instance, and suddenly the relevant animals failed. Caused by _meson_env() filtering out the ASAN_OPTIONS I had set in the build farm configuration's build_env. I temporarily patched my installation, but that doesn't seem like the right fix...

Why are the environment options filtered at all? And, if they need to be, shouldn't at least options set in build_env be included in _meson_env()?

Greetings,

Andres

run_branches.pl like runner that allows multiple trees and multiple configs

Currently I've a buildfarm machine that hosts like 5 animals, with one of them being very slow & expensive. It now has a decent amount of cores and CPUs available, so I'd like to be able to efficiently run multiple branches at the same time. That's especially important for skink, which takes ~6h on master due to the use of valgrind.

run_branches.pl doesn't currently allow for parallelism, so I'd have to setup a cron-job that runs all branches manually. But that has the worst case potential of running like 5 animals * 6 branches at once, when a commit is backpatched all the way - that's obviously too much.

Additionally using separate run_branches.pl invocations builds the most "urgent" branch for each animal separately, but doesn't take into account that there might be more urgent branches for other animals.

What I think would be useful, is one script that a) allows to run multiple animals b) internally understands parallelism. I can code something up for that myself, but it seems more useful to do that in the buildfarm tree. An opinions on how to do that best? Locally I'd probably code something up using gnu parallel, but that's probably not acceptable?

port number collision between latest one-number branches

Hi,

I guess this line is in need of an update with the new single number revision scheme:

client-code/run_build.pl

Line 214 in 6d26211

if ($branch =~ /REL(\d+)_(\d+)/)

as it is now, HEAD and REL_10_STABLE will use the same port.

Bye,
Chris.

Clone the builder's own logs into lastbuild-logs

It's hard to tell why the build failed before generating a status stamp file when doing a --test run; the builder's own logs don't get recorded into the lastbuild-logs dir.

It looks like the point at which the build failed is only recorded as $stage in web-txn.data.

Whether in verbose mode or not, it'd be nice to output the builder's own logs to the log dir to make it easier to tell what's going on when working on a build worker.

(I'm noting this mainly as a task for myself).

Move find_typedef code from run_build.pl to src/tools/find_typedef in the Postgres repo

I would like to make find_typdef available to extension authors, so they can properly use pgindent if they so choose.

After we have moved the code, it would be great if the buildfarm could just use the find_typedef script from Postgres.

Tom Lane suggested this idea, and I will try to be the champion of it. Copy-pasting code is easy to create the new src/tools/find_typedef, but making the buildfarm use this new script is the larger challenge. I was wondering if you had any suggestions :).

OpenBSD - print kern.version

Hi All,

From my build:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=scorpionfly&dt=2019-08-28%2019%3A00%3A06

Problem is that the uname -v information is practically useless on OpenBSD:

uname -m = amd64
uname -r = 6.6
uname -s = OpenBSD
uname -v = GENERIC.MP#257

I think it would be a good idea to get the sysctl kern.version information:

$ sysctl kern.version
kern.version=OpenBSD 6.6-beta (GENERIC.MP) #257: Wed Aug 28 09:12:29 MDT 2019
    [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

#257 doesn't really mean anything at all: https://flak.tedunangst.com/post/OpenBSD-version-numbers

This number is a archaic remnant of long ago. It is mostly useless. For instance, a bug report that specifies “kernel number 27” means little.

I realize this may not be necessary for building Postgresql, but it'll provide a little more information about the animal.

Some CLI options are missing in the help output

According to

client-code/PGBuild/Options.pm

Lines 57 to 80 in 45d612b

	%standard_options = (
	'nosend' => \$nosend,
	'config=s' => \$buildconf,
	'from-source=s' => \$from_source,
	'from-source-clean=s' => \$from_source_clean,
	'force' => \$forcerun,
	'find-typedefs' => \$find_typedefs,
	'keepall!' => \$keepall, # allow --no-keepall
	'verbose:i' => \$verbose,
	'nostatus' => \$nostatus,
	'test' => \$testmode,
	'help' => \$help,
	'quiet' => \$quiet,
	'skip-steps=s' => \$skip_steps,
	'only-steps=s' => \$only_steps,
	'skip-suites=s' => \$skip_suites,
	'config-set=s' => \@config_set,
	'schedule=s' => \$schedule,
	'tests=s' => \$tests,
	'check-warnings!' => \$check_warnings, # allow --nocheck-warnings
	'delay-check' => \$delay_check,
	'show-error-log' => \$show_error_log,
	'avoid-ts-collisions' => \$avoid_ts_collisions,
	);

the following are missing in --help:

• skip-suites
• config-set
• show-error-log
• avoid-ts-collisions

Problem with ActivePerl 5.22 on Windows.

I've discovered following problem with recent ActivePerl on Windows:
run_build.pl fails on the Git stage with message
"ssh cannot resolve host c"
Error occurs when repository is cloned from
$config->{buildroot}/pgmirror.git to working repository,
It seems that windows Git from git-scm.com doesn't properly understand windows paths with forward slashes like
<drive>:/<directory>
and interprets them like <host>:/path.
It does however interpret such paths with backslashes
<drive>:&lt;directory>
correctly.
It looks like behavoir of abspath function is changed in the recent ActivePerl (which is now compiled with mingw).

I've fixed this problem by adding regexp which replaces forward slashers for backslashes just before a clone command if source url looks like \w:/

Verbose errors after failed run

I've a number of VPATH using buildfarm animal setup. Whenever one branch fails, run_build.pl spill a lot of these to stderr:

cannot unlink file for inst/data-C/base/64190/3395: No such file or directory at ./run_build.pl line 519.
cannot restore permissions to 0100600 for inst/data-C/base/64190/3395: No such file or directory at ./run_build.pl line 519.
cannot unlink file for inst/data-C/base/64190/2613_vm: No such file or directory at ./run_build.pl line 519.
cannot restore permissions to 0100600 for inst/data-C/base/64190/2613_vm: No such file or directory at ./run_build.pl line 519.
cannot unlink file for inst/data-C/base/64190/2657: No such file or directory at ./run_build.pl line 519.

The relevant line is:

            rmtree("inst") unless $keepall;

Incomplete logs when TAP suite fails in a step that includes non-TAP actions

These are missing the regress_log_* files due to $SUBJECT:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly&dt=2021-10-07%2016%3A53%3A42
https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=kittiwake&dt=2021-10-26%2012%3A51%3A11&stg=contrib-install-check-en_US.utf8

Feature request: Command / Flag to see if any branch needs to run test

Hi,

I've started to run buildfarm iterations via systemd timers. That allows only running run_branches.pl when it's not already running (with a time difference since the last run, rather than some regular timer), contains an animal into a cgroup, shows resource usage of runs etc.

I'd like to put an ExecCondition so that the whole service doesn't even get started if there's nothing to do. Maybe something like run_branches.pl needed? I've also wanted to know this before, interactively.

Insane?

Thanks,

Andres

fast branch-filtering doesn't work if run prevented due to trigger_exclude

Hi,

While moving a bunch of animals to a new host I just notices that run_branches.pl was "slow" again in no-change builds.

Adding a bunch of debugging prints to run_branches.pl made me realize why:

filtering REL_11_STABLE 0b496bc988 /home/bf/bf-build/francolin/REL_11_STABLE/francolin.lastrun-logs/githead.log
REL_11_STABLE is not up2date a80740a7c9792a622e5ebb75010ea48a5521bf56 0b496bc988
filtering REL_11_STABLE: 0
filtering REL_12_STABLE 1a5ff7be26 /home/bf/bf-build/francolin/REL_12_STABLE/francolin.lastrun-logs/githead.log
REL_12_STABLE is not up2date dfaa705ce423b5c55bb5effac1835d39aeb76767 1a5ff7be26
filtering REL_12_STABLE: 0
filtering REL_13_STABLE 5dac191edf /home/bf/bf-build/francolin/REL_13_STABLE/francolin.lastrun-logs/githead.log
REL_13_STABLE is not up2date 1c89549ca48d04195bfc774534a54cf84b797e8b 5dac191edf
filtering REL_13_STABLE: 0
filtering REL_14_STABLE 5136c3fb57 /home/bf/bf-build/francolin/REL_14_STABLE/francolin.lastrun-logs/githead.log
REL_14_STABLE is not up2date e373e5578bc39af4f20a4b70c5db1895d0e2d3d1 5136c3fb57
filtering REL_14_STABLE: 0
filtering REL_15_STABLE aa26980ca0 /home/bf/bf-build/francolin/REL_15_STABLE/francolin.lastrun-logs/githead.log
REL_15_STABLE is not up2date c772dfe07a9d4e499117f58d47a7c50d56666459 aa26980ca0
filtering REL_15_STABLE: 0
filtering HEAD 5f53b42cfd /home/bf/bf-build/francolin/HEAD/francolin.lastrun-logs/githead.log
Wed Jan  4 00:17:41 2023: francolin:HEAD is up to date.
filtering HEAD: 1
...
================== git log =====================
Updated 6132 paths from the index
HEAD is now at aa26980ca0 Improve documentation of the CREATEROLE attibute.
francolin:REL_15_STABLE [00:13:11] checking if build run needed ...
francolin:REL_15_STABLE [00:13:11] No build required: last status = Tue Jan  3 08:07:07 2023 GMT, current snapshot = Tue Jan  3 19:53:38 2023 GMT, changed files = 0

I think what's happening is that the the last commit on all branches but HEAD currently is a docs-only commit, which ends up getting filtered with the default trigger_exclude. The trigger_exclude processing appears to happens before githead.log is updated, which then leads to run_branches.pl's apply_filters() not realizing the build is actually up2date.

Greetings,

Andres

Rename build-farm.conf to build-farm.conf.template

It'd be useful if the repo renamed build-farm.conf to build-farm.conf.template and added build-farm.conf to .gitignore. That makes it nicer to work with the config file in-tree.

Probably not a big deal given that there's a --config parameter to the client, of course.

Use ExtUtils::MakeMaker or EUMM to package for CPAN

If ExtUtils::MakeMaker, EUMM, or (deprecated) Module::Build are used, then dependencies can be instaled directly from CPAN with a simple perl -f Makefile.PL and make install.

This would be a lot nicer when configuring machines as buildfarm members, especially when making the config automatable over ansible or similar.