ostreedev/ostree

Operating system and container binary deployment and upgrades

ostree's Issues

test failure on some Debian autobuilders: Opening objects/ directory: openat: No such file or directory

My recent upload of ostree to Debian unstable has a weird test failure on i386 and hppa, which I haven't been able to reproduce in an i386 environment locally. (I'm almost as surprised that we still have an operational hppa autobuilder as you are.)

Does this remind you of any previously-seen failures?

ERROR: tests/test-sysroot-c
===========================

++ test -n ''
++ CMD_PREFIX='env LD_PRELOAD=/«PKGBUILDDIR»/tests/libreaddir-rand.so'
+ setup_os_repository archive-z2 syslinux
+ mode=archive-z2
+ bootmode=syslinux
+ shift
++ pwd
+ oldpwd=/var/tmp/tap-test.0YGeKW
+ cd /var/tmp/tap-test.0YGeKW
+ mkdir testos-repo
+ test -n archive-z2
+ env LD_PRELOAD=/«PKGBUILDDIR»/tests/libreaddir-rand.so ostree --repo=testos-repo init --mode=archive-z2

+ cd /var/tmp/tap-test.0YGeKW
+ mkdir osdata
+ cd osdata
+ mkdir -p boot usr/bin usr/lib/modules/3.6.0 usr/share usr/etc
+ echo 'a kernel'
+ echo 'an initramfs'
++ cat boot/vmlinuz-3.6.0 boot/initramfs-3.6.0
++ cut -f 1 -d ' '
++ sha256sum
+ bootcsum=7bd091ad6955c0e7ef41bab65adb016275664b55efc0d10a0c8cb263da3c1e4e
+ export bootcsum
+ mv boot/vmlinuz-3.6.0 boot/vmlinuz-3.6.0-7bd091ad6955c0e7ef41bab65adb016275664b55efc0d10a0c8cb263da3c1e4e
+ mv boot/initramfs-3.6.0 boot/initramfs-3.6.0-7bd091ad6955c0e7ef41bab65adb016275664b55efc0d10a0c8cb263da3c1e4e
+ echo 'an executable'
+ echo 'some shared data'
+ echo 'a library'
+ ln -s usr/bin bin
+ cat
+ echo 'a config file'
+ mkdir -p usr/etc/NetworkManager
+ echo 'a default daemon file'
+ mkdir -p usr/etc/testdirectory
+ echo 'a default daemon file'
+ env LD_PRELOAD=/«PKGBUILDDIR»/tests/libreaddir-rand.so ostree --repo=/var/tmp/tap-test.0YGeKW/testos-repo commit --add-metadata-string version=1.0.9 -b testos/buildmaster/x86_64-runtime -s Build

+ sleep 2
+ echo 'a new executable'
+ env LD_PRELOAD=/«PKGBUILDDIR»/tests/libreaddir-rand.so ostree --repo=/var/tmp/tap-test.0YGeKW/testos-repo commit --add-metadata-string version=1.0.10 -b testos/buildmaster/x86_64-runtime -s Build
error: Opening objects/ directory: openat: No such file or directory
+ cd /var/tmp/tap-test.0YGeKW
+ cp -a osdata osdata-devel
+ cd osdata-devel
+ mkdir -p usr/include
+ echo 'a development header'
+ env LD_PRELOAD=/«PKGBUILDDIR»/tests/libreaddir-rand.so ostree --repo=/var/tmp/tap-test.0YGeKW/testos-repo commit --add-metadata-string version=1.0.9 -b testos/buildmaster/x86_64-devel -s Build
error: Opening objects/ directory: openat: No such file or directory
+ env LD_PRELOAD=/«PKGBUILDDIR»/tests/libreaddir-rand.so ostree --repo=/var/tmp/tap-test.0YGeKW/testos-repo fsck -q
error: Opening objects/ directory: openat: No such file or directory
+ cd /var/tmp/tap-test.0YGeKW
+ mkdir sysroot
+ export OSTREE_SYSROOT=sysroot
+ OSTREE_SYSROOT=sysroot
+ env LD_PRELOAD=/«PKGBUILDDIR»/tests/libreaddir-rand.so ostree admin init-fs sysroot
+ env LD_PRELOAD=/«PKGBUILDDIR»/tests/libreaddir-rand.so ostree admin os-init testos
+ case $bootmode in
+ setup_os_boot_syslinux
+ mkdir -p sysroot/boot/loader.0
+ ln -s loader.0 sysroot/boot/loader
+ touch sysroot/boot/loader/syslinux.cfg
+ mkdir -p sysroot/boot/syslinux
+ ln -s ../loader/syslinux.cfg sysroot/boot/syslinux/syslinux.cfg
+ cd /var/tmp/tap-test.0YGeKW
+ mkdir /var/tmp/tap-test.0YGeKW/httpd
+ cd httpd
+ ln -s /var/tmp/tap-test.0YGeKW ostree
+ env LD_PRELOAD=/«PKGBUILDDIR»/tests/libreaddir-rand.so ostree trivial-httpd --autoexit --daemonize -p /var/tmp/tap-test.0YGeKW/httpd-port
++ cat /var/tmp/tap-test.0YGeKW/httpd-port
+ port=38644
+ echo http://127.0.0.1:38644
+ cd /var/tmp/tap-test.0YGeKW
error: Opening objects/ directory: openat: No such file or directory

(/«PKGBUILDDIR»/tests/.libs/lt-test-sysroot-c:5989): OSTree-ERROR **: Child process exited with code 1
./buildutil/tap-test: line 23:  5989 Trace/breakpoint trap   ${srcd}/${bn} -k --tap
# random seed: R02Sb37ebc76ebece487a987ee20e8707aea
Copying gpghome to /var/tmp/tap-test.0YGeKW
ostree/deploy/testos initialized as OSTree root
1..1
# OSTree-FATAL-ERROR: Child process exited with code 1
ERROR: tests/test-sysroot-c - too few tests run (expected 1, got 0)
ERROR: tests/test-sysroot-c - exited with status 133 (terminated by signal 5?)

(The path isn't really «PKGBUILDDIR» - the autobuilder automatically replaces the real package build directory to make it easier to diff logs.)

One thing that particularly concerns me about this is that it seems like the test is proceeding even though some of the ostree commands failed.

Cannot cancel a pull

While working with GNOME Software and Flatpak, it is noticeable that the pull ('Downloading...' phase of GNOME Software) doesn't stop when clicking the 'Cancel' button.

The PR below fixes this issue.

exception in non-404 error paths

See https://bugzilla.redhat.com/show_bug.cgi?id=1342168#c18

Support default kernel arguments

How can i add some things to kernel cmdline when compose tree?
Or how i control cmdline when boot ostree system?

Add API to refresh ostree config from filesystem

See coreos/rpm-ostree#401.

Racy assertion when updating an app/runtime

From @thiblahute on July 23, 2016 12:55

Sometimes doing:

flatpak update org.gnome.Platform 3.20

I get the following assertion:

OSTree:ERROR:libglnx/glnx-shutil.c:215:mkdir_p_at_internal: assertion failed: (lastslash != NULL)

I am also getting it on our build server:

https://ci.gstreamer.net/job/pitivi-flatpak/309/console

I am running Flatpak 0.6.7 and ostree 2016.6 on arch.

Copied from original issue: flatpak/flatpak#206

Regression in "ostree admin upgrade"

GNOME/libglnx@40ef5f7 Introduced a regression.

Example:

input_text="588 metadata, 6952 content objects fetched; 196612 KiB transferred in 269 seconds"
ncolumns=40
const guint spacelen = ncolumns - input_textlen; // negative value assigned to uint.

This results in a console infinitely printing empty lines.

The /run/ostree-booted API

This API possibly could be improved as it won't work for some use cases, especially when using ostree-prepare-root as PID1 (/run is not mounted). The current approach does nothing if /run is not mounted.

One solution could be to add a new subcommand to the ostree command line - "ostree booted", where it parses /proc/self/mountinfo to check if / -> ostree/deploy

ostree-autocleanups.h not installed

from /usr/include/ostree-1/ostree.h, the #include <ostree-autocleanups.h> does not have the header present.
This is on master as of d5f6104

Tests failure on Debian stable on ARM

Here's the full list, will dive into them as we go along.

ERROR: tests/test-lzma - missing test plan
ERROR: tests/test-lzma - exited with status 134 (terminated by signal 6?)

ERROR: tests/test-basic.sh - too few tests run (expected 57, got 55)
ERROR: tests/test-basic.sh - exited with status 1

ERROR: tests/test-commit-sign.sh - too few tests run (expected 1, got 0)
ERROR: tests/test-commit-sign.sh - exited with status 135 (terminated by signal 7?)

ERROR: tests/test-pull-archive-z.sh - too few tests run (expected 11, got 2)
ERROR: tests/test-pull-archive-z.sh - exited with status 135 (terminated by signal 7?)

ERROR: tests/test-pull-depth.sh - too few tests run (expected 1, got 0)
ERROR: tests/test-pull-depth.sh - exited with status 139 (terminated by signal 11?)

ERROR: tests/test-pull-summary-sigs.sh - too few tests run (expected 7, got 1)
ERROR: tests/test-pull-summary-sigs.sh - exited with status 139 (terminated by signal 11?)

ERROR: tests/test-delta.sh - too few tests run (expected 8, got 0)
ERROR: tests/test-delta.sh - exited with status 1

ERROR: tests/test-prune.sh - too few tests run (expected 2, got 0)
ERROR: tests/test-prune.sh - exited with status 1

Static delta generation broken

After f66906c, static delta generation is broken (on i386 at least). There are 2 issues:

• The call to fcntl with F_DUPFD_CLOEXEC when filename is not specified is missing the minimum fd number argument.
• In the out path, derefencing part_temp_fds->len may segfault if part_temp_fds is still NULL.

No gpg verification of commits if using static deltas

I accidentally got a few commits signed with the wrong gpg key, but you can download these fine, as long as you use static-deltas (the summary is signed with the correct key).

Is this the expected behaviour? I have both gpg-verify and gpg-verify-summary set in the remote config.

I understand that there is some level of trust from the signed summary, but it would cause less confusing errors if gpg-verify=true means we always require signed commits.

'ostree static-delta show' can dump core when given bogus delta

Using RHELAH autobrew 7.3.internal.0.33 which has ostree-2016.10-1.atomic.el7.x86_64

# ostree static-delta list --repo=/srv/cahc e1788f54a981639a0063fe24f21720ea946f52ba3b01211b98d67546e7f3e76f-97f8f19d226215da845c2c4df49847b881a2d203bf36bfcac084be149dc93596
# ostree static-delta show --repo=/srv/cahc foobar
**
OSTree:ERROR:src/libostree/ostree-core.c:1241:ostree_checksum_inplace_to_bytes: assertion failed: (little != -1)
Aborted (core dumped)
# ostree static-delta show --repo=/srv/cahc e1788f54a981639a0063fe24f21720ea946f52ba3b01211b98d67546e7f3e76f-97f8f19d226215da845c2c4df49847b881a2d203bf36bfcac084be149dc93596
({'ostree.endianness': <byte 0x6c>}, uint64 4964323960373116928, [byte 0xe1, 0x78, 0x8f, 0x54, 0xa9, 0x81, 0x63, 0x9a, 0x00, 0x63, 0xfe, 0x24, 0xf2, 0x17, 0x20, 0xea, 0x94, 0x6f, 0x52, 0xba, 0x3b, 0x01, 0x21, 0x1b, 0x98, 0xd6, 0x75, 0x46, 0xe7, 0xf3, 0xe7, 0x6f], [byte 0x97, 0xf8, 0xf1, 0x9d, 0x22, 0x62,
...

wrong releases page on github?

9 hours ago  v2016.10  …  36e8ba1   zip   tar.gz
9 days ago v2016.9  …  dd71999   zip   tar.gz
Latest release
 v2016.8  73eabca Verified 2016.8
@cgwalters cgwalters released this on Aug 9 · 57 commits to master since this release

Why latest so old?

OSTree should clean up old ovl dirs in /var/tmp

Not sure when would be a good time to do this garbage collection. Maybe making it a new OstreeSysrootCleanupFlag ?

Allow ostree admin status unprivileged

Currently you can only run ostree admin status as root since it takes the sysroot lock. I think it's nice to allow unprivileged users to get the status of the system. After all, they could dig around in the repo and do it themselves, anyway.

PR coming.

Allow manual tweaking of delta generation

In flatpak, we typically have two related runtimes, the platform and the corresponding sdk. These share a lot of files, but some are different, and many are only in the sdk.

If using static deltas to download these two it seems like we're downloading everything in the platform twice. It would be nice if we could somehow make the sdk deltas refer to the platform ones, or otherwise set things up so that the sdk deltas have shared files exclusively in some delta parts so they can be completely omitted from the download.

libarchive: make it easy to canonicalize perms

• Directories writable by user
• No world writable files

anything else?

over 9000 blocks of leak in pull

==1235== 2,635,006 (357,552 direct, 2,277,454 indirect) bytes in 14,898 blocks are definitely lost in loss record 2,153 of 2,153
==1235==    at 0x4C2B974: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==1235==    by 0x64583E5: g_malloc0 (gmem.c:124)
==1235==    by 0x4E69FE7: enqueue_one_object_request (ostree-repo-pull.c:1340)
==1235==    by 0x4E67CE1: scan_dirtree_object (ostree-repo-pull.c:492)
==1235==    by 0x4E69D62: scan_one_metadata_object_c (ostree-repo-pull.c:1280)
==1235==    by 0x4E677A2: idle_worker (ostree-repo-pull.c:331)
==1235==    by 0x6452DF9: g_main_dispatch (gmain.c:3154)
==1235==    by 0x6452DF9: g_main_context_dispatch (gmain.c:3769)
==1235==    by 0x6453137: g_main_context_iterate.isra.24 (gmain.c:3840)
==1235==    by 0x64531EB: g_main_context_iteration (gmain.c:3901)
==1235==    by 0x4E6E4D5: ostree_repo_pull_with_options (ostree-repo-pull.c:2725)
==1235==    by 0x42032D: ostree_builtin_pull (ot-builtin-pull.c:260)
==1235==    by 0x414E92: ostree_run (ot-main.c:209)
==1235==    by 0x40AF3E: main (main.c:78)

Possible uninitialized variable used

src/libostree/ostree-repo-commit.c: In function 'ostree_repo_write_commit_detached_metadata':
src/libostree/ostree-repo-commit.c:2084:6: warning: 'data' may be used uninitialized in this function [-Wmaybe-uninitialized]
   if (data == NULL)
      ^

Roll back an update and use of /proc/cmdline

We're trying to implement a roll back scheme so if you deploy a bogus tree you roll back to the previous one.

To do that we create a dracut service that checks if the last boot succeeded and if it didn't, it swaps the bootloader configuration.

At the time we roll back, since we're already booted, /proc/cmdline will have the old tree and ostree-prepare-root will use that causing the bogus tree to boot again. To fix it we bind-mount a file with the new ostree= value on /proc/cmdline.

Since this is pretty hacky, I wanted to start a discussion on how to do it in a cleaner way, passing the information of the tree to boot in another way. In addition, note that ostree admin also relies on the information in /proc/cmdline.

In the future, if we move this logic to the bootloader (instead of doing it in the initramfs) we won't have this problem but we're not quite there yet.

Not always zeroing mtime in checkouts

All mtimes for file objects in bare and bare-user repos are stored with mtime == 0.
This means when you check them out with matching mode (i.e. -U with bare-user, nothing with bare) then we hardlink and the checked out file has mtime 0.

However if the modes don't match, or the repo is archive-z2, then we copy the files on checkout, and this does not zero the mtime (although the code explicitly sets mtime 0 on directories, so this only applies to files).

This should be fixed by zeroing the mtime in the copy case too.

Licensing Debate

Revisiting the licensing question from #305 (comment), what is the reason for OSTree to be LGPL licensed?

Maybe Apple or Google would like to use the code, but in that case it would probably have to be even more permissively licensed, BSD or something.

On the other hand, making it GPL would make it easier to share code with other open-source projects (for example, Git). And the GNU project recommends using GPL for new libraries: http://www.gnu.org/licenses/why-not-lgpl.en.html, particularly for "unique" libraries that will typically only be found on free operating systems.

v0 of recursive deltas

The deltas code/design supports recursion, but we didn't really follow through implementing this. There's both client side and compiler work for this.

The goal should be that we support upgrading from both N-2 and N-1 to N reasonably.

Lower LZMA memory requirements

LZMA has some pretty high memory requirements when compressing and decompressing. ostree's builtin LZMA compression helpers should take this into account when compressing (maybe only allowing memory usage limitation when running the tests, and not in production).

See the "Memory usage" section of http://linux.die.net/man/1/lzma

Disable static deltas by default for local pulls

When doing local pulls, you actually want to hardlink/copy the exact objects rather than generating them from static deltas.

Issues with ref (tmp file cleanup, deleting folders, commit/create conflict with remote)

Currently there are 3 minor issues with refs:

1 ostree_repo_transaction_set_ref and ostree_repo_set_ref_immediate should clean up .tmp files on failure

Currently, if a ref "foo/bar" exists (foo is a folder), and a user tries to create foo, the API functions ostree_repo_transaction_set_ref and ostree_repo_set_ref_immediate in ostree-repo-commit.c would fail, and create a .tmp file (in the format of .tmpABCDE). ostree_repo_transaction_set_ref needs to abort the transaction or the .tmp file is not cleaned up. ostree_repo_set_ref_immediate will always create the .tmp file. This should be cleaned up on failures.

2 allow deleting of folders in ref

Currently, if a ref "foo/bar/baz" exists, and a user either "refs --delete foo" or "refs --delete foo/bar/baz", only the "baz" file gets deleted. All folders (both in local and remote) cannot be deleted unless the user goes into the refs folder and manually rm. The folders don't actually show up in "refs --list" and could cause much hassle. They should be deleted when a) the user specifies to delete or b) when the folder is empty of refs.

3 allow local refs with same name as remote refs/folders, for both commit and ref --create

Currently, if there is a remote ref "remote:foo/bar" or "remote:foo", and the user wants to create a local ref of the name "foo", either through "commit" or "refs --create" (new option), ostree errors saying that the folder "foo" exists, even though it is in the "remote" and not local. The user should be able to create a local ref called foo, which is not affected by the remote repo.

automatic remove own /etc stuff if it equal to tree contents

How can i deal with this use-case:

• i have composed tree on host
• i see that i need some services needs to be enabled, and some configs need to be changed
• i change needed files and enable services
• i see that this changed needs to be present in all my refs and create this modifications when compose tree (for example auto enable units via units section in treefile).
• i deploy updated ref to my host
• ostree admin config-diff displays that i have own services enabled by hand (but i want to discard it and use ref defaults)

Does it possible to add something like E ?

crash when pulling with flatpak

I got a crash when pulling from Flatpak, using ostree 2016.5 on openSUSE Leap 42.1:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f37a1d81cf5 in g_mutex_lock (mutex=mutex@entry=0x6133636366366161) at gthread-posix.c:1336
1336      if G_UNLIKELY (g_atomic_int_add (&mutex->i[0], 1) != 0)
Missing separate debuginfos, use: zypper install glib-networking-debuginfo-2.44.0-2.4.x86_64 glibc-locale-debuginfo-2.19-19.1.x86_64 gnome-keyring-debuginfo-3.16.0-7.1.x86_64 gvfs-debuginfo-1.24.3-7.2.x86_64 libXau6-debuginfo-1.0.8-6.1.x86_64 libarchive13-debuginfo-3.1.2-10.1.x86_64 libassuan0-debuginfo-2.1.1-3.6.x86_64 libattr1-debuginfo-2.4.47-5.4.x86_64 libblkid1-debuginfo-2.25-15.2.x86_64 libbz2-1-debuginfo-1.0.6-32.1.x86_64 libgpgme11-debuginfo-1.6.0-9.1.x86_64 libgsystem0-debuginfo-2015.2-4.2.x86_64 liblzma5-debuginfo-5.0.5-3.5.x86_64 libmount1-debuginfo-2.25-15.2.x86_64 libopenssl1_0_0-debuginfo-1.0.1i-15.1.x86_64 libseccomp2-debuginfo-2.1.1-5.1.x86_64 libselinux1-debuginfo-2.3-5.1.x86_64 libsoup-2_4-1-debuginfo-2.50.0-5.1.x86_64 libsqlite3-0-debuginfo-3.8.10.2-5.2.x86_64 libudev1-debuginfo-210-95.1.x86_64 libuuid1-debuginfo-2.25-15.2.x86_64 libxml2-2-debuginfo-2.9.1-19.1.x86_64 systemd-debuginfo-210-95.1.x86_64
(gdb) bt
#0  0x00007f37a1d81cf5 in g_mutex_lock (mutex=mutex@entry=0x6133636366366161) at gthread-posix.c:1336
#1  0x00007f37a1d3d458 in g_source_destroy_internal (source=0x16d12c0, context=0x6133636366366161, have_lock=0) at gmain.c:1159
#2  0x00007f37a26474c8 in  () at /usr/lib64/libsoup-2.4.so.1
#3  0x00007f37a20187f3 in g_object_unref (_object=0x12d3240) at gobject.c:3137
#4  0x00007f37a28f3e46 in thread_closure_unref (thread_closure=0x12c35e0) at src/libostree/ostree-fetcher.c:137
#5  0x00007f37a28f402f in ostree_fetcher_session_thread (data=0x12c35e0) at src/libostree/ostree-fetcher.c:467
#6  0x00007f37a1d64f65 in g_thread_proxy (data=0x12c1450) at gthread.c:764
#7  0x00007f37a16b60a4 in start_thread (arg=0x7f379aca4700) at pthread_create.c:309
#8  0x00007f37a13eafed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


(gdb) thread apply all bt

Thread 4 (Thread 0x7f379d3c2700 (LWP 20380)):
#0  0x00007f37a13e2bbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f37a1d3fe64 in g_main_context_iterate (priority=2147483647, n_fds=2, fds=0x12f15b0, timeout=-1, context=0x148fff0) at gmain.c:4103
#2  0x00007f37a1d3fe64 in g_main_context_iterate (context=0x148fff0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3803
#3  0x00007f37a1d4019a in g_main_loop_run (loop=0x12e4440) at gmain.c:4002
#4  0x00007f37a2335426 in gdbus_shared_thread_func (user_data=0x148ffc0) at gdbusprivate.c:274
#5  0x00007f37a1d64f65 in g_thread_proxy (data=0x12c11e0) at gthread.c:764
#6  0x00007f37a16b60a4 in start_thread (arg=0x7f379d3c2700) at pthread_create.c:309
#7  0x00007f37a13eafed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f379b6dd700 (LWP 20365)):
#0  0x00007f37a13e6f79 in syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f37a1d81f3e in g_cond_wait_until (cond=cond@entry=0x12d27a8, mutex=mutex@entry=0x12d27a0, end_time=end_time@entry=29989651355) at gthread-posix.c:1442
#2  0x00007f37a1d15dc1 in g_async_queue_pop_intern_unlocked (queue=queue@entry=0x12d27a0, wait=wait@entry=1, end_time=end_time@entry=29989651355) at gasyncqueue.c:422
#3  0x00007f37a1d1634b in g_async_queue_timeout_pop (queue=0x12d27a0, timeout=timeout@entry=15000000) at gasyncqueue.c:543
#4  0x00007f37a1d659ac in g_thread_pool_thread_proxy () at gthreadpool.c:167
#5  0x00007f37a1d659ac in g_thread_pool_thread_proxy (data=<optimized out>) at gthreadpool.c:364
#6  0x00007f37a1d64f65 in g_thread_proxy (data=0x12c1770) at gthread.c:764
#7  0x00007f37a16b60a4 in start_thread (arg=0x7f379b6dd700) at pthread_create.c:309
#8  0x00007f37a13eafed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f37a2f13880 (LWP 20362)):
#0  0x00007f37a137e0d5 in malloc_consolidate (av=av@entry=0x7f37a16a8620 <main_arena>) at malloc.c:4142
#1  0x00007f37a137f4e8 in _int_malloc (av=av@entry=0x7f37a16a8620 <main_arena>, bytes=bytes@entry=4128) at malloc.c:3422
#2  0x00007f37a13807bb in _int_memalign (av=av@entry=0x7f37a16a8620 <main_arena>, alignment=alignment@entry=2048, bytes=bytes@entry=2033) at malloc.c:4402
#3  0x00007f37a1380a46 in memalign_check (alignment=2048, bytes=2032, caller=<optimized out>) at hooks.c:426
#4  0x00007f37a1383612 in __posix_memalign (memptr=memptr@entry=0x7ffda99d7748, alignment=alignment@entry=2048, size=size@entry=2032) at malloc.c:5015
#5  0x00007f37a1d11c13 in slab_allocator_alloc_chunk (memsize=2032, alignment=2048) at gslice.c:1378
#6  0x00007f37a1d11c13 in slab_allocator_alloc_chunk (allocator=0x7f37a20022c0 <allocator>, chunk_size=144, ix=8) at gslice.c:1252
#7  0x00007f37a1d11c13 in slab_allocator_alloc_chunk (chunk_size=chunk_size@entry=144) at gslice.c:1297
#8  0x00007f37a1d5b647 in g_slice_alloc (countp=0x12a6d78, ix=8) at gslice.c:731
#9  0x00007f37a1d5b647 in g_slice_alloc (tmem=<optimized out>, ix=8) at gslice.c:801
#10 0x00007f37a1d5b647 in g_slice_alloc (mem_size=mem_size@entry=136) at gslice.c:996
#11 0x00007f37a1d5b6ae in g_slice_alloc0 (mem_size=mem_size@entry=136) at gslice.c:1032
#12 0x00007f37a28f47e2 in _ostree_fetcher_constructed (object=0x1609200 [OstreeFetcher]) at src/libostree/ostree-fetcher.c:533
#13 0x00007f37a2019092 in g_object_new_internal (class=class@entry=0x12ca500, params=params@entry=0x7ffda99d79c0, n_params=1) at gobject.c:1814
#14 0x00007f37a201af24 in g_object_new_valist (object_type=object_type@entry=19702288, first_property_name=first_property_name@entry=0x7f37a290a288 "config-flags", var_args=var_args@entry=0x7ffda99d7b18)
    at gobject.c:2033
#15 0x00007f37a201b304 in g_object_new (object_type=19702288, first_property_name=first_property_name@entry=0x7f37a290a288 "config-flags") at gobject.c:1617
#16 0x00007f37a28f46b4 in _ostree_fetcher_new (tmpdir_dfd=24, flags=OSTREE_FETCHER_FLAGS_NONE) at src/libostree/ostree-fetcher.c:607
#17 0x00007f37a28c95cd in _ostree_repo_remote_new_fetcher (self=self@entry=0x12c4320 [OstreeRepo], remote_name=remote_name@entry=0x7ffda99d9a61 "gnome", error=error@entry=0x7ffda99d8330)
    at src/libostree/ostree-repo.c:518
#18 0x00007f37a28f9ad5 in ostree_repo_pull_with_options (self=self@entry=0x12c4320 [OstreeRepo], remote_name_or_baseurl=remote_name_or_baseurl@entry=0x7ffda99d9a61 "gnome", options=<optimized out>, progress=progress@entry=0x12c1630 [OstreeAsyncProgress], cancellable=cancellable@entry=0x0, error=error@entry=0x7ffda99d8330) at src/libostree/ostree-repo-pull.c:2372
#19 0x00007f37a28fb781 in ostree_repo_pull_one_dir (self=self@entry=0x12c4320 [OstreeRepo], remote_name=remote_name@entry=0x7ffda99d9a61 "gnome", dir_to_pull=dir_to_pull@entry=0x0, refs_to_fetch=refs_to_fetch@entry=0x7ffda99d8190, flags=flags@entry=OSTREE_REPO_PULL_FLAGS_MIRROR, progress=progress@entry=0x12c1630 [OstreeAsyncProgress], cancellable=cancellable@entry=0x0, error=error@entry=0x7ffda99d8330)
    at src/libostree/ostree-repo-pull.c:1773
#20 0x00007f37a28fb7c2 in ostree_repo_pull (self=self@entry=0x12c4320 [OstreeRepo], remote_name=remote_name@entry=0x7ffda99d9a61 "gnome", refs_to_fetch=refs_to_fetch@entry=0x7ffda99d8190, flags=flags@entry=OSTREE_REPO_PULL_FLAGS_MIRROR, progress=progress@entry=0x12c1630 [OstreeAsyncProgress], cancellable=cancellable@entry=0x0, error=error@entry=0x7ffda99d8330) at src/libostree/ostree-repo-pull.c:1747
#21 0x000000000042154f in flatpak_dir_pull (self=self@entry=0x12c1000 [FlatpakDir], repository=repository@entry=0x7ffda99d9a61 "gnome", ref=ref@entry=0x1389eb0 "runtime/org.gnome.Platform/x86_64/3.20", subpaths=---Type <return> to continue, or q <return> to quit---
subpaths@entry=0x0, repo=repo@entry=0x12c4320 [OstreeRepo], flags=flags@entry=OSTREE_REPO_PULL_FLAGS_MIRROR, progress=0x12c1630 [OstreeAsyncProgress], 
    progress@entry=0x0, cancellable=cancellable@entry=0x0, error=error@entry=0x7ffda99d8330) at common/flatpak-dir.c:1244
#22 0x0000000000426632 in flatpak_dir_install (self=0x12c1000 [FlatpakDir], no_pull=0, no_deploy=0, ref=ref@entry=0x1389eb0 "runtime/org.gnome.Platform/x86_64/3.20", remote_name=remote_name@entry=0x7ffda99d9a61 "gnome", subpaths=0x0, progress=progress@entry=0x0, cancellable=cancellable@entry=0x0, error=0x7ffda99d8330) at common/flatpak-dir.c:3000
#23 0x000000000041049f in flatpak_builtin_install (argc=4, argv=0x7ffda99d8468, cancellable=0x0, error=<optimized out>) at app/flatpak-builtins-install.c:244
#24 0x000000000040e5cd in flatpak_run (argc=4, argc@entry=5, argv=argv@entry=0x7ffda99d8468, res_error=res_error@entry=0x7ffda99d8360) at app/flatpak-main.c:337
#25 0x000000000040de9c in main (argc=5, argv=0x7ffda99d8468) at app/flatpak-main.c:422

Thread 1 (Thread 0x7f379aca4700 (LWP 20381)):
#0  0x00007f37a1d81cf5 in g_mutex_lock (mutex=mutex@entry=0x6133636366366161) at gthread-posix.c:1336
#1  0x00007f37a1d3d458 in g_source_destroy_internal (source=0x16d12c0, context=0x6133636366366161, have_lock=0) at gmain.c:1159
#2  0x00007f37a26474c8 in  () at /usr/lib64/libsoup-2.4.so.1
#3  0x00007f37a20187f3 in g_object_unref (_object=0x12d3240) at gobject.c:3137
#4  0x00007f37a28f3e46 in thread_closure_unref (thread_closure=0x12c35e0) at src/libostree/ostree-fetcher.c:137
#5  0x00007f37a28f402f in ostree_fetcher_session_thread (data=0x12c35e0) at src/libostree/ostree-fetcher.c:467
#6  0x00007f37a1d64f65 in g_thread_proxy (data=0x12c1450) at gthread.c:764
#7  0x00007f37a16b60a4 in start_thread (arg=0x7f379aca4700) at pthread_create.c:309
#8  0x00007f37a13eafed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Refs that do not belong to ostree should not be automatically pruned

There are multiple apps now that also need to store data in the system repo. These apps might not want ostree to automatically prune their refs (see ostree_admin_cleanup()).

We need to make ostree smarter so that it only prunes the refs that belong to it.
See also PR #257.

Static deltas with no checksums

I was experimenting with static delta management and made a delta between 2 commits that were exactly the same but different revisions. The delta was created successfully, but later _ostree_static_delta_parse_checksum_array returned an error because there were no checksums.

$ ostree static-delta show 5824a0ce049681f17d4c7a40c51b8d1338b8d5131f79f29157066cf89f53b9bd-9617d9c15ba643ccf566b2cdf793d46f3b9bc42d392b1ac2940e9311e232e2c8
...
Delta: 5824a0ce049681f17d4c7a40c51b8d1338b8d5131f79f29157066cf89f53b9bd-9617d9c15ba643ccf566b2cdf793d46f3b9bc42d392b1ac2940e9311e232e2c8
Endianness: little
Timestamp: 1469740860
Number of parents: 0
Number of fallback entries: 0
Total Fallback Size: 0 (0 bytes)
Total Fallback Uncompressed Size: 0 (0 bytes)
Number of parts: 1
PartMeta0: nobjects=0 size=61 usize=0
PartPayload0: nmodes=0 nxattrs=0 blobsize=0 opsize=0
error: Invalid checksum array length 0

Should ostree refuse to generate these empty deltas or should the check be relaxed? It looks like _ostree_static_delta_part_execute has an assertion that there are more than 0 checksums, although _ostree_repo_static_delta_part_have_all_objects looks like it would succeed.

pull-progress-changed unreliable

So far I have seen several issues with the pull progress reporting when pulling data from a remote repository (when testing on embedded devices).

1. Expected download size != pulled data size. I have managed to reproduce this on desktop PC with the following test code: https://paste.kde.org/p4yjkksa9

3,2 MB / 18,2 kB

2. ETA is broken in the above case as well: 265876927 days

3. Receiving delta parts: 0/12 does not show progress. (I do not have a test code for reproducing this on desktop PC). On a device sequence of messages is as follows:

Receiving delta parts: 0/12
No progress-changed messages for 2min30s (--disable-fsync does not affect this)
Receiving delta parts: 12/12

The expected behavior here I think is: 0/12 -> 1/12 -> 2/12 -> .. 12/12 ?

These are the same issues I have previously mentioned on the mailing list at: https://mail.gnome.org/archives/ostree-list/2016-July/msg00020.html

ostree export should zero out mtimes in tarball

Since mtimes don't make sense in ostree repos and a checkout will cause a zeroed mtime we should do the same in the tarball headers.

hang in pull

Got this locally once with #495 running tests/test-admin-deploy-syslinux.sh, but it doesn't seem to reproduce easily right now.

(gdb) t a a bt

Thread 3 (Thread 0x7f222b183700 (LWP 26148)):
#0  0x00007f222fdbbc4d in poll () at /lib64/libc.so.6
#1  0x00007f223059da06 in g_main_context_iterate.isra () at /lib64/libglib-2.0.so.0
#2  0x00007f223059dd92 in g_main_loop_run () at /lib64/libglib-2.0.so.0
#3  0x00007f2231c4da41 in ostree_fetcher_session_thread (data=0x218e980) at src/libostree/ostree-fetcher.c:506
#4  0x00007f22305c3cf5 in g_thread_proxy () at /lib64/libglib-2.0.so.0
#5  0x00007f223008e5ba in start_thread () at /lib64/libpthread.so.0
#6  0x00007f222fdc77cd in clone () at /lib64/libc.so.6

Thread 2 (Thread 0x7f222a982700 (LWP 26077)):
#0  0x00007f222fdbbc4d in poll () at /lib64/libc.so.6
#1  0x00007f223059da06 in g_main_context_iterate.isra () at /lib64/libglib-2.0.so.0
#2  0x00007f223059db1c in g_main_context_iteration () at /lib64/libglib-2.0.so.0
#3  0x00007f223059db61 in glib_worker_main () at /lib64/libglib-2.0.so.0
#4  0x00007f22305c3cf5 in g_thread_proxy () at /lib64/libglib-2.0.so.0
#5  0x00007f223008e5ba in start_thread () at /lib64/libpthread.so.0
#6  0x00007f222fdc77cd in clone () at /lib64/libc.so.6

Thread 1 (Thread 0x7f22321f0900 (LWP 26064)):
#0  0x00007f223008f6ad in pthread_join () at /lib64/libpthread.so.0
#1  0x00007f22305e1a67 in g_system_thread_wait () at /lib64/libglib-2.0.so.0
#2  0x00007f22305c415a in g_thread_join () at /lib64/libglib-2.0.so.0
#3  0x00007f2231c4e903 in _ostree_fetcher_finalize (object=0x21b94d0) at src/libostree/ostree-fetcher.c:575
#4  0x00007f2230876117 in g_object_unref () at /lib64/libgobject-2.0.so.0
#5  0x00007f2231c2b75d in ostree_repo_pull_with_options (self=<optimized out>, remote_name_or_baseurl=<optimized out>, options=<optimized out>, progress=progress@entry=0x0, cancellable=cancellable@entry=0x0, error=error@entry=0x7ffc5f2f3458)
    at src/libostree/ostree-repo-pull.c:3026
#6  0x0000000000419f91 in ostree_builtin_pull (argc=<optimized out>, argv=<optimized out>, cancellable=0x0, error=0x7ffc5f2f3458) at src/ostree/ot-builtin-pull.c:260
#7  0x0000000000412dbf in ostree_run (argc=<optimized out>, argc@entry=6, argv=<optimized out>, argv@entry=0x7ffc5f2f35a8, commands=commands@entry=0x62bfc0 <commands>, res_error=res_error@entry=0x7ffc5f2f34a8) at src/ostree/ot-main.c:209
#8  0x000000000040b58f in main (argc=6, argv=0x7ffc5f2f35a8) at src/ostree/main.c:78
(gdb) up
#1  0x00007f22305e1a67 in g_system_thread_wait () from /lib64/libglib-2.0.so.0
(gdb) 
#2  0x00007f22305c415a in g_thread_join () from /lib64/libglib-2.0.so.0
(gdb) 
#3  0x00007f2231c4e903 in _ostree_fetcher_finalize (object=0x21b94d0) at src/libostree/ostree-fetcher.c:575
575             g_thread_join (self->session_thread);
(gdb) p self->session_thread
$1 = (GThread *) 0x21b9ca0
(gdb) p *$
$2 = {func = 0x7f2231c4d990 <ostree_fetcher_session_thread>, data = 0x218e980, joinable = 1, priority = G_THREAD_PRIORITY_LOW}
(gdb) 

improve reterieval of version information (RHBZ 1298200)

When using the rpm-ostree deploy command, the user has to know the available versions which can be deployed to. This can be a published list of versions somewhere or by pulling the commit log for a refspec and then inspecting said log:

# ostree pull --depth=-1 --commit-metadata-only rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
# ostree log rhel-atomic-host/7/x86_64/standard                          
commit ec85fba1bf789268d5fe954aac09e6bd58f718e47a2fcb18bf25073b396e695d
Date:  2015-11-10 16:11:46 +0000
Version: 7.2


commit 23d96474f6775c27cf258e9872330b23f20e80ff4e0b61426debd00ca11a953f
Date:  2015-10-30 18:07:50 +0000
Version: 7.1.6


commit 8060f80ffd480b3806c83dd36c2ec6023384716572cf07037eaf49a781c2cd29
Date:  2015-09-11 18:33:12 +0000
Version: 7.1.5
...

The user experience could be improved if there was an easier mechanism to retrieve the version information using the ostree client.

Suggestions:

• enhance ostree log to automatically retrieve the commit history from the remote if only 1 commit is available locally
• introduce a new sub-command ostree remote log which would retrieve the commit history from the remote
• perhaps introduce an new option to either above idea that would limit the amount of commits to retrieve (i.e. --num-commits X)

Downstream bug - https://bugzilla.redhat.com/show_bug.cgi?id=1298200

failed to build rpm for 2016.8 under fedora 23

+ cd /home/vtolstov/rpmbuild/BUILD
+ rm -rf ostree-2016.8
+ /usr/bin/xz -dc /home/vtolstov/rpmbuild/SOURCES/ostree-2016.8.tar.xz
+ /usr/bin/tar -xof -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd ostree-2016.8
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ /usr/bin/git init -q
+ /usr/bin/git config user.name rpm-build
+ /usr/bin/git config user.email '<rpm-build>'
+ /usr/bin/git add .
+ /usr/bin/git commit -q -a --author 'rpm-build <rpm-build>' -m 'ostree-2016.8 base'
+ /usr/bin/cat /home/vtolstov/rpmbuild/SOURCES/0001-ostree-remount-Explicitly-set-tmp-to-01777.patch
+ /usr/bin/git apply --index -
+ /usr/bin/git commit -q -m 0001-ostree-remount-Explicitly-set-tmp-to-01777.patch --author 'rpm-build <rpm-build>'
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.TNZt7b
+ umask 022
+ cd /home/vtolstov/rpmbuild/BUILD
+ cd ostree-2016.8
+ env NOCONFIGURE=1 ./autogen.sh
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I buildutil -I libglnx ${ACLOCAL_FLAGS}
aclocal: error: aclocal: file 'buildutil/libglnx.m4' does not exist
autoreconf: aclocal failed with exit status: 1
error: Bad exit status from /var/tmp/rpm-tmp.TNZt7b (%build)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.TNZt7b (%build)
vtolstov@yoctocloud:~/.../fedora/fedora-vtolstov/ostree (master *)$ ls bu^C
vtolstov@yoctocloud:~/.../fedora/fedora-vtolstov/ostree (master *)$ ls ~/rpmbuild/
BUILD/     BUILDROOT/ RPMS/      SOURCES/   SPECS/     SRPMS/
vtolstov@yoctocloud:~/.../fedora/fedora-vtolstov/ostree (master *)$ ls ~/rpmbuild/BUILD
BUILD/     BUILDROOT/
vtolstov@yoctocloud:~/.../fedora/fedora-vtolstov/ostree (master *)$ ls ~/rpmbuild/BUILD/ostree-2016.8/buildutil/
attributes.m4   glibtests.m4    libglnx.m4      lt~obsolete.m4  ltsugar.m4      tap-driver.sh
glib-tap.mk     gtk-doc.m4      libtool.m4      ltoptions.m4    ltversion.m4    tap-test
vtolstov@yoctocloud:~/.../fedora/fedora-vtolstov/ostree (master *)$ ls ~/rpmbuild/BUILD/ostree-2016.8/buildutil/libglnx.m4
/home/vtolstov/rpmbuild/BUILD/ostree-2016.8/buildutil/libglnx.m4
vtolstov@yoctocloud:~/.../fedora/fedora-vtolstov/ostree (master *)$ ls -aloh ~/rpmbuild/BUILD/ostree-2016.8/buildutil/libglnx.m4
lrwxrwxrwx. 1 vtolstov 21 Aug 18 15:16 /home/vtolstov/rpmbuild/BUILD/ostree-2016.8/buildutil/libglnx.m4 -> ../libglnx/libglnx.m4

Provide a way to add a ref to an existing commit

Making a note of this here since I again found a need for it. Not sure in which subcommand it would go, but this should be trivial to implement with a single call to ostree_repo_set_ref_immediate().

Summary file races

Originally from flatpak/flatpak#149.

There appear to be multiple races with the summary files.

1. The previous summary signature file is not removed until the new summary is regenerated. A client could get the old signature and new summary in this window.
2. The summary signature can't be generated until after the new summary file is in place. In addition to the race from 1, if the client has gpg-verify-summary=true, then there's a window where no signature exists.

I think there should be a ostree_repo_regenerate_summary variant that also takes GPG keys so that ostree can generate the signature and summary in one pass.

To make the race smaller, this new variant would generate the signature and summary in the temporary directory and rename them. However, that's still racy on the server side, and it doesn't fix the race on the client side where the files may be regenerated between the requests for the summary and signature file.

Here's 2 thoughts about making this really atomic:

The other idea I had to go full atomic would be a summaries directory with a current symlink that can be atomically swapped sure of like the bootloader directory in ostree.

From @alexlarsson:

So, even better would be to have an atomically replaced summary.sig, which contained a metadata field with the unique name of the summary file (we could keep the "summary" name a symlink to the latest summary for backwards compat). Then we could keep old summary files around for a while. This would fix clients racing with the summary being replaced.

CC @alexlarsson

[RFE] 'ostree copy-config` - method for copying contents of /etc between trees

This is born out of projectatomic/centos-release-atomic-host-devel#2 (comment)

The idea is to have the ability to copy the contents of /etc from one tree to another.

Other ideas for this:

• Allow an argument to specify one or more sub-directories rather than all of /etc
• Allow an argument to specify direction of copy, i.e. booted tree->staged tree or staged tree->booted tree

Requires glib 2.40 but needs glib 2.44

https://github.com/ostreedev/ostree/blob/master/configure.ac#L50 says:

GIO_DEPENDENCY="gio-unix-2.0 >= 2.40.0 libgsystem >= 2015.1"

But G_DEFINE_AUTOPTR_CLEANUP_FUNC is used in the code, which makes it need 2.44.

remove hotfix from origin on upgrades

Otherwise it shows up as incorrectly hotfixed.

SIGBUS/use-after-free on ARM

Following on from #378

During the tests, apart from the test failures, which I'll look at again later on, there are a number of crashers, either SIGBUS, or SIGSEGV ones.

From the 9 core dumps I gathered running the test suite, I found those unique SIGBUS ones:

#0  0xb6c66244 in g_mutex_lock (mutex=mutex@entry=0x32323232) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gthread-posix.c:1337
#1  0xb6c34382 in g_source_destroy_internal (source=0x7f8d4d78, context=0x32323232, have_lock=have_lock@entry=0) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gmain.c:1159
#2  0xb6c34874 in g_source_destroy (source=<optimized out>) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gmain.c:1227
#3  0xb6e57042 in soup_session_dispose (object=0x7f5f0898) at soup-session.c:309
#4  0xb6ce49ce in g_object_unref (_object=0x7f5f0898) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./gobject/gobject.c:3133
#5  0xb6ef2282 in thread_closure_unref (thread_closure=0x7f601c18) at src/libostree/ostree-fetcher.c:143
#6  0xb6ef23e6 in ostree_fetcher_session_thread (data=0x7f601c18) at src/libostree/ostree-fetcher.c:486
#7  0xb6c50ffa in g_thread_proxy (data=0x7f5f5d20) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gthread.c:764
#8  0xb6f1bf88 in start_thread (arg=0xb62cb360) at pthread_create.c:311
#9  0xb6b261fc in ?? () at ../ports/sysdeps/unix/sysv/linux/arm/nptl/../clone.S:92 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

#0  0xb6c78244 in g_mutex_lock (mutex=mutex@entry=0xb9b9b9b9) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gthread-posix.c:1337
#1  0xb6c46382 in g_source_destroy_internal (source=0x7f789b00, context=0xb9b9b9b9, have_lock=have_lock@entry=0) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gmain.c:1159
#2  0xb6c46874 in g_source_destroy (source=<optimized out>) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gmain.c:1227
#3  0xb6e69042 in soup_session_dispose (object=0x7f5d5898) at soup-session.c:309
#4  0xb6cf69ce in g_object_unref (_object=0x7f5d5898) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./gobject/gobject.c:3133
#5  0xb6f04282 in thread_closure_unref (thread_closure=0x7f5c6c18) at src/libostree/ostree-fetcher.c:143
#6  0xb6f04eda in _ostree_fetcher_finalize (object=0x7f5cb3b0) at src/libostree/ostree-fetcher.c:537
#7  0xb6cf6a48 in g_object_unref (_object=0x7f5cb3b0) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./gobject/gobject.c:3170
#8  0xb6f05dd0 in _ostree_metalink_finalize (object=0x7f5cf6a0) at src/libostree/ostree-metalink.c:378
#9  0xb6cf6a48 in g_object_unref (_object=0x7f5cf6a0) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./gobject/gobject.c:3170
#10 0xb6eed54a in glnx_local_obj_unref (v=<synthetic pointer>) at ./libglnx/glnx-local-alloc.h:63
#11 ostree_repo_pull_with_options (self=self@entry=0x7f5c4078, remote_name_or_baseurl=<optimized out>, options=<optimized out>, progress=progress@entry=0x0, cancellable=cancellable@entry=0x0, 
    error=error@entry=0xbeca5fd0) at src/libostree/ostree-repo-pull.c:2176
#12 0x7f59da02 in ostree_builtin_pull (argc=2, argv=0xbeca6174, cancellable=<optimized out>, error=0xbeca5fd0) at src/ostree/ot-builtin-pull.c:260
#13 0x7f5988ec in ostree_run (argc=3, argv=0xbeca6174, commands=0x7f5b8008 <commands>, res_error=0xbeca6000) at src/ostree/ot-main.c:209
#14 0x7f593166 in main (argc=4, argv=0xbeca6174) at src/ostree/main.c:78

And a couple of SEGVs too:

#0  0xb6c9f240 in g_mutex_lock (mutex=mutex@entry=0x3747470) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gthread-posix.c:1337
#1  0xb6c6d382 in g_source_destroy_internal (source=0x7f74e778, context=0x3747470, have_lock=have_lock@entry=0) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gmain.c:1159
#2  0xb6c6d874 in g_source_destroy (source=<optimized out>) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gmain.c:1227
#3  0xb6e90042 in soup_session_dispose (object=0x7f597098) at soup-session.c:309
#4  0xb6d1d9ce in g_object_unref (_object=0x7f597098) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./gobject/gobject.c:3133
#5  0xb6f2b282 in thread_closure_unref (thread_closure=0x7f5a9018) at src/libostree/ostree-fetcher.c:143
#6  0xb6f2b3e6 in ostree_fetcher_session_thread (data=0x7f5a9018) at src/libostree/ostree-fetcher.c:486
#7  0xb6c89ffa in g_thread_proxy (data=0x7f59df20) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gthread.c:764
#8  0xb6f54f88 in start_thread (arg=0xb6304360) at pthread_create.c:311
#9  0xb6b5f1fc in ?? () at ../ports/sysdeps/unix/sysv/linux/arm/nptl/../clone.S:92 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

#0  0xb6bdd1da in g_source_unref_internal (source=0x7f6b2898, context=0x0, have_lock=have_lock@entry=0) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gmain.c:2012
#1  0xb6bde296 in g_source_unref (source=<optimized out>) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gmain.c:2063
#2  0xb6e00048 in soup_session_dispose (object=0x7f5c1098) at soup-session.c:310
#3  0xb6c8d9ce in g_object_unref (_object=0x7f5c1098) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./gobject/gobject.c:3133
#4  0xb6e9b282 in thread_closure_unref (thread_closure=0x7f5d2418) at src/libostree/ostree-fetcher.c:143
#5  0xb6e9b3e6 in ostree_fetcher_session_thread (data=0x7f5d2418) at src/libostree/ostree-fetcher.c:486
#6  0xb6bf9ffa in g_thread_proxy (data=0x7f5c8120) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gthread.c:764
#7  0xb6ec4f88 in start_thread (arg=0xb6279360) at pthread_create.c:311
#8  0xb6acf1fc in ?? () at ../ports/sysdeps/unix/sysv/linux/arm/nptl/../clone.S:92 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

#0  0xb6f2d44c in pending_uri_unref (pending=0x0) at src/libostree/ostree-fetcher.c:208
208   if (!g_atomic_int_dec_and_test (&pending->ref_count))
(gdb) bt
#0  0xb6f2d44c in pending_uri_unref (pending=0x0) at src/libostree/ostree-fetcher.c:208
#1  0xb6c6484a in g_hash_table_remove_all_nodes (hash_table=0x7f63ed88, notify=<optimized out>) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/ghash.c:503
#2  0xb6c64ede in g_hash_table_unref (hash_table=0x7f63ed88) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/ghash.c:1042
#3  0xb6f2d2d8 in thread_closure_unref (thread_closure=0x7f64e870) at src/libostree/ostree-fetcher.c:162
#4  0xb6f2deda in _ostree_fetcher_finalize (object=0x7f645398) at src/libostree/ostree-fetcher.c:537
#5  0xb6d1fa48 in g_object_unref (_object=0x7f645398) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./gobject/gobject.c:3170
#6  0xb6f1687a in ostree_repo_pull_with_options (self=self@entry=0x7f64c078, remote_name_or_baseurl=<optimized out>, options=<optimized out>, progress=progress@entry=0x0, cancellable=cancellable@entry=0x0, 
    error=error@entry=0xbeeb6fd0) at src/libostree/ostree-repo-pull.c:2853
#7  0x7f617a02 in ostree_builtin_pull (argc=3, argv=0xbeeb7174, cancellable=<optimized out>, error=0xbeeb6fd0) at src/ostree/ot-builtin-pull.c:260
#8  0x7f6128ec in ostree_run (argc=4, argv=0xbeeb7174, commands=0x7f632008 <commands>, res_error=0xbeeb7000) at src/ostree/ot-main.c:209
#9  0x7f60d166 in main (argc=5, argv=0xbeeb7174) at src/ostree/main.c:78

#0  0xb6c1c240 in g_mutex_lock (mutex=mutex@entry=0x50505050) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gthread-posix.c:1337
#1  0xb6bea382 in g_source_destroy_internal (source=0x7f75adc0, context=0x50505050, have_lock=have_lock@entry=0) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gmain.c:1159
#2  0xb6bea874 in g_source_destroy (source=<optimized out>) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./glib/gmain.c:1227
#3  0xb6e0d042 in soup_session_dispose (object=0x7f5b3898) at soup-session.c:309
#4  0xb6c9a9ce in g_object_unref (_object=0x7f5b3898) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./gobject/gobject.c:3133
#5  0xb6ea8282 in thread_closure_unref (thread_closure=0x7f5c3008) at src/libostree/ostree-fetcher.c:143
#6  0xb6ea8eda in _ostree_fetcher_finalize (object=0x7f5b9338) at src/libostree/ostree-fetcher.c:537
#7  0xb6c9aa48 in g_object_unref (_object=0x7f5b9338) at /build/glib2.0-5Ukv2U/glib2.0-2.42.1/./gobject/gobject.c:3170
#8  0xb6e923aa in ostree_repo_pull_with_options (self=self@entry=0x7f5c0078, remote_name_or_baseurl=0x7f5bfe58 "origin", options=<optimized out>, progress=progress@entry=0x0, cancellable=cancellable@entry=0x0, 
    error=error@entry=0xbebb3fc0) at src/libostree/ostree-repo-pull.c:2653
#9  0x7f58ba02 in ostree_builtin_pull (argc=2, argv=0xbebb4164, cancellable=<optimized out>, error=0xbebb3fc0) at src/ostree/ot-builtin-pull.c:260
#10 0x7f5868ec in ostree_run (argc=5, argv=0xbebb4164, commands=0x7f5a6008 <commands>, res_error=0xbebb3ff0) at src/ostree/ot-main.c:209
#11 0x7f581166 in main (argc=6, argv=0xbebb4164) at src/ostree/main.c:78

The packages are:

ii  libsoup2.4-1:armhf                     2.48.0-1                                                 armhf        HTTP library implementation in C -- Shared library
ii  libglib2.0-0:armhf                     2.42.1-1+b1                                              armhf        GLib library of C routines
ii  ostree                                 2016.6-2.1                                               armhf        content-addressed filesystem for operating system binaries

They all seem related to a refcounting/access after free issue. libglnx bug?

pruning tries to traverse into parent repos

Trying to prune my rpm-ostree pkg cache repo doesn't work because we try to delete objects that came from the parent (but fail because we try to unlinkat() in the pkgcache repo).

An OStree remote without DirectoryIndex fails with a `404 Not Found`

A remote that has been added, that has no index page, and therefore returns a 404, will return an error. This happens when a remote server does not offer a DirectoryIndex.

Either the user has to provide an empty index.html to make it work (index is not needed to retrieve the actual ostree). Or, ostree checks for the existence of [url]/config instead. If this would return a 200, it can be assumed the repository exists and 'works'.

don't execute static deltas for mirroring

Currently we abort:

15:11:37 + ostree --repo=repo remote delete centos-atomic-continuous
15:11:37 + ostree --repo=repo remote add --set=gpg-verify=false centos-atomic-continuous http://artifacts.ci.centos.org/sig-atomic/rdgo/centos-continuous/ostree/repo
15:11:37 + ostree --repo=repo pull --mirror --disable-fsync --depth=0 --commit-metadata-only centos-atomic-continuous centos-atomic-host/7/x86_64/devel/continuous
15:11:37 
15:11:37 OSTree:ERROR:src/libostree/ostree-repo.c:3017:_ostree_repo_read_bare_fd: assertion failed: (self->mode == OSTREE_REPO_MODE_BARE || self->mode == OSTREE_REPO_MODE_BARE_USER)
15:11:37 /home/builder/sig-atomic-buildscripts/centos-ci/libtoolbox.sh: line 3: 21574 Aborted                 ostree --repo=repo pull --mirror --disable-fsync --depth=0 --commit-metadata-only centos-atomic-continuous ${ref}

ostree admin prepare-kexec

We should optionally support kexec. See https://nicst.de/spinning-reboot.html and
https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2016-August/msg00005.html

Failing test test-admin-deploy-karg.sh

Quoting from the test:

Here we're asserting that the host system has a root= kernel
argument. I think it's unlikely that anyone doesn't have one, but
if this is not true for you, please file a bug!

$ cat /proc/cmdline 
initrd=\efi\nixos\zijhk6j55mszzcafhyqslfhmacxi4wlf-initrd-initrd.efi systemConfig=/nix/store/vh5ksfxaf62095s627lcl7zjhjpd04gl-nixos-system-nixos-16.09undefined init=/nix/store/vh5ksfxaf62095s627lcl7zjhjpd04gl-nixos-system-nixos-16.09undefined/init

A similar error occurs in tests/test-admin-instutil-set-kargs.sh.

EBADF during static delta apply

https://mail.gnome.org/archives/ostree-list/2016-July/msg00020.html

openat(3, "refs/remotes", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 15
newfstatat(15, "qt-os", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(15, "qt-os", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 19
close(15)                               = 0
newfstatat(19, "linux", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(19, "linux", O_WRONLY|O_DIRECTORY|O_CLOEXEC|0x400000) = 15
fallocate(15, 0, 0, 65)                 = -1 EOPNOTSUPP (Operation not supported)
fstat(15, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstatfs(15, {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=237798, f_bfree=44086, f_bavail=31798, f_files=61440, f_ffree=37789, f_fsid={297229479, -412021672}, f_namelen=255, f_frsize=4096}) = 0
pwrite(15, "\0", 1, 64)                 = 1
write(15, "6962c73eeebeedccb7fe05cdc57d5309"..., 65) = 65
newfstatat(19, "linux/qt", 0x7ffe6ea4b040, AT_SYMLINK_NOFOLLOW) = -1 EBADF (Bad file descriptor)
close(15)                               = 0
close(19)                               = 0
openat(8, ".", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 15
fstat(15, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl(15, F_GETFL)                      = 0x18800 (flags O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY)
fcntl(15, F_SETFD, FD_CLOEXEC)          = 0
getdents(15, /* 8 entries */, 32768)    = 400
newfstatat(15, "staging-c1e8de4e-d754-45db-9216-18b36ecf1451-YE7LLY-lock", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(15, "staging-c1e8de4e-d754-45db-9216-18b36ecf1451-YE7LLY-lock-lock", O_RDWR|O_CREAT|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC, 0600) = 19
fcntl(19, 0x25 /* F_??? */, 0x7ffe6ea4b140) = 0
fstat(19, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
unlinkat(15, "staging-c1e8de4e-d754-45db-9216-18b36ecf1451-YE7LLY-lock-lock", 0) = 0
close(19)                               = 0
newfstatat(15, "staging-c1e8de4e-d754-45db-9216-18b36ecf1451-9ZLGLY-lock", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(15, "staging-c1e8de4e-d754-45db-9216-18b36ecf1451-9ZLGLY-lock-lock", O_RDWR|O_CREAT|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC, 0600) = 19
fcntl(19, 0x25 /* F_??? */, 0x7ffe6ea4b140) = 0
fstat(19, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
unlinkat(15, "staging-c1e8de4e-d754-45db-9216-18b36ecf1451-9ZLGLY-lock-lock", 0) = 0
close(19)                               = 0
newfstatat(15, "staging-c1e8de4e-d754-45db-9216-18b36ecf1451-AGWYLY-lock", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(15, "staging-c1e8de4e-d754-45db-9216-18b36ecf1451-AGWYLY-lock-lock", O_RDWR|O_CREAT|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC, 0600) = 19
fcntl(19, 0x25 /* F_??? */, 0x7ffe6ea4b140) = 0
fstat(19, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
unlinkat(15, "staging-c1e8de4e-d754-45db-9216-18b36ecf1451-AGWYLY-lock-lock", 0) = 0
close(19)                               = 0
newfstatat(15, "fetcher-341DLY-lock", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(15, "fetcher-341DLY-lock-lock", O_RDWR|O_CREAT|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC, 0600) = 19
fcntl(19, 0x25 /* F_??? */, 0x7ffe6ea4b140) = 0
fstat(19, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
unlinkat(15, "fetcher-341DLY-lock-lock", 0) = 0
close(19)                               = 0
newfstatat(15, "cache", {st_mode=S_IFDIR|0775, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(15, "fetcher-341DLY", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(15, "fetcher-341DLY-lock", O_RDWR|O_CREAT|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC, 0600) = 19
fcntl(19, 0x25 /* F_??? */, 0x7ffe6ea4b140) = -1 EAGAIN (Resource temporarily unavailable)
close(19)                               = 0
getdents(15, /* 0 entries */, 32768)    = 0
close(15)                               = 0
close(14)                               = 0
unlinkat(8, "staging-c1e8de4e-d754-45db-9216-18b36ecf1451-YE7LLY-lock", 0) = 0
close(16)                               = 0
write(13, "\1\0\0\0\0\0\0\0", 8)        = 8
futex(0x65e460, FUTEX_WAKE, 2147483647) = 0
futex(0x65e450, FUTEX_WAKE, 1)          = 1
close(13)                               = 0
close(17)                               = 0
unlinkat(15, "fetcher-341DLY-lock", 0)  = -1 EBADF (Bad file descriptor)
close(18)                               = 0
write(1, "\n", 1
)                       = 1
close(3)                                = 0
close(8)                                = 0
close(9)                                = 0
close(7)                                = 0
unlinkat(5, "ostree/lock", 0)           = 0
close(6)                                = 0
close(5)                                = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
write(2, "\33[31m\33[1merror: \33[22m\33[0mBad fil"..., 45error: Bad file descriptor
) = 45
exit_group(1)                           = ?
+++ exited with 1 +++

First-class support for testing OSTree deployments in a lightweight container

From an IRC conversation on the topic:

[2016-04-19T16:16:10-0400] <walters> systemd-nspawn --register=no -D /ostree/deploy/centos-atomic-host/deploy/378538275fd9029a92435dc8c1efdb7a3486b04de47a2e6ac96752015b8352e5.0 /bin/bash
[2016-04-19T16:16:23-0400] <walters> almost worked...except nspawn barfs at the tmp -> sysroot/tmp symlink
[2016-04-19T16:17:12-0400] <walters> this works as is: systemd-nspawn --bind /tmp:/sysroot/tmp --register=no -D /ostree/deploy/centos-atomic-host/deploy/378538275fd9029a92435dc8c1efdb7a3486b04de47a2e6ac96752015b8352e5.0 /bin/bash
[2016-04-19T16:17:34-0400] <walters> that directory is the previous booted deployment
[2016-04-19T16:17:56-0400] <walters> it'd probably make sense to teach nspawn about the ostree conventions

In my case, adding the --bind /tmp:/sysroot/tmp doesn't seem to help the container boot (systemd-nspawn just gives an opaque error and exits) but if I manually remove /tmp from the tree it boots up properly.

I then run into issues with private networking, but that probably has more to do with the tree I'm using rpm-ostree to build than raw OSTree.

Symlink for test is brittle

Follow-up from #378

Short version, it is possible for the tests not to run against the correct binary.

Instead of running the tests against a specific ostree binary, the tests create a symlink, in a directory, and add that directory to a PATH. Except that depending on libtool, it's possible to screw this up royally.

$ ls -l tests/ostree
lrwxrwxrwx 1 chip chip 33 Jul 11 10:48 tests/ostree -> /home/chip/ostree/.libs/lt-ostree
$ ls -l /home/chip/ostree/.libs/lt-ostree
ls: cannot access /home/chip/ostree/.libs/lt-ostree: No such file or directory
# Symlink likely created in a previous build
$ ls -l /home/chip/ostree/.libs/ostree
-rwxr-xr-x 1 chip chip 778588 Jul 11 14:53 /home/chip/ostree/.libs/ostree
# Yep, that does exist
$ PATH=`pwd`/tests:$PATH ostree
Usage:
  ostree [OPTION...] COMMAND
<snip>
$ PATH=`pwd`/tests:$PATH pull-test.sh
1..11
/home/chip/ostree/tests/pull-test.sh: line 23: test_tmpdir: unbound variable
# Just to double-check that the PATH is taken into account

If tests/ostree is a dangling symlink, then the tests will be run against whatever ostree is first in your path.