Sjoerd Simons [Mon, 17 Oct 2016 20:30:41 +0000 (22:30 +0200)]
OsreeFetcher: Treat 403 as not found
Private Cloudfront instances return 403 for objects which don't exist
rather then a 404.
Change the fetcher to assume 403 is ok for download that are "optional"
rather then erroring out at that step (e.g. trying to download a static
delta if the remote repo doesn't have those)
Sjoerd Simons [Mon, 17 Oct 2016 19:39:38 +0000 (21:39 +0200)]
pull: Add per-remote cookie jar
Optionally read cookie jars for a remote to be used when downloading
data. This can be used for private repositories which require specific
cookies to be present, e.g. repositories hosted on Amazon cloudfront
using signed cookies.
Colin Walters [Tue, 1 Nov 2016 18:17:57 +0000 (14:17 -0400)]
repo: Don't put remote refs in the summary file
I was doing a chain of mirroring like A -> B -> C
And repo B had A as a remote. When I added B as
a remote to C, the summary file of B had a ref
upstream:foo/bar/baz, which caused all pulls from
B to C to fail, since the summary file is only
expected to have refs, not refspecs.
Sjoerd Simons [Sun, 30 Oct 2016 20:06:27 +0000 (21:06 +0100)]
Filter bootloader supplied kernel cmdline options
Various bootloader add kernel commandline options dynamically, filter
these out when grabbing boot options from /proc/cmdline. Specifically
grub adds BOOT_IMAGE and systemd-boot adds initrd.
Simon McVittie [Sat, 29 Oct 2016 17:47:11 +0000 (18:47 +0100)]
Fix some leaks of floating GVariants
ostree_repo_pull_with_options() and ostree_repo_remote_change() don't
sink floating GVariant arguments, and doing so now would be an
ABI change; so don't rely on them to do so.
Leak found with valgrind memcheck.
Signed-off-by: Simon McVittie <smcv@debian.org> Closes: #556
Approved by: cgwalters
Simon McVittie [Sun, 30 Oct 2016 14:14:09 +0000 (14:14 +0000)]
ostree_admin_option_context_parse: explicitly clean up when exiting early
The cleanup attribute doesn't clean up before calling a noreturn
function like exit(). Explicitly clean up the pointer variables
(but don't assume that a simple g_object_unref() would be OK either,
in case the behaviour of the cleanup attribute changes).
This isn't a real leak since we're about to exit anyway, but if we
don't fix it then valgrind memcheck will make the tests fail.
Signed-off-by: Simon McVittie <smcv@debian.org> Closes: #559
Approved by: cgwalters
Simon McVittie [Sat, 29 Oct 2016 20:31:18 +0000 (21:31 +0100)]
_ostree_kernel_args_replace_take: don't leak when replacing
If !existed, then we add arg to kargs->order, where it will be freed
by that array's free-function. However, if the kernel argument did
already exist, we have to either free arg ourselves (and make sure
the old key is what appears in the hash table), or do a linear search
on kargs->order to replace the old key with the new.
Leak found by valgrind memcheck.
Signed-off-by: Simon McVittie <smcv@debian.org> Closes: #559
Approved by: cgwalters
Simon McVittie [Fri, 7 Oct 2016 22:09:29 +0000 (23:09 +0100)]
Distribute test scripts even if we wouldn't run them
This fixes a "make dist" tarball produced on a minimal system and run on a
non-minimal system. Automake knows that files that are only conditionally
included in dist_whatever_WHATEVER are to be distributed, but it does not
do the same for files that are only conditionally included in EXTRA_DIST,
which is how glib-tap.mk's various variables like dist_test_scripts work.
Signed-off-by: Simon McVittie <smcv@debian.org> Closes: #557
Approved by: cgwalters
Simon McVittie [Sat, 8 Oct 2016 13:19:01 +0000 (14:19 +0100)]
Force C.UTF-8 or C locale for tests
Otherwise several tests fail, for example in this build done in a French
locale by Debian's reproducible builds initiative, to check whether
the resulting binaries are identical to what was produced in an
English locale:
(test-pull-metalink)
# error: Erreur à la ligne 1, caractère 1 : Le document doit commencer avec un élément (par ex. <book>)
...
File 'err.txt' doesn't match regexp 'Document must begin with an element'
Signed-off-by: Simon McVittie <smcv@debian.org> Closes: #558
Approved by: cgwalters
static-delta: find a similar filename using what is before '.' or '-'
Improve the heuristic to use only the part before the first '.' when
looking for a similar file in the current directory.
last versions of dracut generate reproducible initramfs files, but we
still fallback to the full file download if there is any minimal
change that causes a different checksum and file name.
This change extends that case to deal better with similar files that
have a different suffix.
This is the difference generating a static delta from
fedora-atomic/f24/x86_64/docker-host to fedora-atomic/f24/x86_64/testing/docker-host
Jonathan Lebon [Thu, 27 Oct 2016 16:03:04 +0000 (12:03 -0400)]
.redhat-ci.yml: add clang
Clang has better detection for unused vars when using auto cleanup
functions. We should eventually just fold this back into the first
testsuite. But let's just turn it on for now, at least until it's
satisfied with the whole codebase.
Colin Walters [Tue, 25 Oct 2016 14:40:45 +0000 (10:40 -0400)]
tests: Skip libarchive/selinux tests if in container without SELinux
I'm doing builds and `make check` inside a Docker container, with
selinux on as a build-time option, but no policy in the container.
This currently aborts. Let's not do that.
(This type of thing is why installed tests are a better model)
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.
In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.
Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.
This fixes https://github.com/ostreedev/ostree/issues/541
If there is a transaction active, then we put writes to detached
metadata into the staging dir, and when reading it we look there
first. This allows transactions to be aborted half-way without
writing the detached metadata into the repository (possibly
overwriting any old metadata from there).
This fixes https://github.com/ostreedev/ostree/issues/526
Commit 1d4f1b8878addd28059c3a3928640491755cd615 started using hardlinks
checkouts of symlinks. However, symlinks are not stored as symlink in the
repo for bare-user repos, so this breaks user-mode checkouts of such repos.
We fix this by checking for !is_symlink in the bare-user case.
This fixes:
https://github.com/ostreedev/ostree/issues/537
Jonathan Lebon [Wed, 19 Oct 2016 16:37:54 +0000 (12:37 -0400)]
add .redhat-ci.yml and .redhat-ci.Dockerfile
Add a YAML file for the new Red Hat CI framework.
Rather than re-installing all the build deps everytime, which takes
time, I added a Dockerfile that we can wire up to the Docker Hub. For
now it lives at `jlebon/ostree-tester:rhci`, but we can move it under
the `projectatomic` org (or a new `ostree` org).
Colin Walters [Tue, 4 Oct 2016 19:23:04 +0000 (15:23 -0400)]
core: Do create hardlinks to symlinks for checkouts
I was noticing a recent performance issue with checkouts
which seemed to be mostly us going back to doing a `fsync()` on
directories.
Regardless, while looking at that, I saw we were spending time
creating new symlinks. Even though symlinks are small, it's still
better to hardlink them.
Going way back in time, the reason we weren't doing this is
because we were hitting `EMFILE` on ext4, but that was for
gnome-continuous which creates *many* build roots. Even
there though, they're just a cache, and we handle `EMFILE`.
For ostree-for-host-system, we don't expect to have many roots (just 3
at most transiently), so hardlinking symlinks does make sense.
Jonathan Lebon [Fri, 14 Oct 2016 13:25:01 +0000 (09:25 -0400)]
OstreeFetcher: provide proxy credentials if needed
There seems to be an issue in libsoup which causes basic auth
credentials to not be passed to the proxy during requests. We thus have
to handle PROXY_UNAUTHORIZED responses and provide the auth ourselves.
Jonathan Lebon [Thu, 13 Oct 2016 16:05:54 +0000 (12:05 -0400)]
ostree-sysroot-deploy.c: delete redundant check
Just noticed this while inspecting the code. The deployments retrieved
by `_ostree_sysroot_list_deployment_dirs_for_os` will forcibly already
have a matching osname since it indirectly uses that same variable to
construct them. Having a check there makes it look like there may be
subtle corner cases, when there aren't.
Jonathan Lebon [Thu, 13 Oct 2016 15:56:46 +0000 (11:56 -0400)]
docs: amend vmlinuz & initramfs naming convention
I was confused while reading the docs how this could work, since in at
least the Fedora/CentOS/RHEL distros, they're named e.g.
initramfs-`uname -r`-$checksum.
If this is true, don't initiate, abort of commit a transaction, instead
it is assumed that the caller initiated the transaction, and that it
will eventually be commited.
This allows you to do multiple pulls or a combination of pulls and
commits in a single transaction.
Colin Walters [Wed, 5 Oct 2016 14:13:33 +0000 (10:13 -0400)]
libglnx: Update to latest
This pulls in a new compilation flag for wrpseudo compatibility. Also
note we need to add some includes since glnx-libcontainer went away,
and with it some includes for `sys/mount.h` etc.
Colin Walters [Wed, 14 Sep 2016 20:08:24 +0000 (16:08 -0400)]
trivial-httpd: Port mostly to fd-relative
We were seeing some weird potential memory corruption in this code
when using it for `rpm-ostree-toolbox installer`, which is almost
certainly not its fault, but let's use it as an excuse to port
(mostly) to fd-relative and away from GFile.
Dropping the last GFile use here is a bit tricky as it does have a
nice high level wrapper around inotify.
Colin Walters [Tue, 4 Oct 2016 19:39:15 +0000 (15:39 -0400)]
checkout: Fix fsync defaults for new API to be off for real
My previous change in https://github.com/ostreedev/ostree/pull/425
actually broke things so we basically used the repository defaults =(
This is a subtle mess since we're only trying to flip things off
for the *new* API.
Clean this up so that the "default repo inheritance" lives only in one
place - in the compat layer for the old checkout API. The new
checkout API defaults to off period, so the repository state is
irrelevant.
Colin Walters [Wed, 14 Sep 2016 00:38:21 +0000 (20:38 -0400)]
sysroot: Drop an unnecessary fsync
While looking at a slow update issue (which I'm guessing is
unpredictable I/O latency in an OpenStack instance), I noticed
in one of the traces we were inside a fsync here.
Dropping the fsync here is just another of a long series of unwinding
them - we `syncfs()` the sysroot fd and `/boot` and we have a big
`sync()` anyways.
Colin Walters [Wed, 14 Sep 2016 01:09:48 +0000 (21:09 -0400)]
sysroot: Drop an fsync for origin file when writing deployments
More fsync pruning. Since we have a public API for writing the origin
file and it did a fsync before, let's preserve that. But when writing
deployments as part of a full transaction, we rely on the global
`syncfs()`, so add an internal function for origin file writing that
doesn't.
Owen W. Taylor [Tue, 13 Sep 2016 13:05:52 +0000 (09:05 -0400)]
ostree_sysroot.c: Don't close sysroot_fd twice.
If ostree_sysroot_unload() was called explicitly, then sysroot_fd
would be closed again at finalization time, possibly closing a
random file descriptor belonging to some other part of the
application.
Owen W. Taylor [Tue, 13 Sep 2016 13:03:53 +0000 (09:03 -0400)]
ostree-repo.c: Fix file descriptor cleanup
0 was used as an "unset" flag for tmp_dir_fd, which is technically
incorrect. For cache_dir_fd, -1 was used as the sentinal but 0
was checked for, resulting in close(-1).
Colin Walters [Mon, 12 Sep 2016 00:33:42 +0000 (20:33 -0400)]
pull: Do allow executing deltas when mirroring into bare{,-user}
In https://github.com/ostreedev/ostree/pull/408 we fixed a
bug where we would crash when trying to execute deltas into
an archive repo (which isn't presently supported).
But that was overly aggressive - we obviously *can* execute deltas
when mirroring into a bare repo. This should fix a regression with
the way flatpak uses mirroring to pull from a user repo into the
system.
Colin Walters [Thu, 8 Sep 2016 17:38:08 +0000 (13:38 -0400)]
sysroot: Avoid double cleanup, and ensure no cleanup if specified
Since forever, we've been doing two cleanups. In
https://github.com/ostreedev/ostree/commit/8ece4d6d51bdbe3e41ab318259276bb83e553aa0
I thought we were doing just one and wanted to go to zero (if specified),
but I actually just dropped one cleanup.
In https://github.com/projectatomic/rpm-ostree/pull/452
@jlebon pointed out the duplication. Fix this by creating a new internal
deploy wrapper that takes cleanup flags.
(Since we already had the "piecemeal cleanup" API internally, let's
frame it in terms of that, rather than passing down a boolean).
Colin Walters [Thu, 8 Sep 2016 16:39:42 +0000 (12:39 -0400)]
fetcher: Fix another finalization deadlock
If the current repo is already up to date (we have no content to
fetch), it's possible for the fetcher to not request any URIs. So
create and then finalize it quickly.
Finalization involves calling `g_main_loop_quit()` +
`g_thread_wait()`. However, if `g_main_loop_quit()` is run *before*
`g_main_loop_run()`, we'll deadlock because GMainLoop assumes in
`_run()` to start things.
This is a common trap - ideally, GMainLoop would record if `_quit()`
was called before `_run()` or something, but doing that now would
likely break people who are expecting quit() -> run() to restart.
In general, we've moved in various GLib-consuming apps to an
explicit "main context iteration with termination condition" model;
see `pull_termination_condition()` in the pull code.
This fixes this race condition.
I verified that an assertion in `_finalize` that more than
zero URIs were requested was hit in multiple test cases, and this patch
has survived a while of make check loops.
Colin Walters [Thu, 1 Sep 2016 20:09:30 +0000 (16:09 -0400)]
repo: Only use mmap() for metadata > 16k
See http://stackoverflow.com/questions/258091/when-should-i-use-mmap-for-file-access
and
https://lwn.net/Articles/591978/
I didn't really notice much performance difference in some small
tests, but I happened to be stracing and realized we were `mmap()`ing
even for 50 bytes which is not very useful, so let's not do it.
mtime of 0 has been the semantics of ostree deployments from basically
the beginning of the project. We (and others, see
flatpak/flatpak@b5204c9) rely on that fact when generating trees.
In particular, this affects caches that use the mtime of the
associated file or directory to determine if the cache is valid. By
arbitrarily changing the mtime of the files to something else, all
the caches we setup in the build are now invalidated. Preseeding
caches is really important to the user experience as it avoids
having the user wait while they're regenerated on first run.
Now, we could change our build infrastructure to preset all the
mtimes to 1 to match this change, but what does that do for our
existing users who are on an ostree that deploys with mtimes of 0?
We could just revert this change at Endless (and the associated one
in Flatpak), and that would be fine for our users. However, if we
point non-Endless users to our apps, they'll have the great
experience of waiting 10 seconds the first time they launch it while
the fontconfig cache is rebuilt unnecessarily.
William Manley [Wed, 31 Aug 2016 16:15:48 +0000 (17:15 +0100)]
ostree-prepare-root: Fix running with musl
musl libc's implementation of `realpath` works by opening the path and then
doing a lookup in `/proc/self/fd` to find the canonical path. This fails
if `/proc` is not mounted. This causes problems for us if
`ostree-prepare-root` is `init` as `/proc` won't be mounted.
We have to mount `/proc` anyway for `/proc/cmdline` so this fix just
expands the scope over which `/proc` is mounted to include both our
`realpath` calls.
William Manley [Wed, 31 Aug 2016 16:07:17 +0000 (17:07 +0100)]
ostree-prepare-root: Error if realpath fails
I've seen it fail with musl which needs `/proc` to be mounted for it to
work. The error messages we're rather confusing before. At least this
now points to the right location.
William Manley [Wed, 31 Aug 2016 17:14:19 +0000 (18:14 +0100)]
switchroot: Fix test-switchroot now autotools can build static
This test previously depended on manually building ostree-prepare-root.
Since 42dab85 we've been able to build static binaries with the usual
autotools build-system. This change reflects the fact that
`ostree-prepare-root` is built into $srcdir rather than `src/switchroot`
where I was building manually.
This test now passes with `./configure --with-static-compiler=gcc` (glibc)
but still fails with `./configure --with-static-compiler=musl-gcc` (musl).
Dan Nicholson [Thu, 1 Sep 2016 18:26:48 +0000 (11:26 -0700)]
build: Set --enable-man during distcheck
If xsltproc is not installed, then ENABLE_MAN will be false and the
generated man pages won't be distributed. Pass --enable-man to enforce
that the man pages will be generated and distributed.
Jonathan Lebon [Tue, 23 Aug 2016 19:55:15 +0000 (15:55 -0400)]
pull: add mirrorlist support
This commit adds mirrorlist support to the fetcher. Users can now
prepend url or/and contenturl by mirrorlist= to interpret the link as a
mirrorlist.
If an object is not found, the fetcher will automatically try the next
mirror in the order given in the list (assuming the order returned by
the server is significant).
Jonathan Lebon [Mon, 22 Aug 2016 20:14:52 +0000 (16:14 -0400)]
pull: drop fetching_sync_uri
This made sense back when we used a main loop even when we needed to
fetch objects synchronously. Nowadays, we no longer actually update
progress before the FETCHING_OBJECTS phase, which is only for async
requests.
This allows us to get rid of fetch_uri_contents_membuf_sync() and to
generalize fetch_uri_contents_utf8_sync() so that it only requires a
fetcher. This will be needed later.
Jonathan Lebon [Thu, 11 Aug 2016 20:01:27 +0000 (16:01 -0400)]
pull code: support contenturl setting
Allow users to pass a --contenturl during `remote add` and store it in
the remote config.
Fish out the contenturl setting from the remote config and use it when
downloading static deltas and objects (except for commit signatures).
The idea here is that items in the trust chain (summary & sigs) can be
fetched from a more secure e.g. TLS-pinned location, while objects
themselves are fetched from another location. Once mirrorlist support is
added, this use-case will become even more advantageous.
OSTree function ostree_repo_pull_with_options starts a
series of operations that makes heavy use of the PullData's
cancellable.
This isn't effective, however, since nowhere in the code
the OtPullData.cancellable field is set. This is visible,
for example, when trying to cancel a Flatpak pull and nothing
happens, because the cancellable is not properly passed
to the pull data.
Fix that by setting the cancellable field of the pull data. It
owns a reference for safety reasons, and unreferences it at the
end of the operation.
William Manley [Tue, 30 Aug 2016 21:36:20 +0000 (22:36 +0100)]
switchroot: Replace custom error printing with err/warn functions from libc
The `warn()` libc extension has exactly the same behaviour as our own
`perrorv` function, but is available in (at least) glibc and musl. As an
added bonus the similar function `err()` which will exit with an error
code afterwards.
This implementation is tidier and allows us to get rid of our own
`perrorv`. It paves the way to removing `ostree-mount-util.c` to simplify
the build scripts.
William Manley [Tue, 30 Aug 2016 19:28:02 +0000 (20:28 +0100)]
ostree-prepare-root: Allow building statically with musl
If the `--with-static-compiler=musl-gcc` configure flag is given.
ostree-prepare-root can be used as init in a system without a populated
/lib. To support this use case we need to link statically as we will be
unable to locate libc.so at run time if it's not installed in /lib.
We support building ostree-prepare-root with a different compiler to the
rest of ostree so we can use musl rather than glibc. This reduces the size
of the executable significantly: from ~700K -> ~30K. We have to use
`_SCRIPTS` here to get autotools to install this as an executable but
without generating rules to make it itself which we have specified
manually.
See https://lists.gnu.org/archive/html/help-gnu-utils/2007-01/msg00007.html
for advice on using autotools in this manner.
William Manley [Tue, 26 Jul 2016 16:42:17 +0000 (17:42 +0100)]
switchroot: Fix building with musl libc
POSIX and GNU define conflicting versions of `strerror_r`. The GNU
version returns the string but doesn't necessilary write into buf.
The POSIX version writes into buf and returns the length but doesn't
necessilary append a terminate the string with a NUL if it's too long
to fit in buf.
This commit fixes building ostree-prepare-root with musl libc. The
stripped static build with musl on my machine is 30K vs. 724K with glibc
static and 11K with glibc shared.
Colin Walters [Mon, 29 Aug 2016 02:22:59 +0000 (22:22 -0400)]
sysroot: Add a flag to suppress post-deploy cleanup
I noticed seeing the output of `prune` twice in rpm-ostree, and had
always wondered why. When reading the rpm-ostree code to fix
something else, reasons, I noticed the reason - we were pruning once
here, and then once after rpm-ostree regenerates its "base" refs.
There's no reason to clean twice, so let's add a flag so rpm-ostree
can suppress doing it inside libostree.
Colin Walters [Sat, 27 Aug 2016 12:50:47 +0000 (08:50 -0400)]
sysroot: Drop unnecessary `dup()` invocation
It's close-on-exec, not close-on-fork. I was clearly confused when
writing this; it works just fine to reference the fd in the child and
`fchdir()` before exec. So drop the unnecessary duplication.
Just noticed this while reading the code for a random other reason.
Gatis Paeglis [Tue, 23 Aug 2016 12:32:35 +0000 (14:32 +0200)]
u-boot: Merge ostree's and systems uEnv.txt
This is a proper fix for:
https://bugzilla.gnome.org/show_bug.cgi?id=755787
With this patch, an admin (system builder) can now:
1) Edit /usr/lib/ostree-boot/uEnv.txt
2) Deploy the new tree. OSTree will append system's uEnv.txt
to the OSTree's managed uEnv.txt (loader/uEnv.txt).
It is common for u-boot systems to read in an extra env
from external /uEnv.txt. The same file OSTree uses to pass
in its env. With this patch /uEnv.txt now contains OSTree's
env + custom env added by system builders.
pull: use same name for parameter and documentation comment
Fixes this warning:
src/libostree/ostree-repo-pull.c:2162: Warning: OSTree: ostree_repo_pull_with_options: unknown parameter 'remote_name_or_baseurl' in documentation comment, should be 'remote_name'
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #472
Approved by: jlebon
Colin Walters [Thu, 25 Aug 2016 15:53:27 +0000 (11:53 -0400)]
lib: Add an API to list only "our" objects, fix prune to use it
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail. There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.
projects / ostree.git / log