William Manley [Tue, 26 Jul 2016 16:42:17 +0000 (17:42 +0100)]
switchroot: Fix building with musl libc
POSIX and GNU define conflicting versions of `strerror_r`. The GNU
version returns the string but doesn't necessilary write into buf.
The POSIX version writes into buf and returns the length but doesn't
necessilary append a terminate the string with a NUL if it's too long
to fit in buf.
This commit fixes building ostree-prepare-root with musl libc. The
stripped static build with musl on my machine is 30K vs. 724K with glibc
static and 11K with glibc shared.
Colin Walters [Mon, 29 Aug 2016 02:22:59 +0000 (22:22 -0400)]
sysroot: Add a flag to suppress post-deploy cleanup
I noticed seeing the output of `prune` twice in rpm-ostree, and had
always wondered why. When reading the rpm-ostree code to fix
something else, reasons, I noticed the reason - we were pruning once
here, and then once after rpm-ostree regenerates its "base" refs.
There's no reason to clean twice, so let's add a flag so rpm-ostree
can suppress doing it inside libostree.
Colin Walters [Sat, 27 Aug 2016 12:50:47 +0000 (08:50 -0400)]
sysroot: Drop unnecessary `dup()` invocation
It's close-on-exec, not close-on-fork. I was clearly confused when
writing this; it works just fine to reference the fd in the child and
`fchdir()` before exec. So drop the unnecessary duplication.
Just noticed this while reading the code for a random other reason.
Gatis Paeglis [Tue, 23 Aug 2016 12:32:35 +0000 (14:32 +0200)]
u-boot: Merge ostree's and systems uEnv.txt
This is a proper fix for:
https://bugzilla.gnome.org/show_bug.cgi?id=755787
With this patch, an admin (system builder) can now:
1) Edit /usr/lib/ostree-boot/uEnv.txt
2) Deploy the new tree. OSTree will append system's uEnv.txt
to the OSTree's managed uEnv.txt (loader/uEnv.txt).
It is common for u-boot systems to read in an extra env
from external /uEnv.txt. The same file OSTree uses to pass
in its env. With this patch /uEnv.txt now contains OSTree's
env + custom env added by system builders.
pull: use same name for parameter and documentation comment
Fixes this warning:
src/libostree/ostree-repo-pull.c:2162: Warning: OSTree: ostree_repo_pull_with_options: unknown parameter 'remote_name_or_baseurl' in documentation comment, should be 'remote_name'
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #472
Approved by: jlebon
Colin Walters [Thu, 25 Aug 2016 15:53:27 +0000 (11:53 -0400)]
lib: Add an API to list only "our" objects, fix prune to use it
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail. There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.
We used to only look at the "gpg-verify" and "gpg-verify-summary"
options when we're passed a local URL. Make these options also have an
effect in the configured remote case.
Colin Walters [Tue, 16 Aug 2016 13:26:16 +0000 (09:26 -0400)]
rofiles-fuse: Rework to be based on nlink
Programs like `useradd` try to `open(/etc/passwd, O_RDWR)` to append,
which didn't work with rofiles-fuse. Thinking about this, I realized
that there's a simpler algorithm for "can we write to this file" which
is "does it have a hardlink count <= 1"?
Switching to this both drops complexity (we no longer need to keep a
hash table of files we created), and also lets useradd work.
Colin Walters [Tue, 16 Aug 2016 17:07:51 +0000 (13:07 -0400)]
test-rofiles-fuse: Actually check out via hardlinks
The test suite was actually doing something before, just
not quite what I intended. Without `-U` for bare-user checkouts
we end up doing a copy.
Now, a future commit will change how rofiles work, which would cause
the test suite to permit inplace mutation for non-hardlinked files.
So let's ensure they *are* hardlinked.
Dan Nicholson [Thu, 11 Aug 2016 19:39:19 +0000 (12:39 -0700)]
repo: Really ignore progress changed user data
The documentation says this is ignored, implying that you should pass
NULL to it. However, the function immediately returns in this case even
though the argument isn't used anywhere.
Gatis Paeglis [Tue, 9 Aug 2016 13:23:04 +0000 (15:23 +0200)]
Move ostree-* executables to /usr/lib/ostree
Why not to use libexecdir?
Because this directory does not exist on some distros or paths
between distros varies. There are several reasons why a well
known path is prefered, for example when generating a kernel
command line:
Dan Nicholson [Tue, 9 Aug 2016 22:41:36 +0000 (15:41 -0700)]
tests: Test that local pulls do not use deltas
Pulls from local repos now default to disabling static deltas so that
objects are copied. To check this is the case, see if the object files
are hardlinked after pulling.
Dan Nicholson [Fri, 5 Aug 2016 13:52:55 +0000 (06:52 -0700)]
pull: Disable static deltas by default for local pulls
For local pulls there's no benefit pulling the static delta over the
individual object files since there's no HTTP overhead. Furthermore,
processing deltas always generates the objects whereas a standard pull
ensures that the exact object files are copied. Using deltas also
prevents hardlinking the objects if the repos exist on the same
filesystem.
Dan Nicholson [Tue, 9 Aug 2016 22:39:41 +0000 (15:39 -0700)]
tests: Ensure deltas for pulling when needed
When testing pulling of deltas, use the new --require-static-deltas
option to pull-local to ensure that deltas are actually used. To support
the require-static-deltas mode, the summary in the remote repo must be
generated.
Colin Walters [Thu, 23 Jun 2016 00:12:31 +0000 (20:12 -0400)]
Final excision of libgsystem dependency
Lots and lots of preparation led to this moment - when nothing
apparent changes for users! Woo!
But seriously, having the extra dependency is a minor annoyance, and
in the big picture I think the libgsystem idea was wrong - we need to
land things in GLib, and use git submodules for API-unstable or
Linux-specific sharing. For a lot of OSTree, the libgsystem `GFile*`
orientation was also wrong, we really want fd-relative.
Colin Walters [Mon, 8 Aug 2016 19:41:09 +0000 (15:41 -0400)]
prune: Retain the tip of each ref even with date pruning
I hit an error with [CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel)
where we were doing time-based pruning. `ostree summary -u` started failing,
and it took me a bit to realize it was because we were pruning
even the tip of old branches, which I was not at all expecting,
and I don't think users will too.
Perhaps in the future we could add some sort of --prune-ref-tips or
something if people wanted it, but I doubt it.
Colin Walters [Thu, 4 Aug 2016 13:37:26 +0000 (09:37 -0400)]
deploy: Use internal recursive copy rather than libgsystem
Since we already had a "recursive copy" implementation here, let's
reuse it rather than the libgsystem `gs_shutil_cp_a()`. Part of the
libglnx porting.
Colin Walters [Fri, 5 Aug 2016 11:08:14 +0000 (07:08 -0400)]
repo: Drop more internally unused GFile members
I forgot to actually remove `config_file` in the previous
commit, the txn lock hasn't been used in a long time, and
for the uncompressed cache, everything uses the fd already.
Simon McVittie [Fri, 5 Aug 2016 21:12:55 +0000 (22:12 +0100)]
travis: run the test suite on various distributions
.travis.yml is obviously still Travis-specific, but tests/ci-* are
designed to be shareable with other CI environments if there is interest
in doing so.
At the moment I'm only testing on Debian and Ubuntu. In principle we
could try a non-Debian-derived Docker container such as Fedora or CentOS
inside travis-ci's Ubuntu environment, similar to what I'm doing
for Debian, but I don't know the correct setup commands to use there.
Simon McVittie [Fri, 5 Aug 2016 19:41:05 +0000 (20:41 +0100)]
Skip tests that use whiteouts under Docker/aufs
ostree's naming convention for whiteouts is similar to what is
done in aufs, which means we can't compose the trees to test this
feature under Docker with the aufs storage driver, as used on
travis-ci.
Signed-off-by: Simon McVittie <smcv@debian.org> Closes: #437
Approved by: cgwalters
Colin Walters [Thu, 4 Aug 2016 20:35:49 +0000 (16:35 -0400)]
lib: Fix a compiler warning introduced from earlier patch
I think we'd end up closing stdin...yuck. Need to investigate
having this be fatal, but CentOS 7 `gcc-4.8.5` doesn't understand
`-Werror=int-conversion`.
Colin Walters [Thu, 4 Aug 2016 11:22:41 +0000 (07:22 -0400)]
lib: Add padding booleans to OstreeRepoCheckoutAtOptions
Since this is a new API, and adding booleans is the most likely thing
we'll do, let's stick some explicit padding for them in here now.
We could use the `unused_ints[]` but it'd be out of order, and this
will more clearly remind people about the padding. The efficiency hit
versus bitfields is annoying, but oh well, not a real world problem.
Colin Walters [Wed, 3 Aug 2016 14:55:35 +0000 (10:55 -0400)]
repo: Flip the fsync default to off for new checkout API
Since we're adding a new API, we have the opportunity to fix
the defaults. We expect clients to do a `syncfs()` or equivalent
on their own now, since it's way more efficient.
Colin Walters [Tue, 2 Aug 2016 18:49:42 +0000 (14:49 -0400)]
deltas: Port compilation to libglnx/fd-relative
This was the last use of libgsystem tmpfile APIs. The change here is
a bit uglier than it needs to be because we support creating the delta
in place as well as in the repo, so we needed to abstract over
locations.
William Manley [Tue, 19 Jul 2016 18:48:23 +0000 (19:48 +0100)]
ostree-prepare-root: Use pivot_root if real sysroot is already mounted at /
This allows ostree-prepare-root outside of the initramfs context where the
real rootfs is already mounted at /. We can't use `mount --move` in this
case because we would be trying to move / into a subdirectory of itself.
William Manley [Mon, 18 Jul 2016 17:09:10 +0000 (18:09 +0100)]
Refactor ostree-prepare-root: Perform chdir to deploy directory earlier
...for simplicity. This way we don't need to keep concatenating
deploy_path to everything. We can just refer relative to the current
working directory.
We need to do this after bind-mounting it over itself otherwise our cwd
is still on the non-bind-mounted filesystem below.
William Manley [Mon, 18 Jul 2016 17:02:12 +0000 (18:02 +0100)]
ostree-prepare-root: Refactor: Create /sysroot.tmp much later
Typically we have our ready made-up up root at
`/sysroot/ostree/deploy/.../` (`deploy_path`) and the real rootfs at
`/sysroot` (`root_mountpoint`). We want to end up with our made-up root
at `/sysroot/` and the real rootfs under `/sysroot/sysroot` as systemd
will be responsible for moving `/sysroot` to `/`.
We need to do this in 3 moves to avoid trying to move `/sysroot` under
itself:
William Manley [Mon, 18 Jul 2016 16:03:11 +0000 (17:03 +0100)]
ostree-prepare-root as init: exec init from deployment if run as PID1
This supports running ostree on embedded platforms without an initrd.
Specificially I'm trying to do bringup on an NVidia Tegra based Jetson TK1
dev board.
William Manley [Wed, 27 Jul 2016 15:40:05 +0000 (16:40 +0100)]
tests: Add basic tests for ostree-prepare-root
These tests use unshare and mount to prepare a fake initrd/early boot
directory structure so we can then test ostree-prepare-root.
Things that are tested:
* Running in an initrd environment
* Running without initrd
* /var and /sysroot being mounted correctly
* /usr being mounted read-only
Things not tested (yet):
* Running as init - this could be accomplished by unsharing the pid
namespace too.
* mounting/unmounting `/proc` if `/proc/cmdline` isn't available
* Persistent overlayfs for `/usr`
* Probably other things
The tests are basic but can be extended in the future as we do more work
on `ostree-prepare-root`.
These tests must be run as root as they require the ability to `mount`
and to `unshare` the mount namespace. Perhaps in the future we can use
user namespaces for this test once they are more widely available.
Colin Walters [Mon, 1 Aug 2016 14:43:49 +0000 (10:43 -0400)]
repo: Make ostree_repo_create() nonfatal on existing repos
In general we want to support "idempotentcy" or "state
synchronization" across interruption. If a repo is only partially
created due to a crash or whatever, it's hard for a user to know that.
Let's just make `ostree_repo_create()` idempotent. Since all we're
doing is a set of `mkdirat()` invocations, it's quite simple.
This also involved porting to fd-relative, which IMO makes the
code a lot clearer.
Dan Nicholson [Sat, 30 Jul 2016 15:14:17 +0000 (10:14 -0500)]
deltas: Allow processing of empty delta parts
If a static delta is generated between 2 commits with the same content,
then the delta will contain 1 part with no checksums. While useless,
this is a valid delta that shouldn't raise an assertion. If the delta
part has no checksums, then there are no objects to recreate and the
processing can be skipped.
Colin Walters [Mon, 27 Jun 2016 01:59:12 +0000 (21:59 -0400)]
libglnx porting: Migrate to new tempfile code
In general this is even cleaner now, though it was better after I
extracted a helper function for the "write tempfile with contents"
bits that were shared between metadata and regular file codepaths.
Colin Walters [Mon, 11 Jul 2016 13:29:18 +0000 (09:29 -0400)]
deltas: Handle untrusted checksums faster and more robustly
When reworking the ostree core [to use O_TMPFILE](https://github.com/ostreedev/ostree/pull/369),
I hit an issue in the way the untrusted delta codepath ends up trying
to re-open the file to checksum it. That's not possible with
`O_TMPFILE` since the fd (which we opened `O_WRONLY`) is the only
accessible reference to the content.
Fix this by changing the delta processing code to update a checksum as
we're doing writes, which is also faster, and ends up simplifying the
code as well.
What would be an even larger simplification here is if we e.g. used a
separate thread calling `write_object()` or something like that; the
main issue I see there is somehow bridging the fact that function
wants a `GInputStream*` but the delta code is generating stream of
writes.
Colin Walters [Sun, 24 Jul 2016 19:26:11 +0000 (15:26 -0400)]
glib.supp: Suppress g_task -> thread leaks
There are variants of this, but right now because `GTask` has its
own pool which isn't necessarily cleaned up on exit, we need
to suppress all of this.
Colin Walters [Sun, 24 Jul 2016 19:00:02 +0000 (15:00 -0400)]
pull-local: Explicitly unref variant rather than relying on floating
This shows up as a leak in valgrind; the callee isn't sinking. In
general through the power of cleanup attributes we can do explicit
cleanup rather than relying on floating refs.
Colin Walters [Sun, 24 Jul 2016 18:54:07 +0000 (14:54 -0400)]
trivial-httpd: Fix leak of option context
Right now our valgrind runs also end up valgrinding the
`trivial-httpd` code, so while it doesn't matter, let's fix this leak
anyways. We need to avoid calling `_exit()` since that won't run the
cleanup functions.
Colin Walters [Fri, 22 Jul 2016 20:58:26 +0000 (16:58 -0400)]
pull: Don't execute static deltas when mirroring
We don't presently support this, since the static delta code assumes
it can just `mmap()` file objects. We could at some point implement
this, but for now just skip executing deltas when doing
`archive -> archive` mirroring.
I noticed this when trying to mirror a repo in Jenkins in
[CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel).
Dan Nicholson [Thu, 14 Jul 2016 16:09:12 +0000 (09:09 -0700)]
core: Add allocating b64 checksum functions
The checksum_b64_inplace variants can't be used in bindings. Provide
versions that allocate and return the output rather than working on a
passed in buffer. These can then be used in GI bindings to get the
ostree modified base64 encodings.
Dan Nicholson [Thu, 14 Jul 2016 19:45:39 +0000 (12:45 -0700)]
lib: Fix version script node ordering
The order of the VERSION nodes is such that the new node name goes first
and the one it depends on goes after the commands. See
https://sourceware.org/binutils/docs/ld/VERSION.html.
Makefile-tests.am: make check uses the built binaries
The tests suite was failing locally as it was using the installed
version of rofiles-fuse, instead of the built one. Create the needed
symlinks in tests/ as we are already doing for the "ostree" binary.
ostree-prepare-root and ostree-remount added for completeness.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #395
Approved by: cgwalters
projects / ostree.git / log