Colin Walters [Wed, 6 Sep 2017 15:37:02 +0000 (11:37 -0400)]
tree-wide: Use helpers for unlinkat()
We have `ot_ensure_unlinked_at()` for the "ignore ENOENT" case, and
`glnx_unlinkat()` otherwise. Port all in-tree callers to one or the other as
appropriate.
Just noticed an unprefixed error in the refs case and decided to do a tree-wide
check.
Colin Walters [Wed, 6 Sep 2017 01:03:18 +0000 (21:03 -0400)]
lib/gpg: Use nicer helper for gpg error messages
The vast majority of invocations of `ot_gpgme_error_to_gio_error()` were paired
with `g_prefix_error()`; let's combine them for the same reason we do
`glnx_throw_errno_prefix()`. For the few cases that don't we might as well add
some prefix.
I also changed it to `return FALSE` in prep for more style porting.
Colin Walters [Tue, 5 Sep 2017 19:01:12 +0000 (15:01 -0400)]
bin/local-pull: Clarify docs, add more tests for corrupted local pulls
I was reading the pull-local command docs and realized it was somewhat unclear
that `--untrusted` *only* applied to local repo pulls; in other words that we
always treat non-local pulls as untrusted.
Tweak the docstring, and add tests that verify this explicitly.
Colin Walters [Wed, 6 Sep 2017 14:24:56 +0000 (10:24 -0400)]
lib/repo: Add apidoc for repo properties
However, they weren't showing up in the output HTML and I have
no idea why; I looked at what we're doing and it looks close enough
to what's going on in `GDBusConnection` that I was using as a reference.
I'm not going to spend a lot of time to debug it right now.
Colin Walters [Fri, 1 Sep 2017 19:01:18 +0000 (15:01 -0400)]
bin/admin: Do sysroot loading during argument parsing
Followup from previous patch - we can now centralize the sysroot loading.
Besides the obvious cleanup value, this is also prep for dropping an
`ostree_sysroot_get_path()` user.
Colin Walters [Fri, 1 Sep 2017 18:57:53 +0000 (14:57 -0400)]
bin/admin: Change init-fs to stop loading a sysroot to init one
This is exactly analogous to the `ostree init` case where
we have `OSTREE_BUILTIN_FLAG_NO_REPO` to avoid trying to load
a repo when we're creating one.
Let's avoid the pointless sysroot for `init-fs`; among other
things this will then let us do `ostree_sysroot_load()` inside
the argument parsing, and drop it from every other user.
Colin Walters [Fri, 1 Sep 2017 20:15:33 +0000 (16:15 -0400)]
tree-wide: Replace archive-z2 with archive
In almost all places. There are just a few exceptions; one tricky bit for
example is that the repo config must still have `mode=archive-z2`, since
`archive` used to mean something else. (We could very likely just get rid of
that check, but eh, later).
I also added a test that one can still do `ostree repo init --mode=archive-z2`.
Ruixin [Thu, 31 Aug 2017 15:44:35 +0000 (15:44 +0000)]
checkout: add an extra checkout_overwrite mode
This is for issue projectatomic/rpm-ostree#365,
an extra option of overwrite mode is added to the checkout command
so that when there is "non-directory" file already exist
during checkout, the error will be handled.
Jonathan Lebon [Wed, 30 Aug 2017 19:20:31 +0000 (15:20 -0400)]
commit: filter out selinux label before commit
The new --selinux-policy added in [0] exposed a subtle issue in the way
we handle labeling during commit. The CI system in rpm-ostree hit this
when trying to make use of it[1].
Basically, because of the way we use a GVariant to represent xattrs, if
a file to be committed already has an SELinux label, the xattr object
ends up with *two* label entries. This of course throws off fsck later
on, since the checksum will have gone over both entries, even though the
on-disk file will only have a single label (in which the second entry
wins).
I confirmed that the `fsck` added in the installed test fails without
the rest of this patch.
Matthew Leeds [Wed, 30 Aug 2017 19:06:45 +0000 (12:06 -0700)]
tests: Make the deployment mutable in test-sysroot.js
We attempt to make deployments mutable in the test suite (as opposed to
immutable which is the default) to make it easier to chmod and clean up
the tmp files after each test. This is normally accomplished by setting
OSTREE_SYSROOT_DEBUG=mutable-deployments in libtest.sh, but that only
affects the environment variables for that bash instance, not the
process running gjs. So in test-sysroot.js OSTREE_SYSROOT_DEBUG wasn't
set when sysroot.deploy_tree() was called, which means the deployment
was made immutable which eventually causes the test to fail. This only
occurs when the test is run by the root user because for non-root users
_ostree_linuxfs_fd_alter_immutable_flag() would silently fail and the
deployment would be mutable.
This commit fixes this issue by setting the environment variable in
tests/test-sysroot.js.
Colin Walters [Wed, 23 Aug 2017 01:52:24 +0000 (21:52 -0400)]
libarchive: Add support for translating paths during commit
For rpm-ostree, I want to move RPM files in `/boot` to `/usr/lib/ostree-boot`.
This is currently impossible without forking the libarchive code. Supporting
this is pretty straightforward; we already had pathname translation in
the libarchive code, we just need to expose it as an option.
On the command line side, I chose to wrap this as a regexp. That should be good
enough for a lot of use cases; sophisticated users should as always be making
use of the API. Note that this required some new `#ifdef LIBARCHIVE` bits to use
the new API. Following previous patterns here, we use the new API only if a
relevant option is enabled, ensuring unit test coverage of both paths.
For the test cases, I ended up changing the accounting to avoid having to
multiply the test count.
Simon McVittie [Tue, 29 Aug 2017 16:03:37 +0000 (17:03 +0100)]
test-basic-user-only: Skip final step if no user xattrs
We don't want to skip the entire test, because the whole point of
bare-user-only is that it works in the absence of xattrs; but we do
need to skip this last stage, which explicitly uses a bare-user
repository.
Signed-off-by: Simon McVittie <smcv@debian.org> Closes: #1120
Approved by: cgwalters
Jonathan Lebon [Wed, 23 Aug 2017 16:15:46 +0000 (12:15 -0400)]
ostree-sysroot: make simple_write_deployment smarter
This is a follow-up to https://github.com/ostreedev/ostree/pull/1097.
We make simple_write_deployment smart enough so that it can be used for
rpm-ostree's purposes. This is mostly an upstreaming of logic that
already existed there.
Notably we correctly append NOT_DEFAULT deployments *after* the booted
deployment and we now support RETAIN_PENDING and RETAIN_ROLLBACK flags
to have more granularity on deployment pruning.
Expose these new flags on the CLI using new options (as well as expose
the previously existing NOT_DEFAULT flag as --not-as-default).
I couldn't add tests for --retain-pending because the merge deployment
is always the topmost one. Though I did check that it worked in a VM.
Matthew Leeds [Tue, 22 Aug 2017 00:08:12 +0000 (17:08 -0700)]
lib/repo-refs: Include remote refs when using collections
When working with collections it can be useful to see remote refs rather
than just local and mirrored ones. This commit changes the "ostree refs
-c" output to include remote refs, and includes remote refs with
collection IDs in summary file generation as well. The former behavior
is consistent with how "ostree refs" works, and the latter behavior is
useful in facilitating P2P updates even when mirrors haven't been
configured.
To accomplish this, OstreeRepoListRefsExtFlags was extended with an
EXCLUDE_REMOTES flag. This was done rather than an INCLUDE_REMOTES flag
so that existing calls to ostree_repo_list_refs_ext continue to have the
same behavior. This flag was added to ostree_repo_list_collection_refs
(which is an experimental API break).
Also, add unit tests for the "refs -c" and summary file behavior, and
update relevant tests.
Colin Walters [Wed, 23 Aug 2017 18:25:43 +0000 (14:25 -0400)]
lib/repo: Add some assertions for ABI sizes
Things like https://sourceware.org/libabigail/manual/abidiff.html
look interesting but in a brief look I couldn't work out
how to conveniently use them for quick ABI sanity checking without
doing a diff from a previous build (which we could do but would be
more involved).
This way will at least catch struct ABI breaks on x86_64 which
I think we'd be most likely to do accidentally when trying
to use one of the previous unused values.
I found the hole values via gdb's `pahole` command.
Colin Walters [Wed, 23 Aug 2017 17:03:39 +0000 (13:03 -0400)]
bin/main: Remove duplicated usage output on unknown commands
It's been this way for a long time, though not forever; I went
back to v2014.11 as a random choice and it worked then. Not
going to bother doing a full archive search for this though.
Anyone who wants more context can help themselves.
Colin Walters [Wed, 23 Aug 2017 17:31:40 +0000 (13:31 -0400)]
build/maint.mk: Comment out setting of LC_ALL
This triggers obscure bugs; really we shouldn't be overriding
the global locale here. In practice, production build systems
will be using fixed locales anyways.
Also, we only use a small subset of this file (`make syntax-check`),
which appears to work OK without this.
I will probably try to work out where to submit this as at
least an issue report for upstream gnulib.
Colin Walters [Wed, 23 Aug 2017 14:09:44 +0000 (10:09 -0400)]
lib/commit: Honor commit filter for libarchive --tar-autocreate-parents
This makes `ostree commit --tree=tar` honor `--owner-uid` and `--owner-gid`
for the root directory.
Prep for further commit filtering work, although mostly for the unit test cases;
this ensures we can use `ostree checkout` after autocreating a root directory.
Colin Walters [Wed, 23 Aug 2017 13:37:44 +0000 (09:37 -0400)]
lib/commit: Remove duplicated function for filter processing
The wrapping here is unnecessary, since `_ostree_repo_commit_modifier_apply()`
already does what this function did. Further, the return type was wrong.
Philip Withnall [Tue, 22 Aug 2017 17:32:43 +0000 (18:32 +0100)]
build: Add distcheck configure flags to fix systemd and bash-completion
When running distcheck, the systemd system-generator and bash-completion
scripts are installed in absolute paths (/usr and /lib) as looked up
from their pkg-config files. This breaks distcheck. Use a
${prefix}-relative path for both of them when configuring for distcheck.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1103
Approved by: cgwalters
Jonathan Lebon [Mon, 21 Aug 2017 19:00:49 +0000 (15:00 -0400)]
tests: fix admin upgrade timestamp file check
In #1055, I tried to be helpful but that didn't work out all too well.
We need to recompute the file path since one is in archive mode and the
other in bare mode.
Colin Walters [Thu, 18 May 2017 22:12:33 +0000 (18:12 -0400)]
lib/pull: Add support for timestamp-check option, use in upgrader
For both flatpak and ostree-as-host, we really want to verify up front during
pulls that we're not being downgraded. Currently both flatpak and
`OstreeSysrootUpgrader` do this before deployments, but at that point we've
already downloaded all the data, which is annoying.
Guy Shapiro [Mon, 21 Aug 2017 06:42:48 +0000 (09:42 +0300)]
lib/sysroot: fix placement for not-default deployment
When using the
OSTREE_SYSROOT_SIMPLE_WRITE_DEPLOYMENT_FLAGS_NOT_DEFAULT flag, the
deployment is said to be added after the booted or merge deployment.
Fix the condition to do so instead of adding it in the second place.
Colin Walters [Sat, 19 Aug 2017 00:51:10 +0000 (20:51 -0400)]
lib/deploy: Add .img to end of initramfs in /usr/lib/modules
Follow up to <https://github.com/ostreedev/ostree/pull/1079>; I was working on
the rpm-ostree updates for this, and I think it's more consistent if we have
`.img` here, since that's a closer match to the "remove $kver" that results in
`vmlinuz`. Also just best practice to have file suffix types where they make
sense.
The astute reader might notice this sneaks in a change where we'd crash if the
legacy bootdir didn't have an initramfs...yeah, should probably have test
coverage of that.
Colin Walters [Tue, 15 Aug 2017 15:22:21 +0000 (11:22 -0400)]
lib/sysroot: Support /usr/lib/modules/$kver for kernel/initramfs
This is the new Fedora kernel standard layout; it has the advantage
of being in `/usr` like `/usr/lib/ostree-boot`, but it's not OSTree
specific.
Further, I think in practice forcing tree builders to compute the checksum is an
annoying stumbling block; since we already switched to e.g. computing checksums
always when doing pulls, the cost of doing another checksum for the
kernel/initramfs is tiny. The "bootcsum" becomes more of an internal
implementation detail.
Now, there is a transition; my current thought for this is that rpm-ostree will
change to default to injecting into both `/usr/lib/ostree-boot` and
`/usr/lib/modules`, and stop doing `/boot`, then maybe next year say we drop the
`/usr/lib/ostree-boot` by default.
A twist here is that the default Fedora kernel RPM layout (and what's in
rpm-ostree today) includes a kernel but *not* an initramfs in
`/usr/lib/modules`. If we looked only there, we'd just find the kernel. So we
need to look in both, and then special case this - pick the legacy layout if we
have `/usr/lib/modules` but not an initramfs.
While here, rework the code to have an `OstreeKernelLayout` struct which makes
dealing with all of the variables nicer.
Colin Walters [Thu, 17 Aug 2017 20:49:26 +0000 (16:49 -0400)]
bin: Squash some -Wuninit warnings with porting to new style
I noticed this with a local build of an RPM:
```
/usr/include/glib-2.0/glib/glib-autocleanups.h:28:3: warning: 'help' may be used uninitialized in this function [-Wmaybe-uninitialized]
g_free (*pp);
^~~~~~~~~~~~
src/ostree/ot-main.c:82:20: note: 'help' was declared here
g_autofree char *help;
^~~~
```
Colin Walters [Thu, 17 Aug 2017 16:16:35 +0000 (12:16 -0400)]
lib: Fix v2017.10 symbols to inherit from v2017.8
I actually don't quite know what the version inheritance really does, but let's
be safe and fix this. I'm being conservative here and fixing it to inherit from
2017.8, skipping .9 since that doesn't have a parent.
Colin Walters [Wed, 16 Aug 2017 16:44:05 +0000 (12:44 -0400)]
lib/deploy: Ignore errors from FITHAW
In the production case since we used `daemon()` our stderr is `/dev/null`¹
there's not much use in logging errors from `FITHAW` or `exit(1)`, and doing so
breaks the test suite which checks the return from `waitpid()`. There's nothing
we can really do if `FITHAW` fails, and in most of those cases `EINVAL`,
`EOPNOTSUPP`, we *shouldn't* do anything anyways.
¹ Though perhaps we should set up the systemd journal, but let's not
go there right now.
Jonathan Lebon [Wed, 16 Aug 2017 13:10:39 +0000 (09:10 -0400)]
ostree-deployment.c: simplify equality check
Just a random cozy patch I made while perusing the codebase. When
determining if two OstreeDeployment objects are the same, rather than
just checking for NULL, we can just directly check for equality of
pointers to also catch the trivial case.
Colin Walters [Wed, 2 Aug 2017 19:44:34 +0000 (15:44 -0400)]
Update libglnx
This is mostly the `copy_file_range` changes plus the Coverity files.
```
Colin Walters (4):
localalloc: Abort on EBADF from close() by default
local-alloc: Remove almost all macros like glnx_free, glnx_unref_variant
console: Fix Coverity NULL deref warning
fdio: Merge systemd code to use copy_file_range(), use FICLONE
Jonathan Lebon (1):
console: trim useless check
Matthew Leeds (1):
dirfd: Fix typo in comment
Philip Withnall (1):
glnx-console: Add missing NULL check before writing out text
```
Matthew Barnes [Tue, 15 Aug 2017 16:26:23 +0000 (16:26 +0000)]
ostree: Add naggy comments to help keep options in sync
/* ATTENTION:
* Please remember to update the bash-completion script (bash/ostree) and
* man page (man/ostree-$COMMANDNAME.xml) when changing the option list.
*/
Colin Walters [Fri, 28 Apr 2017 19:51:32 +0000 (15:51 -0400)]
repo: Introduce ostree_repo_open_at() and ostree_repo_create_at()
This essentially completes our fd-relative conversion.
While here, I cleaned up the semantics of `ostree_repo_create()` and
`ostree_repo_create_at()` to be more atomic - basically various scripts were
testing for the `objects` subdirectory, so let's formalize that.
Jonathan Lebon [Fri, 11 Aug 2017 14:16:17 +0000 (10:16 -0400)]
pull: mark commits from local cache as partial
If one of the localcache repos has the exact same commit we resolved
from the remote, then we need to make sure to mark it as partial so that
we download the full tree.
Colin Walters [Wed, 9 Aug 2017 13:07:11 +0000 (09:07 -0400)]
lib/deploy: Close test suite race condition
Saw this in a PR result; we need to wait for the child to have written its
result to stderr before we exit, otherwise the test suite may not read it in
time.
Colin Walters [Fri, 4 Aug 2017 01:45:50 +0000 (21:45 -0400)]
lib/sysroot: Add journal-msg signal
This will allow us to drop the awful hack in rpm-ostree where we watch our own
stdout. In general, libraries shouldn't write to stdout.
Also we can kill the systemd journal wrapper code. There's some duplication at
each call site now...but it's easier than trying to write a `sd_journal_send()`
wrapper.
I was originally going to have this emit all of the structured data too as a
`GVariant` but decided it wasn't worth it right now.
Philip Withnall [Wed, 9 Aug 2017 13:35:53 +0000 (14:35 +0100)]
lib/gpg-verify: Add an OstreeGpgError error domain
Add a new error domain for GPG signing/verification errors, and use it
throughout libostree for describing verification errors. This replaces
various uses of G_IO_ERROR_FAILED, and one instance of
G_IO_ERROR_NOT_FOUND (for which some code in ot-builtin-show.c had to be
changed to ensure it was still handled correctly).
The use of a separate error domain allows failures in GPG operations to
be handled separately from network failures (where the summary file
could not be found to be downloaded, for example) or timeouts.
I'd like to move the new canonical kernel directory to `/usr/lib/modules/$kver`,
as Fedora has done. The `get_kernel_from_tree()` function now abstracts over
parsing the data (src vs destination filenames, as well as checksum) in
preparation for adding the new case.
In preparation for this, let's change the current test suite to use the
*current* directory of `/usr/lib/ostree-boot`, and also add coverage of `/boot`.
Philip Withnall [Tue, 8 Aug 2017 17:58:52 +0000 (18:58 +0100)]
lib/repo-finder: Emit gpg-verify-summary=false in dynamic remote config
When returning results from finding repos, set gpg-verify-summary=false
in their configs, since any pulls from such remotes will necessarily
involve collection IDs, and hence should be using the unsigned summary
support. In the intended deployment mode for P2P transmission of OSTree
refs, summaries *cannot* be signed, so setting gpg-verify-summary=true
would cause all the pulls to fail.
The unsigned summary support is the move of repository metadata from
the summary file (not spliceable) to the well-known ostree-metadata ref
(spliceable, as it can exist for multiple collection IDs in the same
repository).
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1066
Approved by: cgwalters
XFS doesn't flush the journal on `syncfs()`. GRUB doesn't know how to follow the
XFS journal, so if the filesystem is in a dirty state (possible with xfs
`/boot`, extremely likely with `/`, if the journaled data includes content for
`/boot`, the system may be unbootable if a system crash occurs.
Fix this by doing a `FIFREEZE`+`FITHAW` cycle. Now, most people
probably would have replaced the `syncfs()` invocation with those two
ioctls. But this would have become (I believe) the *only* place in
libostree where we weren't safe against interruption. The failure
mode would be ugly; nothing else would be able to write to the filesystem
until manual intervention.
The real fix here I think is to land an atomic `FIFREEZETHAW` ioctl
in the kernel. I might try a patch.
In the meantime though, let's jump through some hoops and set up
a "watchdog" child process that acts as a fallback unfreezer.
Philip Withnall [Tue, 8 Aug 2017 14:39:34 +0000 (15:39 +0100)]
lib/repo: Fix handling of missing summary files when downloading
The API for downloading a summary file can legitimately return NULL for
the summary file contents when it returns TRUE (success). This indicates
an error 404 — the summary file was not found.
Two call sites were not handling that correctly, which was causing later
assertion failures.
Philip Withnall [Mon, 7 Aug 2017 13:09:07 +0000 (14:09 +0100)]
lib/repo-commit: Drop unreachable conditional branch
(remaining > 0) is asserted by the loop condition, and remaining is not
modified between that check and the G_UNLIKELY — so the condition in the
G_UNLIKELY will always be true.
Philip Withnall [Mon, 7 Aug 2017 18:52:17 +0000 (19:52 +0100)]
lib/repo-refs: Add ostree_repo_remote_list_collection_refs() API
This parallels ostree_repo_remote_list_refs(), but returns a map of
OstreeCollectionRef → checksum, and includes refs from collection IDs
other than the remote repository’s main collection ID.
Use this in OstreeRepoFinderConfig to ensure that refs are matched
against even if they’re stored in the repository summary file’s
collection map, rather than its main ref map. This fixes false negatives
when searching for refs in some situations.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1058
Approved by: cgwalters
Philip Withnall [Mon, 7 Aug 2017 18:50:40 +0000 (19:50 +0100)]
lib/repo-pull: Add a missing precondition
This catches a few failure modes in the pull code a little earlier,
before the incorrectly-NULL repo makes its way into a closure and a
worker thread, where the cause of the problem is harder to track down.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1058
Approved by: cgwalters
Philip Withnall [Mon, 7 Aug 2017 18:49:17 +0000 (19:49 +0100)]
lib/repo-finder: Avoid a potential unref-of-NULL crash
As the comment explains, it’s possible for a result to be freed while
ref_to_checksum is NULL, even though normally the data structure
guarantees it’s non-NULL. This was causing crashes when results were
filtered out of a find-remotes call. Guard against that.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1058
Approved by: cgwalters
Philip Withnall [Mon, 7 Aug 2017 12:37:57 +0000 (13:37 +0100)]
lib/repo-pull: Fix counting of latest commits when finding repos
The intended behaviour of ostree_repo_find_remotes() is to return
results which have the latest version of at least one of the requested
refs. Results which have some of the requested refs, but don’t have the
latest version of any of them, should be ignored. The logic to do this
was broken in the case that a result contained a positive number of the
requested refs, but none of them were the latest version. (It previously
worked when the result contained none of the requested refs.)
Fix the counting to work correctly in both cases.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1058
Approved by: cgwalters
Colin Walters [Mon, 31 Jul 2017 16:22:16 +0000 (12:22 -0400)]
bin/pull: Fix @ override syntax when pulling multiple refs
Coverity spotted an infloop here since we were incrementing `i++`
instead of `j++`. But adding a test revealed other bugs - we need
to keep the arrays in sync.
Colin Walters [Mon, 31 Jul 2017 16:27:31 +0000 (12:27 -0400)]
lib/deltas: Squash Coverity warning for div-by-zero in delta show
If a delta happens to have zero objects, we could end up doing
a divide-by-zero when inferring endianness. In practice,
a zero-object delta isn't possible to generate I think, but
let's make sure the code is defensive all the same.
Matthew Leeds [Fri, 4 Aug 2017 20:18:40 +0000 (13:18 -0700)]
main: Fix subcommand usage output
This commit sets prgname correctly so that the "ostree subcommand
--help" output prints the subcommand rather than just "ostree".
This was removed in commit f0519e541f29 because it tripped the thread
sanitizer, but it's being added back conditionally so most users who
don't compile with -fsanitize=adress see proper help output.
projects / ostree.git / log