Colin Walters [Thu, 28 May 2020 00:41:34 +0000 (00:41 +0000)]
remote-add: Default to explicit sign-verify backends
In https://github.com/ostreedev/ostree/pull/2092/commits/588f42e8c64183dfa1fbaa08cc92c46b691b23c4
we added a way to add keys for sign types when doing
a `remote add`, and in https://github.com/ostreedev/ostree/pull/2105
we extended `sign-verify` to support *limiting* to an explicit
set.
This PR changes the *default* for `remote add` to combine
the two - when providing an explicit `--sign-verify=type`,
we now limit the accepted types to only those.
Colin Walters [Wed, 27 May 2020 12:52:07 +0000 (12:52 +0000)]
tests/kola: Move to tests/kolainst
Follow the precedent set in https://github.com/coreos/rpm-ostree/pull/2106
and rename the directory, to more clearly move away from the
"uninstalled" test model. Prep for Rust-based tests.
The core problem was that a lot of services run early in the
real root and want write access to things like `/var` and `/etc`.
In trying to do remounts while the system is running we introduce
too many race conditions.
Instead, just make the `/etc` bind mount in the initramfs right
after we set up the main root. This is much more natural really,
and avoids all race conditions since nothing is running in the
sysroot yet.
The main awkward part is that since we're not linking
`ostree-prepare-root` to GLib (yet) we have a hacky parser
for the config file. But, this is going to be fine I think.
In order to avoid parsing the config twice, pass state from
`ostree-prepare-root` to `ostree-remount` via a file in `/run`.
Colin Walters [Fri, 15 May 2020 20:43:23 +0000 (20:43 +0000)]
pull: Add support for sign-verify=<list>
The goal here is to move the code towards a model
where the *client* can explicitly specify which signature types
are acceptable.
We retain support for `sign-verify=true` for backwards compatibility.
But in that configuration, a missing public key is just "no signatures found".
With `sign-verify=ed25519` and no key configured, we can
explicitly say `No keys found for required signapi type ed25519`
which is much, much clearer.
Implementation side, rather than maintaining `gboolean sign_verify` *and*
`GPtrArray sign_verifiers`, just have the array. If it's `NULL` that means
not to verify.
Note that currently, an explicit list is an OR of signatures, not AND.
In practice...I think most people are going to be using a single entry
anyways.
Jonathan Lebon [Wed, 20 May 2020 16:37:44 +0000 (12:37 -0400)]
tests/admin-test: Fix --allow-downgrade check
We were doing a check to verify that `ostree admin upgrade` wouldn't
accept a downgrade without `--allow-downgrade`. However, there's no
guarantee that the commit it's upgrading from is older than HEAD^ (what
we're upgrading to). Specifically, if the test runs fast enough, the
timestamps could be equal, since the lowest resolution is seconds.
Rework the test so that we first upgrade to HEAD, which we're sure is at
least 1 second apart from HEAD^, and *then* check that downgrade
protection is enforced.
We also can't use `rev-parse testos/buildmaster/x86_64-runtime` as a way
to know what commit the host is sitting on since the ref might've gone
ahead. Instead, just use `ostree admin status | head -n1`. (I played
with using the `ostree/I/J/K` refs, but those depend on what the
boot/subbootversion is and can easily change if we change previous
tests).
Colin Walters [Wed, 20 May 2020 15:57:10 +0000 (15:57 +0000)]
admin-test: Show err.txt on unexpected failure
In a CI run I think one of these `ostree show` commands is failing.
While that output would be useful, the actual `err.txt` usually
has what we want too.
Colin Walters [Sun, 17 May 2020 13:52:24 +0000 (13:52 +0000)]
pull: Only have API to disable signapi for local pulls
There's a lot of historical baggage associated with GPG verification
and `ostree pull` versus `ostree pull-local`. In particular nowadays,
if you use a `file://` remote things are transparently optimized
to e.g. use reflinks if available.
So for anyone who doesn't trust the "remote" repository, you should
really go through through the regular
`ostree remote add --sign-verify=X file://`
path for example.
Having a mechanism to say "turn on signapi verification" *without*
providing keys goes back into the "global state" debate I brought
up in https://github.com/ostreedev/ostree/issues/2080
It's just much cleaner architecturally if there is exactly one
path to find keys: from a remote config.
So here in contrast to the GPG code, for `pull-local` we explictily
disable signapi validation, and the `ostree_repo_pull()` API just
surfaces flags to disable it, not enable it.
Colin Walters [Thu, 14 May 2020 23:50:40 +0000 (23:50 +0000)]
ci: Install kola tests
This builds on
https://github.com/coreos/coreos-assembler/pull/1441
to install our tests rather than running them from the source
directory. This model will more cleanly allow us to ship
our tests along with a test container or elsewhere, separate
from the source directory.
Also prep for https://github.com/ostreedev/ostree/pull/2048
Jonathan Lebon [Thu, 14 May 2020 19:05:45 +0000 (15:05 -0400)]
lib/upgrader: Pull with `timestamp-check-from-rev`
For the same reason as https://github.com/coreos/rpm-ostree/pull/2094.
What we care most about is that the new commit we pull is newer than the
one we're currently sitting on, not necessarily that it's newer than the
branch itself, which it might not be if e.g. we're trying to deploy a
commit older than the tip but still newer than the deployment (via
`--override-commit`).
Jonathan Lebon [Thu, 14 May 2020 17:44:32 +0000 (13:44 -0400)]
lib/pull: Add `timestamp-check-from-rev`
The way `timestamp-check` works might be too restrictive in some
situations. Essentially, we need to support the case where users want to
pull an older commit than the current tip, but while still guaranteeing
that it is newer than some even older commit.
This will be used in Fedora CoreOS. For more information see:
https://github.com/coreos/rpm-ostree/pull/2094
https://github.com/coreos/fedora-coreos-tracker/issues/481
Colin Walters [Tue, 12 May 2020 22:04:53 +0000 (22:04 +0000)]
finalize: Add RequiresMountsFor=/boot too
In https://bugzilla.redhat.com/show_bug.cgi?id=1827712
some OpenShift CI is seeing `/boot` being unmounted before
`ostree-finalize-staged.service` runs or completes.
We finally tracked this down to a bug elsewhere, but
I think we should add this because it clearly shows
our requirements.
Colin Walters [Tue, 12 May 2020 01:26:00 +0000 (01:26 +0000)]
pull: Further cleanup signapi verification
Previously in the pull code, every time we went to verify
a commit we would re-initialize an `OstreeSign` instance
of each time, re-parse the remote configuration
and re-load its public keys etc.
In most cases this doesn't matter really because we're
pulling one commit, but if e.g. pulling a commit with
history would get a bit silly.
This changes things so that the pull code initializes the
verifiers once, and reuses them thereafter.
This is continuing towards changing the code to support
explicitly configured verifiers, xref
https://github.com/ostreedev/ostree/issues/2080
Colin Walters [Sun, 10 May 2020 13:20:50 +0000 (13:20 +0000)]
signing: Change API to create instances directly
This cleans up the verification code; it was weird how
we'd get the list of known names and then try to create
an instance from it (and throw an error if that failed, which
couldn't happen).
Per https://github.com/ostreedev/ostree/issues/2080#issuecomment-623614483
A huge benefit of ed25519 (and ECC in general) is that keys are very
short - short enough that it's completely reasonable to inline
them into a command line argument.
And I think that's a good model; it makes the keys very visible.
For example, someone could easily copy-paste a commandline
argument from a webpage (secured via TLS) that says to run
`ostree remote add --sign-verify=ed25519=inline:KEY`.
Colin Walters [Fri, 8 May 2020 12:49:55 +0000 (12:49 +0000)]
ostree-prepare-root: Requires=sysroot.mount
With just `After=` we'll still try to run in the scenario
where `sysroot.mount` fails because the rootfs didn't appear.
And this will end up spewing an error which can confuse people
into thinking something is wrong at the ostree level.
This has come up numerous times w/{Fedora,RHEL} CoreOS, most
recently while looking at
https://bugzilla.redhat.com/show_bug.cgi?id=1803130
Colin Walters [Wed, 6 May 2020 18:31:53 +0000 (18:31 +0000)]
commit: Add --mode-ro-executables option
I think we should encourage removing the writable bits from
executables. This has happened to me:
https://thomask.sdf.org/blog/2019/11/09/take-care-editing-bash-scripts.html
And not having the writable bit may help prevent hardlink
corruption with OSTree in some cases.
We can't do this by default, but add a convenient CLI flag
for it.
Dan Nicholson [Thu, 30 Apr 2020 20:28:20 +0000 (14:28 -0600)]
lib: Coerce flags enums to GIR bitfields
The GI scanner decides if an `enum` is really a `bitfield` if it finds
any values that have left shifts. With an `enumeration`, the
introspecting language may error or convert to a different type if the
user tries to combine values. Change all Flags `enum`s to use
left-shifted values so that they're represented as `bitfield`s in the
GIR.
The primary bug here is that you can't combine `REFS_ONLY` and
`NO_PRUNE` when calling `OSTree.Repo.prune()` from an introspected
language.
This is an IABI break since the typelib will change from `enumeration`
to `bitfield`. `OstreeRepoImportFlags` is internal but the change is
included here to prepare for a subsequent name that would require bit
shifting to operate correctly as a flag.
Colin Walters [Wed, 29 Apr 2020 01:45:48 +0000 (01:45 +0000)]
tests/staged-deploy: Cleanup initial state
I'm using [cosa build-fast](https://github.com/coreos/coreos-assembler/pull/1371)
and this test doesn't like starting out with two deployments. Clean
things up to one at the start just to simplify things.
Explicitly expose functions for querying the metadata format
and key name used by OstreeSign object:
- ostree_sign_metadata_format
- ostree_sign_metadata_key
This allows to use the same metadata format and key name
by 3-rd party applications using signapi.
Stefan Agner [Fri, 24 Apr 2020 11:12:47 +0000 (13:12 +0200)]
docs: extend repository types
Clarify where metadata are stored exactly in the `bare-user` case.
Make the first sentence of `bare-user` and `bare-user-only` paragraph
symetric to make it easier to jump to the right paragraph for readers
in a hury. Stree out that `bare-user-only` may loose metadata.
Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
Stefan Agner [Fri, 24 Apr 2020 11:05:15 +0000 (13:05 +0200)]
docs: extend object type documentation
Extend the object type documentation with file endings used for the
individual type. Also clarify in which situation content type objects
are used and why they do not match the SHA256 hash today.
Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
Stefan Agner [Fri, 24 Apr 2020 11:02:05 +0000 (13:02 +0200)]
docs: clarify archive repo type
Today `archive-z2` is still used as the default string to indicate
a `archive` type repository. Make clear that this is the way it is
intended. Otherwise users might think they use an no longer supported
OSTree repository since the mode string is still `archive-z2`.
Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
Colin Walters [Wed, 22 Apr 2020 14:26:23 +0000 (14:26 +0000)]
finalize-staged: Add ProtectHome=yes and ReadOnlyPaths=/etc
Same motivation as
https://github.com/coreos/rpm-ostree/pull/2060
I tried `InaccessiblePaths=/var` first and was very sad to find
out we have one tiny exception that breaks it. Otherwise it'd
be so elegant. Maybe in the future we split out that one thing
to a separate `ostree-finalized-stage-var.service` that's just
`ExecStart=/bin/rm -vf /var/.updated` and is otherwise
`ProtectSystem=strict` etc.
Jonathan Lebon [Fri, 17 Apr 2020 18:29:13 +0000 (14:29 -0400)]
lib/commit: Check that dirent is a directory before cleaning
I've only noticed this by inspection. But I think it's possible for
`cleanup_txn_dir` to get called with the `staging-...-lock` file since
it matches the prefix.
Make the checking here stronger by verifying that it's a directory. If
it's not a directory (lockfile), then follow the default pruning expiry
logic so that we still cleanup stray lockfiles eventually.
Colin Walters [Fri, 17 Apr 2020 01:16:51 +0000 (01:16 +0000)]
lib: Move internal binding verification API to repo.c
`ostree-repo-pull.c` is rather monstrous; I plan to split it
up a bit. There's actually already a `pull-private.h` but
that's just for the binding verification API. I think that one
isn't really pull specific. Let's move it into the "catchall"
`repo.c`.
Colin Walters [Wed, 15 Apr 2020 22:16:58 +0000 (22:16 +0000)]
travis: Add some libsodium coverage
As far as I can tell we're not gating on this right now. From
a quick glance, it looks like Debian stable has `libsodium-dev`
but only Ubuntu eoan does which we're not testing right now.
Colin Walters [Thu, 16 Apr 2020 14:15:04 +0000 (14:15 +0000)]
ci: Build minimal without libsodium too
The goal is to test "no options" build - and eventually tests.
(We're not actually including libsodium in the cosa buildroot right
now, but we may in the future)
Colin Walters [Sun, 12 Apr 2020 18:04:06 +0000 (18:04 +0000)]
Change signature opts to include type, cleanup error handling
Previously we would pass the `verification-key` and `verification-file`
to all backends, ignoring errors from loading keys until we
found one that worked.
Instead, change the options to be `verification-<engine>-key`
and `verification-<engine>-file`, and then
rework this to use standard error handling; barf explicitly if
we can't load the public keys for example. Preserve
the semantics of accepting the first valid signature. The
first signature error is captured, the others are currently
compressed into a `(and %d more)` prefix.
And now that I look at this more closely there's a lot of
duplication between the two code paths in pull.c for verifying;
will dedup this next.
If GPG support is disabled in a build time we should to check if any of
options "gpg_verify" or "gpg_verify_summary" is set to TRUE instead
of checking if they are passed via options while pulling from remote.
Fixed the failure with assertion of `ostree find-remotes --pull --mirror`
calling (`tests/test-pull-collections.sh`) if libostree has been compiled
without GPG support.
Colin Walters [Sun, 5 Apr 2020 22:23:39 +0000 (22:23 +0000)]
commit: Add --base argument
I was trying to followup the `--selinux-policy-from-base` work
to add a `cosa build --fast=overlay` for coreos-assembler,
but hit on the fact that using e.g. `--owner-uid` disables
commit optimizations.
A while ago, https://github.com/ostreedev/ostree/pull/1643 landed
which optimized this for the case where no modifications are provided.
But, we really need the SELinux policy bits, and it's super convenient
to run `ostree commit` as non-root.
It's fairly surprising actually that it's taken us so long to
iterate on a good interface for this "commit changes on top of a base"
model. In practice, many nontrivial cases really end up needing
to do a (hardlink) checkout, and that case is optimized.
But for this coreos-assembler work I want to directly overlay onto
a commit object another commit object.
That previous PR above added exactly the API we need, so let's
expose it in the CLI.
What you can see happening in the test is that we provide
`--owner-uid 42`, but that only applies to directories/files
that were added in the commit.
And now that I look at this, I think what we really want here
is to avoid changing directories that exist in the base, but
eh; in practice the main use here is for `--owner-uid 0` while
committing as non-root; and that works fine with this since
the baseline uid will be zero as well.
grub2: Don't add menu entries if GRUB supports parsing BLS snippets
This is another attempt to avoid having duplicated menu entries caused by
GRUB having support to parse BLS snippets and the 15_ostree script adding
menu entries as well.
The previous attempt was in commit 985a1410029 ("grub2: Exit gracefully if
the configuration has BLS enabled") but that lead to users not having menu
entries at all, due having an old GRUB version that was not able to parse
the BLS snippets.
This happened because the GRUB bootloader is never updated in the ESP as
a part of the OSTree upgrade transaction.
The logic is similar to the previous commit, the 15_ostree script exits if
able to determine that the bootloader can parse the BLS snippets directly.
But this time it will not only check that a BLS configuration was enabled,
but also that a /boot/grub2/.grub2-blscfg-supported file exists. This file
has to be created by a component outside of OSTree that also takes care of
updating GRUB to a version that has proper BLS support.
Colin Walters [Sun, 5 Apr 2020 18:22:49 +0000 (18:22 +0000)]
pull: Update key loading function to match error style
This code wasn't written with idiomatic GError usage; it's not standard
to construct an error up front and continually append to its
message. The exit from a function is usually `return TRUE`,
with error conditions before that.
Updating it to match style reveals what I think is a bug;
we were silently ignoring failure to parse key files.
projects / ostree.git / log