Colin Walters [Sun, 31 May 2020 17:57:22 +0000 (17:57 +0000)]
tests/inst: Add destructive test framework
This adds infrastructure to the Rust test suite for destructive
tests, and adds a new `transactionality` test which runs
rpm-ostree in a loop (along with `ostree-finalize-staged`) and
repeatedly uses either `kill -9`, `reboot` and `reboot -ff`.
The main goal here is to flush out any "logic errors".
So far I've validated that this passes a lot of cycles
using
```
$ kola run --qemu-image=fastbuild-fedora-coreos-ostree-qemu.qcow2 ext.ostree.destructive-rs.transactionality --debug --multiply 8 --parallel 4
```
a number of times.
Jonathan Lebon [Mon, 17 Aug 2020 13:48:15 +0000 (09:48 -0400)]
lib/deploy: Clarify comment re. staging API
Don't mention deprecation in the description for
`ostree_sysroot_deploy_tree` since there are legitimate use cases for it
(e.g. to create the first deployment via `ostree admin deploy`).
Instead, make the comment clearly redirect to the staging API when
booted into the sysroot.
Jonathan Lebon [Mon, 17 Aug 2020 13:48:09 +0000 (09:48 -0400)]
lib/deploy: Clean up kargs override handling
Tighten up how we handle kargs here so it's more clear. When we call
`sysroot_finalize_deployment`, any karg overrides have already been set
on the bootconfig object of the deployment. So re-setting it here is
redundant and confusing.
Without reproducible images, a rebuild of the initrd will create a
different image file (due to things like creation time of the files in
the cpio archive) even if the actual contents in it are exactly the
same, adding an unnecessary download during updates.
Adding 'reproducible=yes' avoids this and creates the same image files
for the same content.
Colin Walters [Sun, 17 May 2020 18:17:37 +0000 (18:17 +0000)]
Add "transient" unlock
I was thinking a bit more recently about the "live" changes
stuff https://github.com/coreos/rpm-ostree/issues/639
(particularly since https://github.com/coreos/rpm-ostree/pull/2060 )
and I realized reading the last debates in that issue that
there's really a much simpler solution; do exactly the same
thing we do for `ostree admin unlock`, except mount it read-only
by default.
Then, anything that wants to modify it does the same thing
libostree does for `/sysroot` and `/boot` as of recently; create
a new mount namespace and do the modifications there.
The advantages of this are numerous. First, we already have
all of the code, it's basically just plumbing through a new
entry in the state enumeration and passing `MS_RDONLY` into
the `mount()` system call.
"live" changes here also naturally don't persist, unlike what
we are currently doing in rpm-ostree.
Philip Withnall [Thu, 16 Jul 2020 15:16:37 +0000 (16:16 +0100)]
pull: Add summary-{,sig-}bytes options to ostree_repo_pull()
These allow the `summary` and `summary.sig` files to be cached at a
higher layer (for example, flatpak) between related pull operations (for
example, within a single flatpak transaction). This avoids
re-downloading `summary.sig` multiple times throughout a transaction,
which increases the transaction’s latency and introduces the possibility
for inconsistency between parts of the transaction if the server changes
its `summary` file part-way through.
In particular, this should speed up flatpak transactions on machines
with high latency network connections, where network round trips have a
high impact on the latency of an overall operation.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
I'm not sure how (or if) this passed before, the job logs have
been GC'd.
This is a bit confusing but basically right now ostree/rpm-ostree's
CI jobs don't use `/srv/fcos` - it might make sense to port
these to `fcosBuild` but that needs investigation.
Colin Walters [Sat, 1 Aug 2020 18:47:36 +0000 (18:47 +0000)]
tests/repo-finder: Explicitly commit empty dir
We were committing the whole tempdir, which seems to fail
in Travis because the GPG agent Unix domain socket ends up there too,
and ostree refuses to commit sockets.
Stefan Agner [Fri, 31 Jul 2020 19:43:09 +0000 (21:43 +0200)]
Show commit checksum of parent, if present
This is useful for ostree log on client side where often not the
full history of a branch is available. It is also helpful for
ostree show to show if a particular commit has a parent.
Jonathan Lebon [Thu, 30 Jul 2020 21:35:20 +0000 (17:35 -0400)]
app: Fix various CLI metavariable names
- Use `REV` instead of `REF` in places where we meant it.
- Fix `commit --parent` actually taking a commit checksum and not a ref.
- Fix `ostree admin switch` using `REF` instead of `REFSPEC`.
Colin Walters [Thu, 16 Jul 2020 21:13:36 +0000 (21:13 +0000)]
pull: Add --per-object-fsync
This is the opposite of
https://github.com/ostreedev/ostree/issues/1184
Motivated by OpenShift seeing etcd performance issues during
OS updates: https://github.com/openshift/machine-config-operator/issues/1897
Basically, if we switch to invoking `fsync()` as we go, it makes
ostree performance worse (in my tests, 31s to write 2G versus 7s if we
delay sync) but it avoids *huge* outliers in `fsync()` time for etcd.
Jonathan Lebon [Thu, 16 Jul 2020 15:42:57 +0000 (11:42 -0400)]
ci: Constrain parallel build jobs
The default `_NPROCESSORS_ONLN` heuristic we have isn't cgroups aware.
So it thinks it has e.g. 40 CPUs when running in a k8s pod. This can
then blow through our allocated resource limits.
Declare some modest amount of RAM and CPU resources and override `make`
parallelism.
This matches what rpm-ostree now does in
https://github.com/coreos/rpm-ostree/pull/2155.
William Manley [Wed, 15 Jul 2020 14:40:14 +0000 (15:40 +0100)]
sysroot: Support /boot on root or as seperate filesystem for syslinux and u-boot
We use a similar trick to having a `sysroot -> .` symlink on the real root
here to support both /boot on root as well as on a separate filesystem. No
matter how it's mounted `/boot/xyz` will always refer to the file you'd
expect.
This is nicer than my previous attempts at this because there's no
configuration nor auto-detection required.
Jonathan Lebon [Tue, 30 Jun 2020 19:05:19 +0000 (15:05 -0400)]
lib/repo: Handle EACCES for POSIX locking
If `glnx_make_lock_file` falls back to `flock`, on NFS this uses POSIX
locks (`F_SETLK`). As such, we need to be able to handle `EACCES` as
well as `EAGAIN` (see `fnctl(2)`).
I think this is what coreos-ostree-importer has been hitting, which runs
on RHEL7 in the Fedora infra and does locking over an NFS share where
multiple apps could concurrently pull things into the repo.
Jonathan Lebon [Wed, 17 Jun 2020 19:48:31 +0000 (15:48 -0400)]
ci: Remove libpaprci/ directory
And move everything that was in it directly in `ci/`. There's a bunch
more cleanups here that we need to do (and more changes to upstream from
the rpm-ostree copies of this).
Colin Walters [Wed, 17 Jun 2020 00:22:49 +0000 (00:22 +0000)]
signapi: Change API to also return a success message
This is the dual of https://github.com/ostreedev/ostree/pull/2129/commits/1f3c8c5b3de978f6e069c24938967f823cce7ee8
where we output more detail when signapi fails to validate.
Extend the API to return a string for success, which we output
to stdout.
This will help the test suite *and* end users validate that the expected
thing is happening.
In order to make this cleaner, split the "verified commit" set
in the pull code into GPG and signapi verified sets, and have
the signapi verified set contain the verification string.
We're not doing anything with the verification string in the
pull code *yet* but I plan to add something like
`ostree pull --verbose` which would finally print this.
Colin Walters [Thu, 11 Jun 2020 18:31:33 +0000 (18:31 +0000)]
tests: Add a pre-signed-pull.sh test
I'm thinking about adding an implementation of ed25519 signatures
with OpenSSL (so we can ship the feature with Fedora CoreOS
without requiring an additional library) and in preparation for
that it's essential that we validate that libsodium-generated
signatures and OpenSSL-generated signatures are compatible.
I don't know if they are yet actually, but the goal of this
new test is to add a pre-generated repository with a signed
commit generated by libsodium.
This will catch if e.g. there's ever a change in libsodium,
or if existing libsodium implementation versions (e.g. the
one in Debian) might differ from what we ship here.
projects / ostree.git / log