Jonathan Lebon [Thu, 13 Apr 2023 21:22:43 +0000 (17:22 -0400)]
lib/sysroot-deploy: Add experimental support for automatic early prune
During the early design of FCOS and RHCOS, we chose a value of 384M
for the boot partition. This turned out to be too small: some arches
other than x86_64 have larger initrds, kernel binaries, or additional
artifacts (like device tree blobs). We'll likely bump the boot partition
size in the future, but we don't want to abandon all the nodes deployed
with the current size.[[1]]
Because stale entries in `/boot` are cleaned up after new entries are
written, there is a window in the update process during which the bootfs
temporarily must host all the `(kernel, initrd)` pairs for the union of
current and new deployments.
This patch determines if the bootfs is capable of holding all the
pairs. If it can't but it could hold all the pairs from just the new
deployments, the outgoing deployments (e.g. rollbacks) are deleted
*before* new deployments are written. This is done by updating the
bootloader in two steps to maintain atomicity.
Since this is a lot of new logic in an important section of the
code, this feature is gated for now behind an environment variable
(`OSTREE_ENABLE_AUTO_EARLY_PRUNE`). Once we gain more experience with
it, we can consider turning it on by default.
This strategy increases the fallibility of the update system since one
would no longer be able to rollback to the previous deployment if a bug
is present in the bootloader update logic after auto-pruning (see [[2]]
and following). This is however mitigated by the fact that the heuristic
is opportunistic: the rollback is pruned *only if* it's the only way for
the system to update.
Colin Walters [Wed, 26 Apr 2023 13:37:22 +0000 (09:37 -0400)]
ci: Turn off errors for deprecated-declarations
Having `-Werror` on in CI only by default has generally worked OK,
but I don't think it's worth trying to immediately scramble to port
when they deprecate APIs.
Motivated in this case by
```
src/libostree/ostree-fetcher-curl.c: In function 'initiate_next_curl_request':
src/libostree/ostree-fetcher-curl.c:876:3: error: 'CURLOPT_PROTOCOLS' is deprecated: since 7.85.0. Use CURLOPT_PROTOCOLS_STR [-Werror=deprecated-declarations]
876 | rc = curl_easy_setopt (req->easy, CURLOPT_PROTOCOLS, (long)(CURLPROTO_HTTP | CURLPROTO_HTTPS | CURLPROTO_FILE));
| ^~
```
Jonathan Lebon [Fri, 14 Apr 2023 16:21:05 +0000 (12:21 -0400)]
tests/kolainst: Add `make localinstall`
When hacking and testing locally with `cosa build-fast` and `kola run`,
I prefer to leave testing framework stuff within the work directory
rather than installed in my pet container. Add a `localinstall` target
for this which puts the tests in `tests/kola`. Then a simple `kola run`
will pick it up.
Jonathan Lebon [Fri, 14 Apr 2023 16:19:30 +0000 (12:19 -0400)]
tests/kola: delete unused .gitignore
AFAICT, I don't see how `runkola.sh` or the Makefile in `tests/kolainst`
can create files in `tests/kola` since it's geared towards installing
under `/usr`.
Jonathan Lebon [Fri, 14 Apr 2023 16:11:05 +0000 (12:11 -0400)]
lib/sysroot-deploy: Nuke `finalize-failure.stamp` on successful finalization
In the unusual case where one is manually finalizing staged deployments,
as can happen in testing, we expect a successful finalization to remove
the failure stamp file.
Jonathan Lebon [Thu, 13 Apr 2023 21:22:40 +0000 (17:22 -0400)]
lib/sysroot-cleanup: Factor out bootfs cleanup
Crawling through the bootfs and the deployment dirs was already mostly
separate. The only inefficiency here is that we now iterate over the
array of active deployments twice when building the hash tables. No
functional change otherwise.
Dan Nicholson [Fri, 7 Apr 2023 15:47:14 +0000 (09:47 -0600)]
fetcher/soup3: Rewrite without threads
soup3 works best using only the async API from a single thread[1].
Rework the fetcher to stop using worker threads. In order to maximize
session usage across requests, sessions will be reused for each main
context.
Dan Nicholson [Wed, 5 Apr 2023 22:26:10 +0000 (16:26 -0600)]
tests: Ensure real GIO backends aren't used
Set a few environment variables during tests to ensure fake GIO backends
are used. This is particularly important with the soup fetcher backend
as it can cause strange test errors in containerized test environments.
Upstream soup has been setting these 3 environment variables for their
tests since 2015.
Jonathan Lebon [Wed, 1 Mar 2023 14:42:46 +0000 (09:42 -0500)]
ci: update for new kolaTestIso()
The `skipMetal4k` and `skipMultipath` parameters no longer do anything.
Instead use `--denylist-test` which is interpreted directly by the
new `kola testiso` code.
This will run more tests than before. We'll likely cut down tests we
consider redundant soon, but that should be transparent to this code.
Simon McVittie [Fri, 17 Mar 2023 12:56:39 +0000 (12:56 +0000)]
write_content_object: Don't assume file info has standard::size
The file info object for symlinks might validly not have this attribute.
If not, behave as though it was 0, matching what happened with older
versions of GLib.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Simon McVittie [Fri, 17 Mar 2023 12:53:42 +0000 (12:53 +0000)]
ostree_raw_file_to_content_stream: Make size default to 0
Some existing code calls into ostree_raw_file_to_content_stream() with
file objects that do not have the standard::size attribute. Since GLib
2.76.0, attempting to access the size of such an object raises a
critical warning. Handle this more gracefully by defaulting the size
to 0, like earlier versions of GLib did.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Colin Walters [Fri, 17 Mar 2023 12:20:16 +0000 (08:20 -0400)]
repo: Fix nullability for remote options
Unfortunately, the nullability of the output value here is
dependent on whether the `default_value` parameter is provided. There's
no way to express this in introspection or Rust.
Evan Anderson [Thu, 16 Mar 2023 02:33:21 +0000 (21:33 -0500)]
core: Ensure glib standard::size attribute is always set
* A recent change in glib [1] requires that the appropriate attribute
be available when calling getters. ostree core only sets this attribute
on regular files, and frequently triggers the critical warning. Solve
this by setting standard::size to zero for non-regular files.
Colin Walters [Tue, 14 Mar 2023 20:51:49 +0000 (16:51 -0400)]
itest-pull-space: Use mkfs.ext4, align to at least 512b
XFS now seems to want filesystems larger than 300MB, so switch
to ext4. Also use `20MiB` so we align to 512b sectors to squash
a `losetup` warning.
Also tweak some of the numbers to still work.
Philip Withnall [Fri, 10 Mar 2023 11:54:34 +0000 (11:54 +0000)]
lib/fetcher-util: Wake up main context when a request is complete
Since the value of `data->done` is not watched by the main context, the
context doesn’t know to wake up from `g_main_context_iteration()` when
that value is changed. The code currently relies on something else
happening to wake the main context up shortly after `data->done` is set.
That doesn’t seem very reliable, so wake the main context up explicitly.
Spotted this while reading the code while trying to debug a stall with
backtrace:
```
5 0x00007ffff68bbbfb in g_main_context_iteration (context=0x60f000136900, may_block=1) at ../../source/glib/glib/gmain.c:4343
6 0x00007fffdc0e50db in _ostree_fetcher_mirrored_request_to_membuf_once (fetcher=0x604001138c10, mirrorlist=0x603001df18b0, filename=0x7fffdc1049e7 "summary.sig", flags=OSTREE_FETCHER_REQUEST_OPTIONAL_CONTENT, if_none_match=0x602002f3b7f0 "\"640a49ff-250\"", if_modified_since=1678395903, out_contents=0x7fffdadd0e80, out_not_modified=0x7fffdadd0e38, out_etag=0x7fffdadd0e28, out_last_modified=0x7fffdadd0e00, max_size=10485760, cancellable=0x6060004bd720, error=0x7fffdadd0ca0) at src/libostree/ostree-fetcher-util.c:95
7 0x00007fffdc0e52e0 in _ostree_fetcher_mirrored_request_to_membuf (fetcher=0x604001138c10, mirrorlist=0x603001df18b0, filename=0x7fffdc1049e7 "summary.sig", flags=OSTREE_FETCHER_REQUEST_OPTIONAL_CONTENT, if_none_match=0x602002f3b7f0 "\"640a49ff-250\"", if_modified_since=1678395903, n_network_retries=5, out_contents=0x7fffdadd0e80, out_not_modified=0x7fffdadd0e38, out_etag=0x7fffdadd0e28, out_last_modified=0x7fffdadd0e00, max_size=10485760, cancellable=0x6060004bd720, error=0x7fffdadd10c0) at src/libostree/ostree-fetcher-util.c:155
8 0x00007fffdc08d937 in _ostree_preload_metadata_file (self=0x61600057bd80, fetcher=0x604001138c10, mirrorlist=0x603001df18b0, filename=0x7fffdc1049e7 "summary.sig", is_metalink=0, if_none_match=0x602002f3b7f0 "\"640a49ff-250\"", if_modified_since=1678395903, n_network_retries=5, out_bytes=0x7fffdadd0e80, out_not_modified=0x7fffdadd0e38, out_etag=0x7fffdadd0e28, out_last_modified=0x7fffdadd0e00, cancellable=0x6060004bd720, error=0x7fffdadd10c0) at src/libostree/ostree-repo-pull.c:3329
9 0x00007fffdc099712 in ostree_repo_remote_fetch_summary_with_options (self=0x61600057bd80, name=0x6020007f4fd0 "eos-apps", options=0x0, out_summary=0x7fffdadd0f88, out_signatures=0x7fffdadd0f80, cancellable=0x6060004bd720, error=0x7fffdadd10c0) at src/libostree/ostree-repo-pull.c:6675
10 0x00007fffdc06887f in ostree_repo_remote_fetch_summary (self=0x61600057bd80, name=0x6020007f4fd0 "eos-apps", out_summary=0x7fffdadd0f88, out_signatures=0x7fffdadd0f80, cancellable=0x6060004bd720, error=0x7fffdadd10c0) at src/libostree/ostree-repo.c:2706
11 0x00007fffdc18f5de in flatpak_dir_remote_fetch_summary (self=0x60c00577e640, name_or_uri=0x6020007f4fd0 "eos-apps", only_cached=0, out_summary=0x7fffdadd10f0, out_summary_sig=0x7fffdadd10e8, cancellable=0x6060004bd720, error=0x7fffdadd10c0) at /opt/gnome/source/flatpak/common/flatpak-dir.c:12235
12 0x00007fffdc1918cb in _flatpak_dir_get_remote_state (self=0x60c00577e640, remote_or_uri=0x6020007f4fd0 "eos-apps", optional=1, local_only=0, only_cached=0, opt_summary_is_index=0, opt_summary=0x0, opt_summary_sig=0x0, cancellable=0x6060004bd720, error=0x7fffdadd15a0) at /opt/gnome/source/flatpak/common/flatpak-dir.c:12789
13 0x00007fffdc19206d in flatpak_dir_get_remote_state_optional (self=0x60c00577e640, remote=0x6020007f4fd0 "eos-apps", only_cached=0, cancellable=0x6060004bd720, error=0x7fffdadd15a0) at /opt/gnome/source/flatpak/common/flatpak-dir.c:12953
14 0x00007fffdc1784f2 in flatpak_dir_update_appstream (self=0x60c00577e640, remote=0x6020007f4fd0 "eos-apps", arch=0x7fffdc239f30 "x86_64", out_changed=0x0, progress=0x6110007479c0, cancellable=0x6060004bd720, error=0x7fffdadd15a0) at /opt/gnome/source/flatpak/common/flatpak-dir.c:5182
```
I don’t think it’ll fix it (calling `g_main_context_wakeup()` on the
context in the debugger didn’t help), but it can’t hurt.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
Corentin Noël [Thu, 9 Feb 2023 15:28:53 +0000 (16:28 +0100)]
libostree: Enhance the annotation coverage
Increase the coverage of the GObject Introspection annotation and most notably
replace the use of "allow-none" with "out" parameters to "nullable" or "optional"
as the previous annotation is deprecated and ambiguous.
Dan Nicholson [Sun, 29 Jan 2023 16:37:03 +0000 (09:37 -0700)]
bin/remote-refs: Add option to print revisions
The only other way to get the remote ref revision from the CLI is to
scrape the output of `ostree remote summary` or pull the commit. The
revision is already there in the summary's ref map, so might as well add
an option to show it.
Dan Nicholson [Sun, 29 Jan 2023 16:23:13 +0000 (09:23 -0700)]
bin/refs: Add option to print revisions
Allow printing the revision along with the ref. This is very convenient
for looping over the refs in a shell as well as for quickly seeing which
refs are pointed to the same commit.
Dan Nicholson [Tue, 21 Jan 2020 19:43:35 +0000 (12:43 -0700)]
repo: Prevent publishing summary without matching signature
Use a temporary directory for the summary and signature file in
`ostree_repo_regenerate_metadata` so that the summary file isn't
published if signing fails. This prevents publishing a summary without a
signature file or leaving a mismatched signature file in place.
Dan Nicholson [Tue, 21 Jan 2020 18:56:09 +0000 (11:56 -0700)]
lib: Provide internal summary signing with separate directory
Refactor the summary signing APIs to use internal versions where the
directory fd containing the summary can be found. The existing signing
APIs still uses the repo directory fd, but this will allow using a
temporary directory for the summary and signature in the new metadata
generating API.
Dan Nicholson [Tue, 15 Oct 2019 22:46:59 +0000 (16:46 -0600)]
repo: Create metadata commit in ostree_repo_regenerate_metadata
Rather than creating the `ostree-metadata` commit in the summary
builtin, do it in the new `ostree_repo_regenerate_metadata` API. The
commit contents are unchanged and the commit is generated before the
summary as before. To keep from triggering an extra summary update,
automatic summary updating is disabled in the transaction.
Since the summary builtin was already using the new API, it will
continue to generate the `ostree-metadata` commit when the repo has a
collection ID. However, the `ostree_repo_regenerate_summary` API will
still only generate the summary file as before.
If a commit is being made during summary generation, then it would
trigger the summary to be generated again. That's either unwanted busy
work or could result in an infinite loop. Add a boolean in
`OstreeRepoTxn` to disable automatic summary generation as seen fit.
Dan Nicholson [Tue, 15 Oct 2019 17:55:22 +0000 (11:55 -0600)]
bin/summary: Use ostree_repo_regenerate_metadata
Call `ostree_repo_regenerate_metadata` with the provided GPG and sign
keys to generate and sign the summary in one call. This changes the
handling when GPGME support is disabled but GPG keys are supplied.
Instead of silently ignoring the option, it will now error. IMO that's
better as callers would otherwise not know that the summary is not GPG
signed.
projects / ostree.git / log