Simon McVittie [Sun, 22 Nov 2020 13:17:24 +0000 (13:17 +0000)]
test-pull-summary-sigs: Set timestamps to serve expected files
If this is not done, the test can fail when the temporary directory is
a tmpfs: for example this happens during build-time testing with /var/tmp
on tmpfs or TEST_TMPDIR pointing to a tmpfs, or installed-tests with
gnome-desktop-testing-runner allocating the test directory on a tmpfs.
In particular, many of Debian's official autobuilders now do the entire
build and test procedure in a chroot hosted on a tmpfs, to improve build
performance and prevent fsync overhead.
In this situation, it appears that overwriting summary.sig with a copy
of summary.sig.2 is not sufficient for the web server to tell the
libostree client that it needs to be re-downloaded. I'm not completely
sure why, because tmpfs does appear to have sub-second-resolution
timestamps, but forcing a distinct mtime is certainly enough to
resolve it.
Resolves: https://github.com/ostreedev/ostree/issues/2245
Bug-Debian: https://bugs.debian.org/975418 Signed-off-by: Simon McVittie <smcv@collabora.com>
Luca BRUNO [Wed, 18 Nov 2020 10:24:19 +0000 (10:24 +0000)]
workflow/release: further refinements
This tweaks the release GH workflow further so that it only triggers
when the `configure.ac` file (which owns the version) changes.
Plus it properly checkouts the PR branch to avoid wrongly looking
at a synthetic merge commit.
pull: Don't save into cache passed in GByte summaries
The cache shouldn't be affected by the user passing in some other
summary as it may not be the "official one".
I ran into this in flatpak where the passed summary was correct, but
the re-saving of the cache updated the mtime of the cached file which
led to later http If-Modified-Since calls failing to update.
Jonathan Lebon [Mon, 2 Nov 2020 19:53:26 +0000 (14:53 -0500)]
Drop use of `volatile`
As detailed in
gitlab.gnome.org/GNOME/glib/-/issues/600#note_877282, volatile
isn't actually needed in these contexts because the atomic operations
already give us strong enough guarantees. In GCC 11, this triggers a
diagnostic due to the volatile qualifier getting dropped anyway.
There is a WIP to do the same in glib:
https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1719
This obsoletes this downstream patch:
https://src.fedoraproject.org/rpms/ostree/c/b8c5a6fb
ostree_repo_gpg_sign_data: Fix API doc argument name
I got:
src/libostree/ostree-repo.c:5232: Warning: OSTree: ostree_repo_gpg_sign_data: unknown parameter 'out_signature' in documentation comment, should be 'out_signatures'
Jonathan Lebon [Tue, 27 Oct 2020 15:58:14 +0000 (11:58 -0400)]
lib/fetch-curl: Unref timeout source
The timeout timer should always be one-shot, so let's just always
destroy it in the callback. The main context has its own ref on it, so
it won't be freed behind its back.
This *should* fix a leak that was brought up in
https://bugzilla.redhat.com/show_bug.cgi?id=1891761.
William Manley [Tue, 14 Jul 2020 12:32:30 +0000 (13:32 +0100)]
Tests: Refactor bootloader-entries-crosscheck
I've made this use functions to make it easier to add support for more
bootloaders. Seeing as there will be a big diff anyway I've also adjusted
the formatting to make it pep8 compliant.
William Manley [Mon, 6 Jul 2020 15:12:29 +0000 (16:12 +0100)]
Add support for explicitly requesting any specific bootloader type
...with the `sysroot.bootloader` configuration option. This can be useful
when converting a system to use `ostree` which doesn't currently have a
bootloader configuration that `ostree` can automatically detect, and is
also useful in combination with the `--sysroot` option when provisioning a
rootfs for systems other than the one you're running `ostree admin deploy`
on.
William Manley [Mon, 6 Jul 2020 14:50:28 +0000 (15:50 +0100)]
Refactor: sysroot.bootloader: Store enum value rather than string
It's easier to extend and it centralises the config parsing. In other
places we will no longer need to use `g_str_equal` to match these values,
a `switch` statement will be sufficient.
pull: Only download summary if we need it for the pull operation
If we have a commit id for all the refs we're pulling, and if we
don't need the summary to list all the refs when mirroring then the
only reason to download the summary is for the list of deltas.
With the new "indexed-deltas" property in the config file (and mirrored
to the summary file) we can detect when we don't need the summary for
deltas and completely avoid downloading it then.
deltas: Set `indexed-deltas` key in the config and summary
Clients can use these during pull and avoid downloading the summary if
needed, or use the indexed-deltas instead of relying on the ones in
the summary which may be left out.
deltas: Make ostree_repo_static_delta_reindex() public
It is useful to be able to trigger this without having to regenerate
the summary. For example, if you are not using summaries, or ar generating
the summaries yourself.
deltas: Update delta indexes when updating summary
When we update the summary file (and its list of deltas) we also update
all delta indexes. The index format is a single `a{sv}` variant identical
to the metadata-part of the summary with (currently) only the
`ostree.static-deltas` key.
Since we expect most delta indexes to change rarely, we avoid
unnecessary writes when reindexing. New indexes are compared to
existing ones and only the changed ones are written to disk. This
avoids unnecessary write load and mtime changes on the repo server.
This gets the subpath for a delta index file, which is of the form
"delta-indexes/$commit.index", that contains all the deltas going
to the particular commit.
Philip Withnall [Fri, 9 Oct 2020 17:46:06 +0000 (18:46 +0100)]
ostree/trivial-httpd: Add Last-Modified/ETag support
This is basic support for the
Last-Modified/ETag/If-Modified-Since/If-None-Match headers. It’s not
high performance, and doesn’t support all of the related caching
features (like the If-Match header, etc.).
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
Philip Withnall [Fri, 9 Oct 2020 17:34:55 +0000 (18:34 +0100)]
tests: Add simple test for summary file caching
This test would have actually passed before the summary file caching
changes (in the previous few commits) were added, as the `summary.sig`
essentially acted as the ETag for the summary file, and itself wasn’t
updated on disk if it didn’t change when querying the server.
Actually testing that the HTTP caching headers are working to reduce
HTTP traffic would require test hooks into the pull code or the
trivial-httpd server, neither of which I have the time to add at the
moment.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
Philip Withnall [Tue, 29 Sep 2020 09:51:26 +0000 (10:51 +0100)]
lib/pull: Hook up HTTP caching headers for summary and summary.sig
As `summary` and `summary.sig` aren’t immutable, HTTP requests to
download them can be optimised by sending the `If-None-Match` and
`If-Modified-Since` headers to avoid unnecessarily re-downloading them
if they haven’t changed since last being checked.
Hook them up to the new support for that in the fetcher.
The `ETag` and `Last-Modified` for each file in the cache are stored as
the `user.etag` xattr and the mtime, respectively. For flatpak, for
example, this affects the cached files in
`~/.local/share/flatpak/repo/tmp/cache/summaries`.
If xattrs aren’t supported, or if the server doesn’t support the caching
headers, the pull behaviour is unchanged from before.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
Philip Withnall [Thu, 24 Sep 2020 16:49:48 +0000 (17:49 +0100)]
libostree: Add support for ETag and Last-Modified headers
Add support in the soup and curl fetchers to send the `If-None-Match`
and `If-Modified-Since` request headers, and pass on the `ETag` and
`Last-Modified` response headers.
This currently introduces no functional changes, but once call sites
provide the appropriate integration, this will allow HTTP caching to
happen with requests (typically with metadata requests, where the data
is not immutable due to being content-addressed). That should reduce
bandwidth requirements.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
Jonathan Lebon [Sat, 3 Oct 2020 15:23:49 +0000 (11:23 -0400)]
Add Packit integration
This is a basic `.packit.yaml` integration file which will allow us have
continuous builds of OSTree in cosa and upstream CI. If things go well,
we'll likely deploy this in other build tools like rpm-ostree.
Prompted by wanting to get #2155 out to unblock
https://github.com/coreos/rpm-ostree/pull/2170.
projects / ostree.git / log