One thing to note is when we parse the admin related functions,
we still keep the old admin related flags, and added a new parameter
to represent the command struct.
This allows us to identify the caller of the function, making it
easier for us to possibly deduplicate the subcommand handling in
the future. A similar approach is done in rpm-ostree:
https://github.com/projectatomic/rpm-ostree/commit/83aeb018c1012c7a43783c09b74ec71bc9c45826
This also makes it easier for us to change the prototype of the function.
If we want to add something new in the future, we won't need to touch every prototype.
Ruixin Bao [Fri, 13 Oct 2017 16:15:13 +0000 (16:15 +0000)]
ostree: Describe subcommands in help output
Added a description argument to all type
of commands. Now when we include -h or --help
for commands that contain subcommands, the description
for those subcommands are shown.
The added subcommands help will be provided to the following commands:
- ostree -h
- ostree admin -h
- ostree admin instutil -h
- ostree remote -h
- ostree static-delta -h
Philip Withnall [Mon, 16 Oct 2017 15:18:03 +0000 (16:18 +0100)]
lib/repo-finder: Add OstreeRepoFinderOverride
This is another OstreeRepoFinder implementation; it returns results from
a given set of URIs. It’s designed to be used for implementing user
overrides to other repo-finders, or for implementing unit tests.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1281
Approved by: mwleeds
Philip Withnall [Thu, 19 Oct 2017 13:04:10 +0000 (14:04 +0100)]
lib/repo-finder-avahi: Fix a leak in a GVariantIter loop
Use g_variant_iter_loop() rather than next(), since it automatically
handles freeing the child memory each iteration. Previously, we leaked
it for all but the last iteration.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1293
Approved by: cgwalters
Colin Walters [Wed, 18 Oct 2017 00:53:27 +0000 (20:53 -0400)]
Disallow refs starting with a non-letter or digit
Change the regexp for validating refs to require at least one letter or digit
before allowing the other special chars in the set `[.-_]`. Names that start
with `.` are traditionally Unix hidden files; let's ignore them under the
assumption they're metadata for some other tool, and we don't want to
potentially conflict with the special `.` and `..` Unix directory entries.
Further, names starting with `-` are problematic for Unix cmdline option
processing; there's no good reason to support that. Finally, disallow `_` just
on general principle - it's simpler to say that ref identifiers must start with
a letter or digit.
We also ignore any existing files (that might be previously created refs) that
start with `.` in the `refs/` directory - there's a Red Hat tool for content
management that injects `.rsync` files, which is why this patch was first
written.
V1: Update to ban all refs starting with a non-letter/digit, and
also add another call to `ostree_validate_rev` in the pull
code.
Dan Nicholson [Wed, 18 Oct 2017 16:17:19 +0000 (16:17 +0000)]
build: Do *.am.inc substitutions from make correctly
The string needs to be escaped with a $ since we want to replace the
literal string $(path). Without this make will run the command with the
value of the variable, which won't match anything in the input Makefile
stub.
Colin Walters [Fri, 13 Oct 2017 21:40:16 +0000 (17:40 -0400)]
lib/commit: (refactor) Clean up delta bare write API
The way `_ostree_repo_open_content_bare()` did both looking for the object and
possibly creating a new fd was just weird and inconsistent with e.g. the pull
code where we always call `has_object()` first.
Just call `has_object()` in the delta paths that used this too, making the
implementation right now a thin wrapper around
`glnx_open_tmpfile_linkable_at()`, but this is prep for a later patch which does
more.
Colin Walters [Mon, 16 Oct 2017 20:08:26 +0000 (16:08 -0400)]
lib/commit: Use direct fd xattr operations again on regular files
A side effect of commit 8fe45362578a43260876134d6547ebd0bb2485c3 is that
we started listing all xattrs even for files with device/inode matches;
further, we did that using the dfd/name which means we went through
the `/proc` path, which is slower and uglier.
Noticed this in strace while looking at adoption code.
Colin Walters [Fri, 13 Oct 2017 22:14:17 +0000 (18:14 -0400)]
tests: Add test-pull-bareuseronly
I was going to fix a bug in the static deltas code and I noticed
we were missing `pull-test.sh` coverage for bareuseronly 🙈.
Obviously fixing this requires duplicating some of the bits we have in
`basic-test.sh`; need to hoist that into `libtest.sh`. For now though let's get
the coverage.
Colin Walters [Mon, 16 Oct 2017 19:29:38 +0000 (15:29 -0400)]
lib/sysroot: Fix error handling when mounting overlayfs fails
This isn't perfect, but at least we fix an error-overwrite error, and in
practice `ostree admin unlock` isn't wrapped by `rpm-ostree` yet, so spew to
stderr is OK.
Colin Walters [Sat, 14 Oct 2017 02:17:56 +0000 (22:17 -0400)]
lib/commit: Implement "adoption" with CONSUME flag
For checkouts that are on the same device, for regular files we can simply
"adopt" existing files. This is useful in the "build from subtrees" pattern that
happens with e.g. `rpm-ostree install` as well as flatpak and gnome-continuous.
New files are things like an updated `ldconfig` cache, etc. And particularly for
`rpm-ostree` we always regenerate the rpmdb, which for e.g. this workstation is
`61MB`.
We probably should have done this from the start, and instead had a `--copy`
flag to commit, but obviously we have to be backwards compatible.
There's more to do here - the biggest gap is probably for `bare-user` repos,
which are often used with things like `rpm-ostree compose tree` for host
systems. But we can do that later.
Dan Nicholson [Mon, 16 Oct 2017 16:24:16 +0000 (16:24 +0000)]
build: Fix bsdiff Makefile circular dependency
The intended use was to have the .am.inc generated from the .am like the
libglnx one. Without this, make was detecting a circular dependency and
dropping the rule:
Colin Walters [Fri, 13 Oct 2017 18:51:34 +0000 (14:51 -0400)]
lib/pull: Also do commit GPG verification before writing
I was working on a patch to do build on the work done to
import content objects async to do the same for metadata, but right
now we basically rely on writing them first to do the GPG verification
when scanning.
Things will be cleaner for that if we can pass the commit object directly into
`scan_commit_object()` and consistently use `gpg_verify_unwritten_commit()`.
We're careful here to continue to do it both ways (but at most one time), to
account for the case where a bad commit has been pulled and written - we need to
keep failing GPG verification there.
Colin Walters [Fri, 13 Oct 2017 19:22:10 +0000 (15:22 -0400)]
lib/pull: Drop duplicate/different error for GPG but no detached meta
Prep for a later patch to do GPG verification before writing commit objects;
`_ostree_repo_gpg_verify_with_metadata()` already handles this, and so dropping
this gives us consistent error messages.
Simon McVittie [Tue, 3 Oct 2017 20:26:50 +0000 (21:26 +0100)]
Make sure *.am.inc are up to date before `make dist`
v2017.12 didn't include test-libglnx-shutil.c, but if you re-run
autogen.sh (as we do in Debian, to update the Autotools build system)
it will try to build it.
Signed-off-by: Simon McVittie <smcv@collabora.com> Closes: #1274
Approved by: cgwalters
Jonathan Lebon [Tue, 10 Oct 2017 18:14:10 +0000 (18:14 +0000)]
lib/checkout: fallback to checksum for UNION_IDENTICAL
There's a subtle issue going on with the way we use `UNION_IDENTICAL`
now in rpm-ostree. Basically, the crux of the issue is that we checkout
the whole tree from the system repo, but then overlay packages by
checking out from the pkgcache repo. This is an easy way to break the
assumption that we will be merging hardlinks from the same repo.
This ends up causing issues like:
https://github.com/projectatomic/rpm-ostree/issues/1047
There, `vim-minimal` is already part of the host and has an object for
`/usr/share/man/man1/ex.1.gz`. `vim-common` has that same file, but
because it's unpacked in the pkgcache repo first, the hardlinks are not
the same.
There are a few ways we *could* work around this in rpm-ostree itself,
e.g. by re-establishing hardlinks when we do the content pull into the
system repo, but it still felt somewhat hacky. Let's just do this the
proper way and fall back to checksumming the target file if needed,
which is what librpm does as well in this case. Note that we only
checksum if they're not hard links, but they're the same size.
Matthew Leeds [Fri, 13 Oct 2017 21:49:07 +0000 (14:49 -0700)]
lib/utils: Check for invalid UTF-8 in filenames
In case a filename contains invalid UTF-8 characters, libostree will
pass it to g_variant_builder_add() in create_tree_variant_from_hashes()
anyway, which leads to a critical warning from glib and an invalid
commit. This commit makes ostree print a useful error and exit instead.
Colin Walters [Wed, 11 Oct 2017 21:02:02 +0000 (17:02 -0400)]
lib/deploy: Use _exit() for FIFREEZE watchdog
This works around an (IMO) SpiderMonkey bug - it tries to
clean up in a shared library destructor, but doesn't install a
`pthread_atfork()` handler to unset its state.
Colin Walters [Sun, 8 Oct 2017 19:55:35 +0000 (15:55 -0400)]
lib/core: Use GBytes for file headers
This simplifies a lot of code; the header function was structured
to write to an input stream, but many callers only wanted the checksum,
so it's simpler (and error-free) to simply allocate a whole buffer
and checksum that.
For the callers that want to write it, it's also still simpler to allocate the
buffer and write the whole thing rather than having this function do the
writing.
A lot of the complexity here again is a legacy of the packfile code, which is
dead.
This is prep for faster regfile commits where we can avoid `G{In,Out}putStream`.
Colin Walters [Sat, 7 Oct 2017 13:36:51 +0000 (09:36 -0400)]
lib/core: (refactor) Drop wrapper and unneeded args for variant writing
Nothing was using the `bytes_written` data (we always discard partially written
tmpfiles), so simplify everything by dropping it. Further, we always passed an
offset of `0`, so drop that argument too. (I believe that this was previously
used by the "pack files" code that we deleted long ago)
Second, we had an unnecessary internal wrapper for this function; drop that too.
Dan Nicholson [Wed, 11 Oct 2017 14:03:10 +0000 (14:03 +0000)]
lib/deploy: Ignore FIFREEZE/FITHAW errors when already in state
If the filesystem is already frozen, FIFREEZE returns EBUSY, and if the
filesystem is already thawed, FITHAW returns EINVAL. It's very unlikely
these issues would arise on a real ostree system since the sysroot would
be locked during the freeze/thaw cycle.
However, when multiple fake sysroots are used during the test suite (run
as root), the tests could race to run the freeze/thaw cycle without
locking. Furthermore, there's no reason why an independent process might
be trying to freeze the filesystem while ostree was deploying. Ignore
but warn for these errors since there's not much ostree can do about it,
anyways.
Colin Walters [Fri, 6 Oct 2017 20:38:08 +0000 (16:38 -0400)]
lib: Add a lighter weight internal checksum wrapper
The faster (OpenSSL/GnuTLS) code lived in a `GInputStream` wrapper, and that
adds a lot of weight (GObject + vtable calls). Move it into a simple
autoptr-struct wrapper, and use it in the metadata path, so we're
now using the faster checksums there too.
This also drops a malloc there as the new API does hexdigest in place to a
buffer.
Prep for more work in the commit path to avoid `GInputStream` for local file
commits, and ["adopting" files](https://github.com/ostreedev/ostree/pull/1255).
Colin Walters [Tue, 3 Oct 2017 21:34:06 +0000 (17:34 -0400)]
commit: Add _CONSUME modifier flag
For many cases of commit, we can actually optimize things by simply "adopting"
the object rather than writing a new copy. For example, in rpm-ostree package
layering.
We can only make that optimization though if we take ownership of the file. This
commit hence adds an API where a caller tells us to do so. For now, that just
means we `unlink()` the files/dirs as we go, but we can now later add the
"adopt" optimization.
Colin Walters [Thu, 5 Oct 2017 15:14:17 +0000 (11:14 -0400)]
lib/deltas: Move variant read helper into the deltas code
What the deltas code is doing is weird/unfortunate. The name
`ot_variant_read()` conflicts too much with `ot_variant_read_fd()`.
Since nothing else uses it, move it into the deltas code.
Colin Walters [Tue, 3 Oct 2017 01:36:10 +0000 (21:36 -0400)]
lib/pull: Change fetcher to return O_TMPFILE
A lot of the libostree code is honestly too complex for its
own good (this is mostly my fault). The way we do HTTP writes
is still one of those. The way the fetcher writes tempfiles,
then reads them back in is definitely one of those.
Now that we've dropped the "partial object" bits in:
https://github.com/ostreedev/ostree/pull/1176 i.e. commit
https://github.com/ostreedev/ostree/commit/0488b4870e80ef575d8b0edf6f2a9e5ad54bf4df
we can simplify things a lot more by having the fetcher
return an `O_TMPFILE` rather than a filename.
For trusted archive mirroring, we need to enable linking
in the tmpfiles directly.
Otherwise for at least content objects they're compressed, so we couldn't link
them in. For metadata, we need to do similar logic to what we have around
`mmap()` to only grab a tmpfile if the size is large enough.
Colin Walters [Wed, 4 Oct 2017 19:06:31 +0000 (15:06 -0400)]
Deduplicate and fix up our use of mmap()
Buried in this large patch is a logical fix:
```
- if (!map)
- return glnx_throw_errno_prefix (error, "mmap");
+ if (map == (void*)-1)
+ return glnx_null_throw_errno_prefix (error, "mmap");
```
Which would have helped me debug another patch I was working
on. But it turns out that actually correctly checking for
errors from `mmap()` triggers lots of other bugs - basically
because we sometimes handle zero-length variants (in detached
metadata). When we start actually returning errors due to
this, things break. (It wasn't a problem in practice before
because most things looked at the zero size, not the data).
Anyways there's a bigger picture issue here - a while ago
we made a fix to only use `mmap()` for reading metadata from disk
only if it was large enough (i.e. `>16k`). But that didn't
help various other paths in the pull code and others that were
directly doing the `mmap()`.
Fix this by having a proper low level fs helper that does "read all data from
fd+offset into GBytes", which handles the size check. Then the `GVariant` bits
are just a clean layer on top of this. (At the small cost of an additional
allocation)
Side note: I had to remind myself, but the reason we can't just use
`GMappedFile` here is it doesn't support passing an offset into `mmap()`.
Jonathan Lebon [Wed, 4 Oct 2017 14:41:15 +0000 (14:41 +0000)]
lib/commit: minor coverity fix
Appease Coverity by using the same condition for both the ternary check
and the if-condition later on. It should be smart enough to figure out
that `dir_enum == NULL` implies that `dfd_iter != NULL` from the
assertion at the top of the function.
Jonathan Lebon [Tue, 3 Oct 2017 21:34:43 +0000 (21:34 +0000)]
ci: really turn on warnings
We didn't have `-Wall` in our `CFLAGS`. It's normally injected by
`configure.ac`, but because we *did* have `-Werror`, it was skipped.
Now, we just turn it on unconditionally directly in `build.sh`.
Jonathan Lebon [Tue, 3 Oct 2017 19:00:32 +0000 (19:00 +0000)]
app/main: use HAVE_LIBCURL_OR_LIBSOUP for pull CLI
We want `pull` to be included as long as we have at least either
`libcurl` or `libsoup` to back it. Of course, this is a moot point for
now since `libsoup` is currently a build requirement.
Philip Withnall [Tue, 3 Oct 2017 14:45:34 +0000 (15:45 +0100)]
lib/repo-commit: Import detached metadata even if hardlink exists
Spotted while reading through the code, it looks like the
copy_detached_metadata() call is accidentally omitted if a hardlink
already exists for the .commit object.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1242
Approved by: cgwalters
Matthew Leeds [Tue, 3 Oct 2017 05:35:17 +0000 (22:35 -0700)]
lib/gpg: Print debug info when reading GPG keys
This commit adds debug output whenever libostree reads GPG keys, which
can come from different locations in the file system. This is especially
helpful in debugging "GPG signatures found, but none are in trusted
keyring" errors, which in my case was caused by OSTree looking in
/usr/local/share/ostree/trusted.gpg.d/ rather than
/usr/share/ostree/trusted.gpg.d/.
Colin Walters [Mon, 2 Oct 2017 19:36:47 +0000 (15:36 -0400)]
lib/pull: Minor cleanup to metadata scanning function, add docs
I'm regretting a bit having the `guint8*csum` variant of checksums
except for the serialized form. Once we start doing processing
it's easier to just have it remain hex.
Do an on-stack conversion for the metadata scanning function; this
drops a malloc and also just looks nicer.
Also add some long-awaited function comments to the two.
Philip Withnall [Mon, 2 Oct 2017 17:04:37 +0000 (18:04 +0100)]
lib/bloom: Add some missing preconditions on n_bytes
These shouldn’t change the bloom filter’s behaviour at all, but make it
a bit more obvious what the programmatical limitations are on the sizes
it can deal with.
In reality, those sizes should never be reached because they won’t fit
in a DNS-SD record.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1239
Approved by: cgwalters
Colin Walters [Mon, 2 Oct 2017 15:24:05 +0000 (11:24 -0400)]
lib/pull: Fix regression with pull-local for nonexistent refs
I was reading the pull code for the last release, and spotted
a bug in commit f923c2e1eaebe0c781f07d34ae1a03f94357bccd - in
the case where the ref doesn't exist, we don't set an error,
tripping an assertion in the main code.
The previous code wanted the ref to always exist, so just flip back the boolean
for "ignore noent". I moved the `g_strchomp()` just into the HTTP path - if a
local repo is corrupted in this way it's something to fix in that repo.
Philip Withnall [Mon, 11 Sep 2017 13:42:52 +0000 (14:42 +0100)]
ostree/summary: Generate an ostree-metadata ref when updating summary
This is the new way of publishing repository metadata, rather than as
additional-metadata in the summary file. The use of an ostree-metadata
ref means that the metadata from multiple upstream collections is not
conflated when doing P2P mirroring of many repositories.
The new ref is only generated if the repository has a collection ID set.
The old summary file continues to be generated for backwards
compatibility (and because it continues to be the canonical ref →
checksum map for the repository).
The new code is only used if configured with --enable-experimental-api.
Includes unit tests.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1158
Approved by: cgwalters
Philip Withnall [Sat, 30 Sep 2017 23:13:17 +0000 (00:13 +0100)]
lib/bloom: Fix a -Wconversion warning in OstreeBloom
Compiling with -Wconversion warns on this line, as the conversion from
guint64 to guint8 is implicit (but safe: there is no bug here, since the
implicit cast is applied after the modulus arithmetic).
Make the cast explicit to silence -Wconversion.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1231
Approved by: cgwalters
Jonathan Lebon [Thu, 28 Sep 2017 18:57:33 +0000 (18:57 +0000)]
tests: check for relabeling rather than overlay
Instead of checking for overlayfs, let's explicitly check for our
ability to relabel files since we now have a `libtest` function to do
this. Also port that logic to `libostreetest`.
Note that overlayfs *does* allow manipulating user xattrs. So ideally,
we should break down `OSTREE_NO_XATTRS` further to distinguish between
tests that use bare repos from other modes.
We check the current directory instead of `/` so that developers can
just point `TEST_TMPDIR` to a non-overlayfs mount point when hacking
from a container.
Jonathan Lebon [Thu, 28 Sep 2017 19:08:06 +0000 (19:08 +0000)]
lib/commit: don't query devino cache for modified files
We can't use the cache if the file we want to commit has been modified
by the client through the file info or xattr modifiers. We would
prematurely look into the cache in `write_dfd_iter_to_mtree_internal`,
regardless of whether any filtering applied.
We remove that path there, and make sure that we only use the cache if
there were no modifications. We rename the `get_modified_xattrs` to
`get_final_xattrs` to reflect the fact that the xattrs may not be
modified.
One tricky bit that took me some time was that we now need to store the
st_dev & st_ino values in the GFileInfo because the cache lookup relies
on it. I'm guessing we regressed on this at some point.
This patch does slightly change the semantics of the xattr callback.
Previously, returning NULL from the cb meant no xattrs at all. Now, it
means to default to the on-disk state. We might want to consider putting
that behind a flag instead. Though it seems like a more useful behaviour
so that callers can only override the files they want to without losing
original on-disk state (and if they don't want that, just return an
empty GVariant).
Colin Walters [Thu, 28 Sep 2017 19:41:02 +0000 (15:41 -0400)]
libarchive: Fix assertion error if used with CANONICAL_PERMISSIONS
I was trying to do a change for rpm-ostree to use
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS`
for container builds with `bare-user-only,` but hit an assertion here
ultimtely because we weren't setting `standard::type`.
Rather than hand-rolling `GFileInfo` creation, use the stat buffer conversion
code which is more robust and used in multiple places already.
Philip Withnall [Mon, 18 Sep 2017 14:43:41 +0000 (15:43 +0100)]
tests: Add a manual integration test for OstreeRepoFinderMount
Test an end-to-end flow of pulling refs from an online
repository → local OS repository → create a USB stick of
them → pull to a local OS repository on another machine.
This is a manual test, as it requires a throwaway USB stick which the
test can format as ext4 or vfat to test the flow works with both file
systems.
Run it as:
MOUNT_INTEGRATION_DEV=/dev/sdb1 make check \
TESTS=tests/test-repo-finder-mount-integration.sh
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1203
Approved by: cgwalters
Colin Walters [Tue, 26 Sep 2017 21:21:59 +0000 (17:21 -0400)]
lib/pull: Don't request deltas for unchanged commits
I noticed this while debugging why I was seeing "2 metadata objects" fetched for
a different PR. I knew 1 was detached meta, but the other turned out to be this.
There's no reason to request a delta if the ref is unchanged.
Colin Walters [Wed, 27 Sep 2017 00:51:24 +0000 (20:51 -0400)]
tests/commit-sign: Update a bit to more modern style
I was trying to debug this while working on another PR. Add `echo ok` lines and
also use `assert_file_has_content` more directly; there's no reason to use an
intermediate `grep` since if it fails it won't print the original file.
Philip Withnall [Thu, 21 Sep 2017 15:08:26 +0000 (16:08 +0100)]
lib/repo-pull: Fix remote names in refspecs from non-mirror P2P pulls
Propagate the refspec_name from the OstreeRemote returned by an
OstreeRepoFinder through to the set_ref() call.
This changes ostree_repo_pull_with_options() to accept the
previously-disallowed combination of passing override-remote-name in
options and also setting a remote name in remote_name_or_baseurl.
ostree_repo_pull_with_options() will continue to pull using the remote
config named in remote_name_or_baseurl as before; but will now use the
remote name from override-remote-name when it’s setting the refs at the
end of the pull. This is consistent with the documentation for
override-remote-name.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1202
Approved by: cgwalters
Philip Withnall [Thu, 21 Sep 2017 15:04:11 +0000 (16:04 +0100)]
lib/remote: Store name of remote providing keyring for dynamic remotes
When pulling from a dynamic (peer to peer) remote, the remote’s name is
set to a unique, generated string which doesn’t exist in repo/config. If
doing a non-mirror pull, however, we don’t want to use this name in the
refspecs for newly created or updated refs — we want to use the name of
the remote which provided the keyring for the pull (this will be a
remote from repo/config whose collection ID matches that being used for
the peer to peer pull).
Store both names in OstreeRemote. The name to use for refspecs is stored
as refspec_name, and is typically NULL unless it differs from name.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1202
Approved by: cgwalters
Philip Withnall [Thu, 21 Sep 2017 15:00:05 +0000 (16:00 +0100)]
lib/repo: Change resolve_keyring_for_collection() to return a remote
Instead of returning just the keyring filename, return the entire
OstreeRemote, which has the keyring filename as one of its members. This
will simplify some upcoming changes, and allows slightly improved debug
logging.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1202
Approved by: cgwalters
Philip Withnall [Thu, 21 Sep 2017 15:01:44 +0000 (16:01 +0100)]
lib/repo-pull: Fix a potential minor leak
If override-remote-name is specified in the options to
ostree_repo_pull_with_options(), but the remote_name_or_baseurl argument
is also set to a remote name, the override-remote-name would be leaked.
Note that this is currently an invalid configuration, so this leak is
basically never hit.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1202
Approved by: cgwalters
Keep track of how many objects we imported, and print that for `ostree
pull-local` (also do this even if noninteractive, like we did for `pull`).
In implementing this at first I used separate variables for import
from repo vs import from localcache, but that broke some of the
tests that checked those values.
It's easier to just merge them; we know from looking at whether or not
`remote_repo_local` is set whether or not we were doing a "HTTP pull with
localcache" versus a true `pull-local` and can use that when rendering status.
Philip Withnall [Tue, 26 Sep 2017 14:17:47 +0000 (15:17 +0100)]
lib/repo-pull: Use resolve() instead of list() in fetch_ref_contents()
This is more efficient in the non-collection case; in the collection
case, the implementation of ostree_repo_resolve_collection_ref() needs
to be rewritten to improve efficiency.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1182
Approved by: cgwalters
Philip Withnall [Tue, 26 Sep 2017 14:15:57 +0000 (15:15 +0100)]
lib/repo-refs: Add first version of ostree_repo_resolve_collection_ref()
This is a parallel for ostree_repo_resolve_rev_ext() which works on
collection–refs. At the moment, the implementation is simple and uses
ostree_repo_list_collection_refs(). In future, it could be rewritten to
check the checksum directly rather than enumerating all
potentially-relevant checksums.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1182
Approved by: cgwalters
Philip Withnall [Mon, 18 Sep 2017 13:01:21 +0000 (14:01 +0100)]
src/pull: Support local pulls for collection–refs
Previously, collection–refs could only be pulled from a repository if it
had a summary file (which listed them). There was no way to pull from a
local repository which doesn’t have a summary file, and where the refs
were stored as refs/remotes/$remote/$ref, with a config section linking
that $remote to the queried collection ID.
Fix that by explicitly supporting pull_data->remote_repo_local in
fetch_ref_contents().
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1182
Approved by: cgwalters
Colin Walters [Tue, 26 Sep 2017 16:41:07 +0000 (12:41 -0400)]
tests,ci: Move "test-basic" (bare mode) to installed test
Our CI uses default Docker, which has SELinux labeling but is rather
evil in returning `EOPNOTSUPP` to any attempts to set `security.selinux`,
even if to the same value.
The previous fire 🔥 for this was: https://github.com/ostreedev/ostree/pull/759
The `bare` repo mode really only makes sense as uid 0, so our installed
test framework is a good match for this. However, the unit tests *do*
work in a privileged container even as non-root, and *also* should
work on SELinux-disabled systems. So let's teach the test framework
how to skip in those situations.
I tested this both in a priv container (my default builder) and an unpriv
container (like our CI).
At the same time, start executing the `test-basic.sh` from an installed test,
so we get better coverage than before.
This is just the start - all of the sysroot tests really need the
same treatment.
Colin Walters [Sat, 23 Sep 2017 14:23:47 +0000 (10:23 -0400)]
lib/pull: Default checksum for archive mirror, add TRUSTED_HTTP flag
I now think commit fab1e113db558cb7d6754e243919558df92d4864 was a mistake;
because it breaks the mental model that at least I'd built up that "local repos
don't have checksums verified, HTTP does".
For example, a problem with this is (with that mental model in place) it's easy
for people who set up mirrors like this to then do local pulls, and at that
point we've done a deployment with no checksum verification.
Further, since then we did PR #671 AKA commit 3d38f03 which is really most of
the speed hit.
So let's switch the default even for this case to doing checksum verification,
and add `ostree pull --http-trusted`. People who are in situations where they
know they want this can find it and turn it on.
Colin Walters [Sat, 23 Sep 2017 13:59:53 +0000 (09:59 -0400)]
lib/pull: Refactor to use OstreeRepoImportFlags internally
Rather than carrying two booleans, just convert `OstreeRepoPullFlags`
into `OstreeRepoImportFlags`. This allows us to drop an internal
wrapper function and just directly call `_ostree_repo_import_object()`.
This though reveals that our mirroring import path doesn't check the
`OSTREE_REPO_PULL_FLAGS_UNTRUSTED` flag...it probably should.
projects / ostree.git / log