Colin Walters [Mon, 30 Jan 2017 10:21:43 +0000 (11:21 +0100)]
travis: Disable tests (but keep builds) on flaky distros
I went through the Travis history a bit, and these seem to be
the flaky ones. The ubuntu one is likely no libsoup patches.
The other one @smcv has partially traced to a GPGME race condition
or something like that.
For the libsoup one; as I say in comments, once we have libcurl, I'd like to
enable that mostly everywhere, which should (hopefully) be more reliable.
Colin Walters [Mon, 30 Jan 2017 09:55:22 +0000 (10:55 +0100)]
lib: Prefix GPG errors with the checksum
I was working on https://bugzilla.redhat.com/show_bug.cgi?id=1393545
and it was annoying that I couldn't know what the new (unsigned)
commit has was until verification succeeded. I could pull it
manually without GPG, but then it'd be sitting in the repo.
Colin Walters [Wed, 18 Jan 2017 18:35:42 +0000 (13:35 -0500)]
tests: Add a big (many objects) pull
This would be more likely to tickle things like
https://github.com/ostreedev/ostree/issues/601
reliably.
Also, while working on the curl backend, I hit on the fact that curl doesn't
queue (by default, you can enable) and will happily create 20000+ concurrent TCP
connections if you try. Having this test would have made that more likely to
fail.
Colin Walters [Thu, 1 Dec 2016 14:28:24 +0000 (09:28 -0500)]
Add support for more selective pruning
There are use cases for having a single repo with branches
with different lifecycles; a simple example of what I was
trying to do in CentOS Atomic Host work is have "stable"
and "devel" branches, were we want to prune devel, but
retain *all* of stable.
This patch is split into two parts - first we add a low level "delete all
objects not in this set" API, and change the current prune API
to use this.
Next, we move more logic into the "ostree prune" command. This paves the way for
demonstrating how more sophisticated algorithms/logic could be developed outside
of the ostree core.
Also, the --keep-younger-than logic already lived in the commandline, so it
makes sense to keep extending it there.
Simon McVittie [Thu, 19 Jan 2017 11:34:57 +0000 (11:34 +0000)]
Make corrupt-repo-ref.js executable
Debian's Lintian packaging consistency check complains that it isn't
executable but has a #! line. In fact it's reasonable to run this
script directly, so make it executable, and put it in a _scripts
variable so it will be installed executable.
Simon McVittie [Thu, 19 Jan 2017 11:23:30 +0000 (11:23 +0000)]
Sourced test snippets: remove shebang and make non-executable
They are installed non-executable, which makes Debian's Lintian
packaging consistency check complain that #! is only useful
in executable scripts. But in fact they are not useful to execute
directly (they rely on setup being done in the script that sources
them), so just chmod them -x.
Colin Walters [Mon, 9 Jan 2017 01:44:20 +0000 (20:44 -0500)]
tests: Don't inject newline in URL
It turns out libsoup strips all whitespace even *inside* a URL. We could do that
for libcurl too but...really, people shouldn't do that. In this test we were
adding the trailing newline into the URL. If someone complains who is using the
libcurl code we can deal with it then.
Colin Walters [Mon, 9 Jan 2017 15:02:19 +0000 (10:02 -0500)]
fetcher: Rework API to use strings for tls keys/db
This is prep for the libcurl porting. `GTlsCertificate/GTlsDatabase` are
abstract classes implemented in glib-networking for gnutls. curl's APIs take
file paths as strings, so it's easier to work on both if we move the GLib TLS
bits into the libsoup code.
Colin Walters [Thu, 19 Jan 2017 01:56:28 +0000 (20:56 -0500)]
pull: Fix theoretical checksum collision for metadata fetches
I was making some other changes in this code, and noticed that we were adding
checksums without object types into the same hash table for metadata. We should
*never* do this with both metadata content objects, since in theory a content
object could have the same hash as metadata.
I don't actually think it's possible in practice for pure metadata to collide,
since they have different structures, but let's do this anyways since it's
conceptually right.
Colin Walters [Tue, 17 Jan 2017 15:16:31 +0000 (10:16 -0500)]
tests: Alias assert_not_reached() -> fatal()
We had a lot of copies of the "echo something 1>&2; exit 1" code even though
`assert_not_reached()` was it. Hence, I think we need a shorter alias for that.
Doing this particularly since I noticed a missing `1` in an `exit 1` call in the
rpm-ostree copy of this.
Colin Walters [Tue, 3 Jan 2017 15:45:58 +0000 (10:45 -0500)]
pull: Rework delta superblock fetches to be async
For the pending libcurl port, the backend is a bit more
sensitive to the main context setup. The delta superblock
fetch here is a synchronous request in the section that's
supposed to be async.
Now, libsoup definitely supports mixing sync and async requests, but it wasn't
hard to help the libcurl port here by making this one async. Now fetchers are
either sync or async.
Colin Walters [Fri, 30 Dec 2016 19:18:34 +0000 (14:18 -0500)]
Split trivial-httpd into separate binary
Working on the libcurl backend, I hit the issue that the trivial-httpd program
depends on libsoup. I briefly considered having two versions, but libcurl is
client only, and moreover trivial-httpd is no longer trivial - it has various
features which are used by the test suite extensively.
Hence, what we'll do is build it as a separate binary which links to libsoup,
and use it during the tests. We *also* currently still provide `ostree
trivial-httpd` since some things use it like `rpm-ostree-toolbox` and the
Cockpit tests.
After those are ported to use some other webserver, I plan to add a build-time
option to drop it.
Colin Walters [Wed, 28 Dec 2016 19:43:28 +0000 (14:43 -0500)]
fetcher: Split lowlevel API into file/membuf variants
The previous commit introduced a single low level API - however,
we can do things in a more optimal way for the curl backend if
we drop the "streaming API" variant. Currently, we only use
it to synchronously splice into a memory buffer...which is pretty
silly when we could just do that in the backend.
The only tweak here is that we have an "add NUL character" flag that is
(possibly) needed when fetching into a membuf.
The code here ends up being better I think, since we avoid the double return
value for the `_finish()` invocation, and now most of the fetcher code (in the
soup case) writes to a `GOutputStream` consistently.
This will again make things easier for a curl backend.
Colin Walters [Fri, 23 Dec 2016 01:34:07 +0000 (20:34 -0500)]
fetcher: Move high level functions into "fetcher-util"
Conceptually these now lay on top of the core API, and don't reference libsoup.
This is preparation for libcurl porting, but it's also just generally better.
Colin Walters [Thu, 22 Dec 2016 22:57:04 +0000 (17:57 -0500)]
fetcher: Hoist core "mirrored request" API to public
This is in preparation for the libcurl port. We're basically making public what
we had internally. The next step here is to create `ostree-fetcher-util.[ch]`
that only operates in terms of this lower level API.
Also drop the `_mirrored` from the function name since it's
the default now.
Colin Walters [Fri, 23 Dec 2016 14:59:30 +0000 (09:59 -0500)]
docs: Fix ostree.version -> version
I think originally I had envisioned this as `ostree.version`, but at the last
minute we changed it to just `version`. That's what all of the code uses, so
let's fix the docs.
Dan Nicholson [Thu, 22 Sep 2016 18:53:19 +0000 (11:53 -0700)]
repo: Fix list_objects annotations
Without the element-type annotations, bindings don't know how to handle
the elements of the hash table. Since the table is created with destroy
functions, the caller does not own the elements, so transfer container
is used.
Dan Nicholson [Wed, 21 Dec 2016 11:59:08 +0000 (05:59 -0600)]
repo: Fix object list keys ownership
ostree_object_name_serialize returns a floating ref, so sink it before
adding it to the hash table so it can properly be freed later when the
hash table is destroyed.
This is particularly a problem for pygobject, which sinks the refs on
variants as it marshals them to native python types. If the ref isn't
already sunk, then the ref count won't increase and a critical warning
will be raised when both the hash table and pygobject try to unref it.
static-delta: Pretend that world unreadable objects are new objects
This will prevent including in the delta the bits to update files that
are not world readable, so that we don't run into a permissions problem
when applying the deltas from a bare-user repository that has a bare
repository set as its parent.
This is the case for Endless when updating flatpak runtimes, as the
temporary directory created in ~/.local/share/flatpak/system-cache will
be of type bare-user with its parent set to /var/lib/flatpak which is a
bare repository in EOS, as it's shared with the one at /ostree/repo.
Colin Walters [Fri, 9 Dec 2016 04:05:39 +0000 (23:05 -0500)]
lib: Squash last use of GFile deltas_dir
I was having this thought today about making more of the OS readonly,
and ultimately if we got to the point where all ostree operations are
through the repo and sysroot dfds, we could have rpm-ostree be the
only process holding those fds open, and have a read-only bind mount
on top.
Anyways, we're not there, likely won't be soon, but this gets us
closer to being fully fd relative.
Colin Walters [Thu, 8 Dec 2016 02:26:43 +0000 (21:26 -0500)]
ci: Combine UBSAN and ASAN by default
I only recently realized this was possible. While we're still seeing
leaks in the CI environment for some reason, adding ASAN gives us
use-after-free detection etc., which is obviously still very useful
even if we're not doing leak checking.
Colin Walters [Thu, 8 Dec 2016 18:35:29 +0000 (13:35 -0500)]
ci: Drop sudo installed tests
This conflicts with the ASAN work...and in general, I think I'd like
to make a new format for tests that require root, and have them be
defined to be in mutable containers or VMs.
Our coverage loss from this isn't much because some of these tests
already required `CAP_SYS_ADMIN` which we didn't have in Docker
anyways.
While we have the patient open, parallelize the regular installed
tests.
Colin Walters [Thu, 8 Dec 2016 02:59:40 +0000 (21:59 -0500)]
Skip gjs-based tests if ASAN is enabled
Unfortunately, introspection uses dlopen(), which doesn't quite
work when the DSO is compiled with ASAN but the outer executable
isn't.
Trying to inject LD_PRELOAD=libasan means the outer executable has to
be leak free...which, yeah, I'm not going to get into running ASAN
today on gjs or pygobject.
So, let's skip those tests - ideally, we still run them in some other
context without the sanitizers. The coverage we have from them is
middling anyways.
Colin Walters [Thu, 8 Dec 2016 19:20:19 +0000 (14:20 -0500)]
repo: Add unconfigured-state to remote config options
This is a migration from the origin version. It's
nicer to have it in the remote, since that's what one
needs to change. Then tools don't need to mess with
the origin file.o
In fact in this scenario one can keep the "media source" like
`file:///install/repo` or whatever, since conceptually that's where it
came from. We're just providing a better error.
Dan Nicholson [Fri, 9 Dec 2016 12:37:47 +0000 (06:37 -0600)]
repo: Fix annotations for remote_fetch_summary functions
These are out parameters, so add the (out) annotation and switch
(nullable) to (optional) since the latter is used for the purpose of
optional out parameters.
Colin Walters [Thu, 8 Dec 2016 15:32:09 +0000 (10:32 -0500)]
tree-wide: Switch to autoptr for GOptionContext
We were leaking in a few places that I noticed in an ASAN run. Also,
this was one of the last non-autoptr cleanup sections we have in
`out:` cleanup sections, making us a lot closer to a potential
full-tree rewrite to `return FALSE`.
Colin Walters [Thu, 8 Dec 2016 02:16:52 +0000 (21:16 -0500)]
build: Make libsoup optional again
The "remote cookies" code broke this. While I'm not sure anyone is
actually using ostree-without-http, it isn't too hard to keep the
build time conditional going. Further, this work is preparatory for
libcurl porting.
Colin Walters [Tue, 6 Dec 2016 16:30:19 +0000 (11:30 -0500)]
lib: Ensure we use _GNU_SOURCE in enum templates
Due to the way glib-mkenums runs the preprocessor itself, it
doesn't pick up the `AC_USE_SYSTEM_EXTENSIONS()` that we have in
`configure.ac`.
This blew up in an obscure way when I later wanted to `#include
"libglnx.h"` in one of the headers, since it needs the `basename()`
from `string.h` which is only available with `_GNU_SOURCE`.
Colin Walters [Mon, 5 Dec 2016 22:22:46 +0000 (17:22 -0500)]
lib: Always checksum content in deltas
This is a follow up to conversation on list - in practice, if we're
backing away from summary signing, then it makes sense to remove the
special casing for checksums in deltas around summary signatures.
This is also related to the recent change to enable GPG checking for
commits in deltas - now we have a more coherent story between the
previous pull path and deltas.
I didn't do any performance checking, and while it's slightly annoying
that we're now doing sha256 on the delta content twice (once for the
part and once per object)...sha256 is pretty fast, I think most users
are I/O bound anyways, and it'd drop even farther if we started using
openssl.
Colin Walters [Mon, 5 Dec 2016 18:08:39 +0000 (13:08 -0500)]
tests/keyfile-utils: Drop tests covering preconditions
The spam in stderr was bothering me, and further at some eventual
point in the future we want to annotate the functions with
`__attribute__((nonnull))` which would then cause tests like these to
become undefined behavior.
The coverage of this isn't worth the log spam basically.
Simon McVittie [Thu, 1 Dec 2016 16:28:27 +0000 (16:28 +0000)]
Terminate individual tests after (10 * $TEST_TIMEOUT_FACTOR) minutes
While using the Automake parallel test harness, if a test hangs for
long enough for an external watchdog to kill the entire build process
(as happens in Debian sbuild after 150 minutes with no activity on
stdout/stderr), the logs will not be shown. If we make an individual
test time out sooner, logs are more likely to be shown.
We use SIGABRT so that the process(es) under test will dump core,
allowing the point at which ostree is blocking to be analyzed.
After 1 minute, if any have not died, we kill them again with SIGKILL.
To support slow platforms and slow debugging tools, if
TEST_TIMEOUT_FACTOR is set, multiply the 10 minute timeout by that.
Signed-off-by: Simon McVittie <smcv@debian.org> Closes: #607
Approved by: cgwalters
Simon McVittie [Mon, 28 Nov 2016 17:27:44 +0000 (17:27 +0000)]
travis-ci: Enable stretch (the future Debian 9), replacing unstable
My goal in building ostree for Debian unstable was that we would
have good coverage of "new code" paths. However, it was removed
for #571 as too much of a moving target. Debian testing is less of
a moving target, and in particular is always internally consistent
(packages are co-installable), which Debian unstable is not guaranteed
to be.
Debian 'stretch' is the future Debian 9, which should be released
next year.
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk> Closes: #600
Approved by: cgwalters
glnx_make_lock_file requires that the dfd passed in survives the
lifetime of the lock. Since dfd_iter.fd gets cleaned up after the
function returns, this isn't the case. dfd_iter.fd should be equivalent
to tmpdir_dfd, since we iter on ".", and that survives past the
function, so just use that instead.
Colin Walters [Fri, 18 Nov 2016 20:07:52 +0000 (15:07 -0500)]
tests: Support TEST_SKIP_CLEANUP=err
I find myself often wanting to debug interactively failing tests.
This makes it more convenient to keep around the temporary directories
just for those tests, rather than accumulating tons of tempdirs from
the successful tests as well.
Colin Walters [Sun, 20 Nov 2016 21:17:22 +0000 (16:17 -0500)]
pull: Do GPG verify commit objects when using deltas
The fact that we weren't doing this is at best an oversight, and
for some deployment models a security vulnerability. Having both
`gpg-verify` and `gpg-verify-summary` shows that we were intending
them to be orthogonal/independent.
Lately I've been advocating moving towards pinned TLS instead of
gpg-signed summaries, and if we follow that path, performing GPG
verification of commit objects even if using deltas is more important,
as it provides an at-rest verifiable authenticity and integrity
mechanism.
Content providers which are signing their summary files and/or using
TLS (particularly pinned TLS) for transport should treat this as a
nice-to-have. However, for providers which are serving content over
plain HTTP and relying on GPG, this is a critical update.
Colin Walters [Thu, 17 Nov 2016 16:40:59 +0000 (11:40 -0500)]
[TSAN] Rework assertions to always access refcount atomically
`-fsanitize=address` complained that the `refcount > 0` assertions
were reading without atomics. We can fix this by reworking them
to read the previous value.