Colin Walters [Sun, 24 Jul 2016 19:00:02 +0000 (15:00 -0400)]
pull-local: Explicitly unref variant rather than relying on floating
This shows up as a leak in valgrind; the callee isn't sinking. In
general through the power of cleanup attributes we can do explicit
cleanup rather than relying on floating refs.
Colin Walters [Sun, 24 Jul 2016 18:54:07 +0000 (14:54 -0400)]
trivial-httpd: Fix leak of option context
Right now our valgrind runs also end up valgrinding the
`trivial-httpd` code, so while it doesn't matter, let's fix this leak
anyways. We need to avoid calling `_exit()` since that won't run the
cleanup functions.
Colin Walters [Fri, 22 Jul 2016 20:58:26 +0000 (16:58 -0400)]
pull: Don't execute static deltas when mirroring
We don't presently support this, since the static delta code assumes
it can just `mmap()` file objects. We could at some point implement
this, but for now just skip executing deltas when doing
`archive -> archive` mirroring.
I noticed this when trying to mirror a repo in Jenkins in
[CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel).
Dan Nicholson [Thu, 14 Jul 2016 16:09:12 +0000 (09:09 -0700)]
core: Add allocating b64 checksum functions
The checksum_b64_inplace variants can't be used in bindings. Provide
versions that allocate and return the output rather than working on a
passed in buffer. These can then be used in GI bindings to get the
ostree modified base64 encodings.
Dan Nicholson [Thu, 14 Jul 2016 19:45:39 +0000 (12:45 -0700)]
lib: Fix version script node ordering
The order of the VERSION nodes is such that the new node name goes first
and the one it depends on goes after the commands. See
https://sourceware.org/binutils/docs/ld/VERSION.html.
Makefile-tests.am: make check uses the built binaries
The tests suite was failing locally as it was using the installed
version of rofiles-fuse, instead of the built one. Create the needed
symlinks in tests/ as we are already doing for the "ostree" binary.
ostree-prepare-root and ostree-remount added for completeness.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #395
Approved by: cgwalters
Colin Walters [Wed, 13 Jul 2016 15:46:16 +0000 (11:46 -0400)]
builtins/commit: Switch to using ostree_repo_write_dfd_to_mtree()
Where we're referencing local files. It's what
ostree_repo_write_directory_to_mtree() is doing internally anyways,
and I consider the `GFile*` based APIs to be generally deprecated for
the most part.
Colin Walters [Tue, 12 Jul 2016 13:43:32 +0000 (09:43 -0400)]
repo: Ensure we set mode for bare-user files before xattrs
When trying to switch ostree to `O_TMPFILE`, I hit the fact that
by default it uses mode `000`. It still works to write to the
open fd of course, but it *doesn't* work to set xattrs because
that code path for some reason in the kernel checks the mode bits.
This only broke for bare-user repos where we tried to set the xattr
before calling `fchmod()`, so just invert those two operations.
Colin Walters [Fri, 8 Jul 2016 14:54:11 +0000 (10:54 -0400)]
tests: Fix karg tests on ostree-booted system
https://github.com/ostreedev/ostree/pull/372 caused these tests to
start failing when the host system is managed using ostree - since the
tests *do* replace the `ostree=` kernel argument.
Colin Walters [Fri, 8 Jul 2016 14:15:47 +0000 (10:15 -0400)]
fetcher: Clear all data for session in session thread
Conceptually the session thread owns the session, so let's clear out
everything predictably there, rather than sometimes having it happen
on the main thread.
Also, this moves up clearing the pending/outstanding queues *before*
we unreference the session, since conceptually they need to reference
it as well.
Based on a patch from: Matthew Barnes <mbarnes@redhat.com>
Colin Walters [Fri, 8 Jul 2016 14:01:47 +0000 (10:01 -0400)]
fetcher: Hold a ref to main context for lifetime of thread
I don't think this fixes the bug I was seeing, but it makes me more
comfortable to know we have a strong ref to the main context across
the thread lifetime, and we only unset the default right before
we go away.
If something in `thread_closure_unref()` used
`g_main_context_get_thread_default()` for example it'd be wrong
before.
Colin Walters [Thu, 7 Jul 2016 21:39:07 +0000 (17:39 -0400)]
tests: Add some test coverage of repeated pulls w/HTTP 500s
Systems like pulp may want to keep retrying in a loop if the server
throws a (hopefully transient) 500, and we need test coverage of
handling these errors versus our existing 404 and 206 coverage.
G_DEFINE_AUTOPTR_CLEANUP_FUNC is a new function in GLib 2.44, but
libglnx contains a backported version of it. A few source files were
however using G_DEFINE_AUTOPTR_CLEANUP_FUNC either without including
libglnx.h, or without including it early enough.
Colin Walters [Fri, 1 Jul 2016 18:39:49 +0000 (14:39 -0400)]
delta: Add --if-not-exists option
I often want to have "idempotent" systems that iterate to a known
state. If after generating a commit, the system is interrupted, I'd
like the next run to still generate a delta. But we don't want to
regenerate if one exists, hence this option.
Dan Nicholson [Mon, 27 Jun 2016 20:06:23 +0000 (13:06 -0700)]
build: Override systemd unit directory for distcheck
distcheck tests that all the files are installed under $prefix. That
doesn't work with the systemd unit directory since the path comes from
pkg-config. Override the setting to be under $prefix in that case.
Dan Nicholson [Fri, 13 May 2016 19:53:01 +0000 (12:53 -0700)]
tests: Improve check for /proc/cmdline kargs
On some systems there may be no root= argument, so the tests for
appending /proc/cmdline arguments will fail. Instead, loop over each of
the arguments in the host's /proc/cmdline and test that they're in the
constructed config file. That will actually test if ostree added all of
the system's /proc/cmdline args correctly. The regex isn't perfect here,
but it's probably good enough for this test.
Dan Nicholson [Fri, 13 May 2016 17:22:02 +0000 (10:22 -0700)]
tests: Ensure mutable deployments from libostreetest
When creating sysroots with libostreetest, we don't get the benefit of
the OSTREE_SYSROOT_DEBUG setting in libtest.sh. That means we'll get
immutable deployments that can't be easily cleaned up.
Ensure the environment variable is set before creating new sysroots. It
would be nice to set the debug flags directly, but that's private API
that isn't currently pulled into libostreetest.
Colin Walters [Sun, 26 Jun 2016 14:25:03 +0000 (10:25 -0400)]
checkout: Add an option to require hardlinks
I've seen a few people hit this and wonder why checkouts are slow/take
space. Really, ensuring this happens is the *point* of OSTree.
Physical copies should be a last resort fallback for very unusual
situations (one of those is rpm-ostree checking out the db since
librpm doesn't know how to read from libostree).
Even I hit the fact that `/var` is a mountpoint disallowing hardlinks
with `/ostree` once and was confused. =)
Add this to the rofiles-fuse test case because it creates a mount
point.
Simon McVittie [Sun, 26 Jun 2016 12:56:05 +0000 (13:56 +0100)]
tests: use our own generated libtool, not the one in $PATH
libtoolize creates a version of libtool for the right architecture
in $(top_builddir), which is guaranteed to be present, and is
guaranteed to match what we are compiling (even during
cross-compilation).
Packaging systems sometimes separate /usr/bin/libtool, which is
specific to one architecture, from the libtool development files
such as libtoolize and ltmain.sh, which are architecture-independent.
For example, in Debian, libtool_*_all.deb contains the files necessary
to libtoolize a package and is depended on by the dh-autoreconf package,
but libtool-bin_*_amd64.deb (or whatever architecture) contains
/usr/bin/libtool and is not normally necessary to depend on.
Signed-off-by: Simon McVittie <smcv@debian.org> Closes: #367
Approved by: cgwalters
Add the functionality to use the same name for refs in local and remote
repos. This helps users keep track of local refs of remote origin, much
like local and remote git branches.
Previously, when a local ref is specified, resolve_refspec would fall
back to searching through remote repos if the ref is not found locally.
This function now takes an extra flag to specify whether it should
search through remote repos. Additionally, ostree_repo_resove_rev_ext
was added to call resolve_refspec with fallback_remote being false, so
refs --create would no longer complain when trying to create a local
ref of the same name as a remote one.
Fix remote repo parsing not being handled correctly on refs --create.
pull: Correctly handle repo->parent_repo when applying static deltas
In flatpak i was using a parent repo, and it failed to update
with ENOENT when dispatching an set-read-source opcode, because the
object it referenced was in the parent repo.
This fixes that by making _ostree_repo_read_bare_fd look
at parent_repo.
We noticed that once a ref folder is created, there is no existing
command that can remove it. For example, once "foo/bar" is created,
even if the user deletes foo or all the refs under foo, the folder
will persist.
Now when the user attempts to create a ref "foo" either through commit
or refs --create, if a folder "foo" exists but is empty of refs, the
folder is removed and the new ref "foo" is created.
New unit tests in tests-ref.sh verify this functionality.
Colin Walters [Tue, 14 Jun 2016 17:12:21 +0000 (13:12 -0400)]
lib: Use sd_journal directly (optionally)
This was the last caller of libgsystem that isn't
`gs_file_get_path_cached()`. I think the use case ostree has where
the same code can be called via command line and via a shared library
*and* via a daemon is rather unusual, so let's just copy the code for
logging from libgsystem into here.
For example rpm-ostree hard depends on a daemon mode, so it'll just
use `sd_journal` directly.
Colin Walters [Fri, 17 Jun 2016 02:16:27 +0000 (22:16 -0400)]
pull: Ensure we always process queue only from main thread
I was easily reproducing a hang on pulls with thousands of requests on
current git master. The initial symptom seemed to be that there are
multiple code paths where we don't invoke
`session_thread_process_pending_queue()`. We really need to do
that any time we remove something from the outstanding queue,
to ensure it gets filled again.
A further issue is that we were tying the lifecycle of the pending
object to the `GTask`, but the task could be unref'd from the main
thread (via a `GSource` on the main thread), and that introduced
threadsafety issues, because the hash table and other data suddenly
could be concurrently modified.
Both of these need to be fixed together. First, we introduce
`Arc<Pending>`, and ensure that both the main and worker threads hold
references.
Second, we ensure that we re-process the queue *immediately* whenever
a task is done, inside the worker thread, rather than doing it
incidentally via an unref. This architecture is quite similar to what
the outside pull code is doing.
Colin Walters [Fri, 17 Jun 2016 13:43:51 +0000 (09:43 -0400)]
tests: Support OT_SKIP_READDIR_RAND
For some reason I'm really not inclined to debug right now, `libtool`
is eating the `-fsanitize=address` argument when linking
`libreaddir-rand.so`, which causes failures since we're trying to
`LD_PRELOAD`.
Colin Walters [Fri, 17 Jun 2016 01:51:13 +0000 (21:51 -0400)]
repo: Avoid a possible divide by zero in progress
The previous code was subject to a divide by zero if less than a
second had passed. Rework it so we only do the divide if more than a
second has passed.
libglnx porting: delete temp files on failure of file creation
We noticed the temp files being left over in ostree when (mistakenly)
trying to create refs with names in use by folders. This fix removes
temp files created by glnx_file_replace_contents_at on failure.
Colin Walters [Thu, 16 Jun 2016 14:07:26 +0000 (10:07 -0400)]
Add "archive" as an alias for "archive-z2"
I find the "-z2" is really a long ago relic of the past when I changed
the format. We no longer have anything to do with the original
`archive`, so let's start allowing people to type `--mode=archive`
which just looks saner.
At some point later I'll update the docs too, but it'll be an annoying
transition period as we'll have to say "On older OSTree, use -z2" etc.
Added the ability to create a ref (much like a git tag) for an
existing commit through "ostree refs EXISTING --create=NEWREF".
Previously the only way to create a new ref was by creating a new commit,
but refs --create allows multiple refs to point to the same commit.
The command will fail if:
- None/more than one existing ref is specified
- The specified EXISTING tag does not exist, or was not specified
- The specified NEWREF already exists, or is the name of a folder
Add unit tests in tests-ref.sh to verify above functionality
Colin Walters [Thu, 9 Jun 2016 18:30:14 +0000 (14:30 -0400)]
lib: Port some manual close() cleanups to be glnx_fd_close
Just noticed this while reading some code, we didn't have many manual
`out: close()` bits left, this pushes us over the edge to autocleanup
almost everywhere.
Simon McVittie [Sun, 12 Jun 2016 09:51:11 +0000 (10:51 +0100)]
libtest: make gpghome in tmpdir writeable
Otherwise, during "make distcheck" it will be read-only (because it
is a copy of the read-only ${srcdir} with permissions preserved), and
deletion will fail during cleanup.
Signed-off-by: Simon McVittie <smcv@debian.org> Closes: #337
Approved by: cgwalters
Simon McVittie [Sun, 12 Jun 2016 09:31:29 +0000 (10:31 +0100)]
autogen.sh: rely on VPATH for submodules' Makefile-*.am.inc
This avoids hard-coding the autogen-time ${srcdir} into our build
system, and might fix "make distcheck".
We don't need to specify ${srcdir} explicitly in the various variables,
because Automake always uses make's VPATH feature to look in both
${builddir} and ${srcdir} at build time.
Signed-off-by: Simon McVittie <smcv@debian.org> Closes: #337
Approved by: cgwalters
Colin Walters [Wed, 11 May 2016 19:54:48 +0000 (15:54 -0400)]
tests: Modernize valgrind infrastructure
The recent memleak fixes motivated me to look at the bitrotted code to
run invocations of `ostree` in the test suite underneath valgrind.
There are a few things here. First, update suppressions file from
libhif, since I recently worked on it.
When running *uninstalled* as we now support, we need
`libtool --mode=execute` in the mix so it expands out to
the uninstalled binary and we don't valgrind the intermediate shell.
However, it's harder than that because we chdir into a tmpdir,
which defeats the libtool logic. AFAICS, the only fix for this
is to determine the realbin path before we chdir, and then unfortunately
we need to change every use of `ostree` to `${OSTREE}` =(
Then this immediately breaks for me on RHEL7 because my ancient
copy of `valgrind-3.10.0-16.el7.x86_64` is unaware of syscall 306, i.e.
`syncfs`.
Mathnerd314 [Sat, 4 Jun 2016 22:16:05 +0000 (16:16 -0600)]
docs: Get API docs working again
This changes around a few things that didn't work for me:
* Section names seem to be ostree-* instead of libostree-*
* Also XML files are ostree-* (they didn't show up at all)
- gtk-doc doesn't seem to parse const _OSTREE_PUBLIC correctly
* pull documentation is now on the actual functions rather than stubs
* Update gitignore with some more files
And there some changes to make gtk-doc give fewer warnings (not finished)
Colin Walters [Fri, 3 Jun 2016 15:00:10 +0000 (11:00 -0400)]
lib: Add a helper for mmap->bytes with openat(), use it in repo
This kills another GSystem consumer...I think down the line I'd like
to do something like "detect whether file is > 1k if so, mmap,
otherwise just readall()" so we can use this helper in more places.
projects / ostree.git / log