Colin Walters [Mon, 7 Dec 2015 15:33:23 +0000 (10:33 -0500)]
build: Delete generated parse-datetime.c file, use AM_V_GEN
Bison is a well known external dependency, so just require it.
Including the generated content in git means it may or may not
be regenerated based randomly on timestamps, etc.
Colin Walters [Fri, 4 Dec 2015 16:14:25 +0000 (11:14 -0500)]
cmdline: Fatally error if the timestamp in a commit is invalid
Previously we were just ignoring this, which hid a bug in
an earlier commit that generated them.
Also change the `commit` program to use both APIs - this
involves extra code, but not too much.
This way, reverting the fix with this on top caused the test suite to
fail. Adding an active test for this would need a custom test program
using the C API, or adding a cmdline flag to the client, neither of
which quite seemed worth it.
Matthew Barnes [Thu, 3 Dec 2015 17:01:38 +0000 (12:01 -0500)]
repo: Fix backwards timestamp in ostree_repo_write_commit()
ostree_repo_write_commit_with_time() converts the timestamp to
big-endian byte order.
ostree_repo_write_commit() was also doing this when calling
ostree_repo_write_commit_with_time(), resulting in a corrupted
commit object (timestamp bytes were backwards).
Matthew Barnes [Mon, 23 Nov 2015 16:04:31 +0000 (11:04 -0500)]
repo: Never delete .commitmeta files
Do not delete a .commitmeta file after removing the last metadata entry.
This way a client will pull the empty .commitmeta file and overwrite old
metadata as expected.
Matthew Barnes [Tue, 17 Nov 2015 00:29:59 +0000 (19:29 -0500)]
repo: Validate checksums have correct length
ostree_checksum_bytes_peek() can return NULL if the checksum has an
incorrect length (most likely from disk corruption) but most callers
are not prepared to handle this and would likely crash.
Use ostree_checksum_bytes_peek_validate() instead, which sets a
GError on an invalid checksum.
deltas: Support including detached metadata in static deltas
This is very useful for the inline-parts case, as you can then include
detached signatures in a single file representing the commit.
It is not as important for the generic pull case, as the detached
metadata is only a single small file. Additionally the detached
metadata is not content referenced and may change after the static
delta file was created, so we need to pull the latest version anyway.
deltas: Support passing filename to ostree_repo_static_delta_execute_offline
If you pass a diriectory it will look for the "superblock" child, otherwise
it will use the file as the superblock. I need this in xdg-app to be able
to install any filename as a bundle.
In this mode the parts are stored in the metadata of the main delta
superblock file. This can be useful if you want a single-file delta
for easy transport, or for http in the case the delta is very small.
repo: create a tombstone commit when deleting a commit
When a commit is deleted and the repo is configured to use tombstone
commits, create one. Delete the tombstone file only if the commit is
pulled again.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Colin Walters [Tue, 13 Oct 2015 01:33:23 +0000 (21:33 -0400)]
sysroot: Write symlinks before calling fsync(), then rename after
There might be a race here in that we create new symlink files *after*
calling `syncfs`, and they are not guaranteed to end up on disk.
Rework the code so that we create symlinks before, and then only
rename them after (and `fsync()` the directory for good measure).
Additional-fixes-by: Giuseppe Scrivano <gscrivan@redhat.com> Tested-by: Giuseppe Scrivano <gscrivan@redhat.com>
This still needs verification that we're fixing a real bug; but I'm
fairly confident this won't make the fsync situation worse.
Daniel Drake [Tue, 21 Jul 2015 14:04:34 +0000 (08:04 -0600)]
prepare-root: set up /boot bind-mount for single partition systems
When booting from a system with /boot on the main partition, set up
an appropriate bind mount during boot. The ostree runtime binary
expects to be able to access the bootloader configs at /boot.
Matthew Barnes [Thu, 1 Oct 2015 23:02:11 +0000 (19:02 -0400)]
upgrader: Allow overriding the commit to pull
Upgrader now looks for an "override-commit" key in the origin file
with a commit checksum, which causes the upgrader to pull and deploy
the specified commit rather than the latest available commit on the
origin refspec.
Gatis Paeglis [Tue, 6 Oct 2015 21:43:25 +0000 (23:43 +0200)]
Don't require /boot/uEnv.txt for u-boot support
The current code checks if /boot/uEnv.txt is a symlink to
decice if sysroot requires u-boot support. Why this is bad:
There are 2 ways to provide a custom env to u-boot from user space:
1) A compiled binary that is sourced from u-boot.
2) A text file (usually /uEnv.txt) that is imported into env from u-boot.
The current OSTree u-boot integration code was designed with the 1st
case in mind.
Many bootscripts provided by an embedded device vendors expect
to find uEnv.txt in the top level directory, it is often hardcoded
when building u-boot and is difficult to change later on. Or in other
cases it is stored in read-only memory so changing it would require
re-flushing boot loader with a new env. So the issue here is that
OSTree's and vendor uEnv.txt want to exist and the same path and OSTree
would throw away any changes added to /uEnv.txt by user on the next
upgrade/deploy.
This patch "hides" away the OSTree's env file loader/uEnv.txt from users
who are used to edditing uEnv.txt at the top level directory. Now to add
OSTree support on such boards you can simply add a custom logic in uEnv.txt
that loads ostree env from /loader/uEnv.txt
This change is backward compatible with the previous ostree releases and
solves the issue described in:
John Hiesey [Wed, 7 Oct 2015 21:43:13 +0000 (14:43 -0700)]
static-delta: Don't run bspatch when output object already exists
There is already a check that the destination object does not
exist in all other cases when processing an incoming static delta.
However, the bspatch case would still try to run and fail. Add
an analogous check to that case as well.
John Hiesey [Wed, 7 Oct 2015 20:30:33 +0000 (13:30 -0700)]
static-delta: Handle LZMA_BUF_ERROR returned by zlib
zlib can return LZMA_BUF_ERROR, which indicates that either
the input or output buffer has size zero. This case should cause
the correct error to be passed back from g_converter_convert
to expand the relevant buffer. Since this error is ambiguous
as to which buffer is too small, an explicit check on the
output buffer size is added as well.
Colin Walters [Thu, 24 Sep 2015 15:00:54 +0000 (11:00 -0400)]
sysroot: Cleanup refs and prune even on last undeployment
I was working on a different test, and ended up being very confused at
the behavior where removing the last deployment didn't remove the last
`ostree/X/X/X` ref pointing to its commit.
There's no reason to special case the last undeployment AFAIK, and the
existing code handles this.
Matthew Barnes [Wed, 23 Sep 2015 23:28:47 +0000 (19:28 -0400)]
fetcher: Track outstanding requests with a table
Track outstanding HTTP requests in a table for easier debugging.
Also fixes a bug discussed in https://bugzilla.gnome.org/755224
where the outstanding request counter was not decremented in the
event of an error, which could result in the fetcher hitting its
max request limit and locking up.
The bug is fixed by removing the request struct from the table in
pending_uri_free(), which is always called regardless of error,
so the outstanding request count is always accurate.
Matthew Barnes [Wed, 23 Sep 2015 21:11:51 +0000 (17:11 -0400)]
fetcher: Rework reference counting
Have OstreeFetcherPendingURI be the GTask's task_data and pass the GTask
around in queues and callback closures. The reference counting before
was a little confusing and this helps clarify it, at least to me.
OstreeFetcherPendingURI no longer needs its own reference count.
Prior to this, the deploy command garbage collected <checksum> since
there's no ref pointing to it, and then ostree_sysroot_deploy_tree()
fails because it can't find the <checksum> commit.
Matthew Barnes [Wed, 9 Sep 2015 18:50:14 +0000 (14:50 -0400)]
sysroot: Add ostree_sysroot_prepare_cleanup()
New public function works like ostree_sysroot_cleanup() EXCEPT FOR
pruning the repository.
Under the hood, add _ostree_sysroot_piecemeal_cleanup() which takes
flags to better control what files are cleaned up. Both public cleanup
functions are now wrappers for _ostree_sysroot_piecemeal_cleanup() with
different flags.
Colin Walters [Tue, 1 Sep 2015 02:37:37 +0000 (22:37 -0400)]
pull: Go back to using one main context
xdg-app was hanging for me with v2015.8, but worked with v2015.7.
I narrowed things down to the GMainLoop/context commit, in which
we started pushing a temporary main context for synchronous
requests internally.
That's never really going to work with libsoup - there needs
to be a single main context which works on the socket. Furthermore,
clients couldn't get progress messages that way.
For *other* internal uses where we added APIs that talk to the remote
repo, we cleanly push a temporary main context.
(Note that I kind of snuck in a change here around the GError handling
in pulls that isn't strictly related but came up in testing)
Colin Walters [Wed, 26 Aug 2015 19:41:39 +0000 (15:41 -0400)]
fetcher: Use 0666 (-umask) for temporary files
There's no reason to keep them hidden. I have a hard policy that
OSTree should *not* be used to carry secrets. Things like host ssh
private keys should be set up out of band by an OS-external
configuration mechanism such as kickstart, cloud-init, etc.
We also assume that hiding binaries is not very useful as most
attackers would be able to find them on the Internet or (for
subscribed content) acting as a customer.
This fixes a bug with mirroring because we changed to take the
unmodified upstream objects rather than uncompress <-> recompress.
John Hiesey [Wed, 26 Aug 2015 19:11:38 +0000 (12:11 -0700)]
static-delta: assert on non-regular files
Now that the computed similar objects are all regular files,
get_unpacked_unlinked_content should never be called on any other
object type. Assert that this is true instead of silently succeeding.
John Hiesey [Wed, 26 Aug 2015 01:15:53 +0000 (18:15 -0700)]
static-delta: Ignore symlinks when computing similar objects
_ostree_delta_compute_similar_objects should not output symlinks.
Previously, a symlink in the "from" commit could be matched to a
real file in the "to" commit, since nothing was filtering symlinks
on the "from" side. This led to failures running the bzdiff
algorithm.
Dan Nicholson [Wed, 26 Aug 2015 16:54:31 +0000 (09:54 -0700)]
tests: Build test-lzma with LZMA flags
test-lzma builds a copy of the compressor and decompressor directly, so
the compiler needs access to the LZMA headers and the linker needs to
link the program with liblzma.
On systems with slow disks, the recursive scanning of directories can
be expensive -- it takes upwards of 2 minutes on our systems. This can
block the main loop for such a long time that it allows the download to
time out...
As such, move all the scanning of objects to a queue, processed from
an idle, to make sure that we don't block the main loop when scanning.
Colin Walters [Wed, 12 Aug 2015 21:03:52 +0000 (17:03 -0400)]
pull: Stop using GMainLoop
First of all, what we were doing with having GMainLoop in the internal
APIs is wrong. Synchronous APIs should always create their own main
context and not iterate the caller's. Doing the latter creates
potential for evil reentrancy issues. Sync API should block, async
API is for not blocking.
Now that's out of the way, fix the pull code to do the clean
```
while (termination_condition (state))
g_main_context_iteration (mainctx, TRUE);
```
model for looping. This is a lot easier to understand and ultimately
more reliable than having other code call `g_main_loop_quit()`, as the
loop condition is in exactly one place.
We can also remove the idle source which only fired once.
Note we have to add a hack here to discard the synchronous session and
create a new one which we only use async.
Colin Walters [Thu, 13 Aug 2015 16:43:14 +0000 (12:43 -0400)]
sysroot: Add an unload() API
This will be used by gnome-continuous at least to drop the reference
to the fd so that unmounting can proceed. See
https://git.gnome.org/browse/gnome-continuous/commit/?id=95e9910ea288d302509ca667e0d190dd89377dd5
projects / ostree.git / log