The problem is that shutdown is not always called, particularly in the test
suite. So, a database connection would be opened, possibly some changes
queued, and then not shut down.
One way this can happen is when using Annex.eval or Annex.run with a new
state. A better fix might be to make both of them call Keys.shutdown
(and be sure to do it even if the annex action threw an error).
Complication: Sometimes they're run reusing an existing state, so shutting
down a database connection could cause problems for other users of that
same state. I think this would need a MVar holding the database handle,
so it could be emptied once shut down, and another user of the database
connection could then start up a new one if it got shut down. But, what if
2 threads were concurrently using the same database handle and one shut it
down while the other was writing to it? Urgh.
Might have to go that route eventually to get the database access to run
fast enough. For now, a quick fix to get the test suite happier, at the
expense of speed.
Joey Hess [Wed, 16 Dec 2015 17:52:43 +0000 (13:52 -0400)]
reorder database shutdown to be concurrency safe
If a DbHandle is in use by another thread, it could be queueing changes
while shutdown is running. So, wait for the worker to finish before
flushing the queue, so that any last-minute writes are included. Before
this fix, they would be silently dropped.
Of course, if the other thread continues to try to use a DbHandle once it's
closed, it will block forever as the worker is no longer reading from the
jobs MVar. So, that would crash with
"thread blocked indefinitely in an MVar operation".
Joey Hess [Wed, 16 Dec 2015 00:20:37 +0000 (20:20 -0400)]
improve temp dir security
http://bugs.debian.org/807341
* Fix insecure temporary permissions when git-annex repair is used in
in a corrupted git repository.
Other calls to withTmpDir didn't leak any potentially private data,
but repair clones the git repository to a temp directory which is made
using the user's umask. Thus, it might expose a git repo that is
otherwise locked down.
* Fix potential denial of service attack when creating temp dirs.
Since withTmpDir used easily predictable temporary directory names,
an attacker could create foo.0, foo.1, etc and as long as it managed to
keep ahead of it, could prevent it from ever returning.
I'd rate this as a low utility DOS attack. Most attackers in a position
to do this could just fill up the disk /tmp is on to prevent anything
from writing temp files. And few parts of git-annex use withTmpDir
anyway, so DOS potential is quite low.
Examined all callers of withTmpDir and satisfied myself that
switching to mkdtmp and so getting a mode 700 temp dir wouldn't break any
of them.
Note that withTmpDirIn continues to not force temp dir to 700.
But it's only used for temp directories inside .git/annex/wherever/
so that is not a problem.
Also re-audited all other uses of temp files and dirs in git-annex.
Joey Hess [Tue, 15 Dec 2015 17:06:52 +0000 (13:06 -0400)]
have clean filter check if the filename was already in use by an old key
The annex object for it may have been modified due to hard link, and
that should be cleaned up when the new version is added. If another
associated file has the old key's content, that's linked into the annex
object. Otherwise, update location log to reflect that content has been
lost.
Joey Hess [Tue, 15 Dec 2015 17:02:33 +0000 (13:02 -0400)]
update inode cache to cover file even when nothing needs to be done to linkAnnex
This covers the case where multiple files have the same content and are
added with git add. Previously only the one that was linked to the annex
got its inode cached; now both are.
Before this fix, that resulted in git saying the file was modified. And
indeed, it didn't have the content it should in the just checked out ref,
because step 3 modified the object file for the old key.
Joey Hess [Fri, 11 Dec 2015 20:12:42 +0000 (16:12 -0400)]
checked getKeysPresent; it's ok for v6 unlocked files
When a v6 unlocked files is removed from the work tree,
unused doesn't show it. When it gets removed from the index,
unused does show it. This is the same as a locked file.
Joey Hess [Thu, 10 Dec 2015 20:12:05 +0000 (16:12 -0400)]
v6 git-annex unlock
Note that the implementation uses replaceFile, so that the actual
replacement of the work tree file is atomic. This seems a good property to
have!
It would be possible for unlock in v6 mode to be run on files that do not
have their content present. However, that would be a behavior change from
before, and I don't see any immediate need to support it, so I didn't
implement it.
Joey Hess [Thu, 10 Dec 2015 18:51:04 +0000 (14:51 -0400)]
check InodeCache in inAnnex et al
This avoids querying the database when the content file doen't exist
(or otherwise fails the provided check). However, it does add overhead of
querying the database, and will certianly impact performance.
Joey Hess [Wed, 9 Dec 2015 21:47:05 +0000 (17:47 -0400)]
use InodeCache when dropping a key to see if a pointer file can be safely reset
The Keys database can hold multiple inode caches for a given key. One for
the annex object, and one for each pointer file, which may not be hard
linked to it.
Inode caches for a key are recorded when its content is added to the annex,
but only if it has known pointer files. This is to avoid the overhead of
maintaining the database when not needed.
When the smudge filter outputs a file's content, the inode cache is not
updated, because git's smudge interface doesn't let us write the file. So,
dropping will fall back to doing an expensive verification then. Ideally,
git's interface would be improved, and then the inode cache could be
updated then too.
Joey Hess [Wed, 9 Dec 2015 21:00:37 +0000 (17:00 -0400)]
add inode cache to the db
Renamed the db to keys, since it is various info about a Keys.
Dropping a key will update its pointer files, as long as their content can
be verified to be unmodified. This falls back to checksum verification, but
I want it to use an InodeCache of the key, for speed. But, I have not made
anything populate that cache yet.
projects / git-annex.git / log