Unfortunately, disabling it by setting it to "" does not work, git
then ignores filter.annex.smudge/clean, and does not pass files through
git-annex at all.
I don't think there is a way to temporarily disable this git config
from the git command line. Which seems like a bug in git.
So, it may be more expensive than anticipated to enable
filter.annex.process, since git checkout etc will pipe all annexed files
being checked out through it.
Joey Hess [Thu, 4 Nov 2021 19:20:26 +0000 (15:20 -0400)]
disable filter.annex.process in restagePointerFile
This means git will run git-annex smudge --clean once per file that is
restaged, which can be slow. But probably *not* as slow as git feeding
all the content of annexed files you've gotten through a pipe to
git-annex filter-process.
The only time this is probably not ideal is after a drop of a bunch of
files, when filter-process would be faster.
Joey Hess [Thu, 4 Nov 2021 19:02:36 +0000 (15:02 -0400)]
add git-annex filter-process
filter-process: New command that can make git add/checkout faster when
there are a lot of unlocked annexed files or non-annexed files, but that
also makes git add of large annexed files slower.
Use it by running: git
config filter.annex.process 'git-annex filter-process'
Fully tested and working, but I have not benchmarked it at all.
And, incremental hashing is not done when git add uses it, so extra work is
done in that case.
Joey Hess [Wed, 3 Nov 2021 19:41:26 +0000 (15:41 -0400)]
git long-running filter process implementation
This module is not used yet, but the plan is to use it for smudge/clean
filtering, at least as an option. In some circumstances, using this
interface may perform better than the interface git-annex is currently
using.
Joey Hess [Mon, 1 Nov 2021 18:40:33 +0000 (14:40 -0400)]
tighter --json parsing for metadata
metadata --batch --json: Reject input whose "fields" does not consist of
arrays of strings. Such invalid input used to be silently ignored.
Used to be that parseJSON for a JSONActionItem ran parseJSON separately
for the itemAdded, and if that failed, did not propagate the error. That
allowed different items with differently named fields to be parsed.
But it was actually only used to parse "fields" for metadata, so that
flexability is not needed.
The fix is just to parse "fields" as-is. AddJSONActionItemFields is needed
only because of the wonky way Command.MetaData adds onto the started json
object.
Note that this line got a dummy type signature added,
just because the type checker needs it to be some type.
itemFields = Nothing :: Maybe Bool
Since it's Nothing, it doesn't really matter what type it is,
and the value gets turned into json and is then thrown away.
Joey Hess [Mon, 1 Nov 2021 17:38:14 +0000 (13:38 -0400)]
metadata --batch: Avoid crashing when a non-annexed file is input
Turns out that CommandStart actions do not have their exceptions caught,
which is why the giveup was causing a crash. Mostly these actions
do not do very much work on their own, but it does seem possible there
are other commands whose CommandStart also throws an exception.
So, my first attempt at a fix was to catch those exceptions. But,
--json-error-messages then causes a difficulty, because in order to output
a json error message, an action needs to have been started; that sets up
the json object that the error message will be included in a field of.
While it would be possible to output an object with just an error field,
this would be json output of a format that the user has no reason to
expect, that happens only in an exceptional circumstance. That is something
I have always wanted to avoid with the json output; while git-annex man
pages don't document what the json looks like, the output has always
been made to be self-describing. Eg, it includes "error-messages":[]
even when there's no errors.
With that ruled out, it doesn't seem a good idea to catch CommandStart
exceptions and display the error to stderr when --json-error-messages
is set. And so I don't know if it makes sense to catch exceptions from that
at all. Maybe I'd have a different opinion if --json-error-messages did not
exist though.
So instead, output a blank line like other batch commands do.
This also leaves open the possibility of implementing support for matching
object with metadata --json, which would also want to output a blank line
when the input didn't match.
Joey Hess [Wed, 27 Oct 2021 18:46:21 +0000 (14:46 -0400)]
move transfer already in progress message to warning
This makes it be displayed in the error-messages field with
--json-error-messages. And with --quiet, it will let it be displayed,
which makes sense because it's telling the user why what they requested
to do has failed to happen.
Joey Hess [Wed, 27 Oct 2021 18:11:34 +0000 (14:11 -0400)]
avoid redundant freezeContent call
This opens the potential for the object file to be in place but
git-annex is interrupted before it can freeze it. git-annex fsck already
fixes that situation, which can also occur when lockContentForRemoval
thaws content.
Reiko Asakura [Tue, 26 Oct 2021 19:46:38 +0000 (15:46 -0400)]
Call freezeContent after move into annex
This change better supports Windows ACL management using
annex.freezecontent-command and annex.thawcontent-command and matches
the behaviour of adding an unlocked file.
By calling freezeContent after the file has moved into the annex,
the file's delete permission can be denied. If the file's delete
permission is denied before moving into the annex, the file cannot
be moved or deleted. If the file's delete permission is not denied after
moving into the annex, it will likely inherit a grant for the delete
permission which allows it to be deleted irrespective of the permissions
of the parent directory.
Joey Hess [Tue, 26 Oct 2021 17:43:50 +0000 (13:43 -0400)]
when private journal file exists, still read from git-annex branch
Fix bug that caused stale git-annex branch information to read when
annex.private or remote.name.annex-private is set.
The private journal file should not prevent reading more current
information from the git-annex branch, but used to.
Note that, overBranchFileContents has to do additional work now, when
there's a private journal file, it reads from the branch redundantly
and more slowly.
Joey Hess [Wed, 20 Oct 2021 16:32:46 +0000 (12:32 -0400)]
avoid using a second db connection for writes
This is a potentially breaking change in a very delicate area. However,
examining the code path for writes, I don't see any benefit to opening a
second db connection for them. If the write throws an exception,
commitDb will retry it with a new db connection.
A potential benefit to not opening a second db connection, beyond using
less resources, is it just might avoid problems in WSL with sqlite that
I have hypothesized are caused by multiple db connections.
Commit 5f9eff3f32e0101ee673c3dde221f0f27f32e3c8 explains why it needs to
shut down the db connection to force the database to be updated on disk:
When closeDb does not get called, garbage collection of DbHandle may not
give the workterThread time to cleanly shut down before git-annex exits,
resulting in a recently written change not reaching disk.
Databases that were SingleWriter still have lock files that are used to
prevent multiple writers.
This does make writing to such databases a bit more expensive,
because the MultiWriter code path that is now used opens a second db
connection in order to write to them.
projects / git-annex.git / log