Joey Hess [Fri, 27 Dec 2019 18:58:10 +0000 (14:58 -0400)]
fix a case where file tracked by git unexpectedly becomes annex pointer file
smudge: When annex.largefiles=anything, files that were already stored in
git, and have not been modified could sometimes be converted to being
stored in the annex. Changes in 7.20191024 made this more of a problem.
This case is now detected and prevented.
Joey Hess [Fri, 20 Dec 2019 22:01:29 +0000 (18:01 -0400)]
Improve file ordering behavior when one parameter is "." and other parameters are other directories
eg, `git-annex get . ..` used to order the files strangly, because it
did not realize that when git ls-files output eg "foo", that should be
grouped with the first set of files and not the second set.
Fixed by making dirContains "." "./foo" = True
which makes sense, because dirContains ".." "../foo" = True
Joey Hess [Fri, 20 Dec 2019 19:01:34 +0000 (15:01 -0400)]
annex.addunlocked expressions
* annex.addunlocked can be set to an expression with the same format used by
annex.largefiles, in case you want to default to unlocking some files but
not others.
* annex.addunlocked can be configured by git-annex config.
Added a git-annex-matching-expression man page, broken out from
tips/largefiles.
A tricky consequence of this is that git-annex add --relaxed
honors annex.addunlocked, but an expression might want to know the size
or content of an url, which it's not going to download. I decided it was
better not to fail, and just dummy up some plausible data in that case.
Performance impact should be negligible. The global config is already
loaded for annex.largefiles. The expression only has to be parsed once,
and in the simple true/false case, it should not do any additional work
matching it.
Joey Hess [Fri, 20 Dec 2019 17:17:28 +0000 (13:17 -0400)]
git-annex-config --set/--unset: No longer change the local git config setting
e53070c1f quietly made it set the local git config too, but that was never
documented anywhere, and it had surprising results. If I set
annex.largefiles globally in a repo, I would expect to be able to change it
in another repo, and the original repo would get the change and use it,
rather than being stuck on the old value set there.
And, if I have a local annex.largefiles and set a different global default,
I'd be surprised to have my local setting overwritten.
annex.securehashesonly does need to be set locally, since it's a security
feature and the global is only a default until it gets set locally. So
special cased.
Joey Hess [Fri, 20 Dec 2019 16:12:31 +0000 (12:12 -0400)]
git-annex config annex.largefiles
annex.largefiles can be configured by git-annex config, to more easily set
a default that will also be used by clones, without needing to shoehorn the
expression into the gitattributes file. The git config and gitattributes
override that.
Whenever something is added to git-annex config, we have to consider what
happens if a user puts a purposfully bad value in there. Or, if a new
git-annex adds some new value that an old git-annex can't parse.
In this case, a global annex.largefiles that can't be parsed currently
makes an error be thrown. That might not be ideal, but the gitattribute
behaves the same, and is almost equally repo-global.
Performance notes:
git-annex add and addurl construct a matcher once
and uses it for every file, so the added time penalty for reading the global
config log is minor. If the gitattributes annex.largefiles were deprecated,
git-annex add would get around 2% faster (excluding hashing), because
looking that up for each file is not fast. So this new way of setting
it is progress toward speeding up add.
git-annex smudge does need to load the log every time. As well as checking
the git attribute. Not ideal. Setting annex.gitaddtoannex=false avoids
both overheads.
Joey Hess [Wed, 18 Dec 2019 20:45:03 +0000 (16:45 -0400)]
more RawFilePath
Remove dup definitions and just use the RawFilePath one. </> etc are
enough faster that it's probably faster than building a String directly,
although I have not benchmarked.
Joey Hess [Wed, 18 Dec 2019 15:29:34 +0000 (11:29 -0400)]
fix fileJournal
My ByteString rewrite oversimplified it, resulting in any _ in a journal
file turning into a / in the git-annex branch, which was often the wrong
filename, or sometimes (//) an invalid filename that git
refused to add.
Joey Hess [Wed, 11 Dec 2019 18:12:22 +0000 (14:12 -0400)]
use filepath-bytestring for annex object manipulations
git-annex find is now RawFilePath end to end, no string conversions.
So is git-annex get when it does not need to get anything.
So this is a major milestone on optimisation.
Benchmarks indicate around 30% speedup in both commands.
Probably many other performance improvements. All or nearly all places
where a file is statted use RawFilePath now.
Joey Hess [Mon, 9 Dec 2019 17:49:05 +0000 (13:49 -0400)]
convert TopFilePath to use RawFilePath
Adds a dependency on filepath-bytestring, an as yet unreleased fork of
filepath that operates on RawFilePath.
Git.Repo also changed to use RawFilePath for the path to the repo.
This does eliminate some RawFilePath -> FilePath -> RawFilePath
conversions. And filepath-bytestring's </> is probably faster.
But I don't expect a major performance improvement from this.
This is mostly groundwork for making Annex.Location use RawFilePath,
which will allow for a conversion-free pipleline.
Joey Hess [Fri, 6 Dec 2019 18:44:42 +0000 (14:44 -0400)]
use RawFilePath getFileStatus for speed
Only done on those calls to getFileStatus that had a RawFilePath, not a
FilePath. The others would probably be just as fast if converted to use
it with toRawFilePath, but I'm not 100% sure.
Note that genInodeCache' uses fromRawFilePath, but that value only gets
used on Windows, so on unix the thunk will never be evaluated.
Joey Hess [Fri, 6 Dec 2019 16:44:18 +0000 (12:44 -0400)]
fix Arbitrary AssociatedFile
Empty filenames were already filtered out as not allowed. But before
the change to ByteString, a NUL could appear in an Arbitrary String,
and so Arbitrary AssociatedFile sometimes generated illegal filenames,
as NUL never appears in a filename. The change to ByteString meant the
String was run through toRawFilePath, which assumes a filename never
contains a NUL. That truncated the String at the NUL, which could
result in an AssociatedFile being generated with an empty filename.
The filtering of NUL added here is not really necessary, because
of the truncation, but it makes explicit that NUL is not allowed.
The real fix is that the suchThat now applies to the final
AssociatedFile, so will catch any empty ones however generated.
This raises the more general question of whether toRawFilePath might
truncate other strings that later get used as filenames. I think new
bugs probably won't be introduced by that. Before, a FilePath that got
read from somewhere (eg an attacker) and contained a NUL would perhaps
be printed out by git-annex, including the NUL, or written to disk
inside a file, or what have you. But as soon as that FilePath gets
passed to any IO action that treats it as a filename, it gets truncated
after the NUL. Eg, writeFile "foo\NULbar" "bar" writes to file "foo".
Now toRawFilePath will make the truncation happen earler, but at most
this will affect what gets printed out or is written to disk inside a
file; actually using the RawFilePath as a filename will not change from
using the FilePath as a filename.
Joey Hess [Fri, 6 Dec 2019 16:14:55 +0000 (12:14 -0400)]
add back lost filtering of multibyte chars in prop_encode_decode_roundtrip
I had thought using ByteString would avoid the problem, but the
quickcheck property is still taking Arbitrary String input, so the use
of ByteString internally doesn't matter.
projects / git-annex.git / log