Joey Hess [Wed, 18 Dec 2019 15:29:34 +0000 (11:29 -0400)]
fix fileJournal
My ByteString rewrite oversimplified it, resulting in any _ in a journal
file turning into a / in the git-annex branch, which was often the wrong
filename, or sometimes (//) an invalid filename that git
refused to add.
Joey Hess [Wed, 11 Dec 2019 18:12:22 +0000 (14:12 -0400)]
use filepath-bytestring for annex object manipulations
git-annex find is now RawFilePath end to end, no string conversions.
So is git-annex get when it does not need to get anything.
So this is a major milestone on optimisation.
Benchmarks indicate around 30% speedup in both commands.
Probably many other performance improvements. All or nearly all places
where a file is statted use RawFilePath now.
Joey Hess [Mon, 9 Dec 2019 17:49:05 +0000 (13:49 -0400)]
convert TopFilePath to use RawFilePath
Adds a dependency on filepath-bytestring, an as yet unreleased fork of
filepath that operates on RawFilePath.
Git.Repo also changed to use RawFilePath for the path to the repo.
This does eliminate some RawFilePath -> FilePath -> RawFilePath
conversions. And filepath-bytestring's </> is probably faster.
But I don't expect a major performance improvement from this.
This is mostly groundwork for making Annex.Location use RawFilePath,
which will allow for a conversion-free pipleline.
Joey Hess [Fri, 6 Dec 2019 18:44:42 +0000 (14:44 -0400)]
use RawFilePath getFileStatus for speed
Only done on those calls to getFileStatus that had a RawFilePath, not a
FilePath. The others would probably be just as fast if converted to use
it with toRawFilePath, but I'm not 100% sure.
Note that genInodeCache' uses fromRawFilePath, but that value only gets
used on Windows, so on unix the thunk will never be evaluated.
Joey Hess [Fri, 6 Dec 2019 16:44:18 +0000 (12:44 -0400)]
fix Arbitrary AssociatedFile
Empty filenames were already filtered out as not allowed. But before
the change to ByteString, a NUL could appear in an Arbitrary String,
and so Arbitrary AssociatedFile sometimes generated illegal filenames,
as NUL never appears in a filename. The change to ByteString meant the
String was run through toRawFilePath, which assumes a filename never
contains a NUL. That truncated the String at the NUL, which could
result in an AssociatedFile being generated with an empty filename.
The filtering of NUL added here is not really necessary, because
of the truncation, but it makes explicit that NUL is not allowed.
The real fix is that the suchThat now applies to the final
AssociatedFile, so will catch any empty ones however generated.
This raises the more general question of whether toRawFilePath might
truncate other strings that later get used as filenames. I think new
bugs probably won't be introduced by that. Before, a FilePath that got
read from somewhere (eg an attacker) and contained a NUL would perhaps
be printed out by git-annex, including the NUL, or written to disk
inside a file, or what have you. But as soon as that FilePath gets
passed to any IO action that treats it as a filename, it gets truncated
after the NUL. Eg, writeFile "foo\NULbar" "bar" writes to file "foo".
Now toRawFilePath will make the truncation happen earler, but at most
this will affect what gets printed out or is written to disk inside a
file; actually using the RawFilePath as a filename will not change from
using the FilePath as a filename.
Joey Hess [Fri, 6 Dec 2019 16:14:55 +0000 (12:14 -0400)]
add back lost filtering of multibyte chars in prop_encode_decode_roundtrip
I had thought using ByteString would avoid the problem, but the
quickcheck property is still taking Arbitrary String input, so the use
of ByteString internally doesn't matter.
Joey Hess [Wed, 4 Dec 2019 18:14:35 +0000 (14:14 -0400)]
optimisation
This was already optimised before, but profiling found that delEntry was
around 1.5% of the total runtime of git-annex whereis. It was being
called once per environment variable per file processed.
Fixed by better caching. Since withIndexFile is almost always run with
the same .git/annex/index file, it can cache the modified environment,
rather than re-modifying it each time called.
Joey Hess [Wed, 4 Dec 2019 18:14:35 +0000 (14:14 -0400)]
optimisation
This was already optimised before, but profiling found that delEntry was
around 1.5% of the total runtime of git-annex whereis. It was being
called once per environment variable per file processed.
Fixed by better caching. Since withIndexFile is almost always run with
the same .git/annex/index file, it can cache the modified environment,
rather than re-modifying it each time called.
Joey Hess [Wed, 27 Nov 2019 20:54:11 +0000 (16:54 -0400)]
use ByteString for git config
The parser and looking up config keys in the map should both be faster
due to using ByteString.
I had hoped this would speed up startup time, but any improvement to
that was too small to measure. Seems worth keeping though.
Note that the parser breaks up the ByteString, but a config map ends up
pointing to the config as read, which is retained in memory until every
value from it is no longer used. This can change memory usage
patterns marginally, but won't affect git-annex.
Joey Hess [Wed, 27 Nov 2019 18:09:11 +0000 (14:09 -0400)]
update based on profiling
While L.toStrict copies, profiling showed it was only around 0.3% of
git-annex find runtime. Does not seem worth optimising that, which would
probably involve either a major refactoring, or a use of
UnsafeInterleaveIO.
Also, it seems to me that the latter would need to read chunks, and
preappend the leftover part to the next chunk. But a strict ByteString
append itself is a copy, so I'm not convinced that would be faster than
L.toStrict.
Joey Hess [Tue, 26 Nov 2019 19:27:22 +0000 (15:27 -0400)]
wip RawFilePath 2x git-annex find speedup
Finally builds (oh the agoncy of making it build), but still very
unmergable, only Command.Find is included and lots of stuff is badly
hacked to make it compile.
Benchmarking vs master, this git-annex find is significantly faster!
Specifically:
num files old new speedup
48500 4.77 3.73 28%
12500 1.36 1.02 66%
20 0.075 0.074 0% (so startup time is unchanged)
That's without really finishing the optimization. Things still to do:
* Eliminate all the fromRawFilePath, toRawFilePath, encodeBS,
decodeBS conversions.
* Use versions of IO actions like getFileStatus that take a RawFilePath.
* Eliminate some Data.ByteString.Lazy.toStrict, which is a slow copy.
* Use ByteString for parsing git config to speed up startup.
It's likely several of those will speed up git-annex find further.
And other commands will certianly benefit even more.
Joey Hess [Mon, 25 Nov 2019 20:18:19 +0000 (16:18 -0400)]
wip RawFilePath
Goal is to make git-annex faster by using ByteString for all the
worktree traversal. For now, this is focusing on Command.Find,
in order to benchmark how much it helps. (All other commands are
temporarily disabled)
Currently in a very bad unbuildable in-between state.
projects / git-annex.git / log