Joey Hess [Wed, 27 Nov 2019 20:54:11 +0000 (16:54 -0400)]
use ByteString for git config
The parser and looking up config keys in the map should both be faster
due to using ByteString.
I had hoped this would speed up startup time, but any improvement to
that was too small to measure. Seems worth keeping though.
Note that the parser breaks up the ByteString, but a config map ends up
pointing to the config as read, which is retained in memory until every
value from it is no longer used. This can change memory usage
patterns marginally, but won't affect git-annex.
Joey Hess [Wed, 27 Nov 2019 18:09:11 +0000 (14:09 -0400)]
update based on profiling
While L.toStrict copies, profiling showed it was only around 0.3% of
git-annex find runtime. Does not seem worth optimising that, which would
probably involve either a major refactoring, or a use of
UnsafeInterleaveIO.
Also, it seems to me that the latter would need to read chunks, and
preappend the leftover part to the next chunk. But a strict ByteString
append itself is a copy, so I'm not convinced that would be faster than
L.toStrict.
Joey Hess [Tue, 26 Nov 2019 19:27:22 +0000 (15:27 -0400)]
wip RawFilePath 2x git-annex find speedup
Finally builds (oh the agoncy of making it build), but still very
unmergable, only Command.Find is included and lots of stuff is badly
hacked to make it compile.
Benchmarking vs master, this git-annex find is significantly faster!
Specifically:
num files old new speedup
48500 4.77 3.73 28%
12500 1.36 1.02 66%
20 0.075 0.074 0% (so startup time is unchanged)
That's without really finishing the optimization. Things still to do:
* Eliminate all the fromRawFilePath, toRawFilePath, encodeBS,
decodeBS conversions.
* Use versions of IO actions like getFileStatus that take a RawFilePath.
* Eliminate some Data.ByteString.Lazy.toStrict, which is a slow copy.
* Use ByteString for parsing git config to speed up startup.
It's likely several of those will speed up git-annex find further.
And other commands will certianly benefit even more.
Joey Hess [Mon, 25 Nov 2019 20:18:19 +0000 (16:18 -0400)]
wip RawFilePath
Goal is to make git-annex faster by using ByteString for all the
worktree traversal. For now, this is focusing on Command.Find,
in order to benchmark how much it helps. (All other commands are
temporarily disabled)
Currently in a very bad unbuildable in-between state.
Read/Show of KeyData is backwards-compatible with Read/Show of Key from before
this change, so Types.Distribution will keep working.
The Eq instance is fixed.
Also, Key has smart constructors, avoiding needing to remember to update
the cached serialization.
Used git-annex benchmark:
find is 7% faster
whereis is 3% faster
get when all files are already present is 5% faster
Generally, the benchmarks are running 0.1 seconds faster per 2000 files,
on a ram disk in my laptop.
(Build-)depend on git >= 2.22 to avoid memory leaked git being bundled or used
Note from Joey:
git-annex still supports git 2.1, but operates in a degraded fashion.
It would be better for backports of the debian package to also
backport a newer git. This dependency is mostly expressing that,
also that any users who might upgrade git-annex should also upgrade
git.
Also worth noting that the i386ancient autobuilder has git 2.1 on it
(best I have been able to manage there), but luckily the epoch is
bumped to 2, so the dependencies will still be satisfied.
That this was not noticed over at least 1 daylight savings time zone
changes makes me wonder if the TSDelta stuff is still needed.
Perhaps the mtime on Windows no longer changes when the time zone is changed?
Joey Hess [Thu, 21 Nov 2019 21:25:20 +0000 (17:25 -0400)]
improve benchmark --databases
* benchmark: Changed --databases to take a parameter specifiying the size
of the database to benchmark.
* benchmark --databases: Display size of the populated database.
* benchmark --databases: Improve the "addAssociatedFile to (new)"
benchmark to really add new values, not overwriting old values.
Joey Hess [Thu, 21 Nov 2019 19:38:06 +0000 (15:38 -0400)]
explict export lists
Eliminated some dead code. In other cases, exported a currently unused
function, since it was a logical part of the API.
Of course this improves the API documentation. It may also sometimes
let ghc optimize code better, since it can know a function is internal
to a module.
364 modules still to go, according to
git grep -E 'module [A-Za-z.]+ where'
Joey Hess [Thu, 21 Nov 2019 18:31:12 +0000 (14:31 -0400)]
avoid running scanUnlockedFiles in bare repo
It's not necessary. And if the bare repo somehow has a pointer
file in it with the same name as a file in HEAD, that file would be
populated, which would be surprising since the file is not really under
git's control.
Joey Hess [Thu, 21 Nov 2019 17:16:42 +0000 (13:16 -0400)]
Fix bug that made bare repos be treated as non-bare when --git-dir was used.
Eg:
git clone url --bare r
git --git-dir r annex init
This resulted in worktree = Just "." and so several things that check
worktree to determine when the repo is bare ran code paths intended for
non-bare. One such code path[1] ran git checkout with --worktree=. which
actually makes it ignore core.bare config, and so the current directory
got populated with a checkout of the master branch in this example. There
was probably also other breakage.
The fix is a bit complicated because whether the repo is bare is not
known until after Git.Config reads the config, but Git.Config handles
setting the RepoLocations's worktree when core.worktree is set. So have
to assume the worktree is the cwd, let core.worktree override that,
and then if the repo turns out to be bare, it's set back to Nothing.
(And then GIT_WORK_TREE can still override all of that.)
[1] switchHEADBack, which runs even when the clone is not from a bare repo.
Joey Hess [Mon, 18 Nov 2019 20:09:09 +0000 (16:09 -0400)]
git-lfs: remember urls, and autoenable remotes using known urls
* git-lfs: The url provided to initremote/enableremote will now be
stored in the git-annex branch, allowing enableremote to be used without
an url. initremote --sameas can be used to add additional urls.
* git-lfs: When there's a git remote with an url that's known to be
used for git-lfs, automatically enable the special remote.
Joey Hess [Fri, 15 Nov 2019 16:01:30 +0000 (12:01 -0400)]
stop displaying rsync progress, and use git-annex's own progress display for local-to-local repo transfers
Reasons to do this include:
1. I've gotten pretty used to git-annex's own progress display, which is
used for all transfers over ssh (except to old git-annex-shell),
and for most special remote transfers. It's getting to seem weird to see
the rsync progress display instead.
2. When -J was used, the rsync output could not be shown, and so there was
no progress display. Now there will be.
Progress will also be displayed now when cp CoW is used. But I'd expect a CoW
copy to typically run so fast that the progress display will barely be
noticable.
Joey Hess [Thu, 14 Nov 2019 23:20:28 +0000 (19:20 -0400)]
fix GIT_EXEC_PATH to point to bundle/git-core
This makes it be the same as in the linux runshell.
It may be that bundle/ used to have all the necessary binaries in it,
but that seems unlikely. I think that the breakage fixed by my previous
commit masked this other breakage, and anything git execs from git-core
(as opposed to being buitin) has never worked in the OSX dmg.
Joey Hess [Thu, 14 Nov 2019 20:15:42 +0000 (16:15 -0400)]
OSX git-annex.app: Fix a regression that broke git-remote-https, git-remote-http, and git-shell
Putting the binaries in bundle/git-core/bin didn't work on OSX,
linker can't find the libraries next to those binaries where it expects to.
So instead put the binaries in the progDir.
Joey Hess [Thu, 14 Nov 2019 15:31:43 +0000 (11:31 -0400)]
Fix a crash (STM deadlock) when -J is used with multiple files that point to the same key
See the comment for a trace of the deadlock.
Added a new StartStage. New worker threads begin in the StartStage.
Once a thread is ready to do work, it moves away from the StartStage,
and no thread will ever transition back to it.
A thread that blocks waiting on another thread that is processing
the same key will block while in the StartStage. That other thread
will never switch back to the StartStage, and so the deadlock is avoided.
projects / git-annex.git / log