Joey Hess [Thu, 29 Oct 2020 14:33:12 +0000 (10:33 -0400)]
more RawFilePath conversion
nukeFile replaced with removeWhenExistsWith removeLink, which allows
using RawFilePath. Utility.Directory cannot use RawFilePath since setup
does not depend on posix.
This commit was sponsored by Graham Spencer on Patreon.
Joey Hess [Wed, 28 Oct 2020 18:18:09 +0000 (14:18 -0400)]
convert to RawByteString
This will break a lot of stuff that uses it, but once fixed should lead
to better performance.
Mostly mechanical.
Changes of note:
* upFrom now uses isPathSeparator, which is better on Windows where
there is not just one
* splitShortExtensions used to take the length of a string,
which would count wide unicode characters as a single character.
Changing to B.length changes that. Note that, git-annex's
annexMaxExtensionLength already changed to the length in bytes
before this change. This function is only used in generating views,
and the small behavior change should not be a problem.
* relHome still uses FilePath because it didn't seem worth changing(?)
This commit was sponsored by Jack Hill on Patreon.
Joey Hess [Tue, 27 Oct 2020 17:39:58 +0000 (13:39 -0400)]
test if magic library works
(When built with it.)
I saw John Thorvald Wodder II doing this in
https://github.com/datalad/datalad-extensions/blob/master/.github/workflows/build-git-annex-windows.yaml
and it seemed like a good idea.
John used an empty file for the binary file, but I think it's
slightly suprising file picks that mime type for it, so opted to instead
use something less ambiguous.
Joey Hess [Mon, 26 Oct 2020 19:37:55 +0000 (15:37 -0400)]
view: Avoid using ':' from metadata when generating a view
Because it's a special character on Windows ("c:").
Use same technique already used for '/' and '\'.
I didn't record how I generated their encoded forms before, so am sure
there was a better way, but the way I did it now is to look at
ghci> encodeFilePath "∕"
"\226\136\149"
And then the difference from that to "\56546\56456\56469"
is adding 56320 to each, to get up to the escaped code plane.
See comment for why I think handling ':' is ok, but that other illegal
windows filenames won't. Note that, this should be enough to make the
test suite always work. Other windows illegal filenames will fail at
checkout time when it tries to put the illegal filename on the
filesystem.
Joey Hess [Mon, 26 Oct 2020 17:24:37 +0000 (13:24 -0400)]
include libmagic in windows installer
This is from https://raw.githubusercontent.com/datalad/datalad-extensions/master/resources/git-annex-magicBundle.patch
which was written by John Thorvald Wodder II.
I have converted line endings from windows, and fixed indentation to use
tabs.
Joey Hess [Mon, 26 Oct 2020 17:06:50 +0000 (13:06 -0400)]
Windows build changed to one done by the datalad-extensions project using Github actions
This is a cleaner build than on Jenkins because the whole environment setup
is handled by the CI config, at least up to the point of "get a random bag
of Windows bytes".
Also, the Jenkins autobuilder has been intermittently failing for a long
time, not due to any problem with git-annex but just a failure to clean up
directories.
Also, this build runs the test suite, and it is (mostly) passing. Test
suite always failed in the jenkins environment.
Also, this build includes libmagic.
Here is the build workflow used by github actions:
https://github.com/datalad/datalad-extensions/blob/master/.github/workflows/build-git-annex-windows.yaml
The libmagic build has its own workflow:
https://github.com/datalad/file-windows/blob/master/.github/workflows/build.yml
(Also cleaned up some windows build cruft I don't use anymore.)
There is no build-version file to link to. I've opened a todo requesting
one: https://github.com/datalad/datalad-extensions/issues/55
Joey Hess [Fri, 23 Oct 2020 22:49:14 +0000 (18:49 -0400)]
remove dead links to youtube-dl
Presumably most users will get it from their linux distribution, or
whatever. Don't want to need to keep updating links if they're going to
rot like this.
But worst, this commit broke the test suite massively. I don't understand how.
git-annex get was failing. Very weirdly, git-annex find in a fresh
clone of an annex repo, during autoinit, was displaying a side message
-- but side messages are disabled when running find.
Joey Hess [Fri, 23 Oct 2020 18:56:12 +0000 (14:56 -0400)]
deal with .git pointer file in Git.CurrentRepo
This fixes the bug.
Note, it's only done when GIT_DIR is set. When it's not set,
Git.Construct already handled it. This is why it was only noticed with this
git submodule command.
This commit was sponsored by Brett Eisenberg on Patreon.
Joey Hess [Fri, 23 Oct 2020 17:36:56 +0000 (13:36 -0400)]
pass --git-dir, rather than changing cwd
If .git is a gitlink file, setting cwd to it will fail, but --git-dir
will succeed. And this is the only place where it sets cwd when running
git, everywhere else already uses --git-dir.
Note that, git-annex's submodule fixup code usually converts gitlink
files to symlinks, so this wasn't usually problem. Still, worth fixing.
This commit was sponsored by Svenne Krap on Patreon.
Joey Hess [Thu, 22 Oct 2020 17:37:09 +0000 (13:37 -0400)]
update chunk log after speculated chunks are verified to be present
Only done in checkPresentChunks, although retrieveChunks could also do
it. Does not seem necessary though, because git-annex never retrives
content without first checking if it's present AFAICR. And really this will
only be needed when using fsck. Puttting it here, rather than in fsck
avoids breaking an abstraction boundary, and is nice and inexpensive.
Joey Hess [Thu, 22 Oct 2020 16:57:58 +0000 (12:57 -0400)]
speculatively use remote's configured chunk size as a fallback
When a special remote has chunking enabled, but no chunk sizes are
recorded (or the recorded ones are not found), speculatively try chunks
using the configured chunk size.
This makes eg, git-annex fsck --from remote be able to fix up the
location log of a file that the git-annex branch does not indicate is
stored on the remote.
Note that fsck does *not* fix up the chunk log to indicate the chunk
size. So, changing the chunk config of the remote after that will still
prevent accessing the chunks stored on it. Maybe fsck should, but I
wanted to start with this and see if it's needed.
Joey Hess [Wed, 21 Oct 2020 14:31:56 +0000 (10:31 -0400)]
move: Improve resuming a move that was interrupted after the object was transferred
In cases where numcopies checks prevented the resumed move from dropping
the object from the source repository, it now relies on a log of recent
moves to replicate the behavior of the interrupted command.
Performance: Probably noticable impact, since it has to add to the log,
check the log, and remove from the log. Seems worth it to avoid this
annoying edge case. The log functions are pretty well optimised to avoid
unncessary work.
An performance improvement to make later would be to avoid cleanup doing
anything if it's not written to the log file, and has confirmed that the
log file does not contain the log line.
This commit was sponsored by Jake Vosloo on Patreon.
Joey Hess [Mon, 19 Oct 2020 17:13:49 +0000 (13:13 -0400)]
don't try to remove pre-commit-annex and post-update-annex-hooks
Those are not installed by git-annex but by the user, and so removal
will never find the default content, and so if the user did install
them, it would display a misleading message.
Seems better, since the user installed them, to let the user remove them
if they want to.
projects / git-annex.git / log