Joey Hess [Wed, 20 Nov 2024 19:00:17 +0000 (15:00 -0400)]
git-remote-annex: Require git version 2.31 or newer
Since old ones had a buggy git bundle command.
In particular, git 2.30.2 has a git bundle that supports --stdin, but does
not read from it, and so fails to create a bundle.
While not using --stdin would perhaps work, it limits the number of revs
that get included in the bundle to the command line length limit.
But the real kicker is that at the same time --stdin got fixed, a bug also
got fixed that made git bundle skip including refs when they had the same
sha as other refs it included. Which would lead to data loss. So best to
avoid that buggy thing.
Joey Hess [Tue, 19 Nov 2024 19:27:26 +0000 (15:27 -0400)]
reuse http url password for p2phttp url when on same host
When remote.name.annexUrl is an annex+http(s) url, that uses the same
hostname as remote.name.url, which is itself a http(s) url, they are
assumed to share a username and password.
This avoids unnecessary duplicate password prompts.
Joey Hess [Tue, 19 Nov 2024 17:01:12 +0000 (13:01 -0400)]
skip git-remote-annex tests on windows
The NullSoftInstaller does not install git-remote-annex. For that
matter, it does not install git-annex-shell either. I don't know quite
how it would make sense to do so, without hard links.
It could contain 3 copies of the same binary.
Joey Hess [Fri, 15 Nov 2024 19:11:56 +0000 (15:11 -0400)]
avoid storing same filename repeatedly in versioned import from S3
Logically, this should make it need a lot less memory when files have
been changed many times. In my tests, it didn't seem to change memory
use at all. Unsure why, it is working. It's possible the Response is not
getting garbage collected due to pinning. But as far as I can see, all
parts of it that are retained get copied in a way that won't keep the
whole thing pinned in memory.
Joey Hess [Fri, 15 Nov 2024 18:19:02 +0000 (14:19 -0400)]
use 80% less memory when importing from a versioned S3 bucket
Same idea as commit eb714c107ba81d805458c770db8d4f22ae12a077, but even
better, because a lot of the response is DeleteMarker, that can be garbage
collected now.
Joey Hess [Thu, 14 Nov 2024 20:09:27 +0000 (16:09 -0400)]
S3 use last Key when there is no Marker element
Fix infinite loop and memory blowup when importing from an unversioned S3
bucket that is large enough to need pagination.
I don't think there actually ever will be a Marker element, a delimiter is
not set.
Probably this code path was never tested with pagination! Also the aws
library's lack of any docs made it easy to mess up.
Versioned buckets seem to not have the same problem. The API docs for
ListObjectVersions say that NextKeyMarker will always be provided when
paginating.
Joey Hess [Wed, 13 Nov 2024 18:09:24 +0000 (14:09 -0400)]
add: When adding a dotfile as a non-large file, mention that it's a dotfile
This is to reduce user confusion when their annex.largefiles matches it,
or is not set.
Note that, when annex.dotfiles is set, but a dotfile is not matched by
annex.largefiles, the "non-large file" message will be displayed. That
makes sense because whether the file is a dotfile does not matter with that
configuration.
Also, this slightly optimised the annex.dotfiles path in passing,
by avoiding the slight slowdown caused by the check added in commit 876d5b6c6fed7d540767783eb3d14469f41bd1c2 in that case.
Joey Hess [Mon, 11 Nov 2024 16:42:26 +0000 (12:42 -0400)]
git-remote-annex: Fix a reversion
Introduced in version 10.20241031 that broke cloning from a special remote
retrieveKeyFile changed to use createAnnexDirectory, which means that the
path passed to it needs to be under .git
git-remote-annex is probably the only thing in git-annex where that was not
the case. And there's no real reason it cannot be the case with it either.
Just use withOtherTmp.
Joey Hess [Wed, 30 Oct 2024 17:51:58 +0000 (13:51 -0400)]
DATA-PRESENT working for exporttree=yes remotes
Since the annex-tracking-branch is pushed first, git-annex has already
updated the export database when the DATA-PRESENT arrives. Which means
that just using checkPresent is enough to verify that there is some file
on the special remote in the export location for the key.
So, the simplest possible implementation of this happened to work!
(I also tested it with chunked specialremotes, which also works, as long
as the chunk size used is the same as the configured chunk size. In that
case, the lack of a chunk log is not a problem. Doubtful this will ever
make sense to use with a chunked special remote though, that gets pretty
deep into re-implementing git-annex.)
Updated the client side upload tip with a missing step, and reorged for clarity.
projects / git-annex.git / log