Joey Hess [Sat, 11 Feb 2023 18:17:13 +0000 (14:17 -0400)]
add warning about Go's past and future data collection
Based on https://github.com/golang/go/discussions/58409, the Go compiler
already defaults to using a google proxy server, which would allow
Google to collect information about what dependencies users are
installing. (Of course they claim they won't.) Two separate environment
settings are needed to turn that off, and users in that thread were
surprised to learn about one of them.
So this warning is already appropriate to some extent.
Also based on the minimisation of user concerns by the golang developers
on that issue and elsewhere, it seems best to assume that they are not
going to be dissuaded from increasing data collection efforts in the future,
even if the blowback prevents this particular attempt.
So this warning should not be removed unless the Go community somehow
extricates itself from Google's control. Or unless ipfs is rewritten in
another language.
Some distros do have ipfs. Unfortunately, Debian appears to be structurally
incapable of packaging it. (8 years and counting;
https://bugs.debian.org/779893). So lots of users will be stuck
installing it from source or having to trust its official binaries.
Joey Hess [Sat, 11 Feb 2023 19:08:31 +0000 (15:08 -0400)]
add a warning about homebrew's use of analytics by default
It apparently displays a notice on first use, but I was surprised to
learn about this behavior today, and I've used it. Displaying a notice
does not make violating users' privacy acceptable.
Joey Hess [Fri, 10 Feb 2023 19:49:15 +0000 (15:49 -0400)]
fix buggy sync to exporttree remote when annex-tracking-branch is not checked out
sync: Fix a bug that caused files to be removed from an importtree=yes
exporttree=yes special remote when the remote's annex-tracking-branch was
not the currently checked out branch.
Joey Hess [Wed, 8 Feb 2023 17:55:55 +0000 (13:55 -0400)]
avoid sync pushing view branches to remotes, and better view branch names
* sync: Avoid pushing view branches to remotes.
* Changed the name of view branches to include the parent branch.
Existing view branches checked out using an old name will still work.
It does not seem useful for sync to push view branches around, because
the information in a view branch can entirely be derived from other
information in git. And sync doesn't push adjusted branches around either.
The better view branch names make it more in line with adjusted branch
names, but were also needed to make fromViewBranch be able to return the
original branch name.
Kept the old view branch names still working. But, when those branches
exist in a repo, sync will still try to push them as before. Avoiding
that would need more complicated and/or expensive changes to sync.
Joey Hess [Tue, 7 Feb 2023 22:09:10 +0000 (18:09 -0400)]
add FIELD?=GLOB to git-annex view usage
And also to vadd usage.
Also added some other things to the usage that were omitted before to
save space.
Adding even FIELD?=GLOB made the git-annex --help list of commands grow
too wide for an 80 column display. So, removed the description of
parameters from that list of commands.
Joey Hess [Tue, 7 Feb 2023 20:28:46 +0000 (16:28 -0400)]
add directory to views for files that lack specified metadata
* view: New field?=glob and ?tag syntax that includes a directory "_"
in the view for files that do not have the specified metadata set.
* Added annex.viewunsetdirectory git config to change the name of the
"_" directory in a view.
When in a view using the new syntax, old git-annex will fail to parse the
view log. It errors with "Not in a view.", which is not ideal. But that
only affects view commands.
annex.viewunsetdirectory is included in the View for a couple of reasons.
One is to avoid needing to warn the user that it should not be changed when
in a view, since that would confuse git-annex. Another reason is that it
helped with plumbing the value through to some pure functions.
annex.viewunsetdirectory is actually mangled the same as any other view
directory. So if it's configured to something like "N/A", there won't be
multiple levels of directories, which would also confuse git-annex.
Joey Hess [Mon, 6 Feb 2023 18:07:33 +0000 (14:07 -0400)]
S3 region=
S3: Support a region= configuration useful for some non-Amazon S3
implementations. This feature needs git-annex to be built with aws-0.24.
datacenter= sets both the AWS hostname and region in one setting, which is
easy when using AWS, but not useful for other hosts. So kept datacenter
as-is, but added this additional config.
Joey Hess [Tue, 24 Jan 2023 18:03:26 +0000 (14:03 -0400)]
use transfer stages for copy --from
See commit e04a9314392037e7b54a098f9709fc6040327411 for an explanation
of why move uses transfer stages for --from, but command stages for
--to. At the point of that commit, copy was actually already using
command stages for everything, so the commit was incorrect about
improving copy --to.
But, the same reasoning about --from applies to copy as to move; when
verification is not done incrementally, download and verification are
the main two stages. The cleanup stage for copy is even less work than
for move (it doesn't drop from the remote).
Joey Hess [Tue, 24 Jan 2023 17:45:01 +0000 (13:45 -0400)]
improve concurrency of move/copy --from --to
Use separate stages for download and upload. In the common case where
it downloads the file from one remote and then uploads to the other,
those are by far the most expensive operations, and there's a decent
chance the two remotes bottleneck on different resources.
Suppose it's being run with -J2 and a bunch of 10 mb files. Two threads
will be started both downloading from the src remote. They will probably
finish at the same time. Then two threads will be started uploading to
the dst remote. They will probably take the same time as well. Before
this change, it would alternate back and forth, bottlenecking on src and dst.
With this change, as soon as the two threads start uploading to dst, two
more threads are able to start, downloading from src. So bandwidth to
both remotes is saturated more often.
Other commands that use transferStages only send in one direction at a
time. So the worker threads for the other direction will sit idle, and
there will be no change in their behavior.
Joey Hess [Mon, 23 Jan 2023 21:07:21 +0000 (17:07 -0400)]
finishing up move --from --to
Lock the local content for drop after getting it from src, to prevent another
process from using the local content as a copy and dropping it from src,
which would prevent dropping the local content after sending it to dest.
Support resuming an interrupted move that downloaded the content from
src, leaving the local content populated. In this case, the location log
has not been updated to say the content is present locally, so we can
assume that it's resuming and go ahead and drop the local content after
sending it to dest.
Note that if a `git-annex get` is being ran at the same time as a
`git-annex move --from --to`, it may get a file just before the move
processes it. So the location log has not been updated yet, and the move
thinks it's resuming. Resulting in local copy being dropped after it's
sent to the dest. This race is something we'll just have to live with,
it seems.
I also gave up on the idea of checking if the location log had been updated
by a `git-annex get` that is ran at the same time. That wouldn't work, because
the location log is precached in the seek stage, so reading it again after
sending the content to dest would not notice changes made to it, unless the cache
were invalidated, which would slow it down a lot. That idea anyway was subject
to races where it would not detect the concurrent `git-annex get`.
So concurrent `git-annex get` will have results that may be surprising.
To make that less surprising, updated the documentation of this feature to
be explicit that it downloads content to the local repository
temporarily.
Joey Hess [Mon, 23 Jan 2023 19:36:06 +0000 (15:36 -0400)]
fully working move --from --to (not release quality)
When the destination already has a copy, it behaves the same as
drop --from really, but display it as a move and implement it
reusing the factored out code from fromPerform.
(Note that willDropMakeItWorse never returns DropAllowed in that
situation, because it's told that dest has a copy. So numcopies is
always checked.)
And when only the source and not the local repo or destination have a
copy, do the full copy from source to local, then copy from local to
dest, then drop from local, then drop from source dance.
This is complicated by fromPerform being hardcoded to assume there is a
local copy, but the local copy has already been dropped. That's why
it uses cleanupfromsrc RemoveNever to avoid the code that makes that
assumption, and finishes with a call to dropfromsrc.
And, since the location log has not yet been updated, checking numcopies
was not working, until I added UnVerifiedRemote dest to the list of
things to check.
This is not yet quite mergeable though. There are two things in the
comment above fromToPerform that are not implemented yet: Checking the
location log before dropping the local copy, and locking the temporary
local copy for drop.
Joey Hess [Mon, 23 Jan 2023 17:45:26 +0000 (13:45 -0400)]
push location log updating up to callers of download
Prep for move --to --from, which needs to download from a src repo
without updating the location log for the local repo, before sending the
content on to the dest repo.
Note that caller of download' already update the log themselves.
See previous commit a422a056f2785999220cde55e07b0fbf999edd44
that pushed it up to download from getViaTmpFrom.
(Also removed in passing a debug print + readline that I accidentially
committed last week on this branch.)
projects / git-annex.git / log