Joey Hess [Thu, 23 Jan 2020 16:35:46 +0000 (12:35 -0400)]
Display a warning when concurrency is enabled but ssh connection caching is not enabled or won't work due to a crippled filesystem
A warning message is unsatisfying. But erroring out is too hard a failure,
especially since it may well work fine if the user has enabled passwordless
ssh.
I did think about falling back to one ssh connection at a time in this
case, but it would have needed a rework of every ssh call, which
seems far overboard for such a niche problem. There's no single place where
git-annex runs ssh, so no one place that it could block a concurrent call
on a semaphore. And, even if it did fall back to one ssh connection at a
time, it seems to me that doing so without warning the user about the
problem just invites bug reports like "git-annex is ignoring my -J2 and
only doing one download at a time". So a warning is needed, and I suppose
is good enough.
Joey Hess [Wed, 22 Jan 2020 20:38:34 +0000 (16:38 -0400)]
handle git-credential prompt in -J mode
If git-credential has it cached and does not prompt, this will
unfortunately result in a brief flicker, as the displayed console
regions are hidden while running it and then re-displayed. Better than a
corrupted display.
Actually, I tried it and don't see a visible flicker, so probably only
over a slow ssh will it be apparent.
Joey Hess [Wed, 22 Jan 2020 17:20:06 +0000 (13:20 -0400)]
better error message when git config fails to parse remote config
Rather than leaking the name of the temp file, just say the config parse
failed, and where the config was downloaded from.
Not closing the bug report because two issues were reported in the same
bug report, because the universe wants me to continually re-read old
unclosed bug reports to waste my time determining what still needs to be
done.
Joey Hess [Wed, 22 Jan 2020 15:39:15 +0000 (11:39 -0400)]
avoid ugly error message
Http remotes that do expose a git config file, but are not initialized
resulted in an ugly and unncessary error message, now sqelched.
When git-annex-shell configlist is run w/o the autoinit field, it may
not generate a uuid for the repository. So in that case, it's not
unexpected for the config it does list to not include a UUID, and
dumping out the config in a warning message is not needed.
If configlist is asked to autoinit and we don't get back a config with a
UUID in it, that suggests some problem, and what we got back may not be
a config at all but some diagnostic message, so it does make sense to
output it then.
Joey Hess [Mon, 20 Jan 2020 21:04:45 +0000 (17:04 -0400)]
--whatelse is a better name than --describe-other-params
The use case is basically the user having forgotten, so --help would be
best, but it would be quite hard to include this in --help, since it may
even have to spin up an external special remote program.
I also considered --umm but typoed it the first time I tried it as
--uum, and while memorable, it's too cutesy. --whatelse is good because
it explicitly asks, what other params, besides the ones I've given?
Joey Hess [Fri, 17 Jan 2020 21:23:19 +0000 (17:23 -0400)]
speed hack
Avoids the external program being started just to use LISTCONFIGS on an
already accepted config.
So initremote/enableremote will still run the external program an extra
time to use LISTCONFIGS, but everything that uses the special remote after
it's initialized will not any longer.
Joey Hess [Fri, 17 Jan 2020 19:30:14 +0000 (15:30 -0400)]
add LISTCONFIGS to external special remote protocol
Special remote programs that use GETCONFIG/SETCONFIG are recommended
to implement it.
The description is not yet used, but will be useful later when adding a way
to make initremote list all accepted configs.
configParser now takes a RemoteConfig parameter. Normally, that's not
needed, because configParser returns a parter, it does not parse it
itself. But, it's needed to look at externaltype and work out what
external remote program to run for LISTCONFIGS.
Note that, while externalUUID is changed to a Maybe UUID, checkExportSupported
used to use NoUUID. The code that now checks for Nothing used to behave
in some undefined way if the external program made requests that
triggered it.
Also, note that in externalSetup, once it generates external,
it parses the RemoteConfig strictly. That generates a
ParsedRemoteConfig, which is thrown away. The reason it's ok to throw
that away, is that, if the strict parse succeeded, the result must be
the same as the earlier, lenient parse.
initremote of an external special remote now runs the program three
times. First for LISTCONFIGS, then EXPORTSUPPORTED, and again
LISTCONFIGS+INITREMOTE. It would not be hard to eliminate at least
one of those, and it should be possible to only run the program once.
Joey Hess [Fri, 17 Jan 2020 17:32:48 +0000 (13:32 -0400)]
avoid getting config parser when there is no config to parse
The benefit here is that external special remotes will need a
LISTCONFIGS request and response to generate their config parser,
and this avoids it being done for all the ones that don't have any
configs.
Note that, a config parser could in theory fail to parse if there are no
configs (none currently do), but a parse failure is already thrown away
when generating the remote list because it's too late. Such problems
have to be caught at initremote/enableremote time, not here.
Joey Hess [Wed, 15 Jan 2020 17:01:22 +0000 (13:01 -0400)]
ported Remote.External
Not yet added anything to the protocol to get a list of remote config
fields; any fields will be accepted and are available for the external
remote to use as before.
There is one minor behavior change.. Before, GETCONFIG could be passed a
field such as type, externaltype, encryption, etc, and would get the
value of that. Now, GETCONFIG only works on fields that don't have a
defined meaning to git-annex, so are passed through to the external
remote. This seems unlikely to affect any external special remotes in
practice.
Joey Hess [Wed, 15 Jan 2020 15:22:36 +0000 (11:22 -0400)]
a few forgotten remote config fields
preferreddir can be used with any special remote, so its parser needs to
be included in the commonFieldParsers.
initremote with uuid= changed to delete that field, so it does not
need to be included in commonFieldParsers. Note that, existing remotes
initialized before this change will have the field in remote.log.
This will not cause problems parsing, because the value will be
Accepted.
Grepping for 'Accepted "' found these, and I'm pretty sure this is all of
them.
Joey Hess [Wed, 15 Jan 2020 15:05:07 +0000 (11:05 -0400)]
use more field functions
Using field functions consistently avoids possibility of typos and also
helps ensure that all fields are added to RemoteConfigParsers (as long
as I have remembered to add them when writing the functions).
Joey Hess [Wed, 15 Jan 2020 14:57:45 +0000 (10:57 -0400)]
include credPairRemoteFields in RemoteConfigParsers
Avoids parse error when the fields are added to RemoteConfig at setup
time and it then gets parsed, also at setup time. After setup time, such
internally added fields are not a problem, because they're Accepted. So
it may not be necessary in all cases to list such internally added
fields, but I think it's a good idea to always do so.
Joey Hess [Tue, 14 Jan 2020 17:18:15 +0000 (13:18 -0400)]
convert configParser to Annex action and add passthrough option
Needed so Remote.External can query the external program for its
configs. When the external program does not support the query,
the passthrough option will make all input fields be available.
Joey Hess [Mon, 13 Jan 2020 16:35:39 +0000 (12:35 -0400)]
wip separate RemoteConfig parsing
Remote now contains a ParsedRemoteConfig. The parsing happens when the
Remote is constructed, rather than when individual configs are used.
This is more efficient, and it lets initremote/enableremote
reject configs that have unknown fields or unparsable values.
It also allows for improved type safety, as shown in
Remote.Helper.Encryptable where things that used to match on string
configs now match on data types.
This is a work in progress, it does not build yet.
The main risk in this conversion is forgetting to add a field to
RemoteConfigParser. That will prevent using that field with
initremote/enableremote, and will prevent remotes that already are set
up from seeing that configuration. So will need to check carefully that
every field that getRemoteConfigValue is called on has been added to
RemoteConfigParser.
(One such case I need to remember is that credPairRemoteField needs to be
included in the RemoteConfigParser.)
Joey Hess [Fri, 10 Jan 2020 18:10:20 +0000 (14:10 -0400)]
be stricter about rejecting invalid configurations for remotes
This is a first step toward that goal, using the ProposedAccepted type
in RemoteConfig lets initremote/enableremote reject bad parameters that
were passed in a remote's configuration, while avoiding enableremote
rejecting bad parameters that have already been stored in remote.log
This does not eliminate every place where a remote config is parsed and a
default value is used if the parse false. But, I did fix several
things that expected foo=yes/no and so confusingly accepted foo=true but
treated it like foo=no. There are still some fields that are parsed with
yesNo but not not checked when initializing a remote, and there are other
fields that are parsed in other ways and not checked when initializing a
remote.
This also lays groundwork for rejecting unknown/typoed config keys.
projects / git-annex.git / log