don't force use of conduit in withUrlOptionsPromptingCreds

Use curl for downloads from git remotes when annex.url-options and other
git configs are set.

If the url needs a password, curl will fail, and git credential will not be
used to prompt for it. But the user can set --netrc in url-options and
put the password in the netrc file.

This also means that url-options settings like -4 will take effect.
That was the case before commit 1883f7ef8f9f617c60832c7f0794b54515fc652d
forced conduit to be used.

diff --git a/Annex/Url.hs b/Annex/Url.hs
index dd418c5d6626c61498cd912bfc5ce11bff9287eb..584482cdc8ea7d40f1b974c86b523ad52d1399c0 100644 (file)
--- a/Annex/Url.hs
+++ b/Annex/Url.hs
@@ -147,6 +147,10 @@ withUrlOptions a = a =<< getUrlOptions
 
 -- When downloading an url, if authentication is needed, uses
 -- git-credential to prompt for username and password.
+--
+-- Note that, when the downloader is curl, it will not use git-credential.
+-- If the user wants to, they can configure curl to use a netrc file that
+-- handles authentication.
 withUrlOptionsPromptingCreds :: (U.UrlOptions -> Annex a) -> Annex a
 withUrlOptionsPromptingCreds a = do
        g <- Annex.gitRepo
@@ -156,12 +160,6 @@ withUrlOptionsPromptingCreds a = do
        a $ uo
                { U.getBasicAuth = \u -> prompter $
                        getBasicAuthFromCredential g cc u
-               -- Can't download with curl and handle basic auth,
-               -- so make sure it uses conduit.
-               , U.urlDownloader = case U.urlDownloader uo of
-                       U.DownloadWithCurl _ -> U.DownloadWithConduit $
-                               U.DownloadWithCurlRestricted mempty
-                       v -> v
                }
 
 checkBoth :: U.URLString -> Maybe Integer -> U.UrlOptions -> Annex Bool

diff --git a/CHANGELOG b/CHANGELOG
index c410c9ec2af5aac6f3b8a82648067edbde587309..d948c20931b147ed4218fb9bcb09dc294889da94 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -8,6 +8,8 @@ git-annex (10.20220823) UNRELEASED; urgency=medium
   * When accessing a git remote over http needs a git credential
     prompt for a password, cache it for the lifetime of the git-annex
     process, rather than repeatedly prompting.
+  * Use curl for downloads from git remotes when annex.url-options
+    and other git configs are set.
 
  -- Joey Hess <id@joeyh.name>  Mon, 29 Aug 2022 15:03:04 -0400
 

diff --git a/doc/bugs/http_remotes_ignore_annex.web-options_--netrc/comment_2_599a66a1833000dc0e54a33511891b36._comment b/doc/bugs/http_remotes_ignore_annex.web-options_--netrc/comment_2_599a66a1833000dc0e54a33511891b36._comment
new file mode 100644 (file)
index 0000000..5580779
--- /dev/null
+++ b/doc/bugs/http_remotes_ignore_annex.web-options_--netrc/comment_2_599a66a1833000dc0e54a33511891b36._comment
@@ -0,0 +1,33 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2022-09-09T19:19:44Z"
+ content="""
+Confirmed this behavior.
+
+It is due to withUrlOptionsPromptingCreds, which forces use of conduit
+rather than curl. The idea there was to use git credentials when basic
+auth is needed. Since those can be provided to conduit but not to curl
+(securely).
+
+But I do think that, if the user has forced use of curl, it ought to use curl.
+Even if the user only set options to `-4`, and so curl is not going to use
+the netrc and will fail the download. I have changed it to do so.
+
+----
+
+This bug report also suggests making git-annex read the netrc file itself.
+Note that git does *not* read the netrc file itself. What it does do is use
+libcurl. git-annex has good reasons to not use libcurl though.
+
+I am not thrilled by the prospect of implementing a parser for netrc
+in git-annex. The file is not even documented on my debian system;
+curl's man page links to a `netrc(5)` but that does not exist.
+
+Aside from git-credential-netrc, there is not a single mention of 
+the netrc file in git's documentation. This is arguably surprising behavior
+on the part of git. 
+
+I feel that git's support for netrc is vestigal and mostly supersceded by
+git credentials.
+"""]]

diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index ac9af004f5f5e1eaa31b3dd55346a1fec52929a6..b6dd7300845318e4eb102ac00726a30458c3d192 100644 (file)
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -1698,12 +1698,15 @@ Remotes are configured using these settings in `.git/config`.
   (rather than the default built-in url downloader).
 
   For example, to force IPv4 only, set it to "-4".
-  Or to make curl use your ~/.netrc file, set it to "--netrc".
 
   Setting this option makes git-annex use curl, but only
   when annex.security.allowed-ip-addresses is configured in a
   specific way. See its documentation.
 
+  Setting this option prevents git-annex from using git-credential
+  for prompting for http passwords. Instead, you can include "--netrc"
+  to make curl use your ~/.netrc file and record the passwords there.
+
 * `annex.youtube-dl-options`
 
   Options to pass to youtube-dl when using it to find the url to download