From: Joey Hess <joeyh@joeyh.name>
Date: Fri, 19 Jan 2024 18:31:54 +0000 (-0400)
Subject: use watchFileSize in Remote.External.retrieveKeyFile
X-Git-Tag: archive/raspbian/10.20250416-2+rpi1~1^2~29^2~20
X-Git-Url: https://dgit.raspbian.org/?a=commitdiff_plain;h=c02df7924803a178f9c3851febe3d44fd67eb7cf;p=git-annex.git

use watchFileSize in Remote.External.retrieveKeyFile

external: Monitor file size when getting content from external special
remotes and use that to update the progress meter, in case the external
special remote program does not report progress.

This relies on 703a70cafa79eb0cf9f3ae676763e1798cb77292 to prevent ever
running the meter backwards.

Sponsored-by: Dartmouth College's DANDI project
---

diff --git a/CHANGELOG b/CHANGELOG
index 8ab8386fb7..c38ba6ecb5 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -15,6 +15,9 @@ git-annex (10.20231228) UNRELEASED; urgency=medium
   * import: --message/-m option.
   * Improve annex.stalldetection to handle remotes that update progress
     less frequently than the configured time period.
+  * external: Monitor file size when getting content from external
+    special remotes and use that to update the progress meter,
+    in case the external special remote program does not report progress.
 
  -- Joey Hess <id@joeyh.name>  Fri, 29 Dec 2023 11:52:06 -0400
 
diff --git a/Remote/External.hs b/Remote/External.hs
index 22a9a61d32..d75c3ebb45 100644
--- a/Remote/External.hs
+++ b/Remote/External.hs
@@ -235,7 +235,7 @@ storeKeyM external = fileStorer $ \k f p ->
 
 retrieveKeyFileM :: External -> Retriever
 retrieveKeyFileM external = fileRetriever $ \d k p ->
-	either giveup return =<< go d k p
+	either giveup return =<< watchFileSize d p (go d k)
   where
 	go d k p = handleRequestKey external (\sk -> TRANSFER Download sk (fromRawFilePath d)) k (Just p) $ \resp ->
 		case resp of
diff --git a/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_5_9fc2d7f4b39615e43bce3993e0a6e647._comment b/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_5_9fc2d7f4b39615e43bce3993e0a6e647._comment
new file mode 100644
index 0000000000..e4e56806bb
--- /dev/null
+++ b/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_5_9fc2d7f4b39615e43bce3993e0a6e647._comment
@@ -0,0 +1,22 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 6"""
+ date="2024-01-19T17:31:41Z"
+ content="""
+Re asymmetric connections, stall detection doesn't need to be
+configured anywhere near the total available bandwidth, so generally it
+would be ok to configure it for a slow uplink even when a fast downlink is
+available. And if you're only interested in detecting total stalls
+(as opposed to some kind of "too slow" condition), "1k" per some unit of
+time is sufficient.
+
+But maybe it does make sense to be able to configure it to abandon slow
+downloads but allow slow uploads. For example, `git-annex get` with
+the content on several remotes, where the download speed from one
+remote is often fast but occasionally slows down, and another remote
+is consistently medium speed.
+So you might set "10gb/1m" for that remote, knowing that if it is slow
+it will abort the download from it and fall back to the medium speed remote.
+But when sending content *to* the variable speed remote, would not want to
+give up only because it was a little slow.
+"""]]
diff --git a/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_6_1b2eb9993e220082f3be5dd568deef3e._comment b/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_6_1b2eb9993e220082f3be5dd568deef3e._comment
new file mode 100644
index 0000000000..d3df8542ec
--- /dev/null
+++ b/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_6_1b2eb9993e220082f3be5dd568deef3e._comment
@@ -0,0 +1,21 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2024-01-19T17:01:27Z"
+ content="""
+> "download", where git-annex actually IIRC keeps its monitoring of the file size as it is arriving
+
+git-annex does do file size monitoring
+in a few cases (when running `cp`, and when downloading
+an url with `curl`), but it does not do it for downloads
+in general, and not for external special remotes specifically.
+
+If it did for external special remotes, and if rclone grows the 
+destination file as it downloads, rather than atomically at the end
+(which I have not verified), that would have avoided this problem.
+Probably it will avoids this type of problem for downloads
+from the majority of external special remotes. Though of course not for
+uploads.
+
+Ok, implemented that for external special remotes.
+"""]]