use watchFileSize in Remote.External.retrieveKeyFile

external: Monitor file size when getting content from external special
remotes and use that to update the progress meter, in case the external
special remote program does not report progress.

This relies on 703a70cafa79eb0cf9f3ae676763e1798cb77292 to prevent ever
running the meter backwards.

Sponsored-by: Dartmouth College's DANDI project

diff --git a/CHANGELOG b/CHANGELOG
index 8ab8386fb71c38d40500a25e6f736ffd63becd3e..c38ba6ecb57f5e80034373247a2ae5f06690e278 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -15,6 +15,9 @@ git-annex (10.20231228) UNRELEASED; urgency=medium
   * import: --message/-m option.
   * Improve annex.stalldetection to handle remotes that update progress
     less frequently than the configured time period.
+  * external: Monitor file size when getting content from external
+    special remotes and use that to update the progress meter,
+    in case the external special remote program does not report progress.
 
  -- Joey Hess <id@joeyh.name>  Fri, 29 Dec 2023 11:52:06 -0400
 

diff --git a/Remote/External.hs b/Remote/External.hs
index 22a9a61d32614fea187f24ae20ed0e2970265b5c..d75c3ebb4558b5e5682c468d7a1bf4018d34cf2d 100644 (file)
--- a/Remote/External.hs
+++ b/Remote/External.hs
@@ -235,7 +235,7 @@ storeKeyM external = fileStorer $ \k f p ->
 
 retrieveKeyFileM :: External -> Retriever
 retrieveKeyFileM external = fileRetriever $ \d k p ->
-       either giveup return =<< go d k p
+       either giveup return =<< watchFileSize d p (go d k)
   where
        go d k p = handleRequestKey external (\sk -> TRANSFER Download sk (fromRawFilePath d)) k (Just p) $ \resp ->
                case resp of

diff --git a/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_5_9fc2d7f4b39615e43bce3993e0a6e647._comment b/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_5_9fc2d7f4b39615e43bce3993e0a6e647._comment
new file mode 100644 (file)
index 0000000..e4e5680
--- /dev/null
+++ b/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_5_9fc2d7f4b39615e43bce3993e0a6e647._comment
@@ -0,0 +1,22 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 6"""
+ date="2024-01-19T17:31:41Z"
+ content="""
+Re asymmetric connections, stall detection doesn't need to be
+configured anywhere near the total available bandwidth, so generally it
+would be ok to configure it for a slow uplink even when a fast downlink is
+available. And if you're only interested in detecting total stalls
+(as opposed to some kind of "too slow" condition), "1k" per some unit of
+time is sufficient.
+
+But maybe it does make sense to be able to configure it to abandon slow
+downloads but allow slow uploads. For example, `git-annex get` with
+the content on several remotes, where the download speed from one
+remote is often fast but occasionally slows down, and another remote
+is consistently medium speed.
+So you might set "10gb/1m" for that remote, knowing that if it is slow
+it will abort the download from it and fall back to the medium speed remote.
+But when sending content *to* the variable speed remote, would not want to
+give up only because it was a little slow.
+"""]]

diff --git a/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_6_1b2eb9993e220082f3be5dd568deef3e._comment b/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_6_1b2eb9993e220082f3be5dd568deef3e._comment
new file mode 100644 (file)
index 0000000..d3df854
--- /dev/null
+++ b/doc/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/comment_6_1b2eb9993e220082f3be5dd568deef3e._comment
@@ -0,0 +1,21 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2024-01-19T17:01:27Z"
+ content="""
+> "download", where git-annex actually IIRC keeps its monitoring of the file size as it is arriving
+
+git-annex does do file size monitoring
+in a few cases (when running `cp`, and when downloading
+an url with `curl`), but it does not do it for downloads
+in general, and not for external special remotes specifically.
+
+If it did for external special remotes, and if rclone grows the 
+destination file as it downloads, rather than atomically at the end
+(which I have not verified), that would have avoided this problem.
+Probably it will avoids this type of problem for downloads
+from the majority of external special remotes. Though of course not for
+uploads.
+
+Ok, implemented that for external special remotes.
+"""]]