When retrival from a chunked remote fails, display the error that occurred when downl...

Rather than the error that occurred when trying to download the unchunked
content, which is less likely to actually be stored in the remote.

Sponsored-by: Boyd Stephen Smith Jr. on Patreon

diff --git a/CHANGELOG b/CHANGELOG
index dfd6b555c81469978da817df310266363fb49696..e8e4ad1d08536eabdcf2f208ce3c97dd37efb451 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -10,6 +10,10 @@ git-annex (8.20211012) UNRELEASED; urgency=medium
   * test: Put gpg temp home directory in system temp directory,
     not filesystem being tested.
   * Avoid crashing tilde expansion on user who does not exist.
+  * When retrival from a chunked remote fails, display the error that
+    occurred when downloading the chunk, rather than the error that
+    occurred when trying to download the unchunked content, which is less
+    likely to actually be stored in the remote.
 
  -- Joey Hess <id@joeyh.name>  Mon, 11 Oct 2021 14:09:13 -0400
 

diff --git a/Remote/Helper/Chunked.hs b/Remote/Helper/Chunked.hs
index ade47f3f4ce8694c5b30d06ab6099af5286e7cf2..b56d43389acc53533e7717d7f667e8ad32fd0255 100644 (file)
--- a/Remote/Helper/Chunked.hs
+++ b/Remote/Helper/Chunked.hs
@@ -286,9 +286,9 @@ retrieveChunks retriever u vc chunkconfig encryptor basek dest basep enc encc
        firstavail Nothing _ [] = giveup "unable to determine the chunks to use for this remote"
        firstavail (Just e) _ [] = throwM e
        firstavail pe currsize ([]:ls) = firstavail pe currsize ls
-       firstavail _ currsize ((k:ks):ls)
+       firstavail pe currsize ((k:ks):ls)
                | k == basek = getunchunked
-                       `catchNonAsync` (\e -> firstavail (Just e) currsize ls)
+                       `catchNonAsync` (\e -> firstavail (Just (pickerr e)) currsize ls)
                | otherwise = do
                        let offset = resumeOffset currsize k
                        let p = maybe basep
@@ -302,10 +302,15 @@ retrieveChunks retriever u vc chunkconfig encryptor basek dest basep enc encc
                                                        fromMaybe 0 $ fromKey keyChunkSize k
                                                getrest p h iv sz sz ks
                        case v of
-                               Left e
-                                       | null ls -> throwM e
-                                       | otherwise -> firstavail (Just e) currsize ls
+                               Left e -> firstavail (Just (pickerr e)) currsize ls
                                Right r -> return r
+         where
+               -- Prefer an earlier exception to a later one, because the
+               -- more probable location is tried first and less probable
+               -- ones later.
+               pickerr e = case pe of
+                       Just pe' -> pe'
+                       Nothing -> e
 
        getrest _ _ iv _ _ [] = return (Right iv)
        getrest p h iv sz bytesprocessed (k:ks) = do

diff --git a/doc/bugs/Improvements_to_S3_glacier_integration.mdwn b/doc/bugs/Improvements_to_S3_glacier_integration.mdwn
index 233acfdbeddd69eb00ff671c4c0ba56715e65bba..983869528df22dfcedb8414df3d4395c6309720b 100644 (file)
--- a/doc/bugs/Improvements_to_S3_glacier_integration.mdwn
+++ b/doc/bugs/Improvements_to_S3_glacier_integration.mdwn
@@ -115,3 +115,5 @@ ongoing-request="false", expiry-date="Mon, 26 Apr 2021 00:00:00 GMT"
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 
 Yes, for loads of stuff. It's awesome, thanks!
+
+> [[closed|done]], see my comment --[[Joey]]

diff --git a/doc/bugs/Improvements_to_S3_glacier_integration/comment_5_9cbe83bbade15b9146d033ceb5d8b05d._comment b/doc/bugs/Improvements_to_S3_glacier_integration/comment_5_9cbe83bbade15b9146d033ceb5d8b05d._comment
new file mode 100644 (file)
index 0000000..b0dfda3
--- /dev/null
+++ b/doc/bugs/Improvements_to_S3_glacier_integration/comment_5_9cbe83bbade15b9146d033ceb5d8b05d._comment
@@ -0,0 +1,19 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2021-10-14T16:17:14Z"
+ content="""
+I have added a note to the S3 documentation about `DEEP_ARCHIVE` and the
+glacier special remote.
+
+I have made git-annex display the exception for the more likely chunked 
+location, rather than the less likely unchunked location, when retrieving
+from both locations fails. Although it's still possible for there to be
+situations where the exception if displays is not for the location where
+the content actually is. Eg, if the chunk size of the remote has
+changed over time.
+
+I think that todo is basically talking about the same desire to make the S3
+remote support these glacier-style storage classes, in one way or another, 
+and so I think this bug report can be closed as otherwise a duplicate of it.
+"""]]

diff --git a/doc/special_remotes/S3.mdwn b/doc/special_remotes/S3.mdwn
index 71d74a533a179dff41be4b9d13aa7ae2fae44d62..e34cbb7f9d413e11bdfd63b78f38c703a7bb8577 100644 (file)
--- a/doc/special_remotes/S3.mdwn
+++ b/doc/special_remotes/S3.mdwn
@@ -44,14 +44,17 @@ the S3 remote.
   
   When using Amazon S3,
   if the remote will be used for backup or archival,
-  and so its files are Infrequently Accessed, "STANDARD_IA" is a
+  and so its files are Infrequently Accessed, `STANDARD_IA` is a
   good choice to save money (requires a git-annex built with aws-0.13.0).
   If you have configured git-annex to preserve
-  multiple [[copies]], also consider setting this to "ONEZONE_IA"
+  multiple [[copies]], also consider setting this to `ONEZONE_IA`
   to save even more money.
 
+  Amazon S3's `DEEP_ARCHIVE` is similar to Amazon Glacier. For that,
+  use the [[glacier]] special remote, rather than this one.
+
   When using Google Cloud Storage, to make a nearline bucket, set this to
-  "NEARLINE". (Requires a git-annex built with aws-0.13.0)
+  `NEARLINE`. (Requires a git-annex built with aws-0.13.0)
 
   Note that changing the storage class of an existing S3 remote will
   affect new objects sent to the remote, but not objects already