]> dgit.raspbian.org Git - git-annex.git/commit
incremental verify for byteRetriever special remotes
authorJoey Hess <joeyh@joeyh.name>
Wed, 11 Aug 2021 17:43:30 +0000 (13:43 -0400)
committerJoey Hess <joeyh@joeyh.name>
Wed, 11 Aug 2021 18:20:38 +0000 (14:20 -0400)
commitc20358b671bb23ba03a82e108d2811c5b42a9bac
tree8d4109c4418cda0cb8a2aa99fe8f8c19d10c2e5b
parent9518aca2f551ff9fc2a90c939716deeb36a861cc
incremental verify for byteRetriever special remotes

Several special remotes verify content while it is being retrieved,
avoiding a separate checksum pass. They are: S3, bup, ddar, and
gcrypt (with a local repository).

Not done when using chunking, yet.

Complicated by Retriever needing to change to be polymorphic. Which in turn
meant RankNTypes is needed, and also needed some code changes. The
change in Remote.External does not change behavior at all but avoids
the type checking failing because of a "rigid, skolem type" which
"would escape its scope". So I refactored slightly to make the type
checker's job easier there.

Unfortunately, directory uses fileRetriever (except when chunked),
so it is not amoung the improved ones. Fixing that would need a way for
FileRetriever to return a Verification. But, since the file retrieved
may be encrypted or chunked, it would be extra work to always
incrementally checksum the file while retrieving it. Hm.

Some other special remotes use fileRetriever, and so don't get incremental
verification, but could be converted to byteRetriever later. One is
GitLFS, which uses downloadConduit, which writes to the file, so could
verify as it goes. Other special remotes like web could too, but don't
use Remote.Helper.Special and so will need to be addressed separately.

Sponsored-by: Dartmouth College's DANDI project
16 files changed:
CHANGELOG
Remote/Adb.hs
Remote/Bup.hs
Remote/Ddar.hs
Remote/Directory.hs
Remote/Directory/LegacyChunked.hs
Remote/External.hs
Remote/GCrypt.hs
Remote/GitLFS.hs
Remote/Glacier.hs
Remote/Helper/Chunked.hs
Remote/Helper/Special.hs
Remote/Hook.hs
Remote/S3.hs
Remote/WebDAV.hs
Types/StoreRetrieve.hs