From: Joey Hess Date: Tue, 5 Oct 2021 21:20:32 +0000 (-0400) Subject: comment X-Git-Tag: archive/raspbian/10.20250416-2+rpi1~1^2~83^2~17 X-Git-Url: https://dgit.raspbian.org/?a=commitdiff_plain;h=250c031d064f3a5f215d3e3d5c032a293f76cc23;p=git-annex.git comment --- diff --git a/doc/bugs/borg_special_remote_memory_usage_high_for_large_borg_repo/comment_6_4b71b012153a71e03c57ae3ed3ce2272._comment b/doc/bugs/borg_special_remote_memory_usage_high_for_large_borg_repo/comment_6_4b71b012153a71e03c57ae3ed3ce2272._comment new file mode 100644 index 0000000000..7d88098501 --- /dev/null +++ b/doc/bugs/borg_special_remote_memory_usage_high_for_large_borg_repo/comment_6_4b71b012153a71e03c57ae3ed3ce2272._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 6""" + date="2021-10-05T20:53:24Z" + content=""" +@tomdhunt so your repo has in the order of 182 million +items for git-annex to track. I do think that is probably too many to be +practical even if this memory problem gets resolved. A list of that many +items is at least 25 gigabytes in size. Add some memory for data structures +and it's hard to see it working with even your enviable 64 gb. + +This brings me back to the idea of only including one item for each key... +Only the item from the most recent archive. +If the oldest archives always are deleted first, that would never leave a +key present in the borg repo without git-annex having a record of the +archive that contained it. + +But if you used borg prune to delete some +intermediate archives, git-annex could no longer know of any existing +archive that contains a key, so getting from the borg repo would fail, +until it re-scanned the whole repo. +git-annex sync could notice when such an intermediate archive +has been deleted, and trigger the re-scan. +"""]]