From 250c031d064f3a5f215d3e3d5c032a293f76cc23 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Tue, 5 Oct 2021 17:20:32 -0400 Subject: [PATCH] comment --- ..._4b71b012153a71e03c57ae3ed3ce2272._comment | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 doc/bugs/borg_special_remote_memory_usage_high_for_large_borg_repo/comment_6_4b71b012153a71e03c57ae3ed3ce2272._comment diff --git a/doc/bugs/borg_special_remote_memory_usage_high_for_large_borg_repo/comment_6_4b71b012153a71e03c57ae3ed3ce2272._comment b/doc/bugs/borg_special_remote_memory_usage_high_for_large_borg_repo/comment_6_4b71b012153a71e03c57ae3ed3ce2272._comment new file mode 100644 index 0000000000..7d88098501 --- /dev/null +++ b/doc/bugs/borg_special_remote_memory_usage_high_for_large_borg_repo/comment_6_4b71b012153a71e03c57ae3ed3ce2272._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 6""" + date="2021-10-05T20:53:24Z" + content=""" +@tomdhunt so your repo has in the order of 182 million +items for git-annex to track. I do think that is probably too many to be +practical even if this memory problem gets resolved. A list of that many +items is at least 25 gigabytes in size. Add some memory for data structures +and it's hard to see it working with even your enviable 64 gb. + +This brings me back to the idea of only including one item for each key... +Only the item from the most recent archive. +If the oldest archives always are deleted first, that would never leave a +key present in the borg repo without git-annex having a record of the +archive that contained it. + +But if you used borg prune to delete some +intermediate archives, git-annex could no longer know of any existing +archive that contains a key, so getting from the borg repo would fail, +until it re-scanned the whole repo. +git-annex sync could notice when such an intermediate archive +has been deleted, and trigger the re-scan. +"""]] -- 2.30.2