cache negative lookups of global numcopies and mincopies

Speeds up eg git-annex sync --content by up to 50%. When it does not need
to transfer or drop anything, it now noops a lot more quickly.

I didn't see anything else in sync --content noop loop that could really
be sped up. It has to cat git objects to keys, stat object files, etc.

Sponsored-by: unqueued on Patreon

diff --git a/Annex.hs b/Annex.hs
index 5a9eac3c32c6ec59f0fa14fe38f71e3c7241c980..74d9f9891f8ca8300ad9b8742e4601664748c81e 100644 (file)
--- a/Annex.hs
+++ b/Annex.hs
@@ -183,8 +183,8 @@ data AnnexState = AnnexState
        , hashobjecthandle :: Maybe (ResourcePool HashObjectHandle)
        , checkattrhandle :: Maybe (ResourcePool CheckAttrHandle)
        , checkignorehandle :: Maybe (ResourcePool CheckIgnoreHandle)
-       , globalnumcopies :: Maybe NumCopies
-       , globalmincopies :: Maybe MinCopies
+       , globalnumcopies :: Maybe (Maybe NumCopies)
+       , globalmincopies :: Maybe (Maybe MinCopies)
        , limit :: ExpandableMatcher Annex
        , timelimit :: Maybe (Duration, POSIXTime)
        , sizelimit :: Maybe (TVar Integer)

diff --git a/CHANGELOG b/CHANGELOG
index 83577218882233afeb1d092c7bac0bf487995c97..862e86628789d9ab9725791fad95e660d9ca5129 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -79,6 +79,8 @@ git-annex (10.20230408) UNRELEASED; urgency=medium
   * Large speed up to importing trees from special remotes that contain a lot
     of files, by only processing changed files.
   * Some other speedups to importing trees from special remotes.
+  * Cache negative lookups of global numcopies and mincopies. 
+    Speeds up eg git-annex sync --content by up to 50%.
 
  -- Joey Hess <id@joeyh.name>  Sat, 08 Apr 2023 13:57:18 -0400
 

diff --git a/Logs/NumCopies.hs b/Logs/NumCopies.hs
index 3c358819bb13d74397141964f3ba3281db7a4de6..7579baf3f385403a48c7d6339d82233ff4c9ce6b 100644 (file)
--- a/Logs/NumCopies.hs
+++ b/Logs/NumCopies.hs
@@ -45,22 +45,22 @@ setGlobalMinCopies new = do
 
 {- Value configured in the numcopies log. Cached for speed. -}
 getGlobalNumCopies :: Annex (Maybe NumCopies)
-getGlobalNumCopies = maybe globalNumCopiesLoad (return . Just)
+getGlobalNumCopies = maybe globalNumCopiesLoad return
        =<< Annex.getState Annex.globalnumcopies
 
 {- Value configured in the mincopies log. Cached for speed. -}
 getGlobalMinCopies :: Annex (Maybe MinCopies)
-getGlobalMinCopies = maybe globalMinCopiesLoad (return . Just)
+getGlobalMinCopies = maybe globalMinCopiesLoad return
        =<< Annex.getState Annex.globalmincopies
 
 globalNumCopiesLoad :: Annex (Maybe NumCopies)
 globalNumCopiesLoad = do
        v <- getLog numcopiesLog
-       Annex.changeState $ \s -> s { Annex.globalnumcopies = v }
+       Annex.changeState $ \s -> s { Annex.globalnumcopies = Just v }
        return v
 
 globalMinCopiesLoad :: Annex (Maybe MinCopies)
 globalMinCopiesLoad = do
        v <- getLog mincopiesLog
-       Annex.changeState $ \s -> s { Annex.globalmincopies = v }
+       Annex.changeState $ \s -> s { Annex.globalmincopies = Just v }
        return v

diff --git a/doc/bugs/importtree_spends_hours_reading_cidsdb/comment_14_e961d2b508e3260670bb61a5ae51d109._comment b/doc/bugs/importtree_spends_hours_reading_cidsdb/comment_14_e961d2b508e3260670bb61a5ae51d109._comment
new file mode 100644 (file)
index 0000000..42efb82
--- /dev/null
+++ b/doc/bugs/importtree_spends_hours_reading_cidsdb/comment_14_e961d2b508e3260670bb61a5ae51d109._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 14"""
+ date="2023-06-06T17:11:35Z"
+ content="""
+There's only one import in the sync, and your output shows it completed
+(with error).
+
+The only other phase of sync that could be run after that and take a lot of
+time is content syncing. You would have to have annex.synccontent set
+somewhere for sync to do that. Do you?
+"""]]

diff --git a/doc/bugs/importtree_spends_hours_reading_cidsdb/comment_15_35d233db5f7410048bd0cf9f5b4fd5cc._comment b/doc/bugs/importtree_spends_hours_reading_cidsdb/comment_15_35d233db5f7410048bd0cf9f5b4fd5cc._comment
new file mode 100644 (file)
index 0000000..2073dca
--- /dev/null
+++ b/doc/bugs/importtree_spends_hours_reading_cidsdb/comment_15_35d233db5f7410048bd0cf9f5b4fd5cc._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 15"""
+ date="2023-06-06T17:31:49Z"
+ content="""
+It would make a lot of sense for --content syncing to be what remains slow.
+That has to scan over all the files and when it decides that it does not
+need to copy the content anywhere, that's a tight loop with no output.
+
+In my repo with 10000 files that was set up by the latest test case,
+`git-annex sync` takes 13 seconds, and with --content it takes 61 seconds.
+
+I optimised a numcopies/mincopies lookup away, and that got it
+down to 28 seconds. 
+
+The cidsdb does not get accessed by the --content scan
+in my testing, although there may be other situations where it does.
+"""]]