queue more changes to keys db

Increasing the size of the queue 10x makes git-annex init 7% faster in a
repository with 86000 annexed files.

The memory use goes up, from 70876 kb to 85376 kb.

diff --git a/CHANGELOG b/CHANGELOG
index eed419b50f2b58ea13f46ce7fea0e562fe8cb237..a8b06f14129c67c1f510740de9599bdc8a33bad2 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,7 +1,7 @@
 git-annex (10.20221105) UNRELEASED; urgency=medium
 
   * Support quettabyte and yottabyte.
-  * Sped up the initial scanning for annexed files by 15%.
+  * Sped up the initial scanning for annexed files by 21%.
 
  -- Joey Hess <id@joeyh.name>  Fri, 18 Nov 2022 12:58:06 -0400
 

diff --git a/Database/Keys/SQL.hs b/Database/Keys/SQL.hs
index 54e1a59bf72804c9de02184a44e0be1b32e90fcf..77afb871a18691d3429cf7fa06c086fa3f16aa24 100644 (file)
--- a/Database/Keys/SQL.hs
+++ b/Database/Keys/SQL.hs
@@ -73,8 +73,8 @@ newtype WriteHandle = WriteHandle H.DbQueue
 queueDb :: SqlPersistM () -> WriteHandle -> IO ()
 queueDb a (WriteHandle h) = H.queueDb h checkcommit a
   where
-       -- commit queue after 1000 changes
-       checkcommit sz _lastcommittime = pure (sz > 1000)
+       -- commit queue after 10000 changes
+       checkcommit sz _lastcommittime = pure (sz > 10000)
 
 -- Insert the associated file.
 -- When the file was associated with a different key before,

diff --git a/doc/bugs/performance_regression__63___init_takes_times_more/comment_13_a79fcbe80060d11582989a9fc31d4a92._comment b/doc/bugs/performance_regression__63___init_takes_times_more/comment_13_a79fcbe80060d11582989a9fc31d4a92._comment
index 02dac893e3bd6020f31951fd688a67e767bb680f..36474beabef4940cc5ca9a8a7dac48983c265bf0 100644 (file)
--- a/doc/bugs/performance_regression__63___init_takes_times_more/comment_13_a79fcbe80060d11582989a9fc31d4a92._comment
+++ b/doc/bugs/performance_regression__63___init_takes_times_more/comment_13_a79fcbe80060d11582989a9fc31d4a92._comment
@@ -12,6 +12,4 @@ This will need some care to be implemented safely...
 
 I benchmarked it, and using insertUnique is no faster, but using insert is.
 This would be a 15% speed up.
-
-Update: Implemented this optimisation.
 """]]

diff --git a/doc/bugs/performance_regression__63___init_takes_times_more/comment_14_8c3b13806adb731435b346a64990527b._comment b/doc/bugs/performance_regression__63___init_takes_times_more/comment_14_8c3b13806adb731435b346a64990527b._comment
new file mode 100644 (file)
index 0000000..8c6505d
--- /dev/null
+++ b/doc/bugs/performance_regression__63___init_takes_times_more/comment_14_8c3b13806adb731435b346a64990527b._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 14"""
+ date="2022-11-18T17:26:03Z"
+ content="""
+Implemented the two optimisations discussed above, and init in that
+repository dropped from 24 seconds to 19 seconds, a 21% speedup.
+"""]]