comment

diff --git a/doc/todo/add_option_to_use_sqlite__39__s_synchronous__61__OFF/comment_2_e262f441e1b224215e68b5707db1d180._comment b/doc/todo/add_option_to_use_sqlite__39__s_synchronous__61__OFF/comment_2_e262f441e1b224215e68b5707db1d180._comment
new file mode 100644 (file)
index 0000000..7752537
--- /dev/null
+++ b/doc/todo/add_option_to_use_sqlite__39__s_synchronous__61__OFF/comment_2_e262f441e1b224215e68b5707db1d180._comment
@@ -0,0 +1,102 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2021-06-23T19:15:55Z"
+ content="""
+Benchmarked with NORMAL:
+
+       joey@darkstar:~/tmp/t>/usr/bin/time ~/git-annex.synchrousNORMAL add 1??? --quiet
+       6.99user 5.09system 0:11.63elapsed 103%CPU (0avgtext+0avgdata 68356maxresident)k
+       143760inputs+40352outputs (819major+404774minor)pagefaults 0swaps
+       joey@darkstar:~/tmp/t>/usr/bin/time ~/git-annex.synchrousNORMAL add 2??? --quiet
+       7.71user 5.15system 0:11.93elapsed 107%CPU (0avgtext+0avgdata 69876maxresident)k
+       11336inputs+42648outputs (9major+414417minor)pagefaults 0swaps
+       joey@darkstar:~/tmp/t>/usr/bin/time ~/git-annex.synchrousNORMAL add 3??? --quiet
+       7.99user 5.16system 0:12.20elapsed 107%CPU (0avgtext+0avgdata 70452maxresident)k
+       11952inputs+44200outputs (8major+415267minor)pagefaults 0swaps
+       joey@darkstar:~/tmp/t>/usr/bin/time ~/git-annex.synchrousNORMAL add 4??? --quiet
+       8.30user 5.25system 0:12.62elapsed 107%CPU (0avgtext+0avgdata 69496maxresident)k
+       17784inputs+45776outputs (9major+416640minor)pagefaults 0swaps
+
+Which is no improvement over git-annex with no pragmas. Actually slower.
+
+       joey@darkstar:~/tmp/t>/usr/bin/time ~/git-annex.orig add 1??? --quiet
+       6.89user 5.36system 0:11.39elapsed 107%CPU (0avgtext+0avgdata 50576maxresident)k
+       47064inputs+40352outputs (5616major+404472minor)pagefaults 0swaps
+       joey@darkstar:~/tmp/u>/usr/bin/time git-annex add 2??? --quiet
+       7.76user 5.09system 0:11.88elapsed 108%CPU (0avgtext+0avgdata 70848maxresident)k
+       12776inputs+42648outputs (9major+414346minor)pagefaults 0swaps
+       joey@darkstar:~/tmp/u>/usr/bin/time git-annex add 3??? --quiet
+       7.90user 5.26system 0:12.14elapsed 108%CPU (0avgtext+0avgdata 71676maxresident)k
+       13824inputs+44200outputs (8major+415258minor)pagefaults 0swaps
+       joey@darkstar:~/tmp/u>/usr/bin/time git-annex add 4??? --quiet
+       8.22user 5.38system 0:12.49elapsed 108%CPU (0avgtext+0avgdata 71652maxresident)k
+       14216inputs+45776outputs (8major+416784minor)pagefaults 0swaps
+
+OFF also benchmarks very close to the same.
+
+       joey@darkstar:~/tmp/v>/usr/bin/time ~/git-annex.synchrousOFF add 1??? --quiet
+       6.85user 5.58system 0:12.01elapsed 103%CPU (0avgtext+0avgdata 71100maxresident)k
+       50080inputs+40352outputs (16major+405312minor)pagefaults 0swaps
+       joey@darkstar:~/tmp/v>/usr/bin/time ~/git-annex.synchrousOFF add 2??? --quiet
+       7.64user 5.31system 0:11.96elapsed 108%CPU (0avgtext+0avgdata 71392maxresident)k
+       12672inputs+42640outputs (8major+414373minor)pagefaults 0swaps
+       joey@darkstar:~/tmp/v>/usr/bin/time ~/git-annex.synchrousOFF add 3??? --quiet
+       8.02user 5.15system 0:12.19elapsed 108%CPU (0avgtext+0avgdata 71556maxresident)k
+       11648inputs+43928outputs (8major+415140minor)pagefaults 0swaps
+       joey@darkstar:~/tmp/v>/usr/bin/time ~/git-annex.synchrousOFF add 4??? --quiet
+       8.24user 5.24system 0:12.41elapsed 108%CPU (0avgtext+0avgdata 71224maxresident)k
+       10952inputs+45304outputs (8major+416560minor)pagefaults 0swaps
+
+One pass did run 0.08s faster, could be due to not syncing but it does
+not seem a significant optimisation, at least not on this SSD.
+
+Should be noted that transactions build up 1000 changes, and that benchmark
+was operating on 1000 files per run, so it probably only wrote one or two
+transactions.
+
+Here's the patch that adds a pragma:
+
+       diff --git a/Database/Handle.hs b/Database/Handle.hs
+       index d7f1822dc..2d66af5e6 100644
+       --- a/Database/Handle.hs
+       +++ b/Database/Handle.hs
+       @@ -1,11 +1,11 @@
+        {- Persistent sqlite database handles.
+         -
+       - - Copyright 2015-2019 Joey Hess <id@joeyh.name>
+       + - Copyright 2015-2021 Joey Hess <id@joeyh.name>
+         -
+         - Licensed under the GNU AGPL version 3 or higher.
+         -}
+        
+       -{-# LANGUAGE TypeFamilies, FlexibleContexts #-}
+       +{-# LANGUAGE TypeFamilies, FlexibleContexts, OverloadedStrings #-}
+        
+        module Database.Handle (
+               DbHandle,
+       @@ -34,6 +34,7 @@ import qualified Data.Text as T
+        import Control.Monad.Trans.Resource (runResourceT)
+        import Control.Monad.Logger (runNoLoggingT)
+        import System.IO
+       +import Lens.Micro
+        
+        {- A DbHandle is a reference to a worker thread that communicates with
+         - the database. It has a MVar which Jobs are submitted to. -}
+       @@ -194,10 +195,13 @@ runSqliteRobustly tablename db a = do
+               maxretries = 100 :: Int
+               
+               rethrow msg e = throwIO $ userError $ show e ++ "(" ++ msg ++ ")"
+       -       
+       +
+       +       conninfo = over extraPragmas (const ["PRAGMA synchronous=OFF"]) $
+       +               mkSqliteConnectionInfo db
+       +
+               go conn retries = do
+                       r <- try $ runResourceT $ runNoLoggingT $
+       -                       withSqlConnRobustly (wrapConnection conn) $
+       +                       withSqlConnRobustly (wrapConnectionInfo conninfo conn) $
+                                       runSqlConn a
+                       case r of
+                               Right v -> return v
+"""]]