enable filter.annex.process in v9

This has tradeoffs, but is generally a win, and users who it causes git add to
slow down unacceptably for can just disable it again.

It needed to happen in an upgrade, since there are git-annex versions
that do not support it, and using such an old version with a v8
repository with filter.annex.process set will cause bad behavior.
By enabling it in v9, it's guaranteed that any git-annex version that
can use the repository does support it. Although, this is not a perfect
protection against problems, since an old git-annex version, if it's
used with a v9 repository, will cause git add to try to run
git-annex filter-process, which will fail. But at least, the user is
unlikely to have an old git-annex in path if they are using a v9
repository, since it won't work in that repository.

Sponsored-by: Dartmouth College's Datalad project

diff --git a/Annex/Version.hs b/Annex/Version.hs
index 944dc14c6f80f7bac41dff07ef17849354b2934b..7f604fefc9b49b0d33780d0b8aa7a10e718557e7 100644 (file)
--- a/Annex/Version.hs
+++ b/Annex/Version.hs
@@ -57,6 +57,11 @@ setVersion (RepoVersion v) = setConfig versionField (show v)
 removeVersion :: Annex ()
 removeVersion = unsetConfig versionField
 
+versionSupportsFilterProcess :: Maybe RepoVersion -> Bool
+versionSupportsFilterProcess (Just v) 
+       | v >= RepoVersion 9 = True
+versionSupportsFilterProcess _ = False
+
 versionNeedsWritableContentFiles :: Maybe RepoVersion -> Bool
 versionNeedsWritableContentFiles (Just v) 
        | v >= RepoVersion 10 = False

diff --git a/CHANGELOG b/CHANGELOG
index b5893ecce9e13c90cfa35c420b4b90995a1dfbcd..8ff21c3bb9f3d66222aa4619ba33054ba8abaefe 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -9,6 +9,11 @@ git-annex (10.20220121) UNRELEASED; urgency=medium
     upgrade, in order to allow time for any old git-annex processes that
     are not aware of the locking change to finish. Or git-annex upgrade
     can be used to upgrade to v10 immediately.
+  * In v9, set filter.annex.process. This makes git add/checkout faster when
+    there are a lot of unlocked annexed files or non-annexed files, but can
+    also makes git add of large files to the annex somewhat slower.
+    If this tradeoff does not work for your use case, you can still unset
+    filter.annex.process.
   * export: When a non-annexed symlink is in the tree to be exported, skip it.
   * import: When the previously exported tree contained a non-annexed symlink,
     preserve it in the imported tree so it does not get deleted.

diff --git a/Config/Smudge.hs b/Config/Smudge.hs
index e124125b650f1b171ecfded53dbb2182ac014913..3a33bcfda30158cc971c6eb0046ae425f0f972b3 100644 (file)
--- a/Config/Smudge.hs
+++ b/Config/Smudge.hs
@@ -1,6 +1,6 @@
 {- Git smudge filter configuration
  -
- - Copyright 2011-2019 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2022 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -16,6 +16,7 @@ import qualified Git.Command
 import Git.Types
 import Config
 import Utility.Directory.Create
+import Annex.Version
 
 import qualified System.FilePath.ByteString as P
 
@@ -32,6 +33,8 @@ configureSmudgeFilter = unlessM (fromRepo Git.repoIsLocalBare) $ do
 
        setConfig (ConfigKey "filter.annex.smudge") "git-annex smudge -- %f"
        setConfig (ConfigKey "filter.annex.clean") "git-annex smudge --clean -- %f"
+       whenM (versionSupportsFilterProcess <$> getVersion)
+               configureSmudgeFilterProcess
        lf <- Annex.fromRepo Git.attributesLocal
        gf <- Annex.fromRepo Git.attributes
        lfs <- readattr lf
@@ -43,6 +46,10 @@ configureSmudgeFilter = unlessM (fromRepo Git.repoIsLocalBare) $ do
   where
        readattr = liftIO . catchDefaultIO "" . readFileStrict . fromRawFilePath
 
+configureSmudgeFilterProcess :: Annex ()
+configureSmudgeFilterProcess =
+       setConfig (ConfigKey "filter.annex.process") "git-annex filter-process"
+
 stdattr :: [String]
 stdattr =
        [ "* filter=annex"

diff --git a/Upgrade/V8.hs b/Upgrade/V8.hs
index f10333cbc98e265e48802263cf9e970ff726350c..57f4951dbb39b33510deb3d01b6460fa3a88cae0 100644 (file)
--- a/Upgrade/V8.hs
+++ b/Upgrade/V8.hs
@@ -9,10 +9,13 @@ module Upgrade.V8 where
 
 import Annex.Common
 import Types.Upgrade
+import Config.Smudge
 
 upgrade :: Bool -> Annex UpgradeResult
 upgrade automatic = do
        unless automatic $
                showAction "v8 to v9"
 
+       configureSmudgeFilterProcess
+
        return UpgradeSuccess

diff --git a/doc/todo/git_smudge_clean_interface_suboptiomal.mdwn b/doc/todo/git_smudge_clean_interface_suboptiomal.mdwn
index 3ff8d873c9f378c1857b6245581e25b0327b45f6..a8e4fee4e5cb1f3d58a27122f6537239ab00729c 100644 (file)
--- a/doc/todo/git_smudge_clean_interface_suboptiomal.mdwn
+++ b/doc/todo/git_smudge_clean_interface_suboptiomal.mdwn
@@ -17,7 +17,8 @@ to git. git-lfs uses it that way.
 The first problem with the interface was that it ran a command once per
 file. This was later fixed by extending it to support long-running filter
 processes, which git-lfs uses. git-annex can also use that interface,
-when `git-annex filter-process` is enabled, but it does not by default.
+when `git-annex filter-process` is enabled. That is the case in v9
+repositories and above.
 
 A second problem with the interface, which affects git-lfs AFAIK, is that
 git buffers the output of the smudge filter in memory before updating the
@@ -81,12 +82,12 @@ And here's the consequences of git-annex's workarounds:
 * It doesn't use the long-running filter process interface by default, 
   so `git add` of a lot of files runs `git-annex smudge --clean` once per file,
   which is slower than it could be. Using `git-annex add` avoids this problem.
-  So does enabling `git-annex filter-process`.
+  So does enabling `git-annex filter-process`, which is default in v9.
 
 * After a git-annex get/drop or a git checkout or pull that affects a lot
   of files, the clean filter gets run once per file, which is again, slower
   than ideal. Enabling `git-annex filter-process` can speed this up
-  in some cases.
+  in some cases, and is default in v9.
 
 * When `git-annex filter-process` is enabled, it cannot use the trick
   described above that `git-annex smudge --clean` uses to avoid git

diff --git a/doc/todo/incremental_hashing_for_add.mdwn b/doc/todo/incremental_hashing_for_add.mdwn
index 37087809640428a3400d1d69fb99cd32851d9f80..fae818593ac3adf391f3e1ed8bac82c09154ace8 100644 (file)
--- a/doc/todo/incremental_hashing_for_add.mdwn
+++ b/doc/todo/incremental_hashing_for_add.mdwn
@@ -1,7 +1,7 @@
-When `git-annex filter-process` is enabled, `git add` pipes the content of
-files into it, but that's thrown away, and the file is read again by git-annex
-to generate a hash. It would improve performance to hash the content
-provided via the pipe.
+When `git-annex filter-process` is enabled (v9 and above), `git add` pipes
+the content of files into it, but that's thrown away, and the file is read
+again by git-annex to generate a hash. It would improve performance to hash
+the content provided via the pipe.
 
 When filter-process is not enabled, `git-annex smudge --clean` reads
 the file to hash it, then reads it a second time to copy it into

diff --git a/doc/todo/v9_changes.mdwn b/doc/todo/v9_changes.mdwn
index 87ac7b39f705b6ff7a6d60e07db3396f726327e7..4b242359f18c08fcf723e0cdca5a3b6eda9eadaf 100644 (file)
--- a/doc/todo/v9_changes.mdwn
+++ b/doc/todo/v9_changes.mdwn
@@ -18,3 +18,5 @@ could change and if it does, these things could be included.
   seem worth it.
 
   May want to implement [[incremental_hashing_for_add]] first.
+
+[[done]] --[[Joey]]