support --batch -J

--batch combined with -J now runs batch requests concurrently for many
commands. Before, the combination was accepted, but did not enable
concurrency. Since the output of batch requests can be in any order, --json
with the new "input" field is recommended to be used, to determine which
batch request each response corresponds to.

If --json is not used, batch mode still runs concurrently, using the usual
concurrent-output. That will not be very useful for most batch mode users,
probably, but who knows.

If a program was using --batch -J before, and was parsing non-json output,
this could break it. But, it was relying on git-annex not supporting
concurrency despite it being enabled, so it should have expected concurrent
output. So, I think that's ok.

annex.jobs does not enable concurrency in --batch mode, because that would
confuse programs that use --batch but don't expect concurrency.

diff --git a/CHANGELOG b/CHANGELOG
index b3bd319f1b134e208a27910565be4e656d4016b5..44a626bdf01618bf976ea338cd89626475a8b7ba 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -2,6 +2,11 @@ git-annex (8.20200909) UNRELEASED; urgency=medium
 
   * --json output now includes a new field "input" which is the input
     value (filename, url, etc) that caused a json object to be output.
+  * --batch combined with -J now runs batch requests concurrently for many
+    commands. Before, the combination was accepted, but did not enable
+    concurrency. Since the output of batch requests can be in any order,
+    --json with the new "input" field is recommended to be used,
+    to determine which batch request each response corresponds to.
 
  -- Joey Hess <id@joeyh.name>  Mon, 14 Sep 2020 13:13:10 -0400
 

diff --git a/CmdLine/Batch.hs b/CmdLine/Batch.hs
index 2cee7720a0e9bf348aaf4c91d7cc57c6bb98980c..62d2e2b87b006146d9481ba902510128943c9263 100644 (file)
--- a/CmdLine/Batch.hs
+++ b/CmdLine/Batch.hs
@@ -8,6 +8,7 @@
 module CmdLine.Batch where
 
 import Annex.Common
+import qualified Annex
 import Types.Command
 import CmdLine.Action
 import CmdLine.GitAnnex.Options
@@ -18,6 +19,8 @@ import Types.FileMatcher
 import Annex.BranchState
 import Annex.WorkTree
 import Annex.Content
+import Annex.Concurrent
+import Types.Concurrency
 
 data BatchMode = Batch BatchFormat | NoBatch
 
@@ -85,6 +88,7 @@ batchInput fmt parser a = go =<< batchLines fmt
 
 batchLines :: BatchFormat -> Annex [String]
 batchLines fmt = do
+       checkBatchConcurrency
        enableInteractiveBranchAccess
        liftIO $ splitter <$> getContents
   where
@@ -92,8 +96,17 @@ batchLines fmt = do
                BatchLine -> lines
                BatchNull -> splitc '\0'
 
+-- When concurrency is enabled at the command line, it is used in batch
+-- mode. But, if it's only set in git config, don't use it, because the
+-- program using batch mode may not expect interleaved output.
+checkBatchConcurrency :: Annex ()
+checkBatchConcurrency = Annex.getState Annex.concurrency >>= \case
+       ConcurrencyCmdLine _ -> noop
+       ConcurrencyGitConfig _ -> 
+               setConcurrency (ConcurrencyGitConfig (Concurrent 1))
+
 batchCommandAction :: CommandStart -> Annex ()
-batchCommandAction = void . callCommandAction . batchCommandStart
+batchCommandAction = commandAction . batchCommandStart
 
 -- The batch mode user expects to read a line of output, and it's up to the
 -- CommandStart to generate that output as it succeeds or fails to do its

diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index 84cbfb611d4fc34cc504db2758598c02529f2487..5240b3e46ce20836df5560861c10b73c7a47ce6d 100644 (file)
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -1014,6 +1014,8 @@ Like other git commands, git-annex is configured via `.git/config`.
 
   Setting this to "cpus" will run one job per CPU core.
 
+  When the `--batch` option is used, this configuration is ignored.
+
 * `annex.queuesize`
 
   git-annex builds a queue of git commands, in order to combine similar