adb: Avoid find failing with "Argument list too long"

The "+" argument only runs the command once, so is not safe to use. Using
";" instead would have been the simplest fix, but also the slowest.

Since my phone has an xargs that supports -0, I piped find to xargs
instead. Unsure how portable this will be, perhaps some android's don't
have xargs -0 or find -printf to send null terminated output.

The business with pipefail is necessary to make a failure of find cause the
import to fail. Probably this works on all androids, but if not, it will
probably just result in a failure of find being ignored. It would be
possible to make ignorefinderror just disable setting pipefail, but then
if some android has a shell that has pipefail enabled by default, ignorefinderror
would not work, so I kept the || true approach for that.

Sponsored-by: Max Thoursie on Patreon

diff --git a/CHANGELOG b/CHANGELOG
index d89ddbc6aaa62128825342bd1386fe8c3000997a..cd9311523feb74fa8dd73317418da420ba4b9600 100644 (file)
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,9 @@
+git-annex (10.20220128) UNRELEASED; urgency=medium
+
+  * adb: Avoid find failing with "Argument list too long"
+
+ -- Joey Hess <id@joeyh.name>  Mon, 31 Jan 2022 13:14:42 -0400
+
 git-annex (10.20220127) upstream; urgency=medium
 
   * New v10 repository version (with v9 as a stepping-stone to it).

diff --git a/Remote/Adb.hs b/Remote/Adb.hs
index 2001727b11dcfdec354aeeee60db80a4b54bc9e6..915ab44b6fb1d795af8cab822a1afbfb7e6ddf4b 100644 (file)
--- a/Remote/Adb.hs
+++ b/Remote/Adb.hs
@@ -307,11 +307,15 @@ listImportableContentsM serial adir c = adbfind >>= \case
                -- won't recurse into the directory
                , File $ fromAndroidPath adir ++ "/"
                , Param "-type", Param "f"
-               , Param "-exec", Param "stat"
-               , Param "-c", Param statformat
-               , Param "{}", Param "+"
+               , Param "-printf", Param "%p\\0"
                ] 
-               (if ignorefinderror then "|| true" else "")
+               (\s -> concat
+                       [ "set -o pipefail; "
+                       , s
+                       , "| xargs -0 stat -c " ++ shellEscape statformat ++ " --"
+                       , if ignorefinderror then " || true" else ""
+                       ]
+               )
 
        ignorefinderror = fromMaybe False (getRemoteConfigValue ignorefinderrorField c)
 
@@ -399,11 +403,11 @@ enumerateAdbConnected = checkAdbInPath [] $ liftIO $
 --
 -- Any stdout from the command is returned, separated into lines.
 adbShell :: AndroidSerial -> [CommandParam] -> Annex (Maybe [String])
-adbShell serial cmd = adbShell' serial cmd ""
+adbShell serial cmd = adbShell' serial cmd id
 
-adbShell' :: AndroidSerial -> [CommandParam] -> String -> Annex (Maybe [String])
-adbShell' serial cmd extra = adbShellRaw serial $
-       (unwords $ map shellEscape (toCommand cmd)) ++ extra
+adbShell' :: AndroidSerial -> [CommandParam] -> (String -> String) -> Annex (Maybe [String])
+adbShell' serial cmd f = adbShellRaw serial $
+       f (unwords $ map shellEscape (toCommand cmd))
 
 adbShellBool :: AndroidSerial -> [CommandParam] -> Annex Bool
 adbShellBool serial cmd =

diff --git a/doc/bugs/special_remote_adb_-_Argument_list_too_long.mdwn b/doc/bugs/special_remote_adb_-_Argument_list_too_long.mdwn
index a2ed5c874e74fec886051f8787246cf126ef4f5d..6ce054d537e4b55cf49803c19f3ef9b016fb393d 100644 (file)
--- a/doc/bugs/special_remote_adb_-_Argument_list_too_long.mdwn
+++ b/doc/bugs/special_remote_adb_-_Argument_list_too_long.mdwn
@@ -52,3 +52,5 @@ local repository version: 8
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 
 This is the first time I've tried it. However, when I restricted the path to /sdcard/DCIM or /sdcard/Music, I didn't have any problems. I could use some assistance regarding how to "copy" a file from one path to another inside an annex. It appears that I could try to resolve and then symlink to the same file, or maybe unlock, copy, and commit both copies of the file. E.g. I'd like to have a copy of the music from my primary "annex/Music" into the "annex/android/Music" tree, then "git rm" the file from "annex/android/Music" when I'm done with it.
+
+> [[fixed|done]] --[[Joey]]

diff --git a/doc/bugs/special_remote_adb_-_Argument_list_too_long/comment_1_34f202f91a16dd4d2bb557aee6fd5a3e._comment b/doc/bugs/special_remote_adb_-_Argument_list_too_long/comment_1_34f202f91a16dd4d2bb557aee6fd5a3e._comment
new file mode 100644 (file)
index 0000000..d8d07e4
--- /dev/null
+++ b/doc/bugs/special_remote_adb_-_Argument_list_too_long/comment_1_34f202f91a16dd4d2bb557aee6fd5a3e._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2022-01-31T16:45:55Z"
+ content="""
+Ah so unfortunately android's `find` does not break the command
+runs by arrgument size like xargs does. So the `+` argument
+can't be used, instead I'll make it pipe through xargs.
+
+(To copy a file, you can use `cp -a` or just `cp`, followed by `git-annex
+add`. either will work.)
+"""]]