From: Joey Hess <joeyh@joeyh.name>
Date: Mon, 2 Jun 2025 18:25:40 +0000 (-0400)
Subject: followup
X-Git-Tag: archive/raspbian/10.20251029-1+rpi1~1^2~7^2~107
X-Git-Url: https://dgit.raspbian.org/?a=commitdiff_plain;h=8166f7925086701213c4d20aaecb010545361023;p=git-annex.git

followup
---

diff --git a/doc/bugs/compute_remote_fails_for_unicode_filenames/comment_1_2da72486f0e83e74871706757b0badb6._comment b/doc/bugs/compute_remote_fails_for_unicode_filenames/comment_1_2da72486f0e83e74871706757b0badb6._comment
new file mode 100644
index 0000000000..e2e06b3473
--- /dev/null
+++ b/doc/bugs/compute_remote_fails_for_unicode_filenames/comment_1_2da72486f0e83e74871706757b0badb6._comment
@@ -0,0 +1,43 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2025-06-02T17:21:37Z"
+ content="""
+> I'm running on Linux and my locale is de_DE.UTF-8:
+> 
+> git-annex was installed using Homebrew.
+
+That's unusual. Linux and Homebrew? I just want to check you didn't
+typo there and mean to say you're on OSX. 
+
+Tried just now (including the same locale setting) and it does not fail for me:
+
+	joey@darkstar:~/tmp/c>git annex addcomputed --to=passthrough "Ä filename with Unicöde chäracters.txt" fails.txt
+	addcomputed passthrough
+	(adding fails.txt...) (checksum...)
+	ok
+	(recording state in git...)
+
+There are 3 possibilities here:
+
+1. The unicode characters are getting stripped out before git-annex is run,
+   eg by your interactive shell or by git.
+2. git-annex is stripping out valid (or invalid) unicode.
+3. "read" or "echo" in your git-annex-compute-passthrough script is
+   for some reason stripping unicode
+
+The best way to track down which of these is the problem is `strace`, so could you please try this:
+
+	strace -o log -f git annex addcomputed --to=passthrough "Ä filename with Unicöde chäracters.txt" foo.txt
+	grep "filename with" log
+
+Here's how that strace looks for me, when the characters are making it through unscathed:
+
+	2395608 execve("/usr/bin/git", ["git", "annex", "addcomputed", "--to=passthrough", "\303\204 filename with Unic\303\266de ch\303\244ra"..., "fails3.txt"], 0x7ffc44897f00 /* 69 vars */) = 0
+	2395609 execve("/home/joey/bin/git-annex", ["/home/joey/bin/git-annex", "addcomputed", "--to=passthrough", "\303\204 filename with Unic\303\266de ch\303\244ra"..., "fails3.txt"], 0x55c3cfdf27c0 /* 70 vars */ <unfinished ...>
+	2395618 execve("/home/joey/bin/git-annex-compute-passthrough", ["git-annex-compute-passthrough", "\303\204 filename with Unic\303\266de ch\303\244ra"..., "fails3.txt"], 0x42000ec610 /* 70 vars */ <unfinished ...>
+	2395618 write(1, "INPUT \303\204 filename with Unic\303\266de "..., 48) = 48
+	2395609 read(16, "INPUT \303\204 filename with Unic\303\266de "..., 8192) = 48
+
+(I commented out the passthrough.log writing from the script to keep the strace easier to follow.)
+"""]]
diff --git a/doc/bugs/compute_remote_fails_for_unicode_filenames/comment_2_a51f8ba4f0b24c21e107bc33db2412ab._comment b/doc/bugs/compute_remote_fails_for_unicode_filenames/comment_2_a51f8ba4f0b24c21e107bc33db2412ab._comment
new file mode 100644
index 0000000000..b4627aeac7
--- /dev/null
+++ b/doc/bugs/compute_remote_fails_for_unicode_filenames/comment_2_a51f8ba4f0b24c21e107bc33db2412ab._comment
@@ -0,0 +1,27 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2025-06-02T17:54:05Z"
+ content="""
+I don't see how git-annex could be stripping even invalid unicode here.
+When it runs the compute program it uses `process` with `CreatePipe`. That
+is documented to use the default encoding. git-annex sets the default
+encoding in `useFileSystemEncoding`.
+
+With that said, git-annex is here using `hGetLineUntilExitOrEOF`, and if
+`hGetChar` ever failed with an encoding error, it does look like that
+would skip over the problem and return the rest of the string.
+
+It would not hurt to throw in a `fileEncoding` on the compute process's
+handles, but I'd really want to be able to reproduce this first.
+
+I have also tried with filenames that are not valid unicode at all, and
+they pass through ok. Eg:
+
+	invalid_byte_sequence=$'\x80\x81'
+	echo hi > invalid$(printf %s $invalid_byte_sequence)
+	git-annex add invalid*
+	git annex addcomputed --to=passthrough invalid* invalidout
+	cat invalidout
+	hi
+"""]]