design work

diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_19_980f4a04543166677ef41fcefd57da03._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_19_980f4a04543166677ef41fcefd57da03._comment
index 90d2ccec5475fa469bcbf642945ac84c5ff892d0..11abd5156d8c5f78a400eb98079faaf8aba3565c 100644 (file)
--- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_19_980f4a04543166677ef41fcefd57da03._comment
+++ b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_19_980f4a04543166677ef41fcefd57da03._comment
@@ -30,5 +30,6 @@ Also, all parts of Messages that remotes use will need to be serialized
 over the transferkeys protocol. Eg, Messages.prompt when the transferkeys
 process is doing something like ssh that is prompting for a password.
 So transferkeys will need its own OutputType to make Messages communicate
-over the pipe.
+over the pipe. (This is a hard thing, so added a todo for it:
+[[todo/serialize_Messages_for_transferkeys]]
 """]]

diff --git a/doc/todo/serialize_Messages_for_transferkeys.mdwn b/doc/todo/serialize_Messages_for_transferkeys.mdwn
new file mode 100644 (file)
index 0000000..0f8f9fb
--- /dev/null
+++ b/doc/todo/serialize_Messages_for_transferkeys.mdwn
@@ -0,0 +1,50 @@
+For [[todo/more_extensive_retries_to_mask_transient_failures]],
+multiple transferkeys processes need to be run. So all console IO needs to
+be serialized and sent back to the main git-annex process from them,
+to avoid concurrent output.
+
+Using --json and --json-progress and --json-error-messages is pretty close
+to what would be needed. That handles progress of transfers, and overall
+success/failure. 
+
+Using --json-error-messages makes most error messages be output in json.
+However, that uses an error-messages array in the json object,
+which error messages are added to as an action runs, and it's only sent at
+the end. So a problem for eg, a relay of stderr output from a command
+(eg ssh password prompt), or for anything that displays a warning that
+should be displayed before the transfer is complete.
+
+Since the existing json is not a perfect fit, it might be better to use a
+custom protocol, implemented as a new output type. Then anything in
+Messages and child modules that looks at output types will support it.
+(Although perhaps some of that stuff is not used by remotes.)
+
+A few notes on implementing that:
+
+* Messages.Progress.mkOutputHandler, which uses mkStderrEmitter,
+  outputs to stderr directly no matter the output type currently.
+  It would need to be changed to support the new output type.
+  (And probably should for concurrent output mode too actually!)
+* So does warningIO, though it's only used in a couple of remotes
+  and rarely. It would be good to find a way to eliminate it.
+
+* Messages.prompt. Which is used by remotes, and would need to 
+  communicate over the pipe to the parent git-annex bidirectionally.
+  Eg, send a message saying the parent needs to prepare for prompt,
+  wait for it to reply saying it has, and then send a message when the
+  prompting is done. (Note that the parent would need to detect if the child
+  process crashed to avoid being locked waiting for the prompt.)
+
+* Messages.Internal.outputMessage is used by several things, and 
+  includes some special parameters used in json mode. Since the parent
+  git-annex might itself have json mode enabled, those parameters will need
+  to be included in the serialization. But those parameters are currently
+  actually functions that manipulate the json object that will be outputted
+  later. So cannot be serialized. Uuuuh.
+
+  Maybe the thing to do is, pass along the --json options to transferkeys,
+  and have a message type for json objects, which it uses to send them
+  to git-annex, which can then output them. outputMessage can handle the
+  new output type by sending the message through the pipe, and also
+  building any json object, and sending it through the pipe once it's done.
+