DATA-PRESENT working for exporttree=yes remotes

Since the annex-tracking-branch is pushed first, git-annex has already
updated the export database when the DATA-PRESENT arrives. Which means
that just using checkPresent is enough to verify that there is some file
on the special remote in the export location for the key.

So, the simplest possible implementation of this happened to work!

(I also tested it with chunked specialremotes, which also works, as long
as the chunk size used is the same as the configured chunk size. In that
case, the lack of a chunk log is not a problem. Doubtful this will ever
make sense to use with a chunked special remote though, that gets pretty
deep into re-implementing git-annex.)

Updated the client side upload tip with a missing step, and reorged for clarity.

diff --git a/Annex/Proxy.hs b/Annex/Proxy.hs
index 546c61dd0ca4c908dd6040649dc01f998a3347da..4f11f617c9f3f99dc0b5a0fdea6f453252b2f636 100644 (file)
--- a/Annex/Proxy.hs
+++ b/Annex/Proxy.hs
@@ -112,10 +112,7 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv oclosedv mexportdb = go
        go :: Annex ()
        go = liftIO receivemessage >>= \case
                Just (CHECKPRESENT k) -> do
-                       tryNonAsync (Remote.checkPresent r k) >>= \case
-                               Right True -> liftIO $ sendmessage SUCCESS
-                               Right False -> liftIO $ sendmessage FAILURE
-                               Left err -> liftIO $ propagateerror err
+                       checkpresent k
                        go
                Just (LOCKCONTENT _) -> do
                        -- Special remotes do not support locking content.
@@ -211,22 +208,14 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv oclosedv mexportdb = go
                                                        nuketmp
                                                        giveup "protocol error"
                                        else store >> nuketmp
-                       Just DATA_PRESENT -> tryNonAsync (verifydatapresent k) >>= \case
-                               Right True -> liftIO $ sendmessage SUCCESS
-                               Right False -> liftIO $ sendmessage FAILURE
-                               Left err -> liftIO $ propagateerror err
+                       Just DATA_PRESENT -> checkpresent k
                        _ -> giveup "protocol error"
 
-       verifydatapresent k = case mexportdb of
-               Just exportdb -> liftIO (Export.getExportTree exportdb k) >>= \case
-                       [] -> verifykey
-                       -- XXX TODO check that one of the export locs is populated,
-                       -- or for an annexobjects=yes special remote, the
-                       -- annexobject file could be populated.
-                       locs -> return True
-               Nothing -> verifykey
-         where
-               verifykey = Remote.checkPresent r k
+       checkpresent k = 
+               tryNonAsync (Remote.checkPresent r k) >>= \case
+                       Right True -> liftIO $ sendmessage SUCCESS
+                       Right False -> liftIO $ sendmessage FAILURE
+                       Left err -> liftIO $ propagateerror err
 
        storeput k af tmpfile = case mexportdb of
                Just exportdb -> liftIO (Export.getExportTree exportdb k) >>= \case

diff --git a/doc/tips/client_side_upload_to_a_special_remote.mdwn b/doc/tips/client_side_upload_to_a_special_remote.mdwn
index 0982850d5a939591d5436fc2550f07c120a7d4b2..25df9bf30e3b5670cece660d7c01f20d3ccfa9c4 100644 (file)
--- a/doc/tips/client_side_upload_to_a_special_remote.mdwn
+++ b/doc/tips/client_side_upload_to_a_special_remote.mdwn
@@ -21,38 +21,45 @@ and then you can upload whatever filenames you want to it, rather than
 needing to use the same filenames git-annex uses for storing keys in a S3
 bucket.
 
+Along with the S3 bucket, you will need a server set up, which is where
+git-annex will run in a git repository. Set up the S3 special remote there.
+And make git-annex on the server a
+[proxy|git-annex-updateproxy]] for the S3 special remote:
+
+    git-annex initremote s3 type=S3 exporttree=yes encryption=none bucket=mybucket
+    git config remote.s3.annex-proxy true
+    git-annex updateproxy
+
+If the special remote is configured with exporttree=yes, be sure to also
+configure the annex-tracking-branch for it on the server:
+
+    git config remote.s3.annex-tracking-branch master
+
 Once the browser uploads the file to S3, you need to add a git-annex
 symlink or pointer file to the git repository. This can be done in the
 browser, using [js-git](https://github.com/creationix/js-git). Generating a
 git-annex key is not hard, just hash the file content before/while
 uploading it, and see [[internals/key_format]]. Write that to a pointer
 file, or make a symlink to the appropriate directory under
-.git/annex/objects (a bit harder). Commit it to git and push to your
-server using js-git.
-
-Now git-annex knows about the file. But it doesn't yet know it's been
-uploaded to the S3 special remote. To do this, you will need have your
-server set up to run git-annex. Set up the S3 special
-remote there. And make git-annex on the server a
-[proxy|git-annex-updateproxy]] for the S3 special remote:
-
-    git-annex initremote s3 type=S3 exporttree=yes encryption=none bucket=mybucket
-    git config remote.s3.annex-proxy true
-    git-annex updateproxy
+.git/annex/objects (a bit harder). Commit it to git and push the branch
+("master" in this example) to your server using js-git.
 
-For the web browser to be able to easily talk with git-annex on the server, 
-you can run [[git-annex p2phttp|git-annex-p2phttp]]. 
+All that's left is to let git-annex know that the file has been uploaded to
+the S3 special remote. To accomplish this, the web browser will need to
+talk with git-annex on the server. The easy way to accomplish that
+is to run [[git-annex p2phttp|git-annex-p2phttp]]. 
 The web browser will be speaking the [[doc/design/P2P_protocol_over_HTTP]].
 
 Make sure you have git-annex 10.20241031 or newer installed. That version
 extended the [[design/p2p_protocol]] with a `DATA-PRESENT` feature, which
 is just what you need.
 
-All the web browser needs to do is `POST /git-annex/$uuid/v4/put`
-with `data-present=true` included in the URL parameters, along with the
-key of the file that was added to the git repository.
-Replace `$uuid` with the UUID of the S3 special remote.
-You can look that up with eg `git config remote.s3.annex-uuid`.
+All the web browser needs to do, after uploading the S3 and pushing the git
+branch to the server, is `POST /git-annex/$uuid/v4/put` with
+`data-present=true` included in the URL parameters, along with the key of
+the file that was added to the git repository. Replace `$uuid` with the
+UUID of the S3 special remote. You can look that up with eg `git config
+remote.s3.annex-uuid`.
 
 When the git-annex HTTP server receives that request, since it is
 configured to be able to proxy for the S3 special remote, it will act the

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 6fa103e933f5e20b3fa258d123b3c1cb1c800018..bb38166c19dfb522ab7bbe4347028cf2af351607 100644 (file)
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -28,22 +28,6 @@ Planned schedule of work:
 
 ## remaining things to do in October
 
-* Streaming uploads to special remotes via the proxy. Possibly; if a
-  workable design can be developed. It seems difficult without changing the
-  external special remote protocol, unless a fifo is used. Make ORDERED
-  response in p2p protocol allow using a fifo?
-
-* Indirect uploads when proxying for special remote is an alternative that
-  would work for OpenNeuro's use case.
-
-* If not implementing upload streaming to proxied special remotes, 
-  this needs to be addressed:
-  When an upload to a cluster is distributed to multiple special remotes,
-  a temporary file is written for each one, which may even happen in
-  parallel. This is a lot of extra work and may use excess disk space.
-  It should be possible to only write a single temp file.
-  (With streaming this wouldn't be an issue.)
-
 * Possibly some of the deferred items listed in following sections:
 
 ## items deferred until later for balanced preferred content and maxsize tracking
@@ -123,6 +107,12 @@ Planned schedule of work:
 * Support using a proxy when its url is a P2P address.
   (Eg tor-annex remotes.)
 
+* When an upload to a cluster is distributed to multiple special remotes,
+  a temporary file is written for each one, which may even happen in
+  parallel. This is a lot of extra work and may use excess disk space.
+  It should be possible to only write a single temp file.
+  (With streaming this wouldn't be an issue.)
+
 ## completed items for October's work on streaming through proxy to special remotes
 
 * Stream downloads through proxy for all special remotes that indicate