checkpresent support for clusters

This assumes that the proxy for a cluster has up-to-date location
logs. If it didn't, it might proxy the checkpresent to a node that no
longer has the content, while some other node still does, and so
it would incorrectly appear that the cluster no longer contains the
content.

Since cluster UUIDs are not stored to location logs,
git-annex fsck --fast when claiming to fix a location log when
that occurred would not cause any problems. And presumably the location
tracking would later get sorted out.

At least usually, changes to the content of nodes goes via the proxy,
and it will update its location logs, so they will be accurate. However,
if there were multiple proxies to the same cluster, or nodes were
accessed directly (or via proxy to the node and not the cluster),
the proxy's location log could certainly be wrong.

(The location log access for GET has the same issues.)

diff --git a/Annex/Cluster.hs b/Annex/Cluster.hs
index 5e9f53a5d995cc76c226072e67bedaaa8f90806b..b832675aa002a6a35bfa0455ea46a5c915221496 100644 (file)
--- a/Annex/Cluster.hs
+++ b/Annex/Cluster.hs
@@ -60,13 +60,8 @@ clusterProxySelector clusteruuid protocolversion = do
                <$> remoteList
        remotesides <- mapM (proxySshRemoteSide protocolversion) remotes
        return $ ProxySelector
-               { proxyCHECKPRESENT = \k -> error "TODO"
-               , proxyGET = \k -> do
-                       locs <- S.fromList <$> loggedLocations k
-                       case filter (flip S.member locs . remoteUUID) remotesides of
-                               -- TODO: Avoid always using same remote
-                               (r:_) -> return (Just r)
-                               [] -> return Nothing
+               { proxyCHECKPRESENT = nodecontaining remotesides
+               , proxyGET = nodecontaining remotesides
                , proxyPUT = \k -> error "TODO"
                , proxyREMOVE = \k -> error "TODO"
                -- Content is not locked on the cluster as a whole,
@@ -75,3 +70,11 @@ clusterProxySelector clusteruuid protocolversion = do
                , proxyLOCKCONTENT = const (pure Nothing)
                , proxyUNLOCKCONTENT = pure Nothing
                }
+  where
+       nodecontaining remotesides k = do
+               locs <- S.fromList <$> loggedLocations k
+               case filter (flip S.member locs . remoteUUID) remotesides of
+                       -- TODO: Avoid always using same remote
+                       (r:_) -> return (Just r)
+                       [] -> return Nothing
+               

diff --git a/P2P/Proxy.hs b/P2P/Proxy.hs
index 89a7199f5f60564f4d66a8bf1e00ab45f5c63721..3e4789da810e90c39a37c733e327d9cca65cc629 100644 (file)
--- a/P2P/Proxy.hs
+++ b/P2P/Proxy.hs
@@ -54,7 +54,7 @@ closeRemoteSide remoteside =
  - actions.
  - -}
 data ProxySelector = ProxySelector
-       { proxyCHECKPRESENT :: Key -> Annex RemoteSide
+       { proxyCHECKPRESENT :: Key -> Annex (Maybe RemoteSide)
        , proxyLOCKCONTENT :: Key -> Annex (Maybe RemoteSide)
        , proxyUNLOCKCONTENT :: Annex (Maybe RemoteSide)
        , proxyREMOVE :: Key -> Annex RemoteSide
@@ -64,7 +64,7 @@ data ProxySelector = ProxySelector
 
 singleProxySelector :: RemoteSide -> ProxySelector
 singleProxySelector r = ProxySelector
-       { proxyCHECKPRESENT = const (pure r)
+       { proxyCHECKPRESENT = const (pure (Just r))
        , proxyLOCKCONTENT = const (pure (Just r))
        , proxyUNLOCKCONTENT = pure (Just r)
        , proxyREMOVE = const (pure r)
@@ -160,9 +160,13 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) prox
 
        proxyclientmessage Nothing = proxydone
        proxyclientmessage (Just message) = case message of
-               CHECKPRESENT k -> do
-                       remoteside <- proxyCHECKPRESENT proxyselector k
-                       proxyresponse remoteside message (const proxynextclientmessage)
+               CHECKPRESENT k -> proxyCHECKPRESENT proxyselector k >>= \case
+                       Just remoteside -> 
+                               proxyresponse remoteside message
+                                       (const proxynextclientmessage)
+                       Nothing -> 
+                               protoerrhandler proxynextclientmessage $
+                                       client $ net $ sendMessage FAILURE
                LOCKCONTENT k -> proxyLOCKCONTENT proxyselector k >>= \case
                        Just remoteside -> 
                                proxyresponse remoteside message 

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index e88f2fa2e0d34e0435a2e4f1c4f0473a40cd26af..e33374f0d8783a8d71d7c66873e38f7ab9df1257 100644 (file)
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -55,7 +55,7 @@ For June's work on [[design/passthrough_proxy]], implementation plan:
 * Basic proxying to special remote support (non-streaming).
 
 * Getting a key from a cluster should proxy from one of the nodes that has
-  it, or from the proxy repository itself if it has the key.
+  it. (done)
 
 * Getting a key from a cluster currently always selects the lowest cost
   remote, and always the same remote if cost is the same. Should
@@ -65,10 +65,6 @@ For June's work on [[design/passthrough_proxy]], implementation plan:
 * Implement upload with fanout and reporting back additional UUIDs over P2P
   protocol.
 
-* On upload to a cluster, as well as fanout to nodes, if the key is
-  preferred content of the proxy repository, store it there.
-  (But not when preferred content is not configured.)
-
 * Implement cluster drops, trying to remove from all nodes, and returning
   which UUIDs it was dropped from. 
 
@@ -85,6 +81,12 @@ For June's work on [[design/passthrough_proxy]], implementation plan:
   check may fail to realize that dropping from multiple nodes does in fact
   make it worse.
 
+* On upload to a cluster, as well as fanout to nodes, if the key is
+  preferred content of the proxy repository, store it there.
+  (But not when preferred content is not configured.)
+  And on download from a cluster, if the proxy repository has the content,
+  get it from there to avoid the overhead of proxying to a node.
+
 * Support proxies-of-proxies better, eg foo-bar-baz.
   Currently, it does work, but have to run `git-annex updateproxy`
   on foo in order for it to notice the bar-baz proxied remote exists,