Joey Hess [Wed, 30 Oct 2024 17:09:50 +0000 (13:09 -0400)]
fix recording present on the proxy when proxying DATA-PRESENT
Apparently the protoerrhandler parameter never runs. Also the const
typo prevented the type checker from complaining that relayPUTRecord was
being called with 1 less parameter than needed.
Joey Hess [Wed, 30 Oct 2024 16:29:37 +0000 (12:29 -0400)]
Fix hang when receiving a large file into a proxied special remote
Only indicate that we're done with the bytestring once it all gets written.
Otherwise, the end of it may get garbage collected before we can process
it, leading to a hang.
This seems to have been introduced in commit cdc4bd7443a4921dd0573a0babedc51c1d62f918. Which oddly was trying to fix a
very similar problem, but specific to a cluster node. In that commit,
things got out of order, with it signaling it was done with the bytestring
before it has written all of it to the file.
My test case for this bug is a directory special remote
with a file being sent to it via a proxy accessed via ssh or http.
The file was 10 mb, and it hung on the last few kb of it not being
received.
I've also tested this fix in the case of proxying to a cluster node
directory special remote over http, which was the case cdc4bd7443a4921dd0573a0babedc51c1d62f918 was dealing with.
Joey Hess [Tue, 29 Oct 2024 19:07:30 +0000 (15:07 -0400)]
document DATA-PRESENT in CHANGELOG
I wonder where else this could be documented? It's kind of a niche
feature, since it needs at least a partial custom implementation of the p2p
protocol or the p2phttp protocol. But it can save a lot of bandwidth and
avoid the proxy needing disk space to buffer files uploaded to a special
remote.
Joey Hess [Tue, 29 Oct 2024 18:53:06 +0000 (14:53 -0400)]
support DATA-PRESENT when proxtying for special remotes
There is a TODO left in the code for exporttree special remotes. If
possible, it should check if one of the export locations contains the
content of the key.
Joey Hess [Tue, 29 Oct 2024 18:20:07 +0000 (14:20 -0400)]
support DATA-PRESENT in the p2p protocol proxy
But not yet when proxying to special remotes.
When proxying for a cluster, the client can store the object on any node
or nodes of the cluster, and send DATA-PRESENT. That gets proxied to each
node, and if any of them agree that they have the data, the proxy will
respond with SUCCESS or SUCCESS-PLUS.
I think it's ok to not check for the proxied remotes supporting protocol
version 4. When there are multiple remotes in a cluster, it behaves as
described above, and if they all respond with ERROR, the result will be
FAILURE. And when not proxying for a cluster, the proxy negotiates the
p2p protocol to be the same version or lower than the proxied remote,
which will prevent sending DATA-PRESENT when it's too old.
Joey Hess [Tue, 29 Oct 2024 17:13:28 +0000 (13:13 -0400)]
implement put data-present parameter in http servant
Changed the protocol docs because servant parses "true" and "false" for
booleans in query parameters, not "1" and "0".
clientPut with datapresent=True is not used by git-annex, and I don't
anticipate it being used in git-annex, except for testing.
I've tested this by making clientPut be called with datapresent=True and
git-annex copy to a remote succeeds once the object file is first
manually copied to the remote. That would be a good test for the test
suite, but running the http client means exposing it to at least
localhost, and would fail if a real http client was already running on
that port.
Joey Hess [Mon, 21 Oct 2024 20:03:29 +0000 (16:03 -0400)]
adjust: Allow any order of options when combining --hide-missing with options like --unlock.
optparse-applicative made this hard, the naive implementation this had
before didn't let --hide-missing come after --unlock. And just adding
additional <|> with --hide-missing coming after --unlock didn't work
either. So need to get some options and then combine them.
Joey Hess [Mon, 21 Oct 2024 14:02:12 +0000 (10:02 -0400)]
p2phttp: Allow unauthenticated users to lock content by default
* p2phttp: Allow unauthenticated users to lock content by default.
* p2phttp: Added --unauth-nolocking option to prevent unauthenticated
users from locking content.
The rationalle for this is that locking is not really a write operation, so
makes sense to allow in a repository that only allows read-only access. Not
supporting locking in that situation will prevent the user from dropping
content from a special remote they control in cases where the other copy of
the content is on the p2phttp server.
Also, when p2phttp is configured to also allow authenticated access,
lockcontent was resulting in a password prompt for users who had no way to
authenticate. And there is no good way to distinguish between the two types
of users client side.
--unauth-nolocking anticipates that this might be abused, and seems better
than disabling unauthenticated access entirely if a server is being
attacked. It may be that rate limiting locking by IP address or similar
would be an effective measure in such a situation. Or just limiting the
number of locks by anonymous users that can be live at any one time. Since
the impact of such an DOS attempt is limited to preventing dropping content
from the server, it seems not a very appealing target anyway.
Joey Hess [Thu, 17 Oct 2024 15:10:28 +0000 (11:10 -0400)]
p2phttp combining unauth and auth options
p2phttp: Support serving unauthenticated users while requesting
authentication for operations that need it. Eg, --unauth-readonly can be
combined with --authenv.
Drop locking currently needs authentication so it will prompt for that.
That still needs to be addressed somehow.
Joey Hess [Tue, 15 Oct 2024 19:35:09 +0000 (15:35 -0400)]
added retrieveKeyFileInOrder and ORDERED to external special remote protocol
I anticipate lots of external special remote programs will neglect
implementing this. Still, it's the right thing to do to assume that some
of them may write files out of order. Probably most external special
remotes will not be used with a proxy. When someone is using one with a
proxy, they can always get it fixed to send ORDERED.
Joey Hess [Tue, 15 Oct 2024 18:38:20 +0000 (14:38 -0400)]
smaller delay in proxy streamer
A one second delay made it seem really choppy and slow when the special
remote was sending content fairly steadily but was bottlenecked on
running gpg on 10 mb chunks.
This does not appreciably increase CPU, although of course if the
special remote is very slow it will add up over time.
It would perhaps be better to use inotify, like tailVerify does.
Joey Hess [Tue, 15 Oct 2024 18:29:06 +0000 (14:29 -0400)]
stream through proxy when using fileRetriever
The problem was that when the proxy requests a key be retrieved to its
own temp file, fileRetriever was retriving it to the key's temp
location, and then moving it at the end, which broke streaming.
So, plumb through the path where the key is being retrieved to.
Joey Hess [Tue, 15 Oct 2024 16:22:34 +0000 (12:22 -0400)]
Sped up proxied downloads from special remotes, by streaming
Currently works for special remotes that don't use fileRetriever. Ones that
do will download to another filename and rename it into place, defeating
the streaming.
This actually benchmarks slightly slower when getting a large file from
a fast proxied special remote. However, when the proxied special remote
is slow, it will be a big win.
Joey Hess [Tue, 15 Oct 2024 15:56:42 +0000 (11:56 -0400)]
implement openFileBeingWritten
This bypasses the usual haskell file locking used to prevent opening a
file for read that is being written to.
This is unfortunately a bit of a hack. But it seems fairly unlikely to
get broken by changes to ghc. I hope. Using fdToHandle' will also work.
This does not work on windows because it uses openFd from posix. It
would probably be possible to implement it for windows too, just opening
the FD using the Win32 library instead. However, whether windows will
allow reading from a file that is also being written to I don't know,
and since in the git-annex case the writer could be another process (eg
external special remote), that might be doing its own locking in
windows, that seems a can of worms I'd prefer not to open.
projects / git-annex.git / log