Joey Hess [Sat, 17 Dec 2016 21:28:08 +0000 (17:28 -0400)]
use PYTHONUNBUFFERED to force python to use sane stdout buffering
Works around https://github.com/warner/magic-wormhole/issues/108
See http://stackoverflow.com/questions/107705/disable-output-buffering
for the gory details. Why a scripting language would chose a default
stdout buffering that differs between terminal and piped output, and
tends to introduce this kind of bug, I don't know.
Joey Hess [Sat, 17 Dec 2016 20:58:05 +0000 (16:58 -0400)]
magic wormhole module
This interacts with it using stdio, which is surprisingly hard.
sendFile does not currently work, due to
https://github.com/warner/magic-wormhole/issues/108
Parsing the output to find the magic code is done as robustly as
possible, and should continue to work unless wormhole radically changes
the format of its codes. Presumably it will never output something that
looks like a wormhole code before the actual wormhole code; that would
also break this. It would be better if there was a way to make
wormhole not mix the code with other output, as requested in
https://github.com/warner/magic-wormhole/issues/104
Only exchange of files/directories is supported. To exchange messages,
https://github.com/warner/magic-wormhole/issues/99 would need to be resolved.
I don't need message exchange however.
On second thought, this was an overcomplication of what should be the
lowest-level primitive. Let's build bi-directional links at the pairing
level with eg magic wormhole.
Joey Hess [Fri, 16 Dec 2016 20:32:29 +0000 (16:32 -0400)]
p2p --link now defaults to setting up a bi-directional link
Both the local and remote git repositories get remotes added
pointing at one-another.
Makes pairing twice as easy!
Security: The new LINK command in the protocol can be sent repeatedly,
but only by a peer who has authenticated with us. So, it's entirely safe to
add a link back to that peer, or to some other peer it knows about.
Anything we receive over such a link, the peer could send us over the
current connection.
There is some risk of being flooded with LINKs, and adding too many
remotes. To guard against that, there's a hard cap on the number of remotes
that can be set up this way. This will only be a problem if setting up
large p2p networks that have exceptional interconnectedness.
A new, dedicated authtoken is created when sending LINK.
This also allows, in theory, using a p2p network like tor, to learn about
links on other networks, like telehash.
This commit was sponsored by Bruno BEAUFILS on Patreon.
Joey Hess [Tue, 13 Dec 2016 15:56:23 +0000 (11:56 -0400)]
Pass annex.web-options to wget and curl after other options, so that eg --no-show-progress can be set by the user to disable the default --show-progress.
Joey Hess [Tue, 13 Dec 2016 15:07:49 +0000 (11:07 -0400)]
metadata --batch: Fix bug when conflicting metadata changes were made in the same batch run.
1 microsecond delay is ugly.. but, maintaining an queue of a list of timestamps
and taking a new one from the queue each time around, or maintaining a timestamp
counter, would probably be slower.
Since https://github.com/aristidb/aws/issues/206 is resolved, this
constraint is no longer necessary. However, http-conduit (>=2.2.0)
requires http-client (>=0.5.0) which introduces some breaking changes.
This commit also implements those changes depending on the version. Fixes: https://git-annex.branchable.com/bugs/Build_with_aws_head_fails/ Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Joey Hess [Fri, 9 Dec 2016 20:03:25 +0000 (16:03 -0400)]
bump maxConnections to 100
10 seemed too low because more than 10 friends could be linked to a repo
over tor, and if all were running the remotedaemon, which makes a
persistent connection for change notification, then the 11th friend
would not be able to access that repo.
100 might be too low, but it's a much larger group of people. And at
that size group, it probably makes sense to structure the network so
that 100 peers are not all trying to access one central node.
Joey Hess [Fri, 9 Dec 2016 19:08:54 +0000 (15:08 -0400)]
only start ref change watcher thread once per P2P connection
This is more efficient. Note that the peer will get CHANGED messages for
all refs changed since the connection opened, even if those changes
happened before it sent NOTIFYCHANGE.
Joey Hess [Fri, 9 Dec 2016 18:52:38 +0000 (14:52 -0400)]
refactor ref change watching
Added to change notification to P2P protocol.
Switched to a TBChan so that a single long-running thread can be
started, and serve perhaps intermittent requests for change
notifications, without buffering all changes in memory.
The P2P runner currently starts up a new thread each times it waits
for a change, but that should allow later reusing a thread. Although
each connection from a peer will still need a new watcher thread to run.
The dependency on stm-chans is more or less free; some stuff in yesod
uses it, so it was already indirectly pulled in when building with the
webapp.
This commit was sponsored by Francois Marier on Patreon.
Joey Hess [Fri, 9 Dec 2016 17:34:00 +0000 (13:34 -0400)]
git-annex-shell, remotedaemon, git remote: Fix some memory DOS attacks.
The attacker could just send a very lot of data, with no \n and it would
all be buffered in memory until the kernel killed git-annex or perhaps OOM
killed some other more valuable process.
This is a low impact security hole, only affecting communication between
local git-annex and git-annex-shell on the remote system. (With either
able to be the attacker). Only those with the right ssh key can do it. And,
there are probably lots of ways to construct git repositories that make git
use a lot of memory in various ways, which would have similar impact as
this attack.
The fix in P2P/IO.hs would have been higher impact, if it had made it to a
released version, since it would have allowed DOSing the tor hidden
service without needing to authenticate.
(The LockContent and NotifyChanges instances may not be really
exploitable; since the line is read and ignored, it probably gets read
lazily and does not end up staying buffered in memory.)
projects / git-annex.git / log