Foolscap

Connection Handlers for SOCKS, Tor, I2P

Foolscap now includes importable connection handlers for SOCKS(5a), Tor, and
I2P. 242, 246, 261

These handlers require additional supporting libraries, so they must be
imported separately, and a setuptools "extra feature" declaration must be
used to ask for the supporting libs. For example, applications which want to
use `tor:` hints (on a host with a Tor daemon running) should have a setup.py
with:

install_requires=["foolscap[tor]"],

and the Tub setup code should do:

from foolscap.connections import tor
tub.addConnectionHintHandler("tor", tor.default_socks())

Full examples and docs are available in docs/connection-handlers.rst.

The default connection-negotiation timeout was increased from 60s to 120s, to
accomodate tor/i2p daemon startup times.

API changes: no more automatic configuration

Foolscap has moved from automatic listener configuration (randomly-allocated
TCP ports, automatically-determined IP address) to using more predictable
manual configuration. In our experience, the automatic configuration only
worked on hosts which had external IP addresses, which (sadly) is not the
case for most computers attached to the modern internet. 252

Applications must now explicitly provide Foolscap with port numbers (for
Tub.listenOn) and hostnames (for Tub.setLocation). Applications are
encouraged to give users configuration controls to teach Foolscap what
hostname and port number it should advertise to external hosts in the FURLs
it creates. See https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2773 for ideas.

The specific API changes were:

- Tub.setLocationAutomatically() has been deprecated
- Listener.getPortnum() has been deprecated
- calling Tub.listenOn("tcp:0") is also deprecated: callers should allocate a
port themselves (the foolscap.util.allocate_tcp_port utility function,
which does not block, has been added for this purpose).

Foolscap tools like "flappserver create" and "flogtool create-gatherer" will
no longer try to deduce their external IP address in an attempt to build
externally-reachable FURLs, and will no longer accept "tcp:0" as a listening
port (they now default to specific port numbers). Instead, they have
--location= and --port arguments. The user must provide '--location' with a
connection-hint string like 'tcp:hostname.example.org:3117' (which is put
into the server's FURLs). This must match the corresponding '--port'
argument, if provided.

- for all tools, if '--port' is provided, it must not be tcp:0
- 'flappserver create' now requires --location, and '--port' defaults to
tcp:3116
- 'flogtool create-gatherer' requires --location, default port is tcp:3117
- 'flogtool create-incident-gatherer' does too, default is tcp:3118

For backwards-compatibility, old flappservers will have "tcp:0" written into
their "BASEDIR/port" file, and an empty string in "BASEDIR/location": these
must then be edited to allow the flappserver to start. For example, write
"tcp:12345" into "BASEDIR/port" to assign a portnumber, and
"tcp:HOSTNAME:12345" into "BASEDIR/location" to expose it in the generated
FURL.

Other API changes

Tub.listenOn() now takes a string or an Endpoint (something that implements
twisted.internet.interfaces.IStreamServerEndpoint). This makes it possible to
listen on non-IPv4 sockets (e.g. IPv6-only sockets, or unix-domain sockets,
or more exotic endpoints), as long as Tub.setLocation() is set to something
which the other end's connection handlers can deal with. 203 243

The "DefaultTCP" handler (which manages normal "tcp:HOST:PORT" connection
hints) has been moved to foolscap.connections.tcp . This makes room for new
Tor/I2P/SOCKS handlers to live in e.g. foolscap.connections.tor . 260

Connection handlers are now allowed to return a Deferred from
hint_to_endpoint(), which should make some handlers easier to write. 262

Note that RemoteReference.notifyOnDisconnect() will be deprecated in the next
release (once all internal uses have been removed from Foolscap).
Applications should stop using it as soon as possible. 42 140 207

Compatibility Changes

This release removes support for the old (py2.4) "sets" module. This was
retained to support applications which were trying to maintain py2.4
compatibility, but it's been so long since this was necessary, it's time to
remove it.

Other Changes

The internal `allocate_tcp_port()` function was fixed: unexpected kernel
behavior meant that sometimes it would return a port that was actually in
use. This caused unit tests to fail randomly about 5% of the time. 258

IPv6 support is nearly complete: listening on a plain TCP port will typically
accept connections via both IPv4 and IPv6, and the DefaultTCP handler will do
a hostname lookup that can use both A and AAAA records. So as long as your
server has a DNS entry that points at its IPv6 address, and you provide the
hostname to Tub.setLocation(), Foolscap will connect over IPv6. There is one
piece missing for complete support: the DefaultTCP connection handler must be
modified to accept square-bracketed numeric IPv6 addresses, for rare
situations where the host has a known (stable) IPv6 address, but no DNS name.

Packaging Fixes

Foolscap now declares a dependency on "twisted[tls]" instead of just
"twisted": the "[tls]" extra means "we need Twisted and its TLS support".
That's how we ask for Twisted to depend upon service_identity and other
supporting packages. By using "[tls]", we no longer need to manually depend
upon service_identity ourselves. If Twisted switches to some other scheme for
TLS support, this will correctly ask for that to be included. (249)

Note that we still depend on pyOpenSSL ourselves, because we need its code to
control certificate validation (if Twisted actually moved away from pyOpenSSL
for TLS, Foolscap might break altogether).

The Twisted dependency was updated to >=16.0.0 (the current version), to get
an important HostnameEndpoint fix (155).

The "flogtool", "flappserver", and "flappclient" executables are now provided
as "entry_points" on all platforms, not just windows. The old bin/* scripts
have been removed. The "flogtool" entrypoint was fixed (a one-character typo
in the setup.py specification): apparently it was always broken on windows
and nobody noticed.

We now use "tox" to run tests, instead of "trial foolscap", although the
latter is still fine when run in a virtualenv into which Foolscap has been
installed (and is what "tox" does under the hood).

This release also moves all source code from "foolscap/" to "src/foolscap/",
which should avoid some confusion as to which code is being tested.
Developers who work from a git checkout should manually "rm -rf foolscap"
after pulling this change, because otherwise the leftover .pyc files are
likely to cause spurious test failures. (250, 251)

partial IPv6 support

Foolscap's outbound connections now use HostnameEndpoint, which means that
connection hints which contain DNS names which map to AAAA (and maybe A6)
records should successfully connect to those IPv6 addresses. There is not yet
any support to *listen* on IPv6 ports, so this probably does not enable IPv6
completely. But a client running this release may be able to connect to
server running some future IPv6-capable release and advertising v6-based
hostnames. (155)

Packaging Fixes

This release fixes a version-string management failure when the "log
publisher" feature was used in a tree built from a release tarball (rather
than from a git checkout). This caused a unit test failure, as well as
operational failures when using `flogtool tail`. Thanks to Ramakrishnan
Muthukrishnan (vu3rdd) for the catch and the patch. (248)

Compatibility Fixes

This release is compatible with Twisted-15.3.0 through 15.5.0. A change in

Point release to deal with PyPI upload problems. No code changes.