Foolscap

bug fixes

Tubs can now connect to themselves

In previous releases, Tubs were unable to connect to themselves: the
following code would fail (the negotiation would never complete, so the
connection attempt would eventually time out after about 30 seconds):

url = mytub.registerReference(target)
d = mytub.getReference(url)

In release 0.0.7, this has been fixed by catching this case and making it use
a special loopback transport (which serializes all messages but does not send
them over a wire). There may be still be problems with this code, in
particular connection shutdown is not completely tested and producer/consumer
code is completely untested.

Tubs can now getReference() the same URL multiple times

A bug was present in the RemoteReference-unslicing code which caused the
following code to fail:

d = mytub.getReference(url)
d.addCallback(lambda ref: mytub.getReference(url))

In particular, the second call to getReference() would return None rather
than the RemoteReference it was supposed to.

This bug has been fixed. If the previous RemoteReference is still alive, it
will be returned by the subsequent getReference() call. If it has been
garbage-collected, a new one will be created.

minor fixes

Negotiation errors (such as having incompatible versions of Foolscap on
either end of the wire) may be reported more usefully.

In certain circumstances, disconnecting the Tub service from a parent service
might have caused an exception before. It might behave better now.

of changes to the negotiation process and the method-calling portion of the
main wire protocol. (you were warned :-). There are still more incompatible
changes to come in future versions as the feature set and protocol
stabilizes. Make sure you can upgrade both ends of the wire until a protocol
freeze has been declared.

Negotiation versions now specify a range, instead of a single number

The two ends of a connection will agree to use the highest mutually-supported
version. This approach should make it much easier to maintain backwards
compatibility in the future.

Negotiation now includes an initial VOCAB table

One of the outputs of connection negotiation is the initial table of VOCAB
tokens to use for abbreviating commonly-used strings into short tokens
(usually just 2 bytes). Both ends have the ability to modify this table at any
time, but by setting the initial table during negotiation we same some
protocol traffic. VOCAB-izing common strings (like 'list' and 'dict') have
the potential to compress wire traffic by maybe 50%.

remote methods now accept both positional and keyword arguments

Previously you had to use a RemoteInterface specification to be able to pass
positional arguments into callRemote(). (the RemoteInterface schema was used
to convert the positional arguments into keyword arguments before sending
them over the wire). In 0.0.6 you can pass both posargs and kwargs over the
wire, and the remote end will pass them directly to the target method. When
schemas are in effect, the arguments you send will be mapped to the method's
named parameters in the same left-to-right way that python does it. This
should make it easier to port oldpb code to use Foolscap, since you don't
have to rewrite everything to use kwargs exclusively.

Schemas now allow =None and =RIFoo

You can use 'None' in a method schema to indicate that the argument or return
value must be None. This is useful for methods that always return None. You
can also require that the argument be a RemoteReference that provides a
particular RemoteInterface. For example:

class RIUser(RemoteInterface):
def get_age():
return int
def delete():
return None

class RIUserDatabase(RemoteInterface):
def get_user(username=str):
return RIUser

Note that these remote interface specifications are parsed at import time, so
any names they refer to must be defined before they get used (hence placing
RIUserDatabase before RIUser would fail). Hopefully we'll figure out a way to
fix this in the future.

Violations are now annotated better, might keep more stack-trace information

Copyable improvements

The Copyable documentation has been split out to docs/copyable.xhtml and
somewhat expanded.

The new preferred Copyable usage is to have a class-level attribute named
"typeToCopy" which holds the unique string. This must match the class-level
"copytype" attribute of the corresponding RemoteCopy class. Copyable
subclasses (or ICopyable adapters) may still implement getTypeToCopy(), but
the default just returns self.typeToCopy . Most significantly, we no longer
automatically use the fully-qualified classname: instead we *require* that
the class definition include "typeToCopy". Feel free to use any stable and
globally-unique string here, like a URI in a namespace that you control, or
the fully-qualified package/module/classname of the Copyable subclass.

The RemoteCopy subclass must set the 'copytype' attribute, as it is used for
auto-registration. These can set copytype=None to inhibit auto-registration.

add Tub.setOption, add logRemoteFailures and logLocalFailures

These options control whether we log exceptions (to the standard twisted log)
that occur on other systems in response to messages that we've sent, and that
occur on our system in response to messages that we've received
(respectively). These may be useful while developing a distributed
application. All such log messages have each line of the stack trace prefixed
by REMOTE: or LOCAL: to make it clear where the exception is happening.

add sarge packaging, improve dependencies for sid and dapper .debs

fix typo that prevented Reconnector from actually reconnecting

API Changes

notifyOnDisconnect() takes args/kwargs

RemoteReference.notifyOnDisconnect(), which registers a callback to be fired
when the connection to this RemoteReference is lost, now accepts args and
kwargs to be passed to the callback function. Without this, application code
needed to use inner functions or bound methods to close over any additional
state you wanted to get into the disconnect handler.

notifyOnDisconnect() returns a "marker", an opaque values that should be
passed into the corresponding dontNotifyOnDisconnect() function to deregister
the callback. (previously dontNotifyOnDisconnect just took the same argument
as notifyOnDisconnect).

For example:

class Foo:
def _disconnect(self, who, reason):
print "%s left us, because of %s" % (who, reason)
def connect(self, url, why):
d = self.tub.getReference(url)
def _connected(rref):
self.rref = rref
m = rref.notifyOnDisconnect(self._disconnect, who, reason=why)
self.marker = m
d.addCallback(_connected)
def stop_caring(self):
self.rref.dontNotifyOnDisconnect(self.marker)

Reconnector / Tub.connectTo()

There is a new connection API for applications that want to connect to a
target and to reconnect to it if/when that connection is lost. This is like
ReconnectingClientFactory, but at a higher layer. You give it a URL to
connect to, and a callback (plus args/kwargs) that should be called each time
a connection is established. Your callback should use notifyOnDisconnect() to
find out when it is disconnected. Reconnection attempts use exponential
backoff to limit the retry rate, and you can shut off reconnection attempts
when you no longer want to maintain a connection.

Use it something like this:

class Foo:
def __init__(self, tub, url):
self.tub = tub
self.reconnector = tub.connectTo(url, self._connected, "arg")
def _connected(self, rref, arg):
print "connected"
assert arg == "arg"
self.rref = rref
self.rref.callRemote("hello")
self.rref.notifyOnDisconnect(self._disconnected, "blag")
def _disconnected(self, blag):
print "disconnected"
assert blag == "blag"
self.rref = None
def shutdown(self):
self.reconnector.stopConnecting()

Code which uses this pattern will see "connected" events strictly interleaved
with "disconnected" events (i.e. it will never see two "connected" events in
a row, nor two "disconnected" events).

The basic idea is that each time your _connected() method is called, it
should re-initialize all your state by making method calls to the remote
side. When the connection is lost, all that state goes away (since you have
no way to know what is happening until you reconnect).

Behavioral Changes

All Referenceable object are now implicitly "giftable"

In 0.0.3, for a Referenceable to be "giftable" (i.e. useable as the payload
of an introduction), two conditions had to be satisfied. 1: the object must
be published through a Tub with Tub.registerReference(obj). 2: that Tub must
have a location set (with Tub.setLocation). Once those conditions were met,
if the object was sent over a wire from this Tub to another one, the
recipient of the corresponding RemoteReference could pass it on to a third
party. Another side effect of calling registerReference() is that the Tub
retains a strongref to the object, keeping it alive (with respect to gc)
until either the Tub is shut down or the object is explicitly de-registered
with unregisterReference().

Starting in 0.0.4, the first condition has been removed. All objects which
pass through a setLocation'ed Tub will be usable as gifts. This makes it much
more convenient to use third-party references.

Note that the Tub will *not* retain a strongref to these objects (merely a
weakref), so such objects might disappear before the recipient has had a
chance to claim it. The lifecycle of gifts is a subject of much research. The
hope is that, for reasonably punctual recipients, the gift will be kept alive
until they claim it. The whole gift/introduction mechanism is likely to
change in the near future, so this lifetime issue will be revisited in a
later release.

Build Changes

The source tree now has some support for making debian-style packages (for
both sid and dapper). 'make debian-sid' and 'make debian-dapper' ought to
create a .deb package.

API Changes

The primary entry point for Foolscap is now the "Tub":

import foolscap
t = foolscap.Tub()
d = t.getReference(pburl)
d.addCallback(self.gotReference)
...

The old "PBService" name is gone, use "Tub" instead. There are now separate
classes for "Tub" and "UnauthenticatedTub", rather than using an "encrypted="
argument. Tubs always use encryption if available: the difference between the
two classes is whether this Tub should use a public key for its identity or
not. Note that you always need encryption to connect to an authenticated Tub.
So install pyopenssl, really.

eventual send operators

Foolscap now provides 'eventually' and 'fireEventually', to implement the
"eventual send" operator advocated by Mark Miller's "Concurrency Among
Strangers" paper (http://www.erights.org/talks/promises/index.html).
eventually(cb, *args, **kwargs) runs the given call in a later reactor turn.
fireEventually(value=None) returns a Deferred that will be fired (with
'value') in a later turn. These behave a lot like reactor.callLater(0,..),
except that Twisted doesn't actually promise that a pair of callLater(0)s
will be fired in the right order (they usually do under unix, but they
frequently don't under windows). Foolscap's eventually() *does* make this
guarantee. In addition, there is a flushEventualQueue() that is useful for
unit tests, it returns a Deferred that will only fire when the entire queue
is empty. As long as your code only uses eventually() (and not callLater(0)),
putting the following in your trial test cases should keep everything nice
and clean:

def tearDown(self):
return foolscap.flushEventualQueue()

Promises

An initial implementation of Promises is in foolscap.promise for
experimentation. Only "Near" Promises are implemented to far (promises which
resolve to a local object). Eventually Foolscap will offer "Far" Promises as
well, and you will be able to invoke remote method calls through Promises as
well as RemoteReferences. See foolscap/test/test_promise.py for some hints.

Bug Fixes

Messages containing "Gifts" (third-party references) are now delivered in the
correct order. In previous versions, the presence of these references could
delay delivery of the containing message, causing methods to be executed out
of order.

The VOCAB-manipulating code used to have nasty race conditions, which should
be all fixed now. This would be more important if we actually used the
VOCAB-manipulating code yet, but we don't.

Lots of internal reorganization (put all slicers in a subpackage), not really
user-visible.

Updated to work with recent Twisted HEAD, specifically changes to sslverify.
This release of Foolscap ought to work with the upcoming Twisted-2.5 .

Incompatible protocol changes

There are now separate add-vocab and set-vocab sequences, which add a single
new VOCAB token and replace the entire table, respectively. These replace the
previous 'vocab' sequence which behaved like set-vocab does now. This would
be an incompatible protocol change, except that previous versions never sent
the vocab sequence anyways. This version doesn't send either vocab-changing
sequence either, but when we finally do start using it, it'll be ready.

Renamed to "Foolscap", extracted from underneat the Twisted packaged,
consolidated API to allow a simple 'import foolscap'. No new features or bug
fixes relative to pb2-0.0.1 .