Foolscap

Compatibility

All releases between 0.1.3 and 0.2.3 (inclusive) are fully wire-compatible.

Bug Fixes

RemoteReference.getSturdyRef() didn't work (due to bitrot). It has been
fixed.

foolscap.logging Changes

This release is mostly about flogging improvements: some bugs and misfeatures
were fixed:

tolerate '%' in log message format strings

Dictionary-style kwarg formatting is now done with a twisted-style style
format= argument instead of happening implicitly. That means the acceptable
ways to call foolscap.logging.log.msg are:

log.msg("plain string")
log.msg("no args means use 0% interpolation")
log.msg("pre-interpolated: %d apples" % apples)
log.msg("posargs: %d apples and %d oranges", apples, oranges)
log.msg(format="kwargs: %(numapples)d apples", numapples=numapples)

The benefit of the latter two forms are that the arguments are recorded
separately in the event dictionary, so viewing tools can filter on the
structured data, think of something like:

[e for e in allEvents if e.get("numapples",0) > 4]

log facility names are now dot-separated, to match stdlib logging
log levels are derived from numerical stdlib logging levels
$FLOGFILE to capture flog events during 'trial' runs

One challenge of the flogging system is that, once an application was changed
to write events to flogging instead of twisted's log, those events do not
show up in the normal places where twisted writes its logfiles. For full
applications this will be less of an issue, because application startup will
tell flogging where events should go (flogging is intended to supplant
twisted logging for these applications). But for events emitted during unit
tests, such as those driven by Trial, these events would get lost.

To address this problem, the 0.2.3 flogging code looks for the "FLOGFILE"
environment variable at import time. This specifies a filename where flog
events (a series of pickled event dictionaries) should be written. The file
is opened at import time, events are written during the lifetime of the
process, then the file is closed at shutdown using a Twisted "system event
trigger" (which happens to be enough to work properly under Trial: other
environments may not work so well). If the FLOGFILE filename ends in .bz2,
the event pickles will be compressed, which is highly recommended because it
can result in a 30x space savings (and e.g. the Tahoe unit test run results
in 90MB of uncompressed events). All 'flogtool' modes know how to handle a
.bz2 compressed flogfile as well as an uncompressed one.

The "FLOGTWISTED" environment variable, if set, will cause this same code to
bridge twisted log messages into the flogfile. This makes it easier to see
the relative ordering of Twisted actions and foolscap/application events.
(without this it becomes very hard to correlate the two sources of events).

The "FLOGLEVEL" variable specifies a minimum severity level that will be put
into the flogfile. This defaults to "1", which puts pretty much everything
into the file. The idea is that, for tests, you might as well record
everything, and use the filtering tools to control the display and isolate
the important events. Real applications will use more sophisticated tradeoffs
between disk space and interpretation effort.

The recommended way to run Trial on a unit test suite for an application that
uses Foolscap is:

FLOGFILE=flog.out FLOGTWISTED=1 trial PACKAGENAME

Note that the logfile cannot be placed in _trial_temp/, because trial deletes
that directory after flogging creates the logfile, so the logfile would get
deleted too. Also note that the file created by $FLOGFILE is truncated on
each run of the program.

Compatibility

All releases between 0.1.3 and 0.2.2 (inclusive) are fully wire-compatible.
New (optional) negotiation parameters were added in 0.2.1 (really in 0.2.0).

Bug Fixes

The new duplicate-connection handling code in 0.2.1 was broken. This release
probably fixes it.

There were other bugs in 0.2.1 which were triggered when a duplicate
connection was shut down, causing remote calls to never be retired, which
would also prevent the Reconnector from doing its job. These should be fixed
now.

Other Changes

Foolscap's connection-negotiation mechanism has been modified to use foolscap
logging ("flog") instead of twisted.log .

Setting the FLOGFILE= environment variable will cause a Foolscap-using
program to write pickled log events to a file of that name. This is
particularly useful when you want to record log events during 'trial' unit
test run. The normal API for setting this file will be added later. The
FLOGTWISTED= environment variable will cause the creation of a twisted.log
bridge, to copy all events from the twisted log into the foolscap log.

The 'flogtool web-view' mode has been enhanced to color-code events according
to their severity, and to format Failure tracebacks in a more-readable way.

should provide much faster reconnection after netquakes. To benefit from
this, both ends must be running foolscap-0.2.1 or newer, however there is an
additional setting (not enabled by default) to improve the behavior of
pre-0.2.1 clients: tub.setOption("handle-old-duplicate-connections", True).

new Reconnector methods

The Reconnector object (as returned by Tub.connectTo) now has three utility
methods that may be useful during debugging. reset() drops the backoff timer
down to one second, causing the Reconnector to reconnect quickly: you could
use this to avoid an hour-long delay if you've just restarted the server or
re-enabled a network connection that was the cause of the earlier connection
failures. getDelayUntilNextAttempt() returns the number of seconds remaining
until the next connection attempt. And getLastFailure() returns a Failure
object explaining why the last connection attempt failed, which may be a
useful diagnostic in trying to resolve the connection problems.

Bug Fixes

There were other minor changes: an OS-X unit test failure was resolved,
CopiedFailures are serialized in a way that doesn't cause constraint
violations, and the figleaf code-coverage tools (used by foolscap developers
to measure how well the unit tests exercise the code base) have been improved
(including an emacs code-used/unused annotation tool).

This release had a fatal bug that wasn't caught by the unit tests, and was
superseded almost immediately by 0.2.1 .

Compatibility

All releases between 0.1.3 and 0.1.7 (inclusive) are fully wire-compatible.

Bug Fixes

slow remote_ methods shouldn't delay subsequent messages (25)

In earlier releases, a remote_ method which runs slowly (i.e. one which
returns a Deferred and does not fire it right away) would have the
unfortunate side-effect of delaying all subsequent calls from the same
Broker. Those later calls would not be delivered until the first message had
completed processing. If, for some reason, that Deferred were never fired,
this Foolscap bug would prevent any other remote_ methods from ever being
called.

This is not how Foolscap's message-ordering logic is designed to work.
Foolscap guarantees in-order *delivery* of messages, but does not require
that they be completed/retired in that same order.

This has now been fixed. The invocation of remote_* is done in-order: any
further sequencing is up to the receiving application.

For example, in the following code:

sender:
rref.callRemote("quick", 1)
rref.callRemote("slow", 2)
rref.callRemote("quick", 3)

receiver:
def remote_quick(self, num):
print num
def remote_slow(self, num):
print num
d = Deferred()
def _done():
print "DONE"
d.callback(None)
reactor.callLater(5.0, _done)
return d

The intended order of printed messages is 1,2,3,DONE . This bug caused the
ordering to be 1,2,DONE,3 instead.

default size limits removed from all Constraints (26)

Constraints in Foolscap serve two purposes: DoS attack mitigation, and strong
typing on remote interfaces to help developers find problems sooner. To
support the former, most container-based Constraints had default size limits.
For example, the default StringConstraint enforced a maximum length of 1000
characters, and the ListConstraint had a maxLength of 30 elements.

In practice, these limits turned out to be more surprising than helpful.
Applications which worked fine in testing would mysteriously break when
subjected to data that was larger than expected. Developers who used
Constraints for their type-checking properties were surprised to discover
that they were getting size limitations as well. In addition, the
DoS-mitigation code in foolscap is not yet complete, so the cost/benefit
ratio of this feature was dubious.

For these reasons, all default size limits have been removed from this

Compatibility

All releases between 0.1.3 and 0.1.6 (inclusive) are fully wire-compatible.

Bug Fixes

Using a schema of ChoiceOf(StringConstraint(2000), None) would fail to accept
strings between 1000 and 2000 bytes: it would accept a short string, or None,
but not a long string. This has been fixed. ChoiceOf() remains a troublesome
constraint: having it is awfully nice, and things like ChoiceOf(str,None)
seem to work, but it is unreliable. Using ChoiceOf with non-terminal children
is not recommended (the garden-path problem is unlikely to be easy to solve):
schemas are not regular expressions.

The debian packaging rules have been fixed. The ones in 0.1.5 failed to run
because of some renamed documentation files.

Minor Fixes

Several minor documentation errors have been corrected. A new 'make api-docs'
target has been added to run epydoc and build HTML versions of the API
documentation.

When a remote method fails and needs to send a traceback over the wire, and
when the traceback is too large, trim out the middle rather than the end,
since usually it's the beginning and the end that are the most useful.