Twisted

===========================

Features
--------

- twisted.python.failure.Failure tracebacks now capture module information, improving compatibility with the Raven Sentry client. (7796)
- twisted.python.failure.Failure objects are now compatible with dis.distb, improving compatibility with post-mortem debuggers. (9599)


Bugfixes
--------

- twisted.internet.interfaces.IReactorSSL.listenSSL now has correct type annotations. (10274)
- twisted.internet.test.test_glibbase.GlibReactorBaseTests now passes. (10317)


Conch
-----

Features
~~~~~~~~

- twisted.conch.ssh now supports using RSA keys with SHA-2 signatures (RFC 8332) when acting as a server. The rsa-sha2-512 and rsa-sha2-256 public key signature algorithms are automatically preferred over ssh-rsa if the client advertises support for them; the actual public keys do not need to change. (9765)
- twisted.conch.ssh now has an alternative Ed25519 implementation using PyNaCl, in order to support platforms that lack OpenSSL >= 1.1.1b. The new "conch_nacl" extra has the necessary dependency. (10208)


Misc
~~~~

- (10313)


Web
---

Features
~~~~~~~~

- Twisted is now compatible with h2 4.x.x. (10182)


Bugfixes
~~~~~~~~

- twisted.web.http had several several defects in HTTP request parsing that could permit HTTP request smuggling. It now disallows signed Content-Length headers, forbids illegal characters in chunked extensions, forbids a ``0x`` prefix to chunk lengths, and only strips spaces and horizontal tab characters from header values. These changes address CVE-2022-24801 and GHSA-c2jg-hw38-jrqq. (10323)


Mail
----

Bugfixes
~~~~~~~~

- twisted.mail.pop3.APOPCredentials is now correctly marked as implementing twisted.cred.credentials.IUsernamHashedPassword, rather than IUsernamePassword. (10305)


Words
-----

No significant changes.


Names
-----

No significant changes.


Trial
-----

Features
~~~~~~~~

- `trial --until-failure --jobs=N` now reports the number of each test pass as it begins. (10312)


Bugfixes
~~~~~~~~

- twisted.trial.unittest.TestCase now discards cleanup functions after running them. Notably, this prevents them from being run an ever growing number of times with `trial -u ...`. (10320)


Misc
~~~~

- 10315, 10321, 10322

===========================

Bugfixes
--------

- twisted.internet.gireactor.PortableGIReactor.simulate and twisted.internet.gtk2reactor.PortableGtkReactor.simulate no longer raises TypeError when there are no delayed called. This was a regression introduced with the migration to Python 3 in which the builtin `min` function no longer accepts `None` as an argument. (9660)
- twisted.conch.ssh.transport.SSHTransportBase now disconnects the remote peer if the
SSH version string is not sent in the first 4096 bytes. (10284, CVE-2022-21716,
GHSA-rv6r-3f5q-9rgx)


Improved Documentation
----------------------

- Add type annotations for twisted.web.http.Request.getHeader. (10270)


Deprecations and Removals
-------------------------

- Support for Python 3.6, which is EoL as of 2021-09-04, has been deprecated. (10303)


Misc
----

- 10216, 10299, 10300


Conch
-----

Misc
~~~~

- 10298


Web
---

No significant changes.


Mail
----

No significant changes.


Words
-----

No significant changes.


Names
-----

No significant changes.


Trial
-----

Bugfixes
~~~~~~~~

- _dist.test.test_workertrial now correctly compare strings via assertEqual() and pass on PyPy3 (10302)

===========================

Features
--------

- Python 3.10 is now a supported platform (10224)
- Type annotations have been added to the twisted.python.fakepwd module. (10287)


Bugfixes
--------

- twisted.internet.defer.inlineCallbacks has an improved type annotation, to avoid typing errors when it is used on a function which returns a non-None result. (10231)
- ``twisted.internet.base.DelayedCall.__repr__`` and ``twisted.internet.task.LoopingCall.__repr__`` had the changes from 10155 reverted to accept non-function callables. (10235)
- Revert the removal of .whl building that was done as part of 10177. (10236)
- The type annotation of the host parameter to twisted.internet.interfaces.IReactorTCP.connectTCP has been corrected from bytes to str. (10251)
- Deprecated ``twisted.python.threading.ThreadPool.currentThread()`` in favor of ``threading.current_thread()``.
Switched ``twisted.python.threading.ThreadPool.currentThread()`` and ``twisted.python.threadable.getThreadID()`` to use `threading.current_thread()`` to avoid the deprecation warnings introduced for ``threading.currentThread()`` in Python 3.10. (10273)


Improved Documentation
----------------------

- twisted.internet.utils.runWithWarningsSupressed behavior of waiting on deferreds has been documented. (10238)
- Sync API docs templates with pydoctor 21.9.0 release, using new theming capabilities. (10267)


Misc
----

- 1681, 9944, 10198, 10218, 10219, 10228, 10229, 10234, 10239, 10240, 10245, 10246, 10248, 10250, 10255, 10277, 10288, 10292


Conch
-----

Bugfixes
--------

- SSHTransportBase.ssh_KEXINIT now uses the remote peer preferred MAC list for negotiation. In previous versions it was only using the local preferred MAC list. (10241)


Features
~~~~~~~~

- twisted.conch.ssh now supports SSH extension negotiation (RFC 8308). (10266)


Bugfixes
~~~~~~~~

- twisted.conch now uses constant-time comparisons for MACs. (8199)
- twisted.conch.ssh.filetransfer.FileTransferServer will now return an ENOENT error status if an SFTP client tries to close an unrecognized file handle. (10293)


Web
---

Bugfixes
~~~~~~~~

- twisted.web.client.RedirectAgent and twisted.web.client.BrowserLikeRedirectAgent now properly remove sensitive headers when redirecting to a different origin. (10294)


Improved Documentation
----------------------

- Add type annotations for twisted.web.client.readBody. (10269)


Deprecations and Removals
~~~~~~~~~~~~~~~~~~~~~~~~~

- twisted.web.client.getPage, twisted.web.client.downladPage, and the associated implementation classes (HTTPPageGetter, HTTPPageDownloader, HTTPClientFactory, HTTPDownloader) have been removed because they do not segregate cookies by domain. They were deprecated in Twisted 16.7.0 in favor of twisted.web.client.Agent. GHSA-92x2-jw7w-xvvx. (10295)


Mail
----

No significant changes.


Words
-----

No significant changes.


Names
-----

No significant changes.


Trial
-----

Bugfixes
~~~~~~~~

- trial.runner.filenameToModule now sets the correct module.__name__ and sys.modules key (10230)

===========================


Features
--------

- Python 3.10b3 is now supported (10224)
- Type hinting was added to twisted.internet.defer, making this is the first release
of Twisted where you might reasonably be able to use mypy without your own custom
stub files (10017)


Bugfixes
--------

- The changes to ``DelayedCall.__repr__`` and ``LoopingCall.__repr__`` from
21.7.0.rc1 were reverted as the wrong assumption that ``__qualname__`` is
available on all the supported Python versions.
(10235)
- The automated release process was updated to generate and release wheel files
to PyPI (10236)
- twisted.internet.defer.inlineCallbacks has an improved type annotation, to avoid typing errors when it is used on a function which returns a non-None result. (10231)
- trial.runner.filenameToModule now sets the correct ``module.__name__`` and ``sys.modules`` key (10230)
- twisted.internet.process can now pause and resume producing in python 3 (9933)
- When installing Twisted it now requires a minimum Python 3.6.7 version to match the version used with automated testing. This is the minimum Python version that we know that Twisted works with. (10098)
- twisted.internet.asyncioreactor.AsyncioSelectorReactor will no longer raise a TypeError like "SelectorEventLoop required, instead got: <uvloop.Loop ...>" (broken since 21.2.0). (10106)
- twisted.web.template.flatten and flattenString will no longer raise RecursionError if a large number of synchronous Deferreds are included in a document. (10125)
- Fix type hint for http.Request.uri (from str to bytes). (10139)
- twisted.web.http_headers.getRawHeaders and twisted.web.http_headers.getAllRawHeaders are now typed to return immutable sequences of header values instead of lists.
twisted.web.http_headers.getRawHeaders is now typed to return a non-optional value if a non-None default value is given. (10142)
- Fixed type hint for addr argument to twisted.internet.interfaces.buildProtocol. (10147)
- twisted.trial._dist.worker.LocalWorker.connectionMade now always writes the
log file using UTF-8 encoding.
In previous versions it was using the system default encoding.
This was causing encoding errors as the distributed trial workers are sending
Unicode data and the system default encoding might not always be Unicode compatible.
For example, it can be CP1252 on Windows. (10157)
- twisted.words.protocols.irc.ctcpExtract was updated to work with PYPY 3.7.4. (10189)
- twisted.conch.ssh.transport.SSHServerTransport and twisted.conch.ssh.transport.SSHClientTransport no longer use the hardcoded
SHA1 digest for non-group key exchanges. (10203)
- haproxy transport wrapper now returns hosts of type str for getPeer() and getHost(), as specified by IPv4Address and IPv6Address documentation. Previously it was returning bytes for the host. (10211)


Improved Documentation
----------------------

- Remove dead link in twisted.internet._dumbwin32proc module docstring (9520)
- Sync API docs templates with pydoctor 21.2.2 release. (10105)
- Twisted IRC channels are now hosted by Libera.Chat. (10213)


Deprecations and Removals
-------------------------

- Python 3.5 is no longer supported. (9958)


Misc
----

- 9816, 9915, 10068, 10085, 10094, 10102, 10107, 10108, 10109, 10110, 10112, 10119, 10120, 10121, 10122, 10123, 10140, 10143, 10145, 10150, 10151, 10155, 10159, 10168, 10169, 10171, 10172, 10173, 10174, 10179, 10194, 10201, 10212, 10215, 10217, 11017


Conch
-----

Misc
~~~~

- 10097


Web
---

Features
~~~~~~~~

- twisted.web.template.renderElement() now accepts any IRequest implementer instead of only twisted.web.server.Request.
Add type hints to twisted.web.template. (10184)


Bugfixes
~~~~~~~~

- The server-side HTTP/1.1 chunking implementation no longer performs quadratic work when input arrives in small chunks, preventing CPU exhaustion. (3795)
- twisted.web.http's chunked encoding support now rejects chunk sizes that are invalid because they look like negative hexadecimal integers. (10130)
- The type hint of twisted.web.server.Request.postpath is now correctly listed as Optional[List[bytes]]. This was incorrect in Twisted v21.2.0. (10136)
- The server-side HTTP/1.1 chunking implementation now rejects invalid chunk boundaries, preventing unbounded buffering. (10137)
- The server-side HTTP/1.1 chunking implementation now limits the length of the chunk size line (which includes chunk extensions) to twisted.web.http.maxChunkSizeLineLength — 1 KiB — so that it may not consume an unbounded amount of memory. (10144)
- Calling twisted.web.server.Site now registers its expiration timeout using the reactor associated with its twisted.web.server.Site. Site now a reactor attribute via its superclass, twisted.web.http.HTTPFactory. (10177)


Misc
~~~~

- 9659, 10100, 10154, 10186


Mail
----

No significant changes.


Words
-----

No significant changes.


Names
-----

No significant changes.

===========================

Features
--------

- The enableSessions argument to twisted.internet.ssl.CertificateOptions now
actually enables/disables OpenSSL's session cache. Also, due to
session-related bugs, it defaults to False. (9583)
- twisted.internet.defer.inlineCallbacks and ensureDeferred will now associate a contextvars.Context with the coroutines they run, meaning that ContextVar objects will maintain their value within the same coroutine, similarly to asyncio Tasks. This functionality requires Python 3.7+, or the contextvars PyPI backport to be installed for Python 3.5-3.6. (9719, 9826)
- twisted.internet.defer.Deferred.fromCoroutine has been added. This is similar to the existing ensureDeferred function, but is named more consistently inside Twisted and does not pass through Deferreds. (9825)
- trial now allows the unittest.skipIf decorator to specify that an entire test class should be skipped. (9829)
- The twisted.python.deprecate.deprecatedKeywordParameter decorator can be used to mark a keyword paramater of a function or method as deprecated. (9844)
- Projects using Twisted can now perform type checking against a Twisted
installation, for example using mypy. (9908)
- twisted.python.util.InsensitiveDict now fully implements MutableMapping. (9919)
- Python 3.8 is now tested and supported. (9955)
- Support a coroutine function in twisted.internet.task.react (9974)
- PyPy 3.7 is now tested and supported. (10093)


Bugfixes
--------

- twisted.web.twcgi.CGIProcessProtocol.processEnded(...) now handles an already-finished request, for example when request.connectionLost(...) was called previously. (9468)
- Twisted's dependency on PyHamcrest has been moved from the base package to the new "test" extra. Consequently the test extra must be installed for Twisted's test suite to pass. (9509)
- Fixed serialization of timedelta, date, and time objects in twisted.spread. (9716)
- twisted.internet.asyncioreactor.AsyncioSelectorReactor now raises an exception if instantiated with an event loop which is not compatible with asyncio.SelectorEventLoop. This fixes the AsyncioSelectorReactor in Python 3.8+ on Windows, where in bp-34687 the default Windows asyncio event loop was changed to ProactorEventLoop. Applications that use AsyncioSelectorReactor on Windows with Python 3.8+ must call asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) before instantiating and running AsyncioSelectorReactor. (9766)
- twisted.internet.process.registerReapProcessHandler and ._BaseProcess.reapProcess will no longer raise a TypeError when processing a None PID (9775)
- INotify will close its file descriptor if a directory is automatically removed by twisted from the watchlist because it's deleted, avoiding orphaned filedescriptors. (9777)
- DelayedCall.reset() is now working properly with asyncioreactor (9780)
- AsyncioSelectorReactor.seconds() now correctly returns an epoch time. (9787)
- The _connDone parameter has been removed from twisted.internet.abstract.FileDescriptor.loseConnection()'s signature in order to match the signature in the base class twisted.internet._newtls.ConnectionMixin loseConnection(). (9849)
- The Gtk3 reactor now runs on Wayland-only sessions (9904)
- Descriptive error messages from twisted.internet.error are now present when running with 'python -OO'. (9918)
- Comparator methods such as __eq__() now always return NotImplemented for uncomparable types. (9919)
- When installing Twisted it now requires a minimum Python 3.5.4 version to match the version used with automated testing. This is the minimum Python version that we know that Twisted works with. (10098)


Improved Documentation
----------------------

- The narrative docs now contains the associated Twisted version and the date
when they were generated. (3945)
- The "Writing a twistd plugin" howto now explains how to deploy twistd plugins using Python packaging and pip (9243)
- (9868, 9873, 9874)
- Fix a typo in "Introduction to Deferreds" document. (9948)
- The Twisted Coding Standard has been changed to refer to The Black code style for guidelines regarding whitespace and line lengths. (9957)
- Exempt ``__repr__``, ``__slots__`` and other ``attrs.define`` related changes from compatibility policy. (9982)
- Fix many docstring mistakes flagged by new sanity checks in pydoctor. (10021)
- Fix a few dozen broken links to API documentation pages. (10057)


Deprecations and Removals
-------------------------

- twisted.cred.credentials.UsernameHashedPassword is now deprecated because it doesn't hash the password, causing it to return the wrong result. (8368)
- twisted.news is now removed from the codebase. (This module was never installed on Python 3.) (9782)
- Support for Python 2.7 has been removed. Twisted now supports only Python versions 3.5/3.6/3.7. (9790)
- twisted.pair.ethernet.IEthernetProtocol.addProto()'s interface was changed to match the existing implementations in the Twisted source code. (9877)
- twisted.python.filepath.FilePath.statinfo was deprecated in Twisted 15.0.0 and has now been removed. (9881)
- The parameters to twisted.internet.base.ReactorBase.addSystemEventTrigger(), twisted.internet.base.ReactorBase.callWhenRunning(), twisted.internet.base.ReactorBase.callLater(), twisted.internet.task.Clock.callLater() have been renamed to match the parameters defined in the following interfaces: twisted.internet.interfaces.IReactorCore, twisted.internet.interfaces.IReactorTime. (9897)
- Functions and types in twisted.python.compat that existed to support the transition from Python 2 to 3 have been deprecated. (9922)
- twisted.logger.LoggingFile.softspace has been deprecated. (10042)
- twisted.python.win32.WindowsError and FakeWindowsError have been deprecated. (10053)
- twisted.mail.pop3client has been renamed to twisted.mail._pop3client, since it has always been a private implementation module. (10054)


Misc
----

- 5356, 6460, 6903, 6986, 7945, 9306, 9512, 9531, 9622, 9652, 9718, 9744, 9768, 9773, 9776, 9778, 9781, 9784, 9785, 9788, 9789, 9791, 9793, 9795, 9796, 9797, 9798, 9800, 9802, 9803, 9808, 9809, 9810, 9811, 9812, 9820, 9823, 9827, 9833, 9837, 9840, 9842, 9846, 9847, 9848, 9850, 9851, 9852, 9854, 9855, 9856, 9857, 9858, 9861, 9862, 9863, 9864, 9865, 9866, 9867, 9869, 9870, 9871, 9872, 9876, 9878, 9879, 9880, 9882, 9883, 9884, 9886, 9889, 9890, 9891, 9892, 9895, 9896, 9898, 9899, 9902, 9903, 9916, 9917, 9921, 9924, 9927, 9928, 9936, 9953, 9954, 9956, 9959, 9960, 9969, 9970, 9971, 9975, 9976, 9977, 9978, 9979, 9980, 9981, 9983, 9985, 9986, 9987, 9988, 9989, 9991, 9992, 9995, 9999, 10000, 10002, 10009, 10010, 10011, 10014, 10015, 10018, 10025, 10027, 10029, 10032, 10033, 10034, 10036, 10038, 10043, 10044, 10046, 10054, 10059, 10060, 10061, 10063, 10064, 10065, 10069, 10080, 10090


Conch
-----

Features
~~~~~~~~

- twisted.conch.ssh now supports Ed25519 keys (requires OpenSSL >= 1.1.1b). (8966)
- twisted.conch.ssh.session.SSHSession can now accept environment variables sent by the client, if the SSH avatar implements the new ISessionSetEnv interface. (9315)
- twisted.conch.ssh.keys.Key.fromString and twisted.conch.ssh.keys.Key.toString now normalize Unicode passphrases as required by NIST 800-63B. (9736)
- twisted.conch.telnet now implements EOR (End of Record) command (RFC 885) (9875)


Bugfixes
~~~~~~~~

- t.c.ssh.filetransfer.FileTransferClient now errbacks any outstanding requests if the connection is lost before a reply is received. (9571)
- t.c.ssh.filetransfer.FileTransferClient immediately errbacks any attempt to send a request on a closed channel. (9572)
- twisted.conch.ssh.session.SSHSession now accepts environment variables also for multiplexed SSH session. (10016)


Improved Documentation
~~~~~~~~~~~~~~~~~~~~~~

- construct and assign portal and checkers consistently in ssh server example (9578)


Misc
~~~~

- 6446, 9571, 9831, 9913


Web
---

Bugfixes
~~~~~~~~

- twisted.web.http.Request.getRequestHostname now supports IPv6 literal hostnames
in HTTP host headers. (6014)
- Fixed unexpected exception by handling subclass of TaskFinished when FileBodyProducer's task stopped twice. (6528)
- Importing twisted.web.client no longer has the side effect of initializing the reactor. (9774)
- Ensure that all calls to connectionLost use a Failure instance in the HTTP 2 code. (9817)
- twisted.web.util.ParentRedirect has been fixed and documented. It was broken by a security fix in Twisted 19.2.0. (9835)
- xmlrpc's Proxy class now verifies HTTPS certificates against the system bundle. (9836)
- twisted.web.twcgi can now handle url parameters in python 3 (9887)
- defer reactor import in twisted.web.xmlrpc (9931)
- twisted.web.RedirectAgent now supports 308 redirects (9940)
- Fixed an error where twisted.web.http.requestReceived() tries to encode a NoneType returned by cgi.parse_multipart when a multipart body does not contain a "content-disposition" definition. (10084)


Improved Documentation
~~~~~~~~~~~~~~~~~~~~~~

- xmlrpc's QueryFactory class is now public, more explanation for xmlrpc's queryFactory, and new xmlrpc-debug.py example script for debugging raw XML-RPC traffic. (9350)
- twisted.web.client.ContentDecoderAgent's documentation has been corrected and improved. (9742)


Misc
~~~~

- 6446, 9758, 9801, 9831, 9834, 9841


Mail
----

Bugfixes
~~~~~~~~

- twisted.mail.smtp.ESMTPSender no longer forces TLSv1.0 when used without explicit context factory. (9740)


Misc
~~~~

- 6446, 9831, 9832, 9900, 9910


Words
-----

Misc
~~~~

- 9901


Names
-----

Features
~~~~~~~~

- twisted.names.hosts.Resolver and twisted.names.hosts.searchFileForAll() now ignore malformed lines in hosts files like /etc/hosts (9752)
- New interface IEncodableRecord combines IEncodable and IRecord, which is useful when using type annotations. (9920)


Bugfixes
~~~~~~~~

- twistd -n dns --pyzone example-domain.com will no longer throw an exception on startup with Python 3. (9783)
- twist dns --pyzone example-domain.com now works on Python 3. (9786)


Misc
~~~~

- 9749

===========================

Bugfixes
--------

- twisted.protocols.amp.BoxDispatcher.callRemote and callRemoteString will no longer return failing Deferreds for requiresAnswer=False commands when the transport they're operating on has been disconnected. (9756)


Improved Documentation
----------------------

- Added a missing hyphen to a reference to the ``--debug`` option of ``pdb`` in the Trial how-to. (9690)
- The documentation of the twisted.cred.checkers module has been extended and corrected. (9724)


Deprecations and Removals
-------------------------

- twisted.news is deprecated. (9405)


Misc
----

- 9634, 9701, 9707, 9710, 9715, 9726, 9727, 9728, 9729, 9735, 9737, 9757


Conch
-----

Features
~~~~~~~~

- twisted.conch.ssh now supports the curve25519-sha256 key exchange algorithm (requires OpenSSL >= 1.1.0). (6814)
- twisted.conch.ssh.keys can now write private keys in the new "openssh-key-v1" format, introduced in OpenSSH 6.5 and made the default in OpenSSH 7.8. ckeygen has a corresponding new --private-key-subtype=v1 option. (9683)


Bugfixes
~~~~~~~~

- twisted.conch.keys.Key.privateBlob now returns the correct blob format for ECDSA (i.e. the same as that implemented by OpenSSH). (9682)


Misc
~~~~

- 9760


Web
---

Bugfixes
~~~~~~~~

- Fixed return type of twisted.web.http.Request.getUser and twisted.web.http.Request.getPassword to binary if no authorization header was found or an exception was thrown (9596)
- twisted.web.http.HTTPChannel now rejects requests (with status code 400 and a drop) that have malformed headers of the form "Foo : value" or ": value". (9646)
- twisted.web.http.Request now correctly parses multipart-encoded form data submitted as a chunked request on Python 3.7+. (9678)
- twisted.web.client.BrowserLikePolicyForHTTPS is now listed in __all__, since it's a user-facing class that anyone could import and extend. (9769)
- twisted.web.http was subject to several request smuggling attacks. Requests with multiple Content-Length headers were allowed (CVE-2020-10108, thanks to Jake Miller from Bishop Fox and ZeddYu Lu for reporting this) and now fail with a 400; requests with a Content-Length header and a Transfer-Encoding header honored the first header (CVE-2020-10109, thanks to Jake Miller from Bishop Fox for reporting this) and now fail with a 400; requests whose Transfer-Encoding header had a value other than "chunked" and "identity" (thanks to ZeddYu Lu) were allowed and now fail with a 400. (9770)


Mail
----

Misc
~~~~

- 9733


Words
-----

Bugfixes
~~~~~~~~

- Fixed parsing of streams with Python 3.8 when there are spaces in namespaces or namespaced attributes in twisted.words.xish.domish.ExpatElementStream (9730)


Names
-----

Bugfixes
~~~~~~~~

- twisted.names.secondary.SecondaryAuthority now accepts str for its domain parameter, so twist dns --secondary now functions on Python 3. (9496)