Web-poet

------------------

This release contains many improvements to the web-poet testing framework,
as well as some other improvements and bug fixes.

Backward-incompatible changes:

* :func:`~.cached_method` no longer caches exceptions for ``async def`` methods.
This makes the behavior the same for sync and async methods, and also makes
it consistent with Python's stdlib caching (i.e. :func:`functools.lru_cache`,
:func:`functools.cached_property`).
* The testing framework now uses the ``HttpResponse-info.json`` file name instead
of ``HttpResponse-other.json`` to store information about HttpResponse
instances. To make tests generated with older web-poet work, rename
these files on disk.

Testing framework improvements:

* Improved test reporting: better diffs and error messages.
* By default, the pytest plugin now generates a test per item attribute
(see :ref:`web-poet-testing-pytest`). There is also an option
(``--web-poet-test-per-item``) to run a test per item instead.
* Page objects with the :class:`~.HttpClient` dependency are now supported
(see :ref:`web-poet-testing-additional-requests`).
* Page objects with the :class:`~.PageParams` dependency are now supported.
* Added a new ``python -m web_poet.testing rerun`` command
(see :ref:`web-poet-testing-tdd`).
* Fixed support for nested (indirect) dependencies in page objects.
Previously they were not handled properly by the testing
framework.
* Non-ASCII output is now stored without escaping in the test fixtures,
for better readability.

Other changes:

* Testing and CI fixes.
* Fixed a packaging issue: ``tests`` and ``tests_extra`` packages were
installed, not just ``web_poet``.

------------------

* Restore the minimum version of ``itemadapter`` from 0.7.1 to 0.7.0, and
prevent a similar issue from happening again in the future.

------------------

* Updated the :ref:`tutorial <tutorial>` to cover recent features and focus on
best practices. Also, a new module was added, :mod:`web_poet.example`, that
allows using page objects while following the tutorial.

* :ref:`web-poet-testing` now covers :ref:`Git LFS <git-lfs>` and
:ref:`scrapy-poet <web-poet-testing-scrapy-poet>`, and recommends
``python -m pytest`` instead of ``pytest``.

* Improved the warning message when duplicate ``ApplyRule`` objects are found.

* ``HttpResponse-other.json`` content is now indented for better readability.

* Improved test coverage for :ref:`fields <fields>`.

------------------

* Add :ref:`a framework for creating tests and running them with pytest
<web-poet-testing>`.

* Support implementing fields in mixin classes.

* Introduce new methods for :class:`web_poet.rules.RulesRegistry`:

* :meth:`web_poet.rules.RulesRegistry.add_rule`
* :meth:`web_poet.rules.RulesRegistry.overrides_for`
* :meth:`web_poet.rules.RulesRegistry.page_cls_for_item`

* Improved the performance of :meth:`web_poet.rules.RulesRegistry.search` where
passing a single parameter of either ``instead_of`` or ``to_return`` results
in *O(1)* look-up time instead of *O(N)*. Additionally, having either
``instead_of`` or ``to_return`` present in multi-parameter search calls would
filter the initial candidate results resulting in a faster search.

* Support :ref:`page object dependency serialization <dep-serialization>`.

* Add new dependencies used in testing and serialization code: ``andi``,
``python-dateutil``, and ``time-machine``. Also ``backports.zoneinfo`` on
non-Windows platforms when the Python version is older than 3.9.

------------------

In this release, the ``handle_urls`` decorator gets an overhaul; it's not
required anymore to pass another Page Object class to
``handle_urls("...", overrides=...)``.

Also, the ``web_poet.field`` decorator gets support for output processing
functions, via the ``out`` argument.

Full list of changes:

* **Backwards incompatible** ``PageObjectRegistry`` is no longer supporting
dict-like access.

* Official support for Python 3.11.

* New ``web_poet.field(out=[...])`` argument which allows to set output
processing functions for web-poet fields.

* The ``web_poet.overrides`` module is deprecated and replaced with
``web_poet.rules``.

* The ``handle_urls`` decorator is now creating ``ApplyRule`` instances
instead of ``OverrideRule`` instances; ``OverrideRule`` is deprecated.
``ApplyRule`` is similar to ``OverrideRule``, but has the following differences:

* ``ApplyRule`` accepts a ``to_return`` parameter, which should be the data
container (item) class that the Page Object returns.
* Passing a string to ``for_patterns`` would auto-convert it into
``url_matcher.Patterns``.
* All arguments are now keyword-only except for ``for_patterns``.

* New signature and behavior of ``handle_urls``:

* The ``overrides`` parameter is made optional and renamed to
``instead_of``.
* If defined, the item class declared in a subclass of
``web_poet.ItemPage`` is used as the ``to_return`` parameter of
``ApplyRule``.
* Multiple ``handle_urls`` annotations are allowed.

* ``PageObjectRegistry`` is replaced with ``RulesRegistry``; its API is changed:

* **backwards incompatible** dict-like API is removed;
* **backwards incompatible** *O(1)* lookups using
``.search(use=PagObject)`` has become *O(N)*;
* ``search_overrides`` method is renamed to ``search``;
* ``get_overrides`` method is renamed to ``get_rules``;
* ``from_override_rules`` method is deprecated;
use ``RulesRegistry(rules=...)`` instead.

* Typing improvements.
* Documentation, test, and warning message improvements.

Deprecations:

* The ``web_poet.overrides`` module is deprecated. Use ``web_poet.rules`` instead.
* The ``overrides`` parameter from ``handle_urls`` is now deprecated.
Use the ``instead_of`` parameter instead.
* The ``OverrideRule`` class is now deprecated. Use ``ApplyRule`` instead.
* ``PageObjectRegistry`` is now deprecated. Use ``RulesRegistry`` instead.
* The ``from_override_rules`` method of ``PageObjectRegistry`` is now deprecated.
Use ``RulesRegistry(rules=...)`` instead.
* The ``PageObjectRegistry.get_overrides`` method is deprecated.
Use ``PageObjectRegistry.get_rules`` instead.
* The ``PageObjectRegistry.search_overrides`` method is deprecated.
Use ``PageObjectRegistry.search`` instead.

------------------

* The BOM encoding from the response body is now read before the response
headers when deriving the response encoding.
* Minor typing improvements.