Pywb

~~~~~~~~~~~~~~~~~~~~~~~

Documentation Updates:

* `Embargo + ACL system updates <https://pywb.readthedocs.io/en/latest/manual/access-control.html>`_

* `New ACL header configuration <https://pywb.readthedocs.io/en/latest/manual/usage.html#config-acl-header>`_

* `Locaalization / Multi-lingual Support Guide <https://pywb.readthedocs.io/en/latest/manual/localization.html>`_


Localization Improvements: (`647 <https://github.com/webrecorder/pywb/pull/647>`_)

* Support for extracting, updating, listing and removing localizable commands via ``wb-manager i18n`` command.

* UI: Add language switch header to all UI templates.

* Mark localizable strings in translatable in existing templates.


Access Control Improvements:

* Support for Embargo System for date-based embargo, overridable via ACL ``allow_ignore_embargo`` `642 <https://github.com/webrecorder/pywb/pull/642>`_

* Support for custom ACL 'user' specified via ``X-pywb-ACL-User`` header passed from frontend proxies.

* Fixes for exact rule matching `629 <https://github.com/webrecorder/pywb/pull/629>`_

* Fixes for ACL for auto-collections `620 <https://github.com/webrecorder/pywb/pull/620>`_


Rewriting Improvements:

* Updated YT rewriting rules `635 <https://github.com/webrecorder/pywb/pull/635>`_

* POST-to-get rewriting consistent with cdxj-indexer, wabac.js/replayweb.page `636 <https://github.com/webrecorder/pywb/pull/636>`_

* Improved fuzzy matching to ensure non-POST requests handled via fuzzy matching.

* Live web: never truncate when reading POST request to avoid hung requests! (Apply limit only on indexing


CDX Server / API Compatibility Fixes:

* XmlQuery: set WARC record length field, if available `633 <https://github.com/webrecorder/pywb/pull/633>`_

* ZipNum: Don't count pages with filter `631 <https://github.com/webrecorder/pywb/pull/631>`_

* Better handle of CDX Server HTTP status `624 <https://github.com/webrecorder/pywb/pull/624>`_

* Better handling of errors from CDX Server API with 400 `623 <https://github.com/webrecorder/pywb/pull/623>`_, `#625 <https://github.com/webrecorder/pywb/pull/625>`_, `#626 <https://github.com/webrecorder/pywb/pull/626>`_, `#630 <https://github.com/webrecorder/pywb/pull/630>`_

* Backwards compatibility of ``fl`` param `621 <https://github.com/webrecorder/pywb/pull/621>`_


Recording Redis Dedup mode:

* Fix dedup index config loading `617 <https://github.com/webrecorder/pywb/pull/617>`_

* Add recording size counter to track any in-flight requests `637 <https://github.com/webrecorder/pywb/pull/637>`_

~~~~~~~~~~~~~~~~~~~~~

* Update to latest wombat.js (3.0.3)

* Dedup Mode: Support for Redis-based dedup index to skip or write revisit records for duplicates, replay from Redis-based index `597 <https://github.com/webrecorder/pywb/pull/597>`_, `#611 <https://github.com/webrecorder/pywb/pull/611>`_

* Rewriting: Updated Rules for youtube and vimeo replay `610 <https://github.com/webrecorder/pywb/pull/610>`_

* CDX Indexing: More efficint cdx sorting `609 <https://github.com/webrecorder/pywb/pull/609>`_

* Set default CDX closest lookup limit to 100 instead of 10 `606 <https://github.com/webrecorder/pywb/pull/606>`_

* UI: Try to avoid css class conflicts in injected banner `604 <https://github.com/webrecorder/pywb/pull/604>`_

* Catch invalid headers in uWSGI `603 <https://github.com/webrecorder/pywb/pull/603>`_

* Config option to support certificate validation when capturing `596 <https://github.com/webrecorder/pywb/pull/596>`_

* Fix indexing POST requests with multipart/form-data without boundary `599 <https://github.com/webrecorder/pywb/pull/599>`_

* New OpenWayback->pywb Transition Guide: `https://pywb.readthedocs.io/en/latest/manual/owb-transition.html <https://pywb.readthedocs.io/en/latest/manual/owb-transition.html>`_

* Sample deployments with Docker Compose for running with Apache, Nginx and OutbackCDX in ``sample-deploy`` directory.

* Update to latest gevent to fix issues with latest python `583 <https://github.com/webrecorder/pywb/pull/583>`_

~~~~~~~~~~~~~~~~~~~~~

* ensure RemoteCDXIndexSource also passes ``matchType`` to upstream

* cdx-indexer: use ``-o`` flag to specify output, not first param (output to stdout by default)

* static paths cleanup, move ``url-polyfill.min.js`` to correct dir (fixes `571 <https://github.com/webrecorder/pywb/issues/571>`_)

* minor fixes to docs

* logo: resize new logo to actual size, add logo via absolute link to ensure it works on pypi also

~~~~~~~~~~~~~~~~~~~~~

* Minor fix: allow timegate content check in `564 <https://github.com/webrecorder/pywb/pull/564>`_ to be ignored (for use with derived classes)

~~~~~~~~~~~~~~~~~~~~~

This release includes significant update, specifically merging of https://github.com/ukwa/pywb branch into this release.
A few selected improvements:

* New Access Control System: https://pywb.readthedocs.io/en/latest/manual/access-control.html

* Support for Localization, configuring multiple languages (not enabled by default): https://github.com/ukwa/ukwa-pywb/blob/master/docs/localization.md

* Support for OpenWayback-style XML-based index source (xmlquery)

* Support for loading from WebHDFS via `webhdfs://` scheme.

* Initial support for a new embeds/transclusions replay system, in combination with warcit: https://github.com/webrecorder/warcit/wiki/Warcit-Video-Audio-Conversion

* Proxy mode improvements: handle OPTIONS requests and CORS `520 <https://github.com/webrecorder/pywb/pull/520>`_

* Memento Prefer header: support for experimental `Prefer` header to select 'raw' or 'rewritten' memento

* Other memento fixes: fix timemap including invalid mementos, correct timegate behavior on top frame `564 <https://github.com/webrecorder/pywb/pull/564>`_

* Fixes for collection metadata display: `509 <https://github.com/webrecorder/pywb/pull/520>`_

* Fix for incorrected WARC record length due to re-serialized headers: `561 <https://github.com/webrecorder/pywb/pull/561>`_

* Filter invalid WARC records `536 <https://github.com/webrecorder/pywb/pull/536>`_

* Updated fuzzy matching rules and wombat client-side rewriting.


For the full changelist, see this PR: `565 <https://github.com/webrecorder/pywb/pull/565>`_

* Access Control System

~~~~~~~~~~~~~~~~~~~~~

* General auto-fetch fixes (503)
- Fixed issue that caused HTTP 404 errors to happen when parsing <link> stylesheet hrefs as sheets (webrecorder/wombat 11)
- Ensured that requests made are cached by the browser (webrecorder/wombat 13 & 15)
- Ensured that the request made by the backing web worker when in proxy mode are not blocked by CORS (webrecorder/wombat 13 & 15)

* SOCKS proxy fixes (504)
- simplify SOCKS config (avoiding global socket monkey patch), default to no cert verify to match non-proxy behavior
- SOCKS proxy can be disabled dynamically by setting SOCKS_DISABLE