Pywb

~~~~~~~~~~~~~~~~~~~~~~

* Rewriting: if no charset specified in original page, don't add charset to allow browser to detect.

* Rewriting: rewrite ``<meta content="">`` attribute if it is a url.

* wb.js: pad shorter timestamp to 14 digits.

* Indexing: fixed exception when indexing empty files.

~~~~~~~~~~~~~~~~~~~~~~

* WombatLocation: overriden properties (href, host, etc...) are enumerable to match Location to support cloning methods.

* WombatLocation: reload() override now works.

* Proxy: Custom ``Pywb-Rewrite-Prefix`` allows adding a custom prefix for proxy mode rewriting

* Proxy: Better error for invalid collection in ip resolve mode

* Warc Indexing Refactor: Allow custom iterators to buffer payload by overriding ``create_payload_buffer()`` to return a writable buffer.

~~~~~~~~~~~~~~~~~~~~~~

* New client-side test system for Wombat.js in place using Karma and SauceLabs with initial set of tests and travis integration.

* Wombat Improvements:
- Better Safari/IE support: accessors overriden only when actually supported in browser, override gracefully skipped otherwise
- Use ``getOwnPropertyDescriptor()`` to get properties in addition to ``__lookupGetter__``, ``__lookupSetter__``
- ``baseURI`` overriden on correct prototype
- ``CSSStyleSheet.href`` override
- ``HTMLAnchorElement.toString()`` override
- Avoid making ``<base>.href`` read-only

* Proxy Mode Improvements:
- To avoid breaking HTTPS envelope, if no content-length provided, chunked encoding is used (HTTP/1.1) or response is buffered and content-length is computed (HTTP/1.0)
- Rewriter: Scheme-only rewriter converts embedded urls to http or https to match the scheme of containing page.
- IP Resolver: Supports IP cache in Redis
- Default resolver set to cookie resolver, eg. ``cookie_resolver: true`` is the default.
- Collection/datetime switching options removed from UI when auth or ip resolvers.

* Encoding: Use webencoding lib to better encode head-insert to match page encoding

* Live Proxy: Support for explicit recording mode, decoupled from using http/https proxy. Enabled when ``LiveRewriter.is_recording()`` is true. By default, http/s proxies imply recording but can be overriden in derived class.

* Rewriting: Convert relative urls for ``rel=canonical`` to absolute urls, even if not rewriting to ensure correct url.

* UI: Use custom webkit scrollbars to minimize scrollbar-in-iframe issues that sometimes occur in Chrome.

* Memento Improvements:
- ``/collinfo.json`` by default returns a JSON spec for all collections as Memento endpoints, in a format compatible with MemGator.
- ``Add /collinfo.json`` endpoint customizable via ``templates/collinfo.json`` and must be enabled with ``enable_coll_info: true``
- 'Not Found' error for timemap query returns empty timemap instead of standard HTML 404.

* WARC Indexing:
- Better detection of content-length < payload, skip to next record boundary and warn, if possible.
- Use ujson if proper version (without forward-slash escaping) is available when writing CDXJ

~~~~~~~~~~~~~~~~~~~~~~

* extensible BlockLoadres: supported 'http', 'https', 's3' and local file system, additional
loaders can now be registered by scheme.

* rewriting fixes:
- wombat: fix occasional style rewrite bug that resulted in leaks.
- strip leading or trailing spaces in url
- charset: default to utf-8 if unknown charset specified in HTML

* live rewrite: LiveRewriter class overridable in config

* WARC indexing: ignore empty records when indexing and continue, rather than stopping at first empty record.

* tests: refactor integration tests to run signficantly faster.

* cdx-indexer

~~~~~~~~~~~~~~~~~~~~~~

* wombat: fix relative '/' rewrite which incorrectly handles rel scheme '//' urls

~~~~~~~~~~~~~~~~~~~~~~

* IPProxyResolver: Support new simple proxy resolver where collection and timestamp stored in server-side cache by IP and set via a rest api through `pywb.proxy` eg: ``curl -x "localhost:8080" http://pywb.proxy/set?ts=2015&coll=all``. No cookies or proxy auth needed in this mode. Useful for Docker-based deployments where virtual IP is fixed. Enabled with ``cookie_resolver: ip`` in ``proxy_options``.

* CDX Server: Add support for timestamp-bounded queries CDX queries ``from=`` and ``to=``, also support calendar query with (inclusive) ranges, eg. ``/2010-2015/example.com``, ``/2010-/example.com/``, ``/-2015/example.com/``.

* Proxy options: add ``use_banner`` to toggle banner insert, and ``use_client_rewrite`` to toggle wombat rewriting in proxy mode. (Client rewriting requires banner insert).

* Proxy and Video: When in proxy mode, load youtube-dl video info via proxy magic host `pywb.proxy`, and ensure CORS support.

* Rewrite: ensure ``<base>`` tag has trailing slash, or add ``<base>`` with trailing slash for host-name only urls, eg: ``http://localhost:8080/example.com``

* Rules: improved blogspot nav and yt rules, rule file cleanup

* Wombat 2.9 improvements, including:

- improved handling of relative paths, '..', '.', '/'
- better support for proxy mode, avoid cross-origin top-frame issues
- rewrite_html() (document.write) override only if any html changed
- improved form action rewrite
- improved rewriting in 'root collection' mode