Pywb

~~~~~~~~~~~~~~~~~~~~~

See the docs at https://pywb.readthedocs.org for more info.

**TODO: more detailed changelist**

~~~~~~~~~~~~~~~~~~~~~~

* Minor fixes from pull requests:
- Better handling of exceptions from in wsgi_wrapper
- Fix CommonCrawl tests
- Fix broken links in README
- Fix travis build (requires certauth<1.2)

~~~~~~~~~~~~~~~~~~~~~~

* Client Rewriting Improvements:
- Better rules for Instagram, Medium
- Fix window.fetch() override
- Work on eval() override (disabled for more testing)

* Add Python 3 classifiers to setup.py

~~~~~~~~~~~~~~~~~~~~~~

* Client-Side Rewriting Improvements:
- Video: More aggressive ``youtube-dl`` rewriting, try video query for any ``<object>`` with flashvars
- proxy: disable most client side rewriting when in proxy mode, keep non-rewriting overrides (random, Date)
- host relative extract: ``extract_orig()`` returns host-relative if url starts with ``/``
- add geolocation and notifications overrides to (auto-disable)
- proxy: use current protocl for video info query.
- fix history check bug: support changing history to exact current origin.
- add ``window.fetch()`` override
- add ``srcset`` attribute rewriting
- ajax: don't add ``X-Pywb-Requested-With`` header to ``data:`` urls
- general JS fixes, add undefined checks before acccessing ``_wb_js``, top frame, and content frame.

* Server-Side Rewriting Improvements:
- www canonicalization: improve regex to include urls containing ``\r``
- memento: fix potential duplicate memento headers
- proxy: when in proxy mode, only rewrite headers related to encoding or cache
- proxy: add special 'proxy_js' rewriter which defaults to no rewriting for proxy mode but allows custom JS rules to still be applied. Used for JS and embedded JS in html.
- WbUrl: add new modifier form starting with ``$`` in addition to ending with ``_``, eg. ``/$mod:foo/http://example.com/``
- ajax: don't rewrite ``text/html`` responses retrieved by ajax requests (when ``X-Pywb-Requested-With`` header is present).

* Static Handler: if ``wsgi.file_wrapper`` fails, fallback to direct streaming of static ocntent.

~~~~~~~~~~~~~~~~~~~~~~

* Template Responses: Calculate ``Content-Length`` correctly from encoded utf-8 text length

* WbUrl: Improved detection of url scheme, don't treat ``a.co/?http://foo`` as having a valid scheme

~~~~~~~~~~~~~~~~~~~~~~

* Cross-Domain Framed Replay
- pywb banner (outer) and content (inner) frames can be served from different domains
- All cross-frame interaction done via ``postMessage``, including url, hash, cookie change notifications

* Server-Side Rewriting:
- Don't rewrite relative urls (unless contain ``../`` or start with ``/``)
- Rewrite svg ``<image>`` tag
- Don't rewrite ``Proxy-Authenticate`` or ``WWW-Authenticate`` headers
- Rewrite ``href`` on any element
- Preserve HTML entities and spaces when rewriting CSS urls
- Content detect: handle ``text/plain`` text as JS or CSS if ``js_`` or ``cs_`` modifiers used
- Improved rewriting of ``on*`` attributes, ensure ``window.`` is added when accessing rewritten objects.

* Client-Side Rewriting:
- Add cookie notification message for cookies with ``Domain=`` to allow server-side handling
- Improved handling of Unicode prefixes, use ``decodeURI``
- History API: properly override go, forward, back and preserve pushState/replaceState
- Ensure client-rewriting for windows created by ``window.open``
- Override ``navigator.sendBeacon``
- Rewrite ``poster`` attr in dynamic elems
- Rewrite ``src`` attr in video ``source`` elems

* Record Loader: Option to convert ARC->WARC records implicitly, return WARC responses (enabled by default)

* Block Loader: Raise exceptions for 4xx or 5xx responses

* CDX API: return not found CDX error as JSON or plain text if using ``output=json`` or ``output=text``