W3lib

-------------------

- Add the ``encoding`` and ``path_encoding`` parameters to
:func:`w3lib.url.safe_download_url` (issue 118)
- :func:`w3lib.url.safe_url_string` now also removes tabs and new lines
(issue 133)
- :func:`w3lib.html.remove_comments` now also removes truncated comments
(issue 129)
- :func:`w3lib.html.remove_tags_with_content` no longer removes tags which
start with the same text as one of the specified tags (issue 114)
- Recommend pytest instead of nose to run tests (issue 124)

-------------------

- Fix url_query_cleaner to do not append "?" to urls without a query string (issue 109)
- Add support for Python 3.7 and drop Python 3.3 (issue 113)
- Add `w3lib.url.add_or_replace_parameters` helper (issue 117)
- Documentation fixes (issue 115)

-------------------

- Add a workaround for CPython segfault (https://bugs.python.org/issue32583)
which affect w3lib.encoding functions. This is technically **backwards
incompatible** because it changes the way non-decodable bytes are replaced
(in some cases instead of two ``\ufffd`` chars you can get one).
As a side effect, the fix speeds up decoding in Python 3.4+.
- Add 'encoding' parameter for w3lib.http.basic_auth_header.
- Fix pypy testing setup, add pypy3 to CI.

-------------------

- Include additional assets used for distribution packages in the source tarball
- Consider ``[`` and ``]`` as safe characters in path and query components
of URLs, i.e. they are not escaped anymore
- Disable codecov project coverage check

-------------------

- Add Python 3.5 and 3.6 support
- Add ``w3lib.url.parse_data_uri`` helper for parsing "data:" URIs
- Add ``w3lib.html.strip_html5_whitespace`` function to strip leading and
trailing whitespace as per W3C recommendations, e.g. for cleaning
"href" attribute values
- Fix ``w3lib.http.headers_raw_to_dict`` for multiple headers with same name
- Do not distribute tests/test_*.pyc artifacts

-------------------

- ``canonicalize_url()`` and ``safe_url_string()``:
strip ":" when no port is specified (as per `RFC 3986`_;
see also https://github.com/scrapy/scrapy/issues/2377)
- ``url_query_cleaner()``: support new ``keep_fragments`` argument
(defaulting to ``False``)