Urlwatch

Added

- A job can now have a `diff_filter` set, which works the same way as the normal
`filter` (and has the same filters available), but applies to the `diff` output
instead of the page content (can be tested with `--test-diff-filter`, needs 2
or more historic snapshots in the cache)
- Documentation now has a section on the configuration settings (`--edit-config`)
- New filter: ``ocr`` to convert text in images to plaintext (using Tesseract OCR)
- New reporters:
- ``ifttt`` to send an event to If This Then That (ifttt.com) (512, by Florian Gaultier)
- ``xmpp`` to send a message using the XMPP (Jabber) protocol (533, by Thorben Günther)

Changed

- The `urlwatch` script (Git only) now works when run from different paths
- Chunking of strings (e.g. for Slack and Telegram) now adds numbering (e.g.
` (1/2)`) to the messages (only if a message is split into multiple parts)
- Unit tests have been migrated from `nose` to `pytest`
and moved from `test/` to `lib/urlwatch/tests/`
- The ``css`` and ``xpath`` filters now accept ``skip`` and ``maxitems`` as subfilter
- The ``shellpipe`` filter now inherits all environment variables (e.g. ``$PATH``)
of the ``urlwatch`` process

Fixed

- The ``html2text`` method ``lynx`` now treats any subfilters with a non-``null``
value as command-line argument ``-key value`` (previously only the value ``true``
was treated like this, and any other values were silently dropped)

Added

- Documentation is now available at [urlwatch.readthedocs.io](https://urlwatch.readthedocs.io)
and shipped in the source tarball under `docs/`; filter examples in the docs are unit-tested
- New filters:
- `reverse`: Reverse input items (default: line-based) with optional `separator`
- `pdf2text`: Convert PDF files to plaintext (must be first filter in chain)
- `shellpipe`: Filter text with arbitrary command-line utilities / shell scripts
- `FilterBase` API improvements for specifying subfilters:
- Add `__supported_subfilters__` for sub filter checking and `--features` output
- Add `__default_subfilter__` to map value-only parameters to dict parameters,
for example the `grep` filter now has a default subfilter named `re`
- Support for using Redis as a cache backend via `--cache=redis://localhost:6379/`

Fixed

- Declare updated Python 3.5 dependency in `setup.py` (already a requirement since urlwatch 2.18)

Changed

- Filter improvements:
- `sort`: Add `reverse` option to reverse the sorting order
- `sort`: Add `separator` option to specify item separator (default is still line-based)
- `beautify`: The `jsbeautifier` (for `<script>` tags) and `cssbeautifier` (for `<style>` tags)
module dependencies are now optional - if they are not installed, beautify only works on the HTML
- Most filters that only had unnamed subfilters (e.g. `grep`) now have a named default subfilter
- Reporter improvements:
- ``pushover``: The message ``priority`` can now be configured
- Travis CI: Set `pycodestyle` version to 2.6.0 to avoid CI breakage when new style checks are added
- Diff results are now runtime cached on a per-job basis, which shouldn't affect behavior, but
could be observed by an external `diff_tool` running at most once per job instead of multiple times
- Jobs with a custom `diff_tool` or a `shellpipe` filter are now ignored if `jobs.yaml` has the
world-writable bit (`o+w`) set or is not owned by the current user (does not apply to Windows);
previously only `shell` jobs were ignored if the permissions/owners were wrong

Deprecated

- String-based filter definitions (e.g. `html2text,grep:Current.*version,strip`) are now
deprecated, it is recommended to use YAML-based dictionary-style listing of filters,
which is more flexible, easier to read and write and more structured

Added
- New filter: `re.sub` that can replace/remove strings using regular expressions
- Support `ignore_timeout_errors` and `ignore_too_many_redirects` for URL jobs (423, by Josh aka Zevlag)
- HTML reporter: Add `viewport` meta tag for improved viewing on mobile devices (432, by Mike Borsetti)
- Optional support for insecure SMTP password storage in the config; use with caution (431)
- Add `matrix` reporter
- New filter: `beautify` that can beautify HTML, JavaScript and CSS

Fixed
- Fix `--test-filter` when the specified job is not found
- Fix another `YAMLLoadWarning` in unit tests (382, by Louis Sautier)
- Documentation updates and typo fixes (by Nate Eagleson)
- Pushover: Fix default device config (Fixes 409 and 372, documented by Richard Goodwin)

Changed
- Nicer formatting of `--features` for jobs with no docstring or many keys
- The XPath and CSS filters now support XML namespaces (404, by Chenfeng Bao)
- Drop support for Python 3.3 and Python 3.4 (new minimum requirement is Python 3.5)
- Use `html.escape` instead of `cgi.escape` (which was removed in Python 3.8; 424, by Chenfeng Bao)
- Allow non-ASCII characters in `format-json` output filter (433, by Mike Borsetti)
- The `keyring` config option for `email` was changed to `auth`; if you have problems
with authentication where none is needed, set `report/email/smtp/auth` to `false`

Added
- XPath/CSS: Support for excluding elements (333, by Chenfeng Bao)
- Add support for using external `diff_tool` on Windows (373, by Chenfeng Bao)
- Document how to use Amazon Simple E-Mail Service "SES" (by mborsetti)
- Compare data with multiple old versions (`compared_versions`, 328, by Chenfeng Bao)

Fixed
- YAML: Fix deprecation warnings (367, by Florent Aide)
- Updated manpage with new options: Authentication, filter tests (Fixes 351)
- Text formatter: Do not emit empty lines for `line_length=0` (Fixes 357)

Changed
- SMTP configuration fix: Only use smtp.user config if it's a non-empty value

Added
- XPath: Handle `/text()` selector (282)
- Document how to specify cookies to README.md (264)
- Text Reporter: `minimal` config option to only print a summary (PR304, fixes 147)
- README.md: Document how to watch Github releases via XPath (266)
- Support for parsing XML/RSS with XPath (Fixes 281)
- Allow explicit setting of `encoding` for URL jobs (PR313, contributes to 306)
- Slack Channel Reporter (PR309)
- ANSI color output on the Windows console via `colorama` (PR296, closes 295)
- Support for using CSS selectors via the `cssselect` module (PR321, closes 273)
- `ignore_http_error_codes` is now an option for URL jobs (PR325, fixes 203)
- `job_defaults` in the config for globally specifying settings (PR345, closes 253)
- Optional `timeout` (in seconds) for URL jobs to specify socket timeout (PR348, closes 340)

Removed
- Support for JSON storage (dead code that was never used in production; PR336)

Changed
- `HtmlReporter` now also highlights links for browser jobs (PR303)
- Allow `--features` and `--edit-*` to run without `urls.yaml` (PR301)
- When a previous run had errors, do not use conditional GETs (PR313, fixes 292)
- Explicitly specify JSON pretty print `separators` for consistency (PR343)
- Use data-driven unit tests/fixtures for easier unit test maintenance (PR344)

Fixed
- Fix migration issues with case-insensitive filesystems (223)
- Correctly reset retry counter when job is added or unchanged (PR291, PR314)
- Fix a `FutureWarning` on Python 3.7 with regard to regular expressions (PR299)
- If the filter list is empty, do not process the filter list (PR308)
- Fix parsing/sanity-checking of `urls.yaml` after editing (PR317, fixes 316)
- Fix Python 3.3 compatibility by depending on `enum34` there (PR311)
- Allow running unit tests on Windows (PR318)
- Fix migration issues introduced by PR180 and 256 (PR323, fixes 267)

Added
- Support for Mailgun regions (by Daniel Peukert, PR280)
- CLI: Allow multiple occurrences of 'filter' when adding jobs (PR278)

Changed
- Fixed incorrect name for chat_id config in the default config (by Robin B, PR276)