Annif

This release includes a new trainable ensemble backend based on a neural network ([nn_ensemble](https://github.com/NatLibFi/Annif/wiki/Backend%3A-nn_ensemble)) and support for default parameters for backend configuration settings, making it easier to configure projects. There are also several performance and memory usage optimizations (e.g. TFIDF training is now around 5x faster!) and some bug fixes.


New features:

* 330/331 Neural network ensemble backend
* 273/324 Default values for backend configuration

Improvements:

* 340 Reduce vector memory usage
* 335 Optimizations to tfidf backend training
* 336 Further optimizations to tfidf backend + rearchitecting
* 318/322 Handle missing or invalid path input in CLI
* 334 Use layer caching in Docker image builds in Drone

Bug fixes:

* 332/333 Eval crash on TFIDF with multiple training files
* 341 Fix Docker image build error by pinning base image version

This release brings improvements to command line functionality and many robustness enhancements. The Docker images in the [quay.io repository](https://quay.io/repository/natlibfi/annif?tab=tags) now have tags in line with release versions and the documentation in ReadTheDocs has both [stable](https://annif.readthedocs.io/en/stable/) (latest release) and [latest](https://annif.readthedocs.io/en/latest/) (current `master`) versions. There are also a lot of small improvements to the Docker image build pipeline.

New features:
- 251/306 CLI `clear` command for clearing project data

Robustness improvements:
- 281/299 Handle invalid lines in training data `.tsv` files
- 282/300 Handling errors in projects.cfg file
- 325 Always use UTF-8 encoding when parsing TSV file document corpora
- 291/296/317 Pin down more specific versions of dependencies

Bug fixes:
- 303/309/313 Fix typos in documentation and bug in training from `.key` files with only subject labels
- 320 Turn AnnifException into an abstract base class and fix message prefix behavior

Container pipeline and ReadTheDocs build related fixes and enhancements
323, 312, 311, 310, 305, 304, 297/298, 314, 319, 295

This release brings support for [running Annif within a Docker container](https://github.com/NatLibFi/Annif/wiki/Usage-with-Docker), [online API documentation published on readthedocs.org](https://annif.readthedocs.io/en/latest/index.html), [a new Vowpal Wabbit based ensemble backend](https://github.com/NatLibFi/Annif/wiki/Backend%3A-vw_ensemble) (still experimental) and a few other small enhancements and fixes. New in this release:

* 278/279/293 Dockerize Annif
* 277/280/285/287 Auto-generate API documentation and publish it on readthedocs.org
* 235/284 Add `vw_ensemble`, Vowpal Wabbit based ensemble backend
* 210/283 Command line option to set path to `projects.cfg`
* 275 Fix joblib warning. Credit: kinow
* 276 Avoid calling update with empty dicts. Credit: kinow

This is a cleanup and bugfix release that renames some methods and commands but brings no real new functionality. New in this release:

* 270/272 The CLI command and REST API method `analyze` has been renamed to `suggest` and the CLI command `analyzedir` to `index`. Also some internal data structures were renamed accordingly. The idea is to clarify terminology so that from now one *analyzing* means only the **preprocessing** (e.g. tokenizing and stemming) done by Analyzers, not the whole operation of assigning/suggesting subjects for a document. For some background, see this [post on annif-users](https://groups.google.com/d/msg/annif-users/tHyPIifu7wo/_Uj-SDTSCwAJ)
* 262/263 Fix some UTF-8 encoding issues, especially on Windows. Now all input files are processed as UTF-8 regardless of the system character set. Credit: Veldhoen
* 271 Clean up unused imports and declare transitive dependencies. Credit: kinow

New in this release:

* 230 / 249 / 252 / 253 / 254 / 256 New `vw_multi` backend based on the [Vowpal Wabbit](https://github.com/VowpalWabbit/vowpal_wabbit) machine learning system, which has to be installed as an optional dependency. This backend is a wrapper for the `oaa`, `ect`, `log_multi` and `multilabel_oaa` multiclass and multilabel classification algorithms implemented by VW. It can use either the document text or the output of other projects (e.g. subjects as determined by another project/backend/model) as the basis for classification.
* 225 / 257 Support for **online learning**, for now implemented only for the new `vw_multi` backend. This includes the new `learn` CLI command as well as a `learn` method in the REST API that can be used to provide additional indexed documents (with subjects/classes) for updating the model.
* 245 The web UI now uses Vue.js instead of jQuery for most dynamic functionality. Credit: kinow
* 258 The web UI can show detailed error messages returned by the REST API. Credit: kinow
* 237 / 246 Access levels for projects: you can now define projects as hidden (they will not be shown by the REST API `projects` list but can still be used if you know the project id) or private (not available via REST at all)
* 247 / 248 Fix bug where `eval` command sometimes reported `nan` values
* Some code cleanup and refactoring

Annif now also has a DOI: https://doi.org/10.5281/zenodo.2578948 - thanks to the [integration between Zenodo and GitHub](https://guides.github.com/activities/citable-code/). This DOI represents all versions of Annif and will always resolve to the latest release; additional DOIs are generated by Zenodo for each specific release from now on.

This is a bugfix release with the following fixes:

* 242 Fix invalid output of `show-project` command
* 243 Really fix the high memory usage problem in `optimize` command
* show more helpful warning when trying to use a backend (fasttext) without the necessary dependencies
* some code cleanup and additional unit tests; coverage is now >99%