Perceval

** New features and improvements: **

* Support for Python 3.5, 3.6.
* New set of backends added:
- **GitLab**
- **Launchpad**
* `Cache` was removed in favor of `Archive`. This new feature stores, in
SQLite databases, each data response received from a remote source. Thus,
it is possible to retrieve original data again without accessing the remote
source.
* A new generic HTTP client (`HttpClient`) is available and shared by those
backends which require to fetch data using that protocol. This client manages
rate limits, sleep times and retries in case of error. It is fully extensible
and configurable.
* With the integration of categories, backends would be able to generate
different types of items. For instance, GitHub generates issue and
pull request items. The option `--category` allows to set which type of
items will be fetched.
* Gmane site shut down its activity in July 2016. Although there were some
actions to revamp it, it is still down. For these reasons, Gmane backend
is no longer maintained and has been removed from the core backends.
* Tests were improved, specially, adding unit tests for Gerrit backend.
* Perceval and GrimoireLab project are now part of CHAOSS community.

** Backend improvements: **

* **askbot**
- add data about accepted answers
* **gerrit**
- rename parameter URL to hostname
* **git**
- add `to-date` option to fetch data up to the given date
- run Git commands setting HOME environment variable
- clone data into a bare repository instead of a work copy
* **github**
- fetch issue comments
- fetch issue/comments reactions
- fetch multiple assignees
- fetch pull request category
- major refactoring reducing the number of requests sent by the client
* **phabricator**
- include project/user information in task transactions

** Bugs fixed: **

* The process for discovering references in Git repositories failed
with those repositories which do not have any. (260)
* When a local Git repository was analyzed by Perceval, the directory where
it was cloned was created inside the local repository. (262)
* Sleep times when rate limit is in use were wrongly calculated in some
cases, generating negative values. (355)
* Pipermail backend failed on inaccessible archive URLs. Now, it skips
those URLs generating warning messages. (358)

** Thanks to: **

* Anvesh Chaturvedi
* David Pose Fernández
* Prabhat Sharma

** New features and improvements: **

* DockerHub added as new backend.
* Fetch the latest commits added in a Git repository using
the argument `latest-items`.

** Bugs fixed: **

* In Slack, comment messages were not processed raising an error
when their UUIDs were computed. These messages do not include a
'user' field on the top layer, which made the backed to fail.
This field can be found inside 'comment' key.
* Some versions of gerrit return number review as an integer.
This value must be converted to string because UUIDs can only be
generated using string values. (144)

** New features and improvements: **

* Common functions used across GrimoireLab projects have been moved
to their own package. This package was named `grimoirelab-toolkit`.
From this version, Perceval depends on this package.

** Backend improvements: **

* **askbot**
- support new URLs schema for comment queries
* **bugzilla**
- set `User-Agent` header in HTTP clients
* **confluence**
- add content URL to each item
* **gerrit**
- add option to disable SSH host keys checks
* **nntp**
- raise `ParseError` exceptions when an encoding error is found
* **rss**
- set `User-Agent` header in HTTP clients

** New features and improvements: **

* New set of backends added:
- Hyperkitty
- NNTP
- Slack
* `RateLimitError` exception added for handling rate limit errors.
* Code was cleaned to follow most of the PEP8 style guidelines.

** Backend improvements: **

* **git**
- retry calls on SSH commands were added to avoid temporal server errors
* **github**
- HHTP 404 errors are managed when user's organizations are fetched
- generic `RateLimitError` exception is used

** Bugs fixed: **

* In Mediawiki backend, the log messages written when a revision is not
found were set to ERROR when the real level should have been WARNING.
* The URL used to fetch jobs in Jenkins was not common to all servers.
* When UUIDs are generated with some input data, some errors may be raised
due to problems encoding invalid characters on the input. To avoid these
problems, a surrogate escape control error has been set when data is
encoded to UTF8. (123)
* Handle Meetup requests rate limit. (126)

** Backend improvements: **

* **bugzillarest**
- messages in client errors were improved
* **git**
- new method `is_detached()`
* **mbox**
- ignore messages with invalid dates on `Date` header
* **phabricator**
- retry requests on HTTP 502 and 503 errors

** Bugs fixed: **

* The `mbox` class from Python's `mailbox` module fails when it tries to
decode non-ascii unix-from headers. This header is used as a separator
between messages. When this error is found, the class stops reading messages
from the mbox. Wrapping `mbox` class to override the way messages are read
was needed to catch the exception and decode the header using UTF-8.
* When a user does not exists on Phabricator, the API does not return an
error. It returns an empty list. The case where an empty list is returned
was not managed by the parser, which raised exceptions.
* In gerrit, the identifier of the change, `Change-Id` (or `id`), is not unique.
What it is unique in a gerrit sever is the number of each change and review.
This `number` is used now instead of `id` as the identifier of a review
* When Git repositories are reset to the current status on upstream, some of
them cannot deal with `origin` reference because it is ambiguous. Replacing
it by `FETCH_HEAD` works on those repositories with defined branches on
the origin.
* Git repositories in detached state do not need to be reset after `git fetch`
is called. This call is now skipped when a repository in this state
is in use. (105)

** New features and improvements: **

* New set of backends added:
- Askbot
- Meetup
- RSS
* Definition of `perceval.backends` namespace and dynamic loading of backends.
These two features allow to have third party backends or packages of
backends that can be imported and used at runtime.
* Mozilla's backends were moved to their own package: `perceval-mozilla`.
* Commands were refactorized generalizing their usage into `BackendCommand`
class which can run any type of backend. This was possible thanks to the
creation of `BackendCommandArgumentParser` class, that defines, manages and
parses those arguments needed to run a command; the definition of `pre_init()`
and `post_init()` methods during the initialization of the instance; and to
the implementation of `setup_cache()` as a public function of the `cache`
module.

** Backend improvements: **

* **bugzilla**
- set maximum number of bugs requested on CSV queries
* **git**
- parse commit trailers
- new methods `is_empty()` and `count_objects()`
- set missing encodings for the command output
- cleaning up of the module
* **jenkins**
- ignore invalid job builds
* **supybot**
- parse action and bot empty lines
- parse user actions with the format `*nick msg`
- generate item ids using the body of the message

** Bugs fixed: **

* The field 'timestamp' on metadata was not generated in UTC. The call
to `datetime.now()` does not generate a timestamp in UTC. It does using
the timezone of the system. The right way is to call to `datetime.utcnow()`
method. (92)
* The docker image for Perceval purged the git package after installing
`perceval`. This made impossible to run the backend for Git because
Perceval needs of `git` command under the hood. (95)
* Git empty repositories threw errors while fetching commits. Those were
raised because on empty repositories, those which do not have any history
or are only initialized, there are some commands that cannot be run, like
a pull or log. If any of this commands is called an error is be returned.
It was fixed checking whether the repository is empty and returning
an empty list of commits for those cases. (102 and 107)