Manubot

Manubot version 0.3.0 updates the schema of output variables & metadata for the `manubot process` command. Now, Pandoc's `header-includes` metadata field is set to provide manuscript-specific metadata that improves indexing by bibliographic databases and assists sharing on social media.

The terminology around citations has been updated. We now refer to identifiers for specific references as "citekeys" or "citation keys". The following external-facing functions have updated names: `manubot.cite.citekey_to_csl_item` and `manubot.cite.standardize_citekey`.

There is a new subcommand `manubot webpage` for managing creation of a webpage directory for manuscripts.

Commits
-------

- a9fc2ea Log user-provided variables at debug level (179)
- 2ffcec1 don't encapsulate header-includes in code block
- e892999 Set Pandoc's header-includes with HTML `<meta>` (138)
- 23e687a Revise metadata.yaml processing (175)
- 74f7a85 Update copyright assertion statement
- 98de2bd miscellaneous edits from gitter reviews
- ebac7ab process: combined pandoc & manubot metadata via load_variables (173)
- 49343c7 readme: improve development commands (171)
- 785b4aa move is_http_url to manubot.util (174)
- 9023b7f process: set / detect manuscript thumbnail image (169)
- 0b7c14a Apply black style to Python codebase (164)
- 43be195 webpage: fix `process` referenced before assignment (166)
- ddd0995 Function to get doi metadata via translation-server
- d5f9871 Travis CI: test on Python 3.8 (162)
- 2f494fe skip test_get_continuous_integration_parameters_* on forks (160)
- 94e3f4c Migrate standardize ids & notes to CSL_Item class (157)
- 04a3574 Generate docs and deploy to gh-pages (153)
- 065ae7f Migrate csl_item_passthrough to CSL_Item.clean (156)
- 40e5b7a test_cite_command_render_stdout: version expected files
- 1aa21dc CSL_Item class for bibliographic metadata of a single reference
- 75b259e TST: pytest option to cache requests (151)
- 0478ce2 Switch pytest to use verbose & colored output (154)
- cf9564d split cite.util into csl_item & citekey submods
- 307b339 TST: read files as utf-8-sig (143)
- 71d4d39 Travis: use API token for PyPI authorization
- df480ae Fix webpage warning during ots upgrade (134)
- 23a0841 Adopt "citekeys" terminology (129)
- efb0adb webpage redirect-template.html: remove blank line
- 00405aa manubot webpage subcommand (132)
- 1b885c6 process.ci: generalize CI variables & support AppVeyor

Contributors
-----------

Authors of commits included in this release:

- [Daniel Himmelstein](https://github.com/dhimmel)
- [Evgeny Pogrebnyak](https://github.com/epogrebnyak)
- [Olga Botvinnik](https://github.com/olgabot)

Manubot version 0.2.4 contains various enhancements and improvements to the citation processing workflow.

New features
------------

Create the new `manubot.pandoc` submodule with code for interacting with the system Pandoc installation (see [GH103](https://github.com/manubot/manubot/pull/103)). This module creates an organized location for Pandoc-related code, which will help with development of creating a Pandoc filter for citation-by-identifier (see [GH99](https://github.com/manubot/manubot/pull/99)).

Manual references can now be supplied in formats other than CSL JSON. Formats supported by `pandoc-citeproc --bib2json` can now be supplied to `manubot process`. See [GH100](https://github.com/manubot/manubot/issues/100) and [GH104](https://github.com/manubot/manubot/pull/104).

Changes
-------

Additional refactoring of the `manubot.cite` submodule has moved the package closer to a well-defining processing pipeline for citations ([GH113](https://github.com/manubot/manubot/pull/113) and [GH114](https://github.com/manubot/manubot/issues/114)). The column names in `citations.tsv` changed to [`manuscript_id`, `detagged_id`, `standard_id`, `short_id`].

Make any missing parent directories for the `--output-directory` and `--cache-directory` arguments of `manubot process`. See [GH102](https://github.com/manubot/manubot/issues/102) and [GH115](https://github.com/manubot/manubot/pull/115).

Read text files using the `utf-8-sig` encoding (to strip BOMs if present). Write text files using `utf-8` encoding. UTF-8 ensures compatibility with Pandoc, which uses it for I/O. Also keeps operation consistent across files / platforms. See [GH125](https://github.com/manubot/manubot/issues/125) and [GH127](https://github.com/manubot/manubot/pull/127).

The README has been updated with improved installation instructions and the Manubot software paper citation. See [GH118](https://github.com/manubot/manubot/pull/118) and [GH121](https://github.com/manubot/manubot/pull/121).

Manubot version 0.2.3 contains various enhancements. In addition, the source code location has moved from <https://github.com/greenelab/manubot> to <https://github.com/manubot/manubot> (see [GH94](https://github.com/manubot/manubot/issues/94)).

New features
------------

Citations of shortDOIs are now supported (see [GH92](https://github.com/manubot/manubot/issues/92) and [GH93](https://github.com/manubot/manubot/pull/93)). [shortDOIs](http://shortdoi.org/), which start with `10/` rather than `10.`, can now be cited just like a DOI. For example, `doi:10/gddkhn` is a supported citation. Manubot expands shortDOI citations to their regular DOIs, e.g. `doi:10.1098/rsif.2017.0387`, such that `manubot process` will treat both the short and regular form as the same citation.

Bug fixes
---------

Queries to Manubot's translation-server now specify `single=1` to enforce returning a single record per persistent identifier (see [GH90](https://github.com/manubot/manubot/pull/90)). Previously, multiple results were [sometimes](https://github.com/zotero/translation-server/issues/65) returned, causing Manubot's JSON CSL retrieval to fail. Furthermore, Zotero child notes are now ignored, fixing another [failure mode](https://github.com/zotero/translation-server/issues/67) for CSL export of Zotero metadata.

Null authors are now allowed in `metadata.yaml` and do not crash Manubot with a TypeError (see [GH91](https://github.com/manubot/manubot/pull/91)).

The codebase has been updated to avoid deprecation warnings in Pandas v0.24 (see [GH95](https://github.com/manubot/manubot/pull/95)).

Setting the header will help upstream resources contact the Manubot developers should our requests be problematic or should downtime be anticipated. Furthermore, it will allow Manubot's translation-server to monitor Manubot usage, including which operating system, Python version, and package version are being used.

Manubot's test suite has been reorganized such that testing modules correspond one-to-one with package modules (see [GH87](https://github.com/greenelab/manubot/pull/87)).

Manubot version 0.2.1 contains several improvements to the package's citation infrastructure.

New features
------------

Support has been added for `raw` citations for references without supported persistent identifiers (see [GH62](https://github.com/greenelab/manubot/issues/62) and [GH74](https://github.com/greenelab/manubot/pull/74)). Raw citations require the user to manually specify the corresponding CSL JSON.

Error messages for invalid citations have been improved (see [GH76](https://github.com/greenelab/manubot/pull/76) and [GH71](https://github.com/greenelab/manubot/issues/71)). More types of incorrect citations are now caught internally before any external APIs are queried to retrieve metadata.

The `manubot cite` command has been updated to generate metadata for all valid citations, while logging error messages for invalid citations (see [GH77](https://github.com/greenelab/manubot/pull/77)). Previously, a single invalid citation would cause the program to exit before outputting references for valid citations.

Bug fixes
---------

Previously, metadata for `pmcid` citations was retrieved from the NCBI Citation Exporter. This service was [taken offline](https://twitter.com/dhimmel/status/1061787168820092929) without notice causing citation retrieval to fail. NCBI replaced the previous service with the [Literature Citation Exporter](https://api.ncbi.nlm.nih.gov/lit/ctxp). The `manubot.cite.pubmed.get_pmc_citeproc` function has been changed to use the new service (see [GH80](https://github.com/greenelab/manubot/pull/80)).

Previously, CSL JSON Items were being generated with empty `date-parts` arrays, which would cause pandoc-citeproc to fail. Manubot's CSL JSON pruning infrastructure has been updated to delete empty `date-parts` arrays (see [GH66](https://github.com/greenelab/manubot/issues/66) and [GH65](https://github.com/greenelab/manubot/pull/65)).

Entrez E-Utils returned integer-encoded months for certain `pmid` citations causing `citeproc_from_pubmed_article` to fail. Both integer and character month encodings are now supported (see [GH72](https://github.com/greenelab/manubot/pull/72)).

Manubot version 0.2.0 introduces subcommands to the command line interface. The previous command `manubot` to process manuscript content for Pandoc is now `manubot process`. New functionality has been added via the `manubot cite` subcommand to retrieve bibliographic metadata for citations (see [GH37](https://github.com/greenelab/manubot/issues/37)). The cite subcommand can either return CSL JSON (default) or formatted references (--render, requires Pandoc, see [GH48](https://github.com/greenelab/manubot/issues/48)).

This release adds support for removing invalid fields from CSL JSON Data, which is enabled by default (see [GH47](https://github.com/greenelab/manubot/issues/47)). Previously, certain citeproc APIs returned CSL JSON with extra fields or fields with invalid values according to the CSL JSON Schema. Now CSL JSON is validated against the schema, with invalid fields removed.

Now PMID & PMCID fields are automatically populated when generating CSL data for DOIs (see [GH45](https://github.com/greenelab/manubot/issues/45)). CSL for DOIs now uses shortDOIs in the URL field.

As this package now supports more varied use cases and workflows, the code has been refactored to use lazy imports (see [GH56](https://github.com/greenelab/manubot/pull/56)). Most functions directly under `manubot.cite` and `manubot.process` have been moved to `util` submodules. `manubot.cite.citation_to_citeproc` and `manubot.cite.standardize_citation` remain for backward compatibility.