Nextstrain-augur

Major Changes

* export, filter, frequencies, refine, traits: From versions 10.0.0 through 21.1.0, arbitrary delimiters for `--metadata` were supported due to internal implementation differences from the advertised CSV and TSV support. Starting with this version, non-CSV/TSV files will no longer be supported by default. To adjust for this breaking change, specify custom delimiters with the new `--metadata-delimiters` flag. [1196][] (victorlin)
* `augur.io.read_metadata`: Previously, this supported any arbitrary delimiters for the metadata. Now, it only supports a list of possible delimiters represented by the new `delimiters` keyword argument, which defaults to `,` and `\t`. [812][] (victorlin)
* refine: The seeding method for `--seed` has been updated. This affects usages that rely on the reproducibility of outputs with the same `--seed` value prior to this version. Outputs from this version onwards should be reproducible until the next implementation change, which we don't expect to happen any time soon. [1207][] (rneher)

Features

* Constrain `bcbio-gff` to >=0.7.0 and allow `Biopython` >=1.81 again. We had to introduce the `Biopython` constraint in v21.0.1 (see [1152][]) due to `bcbio-gff` <0.7.0 relying on the removed `Biopython` feature `UnknownSeq`. [1178][] (corneliusroemer)
* `augur.io.read_metadata` (used by export, filter, frequencies, refine, and traits): Previously, this used the Python parser engine for [`pandas.read_csv()`][]. Updated to use the C engine for faster reading of metadata. [812][] (victorlin)
* curate: Allow custom metadata delimiters with the new `--metadata-delimiters` flag. [1196][] (victorlin)
* Bump the default recursion limit to 10,000. Users can continue to override this limit with the environment variable `AUGUR_RECURSION_LIMIT`. [1200][] (joverlee521)
* clades, export v2: Clade labels + coloring keys are now definable via arguments to augur clades allowing pipelines to use multiple invocations of augur clades resulting in multiple sets of colors and branch labels. How labels are stored in the (intermediate) node-data JSON files has changed. This should be fully backwards compatible for pipelines using augur commands, however custom scripts may need updating. PR [728][] (jameshadfield)
* refine: add flag `--max-iter` to control the maximal number of iterations TreeTime uses to infer time trees. This was previously hard-coded to 2, which is now the default. [1203][] (rneher)
* refine: add flags `--greedy-resolve` and `--stochastic-resolve` to customize polytomy resolution. [1203][], [1207][] (rneher)
* `--greedy-resolve`: resolve polytomies by greedily minimizing tree length (default behavior, unchanged).
* `--stochastic-resolve`: resolve polytomies as random coalescent trees.
* These are mutually exclusive with the pre-existing `--keep-polytomies` flag.

Bug fixes

* filter, frequencies, refine, parse: Previously, ambiguous dates in the future had a limit of today's date imposed on the upper value but not the lower value. It is now imposed on the lower value as well. [1171][] (victorlin)
* refine: `--year-bounds` was ignored in versions 9.0.0 through 20.0.0. It now works. [1136][] (victorlin)
* tree: Input alignment filenames which do not end in `.fasta` are now properly handled when using IQ-TREE. Previously their contents were overwritten first by `augur tree` itself (resulting in truncation) and then by the log output of IQ-TREE (resulting in an error). Thanks to Jon Bråte for reporting this bug. [1206][] (tsibley)
* clades: A number of small bug fixes, improvements to documentation, tests and improved error detection. [1199][] (jameshadfield)

[728]: https://github.com/nextstrain/augur/pull/728
[812]: https://github.com/nextstrain/augur/pull/812
[1136]: https://github.com/nextstrain/augur/issues/1136
[1152]: https://github.com/nextstrain/augur/pull/1152
[1171]: https://github.com/nextstrain/augur/issues/1171
[1178]: https://github.com/nextstrain/augur/pull/1178
[1196]: https://github.com/nextstrain/augur/pull/1196
[1199]: https://github.com/nextstrain/augur/pull/1199
[1200]: https://github.com/nextstrain/augur/pull/1200
[1203]: https://github.com/nextstrain/augur/pull/1203
[1206]: https://github.com/nextstrain/augur/pull/1206
[1207]: https://github.com/nextstrain/augur/pull/1207
[`pandas.read_csv()`]: https://pandas.pydata.org/pandas-docs/version/1.5/reference/api/pandas.read_csv.html

Features

* filter: Add `--empty-output-reporting={error,warn,silent}` option to allow filter to produce empty outputs without raising an error. The default behavior is still to raise an error when filter produces an empty output, so users will have to explicitly pass the "warn" or "silent" value to bypass the error. [1175][] (joverlee521)

Bug fixes

* translate: Fix error handling when features cannot be read from reference sequence file. [1168][] (victorlin)
* translate: Remove an unnecessary check which allowed for inaccurate error messages to be shown. [1169][] (victorlin)
* frequencies: Previously, monthly pivot points calculated from the end of a month may have been shifted by 1-3 days. This is now fixed. [1150][] (victorlin)
* docs: Fix minor formatting issues. [1095][] (victorlin)
* Update development status on PyPI from "3 - Alpha" to "5 - Production/Stable". This should have been done since the beginning of this changelog, but now it is official. [1160][] (corneliusroemer)

[1095]: https://github.com/nextstrain/augur/pull/1095
[1150]: https://github.com/nextstrain/augur/pull/1150
[1160]: https://github.com/nextstrain/augur/pull/1160
[1168]: https://github.com/nextstrain/augur/pull/1168
[1169]: https://github.com/nextstrain/augur/pull/1169
[1175]: https://github.com/nextstrain/augur/pull/1175

Bug fixes

* Constrain Biopython version to <=1.80 so that `augur translate` is not broken by a deprecation of `UnknownSeq` in 1.81. When running `augur translate` with Biopython 1.81, the user will receive an error starting with `ERROR: Package BCBio.GFF not found!` and ending with `TypeError: object of type 'NoneType' has no len()`. [1152][] (corneliusroemer)

[1152]: https://github.com/nextstrain/augur/pull/1152

Major Changes

* measurements export: Supports exporting multiple thresholds per collection via the measurements config and the `--thresholds` option. This change is backwards compatible with previous uses of the `--threshold` option. However, due to the updates to the JSON schema, users will need to update to Auspice v2.43.0 for thresholds to be displayed properly in the measurements panel. [1148][] (joverlee521)

Features

* export v2: Add `--validation-mode={error,warn,skip}` option for more nuanced control of validation. The new "warn" mode performs validation and emits messages about potential problems, but it does not cause the export command to fail even if there are problems. [1135][] (tsibley)

Bug Fixes

* filter, frequencies, refine, parse: Properly handle invalid date errors and output the bad date. [1140][] (victorlin)
* export, validate: Validation errors are now much more human-readable and actually pinpoint the problems. [1134][] (tsibley)

[1134]: https://github.com/nextstrain/augur/pull/1134
[1135]: https://github.com/nextstrain/augur/pull/1135
[1140]: https://github.com/nextstrain/augur/pull/1140
[1148]: https://github.com/nextstrain/augur/pull/1148

Major Changes

* frequencies: Changes the logic for calculating the time points when frequencies are estimated to ensure that the user-provided "end date" is always included. This change in the behavior of the frequencies command fixes a bug where large intervals between time points (e.g., 3 months) could cause recent data to be omitted from frequency calculations. See the pull request for more details included the scientific implications of this bug. [1121][] (huddlej)

[1121]: https://github.com/nextstrain/augur/pull/1121

Features

* titers: Support parsing of thresholded values (e.g., "<80" or ">2560"). [1118][] (huddlej)
* tree: Support bootstrapped trees generated with RAxML via user-provided `--tree-builder-args`. [1127][] (tsibley)

Bug Fixes

* utils: Serialize common numpy data types in `write_json`. [1119][] (victorlin)
* filter: Standardize exit codes from internal error handling. [931][] (victorlin)
* tree: Suppress the `Cannot specify --substitution-model unless using IQTree` warning when `--substitution-model` is left at its default. [1127][] (tsibley)
* tree: Print the underlying error message when tree building fails. [1127][] (tsibley)
* Previously, `numpy` and `scipy` were installed as dependencies of dependencies. Mark them as direct dependencies since they are used directly within Augur. [1120][] (victorlin)

[931]: https://github.com/nextstrain/augur/pull/931
[1118]: https://github.com/nextstrain/augur/pull/1118
[1119]: https://github.com/nextstrain/augur/pull/1119
[1120]: https://github.com/nextstrain/augur/pull/1120
[1127]: https://github.com/nextstrain/augur/pull/1127