Nextstrain-augur

Features

* filter: Add support to group by ISO week (`--group-by week`) during subsampling. [1067][] (victorlin)

Bug Fixes

* filter: Fixed unintended behavior in which grouping by `day` would "work" when used with `month` and/or `year`. Updated so it will be ignored. [1070][] (victorlin)
* filter: Fixed unintended behavior in which grouping by `month` with ambiguous years would "work". Updated so date ambiguity is checked properly for all generated columns. [1072][] (victorlin)

[1067]: https://github.com/nextstrain/augur/pull/1067
[1070]: https://github.com/nextstrain/augur/pull/1070
[1072]: https://github.com/nextstrain/augur/pull/1072

Major Changes

* export: The `--node-data` option may now be given multiple times to provide additional `.json` files. Previously, subsequent occurrences of the option overrode prior occurrences. This is a **breaking change**, although we expect few usages to be impacted. Each occurrence of the option may still specify multiple files at a time. [1010][] (tsibley)

Bug Fixes


* refine: 17.1.0 updated TreeTime to version 0.9.2 and introduced the `refine` flag `--use-fft`. This makes previously costly marginal date inference cheaper. This update adjusts when `refine` runs marginal date inference during its iterative optimization. Without the `use-fft` flag, it will now behave as it did before 17.1.0 (marginal inference only during final iterations). With the `--use-fft` flag, marginal date inference will be used at every step during the iteration if refine is run with `--date-inference marginal` [1034][]. (rneher)
* tree: When using IQtree as tre builder, `--nthreads` now sets the maximum number of threads (IQtree argument `-ntmax`). The actual number of threads to use can be specified by the user through the tree-builder-arg `-nt` which defaults to `-nt AUTO`, causing IQtree to automatically chose the best number of threads to use [1042][] (corneliusroemer)
* Make cvxopt as a required dependency, since it is required for titer models to work [1035][]. (victorlin)
* filter: Fix compatibility with Pandas 1.5.0 which could cause an unexpected `AttributeError` with an invalid `--query` given to `augur filter`. [1050][] (tsibley)
* refine: Add `--verbosity` argument that is passed down to TreeTime to facilitate monitoring and debugging. [1033][] (anna-parker)
* Improve handling of errors from TreeTime. [1033][] (anna-parker)

[1010]: https://github.com/nextstrain/augur/pull/1010
[1033]: https://github.com/nextstrain/augur/pull/1033
[1034]: https://github.com/nextstrain/augur/pull/1034
[1035]: https://github.com/nextstrain/augur/pull/1035
[1042]: https://github.com/nextstrain/augur/pull/1042
[1050]: https://github.com/nextstrain/augur/pull/1050

Features

* refine: Upgrade TreeTime from 0.8.6 to >= 0.9.2 which enables a speedup of timetree inference in marginal mode due to the use of Fast Fourier Transforms [1018][]. (rneher and anna-parker). Use the `refine` flag `--use-fft` to use this feature.

Bug Fixes

* refine, export v1: Use pandas.DataFrame.at instead of .loc for single values [979][]. (victorlin)
* refine: Gracefully handle all exceptions from TreeTime [1023][]. (anna-parker)
* refine: Document branch length units `treetime` expects [1024][]. (anna-parker)
* dates: Raise an error when metadata to `get_numerical_dates()` is not a pandas DataFrame [1026][]. (victorlin)

[979]: https://github.com/nextstrain/augur/pull/979
[1018]: https://github.com/nextstrain/augur/pull/1018
[1023]: https://github.com/nextstrain/augur/pull/1023
[1024]: https://github.com/nextstrain/augur/pull/1024
[1026]: https://github.com/nextstrain/augur/pull/1026

Major Changes

* Moved the following modules to subpackages [1002][]. (joverlee521)
These are technically breaking changes for the API, but they do not change the Augur CLI commands.
* `import.py` -> `import_/__init__.py`
* `import_beast.py` -> `import_/beast.py`
* `measurements.py` -> `measurements/__init__.py` + `measurements/concat.py` + `measurements/export.py`
* Move the following internal functions/classes [1002][]. (joverlee521)
* `augur.add_default_command` -> `argparse_.add_default_command`
* `utils.HideAsFalseAction` -> `argparse_.HideAsFalseAction`
* Subcommands must include a `register_parser` function to add their own parser instead of a `register_arguments` function [1002][]. (joverlee521)
* utils: Remove internal function `utils.read_metadata()` [978][]. (victorlin)
* Use `io.read_metadata()` going forwards.
* To switch to using metadata as a pandas DataFrame (recommended):
* Iterate through strains: `metadata.items()` -> `metadata.iterrows()`
* Check strain presence: `strain in metadata` -> `strain in metadata.index`
* Check field presence: `field in metadata[strain]` -> `field in metadata.columns`
* Get metadata for a strain: `metadata[strain]` -> `metadata.loc[strain]`
* Get field for a strain: `metadata[strain][field]` -> `metadata.at[strain, field]`
* To keep using metadata in a dictionary:
py
metadata = read_metadata(args.metadata)
metadata.insert(0, "strain", metadata.index.values)
columns = metadata.columns
metadata = metadata.to_dict(orient="index")


Features

* export: `--skip-validation` now also skips version compatibility checks [902][]. (corneliusroemer)
* filter: Report names of duplicate strains found during metadata parsing [1008][] (huddlej)
* translate: Add support for Nextclade gene map GFFs [1017][] (huddlej)

Bug Fixes

* filter: Rename internal force inclusion filtering functions [1006][] (victorlin)

[902]: https://github.com/nextstrain/augur/pull/902
[978]: https://github.com/nextstrain/augur/pull/978
[1002]: https://github.com/nextstrain/augur/pull/1002
[1006]: https://github.com/nextstrain/augur/pull/1006
[1008]: https://github.com/nextstrain/augur/pull/1008
[1017]: https://github.com/nextstrain/augur/pull/1017

Bug Fixes

* filter: Move `register_arguments` to the top of the module for better readability [995][]. (victorlin)
* filter: Fix a regression [introduced in 16.0.2](https://github.com/nextstrain/augur/commit/4859b5d70e77cc9a0bb99e741fefb29952058b71) that caused grouping with subsampled max sequences and force-included strains to fail in a data-specific way [#1000][]. (huddlej)

[995]: https://github.com/nextstrain/augur/pull/995
[1000]: https://github.com/nextstrain/augur/pull/1000

Bug Fixes

* The entropy panel was unavailable if mutations were not translated [881][]. This has been fixed by creating an additional `annotations` block in `augur ancestral` containing (nucleotide) genome annotations in the node-data [961][] (jameshadfield)
* ancestral: WARNINGs to stdout have been updated to print to stderr [961][] (jameshadfield)
* filter: Explicitly drop date/year/month columns from metadata during grouping. [967][] (victorlin)
* This fixes a bug [871][] where `augur filter` would crash with a cryptic `ValueError` if `year` and/or `month` is a custom column in the input metadata and also included in `--group-by`.
* filter: Fix duplicates that may appear in metadata when using `--include`/`--include-where` with subsampling [986][] (victorlin)

[881]: https://github.com/nextstrain/augur/issues/881
[961]: https://github.com/nextstrain/augur/pull/961
[967]: https://github.com/nextstrain/augur/pull/967
[871]: https://github.com/nextstrain/augur/issues/871
[986]: https://github.com/nextstrain/augur/pull/986