This release contains **two potentially-breaking changes** for existing usages.
The circumstances and implications of each are described below.
Improvements
* It is now an error (instead of a warning) to use the `--image` option to
`nextstrain build` when using the "native" runner (either explicitly via
`--native` or implicitly via config set by `nextstrain check-setup
--set-default`). The error message is:
The --image option is incompatible with the "native" runner (…).
If you need to use the "native" runner, please omit the --image option.
If you need the --image option, please select another runner (e.g.
with the --docker option) that supports it. Currently --image is
supported by the Docker (--docker) and AWS Batch (--aws-batch)
runners. You can check if your setup supports these runners with
`nextstrain check-setup`.
This is a **potentially-breaking change** as invocations using the "native"
runner with the `--image` option may exist and be working for users as they
expect. If you encounter this new error after upgrading but your results of
running `nextstrain` commands has always been as-expected, then you can
safely drop the `--image` option from your invocations and avoid the new
error.
* When running a Snakemake workflow, `nextstrain build` now defaults
Snakemake's `--cores` option to `all` unless `build`'s own `--cpus` option is
provided. If you provide your own `--cores` (or equivalent, e.g. `-j`)
option to Snakemake via `nextstrain build`, as in `nextstrain build .
--cores 2`, then this new default isn't applicable.
This is a **potentially-breaking change** if you're…
- …using Snakemake version 5.10 or earlier (such as via our standard Docker
runtime image), _and_
- …are not already providing a `--cores` (or equivalent, e.g. `-j`) option
to Snakemake via `nextstrain build`, _and_
- …expect your `nextstrain build` invocations to use only a single CPU/core
instead of all CPUs/cores available on your computer.
If this is the case, you can pass `--cpus 1` to `nextstrain build` to regain
the original behaviour, e.g. `nextstrain build --cpus 1 .`.
This change will allow upgrading of Snakemake in our Docker runtime image
without inflicting the addition of `--cores` (or equivalent) arguments onto
every existing `nextstrain build` invocation that lacks it.
For context, Snakemake [requires the `--cores` option as of
5.11](https://github.com/snakemake/snakemake/issues/283). This has spawned
much discussion
([1](https://github.com/snakemake/snakemake/issues/308),
[2](https://github.com/snakemake/snakemake/issues/312),
[3](https://github.com/snakemake/snakemake/issues/450),
[4](https://github.com/snakemake/snakemake/issues/885))
where it was made clear this is an intentional, permanent change and a
default will not be added. By adding our own default, we can insulate our
users from the upstream change and make Nextstrain builds fast-by-default.
Our `--cpus` option can be used to limit CPU usage back from this default if
necessary, and users can always specify `--cores` (or equivalents)
themselves.
* The `version --verbose` and `check-setup` commands now indicate the default
runner in their output, which is useful context when troubleshooting or just
plain unsure what the default is.
* The `check-setup` command now exits with an error code if the default runner
is not supported. Prior to this it only exited with an error code if no
runners were supported.
Bug fixes
* When using the AWS Batch runner, the `--cpus` and `--memory` options for
`build` now correctly override the corresponding resource requests in newer
style AWS Batch job definitions. Prior to this they would be ignored by AWS
Batch. Older style AWS Batch job definitions were never affected by this and
continue to work with `--cpus` and `--memory` as expected. See
[144](https://github.com/nextstrain/cli/issues/144) for more details.
* A deprecation warning from the `cryptography` module (version 37 and higher)
when running on Python 3.6 is now suppressed as it's just noise to an end
user. This probably moves us closer to dropping 3.6 support ourselves, but
it's not so onerous yet.
* The rST to plain text conversion used to format `--help` text was fixed to
avoid emitting `\r\n` in the wrong context.
* The stdout and stderr streams are now configured at program start to always
emit UTF-8. Previously they used the Python defaults, determined in part by
the system defaults, which often resolved to UTF-8, but not always. The code
base assumes UTF-8, and now the streams are guaranteed to match. In
particular, this fixes `UnicodeEncodeError` issues in some contexts on
Windows even when UTF-8 is supported.
Documentation
* This changelog now sports a preamble to set the scene and provide context for
the content.
Development
* The source repo now uses a `+git` local version part to distinguish
actual releases from installations of unreleased code. Relatedly, the
development builds created by CI use a `+git.${commit}` local version part to
pin down the specific commit from which they were built. This is mostly
helpful when reading CI logs or downloading the builds from the CI artifacts.
* The CI workflow has seen some significant sprucing up and additions,
including sporting a more typical lifecycle with separate steps for build,
test, and release (a new addition). The single test step is also now split
between source tests (unit tests, type checking, linting, etc) and dist tests
(functional tests, integration tests, interface tests, etc). The release
step uploads to PyPI after a release tag is pushed, built, and tested,
replacing manual uploads from a local development environment. Various other
small improvements to CI were also made.
* We now run CI tests on Windows. \o/ It's not perfect, but this should help
avoid basic Windows-only pitfalls which we might not otherwise notice in a
timely fashion.
* Our CI now builds (and tests) standalone installation archives (`.tar.gz` for
Linux and macOS, `.zip` for Windows) comprising of:
1. A `nextstrain` executable containing a bundled Python interpreter + the
Python stdlib + the Nextstrain CLI code + its dependencies.
2. External files (lib, data, etc) that are necessary but can't (for a
variety of reasons) be bundled into the executable.
These installation archives can be downloaded, extracted, and run in-place
without even a Python interpreter being installed on the host computer, hence
the "standalone" moniker.
Currently these are for development/testing/experimentation purposes only.
We include them as assets on GitHub Releases, but do not provide an automated
means of "installing" or unpacking them; those are ultimate goals, but this
is just a first step towards those. If you try out the standalone archives
in the meantime, though, please let us know how it goes (good or bad) by
opening an issue with your experience/feedback/questions.
* GitHub Releases are now created by CI after making a release to PyPI. These
are visible on the GitHub repo's [releases
page](https://github.com/nextstrain/cli/releases) and various other places on
GitHub. Each GitHub Release includes a copy of the relevant changelog
section and release assets like the Python distributions and standalone
installation archives (see above). Releases on GitHub are currently intended
mostly for informational and notification purposes; the primary release
distribution method is still PyPI and sources downstream of PyPI (e.g.
Conda).