Nextstrain-cli

_Unreleased due to [test failures](https://github.com/nextstrain/cli/actions/runs/6238951959). Changes first released as part of 7.3.0.post1._

Improvements

* build: Providing a path to a pathogen build directory is no longer required
when the AWS Batch runtime is in use (e.g. with `--aws-batch`) and both the
`--attach` and `--no-download` options are given. This allows usages which
just want to check job status or logs to stop providing a meaningless/unused
directory.
([305](https://github.com/nextstrain/cli/pull/305))

* The initial setup of the Conda runtime, i.e. with `nextstrain setup conda`,
now explicitly installs the latest available runtime package version. This
was the implicit intent and expectation, but, for various Conda packaging
reasons, it was (silently) not always the result. Now an error will occur if
the latest version is not installable. This matches the existing behaviour
when updating the Conda runtime, i.e. with `nextstrain update conda`.
([312](https://github.com/nextstrain/cli/pull/312))

Bug fixes

* The `HOST` and `PORT` environment variables are now respected by `nextstrain
view`. They provide defaults when `--host` and/or `--port` aren't provided.
([310](https://github.com/nextstrain/cli/pull/310))

* Updated fsspec and s3fs dependencies to avoid version 2023.9.1, since they
caused `nextstrain build --aws-batch` invocations to fail.
([313](https://github.com/nextstrain/cli/pull/313))

Bug fixes

* The Conda runtime now uses Micromamba 1.1.0 (upgraded from 1.0.0) to manage
the runtime environment. The newer version has improved error handling in
edge cases where the previous version silently failed when used during
`nextstrain update`.
([301](https://github.com/nextstrain/cli/pull/301))

* The Singularity runtime now cleans up unnecessary caches during `nextstrain
update` and `nextstrain setup`. This reduces the storage space required by
the runtime and prevents it from growing unbounded over time. However, this
also results in increases in network transfer during `nextstrain update`. We
may mitigate the additional transfer in the future.
([303][])

Documentation

* The command-line `--help` output for commands and the corresponding
documentation pages on the web are more integrated and improved in various
small ways. In particular, command options are more cross-referencable and
directly linkable and the wrap-width of `--help` output is more consistent
and reliably readable.
([299](https://github.com/nextstrain/cli/pull/299))

* We've added a large section of new and expanded [documentation on
runtimes](https://docs.nextstrain.org/projects/cli/page/runtimes/), both in
general and for each individual runtime.
([303][])

* We've added and expanded documentation about configuration, including the
[config file](https://docs.nextstrain.org/projects/cli/page/config/file/) and
the [paths](https://docs.nextstrain.org/projects/cli/page/config/paths/) used
by Nextstrain CLI. Documentation in general now includes more about the
environment variables that can be used to configure Nextstrain CLI.
([303][])


[303]: https://github.com/nextstrain/cli/pull/303

Improvements

* Commands that use a runtime (`nextstrain build`, `nextstrain shell`, and
`nextstrain view`) now support two new options for setting or passing thru
environment variables into the runtime environment:

--env <name>[=<value>]
--envdir <path>

When either of these options are given, the default behaviour of
automatically passing thru several "well-known" environment variables is
disabled. That is, the following "well-known" environment variables are only
automatically passed thru when the new `--env` and `--envdir` options are
_not_ used:

- `AUGUR_RECURSION_LIMIT`
- `AUGUR_MINIFY_JSON`
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`
- `AWS_SESSION_TOKEN`
- `ID3C_URL`
- `ID3C_USERNAME`
- `ID3C_PASSWORD`
- `RETHINK_HOST`
- `RETHINK_AUTH_KEY`

Pass these variables explicitly via `--env` or `--envdir` if you need them in
combination with other `--env` or `--envdir` usage. For more usage details,
use the `--help-all` flag of any of those commands, e.g. `nextstrain build
--help-all`.
([289](https://github.com/nextstrain/cli/pull/289))

* Environment variables are now passed to the Docker and AWS Batch runtimes via
more secure means when the container image in use is new enough to support it
(`nextstrain/base:build-20230613T204512Z` and newer). This ensures the env
values aren't visible in the container's config (e.g. via `docker inspect`,
`aws batch describe-jobs`, the AWS web console). If you're using an older
image, you can update it with `nextstrain update docker`.

For Docker, environment variables are written to an internal and temporary
envdir directory visible only to the current user which is deleted
immediately after use at container start.

For AWS Batch, environment variables are written to a ZIP archive on S3,
alongside but separate from the ZIP archive of the build dir. This env
archive is deleted from S3 immediately after use at container start.

Both of these approaches minimize the amount of time environment variable
values exist outside of memory, persisted to storage (disk, S3).
([289](https://github.com/nextstrain/cli/pull/289))

Bug fixes

* `nextstrain view` now waits (up to 10s) for Auspice to start responding
before automatically opening it in the browser. This should eliminate the
previous behaviour of sometimes opening the browser too soon.
([291](https://github.com/nextstrain/cli/pull/291))

Bug fixes

* `nextstrain update` for the Conda runtime no longer reports an "invalid
version" error. This was a regression introduced in 7.0.0.
([286](https://github.com/nextstrain/cli/pull/286))

This release is mostly a bug fix release for our Conda and Singularity
runtimes. However, it contains a **potentially-breaking change** for existing
usages of the Singularity runtime: **the minimum required Singularity version
has changed from 2.6.0 to 3.0.0**. This change was required for a critical bug
fix. If you do not use the Singularity runtime, there are no
potentially-breaking changes in this release.

Improvements

* `nextstrain shell` now notes which runtime is being entered in its initial
messaging to establish more context for the user (and for developers when
troubleshooting).
([283][])

* The Singularity runtime now checks for the minimum required Singularity
version (3.0.0 with this release) during `nextstrain check-setup`.
([283][])

Bug fixes

* Setup and upgrade of the Conda runtime now only uses stable "main" channel
releases when determining the latest release version, as intended.
Previously, testing and development releases could be selected if they were
newer than the last stable release. Additionally, if there are multiple
builds for a release version, the highest numbered build (i.e. newest) is now
used instead of the lowest.
([280](https://github.com/nextstrain/cli/pull/280))

* The Singularity runtime now works with our container runtime images from
`build-20230411T103027Z` onwards. The Snakemake upgrade in that image
version resulted in "read-only file system" errors which referenced the
user's home directory. Those errors are now fixed.
([283][])

* The prompt for `nextstrain shell`—a stylized variant of the Nextstrain
wordmark—now works when using the Singularity runtime regardless of
Singularity version. Previously Singularity's default prompt of
`Singularity> ` overrode ours when using Singularity versions ≥3.5.3.
([283][])

* More robust command-line processing is used for the Singularity runtime on
Singularity versions ≥3.10.0. Singularity's early (and unexpected)
evaluation of arguments that look like (but aren't) shell variable
substitutions is disabled.
([283][])

Development

* The command lines and environment overrides of many (but not all) process
invocations are now logged when `NEXTSTRAIN_DEBUG` is enabled.
([283][])


[283]: https://github.com/nextstrain/cli/pull/283

Bug fixes

* We've fixed and future-proofed a compatibility bug with a third-party library
that can occur under very specific conditions when `nextstrain build` submits
AWS Batch jobs.
([261](https://github.com/nextstrain/cli/pull/261))

* The update process for the Conda runtime is now more robust and less likely
to get stuck at an old version.
([266](https://github.com/nextstrain/cli/pull/266))