Nf-core

Template

- Critical tweak to add `--dir` declaration to `nf-core lint` GitHub Actions `linting.yml` workflow

General

- Add `--dir` declaration to `nf-core sync` GitHub Actions `sync.yml` workflow

:warning: Major enhancements & breaking changes

This marks the first Nextflow DSL2-centric release of `tools` which means that some commands won't work in full with DSL1 pipelines anymore. Please use a `v1.x` version of `tools` for such pipelines or better yet join us to improve our DSL2 efforts! Here are the most important changes:

- The pipeline template has been completely re-written in DSL2
- A module template has been added to auto-create best-practice DSL2 modules to speed up development
- A whole suite of commands have been added to streamline the creation, installation, removal, linting and version bumping of DSL2 modules either installed within pipelines or the nf-core/modules repo

Template

- Move TODO item of `contains:` map in a YAML string [[1082](https://github.com/nf-core/tools/issues/1082)]
- Trigger AWS tests via Tower API [[1160](https://github.com/nf-core/tools/pull/1160)]

General

- Fixed a bug in the Docker image build for tools that failed due to an extra hyphen. [[1069](https://github.com/nf-core/tools/pull/1069)]
- Regular release sync fix - this time it was to do with JSON serialisation [[1072](https://github.com/nf-core/tools/pull/1072)]
- Fixed bug in schema validation that ignores upper/lower-case typos in parameters [[1087](https://github.com/nf-core/tools/issues/1087)]
- Bugfix: Download should use path relative to workflow for configs
- Remove lint checks for files related to conda and docker as not needed anymore for DSL2
- Removed `params_used` lint check because of incompatibility with DSL2
- Added`modules bump-versions` command to `README.md`
- Update docs for v2.0 release

Modules

- Update comment style of modules `functions.nf` template file [[1076](https://github.com/nf-core/tools/issues/1076)]
- Changed working directory to temporary directory for `nf-core modules create-test-yml` [[908](https://github.com/nf-core/tools/issues/908)]
- Use Biocontainers API instead of quayi.io API for `nf-core modules create` [[875](https://github.com/nf-core/tools/issues/875)]
- Update `nf-core modules install` to handle different versions of modules [1116](https://github.com/nf-core/tools/pull/1116)
- Added `nf-core modules bump-versions` command to update all versions in the `nf-core/modules` repository [[1123](https://github.com/nf-core/tools/issues/1123)]
- Updated `nf-core modules lint` to check whether a `git_sha` exists in the `modules.json` file or whether a new version is available [[1114](https://github.com/nf-core/tools/issues/1114)]
- Refactored `nf-core modules` command into one file per command [1124](https://github.com/nf-core/tools/pull/1124)
- Updated `nf-core modules remove` to also remove entry in `modules.json` file ([1115](https://github.com/nf-core/tools/issues/1115))
- Bugfix: Interactive prompt for `nf-core modules install` was receiving too few arguments
- Added progress bar to creation of 'modules.json'
- Updated `nf-core modules list` to show versions of local modules
- Improved exit behavior by replacing `sys.exit` with exceptions
- Updated `nf-core modules remove` to remove module entry in `modules.json` if module directory is missing
- Create extra tempdir as work directory for `nf-core modules create-test-yml` to avoid adding the temporary files to the `test.yml`
- Refactored passing of command line arguments to `nf-core` commands and subcommands ([1139](https://github.com/nf-core/tools/issues/1139), [#1140](https://github.com/nf-core/tools/issues/1140))
- Check for `modules.json` for entries of modules that are not actually installed in the pipeline [[1141](https://github.com/nf-core/tools/issues/1141)]
- Added `<keywords>` argument to `nf-core modules list` for filtering the listed modules. ([1139](https://github.com/nf-core/tools/issues/1139)
- Added support for a `bump-versions` configuration file [[1142](https://github.com/nf-core/tools/issues/1142)]
- Fixed `nf-core modules create-test-yml` so it doesn't break when the output directory is supplied [[1148](https://github.com/nf-core/tools/issues/1148)]
- Updated `nf-core modules lint` to work with new directory structure [[1159](https://github.com/nf-core/tools/issues/1159)]
- Updated `nf-core modules install` and `modules.json` to work with new directory structure ([1159](https://github.com/nf-core/tools/issues/1159))
- Updated `nf-core modules remove` to work with new directory structure [[1159](https://github.com/nf-core/tools/issues/1159)]
- Restructured code and removed old table style in `nf-core modules list`
- Fixed bug causing `modules.json` creation to loop indefinitly
- Added `--all` flag to `nf-core modules install`
- Added `remote` and `local` subcommands to `nf-core modules list`
- Fix bug due to restructuring in modules template
- Added checks for verifying that the remote repository is well formed
- Added checks to `ModulesCommand` for verifying validity of remote repositories
- Misc. changes to `modules install`: check that module exist in remote, `--all` is has `--latest` by default.

Sync

- Don't set the default value to `"null"` when a parameter is initialised as `null` in the config [[1074](https://github.com/nf-core/tools/pull/1074)]

Tests

- Added a test for the `version_consistency` lint check
- Refactored modules tests into separate files, and removed direct comparisons with number of tests in `lint` tests ([1158](https://github.com/nf-core/tools/issues/1158))

Template

- Add the implicit workflow declaration to `main.nf` DSL2 template [[1056](https://github.com/nf-core/tools/issues/1056)]
- Fixed an issue regarding explicit disabling of unused container engines [[972](https://github.com/nf-core/tools/pull/972)]
- Removed trailing slash from `params.igenomes_base` to yield valid s3 paths (previous paths work with Nextflow but not aws cli)
- Added a timestamp to the trace + timetime + report + dag filenames to fix overwrite issue on AWS
- Rewrite the `params_summary_log()` function to properly ignore unset params and have nicer formatting [[971](https://github.com/nf-core/tools/issues/971)]
- Fix overly strict `--max_time` formatting regex in template schema [[973](https://github.com/nf-core/tools/issues/973)]
- Convert `d` to `day` in the `cleanParameters` function to make Duration objects like `2d` pass the validation [[858](https://github.com/nf-core/tools/issues/858)]
- Added nextflow version to quick start section and adjusted `nf-core bump-version` [[1032](https://github.com/nf-core/tools/issues/1032)]
- Use latest stable Nextflow version `21.04.0` for CI tests instead of the `-edge` release

Download

- Fix bug in `nf-core download` where image names were getting a hyphen in `nf-core` which was breaking things.
- Extensive new interactive prompts for all command line flags [[1027](https://github.com/nf-core/tools/issues/1027)]
- It is now recommended to run `nf-core download` without any cli options and follow prompts (though flags can be used to run non-interactively if you wish)
- New helper code to set `$NXF_SINGULARITY_CACHEDIR` and add to `.bashrc` if desired [[1027](https://github.com/nf-core/tools/issues/1027)]

Launch

- Strip values from `nf-core launch` web response which are `False` and have no default in the schema [[976](https://github.com/nf-core/tools/issues/976)]
- Improve API caching code when polling the website, fixes noisy log message when waiting for a response [[1029](https://github.com/nf-core/tools/issues/1029)]
- New interactive prompts for pipeline name [[1027](https://github.com/nf-core/tools/issues/1027)]

Modules

- Added `tool_name_underscore` to the module template to allow TOOL_SUBTOOL in `main.nf` [[1011](https://github.com/nf-core/tools/issues/1011)]
- Added `--conda-name` flag to `nf-core modules create` command to allow sidestepping questionary [[988](https://github.com/nf-core/tools/issues/988)]
- Extended `nf-core modules lint` functionality to check tags in `test.yml` and to look for a entry in the `pytest_software.yml` file
- Update `modules` commands to use new test tag format `tool/subtool`
- New modules lint test comparing the `functions.nf` file to the template version
- Modules installed from alternative sources are put in folders based on the name of the source repository

Linting

- Fix bug in nf-core lint config skipping for the `nextflow_config` test [[1019](https://github.com/nf-core/tools/issues/1019)]
- New `-k`/`--key` cli option for `nf-core lint` to allow you to run only named lint tests, for faster local debugging
- Merge markers lint test - ignore binary files, allow config to ignore specific files [[1040](https://github.com/nf-core/tools/pull/1040)]
- New lint test to check if all defined pipeline parameters are mentioned in `main.nf` [[1038](https://github.com/nf-core/tools/issues/1038)]
- Added fix to remove warnings about params that get converted from camelCase to camel-case [[1035](https://github.com/nf-core/tools/issues/1035)]
- Added pipeline schema lint checks for missing parameter description and parameters outside of groups [[1017](https://github.com/nf-core/tools/issues/1017)]

General

- Try to fix the fix for the automated sync when we submit too many PRs at once [[970](https://github.com/nf-core/tools/issues/970)]
- Rewrite how the tools documentation is deployed to the website, to allow multiple versions
- Created new Docker image for the tools cli package - see installation docs for details [[917](https://github.com/nf-core/tools/issues/917)]
- Ignore permission errors for setting up requests cache directories to allow starting with an invalid or read-only `HOME` directory

- Running tests twice with `nf-core modules create-test-yml` to catch unreproducible md5 sums [[890](https://github.com/nf-core/tools/issues/890)]
- Fix sync error again where the Nextflow edge release needs to be used for some pipelines
- Fix bug with `nf-core lint --release` (`NameError: name 'os' is not defined`)
- Added linebreak to linting comment so that markdown header renders on PR comment properly
- `nf-core modules create` command - if no bioconda package is found, prompt user for a different bioconda package name
- Updated module template `main.nf` with new test data paths

- Make module template pass the EC linter [[953](https://github.com/nf-core/tools/pull/953)]
- Added better logging message if a user doesn't specificy the directory correctly with `nf-core modules` commands [[942](https://github.com/nf-core/tools/pull/942)]
- Fixed parameter validation bug caused by JSONObject [[937](https://github.com/nf-core/tools/issues/937)]
- Fixed template creation error regarding file permissions [[932](https://github.com/nf-core/tools/issues/932)]
- Split the `create-lint-wf` tests up into separate steps in GitHub Actions to make the CI results easier to read
- Added automated PR comments to the Markdown, YAML and Python lint CI tests to explain failures (tools and pipeline template)
- Make `nf-core lint` summary table borders coloured according to overall pass / fail status
- Attempted a fix for the automated sync when we submit too many PRs at once [[911](https://github.com/nf-core/tools/issues/911)]

- Fixed bug in pipeline linting markdown output that gets posted to PR comments [[914]](https://github.com/nf-core/tools/issues/914)
- Made text for the PR branch CI check less verbose with a TLDR in bold at the top
- A number of minor tweaks to the new `nf-core modules lint` code