Nf-core

Template

- **Major new feature** - Validation of pipeline parameters [[426]](https://github.com/nf-core/tools/issues/426)
- The addition runs as soon as the pipeline launches and checks the pipeline input parameters two main things:
- No parameters are supplied that share a name with core Nextflow options (eg. `--resume` instead of `-resume`)
- Supplied parameters validate against the pipeline JSON schema (eg. correct variable types, required values)
- If either parameter validation fails or the pipeline has errors, a warning is given about any unexpected parameters found which are not described in the pipeline schema.
- This behaviour can be disabled by using `--validate_params false`
- Added profiles to support the [Charliecloud](https://hpc.github.io/charliecloud/) and [Shifter](https://nersc.gitlab.io/development/shifter/how-to-use/) container engines [[#824](https://github.com/nf-core/tools/issues/824)]
- Note that Charliecloud requires Nextflow version `v21.03.0-edge` or later.
- Profiles for container engines now explicitly _disable_ all other engines [[867](https://github.com/nf-core/tools/issues/867)]
- Fixed typo in nf-core-lint CI that prevented the markdown summary from being automatically posted on PRs as a comment.
- Changed default for `--input` from `data/*{1,2}.fastq.gz` to `null`, as this is now validated by the schema as a required value.
- Removed support for `--name` parameter for custom run names.
- The same functionality for MultiQC still exists with the core Nextflow `-name` option.
- Added to template docs about how to identify process name for resource customisation
- The parameters `--max_memory` and `--max_time` are now validated against a regular expression [[793](https://github.com/nf-core/tools/issues/793)]
- Must be written in the format `123.GB` / `456.h` with any of the prefixes listed in the [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#memory)
- Bare numbers no longer allowed, avoiding people from trying to specify GB and actually specifying bytes.
- Switched from cookiecutter to Jinja2 [[880]](https://github.com/nf-core/tools/pull/880)
- Finally dropped the wonderful [cookiecutter](https://github.com/cookiecutter/cookiecutter) library that was behind the first pipeline template that led to nf-core [[#880](https://github.com/nf-core/tools/pull/880)]
- Now rendering templates directly using [Jinja](https://jinja.palletsprojects.com/), which is what cookiecutter was doing anyway

Modules

Initial addition of a number of new helper commands for working with DSL2 modules:

- `modules list` - List available modules
- `modules install` - Install a module from nf-core/modules
- `modules remove` - Remove a module from a pipeline
- `modules create` - Create a module from the template
- `modules create-test-yml` - Create the `test.yml` file for a module with md5 sums, tags, commands and names added
- `modules lint` - Check a module against nf-core guidelines

You can read more about each of these commands in the main tools documentation (see `README.md` or <https://nf-co.re/tools>)

Tools helper code

- Fixed some bugs in the command line interface for `nf-core launch` and improved formatting [[829](https://github.com/nf-core/tools/pull/829)]
- New functionality for `nf-core download` to make it compatible with DSL2 pipelines [[832](https://github.com/nf-core/tools/pull/832)]
- Singularity images in module files are now discovered and fetched
- Direct downloads of Singularity images in python allowed (much faster than running `singularity pull`)
- Downloads now work with `$NXF_SINGULARITY_CACHEDIR` so that pipelines sharing containers have efficient downloads
- Changed behaviour of `nf-core sync` command [[787](https://github.com/nf-core/tools/issues/787)]
- Instead of opening or updating a PR from `TEMPLATE` directly to `dev`, a new branch is now created from `TEMPLATE` and a PR opened from this to `dev`.
- This is to make it easier to fix merge conflicts without accidentally bringing the entire pipeline history back into the `TEMPLATE` branch (which makes subsequent sync merges much more difficult)

Linting

- Major refactor and rewrite of pipieline linting code
- Much better code organisation and maintainability
- New automatically generated documentation using Sphinx
- Numerous new tests and functions, removal of some unnecessary tests
- Added lint check for merge markers [[321]](https://github.com/nf-core/tools/issues/321)
- Added new option `--fix` to automatically correct some problems detected by linting
- Added validation of default params to `nf-core schema lint` [[823](https://github.com/nf-core/tools/issues/823)]
- Added schema validation of GitHub action workflows to lint function [[795](https://github.com/nf-core/tools/issues/795)]
- Fixed bug in schema title and description validation
- Added second progress bar for conda dependencies lint check, as it can be slow [[299](https://github.com/nf-core/tools/issues/299)]
- Added new lint test to check files that should be unchanged from the pipeline.
- Added the possibility to ignore lint tests using a `nf-core-lint.yml` config file [[809](https://github.com/nf-core/tools/pull/809)]

Template

- Finished switch from `$baseDir` to `$projectDir` in `iGenomes.conf` and `main.nf`
- Main fix is for `smail_fields` which was a bug introduced in the previous release. Sorry about that!
- Ported a number of small content tweaks from nf-core/eager to the template [[786](https://github.com/nf-core/tools/issues/786)]
- Better contributing documentation, more placeholders in documentation files, more relaxed markdownlint exceptions for certain HTML tags, more content for the PR and issue templates.

Tools helper code

- Pipeline schema: make parameters of type `range` to `number`. [[738](https://github.com/nf-core/tools/issues/738)]
- Respect `$NXF_HOME` when looking for pipelines with `nf-core list` [[798](https://github.com/nf-core/tools/issues/798)]
- Swapped PyInquirer with questionary for command line questions in `launch.py` [[726](https://github.com/nf-core/tools/issues/726)]
- This should fix conda installation issues that some people had been hitting
- The change also allows other improvements to the UI
- Fix linting crash when a file deleted but not yet staged in git [[796](https://github.com/nf-core/tools/issues/796)]

Tools helper code

- Updated `nf_core` documentation generator for building [https://nf-co.re/tools-docs/](https://nf-co.re/tools-docs/)

Template

- Make CI comments work with PRs from forks [[765](https://github.com/nf-core/tools/issues/765)]
- Branch protection and linting results should now show on all PRs
- Updated GitHub issue templates, which had stopped working
- Refactored GitHub Actions so that the AWS full-scale tests are triggered after docker build is finished
- DockerHub push workflow split into two - one for dev, one for releases
- Updated actions to no longer use `set-env` which is now depreciating [[739](https://github.com/nf-core/tools/issues/739)]
- Added config import for `test_full` in `nextflow.config`
- Switched depreciated `$baseDir` to `$projectDir`
- Updated minimum Nextflow version to `20.04.10`
- Make Nextflow installation less verbose in GitHub Actions [[780](https://github.com/nf-core/tools/pull/780)]

Linting

- Updated code to display colours in GitHub Actions log output
- Allow tests to pass with `dev` version of nf-core/tools (previous failure due to base image version)
- Lint code no longer tries to post GitHub PR comments. This is now done in a GitHub Action only.

Template

- Fix command error in `awstest.yml` GitHub Action workflow.
- Allow manual triggering of AWS test GitHub Action workflows.
- Remove TODO item, which was proposing the usage of additional files beside `usage.md` and `output.md` for documentation.
- Added a Podman profile, which enables Podman as container.
- Updated linting for GitHub actions AWS tests workflows.

Linting

- Made a base-level `Dockerfile` a warning instead of failure
- Added a lint failure if the old `bin/markdown_to_html.r` script is found
- Update `rich` package dependency and use new markup escaping to change `[[!]]` back to `[!]` again

Other

- Pipeline sync - fetch full repo when checking out before sync
- Sync - Add GitHub actions manual trigger option

Second patch release to address some small errors discovered in the pipeline template.
Apologies for the inconvenience.

- Fix syntax error in `/push_dockerhub.yml` GitHub Action workflow
- Change `params.readPaths` -> `params.input_paths` in `test_full.config`
- Check results when posting the lint results as a GitHub comment
- This feature is unfortunately not possible when making PRs from forks outside of the nf-core organisation for now.
- More major refactoring of the automated pipeline sync
- New GitHub Actions matrix parallelisation of sync jobs across pipelines [[673](https://github.com/nf-core/tools/issues/673)]
- Removed the `--all` behaviour from `nf-core sync` as we no longer need it
- Sync now uses a new list of pipelines on the website which does not include archived pipelines [[712](https://github.com/nf-core/tools/issues/712)]
- When making a PR it checks if a PR already exists - if so it updates it [[710](https://github.com/nf-core/tools/issues/710)]
- More tests and code refactoring for more stable code. Hopefully fixes 404 error [[711](https://github.com/nf-core/tools/issues/711)]

Patch release to fix the automatic template synchronisation, which failed in the v1.10 release.

- Improved logging: `nf-core --log-file log.txt` now saves a verbose log to disk.
- nf-core/tools GitHub Actions pipeline sync now uploads verbose log as an artifact.
- Sync - fixed several minor bugs, made logging less verbose.
- Python Rich library updated to `>=4.2.1`
- Hopefully fix git config for pipeline sync so that commit comes from nf-core-bot
- Fix sync auto-PR text indentation so that it doesn't all show as code
- Added explicit flag `--show-passed` for `nf-core lint` instead of taking logging verbosity