Resolwe-bio

=================

Added
-----
- Add ``qorts-qc`` (Quality of RNA-seq Tool-Set QC) process
- Add ``workflow-bbduk-star-fc-quant-single`` and
``workflow-bbduk-star-fc-quant-paired`` processes
- Add independent gene filtering and gene filtering based on Cook's distance
in ``DESeq2`` differential expression process

Changed
-------
- **BACKWARD INCOMPATIBLE**: Move gene filtering by expression count
input to ``filter.min_count_sum`` in ``DESeq2`` differential expression
process
- **BACKWARD INCOMPATIBLE:** Require Resolwe 15.x
- Update ``resolwebio/common:1.1.0`` Docker image:

- add QoRTs (1.3.0) package
- bump MultiQC to 1.7.0
- bump Subread package to 1.6.3
- Expose ``maxns`` input parameter in ``bbduk-single`` and
``bbduk-paired`` processes. Make this parameter available in workflows
``workflow-bbduk-star-featurecounts-qc-single``,
``workflow-bbduk-star-featurecounts-qc-paired``,
``workflow-bbduk-star-featurecounts-single`` and
``workflow-bbduk-star-featurecounts-paired``.
- Save CPM-normalized expressions in ``feature_counts`` process. Control
the default expression normalization type (``exp_type``) using the
``normalization_type`` input.
- Bump MultiQC to version 1.7.0 in ``multiqc`` process
- Use ``resolwebio/rnaseq:4.3.0`` with Subread/featureCounts version
1.6.3 in ``feature_counts`` process


=================

=================

Changed
-------
- Bump STAR aligner version to 2.7.0c in ``resolwebio/rnaseq:4.2.2``
- Processes ``alignment-star`` and ``alignment-star-index`` now use Docker
image ``resolwebio/rnaseq:4.2.2`` which contains STAR version ``2.7.0c``
- Persistence of ``basespace-file-import`` process changed from ``RAW`` to
``TEMP``

Added
-----
- Make ``prepare-geo-chipseq`` work with both
``data:chipseq:callpeak:macs2`` and
``data:chipseq:callpeak:macs14`` as inputs

Fixed
-----
- Report correct total mapped reads and mapped reads percentage in
prepeak QC report for ``data:alignment:bam:bowtie2`` inputs in
``macs2-callpeak`` process


=================

=================

Changed
-------
- Enable multithreading mode in ``alignment-bwa-aln`` and
``alignment-bwa-sw``
- Lineary lower the timeout for BigWig calculation when running on
multiple cores

Fixed
-----
- Remove ``pip`` ``--process-dependency-links`` argument in testenv
settings
- Fix walt getting killed when ``sort`` runs out of memory. The ``sort``
command buffer size was limited to the process memory limit.


=================

=================

Changed
-------

Added
-----
- Add the ``FASTQ`` file validator script to the ``upload-fastq-single``,
``upload-fastq-paired``, ``files-to-fastq-single`` and
``files-to-fastq-paired`` processes
- Add ``spikein-qc`` process
- Add to ``resolwebio/rnaseq:4.1.0`` Docker image:

- ``dnaio`` Python library
- Add to ``resolwebio/rnaseq:4.2.0`` Docker image:

- ERCC table
- common Genialis fonts and css file
- spike-in QC report template
- Set ``MPLBACKEND`` environment variable to ``Agg`` in
``resolwebio/common:1.0.1`` Docker image

Fixed
-----
- Fix the format of the output ``FASTQ`` file in the ``demultiplex.py``
script
- Fix NSC and RSC QC metric calculation for ATAC-seq and paired-end
ChIP-seq samples in ``macs2-callpeak`` and ``qc-prepeak`` processes


=================

=================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Require Resolwe 14.x
- **BACKWARD INCOMPATIBLE:** Remove obsolete processes ``findsimilar``
- **BACKWARD INCOMPATIBLE:** Include ENCODE-proposed QC analysis metrics
methodology in the ``macs2-callpeak`` process. Simplified MACS2
analysis inputs now allow the use of sample relations
(treatment/background) concept to trigger multiple MACS2 jobs
automatically using the ``macs2-batch`` or ``macs2-rose2-batch``
processes.
- **BACKWARD INCOMPATIBLE:** Update ``workflow-atac-seq`` inputs to
match the updated ``macs2-callpeak`` process
- Use ``resolwebio/rnaseq:4.0.0`` Docker image in
``alignment-star-index``, ``bbduk-single``, ``bbduk-paired``,
``cuffdiff``, ``cufflinks``, ``cuffmerge``, ``cuffnorm``,
``cuffquant``, ``cutadapt-custom-single``, ``cutadapt-custom-paired``,
``cutadapt-single``, ``cutadapt-paired``,
``differentialexpression-deseq2``, ``differentialexpression-edger``,
``expression-aggregator``, ``feature_counts``, ``goenrichment``,
``htseq-count``, ``htseq-count-raw``, ``index-fasta-nucl``,
``library-strandedness``, ``pca``, ``regtools-junctions-annotate``,
``rsem``, ``salmon-index``, ``trimmomatic-single``,
``trimmomatic-paired``, ``upload-expression``,
``upload-expression-cuffnorm``, ``upload-expression-star``,
``upload-fasta-nucl``, ``upload-fastq-single``,
``upload-fastq-paired``, ``files-to-fastq-single``,
``files-to-fastq-paired``, ``upload-gaf``, ``upload-genome``,
``upload-gff3``, ``upload-gtf`` and ``upload-obo``
- Order statistical groups in expression aggregator output by sample
descriptor field value
- Use ``resolwebio/biox:1.0.0`` Docker image in ``etc-bcm``,
``expression-dicty`` and ``mappability-bcm`` processes
- Use ``resolwebio/common:1.0.0`` Docker image in ``amplicon-table``,
``mergeexpressions``, ``upload-diffexp``, ``upload-etc``,
``upload-multiplexed-single`` and ``upload-multiplexed-paired``
processes
- Use ``resolwebio/base:ubuntu-18.04`` Docker image in
``create-geneset``, ``create-geneset-venn``, ``mergeetc``,
``prepare-geo-chipseq``, ``prepare-geo-rnaseq``, ``upload-cxb``,
``upload-geneset``, ``upload-header-sam``, ``upload-mappability``,
``upload-snpeff`` and ``upload-picard-pcrmetrics`` processes
- Update GATK4 to version 4.0.11.0 in ``resolwebio/dnaseq:4.1.0`` Docker
image. Install and use JDK v8 by default to ensure compatibility with
GATK4 package.
- Use ``resolwebio/dnaseq:4.1.0`` Docker image in ``align-bwa-trim``,
``coveragebed``, ``filtering-chemut``, ``lofreq``,
``picard-pcrmetrics``, ``upload-master-file``, ``upload-variants-vcf``
and ``vc-gatk4-hc`` processes
- Expose reads quality filtering (q) parameter, reorganize inputs and
rename the stats output file in ``alignment-bwa-aln`` process
- Use ``resolwebio/chipseq:4.0.0`` Docker image in ``chipseq-genescore``,
``chipseq-peakscore``, ``macs14``, ``upload-bed`` and ``qc-prepeak``
processes
- Use ``resolwebio/bamliquidator:1.0.0`` Docker image in
``bamliquidator`` and ``bamplot`` processes

Added
-----
- Add biosample source field to ``sample`` descriptor schema
- Add ``background_pairs`` Jinja expressions filter that accepts a list of
data objects and orders them in a list of pairs (case, background) based on
the background relation between corresponding samples
- Add ``chipseq-bwa`` descriptor schema. This schema specifies the
default inputs for BWA ALN aligner process as defined in ENCODE
ChIP-Seq experiments.
- Add support for MACS2 result files to MultiQC process
- Add ``macs2-batch``, ``macs2-rose2-batch`` and ``workflow-macs-rose``
processes
- Add feature symbols to expressions in ``archive-samples`` process

Fixed
-----
- Make ChIP-seq fields in ``sample`` descriptor schema visible when
ChIPmentation assay type is selected
- Fix handling of whitespace in input BAM file name in script
``detect_strandedness.sh``
- Set available memory for STAR aligner to 36GB. Limit the available
memory for STAR aligner ``--limitBAMsortRAM`` parameter to 90% of the
Docker requirements setting
- Set ``bbduk-single`` and ``bbduk-paired`` memory requirements to 8GB
- Fix wrong file path in ``archive-samples`` process


=================

=================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Remove obsolete processes: ``bsmap``,
``mcall``, ``coverage-garvan``, ``igv``, ``jbrowse-bed``,
``jbrowse-gff3``, ``jbrowse-gtf``, ``jbrowse-bam-coverage``,
``jbrowse-bam-coverage-normalized``, ``jbrowse-refseq``,
``fastq-mcf-single``, ``fastq-mcf-paired``, ``hsqutils-trim``,
``prinseq-lite-single``, ``prinseq-lite-paired``,
``sortmerna-single``, ``sortmerna-paired``, ``bam-coverage``,
``hsqutils-dedup``, ``vc-samtools``, ``workflow-heat-seq`` and
``alignment-tophat2``
- **BACKWARD INCOMPATIBLE:** Remove ``jbrowse-bam-coverage`` process
step from the ``workflow-accel`` workflow. The bigwig coverage track
is computed in ``align-bwa-trim`` process instead.
- **BACKWARD INCOMPATIBLE:** Remove ``resolwebio/utils`` Docker image.
This image is replaced by the ``resolwebio/common`` image.
- **BACKWARD INCOMPATIBLE:** Use ``resolwebio/common`` Docker image
as a base image for the ``resolwebio/biox``, ``resolwebio/chipseq``,
``resolwebio/dnaseq`` and ``resolwebio/rnaseq`` images
- **BACKWARD INCOMPATIBLE:** Remove ``resolwebio/legacy`` Docker image.
- Use sample name as the name of the data object in:

- ``alignment-bwa-aln``
- ``alignment-bowtie2``
- ``qc-prepeak``
- ``macs2-callpeak``
- Attach ``macs2-callpeak``, ``macs14`` and ``rose2`` process data to
the case/treatment sample
- Use ``resolwebio/dnaseq:4.0.0`` docker image in ``align-bwa-trim``
process
- Use ``resolwebio/rnaseq:4.0.0`` docker image in aligners:
``alignment-bowtie``, ``alignment-bowtie2``, ``alignment-bwa-mem``,
``alignment-bwa-sw``, ``alignment-bwa-aln``, ``alignment-hisat2``,
``alignment-star`` and ``alignment-subread``.
- Set memory limits in ``upload-genome``, ``trimmomatic-single`` and
``trimmomatic-paired`` processes
- Improve error messages in differential expression process ``DESeq2``

Added
-----
- Add ``makedb (WALT 1.01)`` - callable as ``makedb-walt``, tool to
create genome index for WALT aligner, to ``resolwebio/rnaseq`` docker
image
- Add ``resolwebio/wgbs`` docker image including the following tools:

- ``MethPipe (3.4.3)``
- ``WALT (1.01)``
- ``wigToBigWig (kent-v365)``
- Add ``resolwebio/common`` Docker image. This image includes common
bioinformatics utilities and can serve as a base image for other,
specialized ``resolwebio`` Docker images: ``resolwebio/biox``,
``resolwebio/chipseq``, ``resolwebio/dnaseq``
and ``resolwebio/rnaseq``.
- Add ``shift`` (user-defined cross-correlation peak strandshift) input
to ``qc-prepeak`` process
- Add ATAC-seq workflow
- Compute index for ``WALT`` aligner on genome upload and support
uploading the index together with the genome
- Add ``Whole genome bisulfite sequencing`` workflow and related WGBS
processes:

- ``WALT``
- ``methcounts``
- ``HMR``
- Add bedClip to `resolwebio/chipseq:3.1.0` docker image
- Add ``resolwebio/biox`` Docker image. This image is based on the
``resolwebio/common`` image and includes Biox Python library for
Dictyostelium RNA-Seq analysis support.
- Add ``resolwebio/snpeff`` Docker image. The image includes
SnpEff (4.3K) tool.
- Add spike-in names, rRNA and globin RNA cromosome names in
``resolwebio/common`` image
- Add UCSC bedGraphtoBigWig tool for calculating BigWig in
``bamtobigwig.sh`` script. In ``align-bwa-trim`` processor set this
option (that BigWig is calculated by UCSC tool instead of deepTools),
because it is much faster for amplicon files. In other processors update
the input parameters for ``bamtobigwig.sh``: ``alignment-bowtie``,
``alignment-bowtie2``, ``alignment-bwa-mem``, ``alignment-bwa-sw``,
``alignment-bwa-aln``, ``alignment-hisat2``, ``alignment-star``
``alignment-subread``, ``upload-bam``, ``upload-bam-indexed`` and
``upload-bam-secondary``.
- In ``bamtobigwig.sh`` don't create BigWig when bam file was aligned on
globin RNA or rRNA (this are QC steps and BigWig is not needed)

Fixed
-----
- **BACKWARD INCOMPATIBLE:** Use user-specificed distance metric in
hierarchical clustering
- Handle integer expression values in hierarchical clustering
- Fix Amplicon table gene hyperlinks for cases where multiple genes
are associated with detected variant
- Handle empty gene name in expression files in PCA
- Fix PBC QC reporting in ``qc-prepeak`` process for a case where
there are no duplicates in the input bam
- Fix ``macs2-callpeak`` process so that user defined fragment lenth
has priority over the ``qc-prepeak`` estimated fragment length when
shifting reads for post-peakcall QC
- Fix ``macs2-callpeak`` to prevent the extension of intervals beyond
chromosome boundaries in MACS2 bedgraph outputs
- Fix warning message in hierarchical clustering of genes to display gene
names


=================