Resolwe-bio

=================

Fixed
-----
- Fix ``htseq-count-raw`` process to correctly map features with associated
feature symbols.


=================

=================

Fixed
-----
- Handle missing gene expression in hierarchical clustering of genes. If one or
more genes requested in gene filter are missing in selected expression files
a warning is issued and hierarchical clustering of genes is computed with the
rest of the genes instead of failing.
- Fix PCA computation for single sample case


=================

=================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Require Resolwe 13.x
- **BACKWARD INCOMPATIBLE:** Remove ``gsize`` input from
``macs2-callpeak`` process and automate genome size selection
- **BACKWARD INCOMPATIBLE:** Set a new default ``sample`` and ``reads``
descriptor schema. Change slug from ``sample2`` to ``sample``, modify group
names, add ``cell_type`` field to the new ``sample`` descriptor schema, and
remove the original ``sample``, ``sample-detailed``, and ``reads-detailed``
descriptor schemas.
- **BACKWARD INCOMPATIBLE:** Unify types of ``macs14`` and
``macs2-callpeak`` processes and make ``rose2`` work with both
- **BACKWARD INCOMPATIBLE:** Remove ``replicates`` input in ``cuffnorm``
process. Use sample relation information instead.
- Use ``resolwebio/chipseq:3.0.0`` docker image in the following processes:

- ``macs14``
- ``macs2-callpeak``
- ``rose2``
- Downgrade primerclip to old version (v171018) in ``resolwebio/dnaseq:3.3.0``
docker image and move it to google drive.
- Move ``bam-split`` process to ``resolwebio/rnaseq:3.7.1`` docker image
- Count unique and multimmaping reads in ``regtools-junctions-annotate``
process

Added
-----
- Add ``qc-prepeak`` process that reports ENCODE3 accepted ChIP-seq and
ATAC-seq QC metrics
- Add QC report to ``macs2-callpeak`` process
- Add combining ChIP-seq QC reports in ``archive-samples`` process
- Add detection of globin-derived reads as an additional QC step in the
``workflow-bbduk-star-featurecounts-qc-single`` and
``workflow-bbduk-star-featurecounts-qc-paired`` processes.
- Add mappings from ENSEMBL or NCBI to UCSC chromosome names and deepTools
(v3.1.0) to ``resolwebio/dnaseq:3.3.0`` docker image
- Add BigWig output field to following processors:

- ``align-bwa-trim``
- ``upload-bam``
- ``upload-bam-indexed``
- ``upload-bam-secondary``
- Add ``replicate_groups`` Jinja expressions filter that accepts a list of
data objects and returns a list of labels determining replicate groups.
- Add 'Novel splice junctions in BED format' output to
``regtools-junctions-annotate`` process, so that user can visualize only
novel splice juntions in genome browsers.

Fixed
-----
- Fix handling of numerical feature_ids (NCBI source) in
``create_expression_set.py`` script
- Make ``chipseq-peakscore`` work with gzipped narrowPeak input from
``macs2-callpeak``
- Use uncompressed FASTQ files as input to STAR aligner to prevent
issues on (network) filesystems without FIFO support


=================

=================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Require Resolwe 12.x
- **BACKWARD INCOMPATIBLE:** Remove obsolete processes: ``assembler-abyss``,
``cutadapt-amplicon``, ``feature_location``, ``microarray-affy-qc``,
``reads-merge``, ``reference_compatibility``, ``transmart-expressions``,
``upload-hmmer-db``, ``upload-mappability-bigwig``,
``upload-microarray-affy``.
- **BACKWARD INCOMPATIBLE:** Remove obsolete descriptor schema: ``transmart``.
- **BACKWARD INCOMPATIBLE:** Remove tools which are not used by any process:
``clustering_leaf_ordering.py``, ``go_genesets.py``, ``VCF_ad_extract.py``,
``volcanoplot.py``, ``xgff.py``, ``xgtf2gff.py``.
- **BACKWARD INCOMPATIBLE:** Management command for inserting features and
mappings requires PostgreSQL version 9.5 or newer
- Update the meta data like name, description, category, etc. of most of the
processes
- Speed-up management command for inserting mappings
- Change location of cufflinks to Google Drive for resolwebio/rnaseq Docker
build
- Calculate alignment statistics for the uploaded alignment (.bam) file in the
``upload-bam``, ``upload-bam-indexed`` and ``upload-bam-secondary`` processes.
- Annotation (GTF/GFF3) file input is now optional for the creation of the
STAR genome index files. Annotation file can be used at the alignment stage
to supplement the genome indices with the set of known features.
- Trigger process warning instead of process error in the case when
``bamtobigwig.sh`` scripts detects an empty .bam file.
- Set the default reads length filtering parameter to 30 bp in the
``rna-seq-bbduk-star-featurecounts`` and ``kapa-rna-seq-bbduk-star-featurecounts``
experiment descriptor schema. Expand the kit selection choice options in the
latter descriptor schema.

Added
-----
- Add ``MultiQC (1.6.0)`` and ``Seqtk (1.2-r94)`` to the
``resolwebio/utils:1.5.0`` Docker image
- Add ``sample2`` descriptor schema which is the successor of the original
``sample`` and ``reads`` descriptor schemas
- Add bedToBigBed and Tabix to resolwebio/rnaseq:3.7.0 docker image
- Add ``HS Panel`` choice option to the ``amplicon-master-file`` descriptor
schema
- Add MultiQC process
- Add process for the Seqtk tool ``sample`` sub-command. This process allows
sub-sampling of ``.fastq`` files using either a fixed number of reads or the
ratio of the input file.
- Add MultiQC analysis step to the ``workflow-bbduk-star-featurecounts-single``
and ``workflow-bbduk-star-featurecounts-single`` processes.
- Add ``workflow-bbduk-star-featurecounts-qc-single`` and
``workflow-bbduk-star-featurecounts-qc-paired`` processes which support
MultiQC analysis, input reads down-sampling (using Seqtk) and rRNA
sequence detection using STAR aligner.
- Add to ``resolwebio/chipseq`` Docker image:

- ``bedtools (2.25.0-1)``
- ``gawk (1:4.1.3+dfsg-0.1)``
- ``picard-tools (1.113-2)``
- ``run_spp.R (1.2) (as spp)``
- ``SPP (1.14)``
- Add ``regtools-junctions-annotate`` process that annotates novel splice
junctions.
- Add ``background`` relation type to fixtures

Fixed
-----
- Track ``source`` information in the ``upload-fasta-nucl`` process.
- When STAR aligner produces an empty alignment file, re-sort the alignment
file to allow successful indexing of the output ``.bam`` file.
- Create a symbolic link to the alignment file in the ``feature_counts`` process,
so that relative path is used in the quantification results. This prevent the
FeatureCounts output to be listed as a separate sample in the MultiQC reports.
- Fix handling of expression objects in ``archive-samples`` process


===================

===================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Require Resolwe 11.x
- **BACKWARD INCOMPATIBLE:** Use read count instead of sampling rate
in strandedness detection
- **BACKWARD INCOMPATIBLE:** Remove ``genome`` input from ``rose2``
process and automate its selection
- **BACKWARD INCOMPATIBLE:** Refactor ``cutadapt-paired`` process
- **BACKWARD INCOMPATIBLE:** Improve leaf ordering performance in gene and
sample hierarchical clustering. We now use exact leaf ordering which has
been recently added to ``scipy`` instead of an approximate in-house
solution based on nearest neighbor algorithm. Add informative warning
and error messages to simplify troubleshooting with degenerate datasets.
- Remove ``igvtools`` from ``resolwebio/utils`` Docker image
- Improve helper text and labels in processes used for sequencing data upload
- Allow using custom adapter sequences in the
``workflow-bbduk-star-featurecounts-single`` and
``workflow-bbduk-star-featurecounts-paired`` processes
- Change chromosome names from ENSEMBL / NCBI to UCSC (example: "1" to
"chr1") in BigWig files. The purpose of this is to enable viewing BigWig
files in UCSC genome browsers for files aligned with ENSEBML or NCBI genome.
This change is done by adding script bigwig_chroms_to_ucsc.py to
bamtobigwig.sh script.
- Reduce RAM requirement in SRA import processes

Added
-----
- Add two-pass mode to ``alignment-star`` process
- Add ``regtools (0.5.0)`` to ``resolwebio/rnaseq`` Docker image
- Add KAPA experiment descriptor schema
- Add ``resdk`` Python 3 package to ``resolwebio/utils`` Docker image
- Add to ``cutadapt-single`` process an option to discard reads having more
'N' bases than specified.
- Add workflows for single-end ``workflow-cutadapt-star-featurecounts-single``
and paired-end reads ``workflow-cutadapt-star-featurecounts-paired``.
Both workflows consist of preprocessing with Cutadapt, alignment
with STAR two pass mode and quantification with featureCounts.
- Add descriptor schema ``rna-seq-cutadapt-star-featurecounts``

Fixed
-----
- **BACKWARD INCOMPATIBLE:** Fix the ``stitch`` parameter handling in
``rose2``
- fix ``upload-gtf`` to create JBrowse track only if GTF file is ok
- Pin ``sra-toolkit`` version to 2.9.0 in ``resolwebio/utils`` Docker image.
- Fix and improve ``rose2`` error messages
- Fail gracefully if bam file is empty when producing bigwig files
- Fail gracefully if there are no matches when mapping chromosome names


===================

===================

Changed
-------
- **BACKWARD INCOMPATIBLE:** Remove management command module
- **BACKWARD INCOMPATIBLE:** Remove filtering of genes with low expression
in PCA analysis
- **BACKWARD INCOMPATIBLE:** Remove obsolete RNA-seq DSS process
- Expand error messages in ``rose2`` process
- Check for errors during download of FASTQ files and use
``resolwebio/utils:1.3.0`` Docker image in import SRA process
- Increase Feature's full name's max length to 350 to support a long full
name of "Complement C3 Complement C3 beta chain C3-beta-c Complement C3
alpha chain C3a anaphylatoxin Acylation stimulating protein Complement C3b
alpha' chain Complement C3c alpha' chain fragment 1 Complement C3dg
fragment Complement C3g fragment Complement C3d fragment Complement C3f
fragment Complement C3c alpha' chain fragment 2" in Ensembl

Added
-----
- Add `exp_set` and `exp_set_json` output fields to expression processes:

- ``feature_counts``
- ``htseq-count``
- ``htseq-count-raw``
- ``rsem``
- ``upload-expression``
- ``upload-expression-cuffnorm``
- ``upload-expression-star``
- Add 'Masking BED file' input to ``rose2`` process which allows
masking reagions from the analysis
- Add ``filtering.outFilterMismatchNoverReadLmax`` input to
``alignment-star`` process
- Add mappings from ENSEMBL or NCBI to UCSC chromosome names to
``resolwebio/rnaseq:3.5.0`` docker image

Fixed
-----
- Fix peaks BigBed output in ``macs14`` process
- Remove duplicated forward of ``alignIntronMax`` input field in
BBDuk - STAR - featureCounts workflow
- Make ``cuffnorm`` process attach correct expression data objects to
samples
- Fix ``upload-gtf`` in a way that GTF can be shown in JBrowse. Because
JBrowse works only with GFF files, input GTF is converted to GFF from
which JBrowse track is created.


===================