********************************************************************
12th March 2015
=====================================================================================================================
1) Bug fixes
=====================================================================================================================
* ``pipeline_printout_graph()`` incompatibility with python3 fixed
* checkpointing did not work correctly with :ref:`split(...) <decorators.split>` and :ref:`subdivide(...) <decorators.subdivide>`
=====================================================================================================================
2) `transform `(..., suffix("xxx"),` :red:`output_dir` `= "/new/output/path")`
=====================================================================================================================
Thanks to the suggestion of Milan Simonovic.
:ref:`transform(..., suffix(...) ) <decorators.transform>` has easy to understand syntax and takes care of all the common use cases
of Ruffus.
However, when we need to place the output in a different directories, we suddenly have to plunge into the deep end and parse file paths using
:ref:`regex() <decorators.regex>` or :ref:`formatter() <new_manual.formatter>`.
Now, :ref:`transform <decorators.transform>` takes an optional ``output_dir`` named parameter so that we can continue to use :ref:`suffix() <new_manual.suffix>` even when the output needs
to go into a new directory.
.. <<Python
.. code-block:: python
:emphasize-lines: 2,3,9
input/a.fasta -> output/a.sam
input/b.fasta -> output/b.sam
starting_files = ["input/a.fasta","input/b.fasta"]
transform(starting_files,
suffix('.fasta'),
'.sam',
output_dir = "output")
def map_dna_sequence(input_file, output_file) :
pass
..
Python
See example ``test\test_suffix_output_dir.py``
=====================================================================================================================
2) Named parameters
=====================================================================================================================
Decorators can take named parameters.
These are self documenting, and improve clarity.
Note that the usual Python rules for function parameters apply:
* Positional arguments must precede named arguments
* Named arguments cannot be used to fill in for "missing" positional arguments
For example the following two functions are identical:
**Positional parameters:**
.. <<Python
.. code-block:: python
merge(prev_task, ["a.summary", "b.summary"], 14, "extra_info", {"a":45, "b":5})
def merge_task(inputs, outputs, extra_num, extra_str, extra_dict):
pass
..
Python
**Named parameters:**
.. <<Python
.. code-block:: python
new style is a bit clearer
merge(input = prev_task,
output = ["a.summary", "b.summary"],
extras = [14, "extra_info", {"a":45, "b":5}]
)
def merge_task(inputs, outputs, extra_num, extra_str, extra_dict):
pass
..
Python
.. warning::
``,extras=`` takes all the *extras* parameters (``14, "extra_info", {"a":45, "b":5}``) as a single list
* :ref:`split(...) <decorators.split>` and :ref:`merge(...) <decorators.merge>`
* *input*
* *output*
* [*extras*\ ]
* :ref:`transform(...) <decorators.transform>` and :ref:`mkdir(...) <decorators.mkdir>`
* *input*
* *filter*
* [*replace_inputs* or *add_inputs*\ ]
* *output*
* [*extras*\ ]
* [*output_dir*\ ]
* :ref:`collate(...) <decorators.collate>` and :ref:`subdivide(...) <decorators.collate>`
* *input*
* *filter*
* *output*
* [*extras*\ ]
* :ref:`originate(...) <decorators.originate>`
* *output*
* [*extras*\ ]
* :ref:`product(...) <decorators.product>`, :ref:`permutations(...) <decorators.permutations>`, :ref:`combinations(...) <decorators.combinations>`, and :ref:`combinations_with_replacement(...) <decorators.combinations_with_replacement>`
* *input*
* *filter*
* [*input2...NNN*\ ] (only for ``product``)
* [*filter2...NNN*\ ] (only for ``product``) where NNN is an incrementing number
* *tuple_size* (except for ``product``)
* [*replace_inputs* or *add_inputs*\ ]
* *output*
* [*extras*\ ]
=============================================
3) New object orientated syntax for Ruffus
=============================================
Ruffus Pipelines can now be created directly using the new ``Pipeline`` and ``Task`` objects instead of via decorators.
.. <<python
.. code-block:: python
:emphasize-lines: 9
make ruffus pipeline
my_pipeline = Pipeline(name = "test")
my_pipeline.transform(task_func = map_dna_sequence,
input = starting_files,
filter = suffix('.fasta'),
output = '.sam',
output_dir = "output")
my_pipeline.run()
..
python
This new syntax is fully compatible and inter-operates with traditional Ruffus syntax using decorators.
Apart from cosmetic changes, the new syntax allows different instances of modular Ruffus sub-pipelines
to be defined separately, in different python modules and then joined together flexible at runtime.
The new syntax and discussion are introduced :ref:`here <new_syntax>`.
********************************************************************