Signac-flow

---------------------

- Add the ``-v/--verbosity`` and ``--show-traceback`` option to the project interface, which allows for more fine-grained control over the message verbosity. The ``--debug`` option is now equivalent to ``-vv --show-traceback``.
- The message verbosity of the project class was overall reduced.
- Global options including (``--debug`` and ``--verbose``) can be used at any place within the project command and must no longer be placed before the sub command, *e.g.*, the following commands are equivalent: ``$ python project.py run --debug`` and ``$ python project.py --debug run``.
- Implement the ``-p/--parallel`` option for the project run command.
- Use cloudpickle when encountering pickling issues during parallel execution (when installed).
- Implement the status ``--ignore-errors`` option.
- Handle changes to the project data space during running execution, *e.g.*, removed jobs.
- Print the ``operation.cmd`` attribute to screen in ``run --pretend`` mode, not ``repr(operation)``.
- Show progressbar while gathering pending operations.

===========

Major changes
-------------

1. The generation of execution and submission scripts is now based on the jinja2_ templating system.
2. The new decorator API for the definition of a :class:`~.flow.FlowProject` class largely reduces the amount of boiler plate code needed to implement FlowProjects.
It also removes the necessity to have at least two modules, *e.g.*, one ``project.py`` and one ``operations.py`` module.
3. Serial execution is now the default for all project sub commands, that includes ``run``, ``script``, and ``submit``.
Parallel execution must be explicitly enabled with the ``--parallel`` option.
4. The ``run`` command executes **all eligible** operations, that means you don't have to run the command multiple times to "cycle" through all pending operations. Accidental infinite loops are automatically avoided.
5. Execution scripts generated with the `script` option are always bundled.
The previous behavior, where the script command would print multiple scripts to screen unless the ``--bundle`` option was provided did not make much sense.

See the full :ref:`changelog <0.6.0>` below for detailed information on all changes.

.. _jinja2: http://jinja.pocoo.org/

How to migrate existing projects
--------------------------------

If your project runs with flow 0.5 without any DeprecationWarnings (that means no messages when running Python with the -W flag), then you don't *have* to do anything. Version 0.6 is mostly backwards compatible to 0.5, with the execption of custom script templating.

Since 0.6 uses jinja2_ to generate execution and submission scripts, the previous method of generating custom scripts by overloading the ``FlowProject.write_script*()`` methods is now deprecated.
That means that if you overloaded any of these functions, the new templating system is disabled, and **flow** will fallback to the old templating system and you won't be able to use jinja2 templates.

If you decide to migrate to the new API, those are the steps you most likely have to take:

1. Replace all ``write_script*()`` methods and replace them with a custom template script, *e.g.*, `templates/script.sh` within your project root directory.
2. Optionally, use the new decorator API instead of ``FlowProject.add_operation`` to add operations to your FlowProject.
3. Optionally, use the new decorator API to define label functions for your FlowProject.
4. Execute your project with the Python ``-w`` option to make DeprecationWarnings visible and address all issues.

We recommend to go through the tutorial on signac-docs_ to learn how to best take advantage of **flow** 0.6.

.. _signac-docs: https://signac-docs.readthedocs.io/en/latest

---------------------

Major updates and new features
++++++++++++++++++++++++++++++

- Use jinja2 as templating engine for the generation of execution and submission scripts.
- Add decorator API for the definition of FlowProject operations and label functions.
- Revise the status view to render on a per job-operation basis, not on a per job basis.
- The <project> run function executes all pending operations, not just the next pending ones.
- The <project> script function no longer supports explicit bundling, all operations are bundled by default.
- The default execution mode for script and submission script bundling is serial, not parallel.
- Add the operations directive parameter, which provides a more generalized interface to specify required resources or other execution related metadata.
- Add support for XSEDE Stampede2.
- Add simple-scheduler, for local testing of scheduled workflows.
- Allow the override of the detected environment with the `SIGNAC_FLOW_ENVIRONMENT` variable.
- The `$ flow init` commad initializes the signac project if not project is found.

API changes (breaking)
++++++++++++++++++++++

- The FlowProject.run() method arguments were changed [1]; the old API is better supported by the new FlowProject.run_operations() function.
- The FlowProject.submit() and .submit_operations() method arguments were changed [1].
- The JobOperation constructor arguments were changed; the old API ist still supported.


API changes (non-breaking)
++++++++++++++++++++++++++

- Unify the job and operation selection API for the run/script/submit commands.
- Add FlowProject.operation() decorator function.
- Add FlowProject.label() docorator function.
- The FlowProject.write_human_readable_statepoints() method is deprecated.
- All FlowProject methods relating to the old templating system are deprecated, that includes all write_script*() methods.
- Add flow.cmd decorator function.
- Add flow.directives decorator function.

[1] A reasonable attempt to support legacy API use is made, but may fail under some circumstances.

---------------------

- Fix issue, where operations with spaces in their name would not be accepted by the SLURM scheduler.
- Add environment profile for XSEDE Bridges.
- Update the environment profile for XSEDE Comet to use the shared queue by default and provide options to specify the memory allocation.
- Improve performance of the project update status function.

---------------------

- Fix issue with the SLURM scheduler, where the queue status could not be parsed.

---------------------

- Fix issue with <project> run, where operation commands consist of multiple shell commands.
- Fix issue where the <project> status output showed negative values for the number of lines omitted (issue 12).
- Raise error when trying to provide a timeout for <project> run in serial execution in combination with Python 2.7; this mode of execution is not supported for Python versions 2.7.x.
- Enforce that the <project> status --overview-max-lines (-m) argument is positive.