Miniwdl

`miniwdl run`

* `--dir` behavior revised to be more predictable: always run in a new timestamp-named subdirectory of the specified directory, unless the latter ends with `.`
* Automatically download input files specified as public http[s]/ftp/gs URIs
* Honor `runtime.maxRetries` to retry task command or docker failures
* `output_links` feature expanded to apply to all nested task/workflow runs, and include Files embedded in structs and other composite output values
* Manipulate ownership/permissions to improve security and user experience:
* as run completes, chown everything in run directory to invoking user and primary group
* run task commands with membership in invoking user's primary group, ensuring access to working directory even if they've dropped privileges (a good docker security practice)
* `--as-me` to force all task commands to run as invoking user (more secure, but blocks commands that assume root e.g. apt-get)
* Revert to always running command in bash (as required by WDL spec) rather than container $SHELL

`miniwdl check`

* Suppress `UnnecessaryQuantifier` for optional `File?` task outputs

`miniwdl run`

* generates a "rerun" script in the run directory, which can be sourced from the shell to re-run with the same inputs (149)
* use docker image's $SHELL to interpret task command, instead of hard-coding bash
* log messages have been revised and reformatted to facilitate programmatic consumption (248)
* probable fix for a sporadic hang under erratic host performance, such as when out of memory (263)
* based on runtime.memory, create host memory reservations to avoid overbooking parallel memory-intensive tasks (250)
* memory reservations affect task scheduling, but aren't enforced limits (unlike runtime.cpu, which creates both reservations and limits)
* shares thread pools to respect task concurrency limits across sub-workflow calls (203)

**Known limitations:**

* Task output files must reside in or under the initial working directory (214)
* Task input files are mounted read-only by default; commands to rename or remove them can succeed only with `--copy-input-files` (210)
* Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (193)
* Scheduler ignores runtime.disks and runtime.maxTries (251)

Please vote on these and/or report other interoperability problems via [Issues](https://github.com/chanzuckerberg/miniwdl/issues)!

`miniwdl run`

* support optional `File?` outputs from tasks (255, https://github.com/openwdl/wdl/pull/310)
* display the available inputs/outputs in source order
* fix duplication in `workflow.log` file

**Known limitations:**

* Task output files must reside in or under the initial working directory (214)
* Task input files are mounted read-only by default; commands to rename or remove them can succeed only with `--copy-input-files` (210)
* Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (193)
* Scheduler considers each tasks's `runtime.cpu` (default 1) versus the available host processors, but ignores `runtime.{memory,disks}` (250)
* Scheduler may exceed the `-` parallelism limit argument when sub-workflows are called, potentially up to host processors (203).

Please vote on these and/or report other interoperability problems via [Issues](https://github.com/chanzuckerberg/miniwdl/issues)!

`miniwdl cromwell`

* Cromwell 47

`WDL`

* permit unbound, optional declarations in task/workflow bodies (outside of inputs) as a way of creating a null value for later use (253)
* `WDL.Type.Any` always coerces to/from optional types
* preserve `Value.expr` references when value is coerced

`miniwdl run`

The built-in runner can now parallelize tasks and sub-workflows on the local host, using [Docker Swarm](https://docs.docker.com/engine/swarm/) locally to schedule containers (which it'll enable automatically). Start with `miniwdl run_self_test` to quickly test the installation viability.

**Known limitations:**

* Task output files must reside in or under the initial working directory (214)
* Task input files are mounted read-only by default; commands to rename or remove them can succeed only with `--copy-input-files` (210)
* Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (193)
* Scheduler considers each tasks's `runtime.cpu` (default 1) versus the available host processors, but ignores `runtime.{memory,disks}` (250)
* Scheduler may exceed the `-` parallelism limit argument when sub-workflows are called, potentially up to host processors (203).

Please vote on these and/or report other interoperability problems via [Issues](https://github.com/chanzuckerberg/miniwdl/issues)!

**Other changes:**

* Add `:latest` tag to `runtime.docker` if no other tag present (232 jdidion)
* Mount all input files into one in-container directory (except when filenames would collide) to improve interoperability using hts index files

`miniwdl cromwell`

* Cromwell 46.1

`miniwdl check`

* Suppress `UnusedImport` warning when imported struct definitions are used (236 DavyCats)

`WDL`

* Fix `str()` on `Apply` AST nodes for certain functions (242 DavyCats)

`miniwdl run`

* reduce likelihood of CPython `RecursionError` due to deep-copying of WDL values (239 nh13)

**Standing limitations:**

* Calls run sequentially (202, 203)
* Task input files are mounted read-only; commands to rename or remove them will fail (210)
* Task output files must reside in or under the initial working directory (214)
* Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (193)

Please vote on these and/or report other interoperability problems via [Issues](https://github.com/chanzuckerberg/miniwdl/issues)!

`miniwdl cromwell`

* accept relative file paths in input JSON file, interpreting them relative to current working directory (230 prihoda)
* Cromwell 45.1

`WDL`

* recognize `version development` and use the most up-to-date grammar available (233 jdidion)

`miniwdl run`

* Improved logging with tty colors and other enhancements
* New subcommand `miniwdl run_self_test` runs a little workflow to test the installation
* Library functions involving file I/O (e.g. `read_string()`, `write_lines()`) are now available in workflows as well as tasks (204)
* Prevent use of host files neither supplied in the workflow inputs nor generated by the workflow (e.g. `/etc/passwd`)
* Fix up type coercions in call inputs and struct instances

**Remaining limitations:**

* Calls run sequentially (202, 203)
* Task input files are mounted read-only; commands to rename or remove them will fail (210)
* Task output files must reside in or under the initial working directory (214)
* Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (193)

Please vote on these and/or report other interoperability problems via [Issues](https://github.com/chanzuckerberg/miniwdl/issues)!

`WDL` package

**Breaking API changes:**

* `WDL.Env` rewritten with an easier, more-Pythonic interface
* Simplify compound `Value` constructors so that e.g. `Value.Array` needs to be told the array item type rather than the full `Type.Array(...)`

We now plan to minimize further breaking API changes to the WDL AST data structures, on the way to a 1.0 version.

**Other changes:**

* Add `__str__` to Expr classes (215 DavyCats)
* String-to-Int/Float coercions allowed, may fail at runtime (198)
* Statically reject use of `stdout()`, `stderr()`, and `glob()` outside of task output sections
* Set `pos : SourcePosition` attribute on the subset of `WDL.Type` objects instantiated by the syntax parser (e.g. `Decl.type` & `StructTypeDef` dinvlad)
* Fix `WDL.resolve_file_import()` pass-through call to `WDL.Tree.resolve_file_import()` (dinvlad)
* Tighten name collision rules for 1.0+ workflow outputs (196)



Housekeeping

* Parallelized unit test suite using pytest
* Cromwell 45, lark-parser 0.7.3