Miniwdl

`miniwdl run`

* New [configuration loader](https://miniwdl.readthedocs.io/en/latest/runner_reference.html#configuration) sourcing from command-line arguments, environment variables, and/or .cfg file.
* [URI input file downloads can be cached](https://miniwdl.readthedocs.io/en/latest/runner_reference.html#file-download-cache) for reuse in multiple runs. (Opt-in via configuration)
* Rigorize error logging and reporting, including `error.json` left in each failed run directory (346)
* Store a copy of the original WDL source files in each top-level run directory (292)
* Task runner plugins are able to manipulate inputs/outputs, commands, etc. ([example](https://github.com/chanzuckerberg/miniwdl/tree/master/examples/plugin_task_omnibus)). Workflow-level and other plugin extension points still to come.
* File URI download plugin interface revised (breaking changes) to fit with the new plugin scheme ([example](https://github.com/chanzuckerberg/miniwdl/tree/master/examples/plugin_download_awscli))

`miniwdl cromwell`

**Deprecation notice:** `miniwdl cromwell` may be retired in the medium-term future; see and comment on 351

`miniwdl localize`

**Deprecation notice:** `miniwdl localize` will probably be retired shortly, superseded by the new URI cache feature.

`miniwdl run`

* Add [example download plugins](https://github.com/chanzuckerberg/miniwdl/tree/master/examples) (API may still change)
* Set docker LogConfig for resource detection container so it works as expected with non-default configuration e.g. ECS (340 kislyuk)

**Standing limitations:**

* Task output files must reside in or under the initial working directory (214)
* Task input files are mounted read-only by default; commands to rename or remove them can succeed only with `--copy-input-files` (210)

Please vote on these and/or report other interoperability problems via [Issues](https://github.com/chanzuckerberg/miniwdl/issues)!

`WDL`

* Statically prevent workflow from calling itself
* Make `size()` work on null inputs, per WDL spec
* lark-parser 0.8.1

`miniwdl run`

* Fix Docker failure with lengthy call names (327 kislyuk)
* Use POSIX ERE for `sub()` (321 DavyCats cf. openwdl/wdl243)
* macOS compatibility improvements
* default `--runtime-cpu-max` and `--runtime-memory-max` to limits of Docker for Mac's virtual machine
* `miniwdl run_self_test` failure displays a hint on overriding `TMPDIR`
* remaining issues tracked in 145

`miniwdl check`

* Whitelist additional task `runtime` keys (323 claymcleod)

`WDL`

* Improve type inference for Map literals (256 nh13)
* Improve type inference for `read_json()` (320 illusional)

`miniwdl run`

* New interface to "plug in" support for additional input file URI schemes such as s3://, with security credentials. [Example here](https://gist.github.com/mlin/7c661521bc93811cc4aa48dcd0b1089a) but the interface may still change.
* The output environments and JSON now refer to Files via the symbolic links created in the `output_links` subdirectory, which in turn refer to the original (hard-linked) files. This means all File outputs exhibit paths within that directory, making them easier to postprocess.
* Input JSON keys for subworkflow calls may now (but still needn't) include the subworkflow name as a namespace following the call name, as expected by Cromwell (193 DavyCats)
* Understand `~` as home directory in command-line file input (kislyuk)
* Remove query string from local filename of WDL imported by URI (312 kislyuk)

**Standing limitations:**

* Task output files must reside in or under the initial working directory (214)
* Task input files are mounted read-only by default; commands to rename or remove them can succeed only with `--copy-input-files` (210)

Please vote on these and/or report other interoperability problems via [Issues](https://github.com/chanzuckerberg/miniwdl/issues)!

`WDL`

* Forbid any call's name from colliding with that of the containing workflow.

[](https://pypi.org/project/miniwdl/) [](https://anaconda.org/conda-forge/miniwdl) **[Getting Started](https://miniwdl.readthedocs.io/en/latest/getting_started.html)**

`miniwdl run`

* Accept HTTP[S] URIs for the `-i` input JSON file
* Fix log file handle leak which would eventually cause 'Too many open files' error (301)
* Improved error reporting in file URI download step
* Document [guidelines](https://miniwdl.readthedocs.io/en/latest/runner_advanced.html) for optimal performance and reliability

**Standing limitations:**

* Task output files must reside in or under the initial working directory (214)
* Task input files are mounted read-only by default; commands to rename or remove them can succeed only with `--copy-input-files` (210)
* Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (193)

Please vote on these and/or report other interoperability problems via [Issues](https://github.com/chanzuckerberg/miniwdl/issues)!

`miniwdl check`

* Prevent StringCoercion and OptionalCoercion warnings on some typical interpolation use cases
* Improve UnusedImport detection

`WDL`

* Permit "comments" in input JSON (ignore keys starting with )
* Allow URI-downloaded WDL to use `..` updir relative imports

`miniwdl localize`

This new tool receives WDL and a Cromwell-style input JSON for it, which may reference Files by URIs. It downloads such URIs to a local directory, and generates equivalent JSON with the local filenames.

Although `miniwdl run` can download URI inputs without this intermediate step, this tool can be helpful to avoid repeating downloads across multiple runs/attempts.

`WDL`

* Permit `"~{sep=' ' array}"` interpolations where `array` item type may be optional (null values are written as empty strings)
* Fix negation/infix operator precedence in WDL boolean expressions
* Forbid identifiers starting with `_`, per WDL spec

`miniwdl run`

* Permit relative imports from WDL files downloaded from URI
* It's now possible to run many workflows from the internet without manually downloading anything in preparation, e.g. `miniwdl run https://raw.githubusercontent.com/my/repo/master/workflow.wdl input1=https://s3.amazonaws.com/mybucket/file.txt`
* `--runtime-defaults` accepts a JSON object or filename with default values for all task runtime sections
* `--max-runtime-{cpu,memory}` renamed to `--runtime-{cpu,memory}-max` for consistency

**Standing limitations:**

* Task output files must reside in or under the initial working directory (214)
* Task input files are mounted read-only by default; commands to rename or remove them can succeed only with `--copy-input-files` (210)
* Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (193)

Please vote on these and/or report other interoperability problems via [Issues](https://github.com/chanzuckerberg/miniwdl/issues)!


Housekeeping

* Sped up [Travis CI builds](https://travis-ci.org/chanzuckerberg/miniwdl) by (i) splitting the suite across multiple jobs and (ii) caching slow-evolving apt/PyPI dependencies in reusable [docker image layers](https://quay.io/repository/mlin/miniwdl_ci?tab=tags).