This major release adds many new features, improvements and bug fixes. It is the new stable release and users are encouraged to upgrade to it. Support for the previous stable release, now considered legacy, will continue but be limited to security fixes.
The important new features are:
- Completely rewritten caching allows for sharing of files between worker processes running on the same worker node. Previously, only a chain of jobs running inside a single worker process would benefit from caching.
- Improved ability to run long-running jobs that provide _services_ to other, regular jobs in the workflow.
- Building on top of the improved service jobs feature, we added support for ephemeral Spark sub-clusters that are formed on-demand during a Toil workflow and can be used by jobs to perform ADAM or other Spark workloads.
- Support for hosting job stores in Google Cloud Storage was added. Install Toil with the `google` extra to enable this support. The Google job store should be considered experimental. The Toil team is working on gaining more experience on the consistency and atomicity guarantees provided by Google Cloud Storage to ready that job store implementation for production use.
- Support for hosting job stores in Azure Storage was demoted to experimental status due to atomicity and consistency issues. The Toil team will be working closely with Microsoft engineers to resolve these.
- Added ability to import and export files into a job store on both the leader prior/after to invoking a workflow and on the workers as part of a workflow. Importing files from `http://`, `https://` and `file://` URLs is supported out of the box. So is exporting to `file://` URLs. Installing Toil with any or all of the `azure`, `aws` or `google` extras extends the import/export facility to files hosted on Azure Blob Storage, S3 and Google Cloud Storage, respectively.
- Toil scripts can now have dependencies and those dependencies are deployed automatically to each worker node. The script and the dependencies must be installed in a virtualenv and must be pure-Python. Native dependencies must be installed by the user on each worker prior to running a workflow. The Toil team uses the CGCloud project to handle those native dependencies.
- CWL support has matured.