Toil

Changelog

CWL
* Update to the latest cwltool (3.0.20201121085451 -> 3.0.20201203173111). 3375
* Add better handling for potentially mis-ordered CWL args. 3395
* Confirm CommandInputParameter expression can receive a File object. 3350

WDL
* Make AnalyzeWDL abstract to allow implementation for different WDL versions 3391

Misc
* Clean up memoize.py. 3374
* Refactor bioio library, simplify logging, and small adjustments to utils. 3351
* Use regular division for memory calculation on LSF. 3387
* Remove unused --nodeOptions arg. 3397
* Get boto S3 args for minio from environment variables. (3370)
* Update EC2 lists. 3376
* Add a retry to AWS destroy. 3379
* Regular division for memory calculation in LSF. 3387
* Add option to blank mem allocation for SLURM. 3399
* Preserve file permissions on imported/exported files 3322

_Bug Fixes_
* Preserve file permissions on imported/exported files. 3322
* Make building a flattened list of live jobs non-recursive. 3394
* Move AWS job store uploads/downloads to boto3 (reopened) 3400
* Use a Docker Hub mirror when pulling Docker images. 3411
* Corrected gridengine maxMEM check to use memorystring 3410

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, douglowe, stevekm , thiagogenez!

Changelog

WDL
* Add WDL cross function. 3360

_Bug Fixes_
* Better error message for insufficient disk. 3362
* Fix maxMEM comparison in LSF. 3369

Thank you to our contributors: mr-c , adamnovak , w-gao , jonathanxu18 !

Caching is now turned on by default.

CWL v1.1 is also now fully supported and passing all conformance tests.

The following batch system names are now deprecated (3225) and replaced by:
- singleMachine -> single_machine
- gridEngine -> grid_engine
- LSF -> lsf
- Mesos -> mesos
- Slurm -> slurm
- Torque -> torque
- HTCondor -> htcondor
- Kubernetes -> kubernetes
- k8s -> kubernetes

Changelog

CWL
* Upgrade to a CWL v1.2.0 capable cwltool (3.0.20200709181526 -> 3.0.20200807132242). 3137
* Export to a bucket if cwl's CreateFile is specified. 3124
* CWL v1.1 Support: loadContents if True for StepValueFrom objects. 3266
* CWL: Fix workdir permissions. 3229
* Test CWL secondary file from s3. 3311
* CWL v1.1 Support: Inplace update has side effect on directory content. 3280
* More CWL v1.2 conformance tests. 3336
* CWL: Reformat w/ Black and better support for tmpdir_prefix with suffix. 3333

WDL
* Add ceil() and floor() WDL functions. 3168
* Create a stdout and stderr file for each WDL task. 3181
* Add WDL write functions and builtin tests. 3236
- write_lines
- write_tsv
- write_json
- write_map
* Add WDL read functions and builtin tests. 3244
- read_lines
- read_tsv
- read_json
- read_map
- read_int
- read_string
- read_float
- read_boolean
* Add transpose() WDL function. 3273
* Add length() WDL function. 3307
* Add sub() WDL function. 3252
* Add range() WDL function. 3277
* Add size() WDL function. 3255
* Add zip() WDL function. 3355
* Implement WDL Pair type. 3304
* Refactor WDL types 3335

Kubernetes
* Adding Labels to Pods/Jobs. 3233
* Make kubernetes & botocore packages fully optional. 3259 3261
* KubeWatch optimization. 3227
* Catch other kubernetes imports. 3261
* Automatically infer singularity for kubernetes. 3279

Misc

* Allow Caching By Default. 3111
* Update dependencies and loosen restrictions slightly. 3139
* Ensure that falsey (but non-null) values are considered. 3143
* Remove pathlib. 3169
* Move AWS job store uploads/downloads to boto3. 3153
* Update prom/node-exporter to use quay.io. 3186
* Add support for TOIL_CUSTOM_INIT_COMMAND. 3183
* Update docker module version to 4.3.1. 3222
* Add 'stream' and 'demux' options to apiDockerCall. 3224
* Use boto3 to create our leader node. 3145
* Prevent job kind filenames from getting too long. 3230
* Add a retry decorator. 3144
* Remove mesos offer messages. 3308
* Job Concept Unification. 3250
* Add reporting for accessed files of failed jobs. 3309
* Add coverage and refactor. 3314
* Expose --statePollingWait param through toil. 3321
* Record the maximum memory used by LSF jobs. 3327
* Remove custom math functions and replace with built ins. 3338
* Define a changelog process. 3316
* SLURM: getting job details from `scontrol show jobs` was non-functional and has now been fixed. This behavior in the case of `sacct` not being configured now works. 3346
* LSF: handle unicode configuration files 3354
* Replace abssympath w/ builtin. 3356

_Bug Fixes_
* Turn off watch code in Kubernetes. 3175
* Check against joining the current thread during CachingFileStore destructor. 3187
* Slurm.py error, undefined stdout - stdout should be stderr in code. 3194
* Fix restart flag error (issue 3094). 3142
* Batch System generates a key error instead of a usage message. 3225
* Remove Python 2 imports in WDL. 3228
* Move checkForDeadlocks() in the leader. 3234
* Fix boto3 migration bugs. 3243
* Fix None check when parsing WDL JSON file. 3272
* Fix broken stdout/stderr in torque job. 3306
* Add doubleMem for LSF jobs that die due to imposed memory limit. 3313
* Add 500 status to the list of retriable SDB BotoServerErrors. 3329
* Add a retry logic for BucketNotEmpty error. 3341
* Fix None check when parsing WDL JSON file. 3272
* Warn when trying to clean non-existent job store 3288
* Only rm a statefile that exists. 3348
* CWL: Remove unprocessed secondaryFiles from both Dict and List types.

Thank you to our contributors: mr-c, arostamianfar, adamnovak, diekhans, w-gao, jonathanxu18, jeffrey856, davidlougheed, altairwei, mberacochea, drkennetz, ionox0!

* Don't log in the inner scheduling loop. 3065
* Handle anticipated 404 errors without retrying. 3067
* Test and reimplement AMI finding via Flatcar JSON feed. 3061
* Bump Enlighten version to partly fix 3069. 3070
* Get wait duration is far too short of a rest for lsf. 3076
* Unify logging format. 3073
* Support CWL 1.1 Listing. 3058
* Enable CWL v1.2.0-dev3 (3.0.20200324120055 -> 3.0.20200530110633). 3092
* Enable CWL v1.2.0-dev4 (3.0.20200530110633 -> 3.0.20200709181526). 3105
* Add --nodeStorageOverrides option. 3096
* More robust CWL filepaths and secondary files. 3114
* Replace custom mkdir_p with the built in. 3123
* Move conditional execution test to before fill_in_defaults. 3117

_Bug Fixes_
* Stop CWL from trying to copy downloaded directories from somewhere local. 3053
* Exit the worker with 1 when jobs fail so the batch system can see. 3052
* Eliminate recursion in root finding to fix 3080. 3081
* Add a Cactus Kubernetes test to the integration tests on Gitlab. 3077
* Chaining doesn't make sense for CWL. 3091
* Fix failed job accounting. 3103
* Improve lsf batchsystem stability. 3101
* Deduplicate updated jobs. 3108
* Redundant virtualenv check. 3135
* Change Slurm command output processing. 3133
* Fix for log file being linked more than once. 3130
* Wrap Kubernetes watches to fix 3125. 3126
* Create dirs that do not exist when bind-mounting to docker. 3120
* Fix database state when file upload fails. 3122

Thank you to our contributors: mr-c, arostamianfar, adamnovak, diekhans, nikhil, drkennetz, jonathanxu18, tobiaszjarosiewicz, ionox0!

* Attempt to make batch system more robust and debuggable. 2959
* Kubernetes Shared Caching. 3012
* Test python3.8 and build a python3.8 appliance. 3028
* Use the original pymesos. 3036
* Detect more kinds of virtual environment and normalize prefixes. 3035
* Do More Hashing. Can't have enough. 3025
* Replace reissued jobs message with better progress indicators. 3044
* Detect insufficient resources and deadlocks more usefully. 3043

_Bug Fixes_
* Don't list specific CWL features, try them all (and fix some CWL running bugs). 3015
* Always honor TOIL_GRIDENGINE_PE and never assume a site-dependent default. 3041

Thank you to our contributors: mr-c, arostamianfar, adamnovak, diekhans!

This release moves to python3 only, dropping all python2.7 compatibility, and also deprecates the command "cwltoil" in favor of "toil-cwl-runner".

* Update cwltool version (==1.0.20190906054215 -> <=2.0.20200126090152). 2969
* Drop the long-deprecated cwltoil. 2843
* Dropped python2.7. 2973
* Port provisioner tests and only test py3.6. 2842
* CWL: Refactor of link merge + conditionals + pickValue. 2845
* Add import subprocess to executor.py. 2976
* Encode mtail stdin to utf-8 and flush. 2978
* Encode framework message as utf-8 as well. 2981
* Make the output files for grid engine batch systems not try and be in per-host directories. 2956
* Use file locks instead of PID polling to see if other processes sharing the cache are alive. 2982
* Size downloads during download. 2989
* Add timeouts to all SQLite connect statements. 2993
* Eliminate sleep time on --restart. 2990
* Add support for --awsEc2ExtraSecurityGroupId. 2997
* Add moveExports option. 2983
* Set worker threads as dameon to prevent hanging process on error. 2998
* Cut the Threads. 2999
* [Part 1] CWL v1.1 Support 2985
* [Part 2] CWL v1.1 Support 3000
* Update Sphinx and let its version float. 3011
* Add --enableUnlimitedPreemptableRetries option. 2896
* Add less and vim to docker. 3017
* Improve and Unify Log Dumping. 3008
* Make dependency on Python3.6+ machine-readable in setup.py. 3021
* Improve slurm update frequency. 3026
* Make Sphinx a make prepare component and not a dependency. 3023
* Use forked pymesos and http-parser dependencies. 3024

_Bug Fixes_
* Fix python3 string/bytes error when using --printLogs. 3005
* Fix connection between Kubernetes and Leader. 3004

Thank you to our contributors: mr-c, arostamianfar, adamnovak, jeffrey856, kaushik-work, glmxndr, johnbradley, cmarkello, diekhans!