Toil

This is the last version of Toil that supports python2.7 and allows the command word "cwltoil" (now "toil-cwl-runner").

* Update ssh and docker setup files. 2958
* Update EC2 instance types. 2962
* Source toil environment as a configuration file. 2960
* Scan for the latest caching database instead of using a hard link. 2948
* Specify py3.6 or py3.x instead of just py3 in appliance. 2966

_Bug Fixes_
* Fix database being locked when trying to write in the CachingFileStore. 2955
* Fix py3 spot market scaling. 2971

Thank you to our contributors: mr-c, arostamianfar, adamnovak, jeffrey856!

Functionally the same as Toil Version 3.22.0.

* Make an appliance for Python 3. 2937
* Use status to properly get the HTTP status code from boto. 2934
* Use atomic file copy when writing to posix file system. 2912
* Improve Kubernetes job timing. 2922
* Increase SQLite DB timeout to 60s. 2913
* Consider pods that are very close to their memory limits to be stuck. 2895
* Constantly reload Kubernetes config. 2869
* Up the disk to the default as well as the memory. 2880
* Tolerate unnecessary bucket cleanup. 2894
* Add checksum verification for AWS job store. 2889
* Change python3.6 ppa to deadsnakes. 2888
* Change Kubernetes job sizing logic. 2877
* Added env var TOIL_KUBERNETES_OWNER. 2872
* Raise job limits to be more than their requests. 2868
* Remove all Azure support from toil. 2860

_Bug Fixes_
* Exclude .nfs files from deferred function search. 2930
* Fix cpu_count() to actually read quotas. 2933
* Compile http-parser so that it doesn't break pip dependencies. 2940
* Use an http-parser for pymesos that builds on 3.7+. 2931
* Fix botocredAdapter. 2904
* Make sure to commit when giving away files. 2914
* Actually call the implemented size polling method. 2916
* Set Docker vars back to ?=. 2908
* LSF: Don't check for Status <EXIT>, as that breaks the error code finding. 2863

Thank you to our contributors: mr-c, arostamianfar, adamnovak, jeffrey856, diekhans!

* Add kubernetes watch. 2848
* Add service account configuration for Kubernetes. 2851
* Head object 404 with encryption. 2853
* Tests for Kubernetes. 2838
* Turn usernames into non-spooky Kubernetes name components. 2836
* Make examples a symlink to make examples discoverable. 2834
* Kubernetes BatchSystem Support. 2786
* Specify GCE coreos image by family name instead of specific version. 2829
* Add toil-cwl-runner --provenance. 2785
* Revise caching more. 2770
* Bump psutil >= 3.0.1, <6. 2790
* Add "failed_" prefix to failed logs. 2792
* Add --writeLogsFromAllJobs option. 2781
* Expose --default-container to toil-cwl-runner. 2784
* Update cwltool from 1.0.20190228155703 -> 1.0.20190815141648. 2764
* Pass jobID to Torque Wrapper. 2762
* Limit Length of Log Names. 2755
* Add CWL options --strict-memory-limit, --relax-path-checks, and --leave-container. 2718
* Update EC2 nodes and script. 2709
* Redesign File Job Store layout. 2669

_Bug Fixes_
* Fix for passing default values between Workflows. 2831
* Ignore checking versioning consistency when cleaning s3 jobstore. 2826
* Ensure terminated ignored nodes are cleared from batch system. 2789
* Fix Slurm sbatch command. 2777
* Typo in gridengine.py. 2778
* CWL fixes. 2783
* Clarify `/data` mountpoint for `dockerCall`. 2747
* Fix CWL tmpdir to be unique per job. 2745
* Protect against potential AWS truncated downloads. 2731
* Fix runningOnEC2 to work with all instances. 2734
* Prevent Toil from zipping entire dirs. 2728
* Changes to lsf.py to make it compatible on lsf9 and lsf10. 2726
* Remove X86_64 type requirement from lsf support. 2723
* Clean up temporary directories cwltoil uses to hold output streams. 2690

Thank you to our contributors: mr-c, arostamianfar, adamnovak, jeffrey856, tthyer, kaushik-work, dleehr, ionox0, psafont, mb1069, tobiaszjarosiewicz, gconcepcion, byee4, smoe, TMiguelT, jvivian-atreca!

* Deprecate subprocessDockerCall(). 2653
* Move wallTime to a label so we can use exact timestamps. 2664
* Capture and log standard output/error from batch systems. 2662
* Add an option to set custom AWS IAM roles. 2656
* Add a ToilNodeType tag to distinguish leader vs. workers. 2658
* Slurm: include both exit code and terminating signal in job return status. 2646
* Update requests package. 2610
* HTCondorBatchSystem(): fixes and features. 2647
* Update cwltool from 1.0.20180820141117 -> 1.0.20190228155703 2609
* Add an option to set disableCaching to false. 2584
* Speed up S3 bucket deletion. 2597
* Tolerate missing jobs/workers in stats expando. 2533
* Add private IP support to node operations. 2569
* Encode logs for gzip. 2595

_Bug Fixes_
* Improve NoSuchFileException usage and robustness. 2593
* Make sure we always have current package lists. 2675
* Make sure to use --tmpdir-prefix to place workdir and jobstore. 2587
* Subprocess dockercall bytestring compatibility for py2/3. 2677
* Address python3 deprecation warnings. 2668
* Fix python3.7 break on __new__ static method. 2659
* Get value of taskId in Mesos' killTask method. 2661
* Stop NamedTemporaryFile tampering. 2638
* StatsAndLogging(): ensure json.load() receives only string input. 2641
* Reduce max size of files allowed to be inlined in SDB. 2440
* Increase hardcoded encapsulate follow-on resources. 2591
* Ensure TOIL_DOCKER_REGISTRY variable is in version_template. 2583
* Use current python executable. 2599

Thank you to our contributors: mr-c, arostamianfar, adamnovak, kwwette, liinus, saimeCS, cdunn2001, Fifan31, jeffrey856!

* Support for CWL s3 imports. 2522
* Support for running custom init commands in docker (allowing auth in private dockers). 2561
* Add --no-read-only option to the CWL runner. 2531
* Add support for assuming roles. 2527
* Added an adapter to allow Boto 2 to use credentials obtained via Boto 3. 2527
* Add a flag for running CWL internal jobs on workers. 2524
* Added CWL tests for http, https, and s3 file imports. 2234
* Begin testing python3 in gitlab. 2518
* Relegate pytest dependency to the dev reqs. 2508
* WDL changes to enable resource usage. 2495
* Sphinx and pathlib2 dependencies. 2490 2493
* Upgrade appliance to Ubuntu 16.04. 2480
* Update cwltool from 1.0.20180820141117 -> 1.0.20181118133959 2469
* Add CWL's --preserve-entire-environment option. 2481
* Bump minimum version of galaxy-lib (17.9.3 -> 17.9.9). 2476
* Added all tutorial scripts to unit testing. 2426
* More python3 support added. 2405
* Update the CoreOS AMI. 2430
* Simplify log creation in leader. 2423
* Support for hardlinks and greater testing for toil status. 2402
* Pymesos port. 2366 2338
* Use displayName for CWLJob to show stats in toil --stats. 2397

_Bug Fixes_
* Import errors. 2558
* Fix boto3 credentials for permanent credentials. 2549
* Fix bug in Toil stats. 2513
* Replace some str objects as real FileIDs. 2500
* Remove linear-time bottleneck from Mesos batch system. 2498
* Slurm and py3 fixes. 2494
* Safeguard unintentional boto imports. 2489
* Fix broken readthedocs build. 2485
* Specify region in prometheus. 2437
* Fixed bytestring-string tagging. 2472
* LSF log job id on error. 2450
* Fix massive slowdown when many jobs are running. 2439
* Make ServiceManager stop blocking while (most) services are established. 2441
* Default to bash on the leader node rather than sh. 2435
* resource.getusage returns KB already, so dividing by 1024 yields MB. 2425
* Recognize conda virtual envs. 2422

Thank you to our contributors: mr-c, arostamianfar, adamnovak, nikhil, lamby, gijzelaerr, joelarmstrong, markasbach, davidjsherman, psafont, jsmedmar, bencvdb, emlys, ionox0, ibebio, MarkFilus!