Luigi

**Note: Broken** due to a runtime error in LuigiConfigParser. See https://github.com/spotify/luigi/issues/2592.

Added:
luigi
* Add some docs to interface.run 2582
* Configure logging via TOML config 2483

luigi.contrib
* Added port property to CopyToTable 2561
* Make it so we can do from luigi.contrib.hdfs import HdfsFlagTarget 2594
* contrib: Add ExternalDailySnapshot 2591

Fixed:
luigi
* (docstring) Update task.py 2589
* Docs: Fixed "Github" to fit to the rest of the doc 2596
* Fix inspect.getargspec() DeprecationWarning in PY3 2579

luigi.contrib
* Fix Travis Moto Test Failures 2586
* Reduce TravisCI Test Runtime 2541

Changed:
luigi
* Make Worker parameter task_process_context an OptionalParameter 2574

luigi.contrib
* S3Client improvements 2569

This is a minor version bump, due to:

* Dropping Python 3.4 and 3.5 from CI, which means no automated tests to ensure compatibility for those versions
* [Security Patch] CORS being disabled by default. A new section of configuration [cors] is introduced to enable custom settings. For details, refer to user group topic: https://groups.google.com/forum/#!topic/luigi-user/ZgfRTpBsVUY

Added
luigi:
* Add Python 3.7 compatibility (2466)
This also drops 3.4 and 3.5 from CI.
* Interpolate environment variables in .cfg config files (2527)

luigi.contrib:
* Add CopyToTable task for MySQL (2553)
* Add HdfsFlagTarget (2559)

luigi.contrib:

Fixed
luigi:
* Fix ReadTheDocs build (2546)
* Make capture_output non-positional in ExternalProgramTask (2547)
luigi.contrib:
* Fix S3Client's _path_to_bucket_and_key to support keys with question marks (2534)
* Fix S3Client.remove - add max batch size (2529)
* Small fix to logging in contrib/ecs.py (2556)
* FIX HdfsAtomicWriteDirPipe.close() when using snakebite and the file do not exist. (2549)

Changed:
luigi:
* [ImgBot] optimizes images (2555)

luigi.contrib:

* Remove s3 bucket validation prior to file upload (2528)
* Refactor s3 copy into sub-methods (2508)

Added
luigi.contrib:
* Added optional choice for hdfs clients (2487)
* s3client check for deprecated host keyword and raise error with the details (2493)
* Add a "capture_output" parameter to ExternalProgramTask (2430)

Fixed
luigi:
* Fix exception when toml lib is not installed (2506)
* Replace direct attribute accessing by using built-n function getattr (2509)
* set upper bound of python-daemon (2536)
luigi.contrib:
* Fix S3Client.copy return value consistency (2488)
* Update MockTarget mode to accept r* or w* (2519)

* revert tornado upgrade (2504)
Upgrading tornado unfortunately breaks older version of Python

Added
luigi:
- Add Data Revenue to the `blogged` list (2472)
- Add default reviewers in CODEOWNERS (2465)
- Optional TOML configs support (2457)
- Add support for multiple requires and inherits arguments (2475)
- Add a visiblity level for luigi.Parameters (2278)
- Make logging of RPC retries configurable 2486
- Added a new event 'progress' (2498)
luigi.contrib:
- Additions to provide support for the Load Sharing Facility (LSF) job scheduler (2373)
- Added default port behaviour for Redshift (2474)
- Add metadata columns to the RDBMS contrib (2440)
- Use passed password when create a redis connection (2489)

Changed
luigi:
- Update supplementary github files to improve repo organization and maintenance (2463)
- Use task_id in Task.__eq__ comparison (2462)
- Replace luigi.Task by RunOnceTask in scheduler_visualisation_test (2476)
- (Breaking change) Bump tornado milestone version (2490)
This changes requires Python version 2.7.9+ and 3.4+
luigi.contrib:
- S3 client refactor (2482)
- Update moto to 1.x milestone version (2471)

Fixed
luigi:
- Fix Scheduler.add_task to overwrite accepts_messages attribute. (2469)
- Fix race condition (2477)
- Fix attribute forwarding for tasks with dynamic dependencies (2478)
luigi.contrib:
- Fix transfer config import (2458)

Removed
luigi:
- Remove long-deprecated scheduler config variable alternatives (2491)

Added
luigi:
- Add a configuration parameter to force multiprocessing (2401)
- Add a configuration parameter to enable/disable the pause button (2399)
- Send messages from scheduler to tasks (via "Send message" UI button) (2426)
- Allow to inject a context manager around `TaskProcess.run` (via `task_process_context` configuration parameter) (2449)
luigi.contrib:
- S3: use Boto3 for the `S3Client` (2423, 2149)
- GCS: add method to push files using multiprocessing (2376)
- HDFS: add `get_merge` to snakebite client (2410)
- Redshift: add schema to DB if it doesn't exist (2439)
- Redshift: add table constraints support (2435)

Fixed
luigi:
- Allow long parameters in task history DB SQL result store (2404)
- Fix `MissingParameterException` when generating execution summary (2415)
- Fix luigid crash due to configuration file parsing (2394)
- Allow explicit parsing of `BoolParameters` (via `luigi.BoolParameter.parsing` variable) (2427)
- Make `ChoiceParameter` check if option is valid within `.normalize` (2454)
- ...and a good deal of documentation fixes and similar.
luigi.contrib:
- BigQuery: fix `bulk_complete` failing when argument is a generator (2441)
- Kubernetes: prevent `KeyError` in `KubernetesJobTask` (2433)
- Kubernetes: don't set `activeDeadlineSeconds` by default (2452)