Luigi

This is mostly a bug-fix release.

---
- Changed behaviour (read bugfix) in local locking https://github.com/spotify/luigi/pull/1842
- Changed behaviour (read bugfix) in disabling workers https://github.com/spotify/luigi/pull/1839
- Many bugfixes to the scheduler, particularly related to the Batch running functionality.

[Here](https://github.com/spotify/luigi/compare/2.3.1...2.3.2) are the changes commit by commit.

This release mainly fixes bugs introduced in the two latest releases and also
some older bugs.

---
- Bugfix regarding sftp https://github.com/spotify/luigi/pull/1825
- Bugfix regarding error emails with smtp https://github.com/spotify/luigi/pull/1821
- Bugfix regarding spark tasks https://github.com/spotify/luigi/pull/1819
- Bugfixes regarding visualiser https://github.com/spotify/luigi/pull/1817 https://github.com/spotify/luigi/pull/1818

[Here](https://github.com/spotify/luigi/compare/2.3.0...2.3.1) are the changes commit by commit.

There's been over a month since the last release. This new release includes a
bunch of new features. What I like the most is that they all come with full and
proper documentation!

luigi:
- Fixed lockfile bugfix introduced in 2.2.0 https://github.com/spotify/luigi/pull/1809 Thanks nugend
- Retcodes improved https://github.com/spotify/luigi/pull/1771 Thanks fabriziodemaria
- Added [NumericalParameter](https://github.com/spotify/luigi/pull/1799)
and [ChoiceParameter](https://github.com/spotify/luigi/pull/1800) Thanks brcopeland
- Per-task configurable retry policies! https://github.com/spotify/luigi/pull/1791 Thanks javrasya
- Luigi can now batch tasks! https://github.com/spotify/luigi/pull/1784 Thanks daveFNbuck

luigi.scheduler:
- We now have a new definition of the UPSTREAM statuses. New intuition is that
`UPSTREAM_FAILED` means that that task cannot run because all downstream
tasks have failed or worse (like being disabled). As an effect there will be
much much fewer tasks considered to have an upstream status. https://github.com/spotify/luigi/pull/1789

There have been a lot of other features, cleanups and bugfixes! See all commits
[here](https://github.com/spotify/luigi/compare/2.2.0...2.3.0).

There's been 3 months since the latest release. Making Google's results on [readthedocs](http://luigi.readthedocs.io/) outdated and gives a stale feeling to luigi. Enjoy updated and hopefully bugfree software. :)

At least read these
- Luigi finally has user-land configurable task status messages https://github.com/spotify/luigi/pull/1625
- Parameters: From now on, you must not pass `None` as the default for a
parameter. Usually, passing the empty string `''` is a sufficient
replacement. If you don't do this luigi will print a deprecation warning.
https://github.com/spotify/luigi/pull/1624.
- Logging for server: Things are greatly improved
now as of https://github.com/spotify/luigi/pull/1633
and https://github.com/spotify/luigi/pull/1636.
Here's my jotted down usage info about it here
https://github.com/spotify/luigi/issues/1752#issuecomment-231266977.

Main changes

luigi:
- More fine grained eventhandlers https://github.com/spotify/luigi/pull/1698
- Range: Finally a proper way to pass along parameters: https://github.com/spotify/luigi/pull/1675
- From this release, we'll also bump the debian verion number. https://github.com/spotify/luigi/pull/1718
- Print your dependency tree as ascii art! https://github.com/spotify/luigi/pull/1680
- We now have a template for PR's! https://github.com/spotify/luigi/pull/1655

luigi.contrib:
- AWS: You can now set the session token: https://github.com/spotify/luigi/pull/1702/files
- Salesforce: Add support for multiple results https://github.com/spotify/luigi/pull/1686
- FTP: Configurable port https://github.com/spotify/luigi/pull/1689
- MSSQL support: https://github.com/spotify/luigi/pull/1650
- Streaming mapreduce: Allow additional archives https://github.com/spotify/luigi/pull/1649
- Streaming mapreduce: Recognize the Google File System formats https://github.com/spotify/luigi/pull/1664
- Streaming mapreduce: `mrrunner.py` is not hardcoded as the binary being run https://github.com/spotify/luigi/pull/1565

Various goodies
- Cleanup: https://github.com/spotify/luigi/pull/1734
- Cleanup: https://github.com/spotify/luigi/pull/1735
- Cleanup: https://github.com/spotify/luigi/pull/1723
- Cleanup: https://github.com/spotify/luigi/pull/1677
- Cleanup: https://github.com/spotify/luigi/pull/1640
- Cleanup: https://github.com/spotify/luigi/pull/1722
- Speedup: https://github.com/spotify/luigi/pull/1737
- Speedup: https://github.com/spotify/luigi/pull/1681
- Docs: https://github.com/spotify/luigi/pull/1729
- Docs: https://github.com/spotify/luigi/pull/1662
- Docs: https://github.com/spotify/luigi/pull/1716
- Bugfix: https://github.com/spotify/luigi/pull/1708
- Bugfix: https://github.com/spotify/luigi/pull/1643
- Improved error message: https://github.com/spotify/luigi/pull/1692

Contribution spirit

A few great examples that show how improvements are well-receieved no matter how small they are. Yet these "small" changes helps hundreads of people reading the docs of luigi. https://github.com/spotify/luigi/pull/1672 https://github.com/spotify/luigi/pull/1642

Other changes

There were [even more changes](https://github.com/spotify/luigi/compare/2.1.1...2.2.0) which we didn't include in these release notes. Like every contribution, whether merged or not we are happy for getting them. So please keep contributing! :)

Last release was only 2 business days ago (as opposed to the 5 months since the one before it). But this release got rushed as of a **security fix**!

In addition to doc fixes:

Additions
- [List and Tupleparameters](https://github.com/spotify/luigi/pull/1614) are now added
- More [Bigquery Options](https://github.com/spotify/luigi/pull/1619)
- The hive-cmd can now [contain spaces](https://github.com/spotify/luigi/pull/1616).

Security bugfix
- The server now have an [explicit whitelist](https://github.com/spotify/luigi/pull/1631) of external commands.
- Previous potential harm: Malicious hackers can run arbitrary code if they have file system (even external mounts!)+network access on the machine running luigid (executed by the user that you run luigid with).

We wait for a while with saying how to use this exploit, giving time to people to apply the bugfix.

Finally, a new PyPI release in a rather long while. Thanks Tarrasch and erikbern, and all the contributors!

Added

luigi:
- Notifications: more emails and proper coloring (1471), improved SMTP handling
- `EnumParameter` (1479), `DictParameter` (1574)
- Support for Python 3.5 (1494)
- Process locking on Alpine Linux (1530) and Windows (1557)
- Visualizer: resources tab (1566), GUI functionality to disable a worker (1564)

luigi.contrib:
- `ExternalBigqueryTask` (1434), `BigqueryCreateViewTask` (1465)
- Luigi tasks for Dataproc, Google's managed Hadoop MapReduce, Spark, Pig, and Hive service (1601)
- `ExternalProgramTask`, `ExternalPythonProgramTask` - commonalities for running any external application or script (1520)
- Support for SFTP (1585)
- Sped-up Hive client using Metastore (1533)
- `OpenerTarget`, a single Luigi target to open multiple file system types (1555)
- `Query` base task (giving rise also to `luigi.contrib.redshift.RedshiftQuery` and `luigi.postgres.PostgresQuery`) (1493)
- `RedshiftUnloadTask` (1527)
- `UploadToSalesforceTask` (1404)
- Support for S3 assumed role (1596)

Changed

luigi:
- Semi-opaque, hashed `task_id` (as opposed to `TaskName(param1=value1, param2=foo bar)`) (1444)
- More explicit way to handle timelike parameters (`date` vs `datetime`) (1473)
- Optimizations in scheduler algorithm

Removed

luigi:
- Old deprecated (2014) stuff around scheduler and its state (1592)

luigi.contrib:
- Deprecated classes `SparkJob`, `Spark1xBackwardCompat`, `Spark1xJob`, `PySpark1xJob` (1442)

Fixed

luigi.contrib:
- Ensure that FTP `RemoteTarget` successfully creates temporary files (meaning, in a directory relative to output) (1515)
- Remove superfluous `init_mapper()`/`init_reducer()` calls in `LocalJobRunner` (1475)
- Humanly format `HadoopJobError` (1528)
- Broken Redshift table creation (1453)
- Improved Salesforce reliability (1597, 1600)
- Missing call to `post_copy()` (1502)

...and a slew of other additions, fixes, improvements and documentation.