Prefect

Released on October 14, 2020.

Features

- Allow for schedules that emit custom Flow Run labels - [3483](https://github.com/PrefectHQ/prefect/pull/3483)

Enhancements

- Use explicit exception chaining - [3306](https://github.com/PrefectHQ/prefect/issues/3306)
- S3List filtering using the LastModified value - [3460](https://github.com/PrefectHQ/prefect/pull/3460)
- Add Gitlab storage - [3461](https://github.com/PrefectHQ/prefect/pull/3461)
- Extend module storage capabilities - [3463](https://github.com/PrefectHQ/prefect/pull/3463)
- Support adding additional flow labels in `prefect register flow` - [3465](https://github.com/PrefectHQ/prefect/pull/3465)
- Strict Type for default value of a Parameter - [3466](https://github.com/PrefectHQ/prefect/pull/3466)
- Enable automatic script upload for file-based storage when using S3 and GCS - [3482](https://github.com/PrefectHQ/prefect/pull/3482)
- Allow for passing labels to `client.create_flow_run` - [3483](https://github.com/PrefectHQ/prefect/pull/3483)
- Display flow group ID in registration output URL instead of flow ID to avoid redirect in UI - [3500](https://github.com/PrefectHQ/prefect/pull/3500)
- Add informative error log when local storage fails to load flow - [3475](https://github.com/PrefectHQ/prefect/pull/3475)

Task Library

- Add cancel flow run task - [3484](https://github.com/PrefectHQ/prefect/issues/3484)
- Add new `BatchSubmit` task for submitting jobs to AWS batch - [3366](https://github.com/PrefectHQ/prefect/pull/3366)
- Add new `AWSClientWait` task for waiting on long-running AWS jobs - [3366](https://github.com/PrefectHQ/prefect/pull/3366)
- Add GetAttr task - [3481](https://github.com/PrefectHQ/prefect/pull/3481)

Fixes

- Fix default profile directory creation behavior - [3037](https://github.com/PrefectHQ/prefect/issues/3037)
- Fix `DaskKubernetesEnvironment` overwriting log attributes for custom specs - [3231](https://github.com/PrefectHQ/prefect/issues/3231)
- Fix default behavior for `dbt_kwargs` in the dbt task to provide an empty string - [3280](https://github.com/PrefectHQ/prefect/issues/3280)
- Fix containerDefinitions environment validation - [3452](https://github.com/PrefectHQ/prefect/pull/3452)
- Raise a better error when calling `flow.register()` from within a `Flow` context - [3467](https://github.com/PrefectHQ/prefect/pull/3467)
- Fix task cancellation on Python 3.8 to properly interrupt long blocking calls - [3474](https://github.com/PrefectHQ/prefect/pull/3474)

Contributors

- [Aaron Richter](https://github.com/rikturr)
- [Alessandro Lollo](https://github.com/https://github.com/AlessandroLollo)
- [Bruno Casarotti](https://github.com/brunocasarotti)
- [Mariia Kerimova](https://github.com/mashun4ek)
- [Max Del Giudice](https://github.com/madelgi)
- [Michal Baumgartner](https://github.com/m1so)
- [Panagiotis Simakis](https://github.com/sp1thas)
- [Raphael Riel](https://github.com/raphael-riel)
- [Shalika Singhal](https://github.com/shalika10)
- [Zach McQuiston](https://github.com/zmac12)
- [heyitskevin](https://github.com/heyitskevin)

Released on October 6, 2020.

Enhancements

- Add option to template task run name at runtime when using backend API - [2100](https://github.com/PrefectHQ/prefect/issues/2100)
- Add `set_task_run_name` Client function - [2100](https://github.com/PrefectHQ/prefect/issues/2100)
- Use `from` to explicitly chain exceptions - [3306](https://github.com/PrefectHQ/prefect/pull/3306)
- Update error message when registering flow to non-existant project - [3418](https://github.com/PrefectHQ/prefect/pull/3418)
- Add `flow.run_config`, an *experimental* design for configuring deployed flows - [3333](https://github.com/PrefectHQ/prefect/pull/3333)
- Allow python path in Local storage - [3351](https://github.com/PrefectHQ/prefect/pull/3351)
- Enable agent registration for server users - [3385](https://github.com/PrefectHQ/prefect/pull/3385)

Task Library

- Add keypair auth for snowflake - [3404](https://github.com/PrefectHQ/prefect/pull/3404)
- Add new `RenameFlowRunTask` for renaming a currently running flow - [3285](https://github.com/PrefectHQ/prefect/issues/3285).

Fixes

- Fix mypy typing for `target` kwarg on base Task class - [2100](https://github.com/PrefectHQ/prefect/issues/2100)
- Fix Fargate Agent not parsing cpu and memory provided as integers - [3423](https://github.com/PrefectHQ/prefect/pull/3423)
- Fix MySQL Tasks breaking on opening a context - [3426](https://github.com/PrefectHQ/prefect/pull/3426)

Contributors

- [Ian Fridge](https://github.com/fridgei)
- [Jack D. Sundberg](https://github.com/jacksund)
- [Juan Calderon-Perez](https://github.com/gabrielcalderon)
- [Max Del Giudice](https://github.com/madelgi)
- [Paras Luthra](https://github.com/luthrap)
- [Tenzin Choedak](https://github.com/tchoedak)

Released on September 29, 2020.

Features

- Allow for scheduling the same flow at the same time with multiple parameter values - [2510](https://github.com/PrefectHQ/prefect/issues/2510)

Enhancements

- Adopt explicit exception chaining in more places - [3306](https://github.com/PrefectHQ/prefect/issues/3306)
- Add `DateTimeParameter` - [3327](https://github.com/PrefectHQ/prefect/pull/3327)

Task Library

- New task for the task library to create an item in Monday - [3387](https://github.com/PrefectHQ/prefect/pull/3387)
- Add option to specify `run_name` for `FlowRunTask` - [3393](https://github.com/PrefectHQ/prefect/pull/3393)

Contributors

- [Nejc Vesel](https://github.com/veseln)
- [Sumit Datta](https://github.com/brainless)

Released on September 22, 2020.

Enhancements

- Allow passing context values as JSON string from CLI - [3347](https://github.com/PrefectHQ/prefect/issues/3347)
- Allow copying of directories into Docker image - [3299](https://github.com/PrefectHQ/prefect/pull/3299)
- Adds schedule filters for month end or month start and specific day - [3330](https://github.com/PrefectHQ/prefect/pull/3330)
- Support configuring executor on flow, not on environment - [3338](https://github.com/PrefectHQ/prefect/pull/3338)
- Support configuring additional docker build commands on `Docker` storage - [3342](https://github.com/PrefectHQ/prefect/pull/3342)
- Support submission retries within the k8s agent - [3344](https://github.com/PrefectHQ/prefect/pull/3344)
- Expose `flow_run_name` to `flow.run()` for local runs - [3364](https://github.com/PrefectHQ/prefect/pull/3364)

Task Library

- Add contributing documentation for task library - [3360](https://github.com/PrefectHQ/prefect/pull/3360)
- Remove duplicate task library documentation in favor of API reference docs - [3360](https://github.com/PrefectHQ/prefect/pull/3360)

Fixes

- Fix issue with constants when copying Flows - [3319](https://github.com/PrefectHQ/prefect/issues/3319)
- Fix `DockerAgent` with `--show-flow-logs` to work on windows/osx (with python >= 3.8) - [3339](https://github.com/PrefectHQ/prefect/pull/3339)
- Fix mypy type checking for tasks created with `prefect.task` - [3346](https://github.com/PrefectHQ/prefect/pull/3346)
- Fix bug in `flow.visualize()` where no output would be generated when running with `PYTHONOPTIMIZE=1` - [3352](https://github.com/PrefectHQ/prefect/pull/3352)
- Fix typo in `DaskCloudProviderEnvironment` logs - [3354](https://github.com/PrefectHQ/prefect/pull/3354)

Deprecations

- Deprecate the use of the `/contrib` directory - [3360](https://github.com/PrefectHQ/prefect/pull/3360)
- Deprecate importing `Databricks` and `MySQL` tasks from `prefect.contrib.tasks`, should use `prefect.tasks` instead - [3360](https://github.com/PrefectHQ/prefect/pull/3360)

Contributors

- [David Severin Ryberg](https://github.com/sevberg)
- [James Lamb](https://github.com/jameslamb)
- [Nejc Vesel](https://github.com/veseln)

Released on September 16, 2020.

Enhancements

- Use explicit exception chaining [3306](https://github.com/PrefectHQ/prefect/issues/3306)
- Quiet Hasura logs with `prefect server start` - [3296](https://github.com/PrefectHQ/prefect/pull/3296)

Fixes

- Fix issue with result configuration not being respected by autogenerated tasks - [2989](https://github.com/PrefectHQ/prefect/issues/2989)
- Fix issue with result templating that failed on task arguments named 'value' - [3034](https://github.com/PrefectHQ/prefect/issues/3034)
- Fix issue restarting Mapped pipelines with no result- [3246](https://github.com/PrefectHQ/prefect/issues/3246)
- Fix handling of Prefect Signals when Task state handlers are called - [3258](https://github.com/PrefectHQ/prefect/issues/3258)
- Allow using `apply_map` under a `case` or `resource_manager` block - [3293](https://github.com/PrefectHQ/prefect/pull/3293)
- Fix bug with interaction between `case` blocks and `Constant` tasks which resulted in some tasks never skipping - [3293](https://github.com/PrefectHQ/prefect/pull/3293)
- Fix bug in `DaskExecutor` where not all client timeouts could be configured via setting `distributed.comm.timeouts.connect` - [3317](https://github.com/PrefectHQ/prefect/pull/3317)

Task Library

- Adds a compression argument to both S3Upload and S3Download, allowing for compression of data upon upload and decompression of data upon download - [3259](https://github.com/PrefectHQ/prefect/issues/3259)

Contributors

- [Eric Lundy](https://github.com/ericlundy87/)
- [James Lamb](https://github.com/jameslamb)
- [Max Del Giudice](https://github.com/madelgi)

Released on September 9, 2020.

Enhancements

- Adds logger to global context to remove friction on running task unit tests - [3256](https://github.com/PrefectHQ/prefect/issues/3256)
- Expand FunctionTask AttributeError Message - [3248](https://github.com/PrefectHQ/prefect/pull/3248)
- Add backend info to diagnostics - [3265](https://github.com/PrefectHQ/prefect/pull/3265)
- Ellipsis Support for GraphQL DSL - [3268](https://github.com/PrefectHQ/prefect/pull/3268)

Task Library

- Add `DatabricksRunNow` task for running Spark jobs on Databricks - [3247](https://github.com/PrefectHQ/prefect/pull/3247)
- Add GitHub `CreateIssueComment` task - [3269](https://github.com/PrefectHQ/prefect/pull/3269)
- Add `S3List` task for listing keys in an S3 bucket - [3282](https://github.com/PrefectHQ/prefect/pull/3282)
- Add `boto_kwargs` to AWS tasks - [3275](https://github.com/PrefectHQ/prefect/pull/3275)

Fixes

- Make identifier optional in `KubernetesAgent.replace_job_spec_yaml()` - [3251](https://github.com/PrefectHQ/prefect/pull/3251)
- Change `https://localhost` to `http://localhost` in the welcome message - [#3271](https://github.com/PrefectHQ/prefect/pull/3271)

Contributors

- [Ashmeet Lamba](https://github.com/ashmeet13)
- [Ashton Sidhu](https://github.com/Ashton-Sidhu)
- [Bas Nijholt](https://github.com/basnijholt)
- [Harutaka Kawamura](https://github.com/harupy)
- [James Lamb](https://github.com/jameslamb)
- [Max Del Giudice](https://github.com/madelgi)
- [Robin Beer](https://github.com/Zaubeerer)
- [Rowan Molony](https://github.com/rdmolony)
- [Shunwen](https://github.com/shunwen)