Prefect

Released August 8, 2019

Features

- Introduce new `flows.checkpointing` configuration setting for checkpointing Tasks in local execution - [1283](https://github.com/PrefectHQ/prefect/pull/1283)
- Introduce new, flexible `Schedule` objects - [1320](https://github.com/PrefectHQ/prefect/pull/1320)

Enhancements

- Allow passing of custom headers in `Client` calls - [1255](https://github.com/PrefectHQ/prefect/pull/1255)
- Autogenerate informative names and tags for Docker images built for Flow storage - [1237](https://github.com/PrefectHQ/prefect/issues/1237)
- Allow mixed-case configuration keys (environment variables are interpolated as lowercase) - [1288](https://github.com/PrefectHQ/prefect/issues/1288)
- Ensure state handler errors are logged informatively - [1326](https://github.com/PrefectHQ/prefect/issues/1326)

Task Library

- Add `BigQueryLoadGoogleCloudStorage` task for loading data into BigQuery from Google Cloud Storage [1317](https://github.com/PrefectHQ/prefect/pull/1317)

Fixes

- Fix issue with logs not always arriving in long-standing Dask clusters - [1244](https://github.com/PrefectHQ/prefect/pull/1244)
- Fix issue with `BuildImage` docker task not actually running to completion - [1243](https://github.com/PrefectHQ/prefect/issues/1243)
- Fix `run --logs` CLI command not exiting on flow run finished state - [1319](https://github.com/PrefectHQ/prefect/pull/1319)

Deprecations

- `OneTimeSchedule` and `UnionSchedule` are deprecated, but remain callable as convenience functions - [1320](https://github.com/PrefectHQ/prefect/pull/1320)
- Old-style schedules can be deserialized as new-style schedules, but support will eventually be dropped - [1320](https://github.com/PrefectHQ/prefect/pull/1320)

Breaking Changes

- `prefect.Client.graphql()` and `prefect.Client.post()` now use an explicit keyword, not `**kwargs`, for variables or parameters - [1259](https://github.com/PrefectHQ/prefect/pull/1259)
- `auth add` CLI command replaced with `auth login` - [1319](https://github.com/PrefectHQ/prefect/pull/1319)

Released July 16, 2019

Features

- Add the Prefect CLI for working with core objects both locally and in cloud - [1059](https://github.com/PrefectHQ/prefect/pull/1059)
- Add RemoteEnvironment for simple executor based executions - [1215](https://github.com/PrefectHQ/prefect/pull/1215)
- Add the ability to share caches across Tasks and Flows - [1222](https://github.com/PrefectHQ/prefect/pull/1222)
- Add the ability to submit tasks to specific dask workers for task / worker affinity - [1229](https://github.com/PrefectHQ/prefect/pull/1229)

Enhancements

- Refactor mapped caching to be independent of order - [1082](https://github.com/PrefectHQ/prefect/issues/1082)
- Refactor caching to allow for caching across multiple runs - [1082](https://github.com/PrefectHQ/prefect/issues/1082)
- Allow for custom secret names in Result Handlers - [1098](https://github.com/PrefectHQ/prefect/issues/1098)
- Have `execute cloud-flow` CLI immediately set the flow run state to `Failed` if environment fails - [1122](https://github.com/PrefectHQ/prefect/pull/1122)
- Validate configuration objects on initial load - [1136](https://github.com/PrefectHQ/prefect/pull/1136)
- Add `auto_generated` property to Tasks for convenient filtering - [1135](https://github.com/PrefectHQ/prefect/pull/1135)
- Disable dask work-stealing in kubernetes via scheduler config - [1166](https://github.com/PrefectHQ/prefect/pull/1166)
- Implement backoff retry settings on Client calls - [1187](https://github.com/PrefectHQ/prefect/pull/1187)
- Explicitly set Dask keys for a better Dask visualization experience - [1218](https://github.com/PrefectHQ/prefect/issues/1218)
- Implement a local cache which persists for the duration of a Python session - [1221](https://github.com/PrefectHQ/prefect/issues/1221)
- Implement in-process retries for Cloud Tasks which request retry in less than one minute - [1228](https://github.com/PrefectHQ/prefect/pull/1228)
- Support `Client.login()` with API tokens - [1240](https://github.com/PrefectHQ/prefect/pull/1240)
- Add live log streaming for `prefect run cloud` command - [1241](https://github.com/PrefectHQ/prefect/pull/1241)

Task Library

- Add task to trigger AWS Step function workflow [1012](https://github.com/PrefectHQ/prefect/issues/1012)
- Add task to copy files within Google Cloud Storage - [1206](https://github.com/PrefectHQ/prefect/pull/1206)
- Add task for downloading files from Dropbox - [1205](https://github.com/PrefectHQ/prefect/pull/1205)

Fixes

- Fix issue with mapped caching in Prefect Cloud - [1096](https://github.com/PrefectHQ/prefect/pull/1096)
- Fix issue with Result Handlers deserializing incorrectly in Cloud - [1112](https://github.com/PrefectHQ/prefect/issues/1112)
- Fix issue caused by breaking change in `marshmallow==3.0.0rc7` - [1151](https://github.com/PrefectHQ/prefect/pull/1151)
- Fix issue with passing results to Prefect signals - [1163](https://github.com/PrefectHQ/prefect/issues/1163)
- Fix issue with `flow.update` not preserving mapped edges - [1164](https://github.com/PrefectHQ/prefect/issues/1164)
- Fix issue with Parameters and Context not being raw dictionaries - [1186](https://github.com/PrefectHQ/prefect/issues/1186)
- Fix issue with asynchronous, long-running mapped retries in Prefect Cloud - [1208](https://github.com/PrefectHQ/prefect/pull/1208)
- Fix issue with automatically applied collections to task call arguments when using the imperative API - [1211](https://github.com/PrefectHQ/prefect/issues/1211)

Breaking Changes

- The CLI command `prefect execute-flow` and `prefect execute-cloud-flow` no longer exist - [1059](https://github.com/PrefectHQ/prefect/pull/1059)
- The `slack_notifier` state handler now uses a `webhook_secret` kwarg to pull the URL from a Secret - [1075](https://github.com/PrefectHQ/prefect/issues/1075)
- Use GraphQL for Cloud logging - [1193](https://github.com/PrefectHQ/prefect/pull/1193)
- Remove the `CloudResultHandler` default result handler - [1198](https://github.com/PrefectHQ/prefect/pull/1198)
- Rename `LocalStorage` to `Local` - [1236](https://github.com/PrefectHQ/prefect/pull/1236)

Contributors

- [Kwangyoun Jung](https://github.com/initialkommit)
- [Anes Benmerzoug](https://github.com/AnesBenmerzoug)

This release fixes a versioning issue caused by a new release of `marshmallow-oneofschema` that is incompatible with the `marshmallow` version pin in 0.5.4.

Released May 28, 2019

Features

- Add new `UnionSchedule` for combining multiple schedules, allowing for complex schedule specifications - [428](https://github.com/PrefectHQ/prefect/issues/428)
- Allow for Cloud users to securely pull Docker images from private registries - [1028](https://github.com/PrefectHQ/prefect/pull/1028)

Enhancements

- Add `prefect_version` kwarg to `Docker` storage for controlling the version of prefect installed into your containers - [1010](https://github.com/PrefectHQ/prefect/pull/1010), [#533](https://github.com/PrefectHQ/prefect/issues/533)
- Warn users if their Docker storage base image uses a different python version than their local machine - [999](https://github.com/PrefectHQ/prefect/issues/999)
- Add flow run id to k8s labels on Cloud Environment jobs / pods for easier filtering in deployment - [1016](https://github.com/PrefectHQ/prefect/pull/1016)
- Allow for `SlackTask` to pull the Slack webhook URL from a custom named Secret - [1023](https://github.com/PrefectHQ/prefect/pull/1023)
- Raise informative errors when Docker storage push / pull fails - [1029](https://github.com/PrefectHQ/prefect/issues/1029)
- Standardized `__repr__`s for various classes, to remove inconsistencies - [617](https://github.com/PrefectHQ/prefect/issues/617)
- Allow for use of local images in Docekr storage - [1052](https://github.com/PrefectHQ/prefect/pull/1052)
- Allow for doc tests and doc generation to run without installing `all_extras` - [1057](https://github.com/PrefectHQ/prefect/issues/1057)

Task Library

- Add task for creating new branches in a GitHub repository - [1011](https://github.com/PrefectHQ/prefect/pull/1011)
- Add tasks to create, delete, invoke, and list AWS Lambda functions [1009](https://github.com/PrefectHQ/prefect/issues/1009)
- Add tasks for integration with spaCy pipelines [1018](https://github.com/PrefectHQ/prefect/issues/1018)
- Add tasks for querying Postgres database [1022](https://github.com/PrefectHQ/prefect/issues/1022)
- Add task for waiting on a Docker container to run and optionally raising for nonzero exit code - [1061](https://github.com/PrefectHQ/prefect/pull/1061)
- Add tasks for communicating with Redis [1021](https://github.com/PrefectHQ/prefect/issues/1021)

Fixes

- Ensure that state change handlers are called even when unexpected initialization errors occur - [1015](https://github.com/PrefectHQ/prefect/pull/1015)
- Fix an issue where a mypy assert relied on an unavailable import - [1034](https://github.com/PrefectHQ/prefect/pull/1034)
- Fix an issue where user configurations were loaded after config interpolation had already taken place - [1037](https://github.com/PrefectHQ/prefect/pull/1037)
- Fix an issue with saving a flow visualization to a file from a notebook - [1056](https://github.com/PrefectHQ/prefect/pull/1056)
- Fix an issue in which mapped tasks incorrectly tried to run when their upstream was skipped - [1068](https://github.com/PrefectHQ/prefect/issues/1068)
- Fix an issue in which mapped tasks were not using their caches locally - [1067](https://github.com/PrefectHQ/prefect/issues/1067)

Breaking Changes

- Changed the signature of `configuration.load_configuration()` - [1037](https://github.com/PrefectHQ/prefect/pull/1037)
- Local Secrets now raise `ValueError`s when not found in context - [1047](https://github.com/PrefectHQ/prefect/pull/1047)

Contributors

- [Zach Angell](https://github.com/zangell44)
- [Nanda H Krishna](https://nandahkrishna.me)
- [Brett Naul](https://github.com/bnaul)
- [Jeremiah Lewis](https://github.com/jlewis91)
- [Dave Hirschfeld](https://github.com/dhirschfeld)

Released May 7, 2019

Features

- Add new `Storage` and `Environment` specifications - [936](https://github.com/PrefectHQ/prefect/pull/936), [#956](https://github.com/PrefectHQ/prefect/pull/956)

Enhancements

- Flow now has optional `storage` keyword - [936](https://github.com/PrefectHQ/prefect/pull/936)
- Flow `environment` argument now defaults to a `CloudEnvironment` - [936](https://github.com/PrefectHQ/prefect/pull/936)
- `Queued` states accept `start_time` arguments - [955](https://github.com/PrefectHQ/prefect/pull/955)
- Add new `Bytes` and `Memory` storage classes for local testing - [956](https://github.com/PrefectHQ/prefect/pull/956), [#961](https://github.com/PrefectHQ/prefect/pull/961)
- Add new `LocalEnvironment` execution environment for local testing - [957](https://github.com/PrefectHQ/prefect/pull/957)
- Add new `Aborted` state for Flow runs which are cancelled by users - [959](https://github.com/PrefectHQ/prefect/issues/959)
- Added an `execute-cloud-flow` CLI command for working with cloud deployed flows - [971](https://github.com/PrefectHQ/prefect/pull/971)
- Add new `flows.run_on_schedule` configuration option for affecting the behavior of `flow.run` - [972](https://github.com/PrefectHQ/prefect/issues/972)
- Allow for Tasks with `manual_only` triggers to be root tasks - [667](https://github.com/PrefectHQ/prefect/issues/667)
- Allow compression of serialized flows [993](https://github.com/PrefectHQ/prefect/pull/993)
- Allow for serialization of user written result handlers - [623](https://github.com/PrefectHQ/prefect/issues/623)
- Allow for state to be serialized in certain triggers and cache validators - [949](https://github.com/PrefectHQ/prefect/issues/949)
- Add new `filename` keyword to `flow.visualize` for automatically saving visualizations - [1001](https://github.com/PrefectHQ/prefect/issues/1001)
- Add new `LocalStorage` option for storing Flows locally - [1006](https://github.com/PrefectHQ/prefect/pull/1006)

Task Library

- None

Fixes

- Fix Docker storage not pulling correct flow path - [968](https://github.com/PrefectHQ/prefect/pull/968)
- Fix `run_flow` loading to decode properly by use cloudpickle - [978](https://github.com/PrefectHQ/prefect/pull/978)
- Fix Docker storage for handling flow names with spaces and weird characters - [969](https://github.com/PrefectHQ/prefect/pull/969)
- Fix non-deterministic issue with mapping in the DaskExecutor - [943](https://github.com/PrefectHQ/prefect/issues/943)

Breaking Changes

- Remove `flow.id` and `task.id` attributes - [940](https://github.com/PrefectHQ/prefect/pull/940)
- Removed old WIP environments - [936](https://github.com/PrefectHQ/prefect/pull/936)
(_Note_: Changes from [936](https://github.com/PrefectHQ/prefect/pull/936) regarding environments don't break any Prefect code because environments weren't used yet outside of Cloud.)
- Update `flow.deploy` and `client.deploy` to use `set_schedule_active` kwarg to match Cloud - [991](https://github.com/PrefectHQ/prefect/pull/991)
- Removed `Flow.generate_local_task_ids()` - [992](https://github.com/PrefectHQ/prefect/pull/992)

Contributors

- None

Released April 19, 2019

Features

- Implement two new triggers that allow for specifying bounds on the number of failures or successes - [933](https://github.com/PrefectHQ/prefect/issues/933)

Enhancements

- `DaskExecutor(local_processes=True)` supports timeouts - [886](https://github.com/PrefectHQ/prefect/issues/886)
- Calling `Secret.get()` from within a Flow context raises an informative error - [927](https://github.com/PrefectHQ/prefect/issues/927)
- Add new keywords to `Task.set_upstream` and `Task.set_downstream` for handling keyed and mapped dependencies - [823](https://github.com/PrefectHQ/prefect/issues/823)
- Downgrade default logging level to "INFO" from "DEBUG" - [935](https://github.com/PrefectHQ/prefect/pull/935)
- Add start times to queued states - [937](https://github.com/PrefectHQ/prefect/pull/937)
- Add `is_submitted` to states - [944](https://github.com/PrefectHQ/prefect/pull/944)
- Introduce new `ClientFailed` state - [938](https://github.com/PrefectHQ/prefect/issues/938)

Task Library

- Add task for sending Slack notifications via Prefect Slack App - [932](https://github.com/PrefectHQ/prefect/issues/932)

Fixes

- Fix issue with timeouts behaving incorrectly with unpickleable objects - [886](https://github.com/PrefectHQ/prefect/issues/886)
- Fix issue with Flow validation being performed even when eager validation was turned off - [919](https://github.com/PrefectHQ/prefect/issues/919)
- Fix issue with downstream tasks with `all_failed` triggers running if an upstream Client call fails in Cloud - [938](https://github.com/PrefectHQ/prefect/issues/938)

Breaking Changes

- Remove `prefect make user config` from cli commands - [904](https://github.com/PrefectHQ/prefect/issues/904)
- Change `set_schedule_active` keyword in Flow deployments to `set_schedule_inactive` to match Cloud - [941](https://github.com/PrefectHQ/prefect/pull/941)

Contributors

- None