Prefect

Released on July 28, 2020.

Features

- Add `flatten` operator for unnesting and flat-maps - [2898](https://github.com/PrefectHQ/prefect/pull/2898)

Enhancements

- Add `retry_on_api_error` flag to client methods - [3012](https://github.com/PrefectHQ/prefect/pull/3012)
- Add `reg_allow_list` option for Docker Agent - [3026](https://github.com/PrefectHQ/prefect/pull/3026#issuecomment-663078217)
- Update FargateTaskEnvironment to throw if task definition is inconsistent with existing task definition - [3031](https://github.com/PrefectHQ/prefect/pull/3031)

Fixes

- Cleanup to ShellTask to close open stdout file which was observable in some cases - [3002](https://github.com/PrefectHQ/prefect/issues/3002)
- Fix check of flow existence in storage object `get_flow` to only occur when provided - [3027](https://github.com/PrefectHQ/prefect/issues/3027)
- Use full name and tag when Docker Storage determines if build was successful - [3029](https://github.com/PrefectHQ/prefect/pull/3029)
- Prevent duplicated agent labels - [3029](https://github.com/PrefectHQ/prefect/pull/3042)

Deprecations

- `prefect.utilities.tasks.unmapped` moved to `prefect.utilities.edges.unmapped` - [2898](https://github.com/PrefectHQ/prefect/pull/2898)

Breaking Changes

- Remove `dbt` extra from dependencies - [3018](https://github.com/PrefectHQ/prefect/pull/3018)

Contributors

- [James Lamb](https://github.com/jameslamb)
- [Spencer Ellinor](https://github.com/zpencerq)
- [Thomas Frederik Hoeck](https://github.com/thomasfrederikhoeck)
- [berosen](https://github.com/berosen)

Changelog

Released on July 21, 2020.

Features

- Add `resource_manager` api for cleaner setup/cleanup of temporary resources - [2913](https://github.com/PrefectHQ/prefect/pull/2913)

Enhancements

- Add `new_flow_context` to FlowRunTask for configurable context - [2941](https://github.com/PrefectHQ/prefect/issues/2941)
- All storage types now support file-based storage - [2944](https://github.com/PrefectHQ/prefect/pull/2944)
- Turn work stealing ON by default on Dask K8s environment - [2973](https://github.com/PrefectHQ/prefect/pull/2973)
- Send regular heartbeats while waiting to retry / dequeue - [2977](https://github.com/PrefectHQ/prefect/pull/2977)
- Cached states now validate based on `hashed_inputs` for more efficient storage - [2984](https://github.com/PrefectHQ/prefect/pull/2984)
- Simplify creation of optional parameters with default of `None` - [2995](https://github.com/PrefectHQ/prefect/pull/2995)

Task Library

- Implement AWSSecretsManager task - [2069](https://github.com/PrefectHQ/prefect/issues/2069)
- Update return value and config for DbtShellTask - [2980](https://github.com/PrefectHQ/prefect/pull/2980)

Fixes

- Don't send idempotency key when running against a local backend - [3001](https://github.com/PrefectHQ/prefect/issues/3001)
- Fix bug in `DaskExecutor` when running with external cluster where dask clients could potentially be leaked - [3009](https://github.com/PrefectHQ/prefect/pull/3009)

Deprecations

- All states have deprecated the usage of `cached_inputs` - [2984](https://github.com/PrefectHQ/prefect/pull/2984)

Breaking Changes

- Remove password from Postgres tasks' initialization methods for security - [1345](https://github.com/PrefectHQ/prefect/issues/1345)

Contributors

- [Robin Beer](https://github.com/Zaubeerer)

Released on July 14, 2020.

Enhancements

- Improve output formatting of `prefect describe` CLI - [2934](https://github.com/PrefectHQ/prefect/pull/2934)
- Add new `wait` kwarg to Flow Run Task for reflecting the flow run state in the task - [2935](https://github.com/PrefectHQ/prefect/pull/2935)
- Separate build-time and run-time job spec details in KubernetesJobEnvironment - [2950](https://github.com/PrefectHQ/prefect/pull/2950)

Task Library

- Implement RunNamespacedJob task for Kubernetes - [2916](https://github.com/PrefectHQ/prefect/pull/2916)
- Add `log_stderr` option to `ShellTask` and `DbtShellTask` for logging the full output from stderr - [2961](https://github.com/PrefectHQ/prefect/pull/2961)

Fixes

- Ensure `is_serializable` always uses same executable for subprocess. - [1262](https://github.com/PrefectHQ/prefect/issues/1262)
- Fix issue with Mapped tasks not always reloading child state results on reruns - [2656](https://github.com/PrefectHQ/prefect/issues/2656)
- Fix `FargateTaskEnvironment` attempting to retrieve authorization token when not present - [2940](https://github.com/PrefectHQ/prefect/pull/2940)
- Fix issue with Metastates compounding - [2965](https://github.com/PrefectHQ/prefect/pull/2965)

Contributors

- [Chris Bowdon](https://github.com/cbowdon)
- [James Lamb](https://github.com/jameslamb)
- [Paweł Cieśliński](https://github.com/pcieslinski)

Released on July 8, 2020.

Enhancements

- Update `flow.slugs` during `flow.replace` - [2919](https://github.com/PrefectHQ/prefect/issues/2919)
- `flow.update` accepts the optional kwarg `merge_parameters` that allows flows to be updated with common `Parameters` - [2501](https://github.com/PrefectHQ/prefect/issues/2501)
- Added poke handler to notify agent process of available flow runs - [2914](https://github.com/PrefectHQ/prefect/pull/2914)
- Add `Cancelling` state for indicating a flow-run that is being cancelled, but may still have tasks running - [2923](https://github.com/PrefectHQ/prefect/pull/2923)

Task Library

- Add `ReadAirtableRow` task - [2843](https://github.com/PrefectHQ/prefect/pull/2843)
- Add `container_name` kwarg to `CreateContainer` Docker task - [2904](https://github.com/PrefectHQ/prefect/pull/2904)
- Adds an `extra_docker_kwargs` argument to `CreateContainer` Docker task - [2915](https://github.com/PrefectHQ/prefect/pull/2915)

Fixes

- Fix issue with short-interval IntervalClocks that had a start_date far in the past - [2906](https://github.com/PrefectHQ/prefect/pull/2906)
- When terminating early, executors ensure all pending work is cancelled/completed before returning, ensuring no lingering background processing - [2920](https://github.com/PrefectHQ/prefect/pull/2920)

Contributors

- [Bradley McElroy](https://github.com/limx0)
- [Itay Livni](https://github.com/gryBox)
- [Matthew Alhonte](https://github.com/mattalhonte)
- [Panagiotis Simakis](https://github.com/sp1thas)
- [Sandeep Aggarwal](https://github.com/asandeep)

Released on June 30, 2020.

Features

- Add `apply_map`, a function to simplify creating complex mapped pipelines - [2846](https://github.com/PrefectHQ/prefect/pull/2846)

Enhancements

- Make storage location inside Docker storage configurable - [2865](https://github.com/PrefectHQ/prefect/pull/2865)
- Send heartbeats on each iteration of the Cloud task runner's retry loop - [2893](https://github.com/PrefectHQ/prefect/pull/2893)

Task Library

- Add option to BigQueryTask to return query as dataframe - [2862](https://github.com/PrefectHQ/prefect/pull/2862)

Fixes

- Add more context keys when running locally so that templating is consistent between local and Cloud runs - [2662](https://github.com/PrefectHQ/prefect/issues/2662)
- Fix Fargate agent not parsing string provided containerDefinitions - [2875](https://github.com/PrefectHQ/prefect/issues/2875)
- Fix Fargate agent providing empty parameters if not set - [2878](https://github.com/PrefectHQ/prefect/issues/2878)
- Fix issue with Queued task runs flooding agents with work - [2884](https://github.com/PrefectHQ/prefect/issues/2884)
- Add missing `prefect register flow` to CLI help text - [2895](https://github.com/PrefectHQ/prefect/pull/2895)

Contributors

- [James Lamb](https://github.com/jameslamb)
- [nathaniel-md](https://github.com/nathaniel-md)

Released on June 25, 2020.

Features

- Task slugs are now stable across rebuilds of the same Flow - [2531](https://github.com/PrefectHQ/prefect/pull/2531)
- Support configuring executors for `LocalEnvironment`, `KubernetesJobEnvironment`, and `FargateTaskEnvironment` - [2805](https://github.com/PrefectHQ/prefect/pull/2805)
- Flows can now be stored and executed using file-based storage - [2840](https://github.com/PrefectHQ/prefect/pull/2840)

Enhancements

- Add option to set `repositoryCredentials` on Fargate Agent `containerDefinitions` - [2822](https://github.com/PrefectHQ/prefect/issues/2822)
- Update GraphQL endpoint to `/graphql` - [2669](https://github.com/PrefectHQ/prefect/pull/2669)
- Allow Cloud Flow Runners to interact properly with Queued runs - [2741](https://github.com/PrefectHQ/prefect/pull/2741)
- Add `Result` serializers - [2755](https://github.com/PrefectHQ/prefect/pull/2755)
- Simplify `DaskExecutor` internals - [2817](https://github.com/PrefectHQ/prefect/pull/2817)
- Set task names in `LocalDaskExecutor` - [2819](https://github.com/PrefectHQ/prefect/pull/2819)
- Flows registered without an image set will default to `all_extras` - [2828](https://github.com/PrefectHQ/prefect/pull/2828)
- Improve error message when sending unauthorized requests to Cloud - [2810](https://github.com/PrefectHQ/prefect/issues/2810)
- Forward state change status back to core - [2839](https://github.com/PrefectHQ/prefect/pull/2839)
- Add GitHub storage for storing flows as files in a GitHub repo - [2840](https://github.com/PrefectHQ/prefect/pull/2840)
- Add `prefect register flow` CLI command for registering flows from files - [2840](https://github.com/PrefectHQ/prefect/pull/2840)
- Add default `GITHUB_ACCESS_TOKEN` secret - [2840](https://github.com/PrefectHQ/prefect/pull/2840)
- Create utility function for getting Kubernetes client - [2845](https://github.com/PrefectHQ/prefect/pull/2845)

Task Library

- Adds a MySQL task using pymysql driver - [2124](https://github.com/PrefectHQ/prefect/issues/2124)
- Add some tasks for working with Google Sheets - [2614](https://github.com/PrefectHQ/prefect/pull/2614)
- Add support for HTML content in the EmailTask - [2811](https://github.com/PrefectHQ/prefect/pull/2811)

Server

- Failing to set a state raises errors more aggressively - [2708](https://github.com/PrefectHQ/prefect/pull/2708)

Fixes

- Fix `all_extras` tag not being set during CI job to build image - [2801](https://github.com/PrefectHQ/prefect/issues/2801)
- Quiet *no candidate Cached states were valid* debug logging - [2815](https://github.com/PrefectHQ/prefect/issues/2815)
- Fix `LocalEnvironment` execute function's use of the flow object - [2804](https://github.com/PrefectHQ/prefect/pull/2804)
- Properly set task names when using `DaskExecutor` - [2814](https://github.com/PrefectHQ/prefect/issues/2814)
- Fix the `LocalDaskExecutor` to only compute tasks once, not multiple times - [2819](https://github.com/PrefectHQ/prefect/pull/2819)
- Generate key names for mapped tasks that work better with Dask's dashboard - [2831](https://github.com/PrefectHQ/prefect/pull/2831)
- Fix FlowRunTask when running against locally deployed Server - [2832](https://github.com/PrefectHQ/prefect/pull/2832)
- Make sure image from Docker storage is always used with KubernetesJobEnvironment - [2838](https://github.com/PrefectHQ/prefect/pull/2838)
- Change Environment.run_flow() to prefer executor from flow's environment - [2849](https://github.com/PrefectHQ/prefect/pull/2849)

Deprecations

- Deprecate `RemoteEnvironment` in favor of `LocalEnvironment` - [2805](https://github.com/PrefectHQ/prefect/pull/2805)
- Deprecate `RemoteDaskEnvironment` in favor of `LocalEnvironment` with a `DaskExecutor` - [2805](https://github.com/PrefectHQ/prefect/pull/2805)
- Deprecate `executor_kwargs` in `KubernetesJobEnvironment` and `FargateTaskEnvironment` in favor of `executor` - [2805](https://github.com/PrefectHQ/prefect/pull/2805)

Breaking Changes

- Remove previously deprecated `SynchronousExecutor` - [2826](https://github.com/PrefectHQ/prefect/pull/2826)

Contributors

- [manesioz](https://github.com/manesioz)
- [Alex Cano](https://github.com/alexisprince1994)
- [James Lamb](https://github.com/jameslamb)
- [Matthew Alhonte](https://github.com/mattalhonte/)
- [Paweł Cieśliński](https://github.com/pcieslinski)