Prefect

Names are hard and stressful.

Changelog

April 05, 2021 <Badge text="beta" type="success" />

Released on April 5, 2021.

Enhancements

- Add terminal flow state handler override - [4198](https://github.com/PrefectHQ/prefect/issues/4198)
- When manually set, `prefect.context.date` will be used to determine dependent values - [4295](https://github.com/PrefectHQ/prefect/pull/4295)
- `prefect.context.date` will be cast to a `DateTime` object if given a parsable string - [4295](https://github.com/PrefectHQ/prefect/pull/4295)
- Expand logging for `DaskExecutor`, including the cluster dashboard address (if available) - [4321](https://github.com/PrefectHQ/prefect/pull/4321)
- Add ability to stream ShellTask logs with level INFO - [4322](https://github.com/PrefectHQ/prefect/pull/4322)
- Add architecture diagram to docs - [4187](https://github.com/PrefectHQ/prefect/issues/4187)
- Speed up flow validation logic - [4347](https://github.com/PrefectHQ/prefect/pull/4347)
- Pin dask upper package versions - [4350](https://github.com/PrefectHQ/prefect/pull/4350)

Task Library

- Add first basic implementation of Soda SQL Scan task to run DQ tests using [soda-sql](https://github.com/sodadata/soda-sql) library
- Add new task KubernetesSecret - [4307](https://github.com/PrefectHQ/prefect/pull/4307)

Fixes

- Fix `DatabricksRunNow` task attribute override behavior - [4309](https://github.com/PrefectHQ/prefect/pull/4309)
- Use default flow labels when triggering flow runs from CLI - [4316](https://github.com/PrefectHQ/prefect/pull/4316)
- Improve ECS agent error messages, and fix bug that prevented using ECS agent on Fargate with ECR - [4325](https://github.com/PrefectHQ/prefect/pull/4325)
- Support imports from local directory when registering/building flows via CLI - [4332](https://github.com/PrefectHQ/prefect/pull/4332)
- Speedup flushing of logs to cloud/server on flow run shutdown, avoiding lost logs on platforms that SIGKILL the process after a short period - [4334](https://github.com/PrefectHQ/prefect/pull/4334)

Contributors

- [Alessandro Lollo](https://github.com/AlessandroLollo)
- [David Zucker](https://github.com/davzucky)
- [Greg Lu](https://github.com/greglu)
- [James Lamb](https://github.com/jameslamb)
- [Sean Talia](https://github.com/TaliaSRTR)

Changelog

Released on March 25, 2021.

Task Library

- Ensures Snowflake Query Task output is serializable [3744] (https://github.com/PrefectHQ/prefect/issues/3744)

Fixes

- Always load task run info prior to running a task - [4296](https://github.com/PrefectHQ/prefect/pull/4296)

Changelog

Released on March 24, 2021.

Features

- Add new improved `prefect register` CLI command. This command supports registering multiple flows at once (through multiple file paths or directories), and also includes a new `--watch` flag for watching files and automatically re-registering upon changes. - [4256](https://github.com/PrefectHQ/prefect/pull/4256)
- New `prefect build` CLI command for building flows. Artifacts produced by this command can then be used by `prefect register` to register flows without requiring the source. - [4282](https://github.com/PrefectHQ/prefect/pull/4282)

Enhancements

- Use explicit exception chaining [3306](https://github.com/PrefectHQ/prefect/issues/3306)
- Add `as_bytes` option to `S3Download` task - [4238](https://github.com/PrefectHQ/prefect/pull/4238)
- Improve error message when a loaded flow doesn't match the version stored in Prefect Cloud/Server - [4259](https://github.com/PrefectHQ/prefect/pull/4259)
- Support setting flow run labels from cli - [4266](https://github.com/PrefectHQ/prefect/pull/4266)
- Support setting default `image` in `--job-template`/`--task-definition` in Kubernetes/ECS agents - [4270](https://github.com/PrefectHQ/prefect/pull/4270)

Task Library

- Adds logging of cell outputs to Jupyter task - [4265] (https://github.com/PrefectHQ/prefect/issues/4265)
- Add `user` and `password` as runtime parameters to Exasol tasks - [4268](https://github.com/PrefectHQ/prefect/pull/4268)

Fixes

- Fix bug where sometimes the global `prefect.context` wouldn't be respected during a flow run - [4287](https://github.com/PrefectHQ/prefect/pull/4287)

Deprecations

- Deprecate the old `prefect register flow` CLI command in favor of `prefect register` - [4256](https://github.com/PrefectHQ/prefect/pull/4256)
- Deprecate `user` and `password` arguments to Exasol task constructors in favor of runtime parameters - [4268](https://github.com/PrefectHQ/prefect/pull/4268)

Contributors

- [Abid Ahmad](https://github.com/abid1998)
- [Alexandr N. Zamaraev](https://github.com/tonal)
- [Jacob Hayes](https://github.com/JacobHayes)
- [Timo S.](https://github.com/sti0)

Released on March 10, 2021.

Enhancements

- Upgrade hasura to 1.3.3 in Prefect Server - [4126](https://github.com/PrefectHQ/prefect/pull/4126)
- Add `--docker-client-timeout` flag to docker agent, for configuring the timeout for all docker API requests - [4232](https://github.com/PrefectHQ/prefect/pull/4232)
- Make `--slug` flag optional in `prefect server create-tenant` - [4240](https://github.com/PrefectHQ/prefect/pull/4240)
- Move `command`, `environment`, `cpu`, `memory`, `execution_role_arn`, and `task_role_arn` configuration for ECS tasks from definition time to run time in the ECS agent - [4211](https://github.com/PrefectHQ/prefect/pull/4211)
- Register (and deregister) a new task definition for every flow run in ECS agent - [4211](https://github.com/PrefectHQ/prefect/pull/4211)

Task Library

- Adds new filesystem `Copy` and `Remove` tasks - [4202](https://github.com/PrefectHQ/prefect/pull/4202)

Fixes

- Don't forward `nout` to mapped tasks - [4206](https://github.com/PrefectHQ/prefect/pull/4206)
- Fix `Task` signature generation in the presence of with variadic kwargs - [4235](https://github.com/PrefectHQ/prefect/pull/4235)
- Ensure `Flow` is serializable using `pickle` - [4209](https://github.com/PrefectHQ/prefect/pull/4209)

Contributors

- [Ben Fogelson](https://github.com/benfogelson)
- [Marwan S.](https://github.com/marwan116)
- [Timo S.](https://github.com/sti0)

Released on March 3, 2021.

Features

- Add command `prefect server config` to output configured docker-compose yaml - [4176](https://github.com/PrefectHQ/prefect/pull/4176)

Enhancements

- Add `project_name` and `project_id` to context during Cloud/Server flow runs - [4083](https://github.com/PrefectHQ/prefect/pull/4083)
- Better error message when flow not found in file - [4182](https://github.com/PrefectHQ/prefect/pull/4182)
- Improve generated names for `GetItem` tasks - [4183](https://github.com/PrefectHQ/prefect/pull/4183)
- Add `base_url` option to `GitHub` storage - [4194](https://github.com/PrefectHQ/prefect/pull/4194)

Task Library

- Add filehandling tasks `Move`, `Unzip`, `Zip` - [4131](https://github.com/PrefectHQ/prefect/pull/4139)
- Add `msg_plain`, `email_to_cc`, and `email_to_bcc` options to `EmailTask` - [4157](https://github.com/PrefectHQ/prefect/pull/4157)
- Add `jar_params` option to `DatabricksRunNow` task - [4157](https://github.com/PrefectHQ/prefect/pull/4178)

Fixes

- Make task slug generation robust to modifying existing task names - [4189](https://github.com/PrefectHQ/prefect/pull/4189)
- Forward `client_options` to `S3Result` from `S3` storage - [4195](https://github.com/PrefectHQ/prefect/pull/4195)

Contributors

- [David Zucker](https://github.com/davzucky)
- [Jacob Hayes](https://github.com/JacobHayes)
- [Joël Luijmes](https://github.com/joelluijmes)
- [Timo S.](https://github.com/sti0)
- [Yogi Patel](https://github.com/ypatel-whitepages)

Released on February 23, 2021.

Fixes

- Dynamically import compression libraries for `CompressedSerializer` - [4150](https://github.com/PrefectHQ/prefect/pull/4150)
- Support passing environment variables containing `=` through agent CLI `--env` flag - [4160](https://github.com/PrefectHQ/prefect/pull/4160)