Prefect

Changelog

Released on November 30, 2021.

Enhancements

- Add `end_time` to `FlowRunView.get_logs` - [5138](https://github.com/PrefectHQ/prefect/pull/5138)
- Update `watch_flow_run` to stream logs immediately instead of waiting for flow run state changes - [5138](https://github.com/PrefectHQ/prefect/pull/5138)
- Allow setting container ports for `DockerRun` - [5130](https://github.com/PrefectHQ/prefect/issues/5130)
- Clarify `ECSRun` documentation, especially the ambiguities in setting IAM roles - [5110](https://github.com/PrefectHQ/prefect/issues/5110)
- Fix deprecated usage of `marshmallow.fields.Dict` in RRule schedules - [4540](https://github.com/PrefectHQ/prefect/issues/4540), [#4903](https://github.com/PrefectHQ/prefect/pull/4903)

Fixes

- Fix connection to local server instances when using `DockerAgent` on linux - [5182](https://github.com/PrefectHQ/prefect/pull/5182)

Task Library

- Add support for triggering [Airbyte](https://airbyte.io/) connection sync jobs using `AirbyteConnectionTask` - [#5078](https://github.com/PrefectHQ/prefect/pull/5078)
- Add artifact publishing to `DbtCloudRunJob` task - [5135](https://github.com/PrefectHQ/prefect/pull/5135)
- Add support for running data quality checks on Spark DataFrames using `soda-spark` - [4901](https://github.com/PrefectHQ/prefect/pull/5144)


Contributors

- [Alessandro Lollo](https://github.com/AlessandroLollo)
- [Nicholas Hemley](https://github.com/iamniche-too)
- [Shahil Mawjee](https://github.com/s-mawjee)

This hotfix release fixes an issue where the kubernetes agent would attempt to load a secret value and fail if it was not present.

See [the PR](https://github.com/PrefectHQ/prefect/pull/5131) for details.

Don't miss [all the exciting changes from 0.15.8](https://github.com/PrefectHQ/prefect/releases/tag/0.15.8) released today as well.

Features

- Add support for rich iCal style scheduling via RRules - [4901](https://github.com/PrefectHQ/prefect/pull/4901)
- Add Google Cloud Vertex agent and run configuration - [4989](https://github.com/PrefectHQ/prefect/pull/4989)

Enhancements

- Allow `Azure` flow storage to overwrite existing blobs - [5103](https://github.com/PrefectHQ/prefect/pull/5103)
- Provide option to specify a dockerignore when using Docker storage - [4980](https://github.com/PrefectHQ/prefect/pull/4980)
- Add keep-alive connections for kubernetes client API connections - [5066](https://github.com/PrefectHQ/prefect/pull/5066)
- Add `idempotency_key` to `create_flow_run` task - [5125](https://github.com/PrefectHQ/prefect/pull/5125)
- Add `raise_final_state` to `wait_for_flow_run` task to reflect child flow run state - [5129](https://github.com/PrefectHQ/prefect/pull/5129)

Task Library

- Bump maximum `google-cloud-bigquery` version to support 2.x - [5084](https://github.com/PrefectHQ/prefect/pull/5084)
- Add `Glob` task for collecting files in directories - [5077](https://github.com/PrefectHQ/prefect/pull/5077)
- Add `DbtCloudRunJob` task for triggering dbt cloud run jobs - [5085](https://github.com/PrefectHQ/prefect/pull/5085)
- Added Kafka Tasks entry to website docs - [5094](https://github.com/PrefectHQ/prefect/pull/5094)

Fixes

- Update the `FlowView` to be more robust to serialized flow changes in the backend - [5116](https://github.com/PrefectHQ/prefect/pull/5116)

Deprecations

- Move artifacts functions to `prefect.backend.artifacts` - [5117](https://github.com/PrefectHQ/prefect/pull/5117)

Server

This release includes a Prefect Server update that updates an upstream dependency to fix a security vulnerability. See the [release changelog](https://github.com/PrefectHQ/server/blob/master/Changelog.md#november-09-2021-) for more details.

Contributors

- [Alessandro Lollo](https://github.com/AlessandroLollo)
- [Bradley Axen](https://github.com/baxen)
- [Damien Ramunno-Johnson](https://github.com/damienrj)
- [Jonas Miederer](https://github.com/jonasmiederer)
- [Josh Wang](https;//github.com/wangjoshuah)
- [Nitay Joffe](https://github.com/nitay)
- [Timo S.](https://github.com/sti0)
- [Brett Naul](https://github.com/bnaul)

Enhancements

- Add flatten support to `apply_map` - [4996](https://github.com/PrefectHQ/prefect/pull/4996)
- Add dask performance report to `DaskExecutor` - [5032](https://github.com/PrefectHQ/prefect/pull/5032)
- Update git storage `repo` parameter to be optional if specifying `git_clone_url_secret_name` - [5033](https://github.com/PrefectHQ/prefect/pull/5033)
- Add `task_run_name` to `prefect.context` - [5055](https://github.com/PrefectHQ/prefect/pull/5055)

Fixes

- Reduce rate limit related failures with the ECS agent - [5059](https://github.com/PrefectHQ/prefect/pull/5059)

Task Library

- Add `data` parameter to `SQLiteQuery` task - [4981](https://github.com/PrefectHQ/prefect/pull/4981)
- Allow `EmailTask` to use insecure internal SMTP servers with `smtp_type="INSECURE"` - [5012](https://github.com/PrefectHQ/prefect/pull/5012)
- Fix Databricks `run_id` mutation during task runs - [4958](https://github.com/PrefectHQ/prefect/issues/4958)
- Add `manual` setting to `FivetranSyncTask` allowing retention of Fivetan scheduling -[5065](https://github.com/PrefectHQ/prefect/pull/5065)

Contributors

- [Daniel Saxton](https://github.com/dsaxton)
- [Emre Akgün](https://github.com/Fraznist)
- [Jessica Smith](https://github.com/NodeJSmith)

Enhancements

- Improve setting the Azure storage connection string - [4955](https://github.com/PrefectHQ/prefect/pull/4955)
- Allow disabling retries for a task with `max_retries=0` when retries are globally configured - [4971](https://github.com/PrefectHQ/prefect/pull/4971)

Task Library

- Allow Exasol Tasks to handle Prefect Secrets directly - [4436](https://github.com/PrefectHQ/prefect/pull/4436)
- Adding [Census](https://www.getcensus.com/) Syncs to the task library - [#4935](https://github.com/PrefectHQ/prefect/pull/4935)

Fixes

- Fix bug where `LocalDaskExecutor` did not respond to a `PrefectSignal` - [4924](https://github.com/PrefectHQ/prefect/pull/4924)
- Fix `PostgresFetch` with headers for one row - [4968](https://github.com/PrefectHQ/prefect/pull/4968)
- Fix bug where `apply_map` could create acyclic flows - [4970](https://github.com/PrefectHQ/prefect/pull/4970)

Contributors

- [Donny Flynn](https://github.com/dflynn20)
- [Noah Holm](https://github.com/noppaz)
- [Timo S.](https://github.com/sti0)

Released on September 2, 2021.

Features

- Python 3.9 docker images are now published - [4896](https://github.com/PrefectHQ/prefect/pull/4896)

Enhancements

- Add `--expose` flag to `prefect server` cli to make the Core server and UI listen to all interfaces - [4821](https://github.com/PrefectHQ/prefect/pull/4821)
- Pass existing/local environment variables to agentless flow runs - [4917](https://github.com/PrefectHQ/prefect/pull/4917)
- Add `--idempotency-key` to `prefect run` - [4928](https://github.com/PrefectHQ/prefect/pull/4928)
- Add support for larger flow registration calls - [4930](https://github.com/PrefectHQ/prefect/pull/4930)
- Ignore schedules by default for CLI flow runs and add flag to run based on schedule for local only runs [4817](https://github.com/PrefectHQ/prefect/pull/4817)

Task Library

- Feature: Added `SnowflakeQueryFromFile` task [3744](https://github.com/PrefectHQ/prefect/pull/4363)
- Enhancement: Log boto exceptions encountered in the in AWS `BatchSubmit` task - [4771](https://github.com/PrefectHQ/prefect/pull/4771)
- Breaking: Legacy Dremio authentication has been updated to the new pattern in `DremioFetch` - [4872](https://github.com/PrefectHQ/prefect/pull/4872)
- Fix: Use runtime arguments over init arguments instead of ignoring them for MySQL Tasks - [4907](https://github.com/PrefectHQ/prefect/pull/4907)
Fixes

- Adjust log limits to match backend logic for better UX - [4900](https://github.com/PrefectHQ/prefect/pull/4900)
- Fix use of `marshmallow.fields.Dict` to use `keys` as a kwarg rather than `key`. - [4903](https://github.com/PrefectHQ/prefect/pull/4903)
- API server settings are passed correctly to task workers when using Prefect Server - [4914](https://github.com/PrefectHQ/prefect/pull/4914)
- Do not attempt attempt to set `host_gateway` if using an unsupported Docker Engine version - [4809](https://github.com/PrefectHQ/prefect/pull/4809)
- Ignore jobs without a `flow_run_id` label in `KubernetesAgent.manage_jobs` - [4934](https://github.com/PrefectHQ/prefect/pull/4934)
Breaking Changes

- Services run by `prefect server` cli are now local by default (listen to localhost instead of 0.0.0.0); use `--expose` if you want to connect from a remote location - [4821](https://github.com/PrefectHQ/prefect/pull/4821)
- The changes in flow registration require Prefect Server 2021.09.02. Prefect Server will need to be upgraded before flows can be registered from this version - [4930](https://github.com/PrefectHQ/prefect/pull/4930)

Contributors

- [Deepyaman Datta](https://github.com/deepyaman)
- [Henri Hannetel](https://github.com/HenriTEL)
- [Johnny Tirado](https://github.com/jclocks)
- [Kathryn Klarich](https://github.com/klarich)
- [Tenzin Choedak](https://github.com/tchoedak)
- [Vincent Xue](https://github.com/xuevin)