Prefect

Released on August 17, 2021.

Docs

- Add a getting started section with a quick start guide for both core and orchestration sections - [4734](https://github.com/PrefectHQ/prefect/pull/4734)
Enhancements

- Expose Snowflake cursor type to SnowflakeQuery task arguments [4786](https://github.com/PrefectHQ/prefect/issues/4786)
- Add ability to use threaded flow heartbeats - [4844](https://github.com/PrefectHQ/prefect/pull/4844)
- Improve behavior when API rate limits are encountered - [4852](https://github.com/PrefectHQ/prefect/pull/4852)
- Allow custom git clone url for `Git` storage - [4870](https://github.com/PrefectHQ/cloud/pull/4870)
- Add `on_worker_status_changed` callback to the `DaskExecutor` - [4874](https://github.com/PrefectHQ/prefect/pull/4874)
- Add `--agent-config-id` to `prefect agent <kubernetes|local> install` - [4876](https://github.com/PrefectHQ/prefect/pull/4876)

Task Library

- Add new prometheus task to push to gateway - [4623](https://github.com/PrefectHQ/prefect/pull/4623)

Fixes

- Fix binding of named volumes to flow containers with Docker agent - [4800](https://github.com/PrefectHQ/prefect/pull/4800)
- Fix `ImportError` typo in dropbox module - [4855](https://github.com/PrefectHQ/prefect/pull/4855)
- Fix default safe char for gitlab storage repo path - [4828](https://github.com/PrefectHQ/prefect/pull/4828)

Contributors

- [Austen Bouza](https://github.com/austen-bouza)
- [David Zucker](https://github.com/davzucky)
- [Jacob Dawang](https://github.com/jdawang)
- [Tom Forbes](https://github.com/orf)
- [Mat](https://github.com/matmiad)

Enhancements

- Add new `evaluation_parameters` parameter to `RunGreatExpectationsValidation` task - [4798](https://github.com/PrefectHQ/prefect/pull/4798)

Fixes

- Fix `create_flow_run` compatibility with auth tokens - [4801](https://github.com/PrefectHQ/prefect/pull/4801)
- Fix auto-quoting for strings that begin with numeric characters - [4802](https://github.com/PrefectHQ/prefect/pull/4802)

Contributors

- [Michal Zawadzki](https://github.com/trymzet)

Enhancements

- Allow CLI registration of flows without starting their schedule `prefect register --no-schedule` - [4752](https://github.com/PrefectHQ/prefect/pull/4752)
- Add `host_config` to `DockerRun` to expose deeper settings for Docker flow runs - [4733](https://github.com/PrefectHQ/prefect/issues/4773)
- Enable loading additional repository files with `Git` storage - [4767](https://github.com/PrefectHQ/prefect/pull/4767)
- Update flow run heartbeats to be robust to exceptions - [4736](https://github.com/PrefectHQ/prefect/pull/4736)
- Allow `prefect build/register` paths to contain globs for recursion - [4761](https://github.com/PrefectHQ/prefect/pull/4761)

Fixes

- Fix duplicate task runs in `FlowRunView.get_all_task_runs` - [4774](https://github.com/PrefectHQ/prefect/pull/4774)
- Fix zombie processes from exited heartbeats - [4733](https://github.com/PrefectHQ/prefect/pull/4733)
- Missing `auth_file` directory is created when saving credentials - [4792](https://github.com/PrefectHQ/prefect/pull/4792)

Task Library

- Add `VaultSecret` task for retrieving secrets from private Vault instances - [4656](https://github.com/PrefectHQ/prefect/pull/4656)


Contributors

- [Gabriel Montañola](https://github.com/gmontanola)
- [Nelson Griffiths](https://github.com/ngriffiths13)
- [Pawel Janowski](https://github.com/pjanowski)
- [Yueh Han Huang](https://github.com/bojne)
- [Chris Ottinger](https://github.com/datwiz)

Enhancements

- Add documentation for querying role and membership info - [4721](https://github.com/PrefectHQ/prefect/pull/4721)
- Checkpoint task results when `SUCCESS` is raised - [4744](https://github.com/PrefectHQ/prefect/pull/4744)

Fixes

- Fix loading of `PREFECT__CLOUD__API_KEY` environment variable when starting agents - [4751](https://github.com/PrefectHQ/prefect/pull/4751)
- Fix bug where the tenant could not be inferred during flow runs while using token auth - [4758](https://github.com/PrefectHQ/prefect/pull/4758)
- Fix bug where an agent using token auth could clear the tenant from disk - [4759](https://github.com/PrefectHQ/prefect/pull/4759)

Released on July 1, 2021.

Features

- Add objects for inspecting flows, flow runs, and task runs without writing queries - [4426](https://github.com/PrefectHQ/prefect/pull/4426)
- Rehaul `prefect run` CLI for executing flows locally and with agents - [4463](https://github.com/PrefectHQ/prefect/pull/4463)
- Add flow run tasks to simplify flow run result passing - [4563](https://github.com/PrefectHQ/prefect/pull/4563)
- Add agentless execution for flow runs - [4589](https://github.com/PrefectHQ/prefect/pull/4589)
- Add `prefect auth create-key` to create API keys - [4643](https://github.com/PrefectHQ/prefect/pull/4643)
- Add `prefect auth list-keys` to list API key metadata - [4643](https://github.com/PrefectHQ/prefect/pull/4643)
- Add `prefect auth revoke-key` to revoke an API key - [4643](https://github.com/PrefectHQ/prefect/pull/4643)
- Add `prefect auth status` command to see the state of your authentication - [4643](https://github.com/PrefectHQ/prefect/pull/4643)

Enhancements

- Improve flow run documentation with new dedicated section - [4492](https://github.com/PrefectHQ/prefect/pull/4492)
- Update `Client` to support API keys - [4643](https://github.com/PrefectHQ/prefect/pull/4643)
- Add API key support to `prefect auth login/logout/switch-tenants` - [4643](https://github.com/PrefectHQ/prefect/pull/4643)
- API keys can be configured in the Prefect config - [4643](https://github.com/PrefectHQ/prefect/pull/4643)
- Change SendGrid `SendEmail` task to use secret value - [4669](https://github.com/PrefectHQ/prefect/pull/4669)
- Add `TenantView` for retrieval of tenant information - [4676](https://github.com/PrefectHQ/prefect/pull/4676)
- Add custom rbac documentation - [4696](https://github.com/PrefectHQ/prefect/pull/4696)
- Exit with non-zero status on flow run failure when watched - [4709](https://github.com/PrefectHQ/prefect/pull/4709)
- Display return code on local agent flow process failure - [4715](https://github.com/PrefectHQ/prefect/pull/4715)

Task Library

- Add `KafkaBatchConsume` and `KafkaBatchProduce` tasks - [4533](https://github.com/PrefectHQ/prefect/pull/4533)

Fixes

- Fix cleanup issue with `Git` storage on Windows - [4665](https://github.com/PrefectHQ/prefect/pull/4665)
- Pass API keys as tokens for compatibility when creating flow run environments - [4683](https://github.com/PrefectHQ/prefect/pull/4683)
- Fix missing event timestamp attribute errors in K8s agent - [4693](https://github.com/PrefectHQ/prefect/pull/4693)
- Fix backwards compatibility for flows without a `terminal_state_handler` - [4695](https://github.com/PrefectHQ/prefect/pull/4695)
- Raise a better exception when a task run result type is not set in `TaskRunView.get_result()` - [4708](https://github.com/PrefectHQ/prefect/pull/4708)

Deprecations

- Deprecate `prefect auth create-token` - [4643](https://github.com/PrefectHQ/prefect/pull/4643)
- Deprecate `prefect auth list-tokens` - [4643](https://github.com/PrefectHQ/prefect/pull/4643)
- Deprecate `prefect auth revoke-token` - [4643](https://github.com/PrefectHQ/prefect/pull/4643)
- Deprecate setting auth tokens in the Prefect config - [4643](https://github.com/PrefectHQ/prefect/pull/4643)
- `prefect.utilities.exceptions` has been deprecated in favor of `prefect.exceptions` - [4664](https://github.com/PrefectHQ/prefect/pull/4664)

Breaking Changes

- Remove deprecated `prefect.environment.storage` module
- Remove deprecated `DockerAgent` kwarg `network`
- Remove deprecated `kubernetes.ResourceManager` class
- Remove deprecated `prefect agent start/install <agent-type>` commands
- Remove deprecated `prefect agent local start` flag `--storage-labels`
- Remove deprecated `DroboxDownload` task kwarg `access_token_secret`
- Remove deprecated `GCS...` tasks kwarg `encryption_key_secret`
- Remove deprecated `prefect.tasks.google` module
- Remove deprecated `prefect.tasks.secret.Secret` class
- Remove deprecated `Scheduler` serializers for Prefect <0.6.0
- Remove deprecated `RunGreatExpectionsCheckpoint` task
- Remove deprecated `OneTimeSchedule` and `UnionSchedule` classes
- Remove deprecated flow run tasks ending in `Task`
- Remove deprecated prefect.utilities.tasks.unmapped; moved to `prefect.utilities.edges.unmapped`
- Prefect state signals now inherit from `BaseException` to prevent accidental capture - [4664](https://github.com/PrefectHQ/prefect/pull/4664)
- `TaskTimeoutError` has been replaced with `TaskTimeoutSignal` - [4664](https://github.com/PrefectHQ/prefect/pull/4664)
- `VersionLockError` has been replaced with `VersionLockMismatchSignal` - [4664](https://github.com/PrefectHQ/prefect/pull/4664)

Contributors

- [Stéphan Taljaard](https://github.com/taljaards)
- [Tenzin Choedak](https://github.com/tchoedak)

Released on June 15, 2021.

Enhancements

- Use `functools.update_wrapper` for `FunctionTask` - [4608](https://github.com/PrefectHQ/prefect/pull/4608)
- Add ability to merge reference tasks when combining two flows - [4644](https://github.com/PrefectHQ/prefect/pull/4644)
- Add client side check for key value size - [4655](https://github.com/PrefectHQ/prefect/pull/4655)
- Ensure stack traces are included in logs during task run exceptions - [4657](https://github.com/PrefectHQ/prefect/pull/4657)
- Add `poll_interval` parameter to `StartFlowRun` to define the polling interval when waiting for the flow run to finish - [4641](https://github.com/PrefectHQ/prefect/pull/4641)
- Add ability to set task timeouts with `timedelta` objects - [4619](https://github.com/PrefectHQ/prefect/pull/4619)

Fixes

- Add ssh documentation - [4539](https://github.com/PrefectHQ/prefect/pull/4539)

Contributors

- [Pawel Janowski, Recursion](https://github.com/pjanowski)
- [Peter Roelants](https://github.com/peterroelants)