Prefect

Released on Dec 17, 2019

Features

- None

Enhancements

- Enhanced treatment of nested and ordered constant values - [1829](https://github.com/PrefectHQ/prefect/pull/1829)
- Add `on_datetime`, `on_date`, and `at_time` filters - [1837](https://github.com/PrefectHQ/prefect/pull/1837)
- Add `--latest` flag for Kubernetes Agent install CLI command - [1842](https://github.com/PrefectHQ/prefect/pull/1842)
- Add `--no-cloud-logs` flag for all agents to optionally opt-out of logs being sent to Prefect Cloud - [1843](https://github.com/PrefectHQ/prefect/pull/1843)
- Agents mark Flow Runs as `Failed` if a deployment error occurs - [1848](https://github.com/PrefectHQ/prefect/pull/1848)
- `Submitted` states from Agents include deployment identifier information - [1848](https://github.com/PrefectHQ/prefect/pull/1848)
- Update heartbeats to respect Cloud flow settings - [1851](https://github.com/PrefectHQ/prefect/pull/1851)
- Add flow run name to `prefect.context` - [1855](https://github.com/PrefectHQ/prefect/pull/1855)
- Add `--namespace` option for Kubernetes Agent start CLI command - [1859](https://github.com/PrefectHQ/prefect/pull/1859)
- Add Prefect job resource configuration for Kubernetes Agent - [1859](https://github.com/PrefectHQ/prefect/pull/1859)

Task Library

- None

Fixes

- Fix Agent deployment errors interrupting full list of found Flow Runs - [1848](https://github.com/PrefectHQ/prefect/pull/1848)
- Fix issue with a single bad log preventing all logs from being sent to Cloud - [1845](https://github.com/PrefectHQ/prefect/pull/1845)
- Fix Kubernetes Agent passing empty default namespace - [1839](https://github.com/PrefectHQ/prefect/pull/1839)

Deprecations

- None

Breaking Changes

- None

Contributors

- None

Released on Dec 11, 2019

Features

- Universal Deploy: Added new Local Agent to run Flows from Local Storage, Azure Storage, S3 Storage, and GCS Storage - [1819](https://github.com/PrefectHQ/prefect/pull/1819)
- Added Azure Blob Storage for Flows - [1831](https://github.com/PrefectHQ/prefect/pull/1831)
- Added GCS Storage for Flows - [1809](https://github.com/PrefectHQ/prefect/pull/1809)
- Added S3 Storage for Flows - [1753](https://github.com/PrefectHQ/prefect/pull/1753)

Enhancements

- Add convenience `parents()` and `children()` classmethods to all State objects for navigating the hierarchy - [1784](https://github.com/PrefectHQ/prefect/pull/1784)
- Add `--rbac` flag to `prefect agent install` for Kubernetes Agent - [1822](https://github.com/PrefectHQ/prefect/pull/1822)
- Add `flow_run_name` to the context - [1815](https://github.com/PrefectHQ/prefect/pull/1815)
- Add `prefect agent install` option to output `supervisord.conf` file for Local Agent - [1819](https://github.com/PrefectHQ/prefect/pull/1819)
- Add convenience `parents()` and `children()` classmethods to all State objects for navigating the hierarchy - [1784](https://github.com/PrefectHQ/prefect/pull/1784)
- Add new `not_all_skipped` trigger and set it as the default for merge tasks - [1768](https://github.com/PrefectHQ/prefect/issues/1768)


Task Library

- Azure Blob tasks now use newer `BlockBlobService` with connection string authentication - [1831](https://github.com/PrefectHQ/prefect/pull/1831)

Fixes

- Fix issue with `flow.visualize()` for mapped tasks which are skipped - [1765](https://github.com/PrefectHQ/prefect/issues/1765)
- Fix issue with timeouts only being softly enforced - [1145](https://github.com/PrefectHQ/prefect/issues/1145), [#1686](https://github.com/PrefectHQ/prefect/issues/1686)
- Log agent errors using `write_run_logs` instead of the deprecated `write_run_log` - [1791](https://github.com/PrefectHQ/prefect/pull/1791)
- Fix issue with `flow.update()` not transferring constants - [1785](https://github.com/PrefectHQ/prefect/pull/1785)

Deprecations

- `flow.deploy` is deprecated in favor of `flow.register` - [1819](https://github.com/PrefectHQ/prefect/pull/1819)

Breaking Changes

- Default Flow storage is now `Local` instead of `Docker` - [1819](https://github.com/PrefectHQ/prefect/pull/1819)
- Docker based `LocalAgent` is renamed `DockerAgent` - [1819](https://github.com/PrefectHQ/prefect/pull/1819)
- `prefect agent start` now defaults to new `LocalAgent` - [1819](https://github.com/PrefectHQ/prefect/pull/1819)

Changelog

Released on Nov 26, 2019

Features

- Add graceful cancellation hooks to Flow and Task runners - [1758](https://github.com/PrefectHQ/prefect/pull/1758)

Enhancements

- Add option to specify a run name for `cloud run` CLI command - [1756](https://github.com/PrefectHQ/prefect/pull/1756)
- Add `work_stealing` option to `DaskKubernetesEnvironment` - [1760](https://github.com/PrefectHQ/prefect/pull/1760)
- Improve heartbeat thread management - [1770](https://github.com/PrefectHQ/prefect/pull/1770)
- Add unique scheduler Job name to `DaskKubernetesEnvironment` - [1772](https://github.com/PrefectHQ/prefect/pull/1772)
- Add informative error when trying to map with the `LocalDaskExecutor` using processes - [1777](https://github.com/PrefectHQ/prefect/pull/1777)

Task Library

- None

Fixes

- Fix issue with heartbeat thread deadlocking dask execution when using a `worker_client` - [1750](https://github.com/PrefectHQ/prefect/pull/1750)
- Fix issue with Environments not calling `run_flow` on Environment stored on Flow object - [1752](https://github.com/PrefectHQ/prefect/pull/1752)
- Fix issue with Docker build context when providing custom docker files - [1762](https://github.com/PrefectHQ/prefect/pull/1762)

Deprecations

- None

Breaking Changes

- None

Contributors

- None

Released on Nov 15, 2019.

Features

- Allow users to provide a custom version group ID for controlling Cloud versioning - [1665](https://github.com/PrefectHQ/prefect/issues/1665)
- Stop autogenerating constant tasks - [1730](https://github.com/PrefectHQ/prefect/pull/1730)

Enhancements

- Raise an informative error when context objects are pickled - [1710](https://github.com/PrefectHQ/prefect/issues/1710)
- Add an option to pass in `run_name` to a flow run to override the auto-generated names when calling `create_flow_run` [1661](https://github.com/PrefectHQ/cloud/pull/1661)
- Add informative logs in the event that a heartbeat thread dies - [1721](https://github.com/PrefectHQ/prefect/pull/1721)
- Loosen Job spec requirements for `KubernetesJobEnvironment` - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Loosen `containerDefinitions` requirements for `FargateTaskEnvironment` - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Local Docker agent proactively fails flow runs if image cannot be pulled - [1395](https://github.com/PrefectHQ/prefect/issues/1395)
- Add graceful keyboard interrupt shutdown for all agents - [1731](https://github.com/PrefectHQ/prefect/pull/1731)
- `agent start` CLI command now allows for Agent kwargs - [1737](https://github.com/PrefectHQ/prefect/pull/1737)
- Add users to specify a custom Dockerfile for Docker storage - [1738](https://github.com/PrefectHQ/prefect/pull/1738)
- Expose `labels` kwarg in `flow.deploy` for convenient labeling of Flows - [1742](https://github.com/PrefectHQ/prefect/pull/1742)

Task Library

- None

Fixes

- `FargateTaskEnvironment` now uses provided `family` for task definition naming - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Fix executor initialization missing `self` in `KubernetesJobEnvironment` - [1713](https://github.com/PrefectHQ/prefect/pull/1713)
- Fix `identifier_label` not being generated on each run for Kubernetes based environments - [1718](https://github.com/PrefectHQ/prefect/pull/1718)
- Fix issue where users could not override their user config path when deploying Docker to Cloud - [1719](https://github.com/PrefectHQ/prefect/pull/1719)
- Respect order of inputs in merge - [1736](https://github.com/~/1736)

Deprecations

- None

Breaking Changes

- None

Contributors

- [Brett Naul](https://github.com/bnaul)

Released on Nov 5, 2019

Features

- None

Enhancements

- Add a `save`/`load` interface to Flows - [1685](https://github.com/PrefectHQ/prefect/pull/1685), [#1695](https://github.com/PrefectHQ/prefect/pull/1695)
- Add option to specify `aws_session_token` for the `FargateTaskEnvironment` - [1688](https://github.com/PrefectHQ/prefect/pull/1688)
- Add `EnvVarSecrets` for loading sensitive information from environment variables - [1683](https://github.com/PrefectHQ/prefect/pull/1683)
- Add an informative version header to all Cloud client requests - [1690](https://github.com/PrefectHQ/prefect/pull/1690)
- Auto-label Flow environments when using Local storage - [1696](https://github.com/PrefectHQ/prefect/pull/1696)
- Batch upload logs to Cloud in a background thread for improved performance - [1691](https://github.com/PrefectHQ/prefect/pull/1691)
- Include agent labels within each flow's configuration environment - [1671](https://github.com/PrefectHQ/prefect/issues/1671)

Task Library

- None

Fixes

- Fix Fargate Agent access defaults and environment variable support - [1687](https://github.com/PrefectHQ/prefect/pull/1687)
- Removed default python version for docker builds - [1705](https://github.com/PrefectHQ/prefect/pull/1705)
- Attempt to install prefect in any docker image (if it is not already installed) - [1704](https://github.com/PrefectHQ/prefect/pull/1704)
- Kubernetes Agent deployment yaml now respects new `prefecthq/prefect` image tagging convention - [1707](https://github.com/PrefectHQ/prefect/pull/1707)

Deprecations

- None

Breaking Changes

- None

Contributors

- None

Released on October 29th, 2019.

Features

- Flow Affinity: Environments and Agents now support labeling for execution specification - [1651](https://github.com/PrefectHQ/prefect/pull/1651)
- Add new Secret Tasks for a pluggable and reusable Secrets API - [1346](https://github.com/PrefectHQ/prefect/issues/1346), [#1587](https://github.com/PrefectHQ/prefect/issues/1587)

Enhancements

- Add the ability to delete task tag limits using the client - [1622](https://github.com/PrefectHQ/prefect/pull/1622)
- Adds an "Ask for help" button with a link to the prefect.io support page - [1637](https://github.com/PrefectHQ/prefect/pull/1637)
- Reduces the size of the `prefecthq/prefect` Docker image by ~400MB, which is now the base Docker image used in Flows - [1648](https://github.com/PrefectHQ/prefect/pull/1648)
- Add a new healthcheck for environment dependencies - [1653](https://github.com/PrefectHQ/prefect/pull/1653)
- Add default 30 second timeout to Client requests - [1672](https://github.com/PrefectHQ/prefect/pull/1672)

Task Library

- Add new Secret Tasks for a pluggable and reusable Secrets API - [1346](https://github.com/PrefectHQ/prefect/issues/1346), [#1587](https://github.com/PrefectHQ/prefect/issues/1587)
- Add support for directly passing credentials to task library tasks, instead of passing secret names - [1667](https://github.com/PrefectHQ/prefect/pull/1673)

Fixes

- Fix defaults for unspecified ARNs in the Fargate Agent - [1634](https://github.com/PrefectHQ/prefect/pull/1634)
- Fix ShellTask return value on empty stdout - [1632](https://github.com/PrefectHQ/prefect/pull/1632)
- Fix issue with some Cloud Secrets not being converted from strings - [1655](https://github.com/PrefectHQ/prefect/pull/1655)
- Fix issue with Agent logging config setting not working - [1657](https://github.com/PrefectHQ/prefect/pull/1657)
- Fix issue with SnowflakeQuery tasks not working - [1663](https://github.com/PrefectHQ/prefect/pull/1663)

Deprecations

- Tasks that accepted the name of a secret (often `credentials_secret`) will raise a deprecation warning - [1667](https://github.com/PrefectHQ/prefect/pull/1673)

Breaking Changes

- Fargate Agent now takes in all boto3 camel case arguments instead of specific snake case options - [1649](https://github.com/PrefectHQ/prefect/pull/1649)
- `kubernetes` is no longer installed by default in deployed flow images - [1653](https://github.com/PrefectHQ/prefect/pull/1653)
- Tasks that accepted the name of a secret (often `credentials_secret`) no longer have a default value for that argument, as it has been deprecated - [1667](https://github.com/PrefectHQ/prefect/pull/1673)

Contributors

- [Tobias Schmidt](https://github.com/royalts)