Prefect

Released on Oct 16, 2019

Features

- Environments now allow for optional `on_start` and `on_exit` callbacks - [1610](https://github.com/PrefectHQ/prefect/pull/1610)

Enhancements

- Raise more informative error when calling `flow.visualize()` if Graphviz executable not installed - [1602](https://github.com/PrefectHQ/prefect/pull/1602)
- Allow authentication to Azure Blob Storage with SAS token - [1600](https://github.com/PrefectHQ/prefect/pull/1600)
- Additional debug logs to `Docker Container` and `Docker Image` tasks - [920](https://github.com/PrefectHQ/prefect/issues/920)
- Changes to Fargate agent to support temporary credentials and IAM role based credentials within AWS compute such as a container or ec2 instance. [1607](https://github.com/PrefectHQ/prefect/pull/1607)
- Local Secrets set through environment variable now retain their casing - [1601](https://github.com/PrefectHQ/prefect/issues/1601)
- Agents can accept an optional `name` for logging and debugging - [1612](https://github.com/PrefectHQ/prefect/pull/1612)
- Added AWS configuration options for Fargate Agent (task_role_arn, execution_role_arn) - [1614](https://github.com/PrefectHQ/prefect/pull/1614)
- Change EmailTask to accept SMTP server settings as well as an email_from kwarg - [1619](https://github.com/PrefectHQ/prefect/pull/1619)

Task Library

- Add `return_all` kwarg to `ShellTask` for optionally returning all lines of stdout - [1598](https://github.com/PrefectHQ/prefect/pull/1598)
- Add `CosmosDBCreateItem`, `CosmosDBReadItems`, `CosmosDBQueryItems` and for interacting with data stored on Azure Cosmos DB - [1617](https://github.com/PrefectHQ/prefect/pull/1617)

Fixes

- Fix issue with running local Flow without a schedule containing cached tasks - [1599](https://github.com/PrefectHQ/prefect/pull/1599)
- Remove blank string for `task_run_id` in k8s resource manager - [1604](https://github.com/PrefectHQ/prefect/pull/1604)
- Fix issue with merge task not working for pandas dataframes and numpy arrays - [1609](https://github.com/PrefectHQ/prefect/pull/1609)

Deprecations

- None

Breaking Changes

- Local Secrets set through environment variable now retain their casing - [1601](https://github.com/PrefectHQ/prefect/issues/1601)

Contributors

- [Mark McDonald](https://github.com/mhmcdonal)
- [Sherman K](https://github.com/shrmnk)

Released on October 3, 2019

Features

- Added `KubernetesJobEnvironment` - [1548](https://github.com/PrefectHQ/prefect/pull/1548)
- Add ability to enforce Task concurrency limits by tag in Prefect Cloud - [1570](https://github.com/PrefectHQ/prefect/pull/1570)
- Added `FargateTaskEnvironment` - [1592](https://github.com/PrefectHQ/prefect/pull/1592)

Enhancements

- Allow the `Client` to more gracefully handle failed login attempts on initialization - [1535](https://github.com/PrefectHQ/prefect/pull/1535)
- Replace `DotDict` with `box.Box` - [1518](https://github.com/PrefectHQ/prefect/pull/1518)
- Store `cached_inputs` on Failed states and call their result handlers if they were provided - [1557](https://github.com/PrefectHQ/prefect/pull/1557)
- `raise_on_exception` no longer raises for Prefect Signals, as these are typically intentional / for control flow - [1562](https://github.com/PrefectHQ/prefect/pull/1562)
- `run cloud` CLI command takes in optional `--parameters` as a file path pointing to a JSON file - [1582](https://github.com/PrefectHQ/prefect/pull/1582)
- Always consider `Constant` tasks successful and unpack them immediately instead of submitting them for execution - [1527](https://github.com/PrefectHQ/prefect/issues/1527)

Task Library

- Add `BlobStorageDownload` and `BlobStorageUpload` for interacting with data stored on Azure Blob Storage - [1538](https://github.com/PrefectHQ/prefect/pull/1538)
- Loosen Kubernetes Tasks' requirement of an API secret key - [1559](https://github.com/PrefectHQ/prefect/pull/1559)
- Add tasks for working in Azure Machine Learning Serviec with Datastores and Datasets - [1590](https://github.com/PrefectHQ/prefect/pull/1590)

Fixes

- Fix issue with certain Pause / Resume / Retry pipelines retrying indefinitely - [1177](https://github.com/PrefectHQ/prefect/issues/1177)
- Kubernetes Agent deployment YAML generation defaults to local Prefect version - [1573](https://github.com/PrefectHQ/prefect/pull/1573)
- Fix issue with custom result handlers not working when called in `cached_inputs` - [1585](https://github.com/PrefectHQ/prefect/pull/1585)

Deprecations

- None

Breaking Changes

- None

Contributors

- [Fredrik Sannholm](https://github.com/frsann)

Changelog

Released on September 19th, 2019.

Features

- Added Fargate agent - [1521](https://github.com/PrefectHQ/prefect/pull/1521)
- Custom user-written environments can be deployed to Prefect Cloud - [1534](https://github.com/PrefectHQ/prefect/pull/1534), [#1537](https://github.com/PrefectHQ/prefect/pull/1537)

Enhancements

- Allow for Agents to correctly run in environments with differently calibrated clocks - [1402](https://github.com/PrefectHQ/prefect/issues/1402)
- Refactor `RemoteEnvironment` to utilize the `get_flow` storage interface - [1476](https://github.com/PrefectHQ/prefect/issues/1476)
- Ensure Task logger is available in context throughout every pipeline step of the run - [1509](https://github.com/PrefectHQ/prefect/issues/1509)
- Skip Docker registry pushing and pulling on empty `registry_url` attribute - [1525](https://github.com/PrefectHQ/prefect/pull/1525)
- Agents now log platform errors to flow runs which cannot deploy - [1528](https://github.com/PrefectHQ/prefect/pull/1528)
- Updating `ShellTask` to work more like Airflow Bash Operator for streaming logs and returning values - [1451](https://github.com/PrefectHQ/prefect/pull/1451)
- Agents now have a verbose/debug logging option for granular output - [1532](https://github.com/PrefectHQ/prefect/pull/1532)
- `DaskKubernetesEnvironment` now allows for custom scheduler and worker specs - [1543](https://github.com/PrefectHQ/prefect/pull/1534), [#1537](https://github.com/PrefectHQ/prefect/pull/1537)

Task Library

- None

Fixes

- Fix map error by removing `imagePullSecrets` from Kubernetes Agent install if not provided - [1524](https://github.com/PrefectHQ/prefect/pull/1524)
- Fix issue with two INFO logs not being associated with the Task Run in Cloud - [1526](https://github.com/PrefectHQ/prefect/pull/1526)
- `execute` CLI command can now load custom environments off of the flow object - [1534](https://github.com/PrefectHQ/prefect/pull/1534)

Deprecations

- None

Breaking Changes

- Update `ShellTask` to return only the last line of stdout, as a string - [1451](https://github.com/PrefectHQ/prefect/pull/1451)

Contributors

- [braunreyes](https://github.com/braunreyes)

Changelog

Released on September 10, 2019

Features

- Improve Windows compatibility for local development and deploying to Prefect Cloud - [1441](https://github.com/PrefectHQ/prefect/pull/1441), [#1456](https://github.com/PrefectHQ/prefect/pull/1456), [#1465](https://github.com/PrefectHQ/prefect/pull/1465), [#1466](https://github.com/PrefectHQ/prefect/pull/1466)

Enhancements

- Add OS platform check to Local Agent for running on Windows machines - [1441](https://github.com/PrefectHQ/prefect/pull/1441)
- Add `--base-url` argument for Docker daemons to `agent start` CLI command - [1441](https://github.com/PrefectHQ/prefect/pull/1441)
- Add environment labels for organizing / tagging different Flow execution environments - [1438](https://github.com/PrefectHQ/prefect/issues/1438)
- Use `-U` option when installing `prefect` in Docker containers to override base image version - [1461](https://github.com/PrefectHQ/prefect/pull/1461)
- Remove restriction that prevented `DotDict` classes from having keys that shadowed dict methods - [1462](https://github.com/PrefectHQ/prefect/pull/1462)
- Added livenessProbe to Kubernetes Agent - [1474](https://github.com/PrefectHQ/prefect/pull/1474)
- Ensure external Dask Clusters do not require Prefect Cloud environment variables to run Cloud flows - [1481](https://github.com/PrefectHQ/prefect/pull/1481)

Task Library

- None

Fixes

- Fix incorrect import in `DaskKubernetesEnvironment` job template - [1458](https://github.com/PrefectHQ/prefect/pull/1458)
- Raise error on Agents started without an appropriate API token - [1459](https://github.com/PrefectHQ/prefect/pull/1459)
- Fix bug when calling `as_nested_dict` on `DotDicts` with an `items` key - [1462](https://github.com/PrefectHQ/prefect/pull/1462)
- Fix `--resource-manager` flag on agent install invalidating `imagePullSecrets` - [1469](https://github.com/PrefectHQ/prefect/pull/1469)
- Fix issue with user-written result handlers in Prefect Cloud preventing some states from being set - [1480](https://github.com/PrefectHQ/prefect/pull/1480)

Deprecations

- None

Breaking Changes

- None

Contributors

- [Joe Schmid](https://github.com/joeschmid)
- [Brett Naul](https://github.com/bnaul)

Released August 30, 2019

Fixes

- Fix issue with reduced mapped tasks not respecting retries - [1436](https://github.com/PrefectHQ/prefect/issues/1436)

Released August 30, 2019

Features

- Added Local, Kubernetes, and Nomad agents - [1341](https://github.com/PrefectHQ/prefect/pull/1341)
- Add the ability for Tasks to sequentially loop - [1356](https://github.com/PrefectHQ/prefect/pull/1356)

Enhancements

- Adds a copy to clipboard button for codeblocks - [213](https://github.com/prefecthq/prefect/issues/213)
- Updates Vuepress to v1.0.3 - [770](https://github.com/prefecthq/prefect/issues/770)
- Introduce configurable default for storage class on Flows - [1044](https://github.com/PrefectHQ/prefect/issues/1044)
- Allow for min and max workers to be specified in `DaskKubernetesEnvironment` - [1338](https://github.com/PrefectHQ/prefect/pulls/1338)
- Use task and flow names for corresponding logger names for better organization - [1355](https://github.com/PrefectHQ/prefect/pull/1355)
- `Paused` states subclass `Scheduled` and can have predefined expirations - [1375](https://github.com/PrefectHQ/prefect/pull/1375)
- Introduce new Flow health checks prior to Cloud deployment - [1372](https://github.com/PrefectHQ/prefect/issues/1372)
- Improve logging functionality to include tracebacks - [1374](https://github.com/PrefectHQ/prefect/issues/1374)
- Improve CLI user experience while working with Cloud - [1384](https://github.com/PrefectHQ/prefect/pull/1384/)
- Users can now create projects from the CLI - [1388](https://github.com/PrefectHQ/prefect/pull/1388)
- Add a health check to confirm that serialized flows are valid prior to Cloud deploy - [1397](https://github.com/PrefectHQ/prefect/pull/1397)
- Add `task_slug`, `flow_id`, and `flow_run_id` to context - [1405](https://github.com/PrefectHQ/prefect/pull/1405)
- Support persistent `scheduled_start_time` for scheduled flow runs when run locally with `flow.run()` - [1418](https://github.com/PrefectHQ/prefect/pull/1418), [#1429](https://github.com/PrefectHQ/prefect/pull/1429)
- Add `task_args` to `Task.map` - [1390](https://github.com/PrefectHQ/prefect/issues/1390)
- Add auth flows for `USER`-scoped Cloud API tokens - [1423](https://github.com/PrefectHQ/prefect/pull/1423)
- Add `AzureResultHandler` for handling results to / from Azure Blob storage containers - [1421](https://github.com/PrefectHQ/prefect/pull/1421)
- Add new configurable `LocalDaskExecutor` - [1336](https://github.com/PrefectHQ/prefect/issues/1336)
- Add CLI commands for working with Prefect Cloud auth - [1431](https://github.com/PrefectHQ/prefect/pull/1431)

Task Library

- Add new `SnowflakeQuery` task for using snowflake data warehouse - [1113](https://github.com/PrefectHQ/prefect/issues/1113)

Fixes

- Fix issue with Docker storage not respecting user-provided image names - [1335](https://github.com/PrefectHQ/prefect/pull/1335)
- Fix issue with local retries in Cloud not always running in-process - [1348](https://github.com/PrefectHQ/prefect/pull/1348)

Deprecations

- Rename `SynchronousExecutor` as `LocalDaskExecutor` - [1434](https://github.com/PrefectHQ/prefect/pull/1434)

Breaking Changes

- Rename `CloudEnvironment` to `DaskKubernetesEnvironment` - [1250](https://github.com/PrefectHQ/prefect/issues/1250)
- Remove unused `queue` method from all executors - [1434](https://github.com/PrefectHQ/prefect/pull/1434)

Contributors

- [Alex Kravetz](http://github.com/akravetz)