Prefect

This release includes a few very exciting enhancements! 🪄

Access default work pool configurations in an air-gapped environment 🌐
Those who run Prefect server in an environment where arbitrary outbound internet traffic is not allowed were previously unable to retrieve up-to-date default work pool configurations (via the UI or otherwise). You can now access the worker metadata needed to access the corresponding work pool configurations in your server even in such an air-gapped environment. Upon each release of `prefect`, the most recent version of this worker metadata will be embedded in the `prefect` package so that it can be used as a fallback if the outbound call to retrieve the real-time metadata fails.

Introducing conditional task retries for enhanced workflow control 🔁
In this release, we're excited to introduce the ability to conditionally retry tasks by passing in an argument to `retry_condition_fn` in your task decorator, enabling more nuanced and flexible retry mechanisms. This adds a significant level of control and efficiency, particularly in handling complex or unpredictable task outcomes. For more information on usage, check out our [docs](https://docs.prefect.io/latest/concepts/tasks/#custom-retry-behavior)!

Other enhancements and fixes 🛠️
We've also added the ability to open your current Prefect Cloud workspace in the browser from the CLI, display work queue status details via CLI, and much more! See the [release notes](https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#release-21413) for details.

Contributors 🧑‍🤝‍🧑
A big thanks to dominictarro and ConstantinoSchillebeeckx for contributing enhancements to this release, as well as to yifanmai, who made their first contribution.

This release brings some exciting enhancements and fixes!

Enhanced UI Customization 📆

- Now featuring relative time spans like "Past 7 days" for better insight into current and upcoming flow runs across the UI.
- Flow Runs Page: Enhanced with customizable date and time filters. Save custom filters with specific date ranges for tailored views
- Flows Page: Updated with similar date and time filtering capabilities

See a demonstration here!
[](https://www.loom.com/share/95113969257d4cffa48ad13f943f950f?sid=b20bc27c-0dc2-40be-a627-a2148942c427)

Get type-checked input from humans in the loop

Human-in-the-loop flows just got an upgrade. You can now pause or suspend a flow and wait for type-checked input. To get started, declare the structure of the input data using a Pydantic model, and Prefect will render a form dynamically in the UI when a human resumes the flow. Form validation will ensure that the data conforms to your Pydantic model, and your flow will receive tbe input.

<img width="472" alt="image" src="https://github.com/PrefectHQ/prefect/assets/97182/ac743557-e872-4b48-a61e-c74c95e076f0">

Prefect's new `RunInput` class powers this experience. `RunInput` is a subclass of Pydantic's `BaseModel`. Here's an example of a `RunInput` that uses dates, literals, and nested Pydantic models to show you what's possible:

python
class Person(RunInput):
first_name: str
last_name: str
birthday: datetime.date
likes_tofu: bool
age: int = Field(gt=0, lt=150)
shirt_size: Literal[ShirtSize.SMALL, ShirtSize.MEDIUM, ShirtSize.LARGE,
ShirtSize.XLARGE]
shirt_color: Literal["red", "blue", "green"]
preferred_delivery_time: datetime.datetime
shipping_address: ShippingAddress
billing_address: BillingAddress | SameAsShipping = Field(
title="", default_factory=SameAsShipping
)


Check out our [guide on how to create human-in-the-loop flows](https://docs.prefect.io/latest/guides/creating-human-in-the-loop-workflows/) to learn more!

Enhancements and Fixes 🛠️
- Fixed environment variable parsing during deployment and prompt and role assignment in the `ContainerInstanceProvisioner`
- Enhanced dashboard header to adapt to various tag and date input sizes
- Addressed an error encountered when deploying flows with options

See the [release notes](https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#release-21412) for details!

🪄 ✨ Customize resource names when provisioning infrastructure for push work pools

In the past few releases, we've added the ability to provision infrastructure for push work pools via the CLI. This release adds the ability to customize the name of the resources created in your cloud environment when provisioning infrastructure for push work pools so you can follow your organization's naming conventions.

To customize your resource names when provisioning infrastructure for a push work pool, follow the interactive prompts:

bash
? Proceed with infrastructure provisioning with default resource names? [Use arrows to move; enter to select]
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃ Options: ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ │ Yes, proceed with infrastructure provisioning with default resource names │
│ > │ Customize resource names │
│ │ Do not proceed with infrastructure provisioning │
└────┴───────────────────────────────────────────────────────────────────────────┘
? Please enter a name for the resource group (prefect-aci-push-pool-rg): new-rg
? Please enter a name for the app registration (prefect-aci-push-pool-app): new-app
? Please enter a prefix for the Azure Container Registry (prefect): newregistry
? Please enter a name for the identity (used for ACR access) (prefect-acr-identity): new-identity
? Please enter a name for the ACI credentials block (new-work-pool-push-pool-credentials): new-aci-block
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ Provisioning infrastructure for your work pool new-work-pool will require: │
│ │
│ Updates in subscription: Azure subscription 1 │
│ │
│ - Create a resource group in location: eastus │
│ - Create an app registration in Azure AD: new-app │
│ - Create/use a service principal for app registration │
│ - Generate a secret for app registration │
│ - Create an Azure Container Registry with prefix newregistry │
│ - Create an identity new-identity to allow access to the created registry │
│ - Assign Contributor role to service account │
│ - Create an ACR registry for image hosting │
│ - Create an identity for Azure Container Instance to allow access to the registry │
│ │
│ Updates in Prefect workspace │
│ │
│ - Create Azure Container Instance credentials block: new-aci-block │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
Proceed with infrastructure provisioning? [y/n]: y
Creating resource group
Resource group 'new-rg' created successfully
Creating app registration
App registration 'new-app' created successfully
Generating secret for app registration
Secret generated for app registration with client ID '03923189-3151-4acd-8d59-76483752cd39'
Creating ACI credentials block
ACI credentials block 'new-aci-block' created in Prefect Cloud
Assigning Contributor role to service account
Service principal created for app ID '25329389-3151-4acd-8d59-71835252cd39'
Contributor role assigned to service principal with object ID '483h4c85-4a8f-4fdb-0394-bd0f0b1202d0'
Creating Azure Container Registry
Registry created
Logged into registry newregistry1702538242q2z2.azurecr.io
Creating identity
Identity 'new-identity' created
Provisioning infrastructure. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Your default Docker build namespace has been set to 'newregistry1702538242q2z2.azurecr.io'.
Use any image name to build and push to this registry by default:

╭─────────────────────────── example_deploy_script.py ───────────────────────────╮
│ from prefect import flow │
│ from prefect.deployments import DeploymentImage │
│ │
│ │
│ flow(log_prints=True) │
│ def my_flow(name: str = "world"): │
│ print(f"Hello {name}! I'm a flow running on an Azure Container Instance!") │
│ │
│ │
│ if __name__ == "__main__": │
│ my_flow.deploy( │
│ name="my-deployment", │
│ work_pool_name="my-work-pool", │
│ image=DeploymentImage( │
│ name="my-image:latest", │
│ platform="linux/amd64", │
│ ) │
│ ) │
╰────────────────────────────────────────────────────────────────────────────────╯
Infrastructure successfully provisioned for 'new-work-pool' work pool!
Created work pool 'new-work-pool'!


Using a push work pool with automatic infrastructure provisioning is a great way to get started with a production-level Prefect set up in minutes! Check out our [push work pool guide](https://docs.prefect.io/latest/guides/deployment/push-work-pools/) for step-by-step instructions on how to get started!

See the following pull requests for implementation details:
- https://github.com/PrefectHQ/prefect/pull/11407
- https://github.com/PrefectHQ/prefect/pull/11381
- https://github.com/PrefectHQ/prefect/pull/11412

📆 🕐 An updated date time input on the workspace dashboard

We've added a new date and time filter to the workspace dashboard that gives greater control over the dashboard. You can now filter by days, hours, and even minutes. You can also specify a specific date and time range to filter by. You can also go backwards and forwards in time using that time window, for example, you can scroll through by hour.

See it in action!
[
](https://www.loom.com/share/ca099d3792d146d08df6fcd506ff9eb2?sid=70797dda-6dc6-4fe6-bf4a-a9df2a0bf230)

See the following pull requests for implementation details:
- https://github.com/PrefectHQ/prefect-ui-library/pull/1937
- https://github.com/PrefectHQ/prefect-design/pull/1048


💥 Enhancements
- Add the ability to publish `KubernetesJob` blocks as work pools — https://github.com/PrefectHQ/prefect/pull/11347
- Add setting to configure a default Docker namespace for image builds — https://github.com/PrefectHQ/prefect/pull/11378
- Add the ability to provision an ECR repository for ECS push work pools — https://github.com/PrefectHQ/prefect/pull/11382
- Add ability to provision an Artifact Registry repository for Cloud Run push work pools — https://github.com/PrefectHQ/prefect/pull/11399
- Add ability to provision an Azure Container Registry for Azure Container Instance push work pools — https://github.com/PrefectHQ/prefect/pull/11387
- Add support for `is_schedule_active` to `flow.deploy` and `flow.serve` — https://github.com/PrefectHQ/prefect/pull/11375
- Allow users to select relative and fixed date ranges to filter the dashboard — https://github.com/PrefectHQ/prefect/pull/11406
- Add support for arbitrary sink types to `prefect.utilities.processutils.stream_text` — https://github.com/PrefectHQ/prefect/pull/11298
- Update the Prefect UI deployments page to add run activity and separate out the deployment and flow names — https://github.com/PrefectHQ/prefect/pull/11394
- Update Prefect UI workspace dashboard filters to use new date range - https://github.com/PrefectHQ/prefect-ui-library/pull/1937

🧰 Fixes
- Fix bug where a pause state reused an existing state ID — https://github.com/PrefectHQ/prefect/pull/11405

🧪 Experimental
- Build out API for creating/reading/deleting flow run inputs — https://github.com/PrefectHQ/prefect/pull/11363
- Integrate flow run input and schema/response mechanics into pause/suspend — https://github.com/PrefectHQ/prefect/pull/11376
- Add typing overloads for pause/suspend methods — https://github.com/PrefectHQ/prefect/pull/11403
- Use bytes for `value` in `create_flow_run_input` — https://github.com/PrefectHQ/prefect/pull/11421
- Validate run input when resuming flow runs — https://github.com/PrefectHQ/prefect/pull/11396
- Run existing deployments via the `Runner` webserver — https://github.com/PrefectHQ/prefect/pull/11333

🧠 Documentation
- Add instructions for automatic infrastructure provisioning to the push work pools guide — https://github.com/PrefectHQ/prefect/pull/11316
- Fix broken links in states concept doc and daemonize guide — https://github.com/PrefectHQ/prefect/pull/11374
- Update agent upgrade guide to include `flow.deploy` and examples — https://github.com/PrefectHQ/prefect/pull/11373
- Update block document names in Moving Data guide — https://github.com/PrefectHQ/prefect/pull/11386
- Rename `Guides` to` How-to Guides` — https://github.com/PrefectHQ/prefect/pull/11388
- Add guide to provision infrastructure for existing push work pools — https://github.com/PrefectHQ/prefect/pull/11365
- Add documentation for required permissions for infrastructure provisioning — https://github.com/PrefectHQ/prefect/pull/11417
- Add docs for managed execution open beta — https://github.com/PrefectHQ/prefect/pull/11397, https://github.com/PrefectHQ/prefect/pull/11426, and https://github.com/PrefectHQ/prefect/pull/11425

🪄 ✨ Azure Container Instance push pool infrastructure provisioning via the CLI

We're introducing an enhancement to the Azure Container Instance push pool experience. You can now conveniently provision necessary Azure infrastructure with the `--provision-infra` flag during work pool creation, automating the provisioning of various Azure resources essential for ACI push pools, including resource groups, app registrations, service accounts, and more.

To provision Azure resources when creating an ACI push pool:

bash
❯ prefect work-pool create my-work-pool --provision-infra --type azure-container-instance:push
? Please select which Azure subscription to use: [Use arrows to move; enter to select]
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃ Name ┃ Subscription ID ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ │ Engineering │ 123 │
│ > │ Azure subscription 1 │ 234 │
└────┴──────────────────────┴──────────────────────────────────────┘
╭───────────────────────────────────────────────────────────────────────────────────────╮
│ Provisioning infrastructure for your work pool my-work-pool will require: │
│ │
│ Updates in subscription Azure subscription 1 │
│ │
│ - Create a resource group in location eastus │
│ - Create an app registration in Azure AD │
│ - Create a service principal for app registration │
│ - Generate a secret for app registration │
│ - Assign Contributor role to service account │
│ - Create Azure Container Instance │
│ │
│ Updates in Prefect workspace │
│ │
│ - Create Azure Container Instance credentials block aci-push-pool-credentials │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────╯
Proceed with infrastructure provisioning? [y/n]: y
Creating resource group
Provisioning infrastructure... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--Resource group 'prefect-aci-push-pool-rg' created in location 'eastus'
Creating app registration
Provisioning infrastructure... ━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20% -:--:--App registration 'prefect-aci-push-pool-app' created successfully
Generating secret for app registration
Provisioning infrastructure... ━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━ 40% 0:00:06Secret generated for app registration with client ID 'abc'
ACI credentials block 'aci-push-pool-credentials' created
Assigning Contributor role to service account...
Provisioning infrastructure... ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 60% 0:00:06Contributor role assigned to service principal with object ID 'xyz'
Creating Azure Container Instance
Provisioning infrastructure... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 80% 0:00:04Container instance 'prefect-acipool-container' created successfully
Creating Azure Container Instance credentials block
Provisioning infrastructure... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Infrastructure successfully provisioned for 'my-work-pool' work pool!
Created work pool 'my-work-pool'!


This marks a step forward in Prefect's Azure capabilities, offering you an efficient and streamlined process for leveraging Azure Container Instances to execute their workflows.

See the following pull request for implementation details:
— https://github.com/PrefectHQ/prefect/pull/11275

🪄 🎩 🐰 Introducing the `provision-infra` sub-command for enhanced push work pool management
This enhancement allows you to directly provision infrastructure for existing push work pools. Rather than recreating a work pool, you can provision necessary infrastructure and
update the existing work pool base job template with the following command:

bash
❯ prefect work-pool provision-infra my-work-pool
? Please select which Azure subscription to use: [Use arrows to move; enter to select]
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃ Name ┃ Subscription ID ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ │ Engineering │ 13d │
│ > │ Azure subscription 1 │ 6h4 │
└────┴──────────────────────┴──────────────────────────────────────┘
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Provisioning infrastructure for your work pool my-work-pool will require: │
│ │
│ Updates in subscription Azure subscription 1 │
│ │
│ - Create a resource group in location eastus │
│ - Create an app registration in Azure AD prefect-aci-push-pool-app │
│ - Create/use a service principal for app registration │
│ - Generate a secret for app registration │
│ - Assign Contributor role to service account │
│ - Create Azure Container Instance 'aci-push-pool-container' in resource group prefect-aci-push-pool-rg │
│ │
│ Updates in Prefect workspace │
│ │
│ - Create Azure Container Instance credentials block aci-push-pool-credentials │
│ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Proceed with infrastructure provisioning? [y/n]: y
...


This PR bolsters support for efficient work pool management across diverse cloud environments, delivering a tool for seamless infrastructure setup.

See the following pull request for implementation details:
- https://github.com/PrefectHQ/prefect/pull/11341
- https://github.com/PrefectHQ/prefect/pull/11355

💥 Enhancements
- Add a `suspend_flow_run` method to suspend a flow run — https://github.com/PrefectHQ/prefect/pull/11291
- Limit the displayed work pool types when `--provision-infra` is used to only show supported work pool types - https://github.com/PrefectHQ/prefect/pull/11350
- Add the ability to publish `Infrastructure` blocks as work pools — https://github.com/PrefectHQ/prefect/pull/11180
- Add the ability to publish `Process` blocks as work pools — https://github.com/PrefectHQ/prefect/pull/11346
- Add a Prefect Cloud event stream subscriber — https://github.com/PrefectHQ/prefect/pull/11332
- Enable storage of key/value information associated with a flow run — https://github.com/PrefectHQ/prefect/pull/11342
- Delete flow run inputs when the corresponding flow run is delete — https://github.com/PrefectHQ/prefect/pull/11352

🧰 Fixes
- Fix the `read_logs` return type to be `List[Log]` — https://github.com/PrefectHQ/prefect/pull/11303
- Fix an issue causing paused flow runs to become stuck in the `Paused` state — https://github.com/PrefectHQ/prefect/pull/11284

🧠 Documentation
- Combine troubleshooting pages — https://github.com/PrefectHQ/prefect/pull/11288
- Add Google Cloud Run V2 option to Serverless guide — https://github.com/PrefectHQ/prefect/pull/11304
- Add `suspend_flow_run` to flows documentation — https://github.com/PrefectHQ/prefect/pull/11300
- Add `work queues` tag to work pools concept page — https://github.com/PrefectHQ/prefect/pull/11320
- Add missing Python SDK CLI items to the docs — https://github.com/PrefectHQ/prefect/pull/11289
- Clarify SCIM + service accounts handling — https://github.com/PrefectHQ/prefect/pull/11343
- Update the work pool concept document — https://github.com/PrefectHQ/prefect/pull/11331

🪄 Automatic infrastructure provisioning for ECS push work pools

Following the introduction of [automatic project configuration for Cloud Run push pools](https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#automatic-project-configuration-for-cloud-run-push-work-pools) last week, we've added the ability to automatically provision infrastructure in your AWS account and set up your Prefect workspace to support a new ECS push pool!

You can create a new ECS push work pool and provision infrastructure in your AWS account with the following command:

bash
prefect work-pool create --type ecs:push --provision-infra my-pool


Using the `--provision-infra` flag will automatically set up your default AWS account to be ready to execute flows via ECS tasks:


╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Provisioning infrastructure for your work pool my-work-pool will require: │
│ │
│ - Creating an IAM user for managing ECS tasks: prefect-ecs-user │
│ - Creating and attaching an IAM policy for managing ECS tasks: prefect-ecs-policy │
│ - Storing generated AWS credentials in a block │
│ - Creating an ECS cluster for running Prefect flows: prefect-ecs-cluster │
│ - Creating a VPC with CIDR 172.31.0.0/16 for running ECS tasks: prefect-ecs-vpc │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Proceed with infrastructure provisioning? [y/n]: y
Provisioning IAM user
Creating IAM policy
Generating AWS credentials
Creating AWS credentials block
Provisioning ECS cluster
Provisioning VPC
Creating internet gateway
Setting up subnets
Setting up security group
Provisioning Infrastructure ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Infrastructure successfully provisioned!
Created work pool 'my-pool'!


If you have yet to try using an ECS push pool, now is a great time!

If you use Azure, don't fret; we will add support for Azure Container Instances push work pools in a future release!

See the following pull request for implementation details:
— https://github.com/PrefectHQ/prefect/pull/11267


🚀 Enhancements
- Make flows list on flows page a scannable table and standardize list headers throughout the app — https://github.com/PrefectHQ/prefect/pull/11274

🛠️ Fixes
- Fix `.serve` crashes due to process limiter — https://github.com/PrefectHQ/prefect/pull/11264
- Fix URL formatting in `GitRepository` when using provider-specific git credentials blocks — https://github.com/PrefectHQ/prefect/pull/11282
- Prevent excessively escaping the Windows executable — https://github.com/PrefectHQ/prefect/pull/11253

This release is a follow-up to 2.14.7 which never made it to PyPI because of an issue with our Github workflow.

Documentation
- Fix broken docs link in serverless worker documentation — https://github.com/PrefectHQ/prefect/pull/11269

**All changes**: https://github.com/PrefectHQ/prefect/compare/2.14.7...2.14.8