Descarteslabs

Workflows

- Fixed a bug causing `descarteslabs.workflows.map.geocontext()` to fail with an import error. This problem
also affected the autoscaling feature of workflows map layers.

Catalog/Scenes/Raster

- Fixed a bug causing downloads of single-band images to fail when utilizing rasterio.

Catalog

- Catalog V2 is now fully supported on the AWS platform, including user ingest.
- Catalog V2 has been enhanced to provide substantially all the functionality of the Scenes API. The `Image` class now
includes methods such as `ndarray` and `download`. A new `ImageCollection` class has been added, mirroring `SceneCollection`.
The various `Search` objects now support a new `collect` method which will return appropriate `Collection` types
(e.g. `ProductCollection`, `BandCollection`, and of course `ImageCollection`). Please see the updated Catalog V2
guide and API documentation for more details.
- Previously, the internal implementation of the `physical_range` attribute on various band types was inconsistent with
that of `data_range` and `display_range`. It has now been made consistent, which means it will either not be set,
or will contain a 2-tuple of float values. It is no longer possible to explicitly set it to `None`.
- Access permissions for bands and images are now managed directly by the product. The `readers`, `writers`, and
`owners` attributes have been removed from all the `*Band` classes as well as the `Image` class. Also the
`Product.update_related_objects_permissions` and `Product.get_update_permissions_status` methods have been removed
as these are no longer necessary or supported.
- All searches for bands (other than derived bands) and images must specify one or more product ids in the filtering.
This requirement can be met by using the `bands()` and `images()` methods of a product to limit the search to that
product, or through a `filter(properties.product_id==...)` clause on the search.
- Products have a new `product_tier` attribute, which can only be set or modified by privileged users.
- The `Image.upload_ndarray` will now accept either an ndarray or a list of ndarrays, allowing multiple files per image.
The band definitions for the product must correspond to the order and properties of the multiple ndarrays.

Scenes

- With the addition of the Scenes functionality to Catalog V2, you are strongly encouraged to migrate your Scenes-based
code to use Catalog V2 instead. Scenes will be deprecated in a future release. Some examples of migrating from Scenes
to Catalog V2 are included in the Catalog V2 guide. In the meantime the Scenes API has been completely reimplemented
to use Catalog V2 under the hood. From a user perspective, existing code using the Scenes API should continue to
function as normal, with the exception of a few differences around some little-used dark corners of the API.
- The Scenes `search_bands` now enforces the use of a non-empty `products=` parameter value. This was previously
documented but not enforced.

Metadata

- With the addition of the Scenes functionality to Catalog V2, you are strongly encouraged to migrate your Metadata-based
code to use Catalog V2 instead. Metadata will be deprecated in a future release.
- As with Catalog and Scenes, one or more products must now be specified when searching for bands or images.

Raster

- The Raster client API now requires a `bands=` parameter for all rastering operations, such as `raster`, `ndarray`
and `stack`. It no longer defaults to all bands defined on the product.

DLTile

- An off-by-1/2-pixel problem was identified in the coordinate transforms underlying
`DLTile.rowcol_to_latlon` and `DLTile.latlon_to_rowcol`. The problem has been corrected,
and you can expect to see slight differences in the results of these two methods.

REST Clients

- All the REST client types, such as `Metadata` and `Raster`, now support `get_default_client()` and `set_default_client()`
instances. This functionality was previously limited to the Catalog V2 `CatalogClient`. Whenever such a client is required,
the client libraries use `get_default_client()` rather than using the default constructor. This makes it easy to
comprehensively redirect the library to use a specially configured client when necessary.

Geo package

- The `GeoContext` types that originally were part of the Scenes package are now available in the new `descarteslabs.geo` package,
with no dependencies on Scenes. This is the preferred location from which to import these classes.

Utils package

- The `descarteslabs.utils` package, added in the previous release for the AWS client only, now exists in the GCP client
as well, and is the preferred location to pick up the `DotDict` and `DotList` classes, the `display` and `save_image` functions,
and the `Properties` class for property filtering in Catalog V2.
- The `display` method now has added support for multi-image plots, see the API documentation for the `figsize`, `nrows`,
`ncols` and `layout_direction` parameters.

Property filtering

- The `property_filtering.GenericProperties` class has been replaced with `property_filtering.Properties`, but remains
for back compatibility.
- Property filters now support `isnull` and `isnotnull` operations. This can be very useful for properties which may or
may not be present, e.g. `properties.cloud_fraction.isnull | properties.cloud_fraction <= 0.2`.

Configuration and Authentication

- The `Config` exceptions `RuntimeError` and `KeyError` were changed to `ConfigError` exceptions
from `descarteslabs.exceptions`.
- `Auth` now retrieves its URL from the `Config` settings. If no valid configuration can be found,
it reverts to the commercial service (`https://iam.descarteslabs.com`).

General
- Dependencies for the descarteslabs library have been updated, but remain constrained to continue to support Python 3.7.
- Numerous bug fixes.

Installation

- The extra requirement options have changed. There are four extra requirement options now, `visualization`, `tables`,
`complete`, and `tests`. `visualization` pulls in extra requirements to support operating in a Jupyter notebook or
environment, enabling interactive maps and graphical displays. It is not required for operating in a "headless"
manner. `tables` pulls in extra requirements to support the `Tables` client. `complete` is the combination of
`visualization` and `tables`. `tests` pulls in extra requirements for running the tests. As always,
`pip install 'descarteslabs[complete]'` will install a fully enabled client.

Configuration

- The Descartes Labs client now supports configuration to support operating in different environments. By default,
the client will configure itself for standard usage against the GCP platform (`"gcp-production"`), except in the case of AWS Marketplace users, for whom
the client will configure itself against the AWS platform (`"aws-production"`).
Alternate environments can be configured by setting the `DESCARTESLABS_ENV` environment variable before starting python, or by using a prelude like

from descarteslabs.config import Settings
Settings.select_env("environment-name")

before any other imports of any part of the descarteslabs client package.
- The new AWS Enterprise Accelerator release currently includes only Auth, Configuration
and the Scenes client.

Auth and Exceptions

- The `descarteslabs.client.auth` package has moved to `descarteslabs.auth`. It is now imported
into the original location at `descarteslabs.client.auth` to continue to work with existing
code, but new code should use the new location.
- The `descarteslabs.client.exceptions` module has moved to `descarteslabs.exceptions`. It is
now imported into the original location at `descarteslabs.client.exceptions` to continue to
work with existing code, but new code should use the new location.

Scenes

- Fixed an issue in `scenes.DLTile.from_shape` where there would be incomplete coverage of certain geometries. The function may now return more tiles than before.
- Added support for the new `all_touched` parameter to the different `GeoContext` types. Default behavior remains the same
as always, but if you set `all_touched=True` this communicates to the raster service that you want the image(s) rastered
using GDAL's `CUTLINE_ALL_TOUCHED` option which will change how source pixels are mapped to output pixels. This mode is
only recommended when using an AOI which is smaller than the source imagery pixel resolution.
- The DLTile support has been fixed to avoid generating gaps when tiling regions that span
a large distance north-to-south and straddle meridians which are boundaries between
UTM zones. So methods such as `DLTile.from_shape` may return more tiles than previously,
but properly covering the region.
- Added support for retrieving products and bands.
- Methods added: `get_product`, `get_band`, `get_derived_band`, `search_products`,
`search_bands`, `search_derived_bands`.
- Disallows search without `products` parameter.
- Scaling support has been enhanced to understand processing levels for newer products. The
`Scene.scaling_parameters` and `SceneCollection.scaling_parameters` methods now accept
a `processing_level` argument, and this will be factored in to the determination of
the default result data type and scaling for all rastering operations such as `Scene.ndarray`
and `SceneCollection.mosaic`.
- If the user provides the `rasterio` package (which implies providing GDAL), then rasterio
will be used to save any downloaded images as GeoTIFF, allowing for the use of compression.
Otherwise, by default the `tifffile` support will be used to generate the GeoTIFF files
but compression is not supported in this mode.
- As the Places client has been deprecated, so has any use of the `place=` parameter supported
by several of the Scenes functions and methods.

Catalog

- (Core users only) Added support for specifying the image index to use when creating a new `Product`.
- Added support for defining per-processing-level `data_type`, `data_range`, `display_range`
and `physical_range` properties on processing level steps.

Discover

- Added support for filtering `Assets` by type and name fields.
- Supported filter types `blob`, `folder`, `namespace`, `sym_link`, `sts_model`, and `vector`. Specifying multiple types will find assets matching any given type.
- The name field supports the following wildcards:
- `*` matches 0 or more of any character.
- `?` matches 1 of any character.
- Find assets matching type of `blob` and having a display name of `file name.json` or `file2name.txt` but **not** `filename.json`:
- `Discover().list_assets("asset/namespace/org:some_org", filters="type=blob&name=file?name.*")`
- `Discover().list_assets("asset/namespace/org:some_org", filters=AssetListFilter(type=AssetType.BLOB, name="file?name.*"))`
- Find assets of type `blob` or `vector`:
- `Discover().list_assets("asset/namespace/org:some_org", filters="type=blob,vector")`
- `Discover().list_assets("asset/namespace/org:some_org", filters=AssetListFilter(type=[AssetType.BLOB, AssetType.VECTOR], name="file?name.*"))`

Metadata

- `Metadata.products` and `Metadata.available_products` now properly implement paging so that
by default, a DotList containing every matching product accessible to the user is returned.

Raster

- If the user provides the `rasterio` package (which implies providing GDAL), then rasterio
will be used to save any downloaded images as GeoTIFF, allowing for the use of compression.
Otherwise, by default the `tifffile` support will be used to generate the GeoTIFF files
but compression is not supported in this mode.

Tables

- Fixed an issue that caused a user's schema to be overwritten if they didn't provide a primary
key on table creation.
- Now uses Discover backend filtering for `list_tables()` instead of filtering on the client to
improve performance.
- `list_tables()` now supports filtering tables by name
- `Tables.list_tables(name="Test*.json")`

Tasks

- New Tasks images for this release bump the versions of several dependencies, please see
the Tasks guide for detailed lists of dependencies.

Workbench

- The new Workbench release bumps the versions of several dependencies.

Workflows

- Added support for the new `all_touched` parameter to the different `GeoContext` types.
See description above under `Scenes`.

General

- The Places client has been deprecated, and use thereof will generate a deprecation warning.
- The older Catalog V1 client has been deprecated, and use thereof will generate a deprecation
warning. Please use the Catalog V2 client in its place.
- Documentation has been updated to include the `AWS Enterprise Accelerator" release.
- With Python 2 far in the rearview mirror, the depedencies on the `six` python package have
been removed throughout the library, the distribution and all tasks images.

Python Versions Supported

- Added support for Python 3.9.
- Removed support for Python 3.6 which is now officially End Of Life.

Workflows
- Added support for organizational sharing. You can now share using the `Organization` type:
- `workflows.add_reader(Organization("some_org"))`

Discover

- Added support for organizational sharing. You can now share using the `Organization` type:
- `asset.share(with_=Organization("some_org"), as_="Viewer")`
- Allow user to list their organization's namespace.
- `Discover().list_asset("asset/namespace/org:some_org")`
- Allow user to list their organization's users.
- `Discover().list_org_users()`

Tables - Added
- Added an **alpha** Tables client. The Tables module lets you organize, upload, and query tabular data and vector geometries. As an alpha release, we reserve the right to modify the Tables client API without any guarantees about backwards compatibility. See the [Tables API](https://docs.descarteslabs.com/descarteslabs/tables/readme.html) and [Tables Guide](https://docs.descarteslabs.com/guides/tables.html) documentation for more details.

Scenes
- Added the `progress=` parameter to the various rastering methods such as `Scene.ndarray`,
`Scene.download`, `SceneCollection.mosaic`, `SceneCollection.stack`, `SceneCollection.download`
and `SceneCollection.download_mosaic`. This can be used to enable or disable the display
of progress bars.

Raster

- Fixed a bug preventing retry-able errors (such as a 429) from being retried.

Catalog

- Allow retrieving Attribute as a class attribute. It used to raise an exception.

Scenes

- Fixed a bug preventing the user from writing JPEG files with smaller than 256x256 tiles.
- Allow specifying a `NoData` value for non-JPEG GeoTIFF files.
- Include band description metadata in created GeoTIFF files.
- Support scaling parameters as lists as well as tuples.
- Add caching of band metadata to drastically reduce the number of metadata queries when creating `SceneCollections`.
- `DLTiles.from_shape` was failing to handle shape objects implementing the `__geo_interface__` API, most notably several of the Workflows `GeoContext` types. These now work as expected.
- Certain kinds of network issues could read to rastering operations raising an `IncompleteRead` exception. This is now correctly caught and retried within the client library.

Tasks

- Users can now use `descarteslabs.tasks.update_credentials()` to update their task credentials in case they became outdated.

Workflows

- We have introduced a hard limit of 120 as the number of outstanding Workflows compute jobs that a single user can have. This limit exists to minimize situations in which a user is unable to complete jobs in a timely manner by ensuring resources cannot be monopolized by any individual user. The API that backs the calls to `compute` will return a `descarteslabs.client.grpc.exceptions.ResourceExhausted` error if the caller has too many outstanding jobs. Prior to this release (1.9.0), these failures would be retried up to some small retry limit. With the latest client release however, the client will fail without retrying on an HTTP 429 (rate limit exceeded) error. For users with large (non-interactive) workloads who don’t mind waiting, we added a new `num_retries` parameter to the `compute` function; when specified, the client will handle any 429 errors and retry up to `num_retries` times.
- Workflows is currently optimized for interactive use cases. If you are submitting large numbers of long-running Workflows compute jobs with `block=False`, you should consider using Tasks and Scenes rather than the Workflows API.
- Removed `ResourceExhausted` exceptions from the list of exceptions we automatically catch and retry on for `compute` calls.

Documentation

- Lots of improvements, additions, and clarifications in the API documentation.