Descarteslabs

Vector

- Fixed a bug in `Table.visualize()` that was causing it to fail.

Due to a number of breaking changes, the version has been bumped to 3.0.0. However, the vast majority
of typical use patterns in typical user code will not require changes. Please review the specifics
below.

Catalog

- The `tags` attributes on Catalog objects can now contain up to 32 elements, each up to 1000 characters long.
But why would you even want to go there?
- *Breaking Change*: Derived bands, never supported in the AWS environment and catalog products, have been
removed.
- The new `Blob.delete_many` method may be used to delete large numbers of blobs efficiently.
- The `Blob.get_or_create` method didn't allow supplying `storage_type`, `namespace`, or `name` parameters.
Now it works as expected, either returning a saved Blob from the Catalog, or an unsaved blob that
you can use to upload and save its data.
- Image methods `ndarray` and `download` no longer pass the image's default geocontext geometry as a cutline.
This is to avoid problems when trying to raster a complete single image in its native CRS and resolution
where imperfect geometries (due to a simplistic projection to EPSG:4326) can cause some boundary pixels
to be masked. When passing in an explicit `GeoContext` to these methods, consider whether any cutline
geometry is required or not, to avoid these issues.

Compute

- `Function` and `Job` objects now have a new `environment` attribute which can be used to define environment
variables for the jobs when they are run.
- *Breaking Change*: The `Function.map` method previously had no bound on how many jobs could be created at one time.
This led to operational problems with very large numbers of jobs. Now it submits jobs in batches (up to 1000
jobs per batch) to avoid request timeouts, and is more robust on retryable errors so that duplicate jobs are not
submitted accidently. There is still no bound on how many jobs you may create with a single call to `Function.map`.
Additionally, since it is possible that some jobs may be successfully submitted, and others not, the return
value, while still behaving as a list of `Job`s, is now a `JobBulkCreateResult` object which has a `is_success`
and an `error` property which can be used to determine if all submissions were successful, what errors may
have occurred, and what jobs have actually been created. Only if the first batch fails hard will the method
raise an exception.
- The `Job.statistics` member is now typed as a `JobStatistics` object.
- The efficiency of deleting many jobs at once has been significantly improved using `Function.delete` and
`Function.delete_jobs`. It is still possible to encounter request timeouts with very large numbers of jobs;
workarounds are now documented in the API documentation for the `Function.delete_jobs` method.
- The `ComputeClient.check_credentials` method has been added, so that the client can determine if valid
user credentials have already been registered with the Compute service.

Vector

- The Vector client library, previously available as the `descarteslabs-vector` package on PyPI, has
now been integrated into the Descartes Labs Python Client (this package). It should no longer be
installed separately.
- Visualization support (`ipyleaflet.Map`) is enabled when `ipyleaflet` is available. It is not
installed by default, but can be installed manually, or by installing the `descarteslabs` python
client with the `viz` extra (e.g. `pip install descarteslabs[viz]`). Note that in order to be
compatible with jupyterlab notebooks, the `visualize()` method no longer returns the layer, it
just adds it to the supplied map.
- The Vector package now has a `VectorClient` API client, with the usual support for `get_default_client()`
and `set_default_client()`. Most constructors and methods now accept an optional `client=` parameter
if you need to use something other than the default client.
- Configuration is now accomplished using the standard `descarteslabs.config` package. In particular,
the `vector_url` setting is used to specify the default Vector host. The `VECTOR_API_HOST` environment
variable is no longer consulted.
- Vector client methods now raise standard `descarteslabs.exceptions` Exception classes rather than
the `descarteslabs.vector.vector_exceptions` classes of the old client.
- The `is_spatial=` parameter previously accepted by many methods and functions is now deprecated
and ignored. It is not required because existing type information always determines if an operation
is spatial or not. Warnings will be generated if it is used.
- Be advised that feature upload and download (query) do not currently support or impose any limits,
and thus allow operations so large and slow that timeouts or other failures may occur. A future
version will implement limits and batching, so that large operations can be supported reliably.
Until then, the user may wish to implement their own batching were possible to avoid encountering
network limits and timeouts.

General

- The old client version v1.12.1 is reaching end of life and will longer be supported as of February 2024.
You can expect the version to stop working at any point after that as legacy backend support is turned off.
- *Breaking Change*: The deprecated `Scenes` client API has been removed.
- *Breaking Change*: The deprecated `Metadata` client API has been removed.
- The minimum required version of `urllib3` has been bumped to 1.26.18 to address a security vulnerability.
- The minimum required version of `shapely` has been bumped to 2.0.0 to address thread safety issues.
- Python 3.7, formerly deprecated, is no longer supported.
- Python 3.12 is not yet officially supported due to the lack of support from `blosc`. However, if you
are able to provide a functional `blosc` on your own, then 3.12 should work.
- Urllib3 2.X is now supported.
- Geopandas, Pydantic, and PyArrow have been added as core dependencies to support the Vector client.
- For those users of the `clear_client_state` function (not common), the bands cache for the Catalog client
is now cleared also.

(note that these release notes are duplicated from the non-public v3.0.0rc0)

Due to a number of breaking changes, the version has been bumped to 3.0.0. However, the vast majority
of typical use patterns in typical user code will not require changes. Please review the specifics
below.

Catalog

- The `tags` attributes on Catalog objects can now contain up to 32 elements, each up to 1000 characters long.
But why would you even want to go there?
- *Breaking Change*: Derived bands, never supported in the AWS environment and catalog products, have been
removed.
- The new `Blob.delete_many` method may be used to delete large numbers of blobs efficiently.
- The `Blob.get_or_create` method didn't allow supplying `storage_type`, `namespace`, or `name` parameters.
Now it works as expected, either returning a saved Blob from the Catalog, or an unsaved blob that
you can use to upload and save its data.
- Image methods `ndarray` and `download` no longer pass the image's default geocontext geometry as a cutline.
This is to avoid problems when trying to raster a complete single image in its native CRS and resolution
where imperfect geometries (due to a simplistic projection to EPSG:4326) can cause some boundary pixels
to be masked. When passing in an explicit `GeoContext` to these methods, consider whether any cutline
geometry is required or not, to avoid these issues.

Compute

- `Function` and `Job` objects now have a new `environment` attribute which can be used to define environment
variables for the jobs when they are run.
- *Breaking Change*: The `Function.map` method previously had no bound on how many jobs could be created at one time.
This led to operational problems with very large numbers of jobs. Now it submits jobs in batches (up to 1000
jobs per batch) to avoid request timeouts, and is more robust on retryable errors so that duplicate jobs are not
submitted accidently. There is still no bound on how many jobs you may create with a single call to `Function.map`.
Additionally, since it is possible that some jobs may be successfully submitted, and others not, the return
value, while still behaving as a list of `Job`s, is now a `JobBulkCreateResult` object which has a `is_success`
and an `error` property which can be used to determine if all submissions were successful, what errors may
have occurred, and what jobs have actually been created. Only if the first batch fails hard will the method
raise an exception.
- The `Job.statistics` member is now typed as a `JobStatistics` object.
- The efficiency of deleting many jobs at once has been significantly improved using `Function.delete` and
`Function.delete_jobs`. It is still possible to encounter request timeouts with very large numbers of jobs;
workarounds are now documented in the API documentation for the `Function.delete_jobs` method.
- The `ComputeClient.check_credentials` method has been added, so that the client can determine if valid
user credentials have already been registered with the Compute service.

Vector

- The Vector client library, previously available as the `descarteslabs-vector` package on PyPI, has
now been integrated into the Descartes Labs Python Client (this package). It should no longer be
installed separately.
- Visualization support (`ipyleaflet.Map`) is enabled when `ipyleaflet` is available. It is not
installed by default, but can be installed manually, or by installing the `descarteslabs` python
client with the `viz` extra (e.g. `pip install descarteslabs[viz]`). Note that in order to be
compatible with jupyterlab notebooks, the `visualize()` method no longer returns the layer, it
just adds it to the supplied map.
- The Vector package now has a `VectorClient` API client, with the usual support for `get_default_client()`
and `set_default_client()`. Most constructors and methods now accept an optional `client=` parameter
if you need to use something other than the default client.
- Configuration is now accomplished using the standard `descarteslabs.config` package. In particular,
the `vector_url` setting is used to specify the default Vector host. The `VECTOR_API_HOST` environment
variable is no longer consulted.
- Vector client methods now raise standard `descarteslabs.exceptions` Exception classes rather than
the `descarteslabs.vector.vector_exceptions` classes of the old client.
- The `is_spatial=` parameter previously accepted by many methods and functions is now deprecated
and ignored. It is not required because existing type information always determines if an operation
is spatial or not. Warnings will be generated if it is used.
- Be advised that feature upload and download (query) do not currently support or impose any limits,
and thus allow operations so large and slow that timeouts or other failures may occur. A future
version will implement limits and batching, so that large operations can be supported reliably.
Until then, the user may wish to implement their own batching were possible to avoid encountering
network limits and timeouts.

General

- The old client version v1.12.1 is reaching end of life and will longer be supported as of February 2024.
You can expect the version to stop working at any point after that as legacy backend support is turned off.
- *Breaking Change*: The deprecated `Scenes` client API has been removed.
- *Breaking Change*: The deprecated `Metadata` client API has been removed.
- The minimum required version of `urllib3` has been bumped to 1.26.18 to address a security vulnerability.
- The minimum required version of `shapely` has been bumped to 2.0.0 to address thread safety issues.
- Python 3.7, formerly deprecated, is no longer supported.
- Python 3.12 is not yet officially supported due to the lack of support from `blosc`. However, if you
are able to provide a functional `blosc` on your own, then 3.12 should work.
- Urllib3 2.X is now supported.
- Geopandas, Pydantic, and PyArrow have been added as core dependencies to support the Vector client.
- For those users of the `clear_client_state` function (not common), the bands cache for the Catalog client
is now cleared also.

Compute

- `Function.delete_jobs` was failing to implement the `delete_results` parameter, so job result blobs
were not being deleted. This has been fixed.
- Add `delete_results` parameter to `Function.delete` for consistency.
- `Job.statistics` field added which contains statistics (CPU, memory, and network utilization) for the
job. This can be used to determine the minimal resources necessary for the `Function` after some
representative runs.

Compute

- Filtering on datetime attributes (such as `Function.created_at`) didn't previously work with anything
but `datetime` instances. Now it also handles iso format strings and unix timestamps (int or float).

General

- Following our lifecycle policy, client versions v1.11.0 and earlier are no longer supported. They may
cease to work with the Platform at any time.

Catalog

- The Catalog `Blob` class now has a `get_data()` method which can be used to retrieve the blob
data directly given the id, without having to first retrieve the `Blob` metadata.

Compute

- *Breaking Change* The status values for `Function` and `Job` objects have changed, to provide a
better experience managing the flow of jobs. Please see the updated Compute guide for a full explanation.
Because of the required changes to the back end, older clients (i.e. v2.0.3) are supported in a
best effort manner. Upgrading to this new client release is strongly advised for all users of the
Compute service.

- *Breaking Change* The base images for Compute have been put on a diet. They are now themselves built
from "slim" Python images, and they no longer include the wide variety of extra Python packages that were
formerly included (e.g. TensorFlow, SciKit Learn, PyTorch). This has reduced the base image size by
an order of magnitude, making function build times and job startup overhead commensurately faster.
Any functions which require such additional packages can add them in as needed via the `requirements=`
parameter. While doing so will increase image size, it will generally still be much better than the prior
"Everything and the kitchen sink" approach. Existing Functions with older images will continue
to work as always, but any newly minted `Function`` using the new client will be using one of the new
slim images.

- Base images are now available for Python3.10 and Python3.11, in addition to Python3.8 and Python3.9.

- Job results and logs are now integrated with Catalog Storage, so that results and logs can be
searched and retrieved directly using the Catalog client as well as using the methods in the Compute
client.

- The new `ComputeResult` class can be used to wrap results from a `Function`, allowing the user to
specify additional attributes for the result which will be stored in the Catalog `Blob` metadata for
the result. This allows the function to specify properties such as `geometry`, `description`,
`expires` and `extra_attributes` for the result `Blob`. The use of `ComputeResult` is not required.

- A `Job` can now be assigned arbitrary tags (strings), and searched based on them.

- A `Job` can now be retried on errors, and jobs track error reasons, exit codes, and execution counts.

- `Function` and `Job` objects can now be filtered by class attributes (ex.
`Job.search().filter(Job.status == JobStatus.PENDING).collect()`).

- The `Job.cancel()` method can now be used to cancel the execution of a job which is currently
pending or running. Pending jobs will immediately transition to `JobStatus.CANCELED` status,
while running jobs will pass through `JobStatus.CANCEL` (waiting for the cancelation to be
signaled to the execution engine), `JobStatus.CANCELING` (waiting for the execution to terminate),
and `JobStatus.CANCELED` (once the job is no longer executing). Cancelation of running jobs is
not guaranteed; a job may terminate successfully, or with a failure or timeout, before it can
be canceled.

- The `Job.result()` method will raise an exception if the job does not have a status of
`JobStatus.SUCCESS`. If `Job.result()` yields an `None` value, this means that there was no
result (i.e. the execution returned a `None`).

- The `Job.result_blob()` will return the Catalog Storage Blob holding the result, if any.

- The `Function` object now has attributes `namespace` and `owner`.

- The `Function.wait_for_completion()` and new `Function.as_completed()` methods provide a richer
set of functionality for waiting on and handling job completion.

- The `Function.build_log()` method now returns the log contents as a string, rather than printing
the log contents.

- The `Job.log()` method now returns the log contents as a list of strings, rather than printing the log
contents. Because logs can be unbounded in size, there's also a new `Job.iter_log()` method which returns
an iterator over the log lines.

- The `requirements=` parameter to `Function` objects now supports more `pip` magic, allowing the use
of special `pip` controls such as `-f`. Also parsing of package versions has been loosened to allow
some more unusual version designators.

- Changes to the `Function.map()` method, with the parameter name change of `iterargs` changed to `kwargs`
(the old name is still honored but deprecated), corrected documentation, and enhancements to support more
general iterators and mappings, allowing for a more functional programming style.

- The compute package was restructured to make all the useful and relevant classes available at the top level.

Utils

- Property filters can now be deserialized as well as serialized.