Datacube

Summary

This release contains mostly bug fixes an documentation improvements.

Full List of Changes

- Added an updated column for trigger based tracking of database row updates in PostgreSQL. (951)
- Changes to the writer driver API. The driver is now responsible for constructing output URIs from user configuration. (960)
- Added a `datacube.utils.geometry.assign_crs` method for better interoperability with other libraries (967)
- Better interoperability with xarray - the `xarray.Dataset.to_netcdf` function should work again (972, 976)
- Add support for unsigned access to public S3 resources from CLI apps (976)
- Usability fixes for indexing EO3 datasets (958)
- Fix CLI initialisation of the Dask Distributed Executor (974)

Summary

Lot's of changes since the 1.7 release.

The two primary changes that are most likely to have backward compatibility issues are:

1. The internal details of how we store geo-registration information on xarray Datasets returned by `dc.load` have changed in a significant way (837, 899).
2. We no longer use GDAL native Python bindings (`osgeo.{ogr,osr}`) and instead rely on `pyproj` and `shapely` as a backend for `Geometry` and `CRS` classes (880).

We no longer store CRS as an object (`datacube.utils.geometry.CRS`) in an attribute dictionary of the `DataArray`, instead it is stored in a string format (WKT) in an attribute of a special `spatial_ref` coordinate. This change allows us to better interface with `xarray` IO libraries. One can now write data loaded by `dc.load` with `xr.to_netcdf(..)` directly and load back with `xr.open_dataset(..)`, all while maintaining geo-registration information (i.e. `.geobox` property).

Also, since CRS information is now stored on the Coordinate, and not on the `DataArray` itself, it survives a greater variety of mathematical operations. Attributes on the `DataArray` would often go missing when doing the most basic of operations, like changing `dtype` of the loaded data, now CRS metadata is preserved in the majority of the cases.

Moving away from the native GDAL Python bindings is primarily motivated by the complexity of the installation of `gdal` python library. Both `shapely` and `pyproj` that replaced it, offer binary wheels, and are therefore much simpler to install.

Summary

Lot's of changes since the 1.7 release.

The two primary changes that are most likely to have backward compatibility issues are:

1. The internal details of how we store geo-registration information on xarray Datasets returned by `dc.load` have changed in a significant way (837, 899).
2. We no longer use GDAL native Python bindings (`osgeo.{ogr,osr}`) and instead rely on `pyproj` and `shapely` as a backend for `Geometry` and `CRS` classes (880).

We no longer store CRS as an object (`datacube.utils.geometry.CRS`) in an attribute dictionary of the `DataArray`, instead it is stored in a string format (WKT) in an attribute of a special `spatial_ref` coordinate. This change allows us to better interface with `xarray` IO libraries. One can now write data loaded by `dc.load` with `xr.to_netcdf(..)` directly and load back with `xr.open_dataset(..)`, all while maintaining geo-registration information (i.e. `.geobox` property).

Also, since CRS information is now stored on the Coordinate, and not on the `DataArray` itself, it survives a greater variety of mathematical operations. Attributes on the `DataArray` would often go missing when doing the most basic of operations, like changing `dtype` of the loaded data, now CRS metadata is preserved in the majority of the cases.

Moving away from the native GDAL Python bindings is primarily motivated by the complexity of the installation of `gdal` python library. Both `shapely` and `pyproj` that replaced it, offer binary wheels, and are therefore much simpler to install.

Full List of Changes

- Changed geo-registration mechanics for arrays returned by `dc.load`
- Migrate geometry and CRS backends from `osgeo.ogr` and `osgeo.osr` to `shapely` and `pyproj` respectively
- Fixes for geometries crossing anti meridian
- EO3 dataset metadata format is now understood by `datacube dataset add`
- New virtual product combinator `reproject` for on-the-fly reprojection of rasters
- Enhancements to the `expressions` transformation in virtual products
- Support `/vsi**` style paths for dataset locations
- Remove old Search Expressions and replace with a simpler implementation based on Lark Parser
- Remove no longer required PyPEG2 dependency
- Change development version numbers generation. Use `setuptools_scm` instead of `versioneer`
- Deprecated `datacube.helpers.write_geotiff`, use `datacube.utils.cog.write_cog` for similar functionality
- Deprecated `datacube.storage.masking`, moved to `datacube.utils.masking`
- Remove S3AIO driver

Summary

Lot's of changes since the 1.7 release.

The two primary changes that are most likely to have backward compatibility issues are:

1. The internal details of how we store geo-registration information on xarray Datasets returned by `dc.load` have changed in a significant way (837, 899).
2. We no longer use GDAL native Python bindings (`osgeo.{ogr,osr}`) and instead rely on `pyproj` and `shapely` as a backend for `Geometry` and `CRS` classes (880).

We no longer store CRS as an object (`datacube.utils.geometry.CRS`) in an attribute dictionary of the `DataArray`, instead it is stored in a string format (WKT) in an attribute of a special `spatial_ref` coordinate. This change allows us to better interface with `xarray` IO libraries. One can now write data loaded by `dc.load` with `xr.to_netcdf(..)` directly and load back with `xr.open_dataset(..)`, all while maintaining geo-registration information (i.e. `.geobox` property).

Also, since CRS information is now stored on the Coordinate, and not on the `DataArray` itself, it survives a greater variety of mathematical operations. Attributes on the `DataArray` would often go missing when doing the most basic of operations, like changing `dtype` of the loaded data, now CRS metadata is preserved in the majority of the cases.

Moving away from the native GDAL Python bindings is primarily motivated by the complexity of the installation of `gdal` python library. Both `shapely` and `pyproj` that replaced it, offer binary wheels, and are therefore much simpler to install.

Full List of Changes

- Changed geo-registration mechanics for arrays returned by `dc.load`
- Migrate geometry and CRS backends from `osgeo.ogr` and `osgeo.osr` to `shapely` and `pyproj` respectively
- New virtual product combinator `reproject` for on-the-fly reprojection of rasters
- Enhancements to the `expressions` transformation in virtual products
- Support `/vsi**` style paths for dataset locations
- Remove old Search Expressions and replace with a simpler implementation based on Lark Parser
- Remove no longer required PyPEG2 dependency
- Change development version numbers generation. Use `setuptools_scm` instead of `versioneer`
- Deprecated `datacube.helpers.write_geotiff`, use `datacube.utils.cog.write_cog` for similar functionality
- Deprecated `datacube.storage.masking`, moved to `datacube.utils.masking`
- Remove S3AIO driver

Testing release process, please ignore this one.

Not a lot of changes since rc1.

- Early exit from `dc.load` on `KeyboardInterrupt`, allows partial loads inside notebook.
- Some bug fixes in geometry related code
- Some cleanups in tests
- Pre-commit hooks configuration for easier testing
- Re-enable multi-threaded reads for `s3aio` driver (set `use_threads=True` in `dc.load(..)`)