Overview
This release includes important updates to the raw-converted data format based on adaptation of convention, an efficiency boost for combining multiple `EchoData` objects, packaging and infrastructure upgrades, and other updates necessitated by these changes.
Enhancement
- Overhaul the `combine_echodata` function (1042, 1116)
- No longer require spinning up dask client under the hood during combine
- Use `xr.concat` directly compatible with delayed computation
- No longer require destination path as input argument
- Overhaul `EchoData.update_platform` (1060)
- Allow adding external variables with arbitrary number of time dimensions, with new time dimensions following an increasing order (e.g., `time3`, `time4`, etc)
- Only allow updating standardized, sonar model-dependent Platform group variables
Raw-converted data format changes
- Drop the `beam` and `ping_time` dimensions for some parameters (1056, 1083)
- The dimensions of these parameters were previously expanded to conform with convention requirements, but this inflated the data volume, especially for in memory operations, and made the calibration code confusing
- Standardize `backscatter_r/i` long_name in the `Sonar/Beam_groupX` group, and correct units (1047)
- Standardize use of `transmit_frequency_start/stop` for all echosounder models (1091)
- Move filter coefficients and decimation factor to variables in EK80 `Vendor_specific` group (1044, 1046, 1105)
- Add new `Provenance` group `combination_*` attributes to combined `EchoData` object, mirroring the `conversion_*` attributes (1113)
- Bring more consistency in the `Platform` group across sensors on conversion (1058, 1061)
- Add missing mandatory variables and other data format changes (1094, 1099, 1101, 1102, 1103, 1104, 1107, 1114)
- Set time arrays explicitly to datetime64[ns] in parsers and remove duplicated timestamp stuff (1117)
Packaging and infrastructure
- Remove Pandas<2 pinning (1080)
- Pin netcdf to >1.6 and add explicit encoding (1112)
- Update CI to prepare for python 3.11 (1108)
- Switch github action for get-changed-files (1118)
- Fix discrepancy on test log output (1119)
Others
- Set and fix encoding for variable encoding (1072)
- Specify handling for expected string type
- Sanitize and set encoding as part of EchoData
- Handling problematic NMEA messages when setting lat/lon in Platform group (1067)
- Update permission check filename to be UUID to make it thread safe (1110)
- use `pulse_form` for EK80 transmit type checks (1091)
- Factor out a normalization factor from pulse compression function for flexible use (1105)
- Add existence checking for angle parameters in `consolidate.add_splitbeam_angle` (1105)
- Revise qc function to clean up reversed time (1065)
- Update matplotlib call to register EK500 colormap (1068)
- Temporarily remove `compute_NASC` for debugging and overhaul (1136)
- Remove the deprecated `preprocess` subpackage (1077, 1082)
- Remove the `ZarrCombine` module since it is no longer used (1117)
- Remove outdated `ui_param` (1071)
Documentation
- Add rendering for example `EchoData` objects for all supported echosounders to show the updated format (1089)
- Restructure TOC to have separate Raw data formats and Data Processing sections, each with new subpages (1093, 1132, 1134)
- Add discussion on z-axis vertical coordinate system and variables (1133)
- Add bibliography page linked from inline citations (1137)