Echopype

Overview

This release includes new features to interface with Echoview ECS files for computing Sv, reorganization of computing functions into new subpackages, addition of data processing level attributes to data products, and other improvements and bug fixes.

New features and major changes
- Allow using ECS for calibrating Simrad echosounders (996, 1004)
- This functionality is in a **beta** testing stage
- Details of implementation may change and bugs are possible
- Expand ECS parser to accept frequency-dependent values in EK80 ECS files
- Overhaul `env_params` to ensure correct intake for calibration (985)
- Now allows using `env_params` entries that are xr.DataArrays
- Move functions previously in `preprocess` subpackage to new subpackages (993). Calling these functions from `preprocess` is deprecated and will be removed in v0.7.1.
- `clean`: `remove_noise`, `estimate_noise`
- `commongrid`: `compute_MVBS`, `compute_MVBS_index_binning`
- Add `commongrid.compute_NASC` (1005)
- The current implementation uses brute force looping for mean Sv computation, this will be refactored and optimized together with other functions requiring the same pattern in an upcoming release
- Add global attributes for data processing levels (1001).
- This functionality is in a **beta** testing stage
- See [data processing level specifications](https://echopype.readthedocs.io/en/stable/processing-levels.html) for functions and conditions under which such attributes are added
- Expand `mask.apply_mask` to handle multi-channel Sv datasets (1010)
- Standardize sonar metadata for EK80 data (992)
- `sonar_serial_number` is now an empty global attribute, no longer a variable, as in the EK60 case
- `transducer_name`, `transducer_serial_number`, `transceiver_serial_number` based on parser parameter `transducer_name`, `transducer_serial_number`, and `serial_number`, respectively

Bug fixes
- Fix scaling bug for `beamwidth_alongship` and `beamwidth_athwartship` from CW-based parameters to values corresponding to center frequency of broadband transmit signals (998)

Tests and infrastructure
- Add more comprehensive tests for `add_location` (1000)
- Add test for splitbeam angle `ek80_CW_power` case (994)
- Add unit and integration tests for `env_params` intake for calibration (985)
- Exclude `test_data` folder in `codespell` pre-commit hook (1016)

Overview

This is a minor release that includes a small bug fix and an enhancement, which removes excessive warnings due to changes in v0.6.4.

Bug fix
- Fix a bug that prevented passing in `env_params` for EK60 calibration (987)

Enhancement
- Handling non-positive values that causes log10 warnings on EK calibration (986)

Overview

This is a release that includes important performance enhancements that allow user-provided calibration parameters for broadband and narrowband calibration, new functionalities to compute and attach split-beam angles to calibrated Sv dataset, perform frequency-differencing and masking, as well as a number of bug fixes and other improvements.

New features
- Allow passing in `cal_params` as a dictionary for narrowband and broadband calibration (955)
- Add default chunk encoding for zarr output (939)
- Add `add_splitbeam_angle` function to the `consolidate` subpackage (916, 971)
- Add a new `mask` subpackage
- Add `apply_mask` function to the `mask` subpackage (912)
- Add `frequency-differencing` function to the `mask` subpackage (901)
- Allow selection of a subset of channels when combining multiple echodata objects (892)
- This is done via the added `channel_selection` input argument to `combine_echodata`
- Add default consolidated flag for `echodata.to_zarr` (855)
- Generalize and improve efficiency of `compute_MVBS` (878)
- Allow `echo_range` that vary with `ping_time`
- Allow `Sv` data that are dask arrays
- Add `.nbytes` to obtain that data size of an `echodata` object (874)
- Add new data variables from the raw data files to facilitate EK80 calibration (944)
- transceiver type, transmit impedance, receive impedance, receiver sampling frequency

Enhancements and other changes
- Unify the order of dimensions of `echo_range` across all sonar models (968)
- Create and use the default echopype home directory (896, 954)
- This default directory is at `~/.echopype`
- Refactor the `calibrate` subpackage (904)
- routines for `env_params` and `cal_params` intake and range computation are now in different modules rather than methods in each classes
- Tidy up `cal_params` related routines (953)
- Revise `env_params` related routines (952)
- Improving pulse compressed broadband Sv computation and echo range computation (944)
- The resulting values are tested against pyEcholab and Echoview outputs
- Note there is unresolved discrepancy between pyEcholab/Echopype outputs with Echoview outputs for the first section of Sv values
- Change input argument names for `open_raw` (962)
- Change `offload_to_zarr` to `use_swap`
- Change `max_zarr_mb` to `max_mb`

Bug fixes
- Handling of provenance attributes in apply-mask and add-depth, especially for testing (930)
- Fix bugs in combining multiple echodata objects (the `combine_echodata` function)
- Fix meta_source_filenames bug and enable (meta)source_filenames appending of path and list (913)
- Fix bugs in `env_params` intake for calibration (952)

Infrastructure
- Update docker build for arm64 mac silicon chips (964)
- Fix windows github actions workflow (947)
- Remove mamba dependency (946)
- Fix pre-commit ci and update RTD config for jupyter-book (934)

Overview

This is a minor release that includes an important performance enhancement for combining large volumes of data residing in individual files into a single entity, a number of bug fixes, and other smaller improvements.

New features
- Overhaul `combine_echodata` function
- Allow combine a large number of `EchoData` objects exceeding memory limits (808, 824, 830)
- Remove reversed time check from combine_echodata (835)
- Add minimal ZarrCombine test (826)
- Order the channel coordinate in file conversion to ensure consistent combination across files (818)
- Revise outdated data combination behavior (797, 799)
- Clean up all coordinate and attribute details under `combine_echodata` function (848, 849)
- Track provanance for filenames of raw data files and auxiliary files
- Propagate `xml_path` as `meta_source_filenames` to combined echodata (814, 852)
- Standardize handling of `source_filenames` in individual and combined echodata as well as downstream datasets (804, 806)

Under the hood enhancements
- Clean up functions for setting encoding in the converted files (851)
- Add `requests` and `aiohttp` to dependency (844)
- Pin `netcdf4` to be <1.6 for pypi package due to ongoing `netcdf-c` problem (843)
- Write `Parsed2Zarr` generated files to `temp_echopype_output/parsed2zarr_temp_files` (832)
- Change `isel` to `sel` to fix `compute_Sv` to allow working with dask array (828)
- Add `open_raw(offload_to_zarr=True)` integration tests (794)

Bug fixes
- Fix regression bug with interpolating environmental variables to `ping_time` grid (837, 856)
- Fix WindowsPath error with compute_Sv when run on Windows (829)
- Fix logic problem in `open_raw(offload_to_zarr=True)` that sometimes cause problems (794, 853)

Infrastructure
- Update CI to use micromamba (805)
- Fix version string on CI (804, 820)
- Rename `ci.yaml` to `build.yaml` for clarity (807)

Overview

This is a minor release that includes a few new features and memory efficiency-related changes that make echopype better.

New features
- Add a new subpackage `consolidate` that contains functions to consolidate data across the calibrated Sv dataset and the corresponding raw-converted file
- Add function `consolidate.add_location` to interpolate location to calibrated dataset (749)
- Add function `consolidate.add_depth` to convert `range_meter` to `depth` with information on transducer tilt and depth (738)
- Move function that swaps the channel with frequency dimension (now as `consolidate.swap_dims_channel_frequency`) (738)
- Add new functionality to allow control of logging outputs (772)

Under the hood enhancements
- Improve memory usage while converting files that require significant NaN-padding and previously would incur very large memory expansion (774)
- This is achieved by directly writing variables that may incur a large memory expansion into a temporary zarr store
- Beta function that will benefit from user feedback
- Overhaul access pattern for EchoData (762)
- Remove previous access pattern for different groups in the raw-converted file
- Starting from this release all groups are accessed with `echodata["GROUP_PATH"]`, e.g., `echodata["Platform"]`, `echodata["Sonar/Beam_group1"]`, etc.
- Make long_name in ds_power for EK80 consistent with other sonar model (771)
- Modify `set_beam()` so it returns a list (780)
- Change the order in `_save_groups_to_file` so the order of groups is preserved when opening a converted netcdf file (779)
- Remove the user option to select NMEA sentences in `open_raw` to ensure raw data is preserved (778)

Infrastructure
- Update CI set up to avoid exceeding GitHub actions memory limitation (761)
- decrease the number of workers from 4 to 2

Overview

This is a minor release that includes important bug fixes, a number of new features, and some leftover data format changes specific to parsed AD2CP data format.

Bug fixes
- We use `datatree` under the hood for the `EchoData` object, but `datatree` v0.0.4 had a bug in accessing subgroups in netCDF data model in Windows OS. This was fixed in `datatree` v0.0.6, and hence we updated our dependency and made other associated changes to support cross platform users (732, 748)
- Fix empty Sv problem related to renaming time coordinate associated with environmental parameters used for calibration; this is a regression bug from v0.6.0 changes (755)
- Fix a bug in `compute_MVBS` in selecting `echo_range` for specific frequency. This is a regression bug from v0.6.0 changes (736)
- Allow parsing data from EK60 with split-beam transducers but without phase/angle data (491, 718)
- Check filter coeffs existence in `SetGroupsEK80.set_vendor` before saving (720, 724)
- Fix the check in `compute_MVBS` for handling different variations of NaN entries in the Sv dataset (753)
- Fix invalid timestamp issue in AD2CP data conversion (733)

New features
- Enhance `update_platform` to support a new use case (location data from fixed location) and add more consistency (741)
- Ability to parse and store RAW4 datagram for EK80 data (714)
- Add utility function for swapping `channel` coordinate with `frequency_nominal` (710)
- Add ES70, ES80, EA640 to allowed data type for calibration (759)

Changes of netCDF data model
- Reorganize AD2CP data variables into different `Sonar/Beam_groupX`s and different first-level groups in a form consistent with v0.6.0 changes for all other sonar models (731); some variables remain to be discussed and may change in future releases (719)

Enhancements
- Refactor AD2CP conversion to improve speed and memory usage through removal of xr.merge (505)
- Update Python requirements in docs to >=3.8 (744)

Infrastructure
- Update PR action to use PR title [all tests ci] to run the entire suite of tests and [skip ci] to skip all tests (721)