Esmvalcore

-------
Highlights

TODO: add highlights

This release includes

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

TODO: add examples of how to deal with these changes

- Allow contiguous representation of extracted regions (:pull:`2230`) by :user:`rebeccaherman1`

Deprecations
~~~~~~~~~~~~

- Refactor regridding (:pull:`2231`) by :user:`schlunma`
- Allow deprecated regridding scheme ``linear_extrapolate`` in recipe checks (:pull:`2324`) by :user:`schlunma`
- Allow deprecated regridding scheme ``unstructured_nearest`` in recipe checks (:pull:`2336`) by :user:`schlunma`

Bug fixes
~~~~~~~~~

- Do not overwrite facets from recipe with CMOR table facets for derived variables (:pull:`2255`) by :user:`bouweandela`
- Fix error message in variable definition check (:pull:`2313`) by :user:`enekomartinmartinez`
- Unify dtype handling of preprocessors (:pull:`2393`) by :user:`schlunma`

CMOR standard
~~~~~~~~~~~~~

- Also read default custom CMOR tables if custom location is specified (:pull:`2279`) by :user:`schlunma`
- Add custom CMOR table for total cloud water (tcw) (:pull:`2277`) by :user:`axel-lauer`
- Add height for sfcWindmax in MPI HighRes models (:pull:`2292`) by :user:`malininae`
- Fixed ``positive`` attribute in custom rtnt table (:pull:`2367`) by :user:`schlunma`
- Fix ``positive`` attributes in custom CMOR variables (:pull:`2380`) by :user:`schlunma`
- Log CMOR check and generic fix output to separate file (:pull:`2361`) by :user:`schlunma`

Computational performance improvements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- More lazy fixes and preprocessing functions (:pull:`2325`) by :user:`bouweandela`
- Made preprocessors :func:`esmvalcore.preprocessor.mask_landsea`, :func:`esmvalcore.preprocessor.mask_landseaice` and :func:`esmvalcore.preprocessor.mask_glaciated` lazy (:pull:`2268`) by :user:`joergbenke`
- More lazy :func:`esmvalcore.preprocessor.extract_levels` preprocessor function (:pull:`2120`) by :user:`bouweandela`
- Use lazy weights for :func:`esmvalcore.preprocessor.climate_statistics` and :func:`esmvalcore.preprocessor.axis_statistics` (:pull:`2346`) by :user:`schlunma`
- Fixed potential memory leak in :func:`esmvalcore.preprocessor.local_solar_time` (:pull:`2356`) by :user:`schlunma`
- Cache regridding weights if possible (:pull:`2344`) by :user:`schlunma`
- Implement lazy area weights (:pull:`2354`) by :user:`schlunma`
- Avoid large chunks in :func:`esmvalcore.preprocessor.climate_statistics` preprocessor function with `period='full'` (:pull:`2404`) by :user:`bouweandela`

Documentation
~~~~~~~~~~~~~

- Use short links in changelog (:pull:`2287`) by :user:`bouweandela`
- National Computing Infrastructure (NCI), Site specific configuration (:pull:`2281`) by :user:`rbeucher`
- Update :func:`esmvalcore.preprocessor.multi_model_statistics` doc with latest changes (new operators, etc.) (:pull:`2321`) by :user:`schlunma`
- Fix Codacy badge (:pull:`2382`) by :user:`bouweandela`
- Change 'mean' to 'percentile' in doc strings of preprocessor statistics (:pull:`2327`) by :user:`lukruh`
- Fixed typo in doc about weighted statistics (:pull:`2387`) by :user:`schlunma`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Fixing missing height 2m coordinates in GFDL-CM4 and KIOST-ESM (:pull:`2294`) by :user:`Karen-A-Garcia`
- Added fix for wrong units of ``clt`` for CIESM and FIO-ESM-2-0 (:pull:`2330`) by :user:`schlunma`
- Cmip6 gfdl_cm4: fix tas height fix to work for concatenated scenarios (:pull:`2332`) by :user:`mwjury`
- Cordex GERICS REMO2015 lon differences above 10e-4 (:pull:`2334`) by :user:`mwjury`
- Download ICON grid without locking (:pull:`2359`) by :user:`bouweandela`
- Added ICON fixes for hfls and hfss (:pull:`2360`) by :user:`diegokam`
- Added ICON fix for rtnt (:pull:`2366`) by :user:`diegokam`
- Expanded ICON extra facets (:pull:`2379`) by :user:`schlunma`
- Add 10m height coordinate to SfcWind GFDL-CM4 instead of 2m height (:pull:`2385`) by :user:`Karen-A-Garcia`
- Cordex wrf381p: fix tas,tasmax,tasmin height (:pull:`2333`) by :user:`mwjury`
- Several minor fixes needed for marine BGC data. (:pull:`2110`) by :user:`ledm`

Installation
~~~~~~~~~~~~

- Updated iris pin to ``iris>=3.6.1`` (:pull:`2286`) by :user:`schlunma`
- Pin pandas yet again avoid new ``2.2.1`` as well (:pull:`2353`) by :user:`valeriupredoi`
- Update Iris pin to avoid using versions with memory issues (:pull:`2408`) by :user:`chrisbillowsMO`
- Pin esmpy <8.6.0 (:pull:`2402`) by :user:`valeriupredoi`

Preprocessor
~~~~~~~~~~~~

- New preprocessor: :func:`esmvalcore.preprocessor.local_solar_time` (:pull:`2258`) by :user:`schlunma`
- Read derived variables from other MIP tables (:pull:`2256`) by :user:`bouweandela`
- Added special unit conversion m -> DU for total column ozone (toz) (:pull:`2270`) by :user:`schlunma`
- Allow cubes as input for :func:`esmvalcore.preprocessor.bias` preprocessor (:pull:`2183`) by :user:`schlunma`
- Add normalization with statistics to many statistics preprocessors (:pull:`2189`) by :user:`schlunma`
- Adding sfcWind derivation from uas and vas (:pull:`2242`) by :user:`malininae`
- Update interval check in resample_hours (:pull:`2362`) by :user:`axel-lauer`
- Broadcast properly ``cell_measures`` when using :func:`esmvalcore.preprocessor.extract_shape` with ``decomposed: True`` (:pull:`2348`) by :user:`sloosvel`
- Compute volume from ``cell_area`` if available (:pull:`2318`) by :user:`enekomartinmartinez`
- Do not expand wildcards for datasets of derived variables where not all input variables are available (:pull:`2374`) by :user:`schlunma`
- Modernize :func:`esmvalcore.preprocessor.regrid_time` and allow setting a common calendar for decadal, yearly, and monthly data (:pull:`2311`) by :user:`schlunma`
- Added unstructured linear regridding (:pull:`2350`) by :user:`schlunma`
- Add preprocessors :func:`esmvalcore.preprocessor.distance_metrics` and :func:`esmvalcore.preprocessor.histogram` (:pull:`2299`) by :user:`schlunma`

Automatic testing
~~~~~~~~~~~~~~~~~

- Increase resources for testing installation from conda-forge (:pull:`2297`) by :user:`bouweandela`
- Pin pandas to avoid broken ``round`` function (:pull:`2305`) by :user:`schlunma`
- Remove team reviewers from conda lock generation workflow in Github Actions (:pull:`2307`) by :user:`valeriupredoi`
- Remove mocking from tests in ``tests/unit/preprocessor/_regrid/test_extract_point.py`` (:pull:`2193`) by :user:`ehogan`
- Pin ``pytest-mypy`` plugin to ``>=0.10.3`` comply with new ``pytest==8`` (:pull:`2315`) by :user:`valeriupredoi`
- Fix regridding test for unstructured nearest regridding on OSX (:pull:`2319`) by :user:`schlunma`
- Fix flaky regrid test by clearing LRU cache after each test (:pull:`2322`) by :user:`valeriupredoi`
- Xfail ``tests/integration/cmor/_fixes/test_common.py::test_cl_hybrid_height_coord_fix_metadata`` while Iris folk fix behaviour (:pull:`2363`) by :user:`valeriupredoi`
- Update codacy reporter orb to latest version (:pull:`2388`) by :user:`valeriupredoi`
- Add calls to ``conda list`` in Github Action test workflows to inspect environment (:pull:`2391`) by :user:`valeriupredoi`
- Pin pandas yet again :panda_face: ``test_icon`` fails again with pandas=2.2.2 (:pull:`2394`) by :user:`valeriupredoi`
- Fixed units of cl test data (necessary since iris>=3.8.0) (:pull:`2403`) by :user:`schlunma`

Improvements
~~~~~~~~~~~~

- Show files of supplementary variables explicitly in log (:pull:`2303`) by :user:`schlunma`
- Remove warning about logging in to ESGF (:pull:`2326`) by :user:`bouweandela`
- Do not read ``~/.esmvaltool/config-user.yml`` if ``--config-file`` is used (:pull:`2309`) by :user:`schlunma`
- Support loading ICON grid from ICON rootpath (:pull:`2337`) by :user:`schlunma`
- Handle warnings about invalid units for iris>=3.8 (:pull:`2378`) by :user:`schlunma`
- Added note on how to access ``index.html`` on remote server (:pull:`2276`) by :user:`schlunma`
- Remove custom fix for concatenation of aux factories now that bug in iris is solved (:pull:`2392`) by :user:`schlunma`
- Use ``iris.FUTURE.save_split_attrs = True`` to remove iris warning (:pull:`2398`) by :user:`schlunma`
- Ignored iris warnings about global attributes (:pull:`2400`) by :user:`schlunma`

.. _changelog-v2-10-0:

-------
Highlights

- All statistics preprocessors support the same operators and have a common
:ref:`documentation <stat_preprocs>`. In addition, arbitrary keyword arguments
for the statistical operation can be directly given to the preprocessor.

- The output webpage generated by the tool now looks better and provides
methods to select and filter the output.

- Improved computational efficiency:

- Automatic rechunking between preprocessor steps to keep the
`graph size smaller <https://docs.dask.org/en/latest/best-practices.html#avoid-very-large-graphs>`_
and the `chunk size optimal <https://blog.dask.org/2021/11/02/choosing-dask-chunk-sizes>`__.
- Reduce the size of the dask graph created by :func:`esmvalcore.preprocessor.anomalies`.
- Preprocessors :func:`esmvalcore.preprocessor.mask_above_threshold`,
:func:`esmvalcore.preprocessor.mask_below_threshold`,
:func:`esmvalcore.preprocessor.mask_inside_range`,
:func:`esmvalcore.preprocessor.mask_outside_range` are now lazy.
- Lazy coordinates bounds are no longer loaded into memory by the CMOR checks and fixes.

This release includes

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Remove the deprecated option ``use_legacy_supplementaries`` (:pull:`2202`) by :user:`bouweandela`

- The recommended upgrade procedure is to remove ``use_legacy_supplementaries`` from config-user.yml
(if it was there) and remove any mention of ``fx_variables`` from the recipe. If automatically defining
the required supplementary variables does not work, define them in the variable or
(``additional_``) ``datasets`` section as described in :ref:`supplementary_variables`.

- Use smarter (units-aware) weights (:pull:`2139`) by :user:`schlunma`

- Some preprocessors handle units better. For details, see the pull request.

- Removed deprecated configuration option ``offline`` (:pull:`2213`) by :user:`schlunma`

- In :ref:`changelog-v2-8-0`, we replaced the old ``offline`` configuration option. From this version on, it stops working.
Please refer to :ref:`changelog-v2-8-0` for upgrade instructions.

- Fix issue with CORDEX datasets requiring different dataset tags for downloads and fixes (:pull:`2066`) by :user:`ljoakim`

- Due to the different facets for CORDEX datasets, there was an inconsistency in the fixing mechanism.
This change requires changes to existing recipes that use CORDEX datasets. Please refer to the pull request for detailed update instructions.

- For the following changes, no user change is necessary

- Remove deprecated way of calling :func:`~esmvalcore.cmor.table.read_cmor_tables` (:pull:`2201`) by :user:`bouweandela`

- Remove deprecated callback argument from preprocessor ``load`` function (:pull:`2207`) by :user:`bouweandela`

- Remove deprecated preprocessor function `cleanup` (:pull:`2215`) by :user:`bouweandela`

Deprecations
~~~~~~~~~~~~

- Clearly separate fixes and CMOR checks (:pull:`2157`) by :user:`schlunma`
- Added new operators for statistics preprocessor (e.g., ``'percentile'``) and allowed passing additional arguments (:pull:`2191`) by :user:`schlunma`

- This harmonizes the operators for all statistics preprocessors. From this version, the new names can be used; the old arguments will stop working from
version 2.12.0. Please refer to :ref:`stat_preprocs` for a detailed description.

Bug fixes
~~~~~~~~~

- Re-add correctly region-extracted cell measures and ancillary variables after :ref:`extract_region` (:pull:`2166`) by :user:`valeriupredoi`, :user:`schlunma`
- Fix sorting of datasets

- Fix sorting of ensemble members in :func:`~esmvalcore.dataset.datasets_to_recipe` (:pull:`2095`) by :user:`bouweandela`
- Fix a problem with sorting datasets that have a mix of facet types (:pull:`2238`) by :user:`bouweandela`
- Avoid a crash if dataset has supplementary variables (:pull:`2198`) by :user:`bouweandela`

CMOR standard
~~~~~~~~~~~~~

- ERA5 on-the-fly CMORizer: changed sign of variables ``evspsbl`` and ``evspsblpot`` (:pull:`2115`) by :user:`katjaweigel`
- Add ``ch4`` surface custom cmor table entry (:pull:`2168`) by :user:`hb326`
- Add CMIP3 institutes names used at NCI (:pull:`2152`) by :user:`rbeucher`
- Added :func:`~esmvalcore.cmor.fixes.get_time_bounds` and :func:`~esmvalcore.cmor.fixes.get_next_month` to public API (:pull:`2214`) by :user:`schlunma`
- Improve concatenation checks

- Relax concatenation checks for ``--check_level=relax`` and ``--check_level=ignore`` (:pull:`2144`) by :user:`sloosvel`
- Fix ``concatenate`` preprocessor function (:pull:`2240`) by :user:`bouweandela`
- Fix time overlap handling in concatenation (:pull:`2247`) by :user:`zklaus`

Computational performance improvements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Make :ref:`threshold_masking` preprocessors lazy (:pull:`2169`) by :user:`joergbenke`

- Restored usage of numpy in `_mask_with_shp` (:pull:`2209`) by :user:`joergbenke`
- Do not realize lazy coordinate bounds in CMOR check (:pull:`2146`) by :user:`sloosvel`
- Rechunk between preprocessor steps (:pull:`2205`) by :user:`bouweandela`
- Reduce the size of the dask graph created by the ``anomalies`` preprocessor function (:pull:`2200`) by :user:`bouweandela`

Documentation
~~~~~~~~~~~~~

- Add reference to release v2.9.0 in the changelog (:pull:`2130`) by :user:`remi-kazeroni`
- Add merge instructions to release instructions (:pull:`2131`) by :user:`zklaus`
- Update `mamba` before building environment during Readthedocs build (:pull:`2149`) by :user:`valeriupredoi`
- Ensure compatible zstandard and zstd versions for .conda support (:pull:`2204`) by :user:`zklaus`
- Remove outdated documentation (:pull:`2210`) by :user:`bouweandela`
- Remove meercode badge from README because their API is broken (:pull:`2224`) by :user:`valeriupredoi`
- Correct usage help text of version command (:pull:`2232`) by :user:`jfrost-mo`
- Add ``navigation_with_keys: False`` to ``html_theme_options`` in Readthedocs ``conf.py`` (:pull:`2245`) by :user:`valeriupredoi`
- Replace squarey badge with roundy shield for Anaconda sticker in README (:pull:`2233`, :pull:`2260`) by :user:`valeriupredoi`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Updated doc about fixes and added type hints to fix functions (:pull:`2160`) by :user:`schlunma`

Installation
~~~~~~~~~~~~

- Clean-up how pins are written in conda environment file (:pull:`2125`) by :user:`valeriupredoi`
- Use importlib.metadata instead of deprecated pkg_resources (:pull:`2096`) by :user:`bouweandela`
- Pin shapely to >=2.0 (:pull:`2075`) by :user:`valeriupredoi`
- Pin Python to <3.12 in conda environment (:pull:`2272`) by :user:`bouweandela`

Preprocessor
~~~~~~~~~~~~

- Improve preprocessor output sorting code (:pull:`2111`) by :user:`bouweandela`
- Preprocess datasets in the same order as they are listed in the recipe (:pull:`2103`) by :user:`bouweandela`

Automatic testing
~~~~~~~~~~~~~~~~~

- [Github Actions] Compress all bash shell setters into one default option per workflow (:pull:`2126`) by :user:`valeriupredoi`
- [Github Actions] Fix Monitor Tests Github Action (:pull:`2135`) by :user:`valeriupredoi`
- [condalock] update conda lock file (:pull:`2141`) by :user:`valeriupredoi`
- [Condalock] make sure mamba/conda are at latest version by forcing a pinned mamba install (:pull:`2136`) by :user:`valeriupredoi`
- Update code coverage orbs (:pull:`2206`) by :user:`bouweandela`
- Revisit the comment-triggered Github Actions test (:pull:`2243`) by :user:`valeriupredoi`
- Remove workflow that runs Github Actions tests from PR comment (:pull:`2244`) by :user:`valeriupredoi`

Improvements
~~~~~~~~~~~~

- Merge v2.9.x into main (:pull:`2128`) by :user:`schlunma`
- Fix typo in citation file (:pull:`2182`) by :user:`bouweandela`
- Cleaned and extended function that extracts datetimes from paths (:pull:`2181`) by :user:`schlunma`
- Add file encoding (and some read modes) at open file step (:pull:`2219`) by :user:`valeriupredoi`
- Check type of argument passed to :func:`~esmvalcore.cmor.table.read_cmor_tables` (:pull:`2217`) by :user:`valeriupredoi`
- Dynamic HTML output for monitoring (:pull:`2062`) by :user:`bsolino`
- Use PyPI's trusted publishers authentication (:pull:`2269`) by :user:`valeriupredoi`

.. _changelog-v2-9-0:

------
Highlights
~~~~~~~~~~
It is now possible to use the
`Dask distributed scheduler <https://docs.dask.org/en/latest/deploying.html>`__,
which can
`significantly reduce the run-time of recipes <https://github.com/ESMValGroup/ESMValCore/pull/2049#pullrequestreview-1446279391>`__.
Configuration examples and advice are available in
:ref:`our documentation <config-dask>`.
More work on improving the computational performance is planned, so please share
your experiences, good and bad, with this new feature in :discussion:`1763`.

This release includes

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Remove deprecated configuration options (:pull:`2056`) by :user:`bouweandela`

- The module ``esmvalcore.experimental.config`` has been removed.
To upgrade, import the module from :mod:`esmvalcore.config` instead.

- The module ``esmvalcore._config`` has been removed.
To upgrade, use :mod:`esmvalcore.config` instead.

- The methods ``esmvalcore.config.Session.to_config_user`` and ``esmvalcore.config.Session.from_config_user`` have been removed.
To upgrade, use :obj:`esmvalcore.config.Session` to access the configuration values directly.

Bug fixes
~~~~~~~~~

- Respect ``ignore_warnings`` settings from the :ref:`project configuration <filterwarnings_config-developer>` in :func:`esmvalcore.dataset.Dataset.load` (:pull:`2046`) by :user:`schlunma`
- Fixed usage of custom location for :ref:`custom CMOR tables <custom_cmor_tables>` (:pull:`2052`) by :user:`schlunma`
- Fix issue with writing index.html when :ref:`running a recipe <running>` with ``--resume-from`` (:pull:`2055`) by :user:`bouweandela`
- Fixed bug in ICON CMORizer that lead to shifted time coordinates (:pull:`2038`) by :user:`schlunma`
- Include ``-`` in allowed characters for bibtex references (:pull:`2097`) by :user:`alistairsellar`
- Do not raise an exception if the requested version of a file is not available for all matching files on ESGF (:pull:`2105`) by :user:`bouweandela`

Computational performance improvements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Add support for :ref:`configuring Dask distributed <config-dask>` (:pull:`2049`, :pull:`2122`) by :user:`bouweandela`
- Make :func:`esmvalcore.preprocessor.extract_levels` lazy (:pull:`1761`) by :user:`bouweandela`
- Lazy implementation of :func:`esmvalcore.preprocessor.multi_model_statistics` and :func:`esmvalcore.preprocessor.ensemble_statistics` (:pull:`968` and :pull:`2087`) by :user:`Peter9192`
- Avoid realizing data in preprocessor function :func:`esmvalcore.preprocessor.concatenate` when cubes overlap (:pull:`2109`) by :user:`bouweandela`

Documentation
~~~~~~~~~~~~~

- Remove unneeded sphinxcontrib extension (:pull:`2047`) by :user:`valeriupredoi`
- Show ESMValTool logo on `PyPI webpage <https://pypi.org/project/ESMValCore/>`__ (:pull:`2065`) by :user:`valeriupredoi`
- Fix gitter badge in README (:pull:`2118`) by :user:`remi-kazeroni`
- Add changelog for v2.9.0 (:pull:`2088` and :pull:`2123`) by :user:`bouweandela`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Pass the :obj:`esmvalcore.config.Session` to fixes (:pull:`1988`) by :user:`schlunma`
- ICON: Allowed specifying vertical grid information in recipe (:pull:`2067`) by :user:`schlunma`
- Allow specifying ``raw_units`` for CESM2, EMAC, and ICON CMORizers (:pull:`2043`) by :user:`schlunma`
- ICON: allow specifying horizontal grid file in recipe/extra facets (:pull:`2078`) by :user:`schlunma`
- Fix tas/tos CMIP6: FIO, KACE, MIROC, IITM (:pull:`2061`) by :user:`pepcos`
- Add fix for EC-Earth3-Veg tos calendar (:pull:`2100`) by :user:`bouweandela`
- Correct GISS-E2-1-G ``tos`` units (:pull:`2099`) by :user:`bouweandela`

Installation
~~~~~~~~~~~~

- Drop support for Python 3.8 (:pull:`2053`) by :user:`bouweandela`
- Add python 3.11 to Github Actions package (conda and PyPI) installation tests (:pull:`2083`) by :user:`valeriupredoi`
- Remove ``with_mypy`` or ``with-mypy`` optional tool for prospector (:pull:`2108`) by :user:`valeriupredoi`

Preprocessor
~~~~~~~~~~~~

- Added ``period='hourly'`` for :func:`esmvalcore.preprocessor.climate_statistics` and :func:`esmvalcore.preprocessor.anomalies` (:pull:`2068`) by :user:`schlunma`
- Support IPCC AR6 regions in :func:`esmvalcore.preprocessor.extract_shape` (:pull:`2008`) by :user:`schlunma`


.. _changelog-v2-8-1:

------
Highlights
~~~~~~~~~~
This release adds support for Python 3.11 and includes several bugfixes.

This release includes:

Bug fixes
~~~~~~~~~

- Pin numpy !=1.24.3 (:pull:`2011`) by :user:`valeriupredoi`
- Fix a bug in recording provenance for the ``mask_multimodel`` preprocessor (:pull:`1984`) by :user:`schlunma`
- Fix ICON hourly data rounding issues (:pull:`2022`) by :user:`BauerJul`
- Use the default SSL context when using the ``extract_location`` preprocessor (:pull:`2023`) by :user:`ehogan`
- Make time-related CMOR fixes work with time dimensions `time1`, `time2`, `time3` (:pull:`1971`) by :user:`schlunma`
- Always create a cache directory for storing ICON grid files (:pull:`2030`) by :user:`schlunma`
- Fixed altitude <--> pressure level conversion for masked arrays in the ``extract_levels`` preprocessor (:pull:`1999`) by :user:`schlunma`
- Allowed ignoring of scalar time coordinates in the ``multi_model_statistics`` preprocessor (:pull:`1961`) by :user:`schlunma`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Add support for hourly ICON data (:pull:`1990`) by :user:`BauerJul`
- Fix areacello in BCC-CSM2-MR (:pull:`1993`) by :user:`remi-kazeroni`

Installation
~~~~~~~~~~~~

- Add support for Python=3.11 (:pull:`1832`) by :user:`valeriupredoi`
- Modernize conda lock file creation workflow with mamba, Mambaforge etc (:pull:`2027`) by :user:`valeriupredoi`
- Pin `libnetcdf!=4.9.1` (:pull:`2072`) by :user:`remi-kazeroni`

Documentation
~~~~~~~~~~~~~
- Add changelog for v2.8.1 (:pull:`2079`) by :user:`bouweandela`

Automatic testing
~~~~~~~~~~~~~~~~~

- Use mocked `geopy.geocoders.Nominatim` to avoid `ReadTimeoutError` (:pull:`2005`) by :user:`schlunma`
- Update pre-commit hooks (:pull:`2020`) by :user:`bouweandela`


.. _changelog-v2-8-0:

------
Highlights
~~~~~~~~~~

- ESMValCore now supports wildcards in recipes and offers improved support for
ancillary variables and dataset versioning thanks to contributions by
:user:`bouweandela`. For details, see
:ref:`Automatically populating a recipe with all available datasets <dataset_wildcards>`
and :ref:`Defining supplementary variables <supplementary_variables>`.
- Support for CORDEX datasets in a rotated pole coordinate system has been
added by :user:`sloosvel`.
- Native :ref:`ICON <read_icon>` output is now made UGRID-compliant
on-the-fly to unlock the use of more sophisticated regridding algorithms,
thanks to :user:`schlunma`.
- The Python API has been extended with the addition of three
modules: :mod:`esmvalcore.config`, :mod:`esmvalcore.dataset`, and
:mod:`esmvalcore.local`, all these features courtesy of
:user:`bouweandela`. For details, see our new
example :doc:`example-notebooks`.
- The preprocessor :func:`~esmvalcore.preprocessor.multi_model_statistics`
has been extended to support more use-cases thanks to contributions by
:user:`schlunma`. For details, see
:ref:`Multi-model statistics <multi-model statistics>`.

This release includes:

Backwards incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Please read the descriptions of the linked pull requests for detailed upgrade instructions.

- The algorithm for automatically defining the ancillary variables and cell
measures has been improved (:pull:`1609`) by :user:`bouweandela`.
If this does not work as expected, more examples of how to adapt your recipes
are given
`here <https://github.com/ESMValGroup/ESMValCore/pull/1609#Backward-incompatible-changes>`__
and in the corresponding sections of the
:ref:`recipe documentation <supplementary_variables>` and the
:ref:`preprocessor documentation <preprocessors_using_supplementary_variables>`.
- Remove deprecated features scheduled for removal in v2.8.0 or earlier
(:pull:`1826`) by :user:`schlunma`.
Removed ``esmvalcore.iris_helpers.var_name_constraint`` (has been deprecated
in v2.6.0; please use :class:`iris.NameConstraint` with the keyword argument
``var_name`` instead) and the option ``always_use_ne_mask`` for
:func:`esmvalcore.preprocessor.mask_landsea` (has been deprecated in v2.5.0;
the same behavior can now be achieved by specifying ``supplementary_variables``.
- No files will be found if a non-existent version of a dataset is specified
(:pull:`1835`) by :user:`bouweandela`. If a ``version`` of a
dataset is specified in the recipe, the tool will now search for exactly that
version, instead of simply using the latest version. Therefore, it is
necessary to make sure that the version number in the directory tree matches
with the version number in the recipe to find the files.
- The default filename template for obs4MIPs has been updated to better match
filenames used in this project in (:pull:`1866`) by :user:`bouweandela`. This
may cause issues if you are storing all the files for obs4MIPs in a
directory with no subdirectories per dataset.

Deprecations
~~~~~~~~~~~~
Please read the descriptions of the linked pull requests for detailed upgrade instructions.

- Various configuration related options that are now available through
:mod:`esmvalcore.config` have been deprecated (:pull:`1769`) by :user:`bouweandela`.
- The ``fx_variables`` preprocessor argument and related features have been
deprecated (:pull:`1609`) by :user:`bouweandela`.
See :pull:`1609Deprecations` for more information.
- Combined ``offline`` and ``always_search_esgf`` into a single option ``search_esgf``
(:pull:`1935`)
:user:`schlunma`. The configuration
option/command line argument ``offline`` has been deprecated in favor of
``search_esgf``. The previous ``offline: true`` is now ``search_esgf: never``
(the default); the previous ``offline: false`` is now
``search_esgf: when_missing``. More details on how to adapt your workflow
regarding these new options are given in :pull:`1935` and the
`documentation <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html?highlight=search_esgf#user-configuration-file>`__.
- :func:`esmvalcore.preprocessor.cleanup` has been deprecated (:pull:`1949`)
:user:`schlunma`. Please do not use this
anymore in the recipe (it is not necessary).

Python API
~~~~~~~~~~

- Support searching ESGF for a specific version of a file and add :obj:`esmvalcore.esgf.ESGFFile.facets` (:pull:`1822`) by :user:`bouweandela`
- Fix issues with searching for files on ESGF (:pull:`1863`) by :user:`bouweandela`
- Move the :mod:`esmvalcore.experimental.config` module to :mod:`esmvalcore.config` (:pull:`1769`) by :user:`bouweandela`
- Add :mod:`esmvalcore.local`, a module to search data on the local filesystem (:pull:`1835`) by :user:`bouweandela`
- Add :mod:`esmvalcore.dataset` module (:pull:`1877`) by :user:`bouweandela`

Bug fixes
~~~~~~~~~

- Import from :mod:`esmvalcore.config` in the :mod:`esmvalcore.experimental` module (:pull:`1816`) by :user:`bouweandela`
- Added scalar coords of input cubes to output of esmpy_regrid (:pull:`1811`) by :user:`schlunma`
- Fix severe bug in :func:`esmvalcore.preprocessor.mask_fillvalues` (:pull:`1823`) by :user:`schlunma`
- Fix LWP of ICON on-the-fly CMORizer (:pull:`1839`) by :user:`schlunma`
- Fixed issue in irregular regridding regarding scalar coordinates (:pull:`1845`) by :user:`schlunma`
- Update product attributes and `metadata.yml` with cube metadata before saving files (:pull:`1837`) by :user:`schlunma`
- Remove an extra space character from a filename (:pull:`1883`) by :user:`bouweandela`
- Improve resilience of ESGF search (:pull:`1869`) by :user:`bouweandela`
- Fix issue with no files found if timerange start/end differs in length (:pull:`1880`) by :user:`bouweandela`
- Add `driver` and `sub_experiment` tags to generate dataset aliases (:pull:`1886`) by :user:`sloosvel`
- Fixed time points of native CESM2 output (:pull:`1772`) by :user:`schlunma`
- Fix type hints for Python versions < 3.10 (:pull:`1897`) by :user:`bouweandela`
- Fixed `set_range_in_0_360` for dask arrays (:pull:`1919`) by :user:`schlunma`
- Made equalized attributes in concatenated cubes consistent across runs (:pull:`1783`) by :user:`schlunma`
- Fix issue with reading dates from files (:pull:`1936`) by :user:`bouweandela`
- Add institute name used on ESGF for CMIP5 CanAM4, CanCM4, and CanESM2 (:pull:`1937`) by :user:`bouweandela`
- Fix issue where data was not loaded and saved (:pull:`1962`) by :user:`bouweandela`
- Fix type hints for Python 3.8 (:pull:`1795`) by :user:`bouweandela`
- Update the institute facet of the CSIRO-Mk3L-1-2 model (:pull:`1966`) by :user:`remi-kazeroni`
- Fixed race condition that may result in errors in :func:`esmvalcore.preprocessor.cleanup` (:pull:`1949`) by :user:`schlunma`
- Update notebook so it uses supplementaries instead of ancillaries (:pull:`1945`) by :user:`bouweandela`

Documentation
~~~~~~~~~~~~~

- Fix anaconda badge in README (:pull:`1759`) by :user:`valeriupredoi`
- Fix mistake in the documentation of :obj:`esmvalcore.esgf.find_files` (:pull:`1784`) by :user:`bouweandela`
- Support linking to "stable" ESMValTool version on readthedocs (:pull:`1608`) by :user:`bouweandela`
- Updated ICON doc with information on usage of extract_levels preprocessor (:pull:`1903`) by :user:`schlunma`
- Add changelog for latest released version v2.7.1 (:pull:`1905`) by :user:`valeriupredoi`
- Update `preprocessor.rst` due to renaming of NCEP dataset to NCEP-NCAR-R1 (:pull:`1908`) by :user:`hb326`
- Replace timerange nested lists in docs with overview table (:pull:`1940`) by :user:`zklaus`
- Updated section "backward compatibility" in `contributing.rst` (:pull:`1918`) by :user:`axel-lauer`
- Add link to ESMValTool release procedure steps (:pull:`1957`) by :user:`remi-kazeroni`
- Synchronize documentation table of contents with ESMValTool (:pull:`1958`) by :user:`bouweandela`

Improvements
~~~~~~~~~~~~

- Support wildcards in the recipe and improve support for ancillary variables and dataset versioning (:pull:`1609`) by :user:`bouweandela`. More details on how to adapt your recipes are given in the corresponding pull request description and in the corresponding sections of the `recipe documentation <https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html#defining-supplementary-variables-ancillary-variables-and-cell-measures>`__ and the `preprocessor documentation <https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/preprocessor.html#preprocessors-using-supplementary-variables>`__.
- Create a session directory with suffix "-1", "-2", etc if it already exists (:pull:`1818`) by :user:`bouweandela`
- Message for users when they use esmvaltool executable from esmvalcore only (:pull:`1831`) by :user:`valeriupredoi`
- Order recipe output in index.html (:pull:`1899`) by :user:`bouweandela`
- Improve reading facets from ESGF search results (:pull:`1920`) by :user:`bouweandela`

Fixes for datasets
~~~~~~~~~~~~~~~~~~

- Fix rotated coordinate grids and `tas` and `pr` for CORDEX datasets (:pull:`1765`) by :user:`sloosvel`
- Made ICON output UGRID-compliant (on-the-fly) (:pull:`1664`) by :user:`schlunma`
- Fix automatic download of ICON grid file and make ICON UGRIDization optional (`default: true`) (:pull:`1922`) by :user:`schlunma`
- Add siconc fixes for EC-Earth3-Veg and EC-Earth3-Veg-LR models (:pull:`1771`) by :user:`egalytska`
- Fix siconc in KIOST-ESM (:pull:`1829`) by :user:`LisaBock`
- Extension of ERA5 CMORizer (variable cl) (:pull:`1850`) by :user:`axel-lauer`
- Add standard variable names for EMAC (:pull:`1853`) by :user:`FranziskaWinterstein`
- Fix for FGOALS-f3-L clt (:pull:`1928`) by :user:`LisaBock`

Installation
~~~~~~~~~~~~

- Add all deps to the conda-forge environment and suppress installing and reinstalling deps with pip at readthedocs builds (:pull:`1786`) by :user:`valeriupredoi`
- Pin netCDF4<1.6.1 (:pull:`1805`) by :user:`bouweandela`
- Unpin NetCF4 (:pull:`1814`) by :user:`valeriupredoi`
- Unpin flake8 (:pull:`1820`) by :user:`valeriupredoi`
- Add iris-esmf-regrid as a dependency (:pull:`1809`) by :user:`sloosvel`
- Pin esmpy<8.4 (:pull:`1871`) by :user:`zklaus`
- Update esmpy import for ESMF v8.4.0 (:pull:`1876`) by :user:`bouweandela`

Preprocessor
~~~~~~~~~~~~
- Allow :func:`esmvalcore.preprocessor.multi_model_statistics` on cubes with arbitrary dimensions (:pull:`1808`) by :user:`schlunma`
- Smarter removal of coordinate metadata in :func:`esmvalcore.preprocessor.multi_model_statistics` preprocessor (:pull:`1813`) by :user:`schlunma`
- Allowed usage of :func:`esmvalcore.preprocessor.multi_model_statistics` on single cubes/products (:pull:`1849`) by :user:`schlunma`
- Allowed usage of :func:`esmvalcore.preprocessor.multi_model_statistics` on cubes with identical ``name()`` and ``units`` (but e.g. different long_name) (:pull:`1921`) by :user:`schlunma`
- Allowed ignoring scalar coordinates in :func:`esmvalcore.preprocessor.multi_model_statistics` (:pull:`1934`) by :user:`schlunma`
- Refactored :func:`esmvalcore.preprocessor.regrid` and removed unnecessary code not needed anymore due to new iris version (:pull:`1898`) by :user:`schlunma`
- Do not realise coordinates during CMOR check (:pull:`1912`) by :user:`sloosvel`
- Make :func:`esmvalcore.preprocessor.extract_volume` work with closed and mixed intervals and allow nearest value selection (:pull:`1930`) by :user:`sloosvel`

Release
~~~~~~~
- Changelog for `v2.8.0rc1` (:pull:`1952`) by :user:`remi-kazeroni`
- Increase version number for ESMValCore `v2.8.0rc1` (:pull:`1955`) by :user:`remi-kazeroni`
- Changelog for `v2.8.0rc2` (:pull:`1959`) by :user:`remi-kazeroni`
- Increase version number for ESMValCore `v2.8.0rc2` (:pull:`1973`) by :user:`remi-kazeroni`
- Changelog for `v2.8.0` (:pull:`1978`) by :user:`remi-kazeroni`
- Increase version number for ESMValCore `v2.8.0` (:pull:`1983`) by :user:`remi-kazeroni`

Automatic testing
~~~~~~~~~~~~~~~~~

- Set implicit optional to true in `mypy` config to avert side effects and test fails from new mypy version (:pull:`1790`) by :user:`valeriupredoi`
- Remove duplicate `implicit_optional = True` line in ``setup.cfg`` (:pull:`1791`) by :user:`valeriupredoi`
- Fix failing test due to missing sample data (:pull:`1797`) by :user:`bouweandela`
- Remove outdated cmor_table facet from data finder tests (:pull:`1798`) by :user:`bouweandela`
- Modernize tests for :func:`esmvalcore.preprocessor.save` (:pull:`1799`) by :user:`bouweandela`
- No more sequential tests since SegFaults were not noticed anymore (:pull:`1819`) by :user:`valeriupredoi`
- Update pre-commit configuration (:pull:`1821`) by :user:`bouweandela`
- Updated URL of ICON grid file used for testing (:pull:`1914`) by :user:`schlunma`

Variable Derivation
~~~~~~~~~~~~~~~~~~~

- Add derivation of sea ice extent (:pull:`1695`) by :user:`sloosvel`


.. _changelog-v2-7-1:

------
Highlights
~~~~~~~~~~

This is a bugfix release where we unpin `cf-units` to allow the latest `iris=3.4.0` to be installed. It also includes an update to the default configuration used when searching the ESGF for files, to account for a recent change of the CEDA ESGF index node hostname. The changelog contains only changes that were made to the ``main`` branch.

Installation
~~~~~~~~~~~~

- Set the version number on the development branches to one minor version more than the previous release (:pull:`1854`) by :user:`bouweandela`
- Unpin cf-units (:pull:`1770`) by :user:`bouweandela`

Bug fixes
~~~~~~~~~

- Improve error handling if an esgf index node is offline (:pull:`1834`) by :user:`bouweandela`

Automatic testing
~~~~~~~~~~~~~~~~~

- Removed unnecessary test that fails with iris 3.4.0 (:pull:`1846`) by :user:`schlunma`
- Update CEDA ESGF index node hostname (:pull:`1838`) by :user:`valeriupredoi`


.. _changelog-v2-7-0: