Climpred

============================

New Features
------------
- :py:meth:`.HindcastEnsemble.verify`,
:py:meth:`.PerfectModelEnsemble.verify`,
:py:meth:`.HindcastEnsemble.bootstrap` and
:py:meth:`.PerfectModelEnsemble.bootstrap`
accept reference ``climatology``. Furthermore, reference ``persistence`` also allows
probabilistic metrics (:issue:`202`, :issue:`565`, :pr:`566`) `Aaron Spring`_.
- Added new metric :py:class:`~climpred.metrics._roc` Receiver Operating
Characteristic as ``metric='roc'``. (:pr:`566`) `Aaron Spring`_.

Bug fixes
---------
- :py:meth:`.HindcastEnsemble.verify` and
:py:meth:`.HindcastEnsemble.bootstrap` accept ``dim`` as ``list``,
``set``, ``tuple`` or ``str`` (:issue:`519`, :pr:`558`) `Aaron Spring`_.
- :py:meth:`.PredictionEnsemble.map` now does not fail silently when
applying a function to all ``xr.Datasets`` of
:py:class:`.PredictionEnsemble`. Instead, ``UserWarning``s are
raised. Furthermore, ``PredictionEnsemble.map(func, *args, **kwargs)``
applies only function to Datasets with matching dims if ``dim="dim0_or_dim1"`` is
passed as ``**kwargs``. (:issue:`417`, :issue:`437`, :pr:`552`) `Aaron Spring`_.
- :py:class:`~climpred.metrics._rpc` was fixed in ``xskillscore>=0.0.19`` and hence is
not falsely limited to 1 anymore (:issue:`562`, :pr:`566`) `Aaron Spring`_.

Internals/Minor Fixes
---------------------
- Docstrings are now tested in GitHub actions continuous integration.
(:issue:`545`, :pr:`560`) `Aaron Spring`_.
- Github actions now cancels previous commits, instead of running the full
testing suite on every single commit. (:pr:`560`) `Aaron Spring`_.
- :py:meth:`.PerfectModelEnsemble.verify` does not add
climpred attributes to skill by default anymore.
(:pr:`560`) `Aaron Spring`_.
- Drop ``python==3.6`` support. (:pr:`573`) `Aaron Spring`_.
- Notebooks are now linted with
`nb_black <https://github.com/dnanhkhoa/nb_black>`_ using
``%load_ext nb_black`` or ``%load_ext lab_black`` for
`Jupyter <https://jupyter.org>`_ notebooks and
`Jupyter <https://jupyter.org>`_ lab.
(:issue:`526`, :pr:`572`) `Aaron Spring`_.
- Reduce dependencies to install climpred.
(:issue:`454`, :pr:`572`) `Aaron Spring`_.
- Examples from documentation available via `Binder <https://mybinder.org/v2/gh/pangeo-data/climpred/master?urlpath=lab%2Ftree%2Fdocs%2Fsource%2Fquick-start.ipynb>`_.
Find further examples in the ``examples`` folder.
(:issue:`549`, :pr:`578`) `Aaron Spring`_.
- Rename branch ``master`` to ``main``. (:pr:`579`) `Aaron Spring`_.

============================

This release is the fixed version for our Journal of Open Source Software (JOSS)
article about ``climpred``, see `review
<https://github.com/openjournals/joss-reviews/issues/2781>`_.

New Features
------------
- Function to calculate predictability horizon
:py:func:`~climpred.predictability_horizon.predictability_horizon` based on condition.
(:issue:`46`, :pr:`521`) `Aaron Spring`_.

Bug fixes
---------
- :py:meth:`.PredictionEnsemble.smooth` now carries ``lead.attrs``
(:issue:`527`, pr:`521`) `Aaron Spring`_.
- :py:meth:`.PerfectModelEnsemble.verify` now works with ``references``
also for geospatial inputs, which returned ``NaN`` before.
(:issue:`522`, pr:`521`) `Aaron Spring`_.
- :py:meth:`.PredictionEnsemble.plot` now shifts composite lead
frequencies like ``days``, ``pentads``, ``seasons`` correctly.
(:issue:`532`, :pr:`533`) `Aaron Spring`_.
- Adapt to ``xesmf>=0.5.2`` for spatial xesmf smoothing. (:issue:`543`, :pr:`548`)
`Aaron Spring`_.
- :py:meth:`.HindcastEnsemble.remove_bias` now carries attributes.
(:issue:`531`, :pr:`551`) `Aaron Spring`_.

============================

Breaking changes
----------------
This version introduces a lot of breaking changes. We are trying to overhaul ``climpred`` to have an intuitive API that also forces users to think about methodology choices when running functions. The main breaking changes we introduced are for :py:meth:`.HindcastEnsemble.verify` and :py:meth:`.PerfectModelEnsemble.verify`. Now, instead of assuming defaults for most keywords, we require the user to define ``metric``, ``comparison``, ``dim``, and ``alignment`` (for hindcast systems). We also require users to designate the number of ``iterations`` for bootstrapping.

- User now has to designate number of iterations with ``iterations=...`` in
:py:meth:`.HindcastEnsemble.bootstrap` (:issue:`384`, :pr:`436`)
`Aaron Spring`_ and `Riley X. Brady`_.
- Make ``metric``, ``comparison``, ``dim``, and ``alignment`` required (previous default
``None``) arguments for :py:meth:`.HindcastEnsemble.verify`
(:issue:`384`, :pr:`436`) `Aaron Spring`_ and `Riley X. Brady`_.
- Metric :py:class:`~climpred.metrics._brier_score` and
:py:func:`~climpred.metrics._threshold_brier_score` now requires callable keyword
argument ``logical`` instead of ``func`` (:pr:`388`) `Aaron Spring`_.
- :py:meth:`.HindcastEnsemble.verify` does not correct ``dim``
automatically to ``member`` for probabilistic metrics.
(:issue:`282`, :pr:`407`) `Aaron Spring`_.
- Users can no longer add multiple observations to
:py:class:`.HindcastEnsemble`. This will make current and future
development much easier on maintainers (:issue:`429`, :pr:`453`) `Riley X. Brady`_.
- Standardize the names of the output coordinates for
:py:meth:`.PredictionEnsemble.verify` and
:py:meth:`.PredictionEnsemble.bootstrap` to ``initialized``,
``uninitialized``, and ``persistence``. ``initialized`` showcases the metric result
after comparing the initialized ensemble to the verification data; ``uninitialized``
when comparing the uninitialized (historical) ensemble to the verification data;
``persistence`` is the evaluation of the persistence forecast
(:issue:`460`, :pr:`478`, :issue:`476`, :pr:`480`) `Aaron Spring`_.
- ``reference`` keyword in :py:meth:`.HindcastEnsemble.verify` should
be choosen from [``uninitialized``, ``persistence``]. ``historical`` no longer works.
(:issue:`460`, :pr:`478`, :issue:`476`, :pr:`480`) `Aaron Spring`_.
- :py:meth:`.HindcastEnsemble.verify` returns no ``skill`` dimension
if ``reference=None`` (:pr:`480`) `Aaron Spring`_.
- ``comparison`` is not applied to uninitialized skill in
:py:meth:`.HindcastEnsemble.bootstrap`.
(:issue:`352`, :pr:`418`) `Aaron Spring`_.

New Features
------------
This release is accompanied by a bunch of new features. Math operations can now be used with our :py:class:`.PredictionEnsemble` objects and their variables can be sub-selected. Users can now quick plot time series forecasts with these objects. Bootstrapping is available for :py:class:`.HindcastEnsemble`. Spatial dimensions can be passed to metrics to do things like pattern correlation. New metrics have been implemented based on Contingency tables. We now include an early version of bias removal for :py:class:`.HindcastEnsemble`.

- Use math operations like ``+-*/`` with :py:class:`.HindcastEnsemble`
and :py:class:`.PerfectModelEnsemble`. See
`demo <prediction-ensemble-object.html>`_
Arithmetic-Operations-with-PredictionEnsemble-Objects. (:pr:`377`) `Aaron Spring`_.
- Subselect data variables from :py:class:`.PerfectModelEnsemble` as
from :py:class:`xarray.Dataset`:
``PredictionEnsemble[["var1", "var3"]]`` (:pr:`409`) `Aaron Spring`_.
- Plot all datasets in :py:class:`.HindcastEnsemble` or
:py:class:`.PerfectModelEnsemble` by
:py:meth:`.PredictionEnsemble.plot` if no other spatial dimensions
are present. (:pr:`383`) `Aaron Spring`_.
- Bootstrapping now available for :py:class:`.HindcastEnsemble` as
:py:meth:`.HindcastEnsemble.bootstrap`, which is analogous to
the :py:class:`.PerfectModelEnsemble` method.
(:issue:`257`, :pr:`418`) `Aaron Spring`_.
- :py:meth:`.HindcastEnsemble.verify` allows all dimensions from
``initialized`` ensemble as ``dim``. This allows e.g. spatial dimensions to be used
for pattern correlation. Make sure to use ``skipna=True`` when using spatial
dimensions and output has NaNs (in the case of land, for instance).
(:issue:`282`, :pr:`407`) `Aaron Spring`_.
- Allow binary forecasts at when calling
:py:meth:`.HindcastEnsemble.verify`,
rather than needing to supply binary results beforehand. In other words,
``hindcast.verify(metric='bs', comparison='m2o', dim='member', logical=logical)``
is now the same as
``hindcast.map(logical).verify(metric='brier_score', comparison='m2o', dim='member'``.
(:pr:`431`) `Aaron Spring`_.
- Check ``calendar`` types when using
:py:meth:`.HindcastEnsemble.add_observations`,
:py:meth:`.HindcastEnsemble.add_uninitialized`,
:py:meth:`.PerfectModelEnsemble.add_control` to ensure that the
verification data calendars match that of the initialized ensemble.
(:issue:`300`, :pr:`452`, :issue:`422`, :pr:`462`)
`Riley X. Brady`_ and `Aaron Spring`_.
- Implement new metrics which have been ported over from
https://github.com/csiro-dcfp/doppyo/ to ``xskillscore`` by `Dougie Squire`_.
(:pr:`439`, :pr:`456`) `Aaron Spring`_

* rank histogram :py:func:`~climpred.metrics._rank_histogram`
* discrimination :py:func:`~climpred.metrics._discrimination`
* reliability :py:func:`~climpred.metrics._reliability`
* ranked probability score :py:func:`~climpred.metrics._rps`
* contingency table and related scores :py:func:`~climpred.metrics._contingency`

- Perfect Model :py:meth:`.PerfectModelEnsemble.verify`
no longer requires ``control`` in :py:class:`.PerfectModelEnsemble`.
It is only required when ``reference=['persistence']``. (:pr:`461`) `Aaron Spring`_.
- Implemented bias removal
:py:class:`~climpred.classes.HindcastEnsemble.remove_bias`.
``remove_bias(how='mean')`` removes the mean bias of initialized hindcasts with
respect to observations. See `example <bias_removal.html>`__.
(:pr:`389`, :pr:`443`, :pr:`459`) `Aaron Spring`_ and `Riley X. Brady`_.

Depreciated
-----------
- ``spatial_smoothing_xrcoarsen`` no longer used for spatial smoothing.
(:pr:`391`) `Aaron Spring`_.
- ``compute_metric``, ``compute_uninitialized`` and ``compute_persistence`` no longer
in use for :py:class:`.PerfectModelEnsemble` in favor of
:py:meth:`.PerfectModelEnsemble.verify` with the ``reference``
keyword instead.
(:pr:`436`, :issue:`468`, :pr:`472`) `Aaron Spring`_ and `Riley X. Brady`_.
- ``'historical'`` no longer a valid choice for ``reference``. Use ``'uninitialized'``
instead. (:pr:`478`) `Aaron Spring`_.

Bug Fixes
---------
- :py:meth:`.PredictionEnsemble.verify` and
:py:meth:`.PredictionEnsemble.bootstrap` now accept
``metric_kwargs``. (:pr:`387`) `Aaron Spring`_.
- :py:meth:`.PerfectModelEnsemble.verify` now accepts
``'uninitialized'`` as a reference. (:pr:`395`) `Riley X. Brady`_.
- Spatial and temporal smoothing :py:meth:`.PredictionEnsemble.smooth`
now work as expected and rename time dimensions after
:py:meth:`~climpred.classes.PredictionEnsembleEnsemble.verify`.
(:pr:`391`) `Aaron Spring`_.
- ``PredictionEnsemble.verify(comparison='m2o', references=['uninitialized',
'persistence']`` does not fail anymore. (:issue:`385`, :pr:`400`) `Aaron Spring`_.
- Remove bias using ``dayofyear`` in
:py:meth:`.HindcastEnsemble.reduce_bias`.
(:pr:`443`) `Aaron Spring`_.
- ``climpred`` works with ``dask=>2.28``. (:issue:`479`, :pr:`482`) `Aaron Spring`_.

Documentation
-------------
- Updates ``climpred`` tagline to "Verification of weather and climate forecasts."
(:pr:`420`) `Riley X. Brady`_.
- Adds section on how to use arithmetic with
:py:class:`.HindcastEnsemble`.
(:pr:`378`) `Riley X. Brady`_.
- Add docs section for similar open-source forecasting packages.
(:pr:`432`) `Riley X. Brady`_.
- Add all metrics to main API in addition to metrics page.
(:pr:`438`) `Riley X. Brady`_.
- Add page on bias removal `Aaron Spring`_.

Internals/Minor Fixes
---------------------
- :py:meth:`.PredictionEnsemble.verify` replaces deprecated
``PerfectModelEnsemble.compute_metric()`` and accepts ``reference`` as keyword.
(:pr:`387`) `Aaron Spring`_.
- Cleared out unnecessary statistics functions from ``climpred`` and migrated them to
``esmtools``. Add ``esmtools`` as a required package. (:pr:`395`) `Riley X. Brady`_.
- Remove fixed pandas dependency from ``pandas=0.25`` to stable ``pandas``.
(:issue:`402`, :pr:`403`) `Aaron Spring`_.
- ``dim`` is expected to be a list of strings in
:py:func:`~climpred.prediction.compute_perfect_model` and
:py:func:`~climpred.prediction.compute_hindcast`.
(:issue:`282`, :pr:`407`) `Aaron Spring`_.
- Update ``cartopy`` requirement to 0.0.18 or greater to release lock on
``matplotlib`` version. Update ``xskillscore`` requirement to 0.0.18 to
cooperate with new ``xarray`` version. (:pr:`451`, :pr:`449`)
`Riley X. Brady`_
- Switch from Travis CI and Coveralls to Github Actions and CodeCov.
(:pr:`471`) `Riley X. Brady`_
- Assertion functions added for :py:class:`.PerfectModelEnsemble`:
:py:func:`~climpred.testing.assert_PredictionEnsemble`. (:pr:`391`) `Aaron Spring`_.
- Test all metrics against synthetic data. (:pr:`388`) `Aaron Spring`_.

============================

Breaking Changes
----------------
- Keyword ``bootstrap`` has been replaced with ``iterations``. We feel that this more
accurately describes the argument, since "bootstrap" is really the process as a whole.
(:pr:`354`) `Aaron Spring`_.

New Features
------------
- :py:class:`.HindcastEnsemble` and
:py:class:`.PerfectModelEnsemble` now use an HTML representation,
following the more recent versions of ``xarray``. (:pr:`371`) `Aaron Spring`_.
- ``HindcastEnsemble.verify()`` now takes ``reference=...`` keyword. Current options are
``'persistence'`` for a persistence forecast of the observations and
``'uninitialized'`` for an uninitialized/historical reference, such as an
uninitialized/forced run. (:pr:`341`) `Riley X. Brady`_.
- We now only enforce a union of the initialization dates with observations if
``reference='persistence'`` for :py:class:`.HindcastEnsemble`.
This is to ensure that the same set of initializations is used by the observations to
construct a persistence forecast. (:pr:`341`) `Riley X. Brady`_.
- :py:func:`~climpred.prediction.compute_perfect_model` now accepts initialization
(``init``) as ``cftime`` and ``int``. ``cftime`` is now implemented into the
bootstrap uninitialized functions for the perfect model configuration.
(:pr:`332`) `Aaron Spring`_.
- New explicit keywords in bootstrap functions for ``resampling_dim`` and
``reference_compute`` (:pr:`320`) `Aaron Spring`_.
- Logging now included for ``compute_hindcast`` which displays the ``inits`` and
verification dates used at each lead (:pr:`324`) `Aaron Spring`_,
(:pr:`338`) `Riley X. Brady`_. See (`logging <alignment.htmlLogging>`__).
- New explicit keywords added for ``alignment`` of verification dates and
initializations. (:pr:`324`) `Aaron Spring`_. See (`alignment <alignment.html>`__)

* ``'maximize'``: Maximize the degrees of freedom by slicing ``hind`` and
``verif`` to a common time frame at each lead. (:pr:`338`) `Riley X. Brady`_.
* ``'same_inits'``: slice to a common init frame prior to computing
metric. This philosophy follows the thought that each lead should be
based on the same set of initializations. (:pr:`328`) `Riley X. Brady`_.
* ``'same_verifs'``: slice to a common/consistent verification time frame prior
to computing metric. This philosophy follows the thought that each lead
should be based on the same set of verification dates. (:pr:`331`)
`Riley X. Brady`_.

Performance
-----------
The major change for this release is a dramatic speedup in bootstrapping functions, led by `Aaron Spring`_. We focused on scalability with ``dask`` and found many places we could compute skill simultaneously over all bootstrapped ensemble members rather than at each iteration.

- Bootstrapping uninitialized skill in the perfect model framework is now sped up
significantly for annual lead resolution. (:pr:`332`) `Aaron Spring`_.
- General speedup in :py:func:`~climpred.bootstrap.bootstrap_hindcast` and
:py:func:`~climpred.bootstrap.bootstrap_perfect_model`: (:pr:`285`) `Aaron Spring`_.

* Properly implemented handling for lazy results when inputs are chunked.

* User gets warned when chunking potentially unnecessarily and/or inefficiently.

Bug Fixes
---------
- Alignment options now account for differences in the historical time series if
``reference='historical'``. (:pr:`341`) `Riley X. Brady`_.

Internals/Minor Fixes
---------------------
- Added a `Code of Conduct <code_of_conduct.html>`__ (:pr:`285`) `Aaron Spring`_.
- Gather ``pytest.fixture in ``conftest.py``. (:pr:`313`) `Aaron Spring`_.
- Move ``x_METRICS`` and ``COMPARISONS`` to ``metrics.py`` and ``comparisons.py`` in
order to avoid circular import dependencies. (:pr:`315`) `Aaron Spring`_.
- ``asv`` benchmarks added for :py:class:`.HindcastEnsemble`
(:pr:`285`) `Aaron Spring`_.
- Ignore irrelevant warnings in ``pytest`` and mark slow tests
(:pr:`333`) `Aaron Spring`_.
- Default ``CONCAT_KWARGS`` now in all ``xr.concat`` to speed up bootstrapping.
(:pr:`330`) `Aaron Spring`_.
- Remove ``member`` coords for ``m2c`` comparison for probabilistic metrics.
(:pr:`330`) `Aaron Spring`_.
- Refactored :py:func:`~climpred.prediction.compute_hindcast` and
:py:func:`~climpred.prediction.compute_perfect_model`. (:pr:`330`) `Aaron Spring`_.
- Changed lead0 coordinate modifications to be compliant with ``xarray=0.15.1`` in
:py:func:`~climpred.reference.compute_persistence`. (:pr:`348`) `Aaron Spring`_.
- Exchanged ``my_quantile`` with ``xr.quantile(skipna=False)``.
(:pr:`348`) `Aaron Spring`_.
- Remove ``sig`` from
:py:func:`~climpred.graphics.plot_bootstrapped_skill_over_leadyear`.
(:pr:`351`) `Aaron Spring`_.
- Require ``xskillscore v0.0.15`` and use their functions for effective sample
size-based metrics. (:pr: `353`) `Riley X. Brady`_.
- Faster bootstrapping without replacement used in threshold functions of
``climpred.stats`` (:pr:`354`) `Aaron Spring`_.
- Require ``cftime v1.1.2``, which modifies their object handling to create 200-400x
speedups in some basic operations. (:pr:`356`) `Riley X. Brady`_.
- Resample first and then calculate skill in
:py:func:`~climpred.bootstrap.bootstrap_perfect_model` and
:py:func:`~climpred.bootstrap.bootstrap_hindcast` (:pr:`355`) `Aaron Spring`_.

Documentation
-------------
- Added demo to setup your own raw model output compliant to ``climpred``
(:pr:`296`) `Aaron Spring`_. See (`here <examples/misc/setup_your_own_data.html>`__).
- Added demo using ``intake-esm`` with ``climpred``.
See `demo <examples/misc/setup_your_own_data.htmlintake-esm-for-cmorized-output>`__.
(:pr:`296`) `Aaron Spring`_.
- Added `Verification Alignment <alignment.html>`_ page explaining how initializations
are selected and aligned with verification data. (:pr:`328`) `Riley X. Brady`_.
See (`here <alignment.html>`__).

============================

New Features
------------
- Add support for ``days``, ``pentads``, ``weeks``, ``months``, ``seasons`` for lead
time resolution. ``climpred`` now requires a ``lead`` attribute "units" to decipher
what resolution the predictions are at. (:pr:`294`) `Kathy Pegion`_ and
`Riley X. Brady`_.

.. :: python

>>> hind = climpred.tutorial.load_dataset("CESM-DP-SST")
>>> hind.lead.attrs["units"] = "years"

- :py:class:`.HindcastEnsemble` now has
:py:meth:`.HindcastEnsemble.add_observations` and
:py:meth:`.HindcastEnsemble.get_observations`
methods. These are the same as ``.add_reference()`` and ``.get_reference()``, which
will be deprecated eventually. The name change clears up confusion, since "reference"
is the appropriate name for a reference forecast, e.g. ``"persistence"``. (:pr:`310`)
`Riley X. Brady`_.

- :py:class:`.HindcastEnsemble` now has ``.verify()`` function, which
duplicates the ``.compute_metric()`` function. We feel that ``.verify()`` is more
clear and easy to write, and follows the terminology of the field.
(:pr:`310`) `Riley X. Brady`_.

- ``e2o`` and ``m2o`` are now the preferred keywords for comparing hindcast ensemble
means and ensemble members to verification data, respectively. (:pr:`310`)
`Riley X. Brady`_.

Documentation
-------------
- New example pages for subseasonal-to-seasonal prediction using ``climpred``.
(:pr:`294`) `Kathy Pegion`_

* Calculate the skill of the MJO index as a function of lead time
(`link <examples/subseasonal/daily-subx-example.html>`__).

* Calculate the skill of the MJO index as a function of lead time for weekly data
(`link <examples/subseasonal/weekly-subx-example.html>`__).

* Calculate ENSO skill as a function of initial month vs. lead time
(`link <examples/monseas/monthly-enso-subx-example.html>`__).

* Calculate Seasonal ENSO skill
(`link <examples/monseas/seasonal-enso-subx-example.html>`__).

- `Comparisons <comparisons.html>`__ page rewritten for more clarity. (:pr:`310`)
`Riley X. Brady`_.

Bug Fixes
---------
- Fixed `m2m` broken comparison issue and removed correction.
(:pr:`290`) `Aaron Spring`_.

Internals/Minor Fixes
---------------------
- Updates to ``xskillscore`` v0.0.12 to get a 30-50% speedup in compute functions that
rely on metrics from there. (:pr:`309`) `Riley X. Brady`_.
- Stacking dims is handled by ``comparisons``, no need for internal keyword
``stack_dims``. Therefore ``comparison`` now takes ``metric`` as argument instead.
(:pr:`290`) `Aaron Spring`_.
- ``assign_attrs`` now carries `dim` (:pr:`290`) `Aaron Spring`_.
- ``reference`` changed to ``verif`` throughout hindcast compute functions. This is more
clear, since ``reference`` usually refers to a type of forecast, such as persistence.
(:pr:`310`) `Riley X. Brady`_.
- ``Comparison`` objects can now have aliases. (:pr:`310`) `Riley X. Brady`_.

============================

Depreciated
-----------
- ``mad`` no longer a keyword for the median absolute error metric. Users should now
use ``median_absolute_error``, which is identical to changes in ``xskillscore``
version 0.0.10. (:pr:`283`) `Riley X. Brady`_
- ``pacc`` no longer a keyword for the p value associated with the Pearson
product-moment correlation, since it is used by the correlation coefficient.
(:pr:`283`) `Riley X. Brady`_
- ``msss`` no longer a keyword for the Murphy's MSSS, since it is reserved for the
standard MSSS. (:pr:`283`) `Riley X. Brady`_

New Features
------------
- Metrics ``pearson_r_eff_p_value`` and ``spearman_r_eff_p_value`` account for
autocorrelation in computing p values. (:pr:`283`) `Riley X. Brady`_
- Metric ``effective_sample_size`` computes number of independent samples between two
time series being correlated. (:pr:`283`) `Riley X. Brady`_
- Added keywords for metrics: (:pr:`283`) `Riley X. Brady`_

* ``'pval'`` for ``pearson_r_p_value``
* ``['n_eff', 'eff_n']`` for ``effective_sample_size``
* ``['p_pval_eff', 'pvalue_eff', 'pval_eff']`` for ``pearson_r_eff_p_value``
* ``['spvalue', 'spval']`` for ``spearman_r_p_value``
* ``['s_pval_eff', 'spvalue_eff', 'spval_eff']`` for ``spearman_r_eff_p_value``
* ``'nev'`` for ``nmse``

Internals/Minor Fixes
---------------------
- ``climpred`` now requires ``xarray`` version 0.14.1 so that the ``drop_vars()``
keyword used in our package does not throw an error. (:pr:`276`) `Riley X. Brady`_
- Update to ``xskillscore`` version 0.0.10 to fix errors in weighted metrics with
pairwise NaNs. (:pr:`283`) `Riley X. Brady`_
- ``doc8`` added to ``pre-commit`` to have consistent formatting on ``.rst`` files.
(:pr:`283`) `Riley X. Brady`_
- Remove ``proper`` attribute on ``Metric`` class since it isn't used anywhere.
(:pr:`283`) `Riley X. Brady`_
- Add testing for effective p values. (:pr:`283`) `Riley X. Brady`_
- Add testing for whether metric aliases are repeated/overwrite each other.
(:pr:`283`) `Riley X. Brady`_
- ``ppp`` changed to ``msess``, but keywords allow for ``ppp`` and ``msss`` still.
(:pr:`283`) `Riley X. Brady`_

Documentation
-------------
- Expansion of `metrics documentation <metrics.html>`_ with much more
detail on how metrics are computed, their keywords, references, min/max/perfect
scores, etc. (:pr:`283`) `Riley X. Brady`_
- Update `terminology page <terminology.html>`_ with more information on metrics
terminology. (:pr:`283`) `Riley X. Brady`_