Heat

In the following, we assume we are about to release Heat v1.3.0.

**PRE-REQUISITES:**

- You need [PyPi](https://pypi.org/), [Test.PyPi](https://test.pypi.org/) account
- All intended PRs are merged, all tests have passed, and the `main` branch is ready for release.

1. We will release all new features in the development branch `main`. Branch off `main` to create a new release branch, e.g.:

bash
git checkout main
git pull
git checkout -b release/1.3.x


2. Update `heat/core/version.py` like this:

python
"""This module contains Heat's version information."""

major: int = 1
"""Indicates Heat's main version."""
minor: int = 2 <-- update to 3
"""Indicates feature extension."""
micro: int = 2 <-- update to 0
"""Indicates revisions for bugfixes."""
extension: str = "dev" <-- set to None
"""Indicates special builds, e.g. for specific hardware."""


3. Commit and push new `version.py` in `release/1.3.x`

4. Update `CITATION.cff` if needed, i.e. add names of non-core contributors.

5. Draft release notes:

- Go to the GitHub repo's [Releases](https://github.com/helmholtz-analytics/heat/releases) page.
- The release notes draft is automated. Click on Edit Draft
- Select the new tag: v1.3.0. Modify Target branch: `release/1.3.x`
- Edit release notes as needed (see older releases)
- Click on Save but do not publish yet

5. Build wheel in your local `heat/` directory, make sure you are on branch `release/1.3.x`.

bash
rm -f dist/*
python -m build


You might have to install the `build` package first (i.e. with `conda install build` or `pip install build`)

6. Upload to Test PyPI and verify things look right. You need to install `twine` first.

bash
twine upload -r testpypi dist/*


`twine` will prompt for your username and password.

7. When everything works, upload to PyPI:

bash
twine upload dist/*


8. Update conda-forge recipe (Need to be listed as maintainer, either ClaudiaComito, mrfh92, JuanPedroGHM)
- Go to https://github.com/conda-forge/heat-feedstock
- A new PR should have been automatically created.
- Changes can be pushed to the PR.
- Make sure the version number is correct.
- Make sure the SHA points to the correct PyPI release.
- Make sure dependencies match.
- Once the PR is done, wait for the CI checks to finish and merge.
- Refer to the conda-forge docs if there are any issues: https://conda-forge.org/docs/maintainer/updating_pkgs.html#pushing-to-regro-cf-autotick-bot-branch

9. Go back to the Release Notes draft and publish them. The new release is out!

- Make sure the CHANGELOG.md got updated, if not, call JuanPedroGHM.
- Check Zenodo to make sure a DOI was created for the release.

10. Now we want to update `main` to include the latest release, we want to modify the version on main so that `minor` is increased by 1, and `extension` is "dev". In this example we want the version on `main` to be:`1.4.0-dev`.

bash
git checkout main
git pull
git checkout -b workflows/update-version-main
git merge release/1.3.x --no-ff --no-commit


Modify `version.py` so that `extension` is `"dev"`. Commit and push the changes.

12. Create a PR with `main` as the base branch.


13. Get approval and merge. You're done!

Patch release

1. Check that all intended branches have been merged to the release branch you want to upgrade, in this example `release/1.3.x`. Branch off from `release/1.3.x` and create a new branch:

bash
git checkout release/1.3.x
git pull
git checkout -b minor-version-update


2. Update `heat/core/version.py` like this:

python
"""This module contains Heat's version information."""

major: int = 1
"""Indicates Heat's main version."""
minor: int = 3
"""Indicates feature extension."""
micro: int = 0 <-- update to 1
"""Indicates revisions for bugfixes."""
extension: str = None
"""Indicates special builds, e.g. for specific hardware."""


3. Commit and push new `version.py` in `minor-version-update`

4. Create a pull request from `minor-version-update` to `release/1.3.x`

- Remember to get a reviewers approval.
- Wait for the test to finish.
- Squash and merge.


5. Draft release notes:

- Go to the GitHub repo's [Releases](https://github.com/helmholtz-analytics/heat/releases) page.
- The release notes draft is automated. Click on Edit Draft
- Select the new tag: `v1.3.1`
- Edit release notes as needed (see older releases)
- Save the changes to the release notes WITHOUT PUBLISHING!!

6. Build wheel in your local `heat/` directory, make sure you are on branch `release/1.3.x`.

bash
rm -f dist/*
python -m build


You might have to install the `build` package first (i.e. with `conda install build` or `pip install build`)

7. Upload to Test PyPI and verify things look right. You need to install `twine` first.

bash
twine upload -r testpypi dist/*


`twine` will prompt for your username and password.

- Look at the testpypi repository and make sure everything is correct (version number, readme, etc.)

8. When everything works, upload to PyPI:

bash
twine upload dist/*


9. Update conda-forge recipe (Need to be listed as maintainer, either ClaudiaComito, mrfh92, JuanPedroGHM)
- Go to https://github.com/conda-forge/heat-feedstock
- A new PR should have been automatically created.
- Changes can be pushed to the PR.
- Make sure the version number is correct.
- Make sure the SHA points to the correct PyPI release.
- Make sure dependencies match.
- Once the PR is done, wait for the CI checks to finish and merge.
- Refer to the conda-forge docs if there are any issues: https://conda-forge.org/docs/maintainer/updating_pkgs.html#pushing-to-regro-cf-autotick-bot-branch


10. Go back to the Release Notes draft and publish them. The new release is out!
- Make sure the CHANGELOG.md got updated in the release branch, in this case `release/1.3.x`, if not, call JuanPedroGHM.
- Check Zenodo to make sure a DOI was created for the release.

11. Now we want to update `main` to the latest version, and we want the version on `main` to be `1.4.0-dev`.
- Create a new branch from `release/1.3.x`, for example `merge-latest-release-into-main`
- Merge `main` into `merge-latest-release-into-main`, resolve conflicts and push.
- Create a PR from `merge-latest-release-into-main`, base branch must be `main`
- Make sure the version number in `merge-latest-release-into-main` is correct (i.e., it matches that in `main`).
- Make sure the CHANGELOG.md in `merge-latest-release-into-main` matches that in the latest release branch, in this example`release/1.3.x`.
- Get a reviewers approval, wait for the CI checks to pass, and merge.

This release includes many important updates (see below). We particularly would like to thank our enthusiastic [GSoC2022](https://summerofcode.withgoogle.com/programs/2022) / tentative GSoC2023 contributors Mystic-Slice neosunhan Sai-Suraj-27 shahpratham AsRaNi1 Ishaan-Chandak 🙏🏼 Thank you so much!

Highlights
- 1155 Support PyTorch 2.0.1 (by ClaudiaComito)
- 1152 Support AMD GPUs (by mtar)
- 1126 [Distributed hierarchical SVD](https://helmholtz-analytics.github.io/heat/2023/06/16/new-feature-hsvd.html) (by mrfh92)
- 1028 Introducing the `sparse` module: Distributed Compressed Sparse Row Matrix (by Mystic-Slice)
- Performance improvements:
- 1125 distributed `heat.reshape()` speed-up (by ClaudiaComito)
- 1141 `heat.pow()` speed-up when exponent is `int` (by ClaudiaComito coquelin77 )
- 1119 `heat.array()` default to `copy=None` (e.g., only if necessary) (by ClaudiaComito neosunhan )
- 970 [Dockerfile and accompanying documentation](https://github.com/helmholtz-analytics/heat/tree/release/1.3.x/docker) (by bhagemeier)

Changelog

Array-API compliance / Interoperability

- 1154 Introduce `DNDarray.__array__()` method for interoperability with `numpy`, `xarray` (by ClaudiaComito)
- 1147 Adopt [NEP29](https://numpy.org/neps/nep-0029-deprecation_policy.html), drop support for PyTorch 1.7, Python 3.6 (by mtar)
- 1119 `ht.array()` default to `copy=None` (e.g., only if necessary) (by ClaudiaComito)
- 1020 Implement `broadcast_arrays`, `broadcast_to` (by neosunhan)
- 1008 API: Rename `keepdim` kwarg to `keepdims` (by neosunhan)
- 788 Interface for [DPPY](https://github.com/IntelPython/DPPY-Spec) interoperability (by coquelin77 fschlimb )

New Features
- 1126 [Distributed hierarchical SVD](https://helmholtz-analytics.github.io/heat/2023/06/16/new-feature-hsvd.html) (by mrfh92)
- 1020 Implement `broadcast_arrays`, `broadcast_to` (by neosunhan)
- 983 Signal processing: fully distributed 1D convolution (by shahpratham)
- 1063 add __eq__ to Device (by mtar)

Bug Fixes

- 1141 `heat.pow()` speed-up when exponent is `int` (by ClaudiaComito)
- 1136 Fixed PyTorch version check in `sparse` module (by Mystic-Slice)
- 1098 Validates number of dimensions in input to `ht.sparse.sparse_csr_matrix` (by Ishaan-Chandak)
- 1095 Convolve with distributed kernel on multiple GPUs (by shahpratham)
- 1094 Fix division precision error in `random` module (by Mystic-Slice)
- 1075 Fixed initialization of DNDarrays communicator in some routines (by AsRaNi1)
- 1066 Verify input object type and layout + Supporting tests (by Mystic-Slice)
- 1037 Distributed weighted `average()` along tuple of axes: shape of `weights` to match shape of input (by Mystic-Slice)

Benchmarking

- 1137 Continous Benchmarking of runtime (by JuanPedroGHM)

Documentation

- 1150 Refactoring for efficiency and readability (by Sai-Suraj-27)
- 1130 Reintroduce Quick Start (by ClaudiaComito)
- 1079 A better README file (by Sai-Suraj-27)


Linear Algebra

- 1126, 1160 [Distributed hierarchical SVD](https://helmholtz-analytics.github.io/heat/2023/06/16/new-feature-hsvd.html) (by mrfh92 ClaudiaComito )

Changes

Communication

- 1058 Fix edge-case contiguity mismatch for Allgatherv (by ClaudiaComito)

Contributors

ClaudiaComito, JuanPedroGHM

Changes

- 1048 Support PyTorch 1.13.0 on branch release/1.2.x (by github-actions)

🐛 Bug Fixes

- 1038 Lanczos decomposition `linalg.solver.lanczos`: Support double precision, complex data types (by ClaudiaComito)
- 1034 `ht.array`, closed loophole allowing `DNDarray` construction with incompatible shapes of local arrays (by Mystic-Slice)

Linear Algebra

- 1038 Lanczos decomposition `linalg.solver.lanczos`: Support double precision, complex data types (by ClaudiaComito)

🧪 Testing

- 1025 mirror repository on gitlab + ci (by mtar)
- 1014 fix: set cuda rng state on gpu tests for test_random.py (by JuanPedroGHM)

Highlights
- [906](https://github.com/helmholtz-analytics/heat/pull/906) PyTorch 1.11 support
- [595](https://github.com/helmholtz-analytics/heat/pull/595) Distributed 1-D convolution: `ht.convolve`
- [941](https://github.com/helmholtz-analytics/heat/pull/941) Parallel I/O: write to CSV file with `ht.save_csv`.
- [887](https://github.com/helmholtz-analytics/heat/pull/887) Binary operations between operands of equal shapes, equal `split` axes, but different distribution maps.
- Expanded memory-distributed [linear algebra](linalg), [manipulations](manipulations) modules.

Bug Fixes
- [826](https://github.com/helmholtz-analytics/heat/pull/826) Fixed `__setitem__` handling of distributed `DNDarray` values which have a different shape in the split dimension
- [846](https://github.com/helmholtz-analytics/heat/pull/846) Fixed an issue in `_reduce_op` when axis and keepdim were set.
- [846](https://github.com/helmholtz-analytics/heat/pull/846) Fixed an issue in `min`, `max` where DNDarrays with empty processes can't be computed.
- [868](https://github.com/helmholtz-analytics/heat/pull/868) Fixed an issue in `__binary_op` where data was falsely distributed if a DNDarray has single element.
- [916](https://github.com/helmholtz-analytics/heat/pull/916) Fixed an issue in `random.randint` where the size parameter does not accept ints.

New features

Arithmetics
- [945](https://github.com/helmholtz-analytics/heat/pull/945) `ht.divide` now supports `out` and `where` kwargs
Communication
- [868](https://github.com/helmholtz-analytics/heat/pull/868) New `MPICommunication` method `Split`
- [940](https://github.com/helmholtz-analytics/heat/pull/940) and [#967](https://github.com/helmholtz-analytics/heat/pull/967) Duplicate `MPI.COMM_WORLD` and `MPI_SELF` to make library more independent.

DNDarray
- [856](https://github.com/helmholtz-analytics/heat/pull/856) New `DNDarray` method `__torch_proxy__`
- [885](https://github.com/helmholtz-analytics/heat/pull/885) New `DNDarray` method `conj`
<a name="linalg"></a> Linear Algebra
- [840](https://github.com/helmholtz-analytics/heat/pull/840) New feature: `vecdot()`
- [842](https://github.com/helmholtz-analytics/heat/pull/842) New feature: `vdot`
- [846](https://github.com/helmholtz-analytics/heat/pull/846) New features `norm`, `vector_norm`, `matrix_norm`
- [850](https://github.com/helmholtz-analytics/heat/pull/850) New Feature `cross`
- [877](https://github.com/helmholtz-analytics/heat/pull/877) New feature `det`
- [875](https://github.com/helmholtz-analytics/heat/pull/875) New feature `inv`
Logical
- [862](https://github.com/helmholtz-analytics/heat/pull/862) New feature `signbit`
<a name="manipulations"></a> Manipulations
- [829](https://github.com/helmholtz-analytics/heat/pull/829) New feature: `roll`
- [853](https://github.com/helmholtz-analytics/heat/pull/853) New Feature: `swapaxes`
- [854](https://github.com/helmholtz-analytics/heat/pull/854) New Feature: `moveaxis`
Printing
- [816](https://github.com/helmholtz-analytics/heat/pull/816) New feature: Local printing (`ht.local_printing()`) and global printing options
- [816](https://github.com/helmholtz-analytics/heat/pull/816) New feature: print only on process 0 with `print0(...)` and `ht.print0(...)`
Random
- [858](https://github.com/helmholtz-analytics/heat/pull/858) New Feature: `standard_normal`, `normal`
Rounding
- [827](https://github.com/helmholtz-analytics/heat/pull/827) New feature: `sign`, `sgn`
Statistics
- [928](https://github.com/helmholtz-analytics/heat/pull/928) New feature: `bucketize`, `digitize`
General
- [876](https://github.com/helmholtz-analytics/heat/pull/876) Fix examples (Lasso and kNN)
- [894](https://github.com/helmholtz-analytics/heat/pull/894) Change inclusion of license file
- [948](https://github.com/helmholtz-analytics/heat/pull/948) Improve CSV write performance.
- [960](https://github.com/helmholtz-analytics/heat/pull/960) Bypass unnecessary communication by replacing `factories.array` with` DNDarray` contruct in random.py