Accelerate

Improved Reproducibility

One critical issue with Accelerate is training runs were different when using an iterable dataset, no matter what seeds were set. v0.24.0 introduces the `dataloader.set_epoch()` function to all `Accelerate` `DataLoaders`, where if the underlying dataset (or sampler) has the ability to set the epoch for reproducability it will do so. This is similar to the implementation already existing in transformers. To use:

python
dataloader = accelerator.prepare(dataloader)
Say we want to resume at epoch/iteration 2
dataloader.set_epoch(2)


For more information see this [PR](https://github.com/huggingface/accelerate/pull/2057), we will update the docs on a subsequent release with more information on this API.

Documentation

* The quick tour docs have gotten a complete makeover thanks to MKhalusova. Take a look [here](https://hf.co/docs/accelerate/quicktour)
* We also now have documentation on how to perform multinode training, see the [launch docs](https://hf.co/docs/accelerate/basic_tutorials/launch)

Internal structure
* Shared file systems are now supported under `save` and `save_state` via the `ProjectConfiguration` dataclass. See 1953 for more info.
* FSDP can now be used for `bfloat16` mixed precision via `torch.autocast`
* `all_gather_into_tensor` is now used as the main gather operation, reducing memory in the cases of big tensors
* Specifying `drop_last=True` will now properly have the desired affect when performing `Accelerator().gather_for_metrics()`


What's Changed
* Update big_modeling.md by kli-casia in https://github.com/huggingface/accelerate/pull/1976
* Fix model copy after `dispatch_model` by austinapatel in https://github.com/huggingface/accelerate/pull/1971
* FIX: Automatic checkpoint path inference issue by BenjaminBossan in https://github.com/huggingface/accelerate/pull/1989
* Fix skip first batch for deepspeed example by SumanthRH in https://github.com/huggingface/accelerate/pull/2001
* [docs] Quick tour refactor by MKhalusova in https://github.com/huggingface/accelerate/pull/2008
* Add basic documentation for multi node training by SumanthRH in https://github.com/huggingface/accelerate/pull/1988
* update torch_dynamo backends by SunMarc in https://github.com/huggingface/accelerate/pull/1992
* Sync states for xpu fsdp by abhilash1910 in https://github.com/huggingface/accelerate/pull/2005
* update fsdp docs by pacman100 in https://github.com/huggingface/accelerate/pull/2026
* Enable shared file system with `save` and `save_state` via ProjectConfiguration by muellerzr in https://github.com/huggingface/accelerate/pull/1953
* Fix save on each node by muellerzr in https://github.com/huggingface/accelerate/pull/2036
* Allow FSDP to use with `torch.autocast` for bfloat16 mixed precision by brcps12 in https://github.com/huggingface/accelerate/pull/2033
* Fix DeepSpeed version to <0.11 by BenjaminBossan in https://github.com/huggingface/accelerate/pull/2043
* Unpin deepspeed by muellerzr in https://github.com/huggingface/accelerate/pull/2044
* Reduce memory by using `all_gather_into_tensor` by muellerzr in https://github.com/huggingface/accelerate/pull/1968
* Safely end training even if trackers weren't initialized by Ben-Epstein in https://github.com/huggingface/accelerate/pull/1994
* Fix integration CI by muellerzr in https://github.com/huggingface/accelerate/pull/2047
* Make fsdp ram efficient loading optional by pacman100 in https://github.com/huggingface/accelerate/pull/2037
* Let drop_last modify `gather_for_metrics` by muellerzr in https://github.com/huggingface/accelerate/pull/2048
* fix docstring by zhangsibo1129 in https://github.com/huggingface/accelerate/pull/2053
* Fix stalebot by muellerzr in https://github.com/huggingface/accelerate/pull/2052
* Add space to docs by muellerzr in https://github.com/huggingface/accelerate/pull/2055
* Fix the error when the "train_batch_size" is absent in DeepSpeed config by LZHgrla in https://github.com/huggingface/accelerate/pull/2060
* remove unused constants by statelesshz in https://github.com/huggingface/accelerate/pull/2045
* fix: remove useless token by rtrompier in https://github.com/huggingface/accelerate/pull/2069
* DOC: Fix broken link to designing a device map by BenjaminBossan in https://github.com/huggingface/accelerate/pull/2073
* Let iterable dataset shard have a length if implemented by muellerzr in https://github.com/huggingface/accelerate/pull/2066
* Allow for samplers to be seedable and reproducable by muellerzr in https://github.com/huggingface/accelerate/pull/2057
* Fix docstring typo by qgallouedec in https://github.com/huggingface/accelerate/pull/2072
* Warn when kernel version is too low on Linux by BenjaminBossan in https://github.com/huggingface/accelerate/pull/2077

New Contributors
* kli-casia made their first contribution in https://github.com/huggingface/accelerate/pull/1976
* MKhalusova made their first contribution in https://github.com/huggingface/accelerate/pull/2008
* brcps12 made their first contribution in https://github.com/huggingface/accelerate/pull/2033
* Ben-Epstein made their first contribution in https://github.com/huggingface/accelerate/pull/1994
* zhangsibo1129 made their first contribution in https://github.com/huggingface/accelerate/pull/2053
* LZHgrla made their first contribution in https://github.com/huggingface/accelerate/pull/2060
* rtrompier made their first contribution in https://github.com/huggingface/accelerate/pull/2069
* qgallouedec made their first contribution in https://github.com/huggingface/accelerate/pull/2072

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.23.0...v0.24.0

Model Memory Estimator

A new model estimation tool to help calculate how much memory is needed for inference has been added. This does not download the pretrained weights, and utilizes `init_empty_weights` to stay memory efficient during the calculation.

Usage directions:

bash
accelerate estimate-memory {model_name} --library {library_name} --dtypes fp16 int8

Or:
python
from accelerate.commands.estimate import estimate_command_parser, estimate_command, gather_data

parser = estimate_command_parser()
args = parser.parse_args(["bert-base-cased", "--dtypes", "float32"])
output = gather_data(args)


🤗 Hub is a first-class citizen

We've made the `huggingface_hub` library a first-class citizen of the framework! While this is mainly for the model estimation tool, this opens the doors for further integrations should they be wanted

`Accelerator` Enhancements:

- `gather_for_metrics` will now also de-dupe for non-tensor objects. See 1937
- `mixed_precision="bf16"` support on NPU devices. See 1949
- New `breakpoint` API to help when dealing with trying to break from a condition on a single process. See 1940
-

Notebook Launcher Enhancements:

- The notebook launcher now supports launching across multiple nodes! See 1913

FSDP Enhancements:

- Activation checkpointing is now natively supported in the framework. See https://github.com/huggingface/accelerate/pull/1891
- `torch.compile` support was fixed. See 1919

DeepSpeed Enhancements:

- XPU/ccl support (1827)
- Easier gradient accumulation support, simply set `gradient_accumulation_steps` to `"auto"` in your deepspeed config, and Accelerate will use the one passed to `Accelerator` instead (1901)
- Support for custom schedulers and deepspeed optimizers (1909)

What's Changed
* Update release instructions by sgugger in https://github.com/huggingface/accelerate/pull/1877
* fix detach_hook by SunMarc in https://github.com/huggingface/accelerate/pull/1880
* Enable power users to bypass device_map="auto" training block by muellerzr in https://github.com/huggingface/accelerate/pull/1881
* Introduce model memory estimator by muellerzr in https://github.com/huggingface/accelerate/pull/1876
* Update with new url for explore by muellerzr in https://github.com/huggingface/accelerate/pull/1884
* Enable a token to be used by muellerzr in https://github.com/huggingface/accelerate/pull/1886
* Add doc on model memory usage by muellerzr in https://github.com/huggingface/accelerate/pull/1887
* Add hub as core dep by muellerzr in https://github.com/huggingface/accelerate/pull/1885
* update import of deepspeed integration from transformers by pacman100 in https://github.com/huggingface/accelerate/pull/1894
* Final nits on model util by muellerzr in https://github.com/huggingface/accelerate/pull/1896
* Fix nb launcher test by muellerzr in https://github.com/huggingface/accelerate/pull/1899
* Add FSDP activation checkpointing feature by arde171 in https://github.com/huggingface/accelerate/pull/1891
* Solve at least one failing test by muellerzr in https://github.com/huggingface/accelerate/pull/1898
* Deepspeed integration for XPU/ccl by abhilash1910 in https://github.com/huggingface/accelerate/pull/1827
* Add PR template by muellerzr in https://github.com/huggingface/accelerate/pull/1906
* deepspeed grad_acc_steps fixes by pacman100 in https://github.com/huggingface/accelerate/pull/1901
* Skip pypi transformers until release by muellerzr in https://github.com/huggingface/accelerate/pull/1911
* Fix docker images by muellerzr in https://github.com/huggingface/accelerate/pull/1910
* Use hosted CI runners for building docker images by muellerzr in https://github.com/huggingface/accelerate/pull/1915
* fix: add debug argument to sagemaker configuration by maximegmd in https://github.com/huggingface/accelerate/pull/1904
* improve help info when run `accelerate config` on npu by statelesshz in https://github.com/huggingface/accelerate/pull/1895
* support logging with mlflow in case of mlflow-skinny installed by ghtaro in https://github.com/huggingface/accelerate/pull/1874
* More CI fun - run all test parts always by muellerzr in https://github.com/huggingface/accelerate/pull/1916
* Expose auto in dataclass by muellerzr in https://github.com/huggingface/accelerate/pull/1914
* Add support for deepspeed optimizer and custom scheduler by pacman100 in https://github.com/huggingface/accelerate/pull/1909
* reduce gradient first for XLA when unscaling the gradients in mixed precision training with AMP. by statelesshz in https://github.com/huggingface/accelerate/pull/1926
* Check for invalid keys by muellerzr in https://github.com/huggingface/accelerate/pull/1935
* clean num devices by SunMarc in https://github.com/huggingface/accelerate/pull/1936
* Bring back pypi to runners by muellerzr in https://github.com/huggingface/accelerate/pull/1939
* Support multi-node notebook launching by ggaaooppeenngg in https://github.com/huggingface/accelerate/pull/1913
* fix the fsdp docs by pacman100 in https://github.com/huggingface/accelerate/pull/1947
* Fix docs by ggaaooppeenngg in https://github.com/huggingface/accelerate/pull/1951
* Protect tensorflow dependency by SunMarc in https://github.com/huggingface/accelerate/pull/1959
* fix safetensor saving by SunMarc in https://github.com/huggingface/accelerate/pull/1954
* FIX: patch_environment restores pre-existing environment variables when finished by BenjaminBossan in https://github.com/huggingface/accelerate/pull/1960
* Better guards for slow imports by muellerzr in https://github.com/huggingface/accelerate/pull/1963
* [`Tests`] Finish all todos by younesbelkada in https://github.com/huggingface/accelerate/pull/1957
* Rm strtobool by muellerzr in https://github.com/huggingface/accelerate/pull/1964
* Implementing gather_for_metrics with dedup for non tensor objects by Lorenzobattistela in https://github.com/huggingface/accelerate/pull/1937
* add bf16 mixed precision support for NPU by statelesshz in https://github.com/huggingface/accelerate/pull/1949
* Introduce breakpoint API by muellerzr in https://github.com/huggingface/accelerate/pull/1940
* fix torch compile with FSDP by pacman100 in https://github.com/huggingface/accelerate/pull/1919
* Add `force_hooks` to `dispatch_model` by austinapatel in https://github.com/huggingface/accelerate/pull/1969
* update FSDP and DeepSpeed docs by pacman100 in https://github.com/huggingface/accelerate/pull/1973
* Flex fix patch for accelerate by abhilash1910 in https://github.com/huggingface/accelerate/pull/1972
* Remove checkpoints only on main process by Kepnu4 in https://github.com/huggingface/accelerate/pull/1974

New Contributors
* arde171 made their first contribution in https://github.com/huggingface/accelerate/pull/1891
* maximegmd made their first contribution in https://github.com/huggingface/accelerate/pull/1904
* ghtaro made their first contribution in https://github.com/huggingface/accelerate/pull/1874
* ggaaooppeenngg made their first contribution in https://github.com/huggingface/accelerate/pull/1913
* Lorenzobattistela made their first contribution in https://github.com/huggingface/accelerate/pull/1937
* austinapatel made their first contribution in https://github.com/huggingface/accelerate/pull/1969
* Kepnu4 made their first contribution in https://github.com/huggingface/accelerate/pull/1974

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.22.0...v0.23.0

Experimental distributed operations checking framework

A new framework has been introduced which can help catch `timeout` errors caused by distributed operations failing *before* they occur. As this adds a tiny bit of overhead, it is an opt-in scenario. Simply run your code with `ACCELERATE_DEBUG_MODE="1"` to enable this. Read more in the [docs](https://huggingface.co/docs/accelerate/main/en/usage_guides/debug), introduced via https://github.com/huggingface/accelerate/pull/1756

`Accelerator.load_state` can now load the most recent checkpoint automatically

If a `ProjectConfiguration` has been made, using `accelerator.load_state()` (without any arguments passed) can now automatically find and load the latest checkpoint used, introduced via https://github.com/huggingface/accelerate/pull/1741

Multiple enhancements to gradient accumulation

In this release multiple new enhancements to distributed gradient accumulation have been added.

* `accelerator.accumulate()` now supports passing in multiple models introduced via https://github.com/huggingface/accelerate/pull/1708
* A util has been introduced to perform multiple forwards, then multiple backwards, and finally sync the gradients only on the last `.backward()` via https://github.com/huggingface/accelerate/pull/1726

FSDP Changes

* FSDP support has been added for NPU and XPU devices via https://github.com/huggingface/accelerate/pull/1803 and https://github.com/huggingface/accelerate/pull/1806
* A new method for supporting RAM-efficient loading of models with FSDP has been added via https://github.com/huggingface/accelerate/pull/1777

DataLoader Changes

* Custom slice functions are now supported in the `DataLoaderDispatcher` added via https://github.com/huggingface/accelerate/pull/1846


What's New?
* fix failing test on 8GPU by statelesshz in https://github.com/huggingface/accelerate/pull/1724
* Better control over DDP's `no_sync` by NouamaneTazi in https://github.com/huggingface/accelerate/pull/1726
* Get rid of calling `get_scale()` by patching the step method of optimizer. by yuxinyuan in https://github.com/huggingface/accelerate/pull/1720
* fix the bug in npu by statelesshz in https://github.com/huggingface/accelerate/pull/1728
* Adding a shape check for `set_module_tensor_to_device`. by Narsil in https://github.com/huggingface/accelerate/pull/1731
* Fix errors when optimizer is not a Pytorch optimizer. by yuxinyuan in https://github.com/huggingface/accelerate/pull/1733
* Make balanced memory able to work with non contiguous GPUs ids by thomwolf in https://github.com/huggingface/accelerate/pull/1734
* Fixed typo in `__repr__` of AlignDevicesHook by KacperWyrwal in https://github.com/huggingface/accelerate/pull/1735
* Update docs by muellerzr in https://github.com/huggingface/accelerate/pull/1736
* Fixed the bug that split dict incorrectly by yuangpeng in https://github.com/huggingface/accelerate/pull/1742
* Let load_state automatically grab the latest save by muellerzr in https://github.com/huggingface/accelerate/pull/1741
* fix `KwargsHandler.to_kwargs` not working with `os.environ` initialization in `__post_init__` by CyCle1024 in https://github.com/huggingface/accelerate/pull/1738
* fix typo by cauyxy in https://github.com/huggingface/accelerate/pull/1747
* Check for misconfiguration of single node & single GPU by muellerzr in https://github.com/huggingface/accelerate/pull/1746
* Remove unused constant by muellerzr in https://github.com/huggingface/accelerate/pull/1749
* Rework new constant for operations by muellerzr in https://github.com/huggingface/accelerate/pull/1748
* Expose `autocast` kwargs and simplify `autocast` wrapper by muellerzr in https://github.com/huggingface/accelerate/pull/1740
* Fix FSDP related issues by pacman100 in https://github.com/huggingface/accelerate/pull/1745
* FSDP enhancements and fixes by pacman100 in https://github.com/huggingface/accelerate/pull/1753
* Fix check failure in `Accelerator.save_state` using multi-gpu by CyCle1024 in https://github.com/huggingface/accelerate/pull/1760
* Fix error when `max_memory` argument is in unexpected order by ranchlai in https://github.com/huggingface/accelerate/pull/1759
* Fix offload on disk when executing on CPU by sgugger in https://github.com/huggingface/accelerate/pull/1762
* Change `is_aim_available()` function to not match aim >= 4.0.0 by alberttorosyan in https://github.com/huggingface/accelerate/pull/1769
* Introduce an experimental distributed operations framework by muellerzr in https://github.com/huggingface/accelerate/pull/1756
* Support wrapping multiple models in Accelerator.accumulate() by yuxinyuan in https://github.com/huggingface/accelerate/pull/1708
* Contigous on gather by muellerzr in https://github.com/huggingface/accelerate/pull/1771
* [FSDP] Fix `load_fsdp_optimizer` by awgu in https://github.com/huggingface/accelerate/pull/1755
* simplify and correct the deepspeed example by pacman100 in https://github.com/huggingface/accelerate/pull/1775
* Set ipex default in state by muellerzr in https://github.com/huggingface/accelerate/pull/1776
* Fix import error when torch>=2.0.1 and `torch.distributed` is disabled by natsukium in https://github.com/huggingface/accelerate/pull/1800
* reserve 10% GPU in `get_balanced_memory` to avoid OOM by ranchlai in https://github.com/huggingface/accelerate/pull/1798
* add support of float memory size in `convert_file_size_to_int` by ranchlai in https://github.com/huggingface/accelerate/pull/1799
* Allow users to resume from previous wandb runs with `allow_val_change` by SumanthRH in https://github.com/huggingface/accelerate/pull/1796
* Add FSDP for XPU by abhilash1910 in https://github.com/huggingface/accelerate/pull/1803
* Add FSDP for NPU by statelesshz in https://github.com/huggingface/accelerate/pull/1806
* Fix pytest import by muellerzr in https://github.com/huggingface/accelerate/pull/1808
* More specific logging in `gather_for_metrics` by dleve123 in https://github.com/huggingface/accelerate/pull/1784
* Detect device map auto and raise a helpful error when trying to not use model parallelism by muellerzr in https://github.com/huggingface/accelerate/pull/1810
* Typo fix by muellerzr in https://github.com/huggingface/accelerate/pull/1812
* Expand device-map warning by muellerzr in https://github.com/huggingface/accelerate/pull/1819
* Update bibtex to reflect team growth by muellerzr in https://github.com/huggingface/accelerate/pull/1820
* Improve docs on grad accumulation by vwxyzjn in https://github.com/huggingface/accelerate/pull/1817
* add warning when using to and cuda by SunMarc in https://github.com/huggingface/accelerate/pull/1790
* Fix bnb import by muellerzr in https://github.com/huggingface/accelerate/pull/1813
* Update docs and docstrings to match `load_and_quantize_model` arg by JonathanRayner in https://github.com/huggingface/accelerate/pull/1822
* Expose a bit of args/docstring fixup by muellerzr in https://github.com/huggingface/accelerate/pull/1824
* Better test by muellerzr in https://github.com/huggingface/accelerate/pull/1825
* Minor idiomatic change for fp8 check. by float-trip in https://github.com/huggingface/accelerate/pull/1829
* Use device as context manager for `init_on_device` by shingjan in https://github.com/huggingface/accelerate/pull/1826
* Ipex bug fix for device properties in modelling by abhilash1910 in https://github.com/huggingface/accelerate/pull/1834
* FIX: Bug with `unwrap_model` and `keep_fp32_wrapper=False` by BenjaminBossan in https://github.com/huggingface/accelerate/pull/1838
* Fix `verify_device_map` by Rexhaif in https://github.com/huggingface/accelerate/pull/1842
* Change CUDA check by muellerzr in https://github.com/huggingface/accelerate/pull/1833
* Fix the noneffective parameter: `gpu_ids` (Rel. Issue 1848) by devymex in https://github.com/huggingface/accelerate/pull/1850
* support for ram efficient loading of model with FSDP by pacman100 in https://github.com/huggingface/accelerate/pull/1777
* Loading logic safetensors by SunMarc in https://github.com/huggingface/accelerate/pull/1853
* fix dispatch for quantized model by SunMarc in https://github.com/huggingface/accelerate/pull/1855
* Update `fsdp_with_peak_mem_tracking`.py by pacman100 in https://github.com/huggingface/accelerate/pull/1856
* Add env variable for `init_on_device` by shingjan in https://github.com/huggingface/accelerate/pull/1852
* remove casting to FP32 when saving state dict by pacman100 in https://github.com/huggingface/accelerate/pull/1868
* support custom slice function in `DataLoaderDispatcher` by thevasudevgupta in https://github.com/huggingface/accelerate/pull/1846
* Include a note to the forums in the bug report by muellerzr in https://github.com/huggingface/accelerate/pull/1871

Significant community contributions

The following contributors have made significant changes to the library over the last release:

* yuxinyuan
* Support wrapping multiple models in `Accelerator.accumulate()` (1708)
* Fix errors when optimizer is not a Pytorch optimizer. (1733)
* Get rid of calling get_scale() by patching the step method of optimizer. (1720)
* NouamaneTazi
* Better control over DDP's `no_sync` (1726)
* abhilash1910
* Add FSDP for XPU (1803)
* Ipex bug fix for device properties in modelling (1834)
* statelesshz
* Add FSDP for NPU (1806)
* fix failing test on 8GPU (1724)
* fix the bug in npu (1728)
* thevasudevgupta
* support custom slice function in `DataLoaderDispatcher` (1846)

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.21.0...v0.22.0

Model quantization with bitsandbytes

You can now quantize any model (no just Transformer models) using Accelerate. This is mainly for models having a lot of linear layers. See the [documentation](https://huggingface.co/docs/accelerate/usage_guides/quantization) for more information!

* Bnb quantization by SunMarc in 1626

Support for Ascend NPUs

Accelerate now supports Ascend NPUs.

* Add Ascend NPU accelerator support by statelesshz in 1676

What's new?

Accelerate now requires Python 3.8+ and PyTorch 1.10+ :

* 🚨🚨🚨 Spring cleaning: Python 3.8 🚨🚨🚨 by muellerzr in 1661
* 🚨🚨🚨 Spring cleaning: PyTorch 1.10 🚨🚨🚨 by muellerzr in 1662


* [doc build] Use secrets by mishig25 in 1551
* Update launch.mdx by LiamSwayne in 1553
* Avoid double wrapping of all accelerate.prepare objects by muellerzr in 1555
* Update README.md by LiamSwayne in 1556
* Fix load_state_dict when there is one device and disk by sgugger in 1557
* Fix tests not being ran on multi-GPU nightly by muellerzr in 1558
* fix the typo when setting the "_accelerator_prepared" attribute by Yura52 in 1560
* [`core`] Fix possibility to pass`NoneType` objects in `prepare` by younesbelkada in 1561
* Reset dataloader end_of_datalaoder at each iter by sgugger in 1562
* Update big_modeling.mdx by LiamSwayne in 1564
* [`bnb`] Fix failing int8 tests by younesbelkada in 1567
* Update gradient sync docs to reflect importance of `optimizer.step()` by dleve123 in 1565
* Update mixed precision integrations in README by sgugger in 1569
* Raise error instead of warn by muellerzr in 1568
* Introduce listify, fix tensorboard silently failing by muellerzr in 1570
* Check for bak and expand docs on directory structure by muellerzr in 1571
* Perminant solution by muellerzr in 1577
* fix the bug in xpu by mingxiaoh in 1508
* Make sure that we only set is_accelerator_prepared on items accelerate actually prepares by muellerzr in 1578
* Expand `prepare()` doc by muellerzr in 1580
* Get Torch version using importlib instead of pkg_resources by catwell in 1585
* improve oob performance when use mpirun to start DDP finetune without `accelerate launch` by sywangyi in 1575
* Update training_tpu.mdx by LiamSwayne in 1582
* Return false if CUDA available by muellerzr in 1581
* fix logger level by caopulan in 1579
* Fix test by muellerzr in 1586
* Update checkpoint.mdx by LiamSwayne in 1587
* FSDP updates by pacman100 in 1576
* Update modeling.py by ain-soph in 1595
* Integration tests by muellerzr in 1593
* Add triggers for CI workflow by muellerzr in 1597
* Remove asking xpu plugin for non xpu devices by abhilash1910 in 1594
* Remove GPU safetensors env variable by sgugger in 1603
* reset end_of_dataloader for dataloader_dispatcher by megavaz in 1609
* fix for arc gpus by abhilash1910 in 1615
* Ignore low_zero option when only device is available by sgugger in 1617
* Fix failing multinode tests by muellerzr in 1616
* Doc to md by sgugger in 1618
* Fix tb issue by muellerzr in 1623
* Fix workflow by muellerzr in 1625
* Fix transformers sync bug with accumulate by muellerzr in 1624
* fixes offload dtype by SunMarc in 1631
* fix: Megatron is not installed. please build it from source. by yuanwu2017 in 1636
* deepspeed z2/z1 state_dict bloating fix by pacman100 in 1638
* Swap disable rich by muellerzr in 1640
* fix autocasting bug by pacman100 in 1637
* fix modeling low zero by abhilash1910 in 1634
* Add skorch to runners by muellerzr in 1646
* add save model by SunMarc in 1641
* Change dispatch_model when we have only one device by SunMarc in 1648
* Doc save model by SunMarc in 1650
* Fix device_map by SunMarc in 1651
* Check for port usage before launch by muellerzr in 1656
* [`BigModeling`] Add missing check for quantized models by younesbelkada in 1652
* Bump integration by muellerzr in 1658
* TIL by muellerzr in 1657
* docker cpu py version by muellerzr in 1659
* [`BigModeling`] Final fix for dispatch int8 and fp4 models by younesbelkada in 1660
* remove safetensor dep on shard_checkpoint by SunMarc in 1664
* change the import place to avoid import error by pacman100 in 1653
* Update broken Runhouse link in examples/README.md by dongreenberg in 1668
* Bnb quantization by SunMarc in 1626
* replace save funct in doc by SunMarc in 1672
* Doc big model inference by SunMarc in 1670
* Add docs for saving Transformers models by deppen8 in 1671
* fix bnb tests by SunMarc in 1679
* Fix workflow CI by muellerzr in 1690
* remove duplicate class by SunMarc in 1691
* update readme in examples by statelesshz in 1678
* Fix nightly tests by muellerzr in 1696
* Fixup docs by muellerzr in 1697
* Improve quality errors by muellerzr in 1698
* Move mixed precision wrapping ahead of DDP/FSDP wrapping by ChenWu98 in 1682
* Add offload for 8-bit model by SunMarc in 1699
* Deepcopy on Accelerator to return self by muellerzr in 1694
* Update tracking.md by stevhliu in 1702
* Skip tests when bnb isn't available by muellerzr in 1706
* Fix launcher validation by abhilash1910 in 1705
* Fixes for issue 1683: failed to run accelerate config in colab by Erickrus in 1692
* Fix the bug where DataLoaderDispatcher gets stuck in an infinite wait when the dataset is an IterDataPipe during multi-process training. by yuxinyuan in 1709
* add multi_gpu decorator by SunMarc in 1712
* Modify loading checkpoint behavior by SunMarc in 1715
* fix version by SunMarc in 1701
* Keep old behavior by muellerzr in 1716
* Optimize `get_scale` to reduce async calls by muellerzr in 1718
* Remove duplicate code by muellerzr in 1717
* New tactic by muellerzr in 1719
* add Comfy-UI by pacman100 in 1723
* add compatibility with peft by SunMarc in 1725

Significant community contributions

The following contributors have made significant changes to the library over the last release:

* LiamSwayne
* Update launch.mdx (1553)
* Update README.md (1556)
* Update big_modeling.mdx (1564)
* Update training_tpu.mdx (1582)
* Update checkpoint.mdx (1587)
* mingxiaoh
* fix the bug in xpu (1508)
* statelesshz
* update readme in examples (1678)
* Add Ascend NPU accelerator support (1676)
* ChenWu98
* Move mixed precision wrapping ahead of DDP/FSDP wrapping (1682)

- Reset dataloader end_of_datalaoder at each iter in 1562 by sgugger

- fix the typo when setting the "_accelerator_prepared" attribute in 1560 by Yura52
- [core] Fix possibility to pass] `NoneType` objects in `prepare` in 1561 by younesbelkada