Accelerate

- Avoid double wrapping of all accelerate.prepare objects by muellerzr in 1555
- Fix load_state_dict when there is one device and disk by sgugger in 1557

Big model inference

Support has been added to run `device_map="auto"` on the MPS device. Big model inference also work with models loaded in 4 bits in Transformers.

* Add mps support to big inference modeling by SunMarc in 1545
* Adds fp4 support for model dispatching by younesbelkada in 1505

4-bit QLoRA Support

* 4-bit QLoRA via bitsandbytes (4-bit base model + LoRA) by TimDettmers in 1458

Distributed Inference Utilities

This version introduces a new `Accelerator.split_between_processes` utility to help with performing distributed infernece with non-tensorized or non-dataloader workflows. Read more [here](https://huggingface.co/docs/accelerate/usage_guides/distributed_inference)

Introduce XPU support for Intel GPU

* Intel GPU support initialization by abhilash1910 in 1118

Add support for the new PyTorch XLA TPU runtime

* Accelerate now supports the latest TPU runtimes 1393, 1385

A new optimizer method: `LocalSGD`

* This is a new wrapper around SGD which enables efficient multi-GPU training in the case when no fast interconnect is possible by searchivarius in 1378

Papers with 🤗 Accelerate

* We now have an entire section of the docs dedicated to official paper implementations and citations using the framework 1399, see it live [here](https://hf.co/docs/accelerate/usage_guides/training_zoo#in-science)

Breaking changes

`logging_dir` has been fully deprecated, please use `project_dir` or a `Project_configuration`

What's new?

* use existing mlflow experiment if exists by Rusteam in 1403
* changes required for DS integration by pacman100 in 1406
* fix deepspeed failing tests by pacman100 in 1411
* Make mlflow logging dir optional by mattplo-decath in 1413
* Fix bug on ipex for diffusers by abhilash1910 in 1426
* Improve Slack Updater by muellerzr in 1433
* Let quality yell at the user if it's a version difference by muellerzr in 1438
* Ensure that it gets installed by muellerzr in 1439
* [`core`] Introducing `CustomDtype` enum for custom dtypes by younesbelkada in 1434
* Fix XPU by muellerzr in 1440
* Make sure torch compiled model can also be unwrapped by patrickvonplaten in 1437
* fixed: ZeroDivisionError: division by zero by sreio in 1436
* fix potential OOM when resuming with multi-GPU training by exhyy in 1444
* Fixes in infer_auto_device_map by sgugger in 1441
* Raise error when logging improperly by muellerzr in 1446
* Fix ci by muellerzr in 1447
* Distributed prompting/inference utility by muellerzr in 1410
* Add to by muellerzr in 1448
* split_between_processes by stevhliu in 1449
* [docs] Replace `state.rank` -> `process_index` by pcuenca in 1450
* Auto multigpu logic by muellerzr in 1452
* Update with cli instructions by muellerzr in 1453
* Adds `in_order` argument that defaults to False, to log in order. by JulesGM in 1262
* fix error for CPU DDP using trainer api. by sywangyi in 1455
* Refactor and simplify xpu device in state by abhilash1910 in 1456
* Document how to use commands with python module instead of argparse by muellerzr in 1457
* 4-bit QLoRA via bitsandbytes (4-bit base model + LoRA) by TimDettmers in 1458
* Fix skip first batch being perminant by muellerzr in 1466
* update conversion of layers to retain original data type. by avisinghal6 in 1467
* Check for xpu specifically by muellerzr in 1472
* update `register_empty_buffer` to match torch args by NouamaneTazi in 1465
* Update gradient accumulation docs, and remove redundant example by iantbutler01 in 1461
* Imrpove sagemaker by muellerzr in 1470
* Split tensors as part of `split_between_processes` by muellerzr in 1477
* Move to device by muellerzr in 1478
* Fix gradient state bugs in multiple dataloader by Ethan-yt in 1483
* Add rdzv-backend by muellerzr in 1490
* Only use IPEX if available by muellerzr in 1495
* Update README.md by lyhue1991 in 1493
* Let gather_for_metrics always run by muellerzr in 1496
* Use empty like when we only need to create buffers by thomasw21 in 1497
* Allow key skipping in big model inference by sgugger in 1491
* fix crash when ipex is installed and torch has no xpu by sywangyi in 1502
* [`bnb`] Add fp4 support for dispatch by younesbelkada in 1505
* Fix 4bit model on multiple devices by SunMarc in 1506
* adjust overriding of model's forward function by prathikr in 1492
* Add assertion when call prepare with deepspeed config. by tensimiku in 1468
* NVME path support for deepspeed by abhilash1910 in 1484
* should set correct dtype to ipex optimize and use amp logic in native… by sywangyi in 1511
* Swap env vars for XPU and IPEX + CLI by muellerzr in 1513
* Fix a bug when parameters tied belong to the same module by sgugger in 1514
* Fixup deepspeed/cli tests by muellerzr in 1526
* Refactor mp into its own wrapper by muellerzr in 1527
* Check tied parameters by SunMarc in 1529
* Raise ValueError on iterable dataset if we've hit the end and attempting to go beyond it by muellerzr in 1531
* Officially support naive PP for quantized models + PEFT by younesbelkada in 1523
* remove ipexplugin, let ACCELERATE_USE_IPEX/ACCELERATE_USE_XPU control the ipex and xpu by sywangyi in 1503
* Prevent using extra VRAM for static device_map by LSerranoPEReN in 1536
* Update deepspeed.mdx by LiamSwayne in 1541
* Update performance.mdx by LiamSwayne in 1543
* Update deferring_execution.mdx by LiamSwayne in 1544
* Apply deprecations by muellerzr in 1537
* Add mps support to big inference modeling by SunMarc in 1545
* [documentation] grammar fixes in gradient_synchronization.mdx by LiamSwayne in 1547
* Eval mode by muellerzr in 1540
* Update migration.mdx by LiamSwayne in 1549

Significant community contributions

The following contributors have made significant changes to the library over the last release:

* will-cromar
* Support TPU v4 with new PyTorch/XLA TPU runtime (1393)
* Support TPU v2 and v3 on new PyTorch/XLA TPU runtime (1385)
* searchivarius
* Adding support for local SGD. (1378)
* abhilash1910
* Intel GPU support initialization (1118)
* Fix bug on ipex for diffusers (1426)
* Refactor and simplify xpu device in state (1456)
* NVME path support for deepspeed (1484)
* sywangyi
* fix error for CPU DDP using trainer api. (1455)
* fix crash when ipex is installed and torch has no xpu (1502)
* should set correct dtype to ipex optimize and use amp logic in native… (1511)
* remove ipexplugin, let ACCELERATE_USE_IPEX/ACCELERATE_USE_XPU control the ipex and xpu (1503)
* Ethan-yt
* Fix gradient state bugs in multiple dataloader (1483)

What's New

- Support for Intel IPEX support has been added, check out the how-to guide [now!](https://huggingface.co/docs/accelerate/main/en/usage_guides/ipex)
- Various modifications have been added to begin work on having 🤗 Accelerate be the foundation for the `Trainer`, keep an eye on the repos to see how our progress is coming along!
- FP8 training is now supported on Ada Lovelance GPUs
- The `wandb` integration now supports logging of images and tables through `tracker.log_images` and `tracker.log_tables` respectively
- Many, many squashed bugs! (see the full detailed report for just what they were)
- **17** new contributors to the framework, congratulations to all who took their first step! 🚀

What's Changed

* Fix pypi image by muellerzr in https://github.com/huggingface/accelerate/pull/1249
* raise error when dataloader with None as batch_size when using DS by pacman100 in https://github.com/huggingface/accelerate/pull/1250
* Handle missing deepspeed config by HeyangQin in https://github.com/huggingface/accelerate/pull/1251
* [`core`] Add Quantization support for `dispatch_model` by younesbelkada in https://github.com/huggingface/accelerate/pull/1237
* Check attribute 'overflow' exists in optimizer. by tensimiku in https://github.com/huggingface/accelerate/pull/1259
* ipex intel extension for pytorch integration by sywangyi in https://github.com/huggingface/accelerate/pull/1255
* fix issue template by stas00 in https://github.com/huggingface/accelerate/pull/1264
* Change error raised to ValueError by sgugger in https://github.com/huggingface/accelerate/pull/1267
* Fix reduce operation by xyfJASON in https://github.com/huggingface/accelerate/pull/1268
* Raise import error if fp8 not available in `has_transfomer_engine_layers` by muellerzr in https://github.com/huggingface/accelerate/pull/1283
* Add missing FP8 options to CLI by muellerzr in https://github.com/huggingface/accelerate/pull/1284
* Update quicktour.mdx by standardAI in https://github.com/huggingface/accelerate/pull/1273
* Minor fix whitespace colon by guspan-tanadi in https://github.com/huggingface/accelerate/pull/1272
* fix attribute error in DataloaderShared by ZhiyuanChen in https://github.com/huggingface/accelerate/pull/1278
* Fix TypeError bug in honor_type by muellerzr in https://github.com/huggingface/accelerate/pull/1285
* Raise more explicit error when transformer_engine isn't installed by muellerzr in https://github.com/huggingface/accelerate/pull/1287
* Expound error on `recursively_apply` by muellerzr in https://github.com/huggingface/accelerate/pull/1286
* Only check for dtype if it has it in get_state_dict by muellerzr in https://github.com/huggingface/accelerate/pull/1288
* [`bnb`] fix bnb slow test by younesbelkada in https://github.com/huggingface/accelerate/pull/1292
* Raise better error on `notebook_launcher` by muellerzr in https://github.com/huggingface/accelerate/pull/1293
* Make note about grad accum and prec in performance documentation by muellerzr in https://github.com/huggingface/accelerate/pull/1296
* fix for load_checkpoint_and_dispatch(device_map=None) by anentropic in https://github.com/huggingface/accelerate/pull/1297
* Set the state device dependant to Accelerator on multigpu by muellerzr in https://github.com/huggingface/accelerate/pull/1220
* add usage guide for ipex plugin by sywangyi in https://github.com/huggingface/accelerate/pull/1270
* Simplify MPS implementation by sgugger in https://github.com/huggingface/accelerate/pull/1308
* Bug fix in setattr by aashiqmuhamed in https://github.com/huggingface/accelerate/pull/1312
* Allow xpu backend by muellerzr in https://github.com/huggingface/accelerate/pull/1313
* Default to nccl by muellerzr in https://github.com/huggingface/accelerate/pull/1314
* offload the previous module hook before the current module is moved to… by williamberman in https://github.com/huggingface/accelerate/pull/1315
* Ensure that dynamo is compatible with mixed precision by muellerzr in https://github.com/huggingface/accelerate/pull/1318
* Upgrade torch version on main tests by muellerzr in https://github.com/huggingface/accelerate/pull/1323
* Add test flag and import check for dynamo by muellerzr in https://github.com/huggingface/accelerate/pull/1322
* ensure module prefixes only match that module by xloem in https://github.com/huggingface/accelerate/pull/1319
* remove repetitive entries from device lists by xloem in https://github.com/huggingface/accelerate/pull/1321
* Fix failing test on main by muellerzr in https://github.com/huggingface/accelerate/pull/1332
* Verbosity, Progress Bar for Loading by xloem in https://github.com/huggingface/accelerate/pull/1329
* Skip failing torch 2.0+ test by muellerzr in https://github.com/huggingface/accelerate/pull/1339
* Remove unused amp import util by muellerzr in https://github.com/huggingface/accelerate/pull/1340
* Fix nested context manager for main_process_first() by flukeskywalker in https://github.com/huggingface/accelerate/pull/1304
* Small progress bar fix by xloem in https://github.com/huggingface/accelerate/pull/1341
* Pop more backend options by muellerzr in https://github.com/huggingface/accelerate/pull/1342
* Support FP8 mixed precision training for Ada Lovelace GPUs by Dango233 in https://github.com/huggingface/accelerate/pull/1348
* using deepspeed.comm for distrbiuted init by pacman100 in https://github.com/huggingface/accelerate/pull/1352
* [`bnb`] Fix bnb slow test by younesbelkada in https://github.com/huggingface/accelerate/pull/1355
* Better check for packages availability by apbard in https://github.com/huggingface/accelerate/pull/1356
* fix: typing issues, and replace deprecated python typing (Optional, Union) to `|` by kiyoon in https://github.com/huggingface/accelerate/pull/1363
* Fix default FSDP_MIN_NUM_PARAMS so it's an int by sam-hieken in https://github.com/huggingface/accelerate/pull/1367
* Special transformers case from args by muellerzr in https://github.com/huggingface/accelerate/pull/1364
* Improve `accelerate env` reporting by muellerzr in https://github.com/huggingface/accelerate/pull/1376
* Seperate out contextmanager generation by muellerzr in https://github.com/huggingface/accelerate/pull/1379
* delete textfile after tests are done by muellerzr in https://github.com/huggingface/accelerate/pull/1381
* Fix flakey thread issue by muellerzr in https://github.com/huggingface/accelerate/pull/1387
* fix config bug for 'mixed_precision' from 'yaml.safe_load()' by ys-eric-choi in https://github.com/huggingface/accelerate/pull/1386
* Log Images and other types to wandb by tcapelle in https://github.com/huggingface/accelerate/pull/962
* Bump torch version by muellerzr in https://github.com/huggingface/accelerate/pull/1392
* Fix gather_obj by muellerzr in https://github.com/huggingface/accelerate/pull/1391
* Update training_zoo.mdx by yuvalkirstain in https://github.com/huggingface/accelerate/pull/1397

New Contributors
* HeyangQin made their first contribution in https://github.com/huggingface/accelerate/pull/1251
* tensimiku made their first contribution in https://github.com/huggingface/accelerate/pull/1259
* xyfJASON made their first contribution in https://github.com/huggingface/accelerate/pull/1268
* standardAI made their first contribution in https://github.com/huggingface/accelerate/pull/1273
* guspan-tanadi made their first contribution in https://github.com/huggingface/accelerate/pull/1272
* anentropic made their first contribution in https://github.com/huggingface/accelerate/pull/1297
* aashiqmuhamed made their first contribution in https://github.com/huggingface/accelerate/pull/1312
* williamberman made their first contribution in https://github.com/huggingface/accelerate/pull/1315
* xloem made their first contribution in https://github.com/huggingface/accelerate/pull/1319
* flukeskywalker made their first contribution in https://github.com/huggingface/accelerate/pull/1304
* Dango233 made their first contribution in https://github.com/huggingface/accelerate/pull/1348
* apbard made their first contribution in https://github.com/huggingface/accelerate/pull/1356
* kiyoon made their first contribution in https://github.com/huggingface/accelerate/pull/1363
* sam-hieken made their first contribution in https://github.com/huggingface/accelerate/pull/1367
* ys-eric-choi made their first contribution in https://github.com/huggingface/accelerate/pull/1386
* tcapelle made their first contribution in https://github.com/huggingface/accelerate/pull/962
* yuvalkirstain made their first contribution in https://github.com/huggingface/accelerate/pull/1397

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.18.0...v0.19.0

What's Changed

- A new `GradientAccumulationPlugin` has been added to handle more configurations with the `GradientState`. Specifically you can optionally disable having `Accelerate` automatically adjust the length of the scheduler relative to gradient accumulation steps through it. Otherwise Accelerate will now automatically handle ensuring that the schedulers built for non-gradient accumulation will work during gradient accumulation
- Some fixes related to the launch configuration and TPU launches were adjusted, and the `dynamo_backend` warning has been silenced.
- Big model inference saw a number of fixes related to linear layers, `drop_last` on linear layers, tied weight loading, and handling of multiple tied parameters
- A new integration example with RunhouseML has been added, read more here: https://github.com/huggingface/accelerate/tree/main/examples#simple-multi-gpu-hardware-launcher

Breaking Changes

- `find_tied_parameters` now deals with groups of tied parameters (instead of only pairs of them). As a result it now returns a list of list of strings instead of a dictionary.

What's New?
* Add documentation around FSDP state dict save behavior by VikParuchuri in https://github.com/huggingface/accelerate/pull/1181
* add `use_orig_params` to FullyShardedDataParallelPlugin by pacman100 in https://github.com/huggingface/accelerate/pull/1184
* Only convert linear layers with weights multiple of 16 by sgugger in https://github.com/huggingface/accelerate/pull/1188
* Set drop last to ensure modulo16 restriction for fp8 by ksivaman in https://github.com/huggingface/accelerate/pull/1189
* Accelerator should not call `to` on modules that wraps `accelerate` loaded models by younesbelkada in https://github.com/huggingface/accelerate/pull/1172
* Fixup passing overlapping args to the script by muellerzr in https://github.com/huggingface/accelerate/pull/1198
* Make the Scheduler adjust the steps taken relative to the gradient accumulation steps by muellerzr in https://github.com/huggingface/accelerate/pull/1187
* Fix tied weights load by sgugger in https://github.com/huggingface/accelerate/pull/1204
* Better error message when using multi-GPU and Accelerate on torch <1.9.1 by muellerzr in https://github.com/huggingface/accelerate/pull/1203
* Fix typo in TPU config by muellerzr in https://github.com/huggingface/accelerate/pull/1202
* Fix example in accumulate method documentation by VikParuchuri in https://github.com/huggingface/accelerate/pull/1211
* ds offload optim fix to use CPUAdam by pacman100 in https://github.com/huggingface/accelerate/pull/1208
* Move when the GradientState test is performed so that it is not None by muellerzr in https://github.com/huggingface/accelerate/pull/1219
* Fix bug in loading launch config by neumyor in https://github.com/huggingface/accelerate/pull/1218
* Fix get_logger kwarg documentation issue by bcol23 in https://github.com/huggingface/accelerate/pull/1222
* docs: add finetuner to ppl who use accelerate by bwanglzu in https://github.com/huggingface/accelerate/pull/1224
* Silence dynamo_backend by muellerzr in https://github.com/huggingface/accelerate/pull/1226
* Add additional check when patching env by Chris-hughes10 in https://github.com/huggingface/accelerate/pull/1229
* Make grad accum steps mutable on the Accelerator object by muellerzr in https://github.com/huggingface/accelerate/pull/1233
* devcontainer: "extensions" has been removed and replaced by customizations by dbpprt in https://github.com/huggingface/accelerate/pull/1075
* remove empty dicts while saving accelerate config by pacman100 in https://github.com/huggingface/accelerate/pull/1236
* backfill ds plugin attributes when using ds_config by pacman100 in https://github.com/huggingface/accelerate/pull/1235
* Change multinode to multigpu in notebook tutorial by muellerzr in https://github.com/huggingface/accelerate/pull/1247
* Hardware Auto-Setup Example/Tutorial for Distributed Launch by carolineechen in https://github.com/huggingface/accelerate/pull/1227
* Handle multiple tied parameters by sgugger in https://github.com/huggingface/accelerate/pull/1241

New Contributors
* hackpert made their first contribution in https://github.com/huggingface/accelerate/pull/1180
* VikParuchuri made their first contribution in https://github.com/huggingface/accelerate/pull/1181
* ksivaman made their first contribution in https://github.com/huggingface/accelerate/pull/1189
* neumyor made their first contribution in https://github.com/huggingface/accelerate/pull/1218
* bcol23 made their first contribution in https://github.com/huggingface/accelerate/pull/1222
* bwanglzu made their first contribution in https://github.com/huggingface/accelerate/pull/1224
* carolineechen made their first contribution in https://github.com/huggingface/accelerate/pull/1227

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v0.17.1...v0.18.0

* Fix CPU error always being raised by muellerzr in 1175
* fixed typo in launch.py tpu_pod_launcher by hackpert in 1180
* Support special mapping of dtypes when preparing device map by sgugger in 1179

PyTorch 2.0 support

This release fully supports the upcoming PyTorch 2.0 release. You can choose to use `torch.compile` or not and then customize the options in `accelerate.config` or via a `TorchDynamoPlugin`.

* update support for torch dynamo compile by pacman100 in 1150

Process Control Enhancements

This release adds a new `PartialState`, which contains most of the capabilities of the `AcceleratorState` however it is designed to be used by the user to assist in any process control mechanisms around it. With this, users also now do not need to have `if accelerator.state.is_main_process` when utilizing classes such as the `Tracking` API, as these now will automatically use only the main process for their work by default.

* Refactor process executors to be in AcceleratorState by muellerzr in 1039

TPU Pod Support (Experimental)

Launching from TPU pods is now supported, please see [this issue](https://github.com/huggingface/accelerate/issues/501#issuecomment-1424614540) for more information

* Introduce TPU Pod launching to `accelerate launch` by muellerzr in 1049

FP8 mixed precision training (Experimental)

This release adds experimental support for FP8 mixed precision training, which requires the [transformer-engine](https://github.com/NVIDIA/TransformerEngine) library as well as a Hopper GPU (or higher).

* Fp8 integration by sgugger in 1086

What's new?

* v0.17.0.dev0 by sgugger (direct commit on main)
* Deepspeed param check by dhar174 in 1015
* enabling `mps` device by default and removing related config by pacman100 in 1030
* fix: links to gradient synchronization by prassanna-ravishankar in 1035
* do not scale gradient in bf16 mode by kashif in 1036
* Pass keywords arguments of backward function deeper to DeepSpeed by DistinctVision in 1037
* Add daily slack notifier for nightlies by muellerzr in 1042
* Make sure direct parameters are properly set on device by sgugger in 1043
* Add `cpu_offload_with_hook` by sgugger in 1045
* Update quality tools to 2023 by sgugger in 1046
* Load tensors directly on device by sgugger in 1028
* Fix cpu_offload_with_hook code snippet by pcuenca in 1047
* Use create_task by muellerzr in 1052
* Fix args by adding in the defaults by muellerzr in 1053
* deepspeed `hidden_size` auto value default fixes by pacman100 in 1060
* Introduce PartialState by muellerzr in 1055
* Flag for deprecation by muellerzr in 1061
* Try with this by muellerzr in 1062
* Update integrations by muellerzr in 1063
* Swap utils over to use PartialState by muellerzr in 1065
* update fsdp docs and removing deepspeed version pinning by pacman100 in 1059
* Fix/implement process-execution decorators on the Accelerator by muellerzr in 1070
* Refactor state and make `PartialState` first class citizen by muellerzr in 1071
* Add error if passed --config_file does not exist by muellerzr in 1074
* SageMaker image_uri is now optional by <NOT FOUND> in 1077
* Allow custom SageMaker Estimator arguments by <NOT FOUND> in 1080
* Fix tpu_cluster arg by muellerzr in 1081
* Update complete_cv_example.py by fcossio in 1082
* Added SageMaker local mode config section by <NOT FOUND> in 1084
* Fix config by muellerzr in 1090
* adds missing "lfs" in pull by CSchoel in 1091
* add multi_cpu support to reduce by alex-hh in 1094
* Update README.md by BM-K in 1100
* Tracker rewrite and lazy process checker by muellerzr in 1079
* Update performance.mdx by fcossio in 1107
* Attempt to unwrap tracker. by pcuenca in 1109
* TensorBoardTracker: wrong arg def by stas00 in 1111
* Actually raise if exception by muellerzr in 1124
* Add test for ops and fix reduce by muellerzr in 1122
* Deep merge SageMaker `additional_args`, allowing more flexible configuration and `env` variable support by dbpprt in 1113
* Move dynamo.optimize to the end of model preparation by ymwangg in 1128
* Refactor `launch` for greater extensibility by Yard1 in 1123
* [Big model loading] Correct GPU only loading by patrickvonplaten in 1121
* Add tee and role to launch by muellerzr in 1132
* Expand warning and grab all GPUs available by default by muellerzr in 1134
* Fix multinode with GPU ids when each node has 1 by muellerzr in 1127
* deepspeed dataloader prepare fix by pacman100 in 1126
* fix ds dist init kwargs issue by pacman100 in 1138
* fix lr scheduler issue by pacman100 in 1140
* fsdp bf16 enable autocast by pacman100 in 1125
* Fix notebook_launcher by muellerzr in 1141
* fix partial state by pacman100 in 1144
* FSDP enhancements and fixes by pacman100 in 1145
* Fixed typos in notebook by SamuelLarkin in 1146
* Include a note in the gradient synchronization docs on "what can go wrong" and show the timings by muellerzr in 1153
* [Safetensors] Relax missing metadata constraint by patrickvonplaten in 1151
* Solve arrow keys being environment dependant for accelerate config by p1atdev (direct commit on main)
* Load custom state to cpu by Guangxuan-Xiao in 1156
* :memo: add a couple more trackers to the docs by nateraw in 1158
* Let GradientState know active dataloaders and reset the remainder by muellerzr in 1162
* Attempt to fix import error when PyTorch is build without `torch.distributed` module by mfuntowicz in 1108
* [`Accelerator`] Fix issue with 8bit models by younesbelkada in 1155
* Document skip_first_batches in the checkpoint usage guides by muellerzr in 1164
* Fix what files get deleted through `total_limit` by muellerzr in 1165
* Remove outdated command directions and use in tests by muellerzr in 1166

Significant community contributions

The following contributors have made significant changes to the library over the last release:

* Yard1
* Refactor `launch` for greater extensibility (1123)