Hivemind

What's Changed
* Fix averager speed for TCP connections by borzunov in https://github.com/learning-at-home/hivemind/pull/373
* Fix "Too many open files" and load state freezing by justheuristic in https://github.com/learning-at-home/hivemind/pull/371
* Prefetch while reading rpc_aggregate_part() outputs by borzunov in https://github.com/learning-at-home/hivemind/pull/370
* Use ModeClient in libp2p DHT in case of --client_mode by borzunov in https://github.com/learning-at-home/hivemind/pull/374
* Integrate p2pd logs and outputs into hivemind logging by borzunov in https://github.com/learning-at-home/hivemind/pull/375
* Split compression strategies into separate classes by justheuristic in https://github.com/learning-at-home/hivemind/pull/366
* Implement colored logs by borzunov in https://github.com/learning-at-home/hivemind/pull/377
* Parametrize max message size for persistent connections by deniskamazur in https://github.com/learning-at-home/hivemind/pull/376
* Make log handlers configurable, shorten entries by borzunov in https://github.com/learning-at-home/hivemind/pull/378
* Enable log handler in benchmarks and run_server by borzunov in https://github.com/learning-at-home/hivemind/pull/380
* Fix step_tolerance in CollaborativeOptimizer by justheuristic in https://github.com/learning-at-home/hivemind/pull/383
* Fix pickle vulnerability by deniskamazur in https://github.com/learning-at-home/hivemind/pull/386
* Remove arguments with default values from example instructions by borzunov in https://github.com/learning-at-home/hivemind/pull/388
* Implement weight as part of the allreduce protocol, not matchmaking by justheuristic in https://github.com/learning-at-home/hivemind/pull/384
* Support different AMP & buffer configurations in one experiment, fix minor bugs by justheuristic in https://github.com/learning-at-home/hivemind/pull/389
* Fix codecov_in_develop_mode with pip>=21.2 by justheuristic in https://github.com/learning-at-home/hivemind/pull/393
* Fix minor issues in documentation by borzunov in https://github.com/learning-at-home/hivemind/pull/392
* Apply averager updates asynchronously by justheuristic in https://github.com/learning-at-home/hivemind/pull/395
* Fix schema typing by justheuristic in https://github.com/learning-at-home/hivemind/pull/396
* backport PerformanceEMA from server_side_averaging by justheuristic in https://github.com/learning-at-home/hivemind/pull/397
* Add an option to pre-schedule averaging by justheuristic in https://github.com/learning-at-home/hivemind/pull/398
* Move DHT to dht/dht.py, update DHT figure by justheuristic in https://github.com/learning-at-home/hivemind/pull/399
* [hotfix] replace StepControl.can_modify with began_allreduce by justheuristic in https://github.com/learning-at-home/hivemind/pull/402
* move PerformanceEMA to utils, TrainingAverager to optim, update utils by justheuristic in https://github.com/learning-at-home/hivemind/pull/405
* Add GradientAverager with support for delayed averaging by justheuristic in https://github.com/learning-at-home/hivemind/pull/404
* [hivemind.Optimizer] TrainingStateAverager by justheuristic in https://github.com/learning-at-home/hivemind/pull/407
* Catch OSError in MPFuture by artek0chumak in https://github.com/learning-at-home/hivemind/pull/409
* [hivemind.Optimizer] ProgressTracker by justheuristic in https://github.com/learning-at-home/hivemind/pull/408
* Fix minor bugs in GradientAverager by justheuristic in https://github.com/learning-at-home/hivemind/pull/410
* Make target group size optional by justheuristic in https://github.com/learning-at-home/hivemind/pull/412
* Prepare GradScaler for hivemind.Optimizer by justheuristic in https://github.com/learning-at-home/hivemind/pull/413
* Patch recursive cancel in StepControl by justheuristic in https://github.com/learning-at-home/hivemind/pull/411
* Replace the invalid link to discord by artek0chumak in https://github.com/learning-at-home/hivemind/pull/414
* Implement state sharing priority by justheuristic in https://github.com/learning-at-home/hivemind/pull/415
* Implement core functionality of hivemind.Optimizer by justheuristic in https://github.com/learning-at-home/hivemind/pull/403
* DHT Benchmark with asynchronous w/r by MuXauJl11110 in https://github.com/learning-at-home/hivemind/pull/406
* Hotfix: load_state_from_peers with offload_optimizer by justheuristic in https://github.com/learning-at-home/hivemind/pull/417
* Improve Optimizer docs, update quickstart to use Optimizer by justheuristic in https://github.com/learning-at-home/hivemind/pull/416
* Quickstart: typos and references by justheuristic in https://github.com/learning-at-home/hivemind/pull/420
* Remove trailing dots in log messages and errors by borzunov in https://github.com/learning-at-home/hivemind/pull/419
* Do not log caller for INFO messages by borzunov in https://github.com/learning-at-home/hivemind/pull/418
* Improve hivemind.optim.experimental and averager stability by borzunov in https://github.com/learning-at-home/hivemind/pull/421
* Add minor tweaks learned from the NeurIPS demo run by justheuristic in https://github.com/learning-at-home/hivemind/pull/422
* Improve All-Reduce fault-tolerance by justheuristic in https://github.com/learning-at-home/hivemind/pull/423
* Fix All-Reduce fault-tolerance: catch Exception instead of BaseException by justheuristic in https://github.com/learning-at-home/hivemind/pull/424
* Fix Task was destroeyd but is pending (put items) by justheuristic in https://github.com/learning-at-home/hivemind/pull/427
* Use hivemind.Optimizer in examples/albert by mryab in https://github.com/learning-at-home/hivemind/pull/426

New Contributors
* artek0chumak made their first contribution in https://github.com/learning-at-home/hivemind/pull/409
* MuXauJl11110 made their first contribution in https://github.com/learning-at-home/hivemind/pull/406

**Full Changelog**: https://github.com/learning-at-home/hivemind/compare/0.10.0...1.0.0

This release contains the following new features and bugfixes:

* Fix deadlocks in DecentralizedAverager and MPFuture (331) (borzunov justheuristic)
* Resolve deadlock in MPFuture (337) (justheuristic borzunov yhn112)
* Convert averager to libp2p backend (323) (borzunov mryab)
* Refactor naming and serialization for PeerIDs (339) (borzunov)
* Set default DHT num_workers = 4 (342) (borzunov deniskamazur justheuristic mryab)
* Fix typo in dht.md (345) (justheuristic)
* Fix some warnings related to asyncio (346) (borzunov)
* Speed up P2P client creation (343) (deniskamazur borzunov)
* Propagate startup errors from DHT and averager processes (347) (borzunov)
* Add less comparator for PeerID (353) (deniskamazur borzunov)
* Fix minor asyncio issues in averager (356) (borzunov justheuristic)
* Optimize unary handlers with persistent connections to P2P daemon (328) (deniskamazur)
* Fix import error breaking AllReduceRunner._send_error_to_peer() (360) (borzunov)
* Fix logger warning in P2P (361) (borzunov)
* Disable QUIC (355) (borzunov)
* Disable elasticity for averaging, add error handling (362) (justheuristic mryab)
* Improve Matchmaking finalizers (357) (borzunov)
* Allow to specify P2P identity file (363) (borzunov)
* Fix loglevel for a message in _read_from_persistent_conn() (364) (borzunov)

This release contains the following features and bugfixes:
* Add p2pd to package_data (287) (mryab)
* Add per-tensor compression, make All-Reduce faster and more flexible (272) (justheuristic mponty mryab yhn112 borzunov)
* Fix race condition while reserving ports in P2P (299) (borzunov)
* Add graceful shutdown to DHT and Averager (301) (justheuristic mryab)
* Make checkpointing optional in example (303) (yhn112)
* Refactor MPFuture to use a single pipe/thread per process (298) (justheuristic borzunov mryab yhn112)
* Split hivemind.client into hivemind.averaging and hivemind.moe (304) (mryab)
* Update readthedocs with hivemind.optim (288) (yhn112 justheuristic)
* Minor fixes in examples/albert (308) (yhn112)
* Upload the model with push_to_hub in examples (297) (leshanbog mryab justheuristic)
* Account for multi-gpu devices in examples/albert (309) (justheuristic)
* Convert DHT to libp2p backend (296) (borzunov skobellev)
* Simplify argument parsing, update docs in ALBERT example (315) (mryab justheuristic yhn112)
* Improve P2P handler throughput and interface (316) (borzunov)
* Remove shared memory from MPFuture, fix minor bugs (317) (justheuristic borzunov mryab)
* Implement protobuf-based stream handlers over libp2p backend (318) (borzunov)
* Refactor for v0.9.10 and fix example (319) (justheuristic borzunov)
* Update quickstart tutorials and acknowledgements (307) (justheuristic yhn112 borzunov mryab)

This release contains the following improvements and bugfixes:

* Add relay options to P2P (268) (deniskamazur)
* Add packaging to requirements (269) (deniskamazur)
* Disable p2pd compilation by default (270) (yhn112 justheuristic)
* Measure testing coverage on pull request (271) (yhn112)
* Update p2pd md5 checksum (273) (deniskamazur)
* Use logging in benchmarks, fix libp2p-related issues (280) (justheuristic)
* Add BibTeX reference for the library to README (283) (mryab)
* Fix Codecov (282) (yhn112)
* Remove use of packaging module (284) (borzunov)
* Support auxiliary peers in CollaborativeOptimizer (279) (yhn112 justheuristic mryab)

This release contains the following improvements and bugfixes:

* Implement combining validators (249) (borzunov)
* Decentralized adaptive optimizers (243) (nevec)
* Add nltk to ALBERT example's requirements (251) (borzunov)
* Protect training progress and metrics with signatures and DHT schema validation (250) (borzunov)
* Add state checkpointing and uploading in coordinator (241) (leshanbog mryab)
* Fix random freezes in averager.step, improve error handling (254) (justheuristic yhn112 borzunov mryab)
* Fix device in Switch-MoE, overhaul Server architecture (256) (mryab)
* Log more stats for user, move performance stats to examples (257) (yhn112)
* Implement authorization for a moderated Hivemind network (255) (borzunov)
* Improve error handling, remove deprecated functionality (261) (justheuristic mryab)
* Log correct loss in examples/albert/run_first_peer.py (265) (borzunov)
* Fixed nan when compressing the tensor of zeros (266) (Vsevolod-pl)
* Support auxiliary participants in AllReduceProtocol (260) (foksly)
* Log collaboration step to Wandb, store metrics only if peer is synchronized (267) (borzunov yhn112 justheuristic)
* Add initial support for connecting via libp2p (238) (MaximKsh deniskamazur skobellev leshanbog borzunov mryab yhn112)

This release contains the following improvements and bugfixes:
* Add RSA signature protection for DHT records (187) (borzunov)
* Improve Runtime exception handling (207) (mryab)
* Implement basic decentralized optimizers (210) (justheuristic, mryab)
* Add gradient clipping support to ExpertBackend (214) (mryab)
* Convert SerializerBase to an abstract class (212) (mryab)
* Replace FeedforwardBlock with a correct implementation (211) (mryab)
* Disentangle DecentralizedAverager components, add averaging weights (217) (justheuristic mryab)
* Add CollaborativeOptimizer, TrainingAverager (215) (leshanbog nevec mryab)
* Move compression-related code to hivemind.utils.compression (213) (mryab)
* Prevent DecentralizedSGD from accidentally skipping a fraction of training batches (218) (ploshkin)
* Add uniform compression (202) (mponty)
* Add gradient buffers to CollaborativeOptimizer (220) (justheuristic)
* Improve zero_grad behavior in CollaborativeOptimizer (221) (justheuristic)
* Reset gradient buffers when synchronizing with peers (222) (justheuristic)
* Add tool for custom user experts (189) (romakail justheuristic)
* Delta gradients transmission (225) (Vsevolod-pl)
* Statistics averaging (229) (nevec)
* Ensure version-consistent result rounding in load_balance_peers (230) (justheuristic mryab)
* Add Switch Transformers-like RemoteMixtureOfExperts (228) (mryab)
* Add example for collaborative ALBERT training (226) (leshanbog yhn112 nevec mryab)
* Fix loss metric calculation (240) (yhn112)
* Add DHT schema validator (227) (borzunov)
* Fix server hanging in certain cases when connection is lost (247) (justheuristic)
* Add Dockerfile, refactor tests (245) (mryab)
* Fix incorrect data types/values in RemoteSwitchMixtureOfExperts (246) (mryab)