Modin

This release contains a few important improvements for MPI backend.

Key Features and Updates
------------------------
* Stability and Bugfixes
* FIX-402: Fix MPICH version check (403)
* FIX-404: Add psutil as a required dependency for unidist[mpi] (406)
* FIX-407: Make data put into MPI object store immutable (409)
* Performance enhancements
* PERF-360: Re - Implement asyncronous get request with a fix (405)
* Refactor Codebase
* REFACTOR-000: Refactor remote function class for MPI backend (410)

Contributors
------------
YarShev
Retribution98
arunjose696

This release contains many performance enhancements and minor bugfixes. Several new features were also added this release.


Bugfixes + Pandas Concordance (🐛 + 🐼)
* Fix sorting for MultiIndex (705)
* Fix parallel read_sql on Postgres tables (707)
* Add fix for axis_partition operations after a join/concat (704)
* Fix case where SQLAlchemy connections are passed in to read_sql (712)
* Fix index mismatch for add (710)
* Fixes indexing with boolean list in loc (735)
* Default to pandas on MultiLevelIndex `reset_index` (739)


New Functionality ✨
* reading gzipped csv files (682)
* Add support for text manipulation operations (713)
* Add support for `dot` (719)
* Add to_numpy method (718)
* Read_csv with compression "bz2", "zip", and "xz" (722)


Backend enhancements + Performance 🚀
* Improve performance of `__setitem__` on new column (701)
* Improve fillna performance depending on arguments passed in (709)
* Add metadata to be returned by operations along entire axis (699)
* Add a way to compute pre-compute metadata on read_csv (714)
* Add metadata to object when reading from parquet (716)
* Improve performance of dtypes computation by collecting at data ingest (717)
* Update partition width calculation for read csv with a ray engine (728)
* Add to_numpy to Frame Manager, Query Compiler, and BaseDataFrame (726)

Code Quality 💯
* Change datamanager to querycompiler (702)
* Code quality/reorganize query compiler (706)

Dependencies 🔗
* Update ray version to 0.7.1 (697)


Contributors this release

The following users contributed code to Modin since the last release.

dulinda (First time contributor) 🌟
RehanSD (First time contributor) 🌟
williamma12 (Maintainer)
devin-petersohn (Maintainer)

🎉🎉 Thank you! 🎉🎉

This release includes several fixes to regressions and documentation. We also have preliminary support for an autoscaling cluster (661). Performance `groupby` + `sum`, `count`, and other dimension reducing operations was increased by up to 10x from the previous implementation (659).

Bugfixes + Pandas Concordance (🐛 + 🐼)
* Fix `usecols` when the string name of the column is provided (652)
* Fixes full axis reduce functions with empty row and/or column partitions (663)
* Fix memory_usage() for transposed dataframes (662)
* Default to pandas when trying to get tuple from Series (689)
* Fix dtypes on empty dataframes calls to to_pandas (688)

New Functionality ✨
* Initial ray autoscaler support (661) 🎉

Backend enhancements + Performance 🚀
* Improve performance of Groupby (659) 🎉
* Fix internal indices calculation for non-compute partitions (691)

Documentation 📑
* Update documentation for DataFrame methods (643)
* Update Documentation structure (665, 666, 667, 668)
* Update utilities documentation for pandas on ray (669)

Testing 📈
* Remove extraneous teardown module for parquet (648)

Regressions ↩️
* Fix binary operations after transpose (676)
* Fix issue with `sort_values` after transpose (679)
* Fix `concat` when QueryCompiler is transposed (681)
* Fix `concat` with all Series and axis=1 (684)
* Fix how we compute the block_widths/lengths after single update (693)

Contributors this release

The following users contributed code to Modin since the last release.

williamma12 (Committer)
devin-petersohn (Admin)

🎉🎉 Thank you! 🎉🎉

This release is a hotfix for a bug/regression introduced in 0.5.1.

Bugfixes + Pandas Concordance (🐛 + 🐼)
* Fix Parquet reader for partitioned files (644)

This release contains an important bugfix for MPI backend and
adds psutil as a required dependency for unidist[mpi] package.

Key Features and Updates
------------------------
* Stability and Bugfixes
* FIX-https://github.com/modin-project/unidist/issues/402: Fix MPICH version check (https://github.com/modin-project/unidist/pull/403)
* FIX-https://github.com/modin-project/unidist/issues/404: Add psutil as a required dependency for unidist[mpi] (https://github.com/modin-project/unidist/pull/406)

Contributors
------------
YarShev
Retribution98

This release adds support for a shared object store for MPI backend,
which can be enabled with UNIDIST_MPI_SHARED_OBJECT_STORE environment variable,
contains a lot of performance enhancements, includes several features,
important documentation updates and a bunch of bugfixes.

Key Features and Updates
------------------------
* Stability and Bugfixes
* FIX-329: Allow remaining communication requests to complete (330)
* FIX-343: Get rid of the hacks in the MPI backend (344)
* FIX-345: Test both modes of MPI execution on Linux only (346)
* FIX-349: Remove the warning regarding MpiPickleThreshold (350)
* FIX-354: Fix unidist.wait for MPI backend (355)
* FIX-363: Fix DataID usage between different MPI processes (364)
* FIX-371: Fix check for out of bounds reserved memory in shared storage (372)
* FIX-375: Fix running an MPI cluster without using shared memory (377)
* FIX-340: Fix hangs at low UNIDIST_MPI_SHARED_OBJECT_STORE_THRESHOLD (380)
* FIX-376: Partition buffers of size larger than the _bigmpi.blocksize into blocks while sending and receiving with MPI (383)
* FIX-337: Fix MpiHosts parameter usage (378)
* FIX-000: Revert PR 387 (388)
* FEAT-392: Raise a warning if shared object storage is enabled but the MPI library does not support it (393)
* FIX-389: Revert async get changes made in 359 (390)
* FIX-394: Fix CI running on Windows (395)
* FIX-396: Fix Split_type call for Unidist on MSMPI (397)
* Performance enhancements
* PERF-365: Cache info about sent metadata of shared data (366)
* PERF-360: Implement asyncronous get request (359)
* PERF-367: Use std::fill to fill service shared buffer with a given value (373)
* New Features
* FEAT-285: Implement shared object storage for MPI backend (286)
* FEAT-341: Add the ability to specify environment variables for MPI workers (352)
* FEAT-347: Get rid of mpi4py-mpich dependency (358)
* FEAT-386: Make MPI shared object store default option (387)
* Refactor Codebase
* REFACTOR-000: Remove unnecessary build step in CI (339)
* REFACTOR-000: Introduce task_info metadata package (348)
* REFACTOR-000: Remove unncessary else branch in mpi init (351)
* REFACTOR-000: Change mpi_state.comm to mpi_state.global_comm (379)
* REFACTOR-000: Change IsMpiSpawnWorkers to MpiSpawn to make it more concise (382)
* Documentation improvements
* DOCS-000: Update the workflow status badge (332)
* DOCS-000: Add the downloads badge in readme (333)
* DOCS-384: Fix warnings when building documentation (385)
* DOCS-336: Add documentation about shared object storage (391)

Contributors
------------
YarShev
Retribution98
arunjose696