Gensim

**:warning: 3.8.x will be the last Gensim version to support Py2.7. Starting with 4.0.0, Gensim will only support Py3.5 and above.**

This is primarily a bugfix release to bring back Py2.7 compatibility to gensim 3.8.

:red_circle: Bug fixes

* Bring back Py27 support (PR [2812](https://github.com/RaRe-Technologies/gensim/pull/2812), [mpenkov](https://github.com/mpenkov))
* Fix wrong version reported by setup.py (Issue [2796](https://github.com/RaRe-Technologies/gensim/issues/2796))
* Fix missing C extensions (Issues [2794](https://github.com/RaRe-Technologies/gensim/issues/2794) and [#2802](https://github.com/RaRe-Technologies/gensim/issues/2802))

:+1: Improvements

* Wheels for Python 3.8 ([menshikh-iv](https://github.com/menshikh-iv))
* Prepare for removal of deprecated `lxml.etree.cElementTree` (PR [2777](https://github.com/RaRe-Technologies/gensim/pull/2777), [tirkarthi](https://github.com/tirkarthi))

:books: Tutorial and doc improvements

* Update test instructions in README (PR [2814](https://github.com/RaRe-Technologies/gensim/pull/2814), [piskvorky](https://github.com/piskvorky))

:warning: Deprecations (will be removed in the next major release)

* Remove
- `gensim.models.FastText.load_fasttext_format`: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)
- `gensim.models.wrappers.fasttext` (obsoleted by the new native `gensim.models.fasttext` implementation)
- `gensim.examples`
- `gensim.nosy`
- `gensim.scripts.word2vec_standalone`
- `gensim.scripts.make_wiki_lemma`
- `gensim.scripts.make_wiki_online`
- `gensim.scripts.make_wiki_online_lemma`
- `gensim.scripts.make_wiki_online_nodebug`
- `gensim.scripts.make_wiki` (all of these obsoleted by the new native `gensim.scripts.segment_wiki` implementation)
- "deprecated" functions and attributes

* Move
- `gensim.scripts.make_wikicorpus` ➡ `gensim.scripts.make_wiki.py`
- `gensim.summarization` ➡ `gensim.models.summarization`
- `gensim.topic_coherence` ➡ `gensim.models._coherence`
- `gensim.utils` ➡ `gensim.utils.utils` (old imports will continue to work)
- `gensim.parsing.*` ➡ `gensim.utils.text_utils`

---

:red_circle: Bug fixes

* Pin `smart_open` version for compatibility with Py2.7

:warning: Deprecations (will be removed in the next major release)

* Remove
- `gensim.models.FastText.load_fasttext_format`: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)
- `gensim.models.wrappers.fasttext` (obsoleted by the new native `gensim.models.fasttext` implementation)
- `gensim.examples`
- `gensim.nosy`
- `gensim.scripts.word2vec_standalone`
- `gensim.scripts.make_wiki_lemma`
- `gensim.scripts.make_wiki_online`
- `gensim.scripts.make_wiki_online_lemma`
- `gensim.scripts.make_wiki_online_nodebug`
- `gensim.scripts.make_wiki` (all of these obsoleted by the new native `gensim.scripts.segment_wiki` implementation)
- "deprecated" functions and attributes

* Move
- `gensim.scripts.make_wikicorpus` ➡ `gensim.scripts.make_wiki.py`
- `gensim.summarization` ➡ `gensim.models.summarization`
- `gensim.topic_coherence` ➡ `gensim.models._coherence`
- `gensim.utils` ➡ `gensim.utils.utils` (old imports will continue to work)
- `gensim.parsing.*` ➡ `gensim.utils.text_utils`

---

:red_circle: Bug fixes

* Fix usage of base_dir instead of BASE_DIR in _load_info in downloader. ([movb](https://github.com/movb), [#2605](https://github.com/RaRe-Technologies/gensim/pull/2605))
* Update the version of smart_open in the setup.py file ([AMR-KELEG](https://github.com/AMR-KELEG), [#2582](https://github.com/RaRe-Technologies/gensim/pull/2582))
* Properly handle unicode_errors arg parameter when loading a vocab file ([wmtzk](https://github.com/wmtzk), [#2570](https://github.com/RaRe-Technologies/gensim/pull/2570))
* Catch loading older TfidfModels without smartirs ([bnomis](https://github.com/bnomis), [#2559](https://github.com/RaRe-Technologies/gensim/pull/2559))
* Fix bug where a module import set up logging, pin doctools for Py2 ([piskvorky](https://github.com/piskvorky), [#2552](https://github.com/RaRe-Technologies/gensim/pull/2552))

:books: Tutorial and doc improvements

* Fix usage example in phrases.py ([piskvorky](https://github.com/piskvorky), [#2575](https://github.com/RaRe-Technologies/gensim/pull/2575))

:+1: Improvements

* Optimize Poincare model training ([koiizukag](https://github.com/koiizukag), [#2589](https://github.com/RaRe-Technologies/gensim/pull/2589))

:warning: Deprecations (will be removed in the next major release)

* Remove
- `gensim.models.FastText.load_fasttext_format`: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)
- `gensim.models.wrappers.fasttext` (obsoleted by the new native `gensim.models.fasttext` implementation)
- `gensim.examples`
- `gensim.nosy`
- `gensim.scripts.word2vec_standalone`
- `gensim.scripts.make_wiki_lemma`
- `gensim.scripts.make_wiki_online`
- `gensim.scripts.make_wiki_online_lemma`
- `gensim.scripts.make_wiki_online_nodebug`
- `gensim.scripts.make_wiki` (all of these obsoleted by the new native `gensim.scripts.segment_wiki` implementation)
- "deprecated" functions and attributes

* Move
- `gensim.scripts.make_wikicorpus` ➡ `gensim.scripts.make_wiki.py`
- `gensim.summarization` ➡ `gensim.models.summarization`
- `gensim.topic_coherence` ➡ `gensim.models._coherence`
- `gensim.utils` ➡ `gensim.utils.utils` (old imports will continue to work)
- `gensim.parsing.*` ➡ `gensim.utils.text_utils`

---

:star2: New Features

* Enable online training of Poincare models ([koiizukag](https://github.com/koiizukag), [#2505](https://github.com/RaRe-Technologies/gensim/pull/2505))
* Make BM25 more scalable by adding support for generator inputs ([saraswatmks](https://github.com/saraswatmks), [#2479](https://github.com/RaRe-Technologies/gensim/pull/2479))
* Allow the Gensim dataset / pre-trained model downloader `gensim.downloader` to run offline, by introducing a local file cache ([mpenkov](https://github.com/mpenkov), [#2545](https://github.com/RaRe-Technologies/gensim/pull/2545))
* Make the `gensim.downloader` target directory configurable ([mpenkov](https://github.com/mpenkov), [#2456](https://github.com/RaRe-Technologies/gensim/pull/2456))
* Add `nmslib` indexer ([masa3141](https://github.com/masa3141), [#2417](https://github.com/RaRe-Technologies/gensim/pull/2417))

:red_circle: Bug fixes

* Fix `smart_open` deprecation warning globally ([itayB](https://github.com/itayB), [#2530](https://github.com/RaRe-Technologies/gensim/pull/2530))
* Fix AppVeyor issues with Windows and Py2 ([mpenkov](https://github.com/mpenkov), [#2546](https://github.com/RaRe-Technologies/gensim/pull/2546))
* Fix `topn=0` versus `topn=None` bug in `most_similar`, accept `topn` of any integer type ([Witiko](https://github.com/Witiko), [#2497](https://github.com/RaRe-Technologies/gensim/pull/2497))
* Fix Python version check ([charsyam](https://github.com/charsyam), [#2547](https://github.com/RaRe-Technologies/gensim/pull/2547))
* Fix typo in FastText documentation ([Guitaricet](https://github.com/Guitaricet), [#2518](https://github.com/RaRe-Technologies/gensim/pull/2518))
* Fix "Market Matrix" to "Matrix Market" typo. ([Shooter23](https://github.com/Shooter23), [#2513](https://github.com/RaRe-Technologies/gensim/pull/2513))
* Fix auto-generated hyperlinks in `CHANGELOG.md` ([mpenkov](https://github.com/mpenkov), [#2482](https://github.com/RaRe-Technologies/gensim/pull/2482))

:books: Tutorial and doc improvements

* Generate documentation for the `gensim.similarities.termsim` module ([Witiko](https://github.com/Witiko), [#2485](https://github.com/RaRe-Technologies/gensim/pull/2485))
* Simplify the `Support` section in README ([piskvorky](https://github.com/piskvorky), [#2542](https://github.com/RaRe-Technologies/gensim/pull/2542))

:+1: Improvements

* Pin sklearn version for Py2, because sklearn dropped py2 support ([mpenkov](https://github.com/mpenkov), [#2510](https://github.com/RaRe-Technologies/gensim/pull/2510))


:warning: Deprecations (will be removed in the next major release)

* Remove
- `gensim.models.FastText.load_fasttext_format`: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)
- `gensim.models.wrappers.fasttext` (obsoleted by the new native `gensim.models.fasttext` implementation)
- `gensim.examples`
- `gensim.nosy`
- `gensim.scripts.word2vec_standalone`
- `gensim.scripts.make_wiki_lemma`
- `gensim.scripts.make_wiki_online`
- `gensim.scripts.make_wiki_online_lemma`
- `gensim.scripts.make_wiki_online_nodebug`
- `gensim.scripts.make_wiki` (all of these obsoleted by the new native `gensim.scripts.segment_wiki` implementation)
- "deprecated" functions and attributes

* Move
- `gensim.scripts.make_wikicorpus` ➡ `gensim.scripts.make_wiki.py`
- `gensim.summarization` ➡ `gensim.models.summarization`
- `gensim.topic_coherence` ➡ `gensim.models._coherence`
- `gensim.utils` ➡ `gensim.utils.utils` (old imports will continue to work)
- `gensim.parsing.*` ➡ `gensim.utils.text_utils`

:red_circle: Bug fixes

* Fix fasttext model loading from gzip files ([mpenkov](https://github.com/mpenkov), [#2476](https://github.com/RaRe-Technologies/gensim/pull/2476))
* Fix misleading `Doc2Vec.docvecs` comment ([gojomo](https://github.com/gojomo), [#2472](https://github.com/RaRe-Technologies/gensim/pull/2472))
* NMF bugfix ([mpenkov](https://github.com/mpenkov), [#2466](https://github.com/RaRe-Technologies/gensim/pull/2466))
* Fix `WordEmbeddingsKeyedVectors.most_similar` ([Witiko](https://github.com/Witiko), [#2461](https://github.com/RaRe-Technologies/gensim/pull/2461))
* Fix LdaSequence model by updating to num_documents ([Bharat123rox](https://github.com/Bharat123rox), [#2410](https://github.com/RaRe-Technologies/gensim/pull/2410))
* Make termsim matrix positive definite even with negative similarities ([Witiko](https://github.com/Witiko), [#2397](https://github.com/RaRe-Technologies/gensim/pull/2397))
* Fix the off-by-one bug in the TFIDF model. ([AMR-KELEG](https://github.com/AMR-KELEG), [#2392](https://github.com/RaRe-Technologies/gensim/pull/2392))
* Update legacy model loading ([mpenkov](https://github.com/mpenkov), [#2454](https://github.com/RaRe-Technologies/gensim/pull/2454), [#2457](https://github.com/RaRe-Technologies/gensim/pull/2457))
* Make `matutils.unitvec` always return float norm when requested ([Witiko](https://github.com/Witiko), [#2419](https://github.com/RaRe-Technologies/gensim/pull/2419))

:books: Tutorial and doc improvements

* Update word2vec.ipynb ([asyabo](https://github.com/asyabo), [#2423](https://github.com/RaRe-Technologies/gensim/pull/2423))

:+1: Improvements

* Adding type check for corpus_file argument ([saraswatmks](https://github.com/saraswatmks), [#2469](https://github.com/RaRe-Technologies/gensim/pull/2469))
* Clean up FastText Cython code, fix division by zero ([mpenkov](https://github.com/mpenkov), [#2382](https://github.com/RaRe-Technologies/gensim/pull/2382))

:warning: Deprecations (will be removed in the next major release)

* Remove
- `gensim.models.FastText.load_fasttext_format`: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)
- `gensim.models.wrappers.fasttext` (obsoleted by the new native `gensim.models.fasttext` implementation)
- `gensim.examples`
- `gensim.nosy`
- `gensim.scripts.word2vec_standalone`
- `gensim.scripts.make_wiki_lemma`
- `gensim.scripts.make_wiki_online`
- `gensim.scripts.make_wiki_online_lemma`
- `gensim.scripts.make_wiki_online_nodebug`
- `gensim.scripts.make_wiki` (all of these obsoleted by the new native `gensim.scripts.segment_wiki` implementation)
- "deprecated" functions and attributes

* Move
- `gensim.scripts.make_wikicorpus` ➡ `gensim.scripts.make_wiki.py`
- `gensim.summarization` ➡ `gensim.models.summarization`
- `gensim.topic_coherence` ➡ `gensim.models._coherence`
- `gensim.utils` ➡ `gensim.utils.utils` (old imports will continue to work)
- `gensim.parsing.*` ➡ `gensim.utils.text_utils`