Deeppavlov

Breaking Changes
Removed Models
- FAQ models, [tfidf_vectorizer](https://github.com/deeppavlov/DeepPavlov/blob/1.0.2/deeppavlov/configs/embedder/tfidf_vectorizer.json), [tfidf_logreg_autofaq_psearch](https://github.com/deeppavlov/DeepPavlov/blob/1.0.2/deeppavlov/configs/paramsearch/tfidf_logreg_autofaq_psearch.json) and [cv_tfidf_autofaq](https://github.com/deeppavlov/DeepPavlov/blob/1.0.2/deeppavlov/configs/cv/cv_tfidf_autofaq.json) ([#1608](https://github.com/deeppavlov/DeepPavlov/pull/1608))

Removed Components
- [ru_tokenizer](https://github.com/deeppavlov/DeepPavlov/blob/1.0.2/deeppavlov/models/tokenizers/ru_tokenizer.py) ([#1608](https://github.com/deeppavlov/DeepPavlov/pull/1608))

Major Features and Improvements
- **Python 3.10** support ([1614](https://github.com/deeppavlov/DeepPavlov/pull/1614)).
- Few-shot classification [example](https://github.com/deeppavlov/DeepPavlov/blob/1.1.0/deeppavlov/configs/faq/fasttext_logreg.json) and few-shot sampling support added to [basic_classification_iterator](https://github.com/deeppavlov/DeepPavlov/blob/1.1.0/deeppavlov/dataset_iterators/basic_classification_iterator.py) ([#1608](https://github.com/deeppavlov/DeepPavlov/pull/1608)).

Other Changes
- `pymorphy2` is replaced with `spacy` in [entity_linker](https://github.com/deeppavlov/DeepPavlov/blob/1.1.0/deeppavlov/models/entity_extraction/entity_linking.py#L34), [ru_adj_to_noun](https://github.com/deeppavlov/DeepPavlov/blob/1.1.0/deeppavlov/models/kbqa/tree_to_sparql.py#L38) and [answer_types_extractor](https://github.com/deeppavlov/DeepPavlov/blob/1.1.0/deeppavlov/models/kbqa/type_define.py#L26) components ([1618](https://github.com/deeppavlov/DeepPavlov/pull/1618)).

Fixes
- Fixed kbqa_cq_ru on Python 3.8/3.9 ([1609](https://github.com/deeppavlov/DeepPavlov/pull/1609))

Documentation
- Added python inference pipelines example ([1613](https://github.com/deeppavlov/DeepPavlov/pull/1613)).

Other changes
- Updated model files upload script ([1607](https://github.com/deeppavlov/DeepPavlov/pull/1607)).

Major Features and Improvements
- Added `-i`/`--install` CLI argument and `install` argument to `deeppavlov.build_model`, `deeppavlov.evaluate_model`, `deeppavlov.train_model` to install model requirements before interaction with model ([1603](https://github.com/deeppavlov/DeepPavlov/pull/1603)).

Bug Fixes and Other Changes
- Reduced library verbosity: redundant logging `info` messages replaced with `debug` ones. Set `nltk.download` to quiet mode. ([1601](https://github.com/deeppavlov/DeepPavlov/pull/1601)).
- Replaced `docs/features/models/classifiers.rst` with `docs/features/models/classification.ipynb`. Fixed minor typos in documentation, removed `skill` concept ([1600](https://github.com/deeppavlov/DeepPavlov/pull/1600)).
- Removed `/examples` from the README.md links ([1602](https://github.com/deeppavlov/DeepPavlov/pull/1602)).

- Added Python 3.8 and 3.9 support and library requirements are optimized in 1525 and 1578.
- Removed all TensorFlow components and default trainer replaced with `torch_trainer` in 1574.
- Added Russian SuperGLUE models and submission generation in 1577.
- Added NER case-agnostig config in 1570.
- Added external metrics support in 1546.
- Nested config overwriting mechanism in 1561.
- Refactoring of the training logging in 1572.
- KBQA models migrated to PyTorch in 1569.
- Classification models migrated to PyTorch in 1565.
- NER models migrated to PyTorch in 1545.
- Context question answering models migrated to PyTorch in 1539.
- Entity Linking migrated to PyTorch and reduced RAM and VRAM consumption in 1516.
- Added config deprecation mechanism in 1547.
- `torch_bert_ranker` now uses the same Hugging Face Transformers version as the rest of the components in 1532.
- Models and components removed in 1523, 1526, 1534, 1533, 1537, 1544, 1560, 1563, 1564, 1573.
- Fixed a problem with pre-trained BERT models by DeepPavlov in 1502. Resolves 1275 and 1390.
- Fixed en_core_web_sm load error during tests in 1524.
- Removed Telegram, MSBot Framework, Yandex Alice and Amazon Alexa connectors in 1548.
- Documentation updated in 1517, 1529.

**Note**: DeepPavlov `1.0.0` is not released yet!

Renamed models:
When `a.json` is renamed to `b.json`, original `b.json` is removed.
- [squad_ru_torch_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/squad/squad_ru_torch_bert.json) -> [squad_ru_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/squad/squad_ru_bert.json)
- [ner_rus_bert_torch](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/ner_rus_bert_torch.json) -> [ner_rus_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/ner_rus_bert.json)
- [insults_kaggle_bert_torch](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/insults_kaggle_bert_torch.json) -> [insults_kaggle_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/insults_kaggle_bert.json)

TensorFlow replaced by PyTorch
- [squad_ru_bert_infer](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/squad/squad_ru_bert_infer.json)

Removed models:
**asr**
- [asr.json](https://github.com/deepmipt/DeepPavlov/blob/0.17.2/deeppavlov/configs/nemo/asr.json)
- [tts.json](https://github.com/deepmipt/DeepPavlov/blob/0.17.2/deeppavlov/configs/nemo/tts.json)
- [asr_tts.json](https://github.com/deepmipt/DeepPavlov/blob/0.17.2/deeppavlov/configs/nemo/asr_tts.json)

**elmo**
- [rusentiment_elmo_twitter_cnn](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/rusentiment_elmo_twitter_cnn.json)
- [elmo_en_1billion](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/embedder/elmo_en_1billion.json)
- [elmo_ru_news](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/embedder/elmo_ru_news.json)
- [elmo_ru_twitter](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/embedder/elmo_ru_twitter.json)
- [elmo_ru_wiki](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/embedder/elmo_ru_wiki.json)

**classifiers**
- [insults_kaggle](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/insults_kaggle.json)
- [insults_kaggle_conv_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/insults_kaggle_conv_bert.json)
- [intents_dstc2](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/intents_dstc2.json)
- [intents_dstc2_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/intents_dstc2_bert.json)
- [intents_dstc2_big](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/intents_dstc2_big.json)
- [intents_sample_csv](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/intents_sample_csv.json)
- [intents_sample_json](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/intents_sample_json.json)
- [intents_snips](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/intents_snips.json)
- [intents_snips_big](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/intents_snips_big.json)
- [intents_snips_sklearn](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/intents_snips_sklearn.json)
- [intents_snips_tfidf_weighted](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/intents_snips_tfidf_weighted.json)
- [relation_prediction_rus](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/relation_prediction_rus.json)
- [ru_obscenity_classifier](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/ru_obscenity_classifier.json)
- [rusentiment_bigru_superconv](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/rusentiment_bigru_superconv.json)
- [rusentiment_cnn](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/rusentiment_cnn.json)
- [sentiment_imdb_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/sentiment_imdb_bert.json)
- [sentiment_imdb_conv_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/sentiment_imdb_conv_bert.json)
- [sentiment_sst_multi_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/sentiment_sst_multi_bert.json)
- [sentiment_twitter_bert_emb](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/sentiment_twitter_bert_emb.json)
- [sentiment_twitter_preproc](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/sentiment_twitter_preproc.json)
- [sentiment_yelp_conv_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/sentiment_yelp_conv_bert.json)
- [sentiment_yelp_multi_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/sentiment_yelp_multi_bert.json)
- [sst_torch_swcnn](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/sst_torch_swcnn.json)
- [topic_ag_news](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/topic_ag_news.json)
- [yahoo_convers_vs_info](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/yahoo_convers_vs_info.json)
- [yahoo_convers_vs_info_bert](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/classifiers/yahoo_convers_vs_info_bert.json)

**doc_retrieval**
- [en_ranker_tfidf_enwiki20161221](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/doc_retrieval/en_ranker_tfidf_enwiki20161221.json)

**go_bot**
- [database_dstc2](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/go_bot/database_dstc2.json)
- [gobot_dstc2](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/go_bot/gobot_dstc2.json)
- [gobot_dstc2_best](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/go_bot/gobot_dstc2_best.json)
- [gobot_dstc2_best_json_nlg](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/go_bot/gobot_dstc2_best_json_nlg.json)
- [gobot_simple_dstc2](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/go_bot/gobot_simple_dstc2.json)

**morpho_tagger**
- [morpho_ar](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_ar.json)
- [morpho_cs](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_cs.json)
- [morpho_de](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_de.json)
- [morpho_en](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_en.json)
- [morpho_es_ancora](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_es_ancora.json)
- [morpho_fr](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_fr.json)
- [morpho_hi](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_hi.json)
- [morpho_hu](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_hu.json)
- [morpho_it](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_it.json)
- [morpho_ru_syntagrus](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_ru_syntagrus.json)
- [morpho_ru_syntagrus_pymorphy](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_ru_syntagrus_pymorphy.json)
- [morpho_ru_syntagrus_pymorphy_lemmatize](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_ru_syntagrus_pymorphy_lemmatize.json)
- [morpho_tr](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/morpho_tagger/UD2.0/morpho_tr.json)

**ner**
- [ner_conll2003_pos](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/ner_conll2003_pos.json)
- [ner_dstc2](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/ner_dstc2.json)
- [ner_few_shot_ru](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/ner_few_shot_ru.json)
- [ner_few_shot_ru_simulate](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/ner_few_shot_ru_simulate.json)
- [ner_kb_rus](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/ner_kb_rus.json)
- [ner_lcquad_bert_probas](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/ner_lcquad_bert_probas.json)
- [ner_ontonotes_m1](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/ner_ontonotes_m1.json)
- [slotfill_dstc2](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/slotfill_dstc2.json)
- [slotfill_dstc2_raw](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/slotfill_dstc2_raw.json)
- [slotfill_simple_dstc2_raw](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/slotfill_simple_dstc2_raw.json)
- [slotfill_simple_rasa_raw](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/slotfill_simple_rasa_raw.json)
- [vlsp2016_full](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ner/vlsp2016_full.json)

**odqa**
- [en_odqa_infer_enwiki20161221](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/odqa/en_odqa_infer_enwiki20161221.json)

**ranking**
- [paraphrase_ident_paraphraser](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ranking/paraphrase_ident_paraphraser.json)
- [paraphrase_ident_paraphraser_interact](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ranking/paraphrase_ident_paraphraser_interact.json)
- [ranking_ubuntu_v2_mt](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ranking/ranking_ubuntu_v2_mt.json)
- [ranking_ubuntu_v2_mt_interact](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/ranking/ranking_ubuntu_v2_mt_interact.json)

**spelling_correction**
- [brillmoore_kartaslov_ru](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/spelling_correction/brillmoore_kartaslov_ru.json)
- [brillmoore_kartaslov_ru_custom_vocab](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/spelling_correction/brillmoore_kartaslov_ru_custom_vocab.json)
- [brillmoore_kartaslov_ru_nolm](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/spelling_correction/brillmoore_kartaslov_ru_nolm.json)

**squad**
- [squad_bert_uncased](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/squad/squad_bert_uncased.json)
- [squad_zh_bert_mult](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/squad/squad_zh_bert_mult.json)
- [squad_zh_bert_zh](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/configs/squad/squad_zh_bert_zh.json)

Removed components:
- [ner_few_shot_iterator](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/dataset_iterators/ner_few_shot_iterator.py)
- [ru_obscenity_classifier](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/models/classifiers/ru_obscenity_classifier.py)
- [snips_intents_iterator](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/dataset_iterators/snips_intents_iterator.py)
- [snips_ner_iterator](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/dataset_iterators/snips_ner_iterator.py)
- [snips_reader](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/dataset_readers/snips_reader.py)
- [mpm_nn](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/models/ranking/mpm_siamese_network.py)
- [bilstm_gru_nn](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/models/ranking/bilstm_gru_siamese_network.py)
- [morpho_tagger](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/models/morpho_tagger/morpho_tagger.py)
- [siamese_predictor](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/models/ranking/siamese_predictor.py)
- [elmo_embedder](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/models/embedders/elmo_embedder.py)
- [ner_bio_converter](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/models/ner/bio.py)
- [ner_svm](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/models/ner/svm.py)
- [jieba_tokenizer](https://github.com/deepmipt/DeepPavlov/blob/0.17.1/deeppavlov/models/tokenizers/jieba_tokenizer.py)
- [base64_decode_bytesIO](https://github.com/deepmipt/DeepPavlov/blob/08555428388fed3c7b036c0a82a70a25efcabcff/deeppavlov/models/nemo/common.py#L35)
- [bytesIO_encode_base64](https://github.com/deepmipt/DeepPavlov/blob/08555428388fed3c7b036c0a82a70a25efcabcff/deeppavlov/models/nemo/common.py#L53)
- [nemo_asr](https://github.com/deepmipt/DeepPavlov/blob/08555428388fed3c7b036c0a82a70a25efcabcff/deeppavlov/models/nemo/asr.py#L145)
- [nemo_tts](https://github.com/deepmipt/DeepPavlov/blob/08555428388fed3c7b036c0a82a70a25efcabcff/deeppavlov/models/nemo/tts.py#L133)

Other features:
- Python 3.8 and 3.9 support for non-tensorflow-based models
- [Upload DeepPavlov BERT models with MLM & NSP heads parameters](https://github.com/deepmipt/DeepPavlov/pull/1502/files)
- removed [examples](https://github.com/deepmipt/DeepPavlov/tree/0.17.1/examples) directory
- minor fixes and improvements