SpeechToText
Two new models are released as part of the S2T implementation: `Speech2TextModel` and `Speech2TextForConditionalGeneration`, in PyTorch.
Speech2Text is a speech model that accepts a float tensor of log-mel filter-bank features extracted from the speech signal. It’s a transformer-based seq2seq model, so the transcripts/translations are generated autoregressively.
The Speech2Text model was proposed in [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://arxiv.org/abs/2010.05171) by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino.
Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=speech_to_text
- Speech2TextTransformer 10175 (patil-suraj)
M2M100
Two new models are released as part of the M2M100 implementation: `M2M100Model` and `M2M100ForConditionalGeneration`, in PyTorch.
M2M100 is a multilingual encoder-decoder (seq-to-seq) model primarily intended for translation tasks.
The M2M100 model was proposed in [Beyond English-Centric Multilingual Machine Translation](https://arxiv.org/abs/2010.11125) by Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, Armand Joulin.
Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=m2m_100
- Add m2m100 10236 (patil-suraj)
I-BERT
Six new models are released as part of the I-BERT implementation: `IBertModel`, `IBertForMaskedLM`, `IBertForSequenceClassification`, `IBertForMultipleChoice`, `IBertForTokenClassification` and `IBertForQuestionAnswering`, in PyTorch.
I-BERT is a quantized version of RoBERTa running inference up to four times faster.
The I-BERT framework in PyTorch allows to identify the best parameters for quantization. Once the model is exported in a framework that supports int8 execution (such as TensorRT), a speedup of up to 4x is visible, with no loss in performance thanks to the parameter search.
The I-BERT model was proposed in [I-BERT: Integer-only BERT Quantization](https://arxiv.org/abs/2101.01321) by Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney and Kurt Keutzer.
Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=ibert
- I-BERT model support 10153 (kssteven418)
- [IBert] Correct link to paper 10445 (patrickvonplaten)
- Add I-BERT to README 10462 (LysandreJik)
Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=speech_to_text
mBART-50
MBart-50 is created using the original mbart-large-cc25 checkpoint by extending its embedding layers with randomly initialized vectors for an extra set of 25 language tokens and then pretrained on 50 languages.
The MBart model was presented in [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://arxiv.org/abs/2008.00401) by Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, Angela Fan.
Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=mbart-50
- Add mBART-50 10154 (patil-suraj)
DeBERTa-v2
Fixe new models are released as part of the DeBERTa-v2 implementation: `DebertaV2Model`, `DebertaV2ForMaskedLM`, `DebertaV2ForSequenceClassification`, `DeberaV2ForTokenClassification` and `DebertaV2ForQuestionAnswering`, in PyTorch.
The DeBERTa model was proposed in [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen. It is based on Google’s BERT model released in 2018 and Facebook’s RoBERTa model released in 2019.
It builds on RoBERTa with disentangled attention and enhanced mask decoder training with half of the data used in RoBERTa.
Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=deberta-v2
- Integrate DeBERTa v2(the 1.5B model surpassed human performance on Su… 10018 (BigBird01)
- DeBERTa-v2 fixes 10328 (LysandreJik)
Wav2Vec2
XLSR-Wav2Vec2
The XLSR-Wav2Vec2 model was proposed in [Unsupervised Cross-Lingual Representation Learning For Speech Recognition](https://arxiv.org/abs/2006.13979) by Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, Michael Auli.
The checkpoint corresponding to that model is added to the model hub: [facebook/
wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53)
- [XLSR-Wav2Vec2] Add multi-lingual Wav2Vec2 models 10648 (patrickvonplaten)
Training script
A fine-tuning script showcasing how the Wav2Vec2 model can be trained has been added.
- Add Fine-Tuning for Wav2Vec2 10145 (patrickvonplaten)
Further improvements
The Wav2Vec2 architecture becomes more stable as several changes are done to its architecture. This introduces feature extractors and feature processors as the pre-processing aspect of multi-modal speech models.
- Deprecate Wav2Vec2ForMaskedLM and add Wav2Vec2ForCTC 10089 (patrickvonplaten)
- Fix example in Wav2Vec2 documentation 10096 (abhishekkrthakur)
- [Wav2Vec2] Remove unused config 10457 (patrickvonplaten)
- [Wav2Vec2FeatureExtractor] smal fixes 10455 (patil-suraj)
- [Wav2Vec2] Improve Tokenizer & Model for batched inference 10117 (patrickvonplaten)
- [PretrainedFeatureExtractor] + Wav2Vec2FeatureExtractor, Wav2Vec2Processor, Wav2Vec2Tokenizer 10324 (patrickvonplaten)
- [Wav2Vec2 Example Script] Typo 10547 (patrickvonplaten)
- [Wav2Vec2] Make wav2vec2 test deterministic 10714 (patrickvonplaten)
- [Wav2Vec2] Fix documentation inaccuracy 10694 (MikeG112)
AMP & XLA Support for TensorFlow models
Most of the TensorFlow models are now compatible with automatic mixed precision and have XLA support.
- Add AMP for TF Albert 10141 (jplu)
- Unlock XLA test for TF ConvBert 10207 (jplu)
- Making TF BART-like models XLA and AMP compliant 10191 (jplu)
- Making TF XLM-like models XLA and AMP compliant 10211 (jplu)
- Make TF CTRL compliant with XLA and AMP 10209 (jplu)
- Making TF GPT2 compliant with XLA and AMP 10230 (jplu)
- Making TF Funnel compliant with AMP 10216 (jplu)
- Making TF Lxmert model compliant with AMP 10257 (jplu)
- Making TF MobileBert model compliant with AMP 10259 (jplu)
- Making TF MPNet model compliant with XLA 10260 (jplu)
- Making TF T5 model compliant with AMP and XLA 10262 (jplu)
- Making TF TransfoXL model compliant with AMP 10264 (jplu)
- Making TF OpenAI GPT model compliant with AMP and XLA 10261 (jplu)
- Rework the AMP for TF XLNet 10274 (jplu)
- Making TF Longformer-like models compliant with AMP 10233 (jplu)
SageMaker Trainer for model parallelism
We are rolling out experimental support for model parallelism on SageMaker with a new `SageMakerTrainer` that can be used in place of the regular `Trainer`. This is a temporary class that will be removed in a future version, the end goal is to have `Trainer` support this feature out of the box.
- Add SageMakerTrainer for model paralellism 10122 (sgugger)
- Extend trainer logging for sm 10633 (philschmid)
- Sagemaker Model Parallel tensoboard writing fix 10403 (mansimane)
- Multiple fixes in SageMakerTrainer 10687 (sgugger)
- Add DistributedSamplerWithLoop 10746 (sgugger)
General improvements and bugfixes
- [trainer] deepspeed bug fixes and tests 10039 (stas00)
- Removing run_pl_glue.py from text classification docs, include run_xnli.py & run_tf_text_classification.py 10066 (cbjuan)
- remove token_type_ids from TokenizerBertGeneration output 10070 (sadakmed)
- [deepspeed tests] transition to new tests dir 10080 (stas00)
- Added integration tests for Pytorch implementation of the ELECTRA model 10073 (spatil6)
- Fix naming in TF MobileBERT 10095 (jplu)
- [examples/s2s] add test set predictions 10085 (patil-suraj)
- Logging propagation 10092 (LysandreJik)
- Fix some edge cases in report_to and add deprecation warnings 10100 (sgugger)
- Add head_mask and decoder_head_mask to TF LED 9988 (stancld)
- Replace strided slice with tf.expand_dims 10078 (jplu)
- Fix Faiss Import 10103 (patrickvonplaten)
- [RAG] fix generate 10094 (patil-suraj)
- Fix TFConvBertModelIntegrationTest::test_inference_masked_lm Test 10104 (abhishekkrthakur)
- doc: update W&B related doc 10086 (borisdayma)
- Remove speed metrics from default compute objective [WIP] 10107 (shiva-z)
- Fix tokenizers training in notebooks 10110 (n1t0)
- [DeepSpeed docs] new information 9610 (stas00)
- [CI] build docs faster 10115 (stas00)
- [scheduled github CI] add deepspeed fairscale deps 10116 (stas00)
- Line endings should be LF across repo and not CRLF 10119 (LysandreJik)
- Fix TF LED/Longformer attentions computation 10007 (jplu)
- remove adjust_logits_during_generation method 10087 (patil-suraj)
- [DeepSpeed] restore memory for evaluation 10114 (stas00)
- Update run_xnli.py to use Datasets library 9829 (Qbiwan)
- Add new community notebook - Blenderbot 10126 (lordtt13)
- [DeepSpeed in notebooks] Jupyter + Colab 10130 (stas00)
- [examples/run_s2s] remove task_specific_params and update rouge computation 10133 (patil-suraj)
- Fix typo in GPT2DoubleHeadsModel docs 10148 (M-Salti)
- [hf_api] delete deprecated methods and tests 10159 (julien-c)
- Revert propagation 10171 (LysandreJik)
- Conversion from slow to fast for BPE spm vocabs contained an error. 10120 (Narsil)
- Fix typo in comments 10157 (mrm8488)
- Fix typo in comment 10156 (mrm8488)
- [Doc] Fix version control in internal pages 10124 (sgugger)
- [t5 tokenizer] add info logs 9897 (stas00)
- Fix v2 model loading issue 10129 (BigBird01)
- Fix datasets set_format 10178 (sgugger)
- Fixing NER pipeline for list inputs. 10184 (Narsil)
- Add new model to labels that should not stale 10187 (LysandreJik)
- Check TF ops for ONNX compliance 10025 (jplu)
- [RAG] fix tokenizer 10167 (patil-suraj)
- Fix TF template 10189 (jplu)
- fix run_seq2seq.py; porting trainer tests to it 10162 (stas00)
- Specify dataset dtype 10195 (LysandreJik)
- [CI] make the examples sub-group of tests run always 10196 (stas00)
- [WIP][examples/seq2seq] move old s2s scripts to legacy 10136 (patil-suraj)
- set tgt_lang of MBart Tokenizer for summarization 10205 (HeroadZ)
- Store FLOS as floats to avoid overflow. 10213 (sgugger)
- Fix add_token_positions in custom datasets tutorial 10217 (joeddav)
- [trainer] fix ignored columns logger 10219 (stas00)
- Factor out methods 10215 (LysandreJik)
- Fix head masking for TFT5 models 9877 (stancld)
- [CI] 2 fixes 10248 (stas00)
- [trainer] refactor place_model_on_device logic, add deepspeed 10243 (stas00)
- [Trainer] doc update 10241 (stas00)
- Reduce the time spent for the TF slow tests 10152 (jplu)
- Introduce warmup_ratio training argument 10229 (tanmay17061)
- [Trainer] memory tracker metrics 10225 (stas00)
- Script for distilling zero-shot classifier to more efficient student 10244 (joeddav)
- [test] fix func signature 10271 (stas00)
- [trainer] implement support for full fp16 in evaluation/predict 10268 (stas00)
- [ISSUES.md] propose using google colab to reproduce problems 10270 (stas00)
- Introduce logging_strategy training argument 10267 (tanmay17061)
- [CI] Kill any run-away pytest processes 10281 (stas00)
- Patch zero shot distillation script cuda issue 10284 (joeddav)
- Move the TF NER example 10276 (jplu)
- Fix example links in the task summary 10291 (sgugger)
- fixes 10303 10304 (cronoik)
- [ci] don't fail when there are no zombies 10308 (stas00)
- fix typo in conversion script 10316 (tagucci)
- Add note to resize token embeddings matrix when adding new tokens to voc 10331 (LysandreJik)
- Deprecate prepare_seq2seq_batch 10287 (sgugger)
- [examples/seq2seq] defensive programming + expand/correct README 10295 (stas00)
- [Trainer] implement gradient_accumulation_steps support in DeepSpeed integration 10310 (stas00)
- Loading from last checkpoint functionality in Trainer.train 10334 (tanmay17061)
- [trainer] add Trainer methods for metrics logging and saving 10266 (stas00)
- Fix evaluation with label smoothing in Trainer 10338 (sgugger)
- Fix broken examples/seq2seq/README.md markdown 10344 (Wikidepia)
- [bert-base-german-cased] use model repo, not external bucket 10353 (julien-c)
- [Trainer/Deepspeed] handle get_last_lr() before first step() 10362 (stas00)
- ConvBERT fix torch <> tf weights conversion 10314 (abhishekkrthakur)
- fix deprecated reference `tokenizer.max_len` in glue.py 10220 (poedator)
- [trainer] move secondary methods into a separate file 10363 (stas00)
- Run GA on every push even on forks 10383 (LysandreJik)
- GA: only run model templates once 10388 (LysandreJik)
- Bugfix: Removal of padding_idx in BartLearnedPositionalEmbedding 10200 (mingruimingrui)
- Remove unused variable in example for Q&A 10392 (abhishekkrthakur)
- Ignore unexpected weights from PT conversion 10397 (LysandreJik)
- Add support for ZeRO-2/3 and ZeRO-offload in fairscale 10354 (sgugger)
- Fix None in add_token_positions - issue 10210 10374 (andreabac3)
- Make Barthez tokenizer tests a bit faster 10399 (sgugger)
- Fix run_glue evaluation when model has a label correspondence 10401 (sgugger)
- [ci, flax] non-existing models are unlikely to pass tests 10409 (julien-c)
- [LED] Correct Docs 10419 (patrickvonplaten)
- Add Ray Tune hyperparameter search integration test 10414 (krfricke)
- Ray Tune Integration Bug Fixes 10406 (amogkam)
- [examples] better model example 10427 (stas00)
- Fix conda-build 10431 (LysandreJik)
- [run_seq2seq.py] restore functionality: saving to test_generations.txt 10428 (stas00)
- updated logging and saving metrics 10436 (bhadreshpsavani)
- Introduce save_strategy training argument 10286 (tanmay17061)
- Adds terms to Glossary 10443 (darigovresearch)
- Fixes compatibility bug when using grouped beam search and constrained decoding together 10475 (mnschmit)
- Generate can return cross-attention weights too 10493 (Mehrad0711)
- Fix typos 10489 (WybeKoper)
- [T5] Fix speed degradation bug t5 10496 (patrickvonplaten)
- feat(docs): navigate with left/right arrow keys 10481 (ydcjeff)
- Refactor checkpoint name in BERT and MobileBERT 10424 (sgugger)
- remap MODEL_FOR_QUESTION_ANSWERING_MAPPING classes to names auto-generated file 10487 (stas00)
- Fix the bug in constructing the all_hidden_states of DeBERTa v2 10466 (felixgwu)
- Smp grad accum 10488 (sgugger)
- Remove unsupported methods from ModelOutput doc 10505 (sgugger)
- Not always consider a local model a checkpoint in run_glue 10517 (sgugger)
- Removes overwrites for output_dir 10521 (philschmid)
- Rework TPU checkpointing in Trainer 10504 (sgugger)
- [ProphetNet] Bart-like Refactor 10501 (patrickvonplaten)
- Fix example of custom Trainer to reflect signature of compute_loss 10537 (lewtun)
- Fixing conversation test for torch 1.8 10545 (Narsil)
- Fix torch 1.8.0 segmentation fault 10546 (LysandreJik)
- Fixed dead link in Trainer documentation 10554 (jwa018)
- Typo correction. 10531 (cliang1453)
- Fix embeddings for PyTorch 1.8 10549 (sgugger)
- Stale Bot 10509 (LysandreJik)
- Refactoring checkpoint names for multiple models 10527 (danielpatrickhug)
- offline mode for firewalled envs 10407 (stas00)
- fix tf doc bug 10570 (Sniper970119)
- [run_seq2seq] fix nltk lookup 10585 (stas00)
- Fix typo in docstring for pipeline 10591 (silvershine157)
- wrong model used for BART Summarization example 10582 (orena1)
- [M2M100] fix positional embeddings 10590 (patil-suraj)
- Enable torch 1.8.0 on GPU CI 10593 (LysandreJik)
- tokenization_marian.py: use current_spm for decoding 10357 (Mehrad0711)
- [trainer] fix double wrapping + test 10583 (stas00)
- Fix version control with anchors 10595 (sgugger)
- offline mode for firewalled envs (part 2) 10569 (stas00)
- [examples tests] various fixes 10584 (stas00)
- Added max_sample_ arguments 10551 (bhadreshpsavani)
- [examples tests on multigpu] resolving require_torch_non_multi_gpu_but_fix_me 10561 (stas00)
- Check layer types for Optimizer construction 10598 (sgugger)
- Speedup tf tests 10601 (LysandreJik)
- [docs] How to solve "Title level inconsistent" sphinx error 10600 (stas00)
- [FeatureExtractorSavingUtils] Refactor PretrainedFeatureExtractor 10594 (patrickvonplaten)
- fix flaky m2m100 test 10604 (patil-suraj)
- [examples template] added max_sample args and metrics changes 10602 (bhadreshpsavani)
- Fairscale FSDP fix model save 10596 (sgugger)
- Fix tests of TrainerCallback 10615 (sgugger)
- Fixes an issue in `text-classification` where MNLI eval/test datasets are not being preprocessed. 10621 (allenwang28)
- [M2M100] remove final_logits_bias 10606 (patil-suraj)
- Add new GLUE example with no Trainer. 10555 (sgugger)
- Copy tokenizer files in each of their repo 10624 (sgugger)
- Document Trainer limitation on custom models 10635 (sgugger)
- Fix Longformer tokenizer filename 10653 (LysandreJik)
- Update README.md 10647 (Arvid-pku)
- Ensure metric results are JSON-serializable 10632 (sgugger)
- S2S + M2M100 should be available in tokenization_auto 10657 (LysandreJik)
- Remove special treatment for custom vocab files 10637 (sgugger)
- [S2T] fix example in docs 10667 (patil-suraj)
- W2v2 test require torch 10665 (LysandreJik)
- Fix Marian/TFMarian tokenization tests 10661 (LysandreJik)
- Fixes Pegasus tokenization tests 10671 (LysandreJik)
- Onnx fix test 10663 (mfuntowicz)
- Fix integration slow tests 10670 (sgugger)
- Specify minimum version for sacrebleu 10662 (LysandreJik)
- Add DeBERTa to MODEL_FOR_PRETRAINING_MAPPING 10668 (jeswan)
- Fix broken link 10656 (WybeKoper)
- fix typing error for HfArgumentParser for Optional[bool] 10672 (bfineran)
- MT5 integration test: adjust loss difference 10669 (LysandreJik)
- Adding new parameter to `generate`: `max_time`. 9846 (Narsil)
- TensorFlow tests: having from_pt set to True requires torch to be installed. 10664 (LysandreJik)
- Add auto_wrap option in fairscale integration 10673 (sgugger)
- fix: 10628 expanduser path in TrainingArguments 10660 (PaulLerner)
- Pass encoder outputs into GenerationMixin 10599 (ymfa)
- [wip] [deepspeed] AdamW is now supported by default 9624 (stas00)
- [Tests] RAG 10679 (patrickvonplaten)
- enable loading Mbart50Tokenizer with AutoTokenizer 10690 (patil-suraj)
- Wrong link to super class 10709 (cronoik)
- Distributed barrier before loading model 10685 (sgugger)
- GPT2DoubleHeadsModel made parallelizable 10658 (ishalyminov)
- split seq2seq script into summarization & translation 10611 (theo-m)
- Adding required flags to non-default arguments in hf_argparser 10688 (Craigacp)
- Fix backward compatibility with EvaluationStrategy 10718 (sgugger)
- Tests run on Docker 10681 (LysandreJik)
- Rename zero-shot pipeline multi_class argument 10727 (joeddav)
- Add minimum version check in examples 10724 (sgugger)
- independent training / eval with local files 10710 (riklopfer)
- Flax testing should not run the full torch test suite 10725 (patrickvonplaten)