- 1607 `pip install autogluon.text` will error on import if installed standalone due to missing `autogluon.features` as a dependency. To fix: `pip install autogluon.features`. This will be resolved in v0.4.1 release.
Changes
General
- [AutoGluon now supports Windows OS!](https://auto.gluon.ai/0.4.0/index.html) Both CPU and GPU are supported on Windows.
- AutoGluon now supports Python 3.9. Python 3.6 is no longer supported.
- AutoGluon has migrated from MXNet to PyTorch for all deep learning models resulting in major speedups.
- [AutoGluon v0.4 Cheat Sheet](https://auto.gluon.ai/stable/autogluon-cheat-sheet.pdf): Get started faster than ever before with this handy reference page!
- [New tutorials](https://auto.gluon.ai/0.4.0/tutorials/cloud_fit_deploy/index.html) showcasing cloud training and deployment with AWS SageMaker and Lambda.
Text
AutoGluon-Text is refactored with [PyTorch Lightning](https://www.pytorchlightning.ai/). It now supports backbones in [huggingface/transformers](https://huggingface.co/docs/transformers/index). The new version has better performance, faster training time, and faster inference speed. In addition, AutoGluon-Text now supports solving multilingual problems and a new `AutoMMPredictor` has been implemented for automatically building multimodal DL models.
- **Better Performance**
- Compared with TextPredictor in AutoGluon 0.3, TextPredictor in AutoGluon 0.4 has **72.22%** win-rate in the [multimodal text-tabular benchmark published in NeurIPS 2021](https://arxiv.org/abs/2111.02705). If we use `presets="high_quality"`, the win-rate increased to **77.8%** thanks to the [DeBERTa-v3 backbone](https://arxiv.org/abs/2111.09543).
- In addition, we resubmitted our results to [MachineHack: Product Sentiment Analysis](https://machinehack.com/hackathon/product_sentiment_classification_weekend_hackathon_19/overview
), ["MachineHack: Predict the Price of Books"](https://machinehack.com/hackathon/predict_the_price_of_books/overview
), and ["Kaggle: Mercari Price Suggestion"](https://www.kaggle.com/c/mercari-price-suggestion-challenge). With three lines of code, AutoGluon 0.4 is able to achieve top places in these competitions (1st, 2nd, 2nd correspondingly). The results obtained by AutoGluon 0.4 also consistently outperform the results obtained by AutoGluon 0.3.
- **Faster Speed**
- The new version has **~2.88x** speedup in training and **~1.40x** speedup in inference. With g4dn.12x instance, the model can achieve an additional 2.26x speedup with 4 GPUs.
- **Multilingual Support**
- AutoGluon-Text now supports solving multilingual problems via cross-lingual transfer ([Tutorial](https://auto.gluon.ai/0.4.0/tutorials/text_prediction/multimodal_text.html)). This is triggered by setting `presets="multilingual"`. You can now train a model on the English dataset and directly apply the model on datasets in other languages such as German, Japanese, Italian, etc.
- **AutoMMPredictor for Multimodal Problems**
- Support an experimental AutoMMPredictor that supports fusion image backbones in [timm](https://github.com/rwightman/pytorch-image-models/tree/master/timm), text backbone in [huggingface/transformers](https://huggingface.co/docs/transformers/index), and multimodal backbones like [CLIP](https://openai.com/blog/clip/) ([Tutorial](https://auto.gluon.ai/0.4.0/tutorials/text_prediction/automm.html)). It may perform better than ensembling ImagePredictor + TextPredictor.
- **Other Features**
- Support continuous training from an existing checkpoint. You may just call `.fit()` again after a previous trained model has been loaded.
Thanks to zhiqiangdon and sxjscience for contributing the AutoGluon-Text refactors! (1537, 1547, 1557, 1565, 1571, 1574, 1578, 1579, 1581, 1585, 1586)
Tabular
AutoGluon-Tabular has been majorly enhanced by numerous optimizations in 0.4. In summation, these improvements have led to a:
- **~2x** training speedup in Good, High, and Best quality presets.
- **~1.3x** inference speedup.
- **63%** win-rate vs AutoGluon 0.3.1 (Results from [AutoMLBenchmark](https://github.com/openml/automlbenchmark))
- **93%** win-rate vs AutoGluon 0.3.1 on datasets with >=100,000 rows of data (!!!)
Specific updates:
- Added `infer_limit` and `infer_limit_batch_size` as new fit-time constraints ([Tutorial](https://auto.gluon.ai/0.4.0/tutorials/tabular_prediction/tabular-indepth.html#inference-speed-as-a-fit-constraint)). This allows users to specify
the desired end-to-end inference latency of the final model and AutoGluon will automatically train models
to satisfy the constraint. This is extremely useful for online-inference scenarios where you need to satisfy an
end-to-end latency constraint (for example 50ms). Innixma (1541, 1584)
- Implemented automated semi-supervised and transductive learning in TabularPredictor.
[Try it out](https://auto.gluon.ai/0.4.0/api/autogluon.predictor.html#autogluon.tabular.TabularPredictor.fit_pseudolabel) via `TabularPredictor.fit_pseudolabel(...)`! DolanTheMFWizard (1323, 1382)
- Implemented automated feature pruning (i.e. feature selection) in TabularPredictor.
Try it out via `TabularPredictor.fit(..., feature_prune_kwargs={})`! truebluejason (1274, 1305)
- Implemented automated model calibration to improve AutoGluon's predicted probabilities for classification problems.
This is enabled by default, and can be toggled via the `calibrate` fit argument. DolanTheMFWizard (1336, 1374, 1502)
- Implemented parallel bag training via Ray. This results in a ~2x training speedup when bagging is enabled
compared to v0.3.1 with the same hardware due to more efficient usage of resources
for models that cannot effectively use all cores. yinweisu (1329, 1415, 1417, 1423)
- Added adaptive early stopping logic which greatly improves the quality of models within a time budget. Innixma (1380)
- Added automated model calibration in quantile regression. taesup-aws (1388)
- Enhanced datetime feature handling. willsmithorg (1446)
- Added support for custom confidence levels in feature importance. jwmueller (1328)
- Improved neural network HPO search spaces. jwmueller (1346)
- Optimized one-hot encoding preprocessing. Innixma (1376)
- Refactored `refit_full` logic to majorly simplify user model contributions and improve multimodal support with advanced presets. Innixma (1567)
- Added experimental TabularPredictor config helper. gradientsky (1491)
- New Tutorials
- [GPU training tutorial for tabular models](https://auto.gluon.ai/0.4.0/tutorials/tabular_prediction/tabular-gpu.html). gradientsky (#1527)
- [Feature preprocessing tutorial](https://auto.gluon.ai/0.4.0/tutorials/tabular_prediction/tabular-feature-engineering.html). willsmithorg (#1478)
Tabular Models
NEW: TabularNeuralNetTorchModel (alias: 'NN_TORCH')
As part of the migration from MXNet to Torch, we have created a Torch based counterpart
to the prior MXNet tabular neural network model. This model has several major advantages, such as:
- **1.9x** faster training speed
- **4.7x** faster inference speed
- **51%** win-rate vs MXNet Tabular NN
This model has replaced the MXNet tabular neural network model in the default hyperparameters configuration,
and is enabled by default.
Thanks to jwmueller and Innixma for contributing TabularNeuralNetTorchModel to AutoGluon! (1489)
NEW: VowpalWabbitModel (alias: 'VW')
VowpalWabbit has been added as a new model in AutoGluon. VowpalWabbit is not installed by default, and must be installed separately.
VowpalWabbit is used in the `hyperparameters='multimodal'` preset, and the model is a great option to use for datasets containing text features.
To install VowpalWabbit, specify it via `pip install autogluon.tabular[all, vowpalwabbit]` or `pip install "vowpalwabbit>=8.10,<8.11"`
Thanks to killerSwitch for contributing VowpalWabbitModel to AutoGluon! (1422)
XGBoostModel (alias: 'XGB')
- Optimized model serialization method, which results in 5.5x faster inference speed and halved disk usage. Innixma (1509)
- Adaptive early stopping logic leading to 54.7% win-rate vs prior implementation. Innixma (1380)
- Optimized training speed with expensive metrics such as F1 by ~10x. Innixma (1344)
- Optimized num_cpus default to equal physical cores rather than virtual cores. Innixma (1467)
CatBoostModel (alias: 'CAT')
- CatBoost now incorporates callbacks which make it more stable and resilient to memory errors,
along with more advanced adaptive early stopping logic that leads to 63.2% win-rate vs prior implementation. Innixma (1352, 1380)
LightGBMModel (alias: 'GBM')
- Optimized training speed with expensive metrics such as F1 by ~10x. Innixma (1344)
- Adaptive early stopping logic leading to 51.1% win-rate vs prior implementation. Innixma (1380)
- Optimized num_cpus default to equal physical cores rather than virtual cores. Innixma (1467)
FastAIModel (alias: 'FASTAI')
- Added adaptive batch size selection and epoch selection. gradientsky (1409)
- Enabled HPO support in FastAI (previously HPO was not supported for FastAI). Innixma (1408)
- Made FastAI training deterministic (it is now consistently seeded). Innixma (1419)
- Fixed GPU specification in FastAI to respect the num_gpus parameter. Innixma (1421)
- Forced correct number of threads during fit and inference to avoid issues with global thread updates. yinweisu (1535)
LinearModel (alias: 'LR')
Linear models have been accelerated by **20x** in training and **20x** in inference thanks to a variety of optimizations.
To get the accelerated training speeds, please install [scikit-learn-intelex](https://github.com/intel/scikit-learn-intelex) via `pip install "scikit-learn-intelex>=2021.5,<2021.6"`
Note that currently LinearModel is not enabled by default in AutoGluon,
and must be specified in `hyperparameters` via the key `'LR'`.
Further testing is planned to incorporate LinearModel as a default model in future releases.
Thanks to the `scikit-learn-intelex` team and Innixma for the LinearModel optimizations! (1378)
Vision
- Refactored backend logic to be more robust. yinweisu (1427)
- Added support for inference via CPU. Previously, inferring without GPU would error. yinweisu (1533)
- Refactored HPO logic. Innixma (1511)
Miscellaneous
- AutoGluon no longer depends on ConfigSpace, cython, dill, paramiko, autograd, openml, d8, and graphviz.
This greatly simplifies installation of AutoGluon, particularly on Windows.
- Entirely refactored HPO logic to break dependencies on ConfigSpace and improve stability and ease of development.
HPO has been simplified to use random search in this release while we work on
re-introducing the more advanced HPO methods such as bayesopt in a future release.
Additionally, removed 40,000 lines of out-dated code to streamline future development.
Innixma (1397, 1411, 1414, 1431, 1443, 1511)
- Added `autogluon.common` to simplify dependency management for future submodules. Innixma (1386)
- Removed `autogluon.mxnet` and `autogluon.extra` submodules as part of code cleanup. Innixma (1397, 1411, 1414)
- Refactored logging to avoid interfering with other packages. yinweisu (1403)
- Fixed logging output on Kaggle, previously no logs would be displayed while fitting AutoGluon in a Kaggle kernel. Innixma (1468)
- Added platform tests for Linux, MacOS, and Windows. yinweisu (1464, 1506, 1513)
- Added [ROADMAP.md](https://github.com/awslabs/autogluon/blob/master/ROADMAP.md) to highlight past, present, and future feature prioritization and progress to the community. Innixma (#1420)
- Various documentation and CI improvements
- jwmueller (1379, 1408, 1429)
- gradientsky (1383, 1387, 1471, 1500)
- yinweisu (1441, 1482, 1566, 1580)
- willsmithorg (1476, 1483)
- Xilorole (1526)
- Innixma (1452, 1453, 1528, 1577, 1584, 1588, 1593)
- Various backend enhancements / refactoring / cleanup
- DolanTheMFWizard (1319)
- gradientsky (1320, 1366, 1385, 1448, 1488, 1490, 1570, 1576)
- mseeger (1349)
- yinweisu (1497, 1503, 1512, 1563, 1573)
- willsmithorg (1525, 1543)
- Innixma (1311, 1313, 1327, 1331, 1338, 1345, 1369, 1377, 1380, 1408, 1410, 1412, 1419, 1425, 1428, 1462, 1465, 1562, 1569, 1591, 1593)
- Various bug fixes
- jwmueller (1314, 1356)
- yinweisu (1472, 1499, 1504, 1508, 1516)
- gradientsky (1514)
- Innixma (1304, 1325, 1326, 1337, 1365, 1395, 1405, 1587, 1599)