This is the release note of [v3.3.0](https://github.com/optuna/optuna/milestone/57?closed=1).
Highlights
CMA-ES with Learning Rate Adaptation
A new variant of CMA-ES has been added. By setting the `lr_adapt` argument to `True` in `CmaEsSampler`, you can utilize it. For multimodal and/or noisy problems, adapting the learning rate can help avoid getting trapped in local optima. For more details, please refer to 4817. We want to thank nomuramasahir0, one of the authors of LRA-CMA-ES, for his great work and the development of [cmaes](https://github.com/CyberAgentAILab/cmaes) library.
<img width="513" alt="256118903-6796d0c4-3278-4d99-bdb2-00b6fe0fa13b" src="https://github.com/optuna/optuna/assets/5564044/50ed3200-2e02-4b10-8ad1-1f237cb3f3ea">
Hypervolume History Plot for Multiobjective Optimization
In multiobjective optimization, the history of hypervolume is commonly used as an indicator of performance. Optuna now supports this feature in the visualization module. Thanks to y0z for your great work!
Constrained Optimization Support for Visualization Functions
| Plotly | matplotlib |
| --- | --- |
|  | <img width="1056" alt="254270811-e85c3c5e-44e5-4a04-ba8a-f6ea2c53611f (1)" src="https://github.com/optuna/optuna/assets/5564044/c043c79b-a6ad-46bc-92f5-fd54ee61f995"> |
Some samplers support constrained optimization, however, many other features cannot handle it. We are continuously enhancing support for constraints. In this release, `plot_optimization_history` starts to consider constraint violations. Thanks to hrntsm for your great work!
python
import optuna
def objective(trial):
x = trial.suggest_float("x", -15, 30)
y = trial.suggest_float("y", -15, 30)
v0 = 4 * x**2 + 4 * y**2
trial.set_user_attr("constraint", [1000 - v0])
return v0
def constraints_func(trial):
return trial.user_attrs["constraint"]
sampler = optuna.samplers.TPESampler(constraints_func=constraints_func)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=100)
fig = optuna.visualization.plot_optimization_history(study)
fig.show()
Streamlit Integration for Human-in-the-loop Optimization
<img width="1127" alt="streamlit_integration" src="https://github.com/optuna/optuna/assets/5564044/e8ea5d13-c834-4ed3-8c7b-24ab07c37105">
[Optuna Dashboard v0.11.0](https://github.com/optuna/optuna-dashboard/releases/tag/v0.11.0) provides the tight integration with [Streamlit](https://streamlit.io/) framework. By using this feature, you can create your own application for human-in-the-loop optimization. Please check out [the documentation](https://optuna-dashboard.readthedocs.io/en/latest/api.html#streamlit) and [the example](https://github.com/optuna/optuna-dashboard/tree/main/examples/streamlit_plugin) for details.
Breaking Changes
- Move mxnet (https://github.com/optuna/optuna-integration/pull/31)
- Remove mxnet (4790)
- Remove `ordered_dict` argument from `IntersectionSearchSpace` (4846)
New Features
- Add `logei_candidate_func` and make it default when available (4667)
- Support `JournalFileStorage` and `JournalRedisStorage` on CLI (4696)
- Implement hypervolume history plot for matplotlib backend (4748, thanks y0z!)
- Add `cv_results_` to `OptunaSearchCV` (4751, thanks jckkvs!)
- Add `optuna.integration.botorch.qnei_candidates_func` (4753, thanks kstoneriv3!)
- Add hypervolume history plot for `plotly` backend (4757, thanks y0z!)
- Add `FileSystemArtifactStore` (4763)
- Sort params on fetch (4775)
- Add constraints support to `_optimization_history_plot` (4793, thanks hrntsm!)
- Bump up `LightGBM` version to v4.0.0 (4810)
- Add constraints support to `matplotlib._optimization_history_plot` (4816, thanks hrntsm!)
- Introduce CMA-ES with Learning Rate Adaptation (4817)
- Add `upload_artifact` api (4823)
- Add `before_trial` (4825)
- Add `Boto3ArtifactStore` (4840)
- Display best objective value in contour plot for a given param pair, not the value from the most recent trial (4848)
Enhancements
- Speed up `logpdf` in `_truncnorm.py` (4712)
- Speed up `erf` (4713)
- Speed up `get_all_trials` in `InMemoryStorage` (4716)
- Add a warning for a progress bar not being displayed 4679 (4728, thanks rishabsinghh!)
- Make `BruteForceSampler` consider failed trials (4747)
- Use shallow copy in `_get_latest_trial` (4774)
- Speed up `plot_hypervolume_history` (4776)
Bug Fixes
- Solve issue 4557 - error_score (4642, thanks jckkvs!)
- Fix `BruteForceSampler` for pruned trials (4720)
- Fix `plot_slice` bug when some of the choices are numeric (4724)
- Make `LightGBMTuner` reproducible (4795)
Installation
- Bump up python version (https://github.com/optuna/optuna-integration/pull/34)
Documentation
- Remove `jquery-extension` (4691)
- Add FAQ on combinatorial search space (4723)
- Fix docs (4732)
- Add `plot_rank` and `plot_timeline` plots to visualization tutorial (4735)
- Fix typos found in `integration/sklearn.py` (4745)
- Remove `study.n_objectives` from document (4796)
- Add lower version constraint for `sphinx_rtd_theme` (4853)
- Artifact docs (4855)
Examples
- Run DaskML example with Python 3.11 (https://github.com/optuna/optuna-examples/pull/188)
- Show more information in terminator examples (https://github.com/optuna/optuna-examples/pull/192)
- Drop support for Python 3.7 on Haiku (https://github.com/optuna/optuna-examples/pull/198)
- Add `LICENSE` file (https://github.com/optuna/optuna-examples/pull/200)
Tests
- Remove unnecessary `pytestmark` (https://github.com/optuna/optuna-integration/pull/29)
- Add `GridSampler` test for failed trials (4721)
- Follow up PR 4642 by adding a unit test to confirm `OptunaSearchCV` behavior (4758)
- Fix `test_log_gass_mass` with SciPy 1.11.0 (4766)
- Fix Pytorch lightning unit test (4780)
- Remove skopt (4792)
- Rename test directory (4839)
Code Fixes
- Simplify the type annotations in `benchmarks` (4703, thanks caprest!)
- Unify sampling implementation in `TPESampler` (4717)
- Get values after `_get_observation_pairs` (4742)
- Remove unnecessary period (4746)
- Handle deprecated argument `early_stopping_rounds` (4752)
- Separate dominate function from `_fast_non_dominated_sort()` (4759)
- Separate `after_trial` strategy (4760)
- Remove unused attributes in `TPESampler` (4769)
- Remove `pkg_resources` (4770)
- Use trials as argument of `_calculate_weights_below_for_multi_objective` (4773)
- Fix type annotation (4797, thanks taniokay!)
- Follow up separation of after trial strategy (4803)
- Loose coupling nsgaii child generation (4806)
- Remove `_study_id` parameter from `Trial` class (4811, thanks adjeiv!)
- Loose coupling nsgaii elite population selection (4821)
- Fix checks integration (4826)
- Remove `OrderedDict` (4838, thanks taniokay!)
- Fix typo (4842, thanks wouterzwerink!)
- Followup child generation strategy (4856)
- Remove `samplers._search_space.IntersectionSearchSpace` (4857)
- Add experimental decorators to artifacts functionalities (4858)
Continuous Integration
- Output dependency tree (https://github.com/optuna/optuna-integration/pull/9)
- Use OIDC (https://github.com/optuna/optuna-integration/pull/33)
- Drop Python 3.7 support (https://github.com/optuna/optuna-integration/pull/35)
- Enhance speed benchmark for storages (4778)
- Drop Python 3.7 on `tests-integration` (4784)
- Remove unused `type:ignore`s (4787)
- Restrict numpy version < 1.24 (4788)
- Upgrade redis version (4805)
- Add version constraints on LightGBM (4807)
- Follow-up 4807 : Fix windows-tests and mac-tests (4809)
- Support 3.11 integration (4820)
- Support flake8 6.1.0 (4847)
Other
- Bump up version number to 3.3.0dev (https://github.com/optuna/optuna-integration/pull/27)
- Bump up version number to 3.3.0 (https://github.com/optuna/optuna-integration/pull/36)
- Bump up version number to 3.3.0dev (4710)
- Bump the version up to v3.3.0 (4860)
Thanks to All the Contributors!
This release was made possible by the authors and the people who participated in the reviews and discussions.
Alnusjaponica, HideakiImamura, adjeiv, c-bata, caprest, contramundum53, cross32768, eukaryo, gen740, hrntsm, jckkvs, knshnb, kstoneriv3, nomuramasahir0, not522, nzw0301, rishabsinghh, taniokay, toshihikoyanase, wouterzwerink, xadrianzetx, y0z