* General NAS programming interface
* Add `enas-mode` and `oneshot-mode` for NAS interface: [PR 1201](https://github.com/microsoft/nni/pull/1201issue-291094510)
* [Gaussian Process Tuner with Matern kernel](https://github.com/microsoft/nni/blob/master/docs/en_US/Tuner/GPTuner.md)
* Multiphase experiment supports
* Added new training service support for multiphase experiment: PAI mode supports multiphase experiment since v0.9.
* Added multiphase capability for the following builtin tuners:
* TPE, Random Search, Anneal, Naïve Evolution, SMAC, Network Morphism, Metis Tuner.
For details, please refer to [Write a tuner that leverages multi-phase](https://github.com/microsoft/nni/blob/master/docs/en_US/AdvancedFeature/MultiPhase.mdwrite-a-tuner-that-leverages-multi-phase)
* Web Portal
* Enable trial comparation in Web Portal. For details, refer to [View trials status](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/WebUI.mdview-trials-status)
* Allow users to adjust rendering interval of Web Portal. For details, refer to [View Summary Page](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/WebUI.mdview-summary-page)
* show intermediate results more friendly. For details, refer to [View trials status](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/WebUI.mdview-trials-status)
* [Commandline Interface](https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/Nnictl.md)
* `nnictl experiment delete`: delete one or all experiments, it includes log, result, environment information and cache. It uses to delete useless experiment result, or save disk space.
* `nnictl platform clean`: It uses to clean up disk on a target platform. The provided YAML file includes the information of target platform, and it follows the same schema as the NNI configuration file.
Bug fix and other changes
* Tuner Installation Improvements: add [sklearn](https://scikit-learn.org/stable/) to nni dependencies.
* (Bug Fix) Failed to connect to PAI http code - [Issue 1076](https://github.com/microsoft/nni/issues/1076)
* (Bug Fix) Validate file name for PAI platform - [Issue 1164](https://github.com/microsoft/nni/issues/1164)
* (Bug Fix) Update GMM evaluation in Metis Tuner
* (Bug Fix) Negative time number rendering in Web Portal - [Issue 1182](https://github.com/microsoft/nni/issues/1182), [Issue 1185](https://github.com/microsoft/nni/issues/1185)
* (Bug Fix) Hyper-parameter not shown correctly in WebUI when there is only one hyper parameter - [Issue 1192](https://github.com/microsoft/nni/issues/1192)
* Support NNI on Windows for PAI/Remote mode
* NNI running on windows for remote mode
* NNI running on windows for PAI mode
* [Advanced features for using GPU](https://github.com/microsoft/nni/blob/v0.8/docs/en_US/ExperimentConfig.mdconfiguration-spec)
* Run multiple trial jobs on the same GPU for local and remote mode
* Run trial jobs on the GPU running non-NNI jobs
* Kubeflow v1beta2 operator
* Support Kubeflow TFJob/PyTorchJob v1beta2
* [General NAS programming interface](https://github.com/microsoft/nni/blob/v0.8/docs/en_US/GeneralNasInterfaces.md)
* Provide NAS programming interface for users to easily express their neural architecture search space through NNI annotation
* Provide a new command `nnictl trial codegen` for debugging the NAS code
* Tutorial of NAS programming interface, example of NAS on mnist, customized random tuner for NAS
* Support resume tuner/advisor's state for experiment resume
* For experiment resume, tuner/advisor will be resumed by replaying finished trial data
* Web Portal
* Improve the design of copying trial's parameters
* Support 'randint' type in hyper-parameter graph
* Use should ComponentUpdate to avoid unnecessary render
Bug fix and other changes
* Bug fix that `nnictl update` has inconsistent command styles
* Support import data for SMAC tuner
* Bug fix that experiment state transition from ERROR back to RUNNING
* Fix bug of table entries
* Nested search space refinement
* Refine 'randint' type and support lower bound
* [Comparison of different hyper-parameter tuning algorithm](https://github.com/microsoft/nni/blob/v0.8/docs/en_US/CommunitySharings/HpoComparision.md)
* [Comparison of NAS algorithm](https://github.com/microsoft/nni/blob/v0.8/docs/en_US/CommunitySharings/NasComparision.md)
* [NNI practice on Recommenders](https://github.com/microsoft/nni/blob/v0.8/docs/en_US/CommunitySharings/NniPracticeSharing/RecommendersSvd.md)
* [Support NNI on Windows](./docs/en_US/WindowsLocalMode.md)
* NNI running on windows for local mode
* [New advisor: BOHB](./docs/en_US/bohbAdvisor.md)
* Support a new advisor BOHB, which is a robust and efficient hyperparameter tuning algorithm, combines the advantages of Bayesian optimization and Hyperband
* [Support import and export experiment data through nnictl](./docs/en_US/NNICTLDOC.mdexperiment)
* Generate analysis results report after the experiment execution
* Support import data to tuner and advisor for tuning
* [Designated gpu devices for NNI trial jobs](./docs/en_US/ExperimentConfig.mdlocalConfig)
* Specify GPU devices for NNI trial jobs by gpuIndices configuration, if gpuIndices is set in experiment configuration file, only the specified GPU devices are used for NNI trial jobs.
* Web Portal enhancement
* Decimal format of metrics other than default on the Web UI
* Hints in WebUI about Multi-phase
* Enable copy/paste for hyperparameters as python dict
* Enable early stopped trials data for tuners.
* NNICTL provide better error message
* nnictl provide more meaningful error message for yaml file format error
* Unable to kill all python threads after nnictl stop in async dispatcher mode
* nnictl --version does not work with make dev-instal
* All trail jobs status stays on 'waiting' for long time on PAI platform
* [Version checking](https://github.com/microsoft/nni/blob/master/docs/en_US/TrainingService/PaiMode.mdversion-check)
* check whether the version is consistent between nniManager and trialKeeper
* [Report final metrics for early stop job](https://github.com/Microsoft/nni/issues/776)
* If includeIntermediateResults is true, the last intermediate result of the trial that is early stopped by assessor is sent to tuner as final result. The default value of includeIntermediateResults is false.
* [Separate Tuner/Assessor](https://github.com/Microsoft/nni/issues/841)
* Adds two pipes to separate message receiving channels for tuner and assessor.
* Make log collection feature configurable
* Add intermediate result graph for all trials
* [Add shmMB config key for PAI](https://github.com/Microsoft/nni/issues/842)
* Fix the bug that doesn't show any result if metrics is dict
* Fix the number calculation issue for float types in hyperband
* Fix a bug in the search space conversion in SMAC tuner
* Fix the WebUI issue when parsing experiment.json with illegal format
* Fix cold start issue in Metis Tuner
* Add release note.
* Fix Metis tuner cold start issue.
* Curve fitting assessor performance improvement.
* Chinese version document: https://nni.readthedocs.io/zh/latest/
* Debuggability/serviceability document: https://nni.readthedocs.io/en/latest/Tutorial/HowToDebug.html
* Tuner assessor reference: https://nni.readthedocs.io/en/latest/sdk_reference.htmltuner
Bug Fixes and Other Changes
* Fix a race condition bug that does not store trial job cancel status correctly.
* Fix search space parsing error when using SMAC tuner.
* Fix cifar10 example broken pipe issue.
* Add unit test cases for nnimanager and local training service.
* Add integration test azure pipelines for remote machine, PAI and kubeflow training services.
* Support Pylon in PAI webhdfs client.
* Making [log directory](https://github.com/Microsoft/nni/blob/v0.5.1/docs/ExperimentConfig.md) configurable
* Support [different levels of logs](https://github.com/Microsoft/nni/blob/v0.5.1/docs/ExperimentConfig.md), making it easier for debugging
* Reorganized documentation & New Homepage Released: https://nni.readthedocs.io/en/latest/
* Chinese users are able to learn NNI with the translated Chinese doc:
_Dear Contributors_: _We'd love to provide more language translations, contribute to NNI with more languages_ =)
Bug Fixes and Other Changes
* Fix the bug of installation in python virtualenv, and refactor the installation logic
* Fix the bug of HDFS access failure on PAI mode after PAI is upgraded.
* Fix the bug that sometimes in-place flushed stdout makes experiment crash
New tuner and assessor supports
* Support [Metis tuner](docs/HowToChooseTuner.mdMetisTuner) as a new NNI tuner. Metis algorithm has been proofed to be well performed for **online** hyper-parameter tuning.
* Support [ENAS customized tuner](https://github.com/countif/enas_nni), a tuner contributed by github community user, is an algorithm for neural network search, it could learn neural network architecture via reinforcement learning and serve a better performance than NAS.
* Support [Curve fitting assessor](docs/HowToChooseTuner.mdCurvefitting) for early stop policy using learning curve extrapolation.
* Advanced Support of [Weight Sharing](docs/AdvancedNAS.md): Enable weight sharing for NAS tuners, currently through NFS.
Training Service Enhancement
* [FrameworkController Training service](docs/FrameworkControllerMode.md): Support run experiments using frameworkcontroller on kubernetes
* FrameworkController is a Controller on kubernetes that is general enough to run (distributed) jobs with various machine learning frameworks, such as tensorflow, pytorch, MXNet.
* NNI provides unified and simple specification for job definition.
* MNIST example for how to use FrameworkController.
User Experience improvements
* A better trial logging support for NNI experiments in PAI, Kubeflow and FrameworkController mode:
* An improved logging architecture to send stdout/stderr of trials to NNI manager via Http post. NNI manager will store trial's stdout/stderr messages in local log file.
* Show the link for trial log file on WebUI.
* Support to show final result's all key-value pairs.
New tuner support
* Support [network morphism](https://github.com/Microsoft/nni/tree/master/src/sdk/pynni/nni/networkmorphism_tuner)
Training Service Improvements
* Migrate [Kubeflow training service](https://github.com/microsoft/nni/blob/master/docs/en_US/TrainingService/KubeflowMode.md)'s dependency from kubectl CLI to [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/) client
* [Pytorch-operator](https://github.com/kubeflow/pytorch-operator) support for Kubeflow training service.
* Improvement on local code files uploading to OpenPAI HDFS.
* Enable modify concurrency number during experiment.
* Add feedback link to NNI github 'create issue' page
* Enable customize top 10 trials regarding to metric numbers (largest or smallest)
* Enable download logs for dispatcher & nnimanager
* Enable automatic scaling of axes for metric number
* [FashionMnist](https://github.com/Microsoft/nni/tree/master/examples/trials/network_morphism), work together with network morphism tuner.
* [Distributed MNIST example](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist-distributed-pytorch) written in PyTorch.
* Show version information both in nnictl and WebUI. You can run **nnictl -v** to show your current installed NNI version.
* Bug Fix
• fix the bug that WebUI doesn't show latest trial job status, which is caused by OpenPAI token expiration.
• Update annotation to support displaying real choice in searchspace
New Platform Support
* Support launch NNI experiment on MAC
New Tuner Support
* [Grid search tuner](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/gridsearch_tuner/gridsearch_tuner.py)
* [Hyperband tuner](https://github.com/Microsoft/nni/tree/master/src/sdk/pynni/nni/hyperband_advisor)
New Training Service Support
* [Kubeflow Training service](https://github.com/microsoft/nni/blob/master/docs/en_US/TrainingService/KubeflowMode.md)
* Support tf-operator
* [Distributed trial example](https://github.com/Microsoft/nni/blob/master/examples/trials/mnist-distributed/dist_mnist.py) on Kubeflow
OpenPAI Training Service Improvements
* Support NNI Manager IP configuration(nniManagerIp) in PAI cluster config file, to fix the issue that user’s machine has no eth0 device
* File number in codeDir is capped to 1000 now, to avoid user mistakenly fill root dir for codeDir
* Don’t print useless ‘metrics is empty’ log int PAI job’s stdout. Only print useful message once new metrics are recorded, to reduce confusion when user checks PAI trial’s output for debugging purpose
* Add timestamp at the beginning of each log entry in trial keeper.
* Show trial's hdfsLogPath
WebUI Improvements and updates
* Download experiment parameters
* UI support for hyperband tuner
* Show experiment error message
* Show line numbers in search space and trial profile
* Support search a specific trial by trial number
* Remove tensorboard button
* Asynchronous dispatcher
* Docker file update: added pytorch library
* Refactor 'nnictl stop' process, send SIGTERM to nni manager process, rather than calling stop Rest API
* Updated several examples
* Fix the bug that Medianstop assessor does not work
* Fix tuner path in ga_squad example
* Fix the bug induced by changed nni APIs in mnist_smarparam example
* Fix bug in command `nnictl update ...`
* Support running multiple experiments simultaneously. You can run multiple experiments by specifying a unique port for each experiment:
nnictl create --port 8081 --config <config file path>
You can still run the first experiment without '--port' parameter:
nnictl create --config <config file path>
* A builtin Batch Tuner which iterates all parameter combination, can be used to submit batch trial jobs.
* nni.report_final_result(result) API supports more data types for result parameter, it can be of following types:
* A python dict containing 'default' key, the value of 'default' key should be of type int or float. The dict can contain any other key value pairs.
* Continuous Integration
* Switched to Azure pipelines
* New nni.get_sequence_id() API. Each trial job is allocated a unique sequence number, which can be retrieved by nni.get_sequence_id() API.
* Download experiment result from WebUI
* Add trial examples using sklearn and NNI together
* Support updating max trial number
* Kaggle competition TGS Salt code as an example
* NNI Docker image:
docker pull msranni/nni:latest
* <span style="color:red">API nn.get_parameters() is renamed to nni.get_next_parameter(), this is a broken change, all examples of prior releases can not run on v0.3.2, please clone nni repo to get new examples.</span>
git clone -b v0.3.2 https://github.com/Microsoft/nni.git
[Known Issues in release 0.3.2](https://github.com/Microsoft/nni/labels/nni030knownissues).
* Support for [OpenPAI](https://github.com/Microsoft/pai) (aka pai) Training Service
* Support training services on pai mode. NNI trials will be scheduled to run on OpenPAI cluster
* NNI trial's output (including logs and model file) will be copied to OpenPAI HDFS for further debugging and checking
* Support [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) tuner
* [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO to handle categorical parameters. The SMAC supported by NNI is a wrapper on [SMAC3](https://github.com/automl/SMAC3)
* Support NNI installation on [conda](https://conda.io/docs/index.html) and python virtual environment
* Update ga squad example and related documentation
* WebUI UX small enhancement and bug fix
[Known Issues in release 0.2.0](https://github.com/Microsoft/nni/labels/nni020knownissues).
Initial release of Neural Network Intelligence (NNI).
* Installation and Deployment
* Support pip install and source codes install
* Support training services on local mode(including Multi-GPU mode) as well as multi-machines mode
* Tuners, Accessors and Trial
* Support AutoML algorithms including: hyperopt_tpe, hyperopt_annealing, hyperopt_random, and evolution_tuner
* Support assessor(early stop) algorithms including: medianstop algorithm
* Provide Python API for user defined tuners and accessors
* Provide Python API for user to wrap trial code as NNI deployable codes
* Provide a command line toolkit 'nnictl' for experiments management
* Provide a web UI for viewing experiments details and managing experiments
* Continuous Integration
* Support CI by providing out-of-box integration with [travis-ci](https://github.com/travis-ci) on ubuntu
* Support simple GPU job scheduling
[Known Issues in release 0.1.0](https://github.com/Microsoft/nni/labels/nni010knownissues).