Trains

Features
- Add Hydra support 219
- Add cifar ignite example 237
- Add auto extraction of `tar.gz` files when using `StorageManager` 237
- Add `Task.init()` argument `auto_connect_streams` controlling stdout/stderr/logging capture 181
- Add carriage return flush support using the `sdk.development.worker.console_cr_flush_period` configuration setting 181
- Add `Task.create_function_task()` to allow creating a new task using a function and arguments to be executed remotely 230
- Allow disabling SSL certificates verification using `Task.setup_upload()` argument `verify` or AWS S3 bucket configuration `verify` property 256
- Add `StorageManager.get_files_server()`
- Add `Task.get_project_id()` using project name
- Add `project_name` argument to `Task.set_project()`
- Add `Task.connect()` support for class / instance objects
- Add `Task get_configuration_object() and `Task.set_configuration_object()` for easier automation
- Improve Auto-Scaler - allow extra configurations, key name and security group are now optional, defaults using empty strings
- Use a built-in matplotlib convertor
- Add reporting text as debug sample example

Bug Fixes
- Fix Optuna HPO parameter serializing 254
- Fix connect dictionary `''` cast to `None` 258
- Fix lightgbm binding keyword argument issue 251
- Fix artifact preview if artifact body is remote URI 239
- Fix infinite recursion in `StorageManager` upload 253
- Fix keras reusing model object only if the filename is the same 252
- Fix running remotely with no configuration should not crash but output a warning 243
- matplotlib
- Fix matplotlib 3.3.3 support
- Fix global figure enumeration
- Fix binding without a title reported a single plot (`untitled 00`) instead of increasing the counter
- Fix Python 2.7/3.5 support
- Fix quote issue when reporting debug images
- Fix replace quote safe characters in upload file to include `;=$`
- Fix `at_exit` called from another process should be ignored
- Fix `Task.set_tags()` for completed / published tasks
- Fix `Task.add_tags()` not working when running remotely
- Fix `Task.set_user_properties()` docstring and interface
- Fix preview with JSON (dict) artifacts did not store the artifact
- Fix `Logger.report_text()` on task created using `Task.create()` was not supported
- Fix initialization for torch: only call torch `get_worker_info` if torch was loaded
- Fix flush (wait) on auxiliary task (obtained using` Task.get_task()`) should wait on all upload events
- Fix server was not updated with the defaults from the code when running remotely and configuration section is missing
- Fix connect dict containing `None` default values, blocked the remote execution from passing string instead of None
- Fix `Task.upload_artifact()` argument `delete_after_upload=True` used in conjunction with `wait_for_upload=True` was not supported

Features

- Add LightGBM support
- Add initial Hydra support 219
- Add synchronous support for `Task.upload_artifact()` 231
- Add `sdk.development.store_code_diff_from_remote` (default `false`) to store diff from remote HEAD instead of local HEAD 222
- Add `sdk.development.detect_with_conda_freeze` (default `true`) for full conda freeze (requires trains-agent >= 16.2)
- Add user properties support using `Task.get_user_properties`, `Task.set_user_properties()` and `Task.delete_user_properties()`
- Add `Logger.report_table()` support for table as list of lists
- Add support to split DAG and Table in pipeline DAG plot. Pipeline DAG single nodes are now round circles below the DAG graph.
- Add Pipeline/Optimization can be attached to any Task (not just the current task)
- Add `force_download` flag to `StorageManager.get_local_copy()`
- Add control over the artifact preview using `Task.upload_artifact()` `preview` argument
- Add `Logger.report_matplotlib_figure()` with examples
- Add `Task.set_task_type()`
- AWS auto-scaler
- Add key pair and security groups support
- Add multi-line support for both extra bash script and extra `trains.conf` data
- Update examples

Bug Fixes

- Fix `Task.update_output_model()` wrong argument order 220
- Fix initializing task on argparse parse in remote mode. Do not call `Task.init()` to avoid auto connect, use `Task.get_task()` instead
- Fix detected task cwd outside of repository root folder
- Fix `Task.connect(dict)` to place non-existing entries on the section name instead of General
- Fix `Task.clone()` support for trains-server < 0.16
- Fix `StorageManager` cache extract zipped artifacts. Use modified time instead of access time for cached files
- Fix diff command output was stripped
- Make sure local packages with multi-files are marked as `package`
- Fix `Task.set_base_docker()` should be skipped when running remotely
- Fix ArgParser binding handling of string argument with boolean default value (affects Pytorch Lightning integration)
- When using `detect_with_pip_freeze` make sure that `package file://` lines are replaced with `package==x.y.z` as local file will probably not be available
- Fix git packages to new pip standard `package git+`
- Improve conda package naming `_` and `-` support
- Do not add specific setuptools version to requirements (pip can't install it anyway)
- Fix image URL quoting when uploading from a file path

Features

- Add `Task.set_resource_monitor_iteration_timeout()` to set ResourceMonitor iteration wait duration timeout 208
- Add PyTorch Lightning save/restore model binding 212
- Add `git diff` for repository submodule (requires git 2.14 or above)
- Add `TrainsJob.is_completed()` and `TrainsJob.is_aborted()`
- Add `Task.logger` property
- Add Pipeline Controller automation and example (see ***here***)
- Add improved trace filtering capabilities in `trains.debugging.trace.trace_trains()`
- Add default help per argument (if not provided) in ArgParser binding
- Deprecate `Task.reporter`
- Update PyTorch example
- Remove warning on skipped auto-magic model logging 206
- Support Keras restructuring for Network, Model and Sequential
- Update autokeras requirements according to https://github.com/keras-team/autokerasinstallation

Bug Fixes

- Fix joblib auto logging models failing on compressed streams 203
- Fix sending empty reports 205
- Fix scatter2d sub-sampling and rounding
- Fix plots reporting
* `NaN` representation (matplotlib conversion)
* Limit the number of digits in a plot to reduce plot size (using `sdk.metrics.plot_max_num_digits` configuration value)
- Fix `Task.wait_for_status()` to reload after it ends
- Fix thread wait Ctrl-C interrupt did not exit process
- Improve Windows support for installed packages analysis
- Fix auto model logging using relative path
- Fix Hyper-parameter Optimization example
- Fix `Task.clone()` when working with TrainsServer < 0.16.0
- Fix pandas artifact handling
* Avoid adding `unnamed:0` column
* Return original pandas object
- Fix `TrainsJob` hyper-params overriding order was not guaranteed
- Fix ArgParse auto-connect to support default function type

Features and Bug Fixes

- Fix `typing` dependency for Python<3.5 184
- Fix `git+https` requirements handling, resolve `top_level.txt` package name (`kerastuner` from git was not detected)
- Fix `Task.get_reported_console_output()` for new Trains Server API v2.9
- Fix cache handling for different partitions/drives/devices
- Disable offline mode when running remotely (i.e. executed by Trains Agent)
- Fix artifact upload to only use file stream when not uploading a locally stored file (multipart upload is not supported on stream upload) 189
- Fix double-escaped model design text when connecting OutputModel
- Enhance HyperParameter optimizer

This release is part of the [Trains Server v0.16.1](https://github.com/allegroai/trains-server/releases/tag/0.16.1) release and is included in the pre-packaged docker image and other deployment methods (see *Trains Server and Trains Agent deployment options* under [Deploying Trains Overview](https://allegro.ai/docs/deploying_trains/trains_deploy_overview/)).

Features

- Add continuing of previously executed experiments. Add `Task.init()` argument `continue_last_task` to continue a previously used Task 160
- Allow Task editing/creation from code. `Task.export_task/import_task/update_task()` 128
- Add offline mode. Use `Task.set_offline()` and `Task.import_offline_session()`
- Support setting offline mode via `TRAINS_OFFLINE_MODE=1` environment variable
- Support setting offline API version via `TRAINS_OFFLINE_MODE=2.9` environment variable
- Automatically pickle all objects when uploading as artifacts, `task.upload_artifact()` argument `auto_pickle=True` 153
- Add multiple sections/groups support for Task hyper-parameters using `Task.connect()`
- Add multiple configurations (files) using `Task.connect_configuration`
- Allow enabling OS environment logging using the `sdk.development.log_os_environments` configuration parameter (complements the `TRAINS_LOG_ENVIRONMENT` environment variable)
- Add Optuna support for hyper-parameter optimization controller. `OptimizerOptuna` is now the default optimizer.
- Add initial Keras-Tuner support (keras-team/keras-tuner334)
- Add automatic FastAI logging. It is disabled if tensorboard is loaded (assuming TensorBoradLogger will be used)
- Support Tensorboard text logging (`add_text()`) as debug samples (`.txt` files), instead of as console output
- Allow for more standard confusion matrix reporting. `Logger.report_confusion_matrix()` argument `yaxis_reversed` (flips the confusion matrix if `True`, default `False`) 165
- Add support for Trains Server 0.16.0 (API v2.9 support)
- Allow disabling Trains update message from the log using the `TRAINS_SUPPRESS_UPDATE_MESSAGE` environment variable 157
- Add AWS EC2 Auto-Scaler service wizard and Service
- Improved and updated examples
- Add Keras Tuner CIFAR10 example
- Add FastAI example
- Update PyTorch Jupyter notebook examples 150
- Support global requirements detection using `pip freeze` (set `sdk.development.detect_with_pip_freeze` configuration in `trains.conf`)
- Add `Task.get_projects()` to get all projects in the system, sorted by last update time

Bug Fixes

- Fix UTC to time stamp in comment 152
- Fix and enhance GPU monitoring
- Fix GPU stats on Windows machines 177
- More robust GPU monitoring 170
- Fix filename too long bug https://github.com/allegroai/trains-server/issues/49
- Fix TensorFlow image logging to allow images with no width/height/color metadata 182
- Fix multiprocessing Pool throw exception in pool hangs execution. Call original signal handler and re-flush `stdout`
- Fix `plotly` support for `matplotlib` 3.3
- Add Python 2.7 support for `get_current_thread_id()`
- Update examples requirements
- Fix and improve signal handling
- Fix Tensorboard 2D convolution histogram, improve histogram accuracy on very small histograms
- Fix auto logging multiple argparse calls before `Task.init()`
- Limit experiment Git diff logging to 500Kb. If larger than 500Kb, diff section will contain a warning and entire diff will be uploaded as an artifact named `auxiliary_git_dif`
- Fix requirements detection
- Fix Trains installed from `git+`
- Fix when Trains is not directly imported
- Fix multiple `-e` packages were not detected (only the first one)
- Fix running with Trains in `PYTHONPATH` resulted in double entry of trains
- Fix `Task.set_base_docker()` on main task to do nothing when running remotely