Lumin

Important changes

Breaking

Additions

Removals

Fixes

Changes

Depreciations

Comments

Important changes

- `EvalMetrics` revised to inherit from `Callback` and be called on validation data after every epoch. User-written `EvalMetrics` willneed to be adjusted to work with the new calling method: adjust `evaluate` method and constructor may need to be adjusted; see existing metrics to see how.

Breaking

- `eval_metrics` argument in `train_models` renamed to `metric_partials` and now takes a list of partial `EvalMetrics`
- User-written `EvalMetrics` will need to be adjusted to work with the new calling method: adjust `evaluate` method and constructor may need to be adjusted; see existing metrics to see how.

Additions

- `OneCycle` now has a `cycle_ends_training` which allows training to continue at the final LR and Momentum. keeping at default of `True` ends the training once the cycle is complete, as usual.
- `to_np` now returns `None` when input tensor is `None`
- `plot_train_history` now plots metric evolution for validation data

Removals

Fixes

- `Model` now creates `cb_savepath` is it didn't already exist
- Bug in `PredHandler` where predictions were kept on device leading to increased memory usage
- Version issue in matplotlib affecting plot positioning

Changes

Depreciations

- V0.8:
- All `EvalMetrics` depreciated with metric system. They have been copied and renamed to Old* for compatibility with the old model training system.
- `OldEvalMetric`: Replaced by `EvalMetric`
- `OldMultiAMS`: Replaced by `MultiAMS`
- `OldAMS`: Replaced by `AMS`
- `OldRegPull`: Replaced by `RegPull`
- `OldRegAsProxyPull`: Replaced by `RegAsProxyPull`
- `OldRocAucScore`: Replaced by `RocAucScore`
- `OldBinaryAccuracy`: Replaced by `BinaryAccuracy`

Comments

Important changes

- Model training and callbacks have significantly changed:
- `Model.fit` now expects to perform the entire training proceedure, rather than just single epochs.
- A lot of the functionality of the old training method `fold_train_ensemble` is now delegated to `Model.fit`.
- A new ensemble training method `train_models` has replaced `fold_train_ensemble`. It provied a similar API, but aims to be more understandable to users.
- `Model.fit` is now 'stateful': a `fit_params` class is created containing all the information and data relevant to training the model and trainig methods change their actions according to `fit_params.state` ('train', 'valid', and 'test')
- Callbacks now have greater potential: They have more action points during the training cycle, where they can affect training behaviour, and they have access to `fit_params`, allowing them to modify more aspects of the training and have indirect access to all other callbacks.
- The "tick" for the training loop is now one epoch, i.e. validation loss is computed after the entire use of the training data (as opposed to after every sub-epoch), cyclic callbacks now work on the scale of epochs, rather than sub-epochs. Due to the data being split into folds, the concept of a sup-epoch still exists, but the APIs are now simplified for the user (previously they were a mixture of sup-epoch and epoch arguments).
- For users who do not wish to transition to the new model behaviour, the existing behaviour can still be achieved by using the `Old*` models and classes. See the depreciations section for the full list.
- Input masks (present if e.g using feature subsampling in `Model`Builder`)
- `BatchYielder` now takes an `input_mask` argument to filter inputs
- `Model` prediction methods no longer take input mask arguments, instead the input mask (if present) is automatically used. If users have already filtered their data, they should manually remove the input mask from the model (i.e. set it to None)
- Callbacks which take arguments related to (sub-)epochs (e.g. cycle length, scale, time to renewal. etc. for `CycleLR`, `OneCycle`, etc. and `SWA`) now take these arguments in terms of epochs. I.e. a OneCycle schedule with 9 training folds, running for 15 epochs would previously require e.g. `lenghts=(45,90)` in order to complete the cycle in 15 epochs (135 subepochs). Now it is specified as simply `lenghts=(5,10)`. Additionally, these arguments must be integers. Floats will be coerced to integers with warning.
- `lr_find` now runds over all training folds, instead of just 1

Breaking

- Heavy renaming of methods and classes due to changes in model trainng and callbacks.

Additions

- `__del__` method to `FowardHook` class
- `BatchYielder`:
- Now takes an `input_mask` argument to filter inputs
- Now takes an argument allowing incomplete batches to be yielded
- Target array can now be None
- `Model`:
- now takes a `bs` argument for `evaluate`
- predictions can now be modified by passing a `PredHandler` callback to `pred_cb`. The default one simply returns the model predicitons, however other actions could be defined by the user, e.g. performing argmax for multiclass classifiers.

Removals

- `Model`:
- Now no longer takes `callbacks` and `mask_inputs` as arguments for `evaluate`
- `evaluate_from_by` removed, just call `evaluate`
- Callbacks no longer take model and plot_settings arguments during initialisation. These should be added by calling the relevant setters. `Model` will call them when relevant.

Fixes

- Potential bug in convolutional models where checking the out size of the head would affect the batchnorm averaging
- Potential bug in `plot_sample_pred` to do with bin ranges
- `ForwardHook` not working with passed hook functions

Changes

- `BinaryLabelSmooth` now only applies smoothing during training and not in validation
- `Ensemble`
- `from_results` and `build_ensemble` now no longer take `location` as an argument. Instead, results should contain the savepath for the models
- `_build_ensemble` is now private
- `Model`:
- `predict_array` and `predict_folds` are now private
- `fit` now expects to perform the entire fitting of the model, rather than just one sup-epoch. Additionally, validation loss is now computed only at the end of the epoch, rather that previously where it was computed after each fold.
- `SWA` `renewal_period` should now be None in order to prevent a second average being tracked (previously was negative)
- Some examples have been renamed, and copies using the old model fitting proceedure and old callbacks are available in `examples/old`
- `lr_find` now runds over all training folds, instead of just 1

Depreciations

- V0.8:
- Many classes and methods depreciated with new model. They have been copied and renamed to Old*.
- `OldAbsModel`: Replaced by `AbsModel`
- `OldModel`: Replaced by `Model`
- `OldAbsCallback`: Replaced by `AbsCallback`
- `OldCallback`: Replaced by `Callback`
- `OldBinaryLabelSmooth`: Replaced by `BinaryLabelSmooth`
- `OldSequentialReweight`: Will not be replaced
- `SequentialReweightClasses`: Will not be replaced
- `OldBootstrapResample`: Replaced by `BootstrapResample`
- `OldParametrisedPrediction`: Replaced by `ParametrisedPrediction`
- `OldGradClip`: Replaced by `GradClip`
- `OldLsuvInitL` Replaced by `LsuvInit`
- `OldAbsCyclicCallback`: Replaced by `AbsCyclicCallback`
- `OldCycleLR`: Replaced by `CycleLR`
- `OldCycleMom`: Replaced by `CycleMom`
- `OldOneCycle`: Replaced by `OneCycle`
- `OldLRFinder`: Replaced by `LRFinder`
- `fold_lr_find`: Replaced by `lr_find`
- `fold_train_ensemble`: Replaced by `train_models`
- `OldMetricLogger`: Replaced by `MetricLogger`
- `AbsModelCallback`: Will not be replaced
- `OldSWA`: Replaced by `SWA`
- `old_plot_train_history`: Replaced by `plot_train_history`
- `OldEnsemble`: Replaced by `Ensemble`

Comments

Important changes

- `auto_filter_on_linear_correlation` now examines **all** features within correlated clusters, rather than just the most correlated pair. This means that the function now only needs to be run once, rather than the previously recommended multiple rerunning.
- Moved to Scikit-learn 0.22.2, and moved, where possible, to keyword argument calls for sklearn methods in preparation for 0.25 enforcement of keyword arguments
- Fixed error in patience when using cyclical LR callbacks, now specify the number of cycles to go without improvement. Previously had to specify 1+number.
- Matrix data is no longer passed through `np.nan_to_num` in `FoldYielder`. Users should ensure that all values in matrix data are not NaN or Inf
- Tensor data:
- `df2foldfile`, `fold2foldfile`, and 'add_meta_data` can now support the saving of arbitrary matrices as a matrix input
- Pass a `numpy.array` whose first dimension matches the length of the DataFrame to the `tensor_data` argument of `df2foldfile` and a name to `tensor_name`.
The array will be split along the first dimension and the sub-arrays will be saved as matrix inputs in the resulting foldfile
- The matrices may also be passed as sparse format and be densified on loading by FoldYielder

Breaking

- `plot_rank_order_dendrogram` now returns sets of all features in cluster with distance over the threshold, rather than just the closest features in each cluster

Additions

- Addition of batch size parameter to `Ensemble.predict*`
- Lorentz Boost Network (https://arxiv.org/abs/1812.09722):
- `LorentzBoostNet` basic implementation which learns boosted particles from existing particles and extracts features from them using fixed kernel functions
- `AutoExtractLorentzBoostNet` which also learns the kernel-functions during training
- Classification `Eval` classes:
- `BinaryAccuracy`: Computes and returns the accuracy of a single-output model for binary classification tasks.
- `RocAucScore`: Computes and returns the area under the Receiver Operator Characteristic curve (ROC AUC) of a classifier model.
- `plot_binary_sample_feat`: a version of `plot_sample_pred` designed for plotting feature histograms with stacked contributions by sample for
background.
- Added compression arguments to `df2foldfile`, `fold2foldfile`, and `save_to_grp`
- Tensor data:
- `df2foldfile`, `fold2foldfile`, and 'add_meta_data` can now support the saving of arbitrary matrices as a matrix input
- Pass a `numpy.array` whose first dimension matches the length of the DataFrame to the `tensor_data` argument of `df2foldfile` and a name to `tensor_name`.
The array will be split along the first dimension and the sub-arrays will be saved as matrix inputs in the resulting foldfile
- The matrices may also be passed as sparse format and be densified on loading by FoldYielder
- `plot_lr_finders` now has a `log_y` argument for logarithmic y-axis. Default `auto` set log_y if maximum fractional difference between losses is greater than 50
- Added new rescaling options to `ClassRegMulti` using linear outputs and scaling by mean and std of targets
- `LsuvInit` now applies scaling to `nn.Conv3d` layers
- `plot_lr_finders` and `fold_lr_find` now have options to save the resulting LR finder plot (currently limited to png due to problems with pdf)
- Addition of AdamW and an optimiser, thanks to [kiryteo](https://github.com/kiryteo)
- Contribution guide, thanks to [kiryteo](https://github.com/kiryteo)
- OneCycle `lr_range` now supports a non-zero final LR; just supply a three-tuple to the `lr_range` argument.
- `Ensemble.from_models` classmethod for combining in-memory models into an Ensemble.

Removals

- `FeatureSubsample`
- `plots` keyword in `fold_train_ensemble`

Fixes

- Docs bug for nn.training due to missing ipython in requirements
- Bug in LSUV init when running on CUDA
- Bug in TF export based on searching for fullstops
- Bug in model_bar update during fold training
- Quiet bug in 'MultHead' when matrix feats were not listed first; map construction indexed self.matrix_feats not self.feats
- Slowdown in `ensemble.predict_array` which caused the array to get sent to device in during each model evaluations
-`Model.get_param_count` now includes mon-trainable params when requested
- Fixed bug in `fold_lr_find` where LR finders would use different LR steps leading to NaNs when plotting in `fold_lr_find`
- `plot_feat` used to coerce NaNs and Infs via `np.nan_to_num` prior to plotting, potentially impacting distributions, plotting scales, moments, etc. Fixed so that nan and inf values are removed rather than coerced.
- Fixed early-stopping statement in `fold_train_ensemble` to state the number as "sub-epochs" (previously said "epochs")
- Fixed error in patience when using cyclical LR callbacks, now specify the number of cycles to go without improvement. Previously had to specify 1+number.
- Unnecessary warning `df2foldfile` when no strat-key is passed.
- Saved matrices in `fold2foldfile` are now in float32
- Fixed return type of `get_layers` methods in `RNNs_CNNs_and_GNNs_for_matrix_data` example
- Bug in `model.predict_array` when predicting matrix data with a batch size
- Added missing indexing in `AbsMatrixHead` to use `torch.bool` if PyTorch version is >= 1.2 (was `uint8` but now depreciated for indexing)
- Errors when running in terminal due to trying to call `.show` on fastprogress bars
- Bug due to encoding of readme when trying to install when default encoder is ascii
- Bug when running `Model.predict` in batches when the data contains less than one batch
- Include missing files in sdist, thanks to [thatch](https://github.com/thatch)
- Test path correction in example notebook, thanks to [kiryteo](https://github.com/kiryteo)
- Doc links in `hep_proc`
- Error in `MultiHead._set_feats` when `matrix_head` does not contain 'vecs' or 'feats_per_vec' keywords
- Compatibility error in numpy >= 1.18 in `bin_binary_class_pred` due to float instead of int
- Unnecessary second loading of fold data in `fold_lr_find`
- Compatibility error when working in PyTorch 1.6 based on integer and true division
- SWA not evaluating in batches when running in non-bulk-move mode
- Moved from `normed` to `density` keywords for matplotlib

Changes

- `ParametrisedPrediction` now accepts lists of parameterisation features
- `plot_sample_pred` now ensures that signal and background have the same binning
- `PlotSettings` now coerces string arguments for `savepath` to `Path`
- Added default value for `targ_name` in `EvalMetric`
- `plot_rank_order_dendrogram`:
- Now uses "optimal ordering" for improved presentation
- Now returns sets of all features in cluster with distance over the threshold, rather than just the closest features in each cluster
- `auto_filter_on_linear_correlation` now examines **all** features within correlated clusters, rather than just the most correlated pair. This means that the function now only needs to be run once, rather than the previously recommended multiple rerunning.
- Improved data shuffling in `BatchYielder`, now runs much quicker
- Slight speedup when loading data from foldfiles
- Matrix data is no longer passed through `np.nan_to_num` in `FoldYielder`. Users should ensure that all values in matrix data are not NaN or Inf

Depreciations

Comments

- RFPImp still imports from `sklearn.ensemble.forest` which is depreciated, and possibly part of the private API. Hopefully the package will remedy this in time for depreciation. For now, future warnings are displayed.

Important changes

- New live plot for losses during training (`MetricLogger`):
- Provides additional information
- Only updates after every epoch (previously every subepoch) reducing training times
- Nicer appearance and automatic log scale for y-axis

Breaking

Additions

- New live plot for losses during training (`MetricLogger`):
- Provides additional information
- Only updates after every epoch (previously every subepoch) reducing training times
- Nicer appearance and automatic log scale for y-axis

Removals

Fixes

- Fixed error in documentation which removed the ToC for the nn module

Changes

Depreciations

- `plots` argument in `fold_train_ensemble`. The plots argument is now depreciated and ignored. Loss history will always be shown, lr history will no longer be shown separately, and live feedback is now controlled by the four live_fdbk arguments. This argument will be removed in V0.6.

Comments

Important changes

- Added support for processing and embedding of matrix data
- `MultiHead` to allow the use of multiple head blocks to handle input data containing flat and matrix inputs
- `AbsMatrixHead` abstract class for head blocks designed to process matrix data
- `InteractionNet` a new head block to apply interaction graph-nets to objects in matrix form
- `RecurrentHead` a new head block to apply recurrent layers (RNN, LSTM, GRU) to series objects in matrix form
- `AbsConv1dHead` a new abstract class for building convolutional networks from basic blocks to apply to object in matrix form.
- Meta data:
- `FoldYielder` now checks its foldfile for a `meta_data` group which contains information about the features and inputs in the data
- `cont_feats` and `cat_feats` now no longer need to be passed to `FoldYielder` during initialisation of the foldfile contains meta data
- `add_meta_data` function added to write meta data to foldfiles and is automatically called by `df2foldfile`
- Improved usage with large datasets:
- Added`Model.evaluate_from_by` to allow batch-wise evaluation of loss
- `bulk_move` in `fold_train_ensemble` now also affects the validation fold, i.e. `bulk_move=False` no longer preloads the validation fold, and validation loss is evaluated using `Model.evaluate_from_by`
- `bulk_move` arguments added to `fold_lr_find`
- Added batch-size argument to Model predict methods to run predictions in batches

Breaking

- `FoldYielder.get_df()` now returns any NaNs present in data rather than zeros unless `nan_to_num` is set to `True`
- Zero bias init for bottlenecks in `MultiBlock` body

Additions

- `__repr__` of `Model` now detail information about input variables
- Added support for processing and embedding of matrix data
- `MultiHead` to allow the use of multiple head blocks to handle input data containing flat and matrix inputs
- `AbsMatrixHead` abstract class for head blocks designed to process matrix data
- `InteractionNet` a new head block to apply interaction graph-nets to objects in matrix form
- `RecurrentHead` a new head block to apply recurrent layers (RNN, LSTM, GRU) to series objects in matrix form
- `AbsConv1dHead` a new abstract class for building convolutional networks from basic blocks to apply to object in matrix form.
- Meta data:
- `FoldYielder` now checks its foldfile for a `meta_data` group which contains information about the features and inputs in the data
- `cont_feats` and `cat_feats` now no longer need to be passed to `FoldYielder` during initialisation of the foldfile contains meta data
- `add_meta_data` function added to write meta data to foldfiles and is automatically called by `df2foldfile`
- `get_inputs` method to `BatchYielder` to return the inputs, optionally on device
- Added LSUV initialisation, implemented by `LsuvInit` callback


Removals

Fixes

- `FoldYielder.get_df()` now returns any NaNs present in data rather than zeros unless `nan_to_num` is set to `True`
- Various typing fixes`
- Body and tail modules not correctly freezing
- Made `Swish` to not be inplace - seemed to cause problems sometimes
- Enforced fastprogress version; latest version renamed a parameter
- Added support to `df2foldfile` for missing `strat_key`
- Added support to `fold2foldfile` for missing features
- Zero bias init for bottlenecks in `MultiBlock` body


Changes

- Slight optimisation in `FullyConnected` when not using dense or residual networks
- `FoldYielder.set_foldfile` is now a private function `FoldYielder._set_foldfile`
- Improved usage with large datasets:
- Added`Model.evaluate_from_by` to allow batch-wise evaluation of loss
- `bulk_move` in `fold_train_ensemble` now also affects the validation fold, i.e. `bulk_move=False` no longer preloads the validation fold, and validation loss is evaluated using `Model.evaluate_from_by`
- `bulk_move` arguments added to `fold_lr_find`
- Added batch-size argument to Model predict methods to run predictions in batches

Depreciations

Comments

Targeting V0.4 Hypothetically Useful But Of Limited Actual Utility

Important changes

- Moved to Pandas 0.25.0
- Moved to Seaborn 0.9.0
- Moved to Scikit-learn 0.21.0

Breaking

Additions

- `rf_check_feat_removal` method to check whether one of several (correlated) features can safely be ignored
- `rf_rank_features`:
- `n_max_display` to `rf_rank_features` to adjust number of features displayed in plot
- `plot_results`, `retrain_on_import_feats`, and `verbose` to control printed outputs of function
- Can now take preset RF params, rather than optimising each time
- Control over x-axis label in `plot_importance`
- `repeated_rf_rank_features`
- `get_df` function to `LRFinder`
- Ability to use dictionaries for `PlotSettings.style`
- `plot_rank_order_dendrogram`:
- added threshold param to control plotting colour and return
- returns list of paris of correlated features
- `FoldYielder`
- Method to list columns in foldfile
- option to initialise using a string or path for the foldfile
- close method to close the foldfile
- New methods to `hep_proc` focussing on vectoriesed transformations and operatins of Lorentz Vectors
- `subsample_df` to sub sample a data frame (with optional stratification and replacement)
- Callbacks during prediction:
- `on_pred_begin` and `on_pred_end` methods added to `AbsCallback` which are called during `Model.predict_array`
- `Model.predict`, `Model.predict_folds`, `Model.predict_array` now take a list of instantiated callbacks to apply during prediciton
- `Ensemble.predict`, `Ensemble.predict_folds`, `Ensemble.predict_array` now take a list of instantiated callbacks to apply during prediciton
- `ParametrisedPrediction` callback for setting a single parameterisation feature to a set value during model prediction
- y-axis limit argument to `plot_1d_partial_dependence`
- `auto_filter_on_linear_correlation`
- `auto_filter_on_mutual_dependence`

Removals

- Passing `eta` argument to `to_pt_eta_phi`: now inferred from data
- `Embedder` renamed to `CatEmbedder`
- `cat_args` and `n_cont_in` arguments in `ModelBuilder`: Use `cat_embedder` and `cont_feats` instead
- `callback_args` argument in `fold_train_ensemble`: Use `callback_partials` instead
- `binary_class_cut` renamed to `binary_class_cut_by_ams`
- `plot_dendrogram` renamed to `plot_rank_order_dendrogram`

Fixes

- Remove mutable default paramert for `get_opt_rf_params`
- Missing `n_estimators` in call to `get_opt_rf_params` to `rf_rank_features`
- Added string interpretation check when loading `ModelBuilder` saved in pre-v0.3.1 versions
- `rf_rank_features` importance cut now >= threshold, was previously >
- `plot_rank_order_dendrogram` now clusters by absolute Spearman's rank correlation coeficient
- `feat_map` to `self.feat_map` in `MultiBlock.__init__`
- Bias initialisation for sigmoids in `ClassRegMulti` corrected to zero, was 0.5
- Removed uncertainties from the moments shown by `plot_feat` when plotting with weights; uncertainties were underestimated

Changes

- Improved `plot_lr_finders`
- Moved to Pandas 0.25.0
- Moved to Seaborn 0.9.0
- Moved to Scikit-learn 0.21.0
- `model_builder.get_model` now returns a 4th object, an input_mask
- Feature subsampling:
- Moved to `ModelBuilder` rather than `FeatureSubsample` callback: required to handle `MultiBlock` models
- Now allows a list of features to always be present in model via `ModelBuilder.guaranteed_feats`
- `plot_1d_partial_dependence` and `plot_2d_partial_dependence` now better handle weighted resampling of data: replacement sampling, and auto fix when `wgt_name` specified but no `sample_sz`

Depreciations

- `FeatureSubsample` in favour of `guaranteed_feats` and `cont_subsample_rate` in `ModelBuilder`. Will be removed in v0.6.

Comments