💡 New features 💡
Dependencies
* RSMTool is now compatible with [SKLL v2.1](https://skll.readthedocs.io/en/latest/)
* All dependencies other than `skll` are now unpinned.
* RSMTool now supports Python versions 3.6, 3.7 and 3.8.
Interactive generation of configuration files
* Configuration files for `rsmtool`, `rsmeval`, `rsmpredict`, `rsmcompare` and `rsmsummarize` can now be generated automatically, either interactively or non-interactively. This exciting new functionality makes it easier to keep track of the many configuration options available in RSMTool and greatly simplifies the process of setting up the experiment. Watch the [video](https://asciinema.org/a/313107) demonstrating the new interactive generation or read the [documentation](https://rsmtool.readthedocs.io/en/stable/automated_configuration.html#autogen-configuration).
Passing hyperparameters to SKLL models
* It is now possible to pass custom hyperparameter values to [`skll` learners](https://skll.readthedocs.io/en/latest/run_experiment.html#learners) used through RSMTool. This is done using a new configuration field [`skll_fixed_parameters`](https://rsmtool.readthedocs.io/en/stable/usage_rsmtool.html#skll-fixed-parameters-optional). The parameters are also displayed in the report.
Generalized version of PRMSE
* The formula for [PRMSE](https://rsmtool.readthedocs.io/en/stable/evaluation.html#proportional-reduction-in-mean-squared-error-for-true-scores-prmse) has been updated to a more general version derived by Matthew S. Johnson that allows computation of PRMSE for any number of raters. For two raters, the formula returns the same result as the formula used in previous versions of the tool.
* The API now provides a new function [`prmse_true()`](https://rsmtool.readthedocs.io/en/stable/api.html#rsmtool.utils.prmse.prmse_true) which accepts scikit-learn style parameters and returns the PRMSE value.
* It is now possible to supply error variance of human raters necessary to compute PRMSE. This can be useful when the experiments require computing this parameter on data other than the evaluation set. This can be done via the [`rater_error_variance`](https://rsmtool.readthedocs.io/en/stable/usage_rsmtool.html#rater-error-variance-optional) field in the configuration file or by passing the variance as a parameter to [`prmse_true()`](https://rsmtool.readthedocs.io/en/stable/api.html#rsmtool.utils.prmse.prmse_true).
Changes to RSMTool reports
* The report now always displays the headers for the "Consistency" and "True score evaluations" sections. If no second score is available, the report will indicate this. If you do not want these section headers to appear in your report, use the [`general_section`](https://rsmtool.readthedocs.io/en/stable/usage_rsmtool.html#general-sections-optional) field to exclude these sections. *TIP*: If you use automatic configuration generation, you configuration file will contain the full list of available sections that you can edit to exclude unnecessary sections.
💥 Incompatible Changes 💥
File formats
* `rsmcompare` and `rsmsummarize` no longer support experiments that were generated with earlier versions of RSMTool. You will need to re-run the experiments that you want to compare or summarize.
* `rsmtool` no longer supports old-style configuration files (not used since v5.5 or earlier).
* `rsmtool` no longer supports feature files in `.json` format (not used since v5.5 or earlier).
* The Intermediate file containing true score evaluations [`true_score_eval`](https://rsmtool.readthedocs.io/en/stable/usage_rsmtool.html#evaluations-based-on-test-theory) no longer contains variance of human scores. This information can still be obtained from [`consistency`](https://rsmtool.readthedocs.io/en/stable/usage_rsmtool.html#human-human-consistency) files.
API Changes
* The `Configuration` and `ConfigurationParser` objects in the [
`configuration_parser`](https://rsmtool.readthedocs.io/en/stable/api.html#module-rsmtool.configuration_parser) module have been fully refactored. A new `Configuration` object can now be instantiated using a dictionary with keys using the same name as the fields in the [configuration file]((https://rsmtool.readthedocs.io/en/stable/usage_rsmtool.html#experiment-configuration-file)) . Validation and normalization is now done as part of initialization. See [this PR](https://github.com/EducationalTestingService/rsmtool/pull/388) for more detail.
* `Configuration` objects no longer have a `filepath` attribute. Use the `configdir` attribute to indicate what any relative paths in the dictionary are relative to.
* Functions in the erstwhile `rsmtool.utils` module have been moved to new locations. This includes several functions for computing evaluation metrics (``agreement``, ``difference_of_standardized_means``, ``partial_correlations``, ``quadratic_weighted_kappa``, and ``standardized_mean_difference``). See the [API documentation](https://rsmtool.readthedocs.io/en/stable/api.html#from-utils-module) for the new location of these functions.
* The API for computing PRMSE has changed. See the [API documentation](https://rsmtool.readthedocs.io/en/stable/api.html#from-prmse-module) for new functions.
🛠Bugfixes & Improvements ðŸ›
* v7.1.0 did not allow `run_*` functions to accept `pathlib.Path` objects for paths to configuration files. This is now allowed.
* Error messages and warnings produced by RSMTool are now more meaningful and consistent.
* Multiple changes to improve code readability and consistency.