Sherpa

Note that Sherpa can be built against XSPEC 12.13.1 and update some
parameter ranges or defaults to match the new values from this release.

1836 - Allow ARF and RMF to be written out
Support writing out ARF and RMF data to FITS files (and, for ARF, ASCII
files). Fix 1699, 1695, and 203.

1839 - ui: add get_plot_prefs/get_contour_prefs calls
Add the get_plot_prefs and get_contour_prefs call to simplify access to
the plot and contour preferences.

1840 - Fix 1828 - histogram plot issue
Fix up the histogram display for the unlikely case when the bin edges have
at least one gap, the low edges are integer values, and the high edges are
floating-point values.

1841 - Minor refresh of AstroPy I/O code for better FITS and ASCII output
Minor rework of the sherpa.astro.io.pyfits_backend module to improve on
the output (both FITS and ASCII), such as a closer mapping of the data
type used on disk and written out to file, and some changes to the header
lines of ASCII files.

1844 - XSPEC: allow table model redshift parameter range to change (fix 1814)
Allow the redshift range of XSPEC table models to be changed outside the
default 0 to 5 range. This restores a feature that was lost in the 4.15
release. Fix 1814

1848 - Tests: add regrid corner case tests
Add tests for corner-cases for the regrid method. Many of these tests are
currently regression tests, and check that the code behavior has not
changed, rather than that it works correctly. There is no functional
change to the code.

1851 - XSPEC: table model improvements
Address issues with the XSPEC table model support: models that set an
Escale parameter (feature added December 2020) are now supported
(previously they could cause a segmentation fault) and ensure that
parameters with a negative delta value are marked as frozen. Fix 1850 and
1852.

1853 - XSPEC: update model parameters to match XSPEC 12.13.1 / HEASOFT 6.32
Update the XSPEC parameter names and ranges to match those from XSPEC
12.13.1 / HEASOFT 6.32.

1854 - models: change defaults for Box1D parameters
The xlow parameter of Box1D now defaults to 1 and the ampl parameter has a
greatly-increased range (from the previous version of -1 to 1).

1855 - group_counts et al now default to using the existing mask as the
tabstops, unless explicitly given
When calling group_counts, group_width, ..., use the current mask as the
tabStops argument by default (actually, the inverse of the mask), that is,
when the tabStops argument is set to None (the default). This means that
the grouping is only applied to the selected channels, and means that the
filter range is less likely to change when filtered data is grouped.
This is a change in behaviour. To get the previous behaviour users need to
set tabStops to an array of zeros (length being the number of channels):
e.g. for Chandra ACIS imaging-mode data, group_counts(20, tabStops=[0] *
1024) or group_counts(20, tabStops=np.zeros(1024)). Fix 417 956 1081

1856 - XSPEC: fix link for XSPEC gauss/zgauss models
Fix the link to the XSPEC documentation for the gauss and zgauss models.

1858 - Correctly handle XSPEC table models with both ESCALE and Z parameters
Update XSPEC table model support for models with both ESCALE and REDSHIFT
keywords set to work correctly. Fix 1857

1861 - Allow XSPEC table models to be created and written out
Provide experimental support for creating XSPEC table models (additive or
multiplicative). The output routines can be found in the sherpa.astro.ui.
xstable module. Note that XSPEC support is not required to write out an
XSPEC table (but it is to be able to read it back in and use it). Fix 1859

1866 - Switch to conda-forge and use new Conda channel
Removes use of defaults channel and replaces it with conda-forge; removes
references to anaconda

Add explicit support for XSPEC 12.13.0. Sherpa can be built against XSPEC
12.13.0 without this PR, but this adds in support for the new cglumin
convolution model added in this release.

1637 - report the filter changes when grouping is changed for the ui
Report the filter when data is grouped with UI commands - such as group,
group_counts, and set_grouping - to match the notice/ignore behavior added in
1562. The logic for setting the background group flag has been moved from the
UI layer to the DataPHA class. Addresses 1632.

1644 - Deployments via GH Actions
Dropped support for RH7 and macOS 10.15

1656 - CIAOX deployments require force. As they deployed for main already.
Based on the current scheme used for the build number, we need to force uploads
for CIAOX as they have already been deployed for main. This was already done for
the previous GitLab builds.

1660 - Associated group fixes
Clean up of the notice/ignore and group/ungroup code to avoid unnecessary work,
move logic from the UI layer into the DataPHA class, and ensure that if the
grouping field is set to None that the data is not grouped. Fix 1641, 1657,
and 1659

1662 - Add grouping tests
Add a number of tests to check out corner case behavior, primarily for grouping
PHA data but there's also a number of routines from the UI code that have missed
some tests.

1664 - Allow save_model/source/resid to work with 2D data (fix 1642)
Fix save_model/source/resid calls when used with 2D data sets. Fix 1642

1665 - Fix an issue with show_bkg (fix 1645)
The show_bkg routine could fail when the set of backgrounds is not the same for
all datasets (e.g. one has a background and another does not). Fix 1645

1666 - model-wrapped symbols now reference the actual model (fix 215)
Wrapped models, accessible from the UI layer, now reference the wrapped model
rather than being devoid of any useful information. Fix 215

1672 - models: f-string changes
Change the models code to use f-strings where appropriate rather than other methods
of string interpolation. There is no change in behavior.

1675 - Bump ds9 to 8.4.1 - Ubuntu 20 and macOS BigSur
Bump to the latest DS9 version and update to ubuntu20 version. This resolves 1674

1676 - update model macro definitions for compiled code
Re-work the macro definitions used to build the compiled models and use this to
clean up the handling of the XSPEC models, in particular table models. There is
no functional change.

1681 - XSPEC test cleanup
Minor clean up to some XSPEC tests to make better use of pytest.

1689 - Address regression created by 1672
Internal change to the Sherpa-specific logging code, to ensure that logged
message includes all components. This fixes 1688 which was a regression created
when 1672 was recently merged.

1693 - Test cleanup related to 1334
Complete the cleanup related to removing the 1334 workaround which was added in
1336. These two tests were missed from 1472 (which was meant to remove these
changes). The two test files have also seen some minor style changes (no change
in behavior).

1696 - Docs: fix 1694
Correct a link in the RTD documentation.

1697 - Ds9 test tweaks
Minor cleanup of the DS9 tests (no functional change)

Allows Sherpa to be built against XSPEC 12.11.1. There are no new or changed models
in this release compared to XSPEC 12.11.0.

899 - Update likelihood descriptions
Updates likelihood description in several doc-strings and clarifies descriptions
of statistics.

900 - Ensure that 1D and 2D models are not combined in an expression.
Checks that models have the correct dimensionality when combining them, so
expressions like gauss2d.src + const1d.bgnd will now raise a ModelErr.

906 - Improve and add support for histogram plots
Updates to the display of histogram-style plots, in particular for 1D integrated
datasets and some model display for PHA data. The histograms now cover the full
length of each bin (previously they only showed half the bin for the first and
last bins), and gaps in the histogram (where the high edge of a bin is less than
the lower edge of the next bin) are now correctly displayed.

907 - FEATURE: support alpha transparency for matplotlib plots
Supports the 'alpha' preference setting for most plots and contours generated
by Matplotlib

909 - Refactor: remove _get_model/source methods
Removes the _get_source and _get_model methods as they are the same as
get_source and get_model

910 - Docs: fix several minor issues
Documentation clean up - the load_template_interpolator function was named
incorrectly in the example, and the examples for
get_source_component_plot/get_model_component_plot were missing the trailing
_plot for the function names

911 - Docstring: changes for ahelp
Formatting changes to the docstrings for several routines that are useful for
SDS in generating ahelp files

918 - fix an issue with PHA filtering that affects plot_model
Fixes an issue when applying filters to generate the plot_model and
plot_model_component plots for PHA datasets

919 - enable regrid for the BinaryOpModel class (rebased 798)
Enables composite models (created by a binary operation between two models) to
be regridded. The composite model is evaluated at the new grid, and it is only
the combined model expression that is rebinned to the data grid.

922 - channel settings with grouped PHA data and model plotting
Fixes a bug when filtering a grouped PHA dataset using analysis=channel. The
selected bin ranges did not always match the versions you would have received
when doing the same operation with energy or wavelength analysis (the first or
last bin may have been different).

924 - Updated to ignore the .vscode directory
updates system .gitignore to skip over vscode directories

929 - Support overplot option in plot_fit_xxx (issue 700)
The overplot argument can now be used with the plot_fit_xxx and plot_bkg_fit_xxx
routines (e.g. plot_fit_ratio).

931 - Update Data1DInt / DataPHA data plots to use the histogram plot style
Switch the plot_data/plot_bkg plots to draw the data as histograms for Data1DInt
and DataPHA plots. This will change the behavior of code that accesses the plot
data - e.g. get_data_plot() or the dataplot element of get_fit_plot() - since for
DataPHA and Data1DInt datasets the data will no-longer have an x attribute but
xlo and xhi. To reduce the need for code changes for existing scripts - as many
people use get_model_plot and get_data_plot to get the data - the histogram
plots will return (xlo+xhi)/2 when asked for the x attribute.

939 - Docs: note XSPEC convolution models are new in 4.12.2
Documentation only change noting that the support for XSPEC convolution models
is new to 4.12.2.

940 - Minor documentation improvements
Fixed several minor issues in the existing documentation

944 - Improve handling of the default id with PHA background datasets (fix issue 943)
The sherpa.astro.ui.set_default_id call no-longer sets the default identifier for
background ids, which are now kept as the value 1. This avoids several issues when
using set_default_id with the background components of PHA datasets

950 - Documentation fixes
Minor documentation updates and adding ReSampleData to the RTD documentation

953 - docstring fixes
Cleans up typos in documentation

957 - Docs: include the example notebook in the RTD pages
Adds the example notebook (SherpaQuickStart.ipynb) to the Sphinx documentation
under the "Notebooks" heading

961 - Update notebook support for Data1DInt/PHA data
Improves the display of Data1DInt and DataPHA objects when displayed directly
by Jupyter notebook and add a new notebook showing off the notebook support

963 -Docs: note plot_fit_xxx overplot change and add info on notebook support
Adds documentation updates including notes on overplot support in plot_fit_xxx
and plot_bkg_fit_xxx functions and RTD notes on adding notebooks

964 - Docs: avoid invalid escape sequence warning
Very-minor tweak to the Chi2 docstring

968 - Fix 2d image filtering (fix 965)
Fix problems with ignore2d and notice2d when multiple regions are used.

969 - Support Python 3.8 for MacOSX
Add a new multiprocessing_start_method option to sherpa.rc and
sherpa-standalone.rc and init code to set start multiprocessing method
to fork by default.

971 - Docs: fix sphinx warnings
Fix documentation in sherpa.sim.sample on the ReadTheDocs site.

979 - Improve plots of PHA data sets and models when the response grid is not ideal
Improve the display (plots) of PHA data and models when the response grid contains minor numerical differences.
(fixes 977)

984 - Address set_xlog/ylog problems with DataPHA/Data1DInt classes (981)
Fix the set_xlog/ylog routines for PHA and 1D integrated datasets.

3.0.9. Without this change models such as XSapec would return all 0's (unless
the AtomDB version was set manually by the user).

534 XSPEC 12.10.1 support and mtable fix
Sherpa now supports XSPEC 12.10.1. Several issues were fixed to ensure
compatibility with this release. Note that there are known issues with version
12.10.1 (no patches), so at least v12.10.1a must be used.

537 / 552 sherpa.rc changes
Sherpa does not read the verbosity.level setting from sherpa.rc anymore but
the option is still in the file for backward compatibility

539 Fix XSPEC models: mtable table models and ismabs (fix 535, 538, 540)
Fix to address issues with XSpec- 535 XSPEC multiplicative table models
(mtable models loaded with load_xstable_model); 538 incorrect evaluation of
the ismabs model; 540 the kerrd model evaluated to 0 for XSPEC 12.10.0 and
later.

546 Small code cleanup in LevMar
small code cleanup in the LevMar class to match the other classes in that
file.

564 / 567 Harmonise the handling of dof is zero or negative in fit and calc_stat_info (fix 565)
The `fit` and `calc_stat_info` methods now use the same code to calculate
the reduced statistic (`rstat`) and "quality of fit" (`qval`) values. This
avoids a `TypeError` when the number of degrees of freedom is negative in
`calc_stat_info` and ensures that `NaN` is returned for both values when the
goodness-of-fit calculation has failed (e.g. the reduced statistic is zero or
less or the statistic is negative), whereas in previous versions the `qval`
value could be `None` or `1` depending on the code path.

576 Fix regression with responses `get_x` method
Version 4.10.1 introduced a regression which made the `get_x()` method fail
when called on RMF and ARF objects. This has been resolved.

583 Update pager code (fix 445 561)
Replace the use of an external pager (such as `more` or `less`) with a
Python one. This means that Sherpa no-longer uses the PAGER environment
variable, and that the screen output from the `show_*` series of commands
should now appear in the correct location when using a Jupyter notebook or the
spyder application.

Update to use newer ds9 version

1888 - UI: set_analysis reports the filter change - fix 1590
The sherpa.astro.ui.set_analysis call now reports the new filter. It will
also now error out if no data has been loaded. Fix 1590 and 1368.

1890 - Convert a RMF to an image
Add the sherpa.astro.instrument.rmf_to_image and sherpa.astro.instrument.
rmf_to_matrix routines for converting a RMF to DataIMG or a 2D matrix
respectively. Fix 1619. The create_non_delta_rmf routine in the same
module no-longer assumes the matrix is square. Fix 1889.

1891 - ui: add calc_bkg_stat - fix 333
Adds the calc_bkg_stat, calc_bkg_stat_info, and get_bkg_stat_info
routines. Fix 333.

1892 - data: group backgrounds before source - fix 1881
Change the ordering of operations when grouping background PHA datasets.
Fix 1881

1899 - Bokeh class
Provides a new backend plotting class which utilizes the Bokeh
visualization package.

1906 - Remove hardcoded matplotlib for _repr_html_ of RMF and DataImage2D
Implement RMFPlot and DataIMGPlot classes and associated UI functions
(plot_rmf, get_rmf_plot). Fix 1385

1908 - Add versionadded note to set_plot_backend
Added note that the set_plot_backend command is new in 4.16

1909 - Add security warnings to restore functions
Added documentation warning of inherent risks of importing binary files
when using restore functions.

1911 - IO: add missing import for pyfits backend
Ensure a corner-case error condition when writing out responses fails with
the correct error.

1912 - Warn when reading in a multi-matrix rmf
Ensure that attempts to read in a multi-matrix RMF will warn the user that
they are not yet supported.

This release of Sherpa includes various enhancements, documentation updates, bug fixes, and infrastructure changes.

* enhancements:
* the grouping commands like group_counts and group_snr now default to only grouping within the noticed range of channels, which is a change in behaviour
* new plotting backend: Users can now choose between matplotlib and bokeh (experimental) support use of arbitrary python functions when linking model parameters
* updates to fake_pha, save_all, allowing to write out RMF/ARF files
* get_plot_prefs and get_contour_prefs call to simplify access to the plot and contour preferences
* implementation of RMFPlot and DataIMGPlot classes and associated UI functions (plot_rmf, get_rmf_plot)
* update support for XSPEC to include version 12.13.1, allow XSPEC table models which include the ESCALE parameter, and provide experimental support for writing out XSPEC table models.
* documentation changes:
* added doc strings for sherpa data classes
* several updates to in code and RTD documentation
* Infrastructure changes:
* revamp of plotting backends from modules to classes and adding support for multiple backends
* changes to use the NumPy random generator API
* bug fixes:
* PHA source plot Y axis scaling (1825)
* fix model display for grouped data (1779, 1784)
* Change in the ordering of operations when grouping background PHA datasets. (1881)

Details
-------

1382 - Plot changes: Step 1 A- Draft API for backend classes
Plotting infrastructure changes that make the plotting backends classes
instead of modules.

1617 - Pass keyword arguments to models
Allow keyword arguments to be passed to models and ensure these arguments
are respected by the model cache.

1629 - Note when a XSPEC model is changed to F77 form (double precision)
Update a script used when updating the XSPEC support to point out models
that use the double-precision FORTRAN support. There is no functional
change to the code.

1653 - Allow arbitrary functions for linking parameters
Provide a way to use any Python function to be used for linking
parameters. Fix 1652

1730 - Update xspec model creation
Update the code created by the sherpa.astro.utils.xspec routines to
account for changes in the XSPEC interface in Sherpa. This only affects
users who call these routines directly.

1734 - Unify the fake and fake_pha calls
The fake_pha call now accepts a method argument to match the fake call.

1735 - Allow use of the NumPy random generator API
Take advantage of the NumPy random-number generator classes to provide
random numbers, and move away from using the global state provided by
numpy.random.seed and random.seed routines.

1738 - Return more-useful information when an XSPEC model fails
Include the model name and parameter values in the error message of a failed XSPEC model.

1750 - Replace pkg_resources as it is deprecated
Remove the use of the deprecated pkg_resources module and replace with use
of importlib.resources. Fix 1723

1763 - Allow load_image to change the coordinate system
Ensure that the coordinate setting used in load_image calls is properly
handled (when not set to the default value of "logical"). Fix 1762

1770 - Drop python2 ism in the fit code
Internal clean up of the fit code to replace a Python 2 method with a
Python 3 version and to use f-strings.

1771 - utils: make sure OutOfBoundErr is an actual exception
Ensure that OutOfBoundErr is an exception (this is an internal feature).

1772 - Tweak resample data code
Internal changes to the ReSampleData class to better handle invalid input.

1773 - Internal clean up of estmethods
Code clean up of sherpa.estmethods.

1775 - Change a single test to use explicit rather than implicit random state
from NumPy
Simplify a single test. There is no functional change.

1776 - Move the parallel code into a separate module
Move the parallal_map code from sherpa.utils to the new sherpa.utils.
parallel module (the symbols are still accessible from the original
location but code should be updated to use the new location, including the
renames of _ncpus and _multi to ncpus and multi).1777 - Change get_sample
to require named arguments

1778 - Optmethods cleanup
Move testing code out of the optmethods modules and into a separate test,
along with a number of code clean ups. There is no functional change.

1780 - Fix model plot display when factor is set for grouped PHA data (issue
1779)
Ensure that model plots can be created for PHA data when set_analysis
routine is used to set factor > 0 and the data is grouped. Fixes 1779 and
1784

1781 - utils: remove the numpy_convolve routine
Remove the sherpa.utils.numpy_convolve routine.

1785 - Add docs to Sherpa data classes
Add docstring to most Sherpa dataclasses.

1791 - Tests: ensure repeatable test of get_draws
Ensure tests of get_draws use a fixed seed to make the results
reproducible. There is no functional changes in this commit.

1794 - docs: note requirement on ar for building Sherpa
Add ar to the list of requirements for building Sherpa. Fix 1792

1795 - Add docs to describe the inputs for the ``calc`` function of a user
model
Add docs that describe the arguments of the calc function for user models.

1797 - model: explicitly send parameter argument to calc
Ensure that the calc method for models begins with the parameter array
(explicitly given) rather than extracting it from the named arguments sent
to the routine. This should not change any user code.

1800 - utils: minor code clean up
Internal clean up of the sherpa.ui.utils module.

1805 - Clean up display of dataset ids during fits
Ensure dataset ids are listed without leading brackets in the fit and
staterror string outputs (remove an accidental change made in 1770). Fix
1804

1806 - Activate sphinx directive to make plots
Use the Sphinx plot directive to generate (some of) the plots in the docs.

1807 - NumPy 1.25 support
Remove deprecation warnings seen when using NumPy 1.25.0.

1808 - Use Sherpa install instead of code only to build docs
Switches from building the documentations from the source directory, to
building it from an installed version of Sherpa.

1809 - Update the hardcoded Zenodo releases to include the 4.14/4.15 releases
Update the stored Zenodo releases to include the 4.14.* and 4.15.* Sherpa
releases. Users should see no difference (other than avoiding the need for
an http call to retrieve this information) when calling sherpa.citation().

1810 - RTD: fix script name for XSPEC updates
Improve the documentation on how to update to a new XSPEC version by
ensuring the correct helper script is run in an example.

1811 - ui: use the default pickle version when dumping the state
Use the default version when pickling data (used by the UI save call)
rather than an old version. Fix 1243

1812 - Autoupdate copyright year in sphinx docs
Autoupdate copyright year in sphinx docs.

1813 - Fix ReadTheDocs warnings and then make RTD fail on warnings in CI
Fix ReadTheDocs warnings and then make RTD fail on warnings in CI.

1819 - CI: hide NumPy 1.25 array ndim>0 deprecation warnings
Allow the tests to run with NumPy 1.25 and with Crates (from CIAO 4.15) as
the I/O backend. There are no functional changes.

1821 - parallel: tweak ordering of checks for parallelization
Internal change to the decision of whether to use the multiprocessing
module to run code or not.

1823 - sherpa_test: tweak how pytest/plugins are installed
Internal changes to how we check for test requirements used by the
sherpa_test script.

1824 - remove non-existent datastack plotting backend initialization attribute
Clean up error condition by removing the sherpa.plot.backend.
initialize_backend attribute as no longer exists.

1828 - Improve PHA source plots: support type=counts and factor above 1
Source plots for PHA data now follow the rate and factor analysis
settings. Prior to this the rate setting was ignored and the factor
setting produced incorrect values for factor=1 and factor=2). Fix 1825

1829 - Limit precision in the comparison of string outputs
Update the tests to support running on ARM/AARCH64 platforms where numeric
differences can complicate string comparisons. Fix 1815.

1832 - astropy: rework RMF input (fix 1344)
Allow the AstroPy backend to read in HRC-I RMF data. Fixes 1344.