Sherpa

138: improve and fix issues in `save_all` function.
- added a new argument to `save_all`: if `outfile` is `None` then the `outfh`
argument is used
to define the output handle (the argument can be any file-like argument, such
as a file
handle like `sys.stdout` or the output of `open`, or a `StringIO` object)
- setting the `clobber` argument to `save_all` now means that the output file
(the `outfile`
argument, if not `None`) is deleted if it already exists; prior to this, the
file would be
appended to instead
- the source expression is now saved correctly for most cases (e.g. when not
using `set_full_model`); this is bug 97 but also affects non-PHA data sets
- the background model expression was not always written out correctly when
using PHA
data sets
- quality and grouping arrays of PHA data sets are now stored as 16-byte
integers rather
than a floating-point value (this has no affect on the results, but matches
the OGIP standard)
- fixed up saving the grouping and quality arrays of background PHA data sets
(this would only
be an issue if the background is being fit, rather than subtracted)
- basic data sets created with the `load_arrays` function are now written out by
`save_all`
as part of the script; this is intended for small datasets and may have
problems with
precision if used with floating-point arrays
- calls to `load_psf` are now correctly restored (they may not have been written
out correctly
if multiple data sets were loaded)
- user models are now written out to disk; this consists of two parts:
- writing out the function that defines the model, which may or may not be
possible (if
not, a place-holder function is added to the output and a warning
displayed).
- the necessary calls to `load_user_model` and `add_user_pars` are now
included in the output
- the Python code created by `save` all has undergone several minor changes:
- it now explicitly imports the `sherpa.astro.ui` module, so that it can be
run from the
IPython prompt using the `%run <filename>` command, or directly as `python <filename>`
- it uses the `create_model_component` function rather than `eval` to create
model
components (this is CXC bug 12146)
- many optional arguments to functions are now given as `name=value` rather
than
being a positional argument, to make it clearer what the script is doing.
- calls to `load_data` have been replaced by more-specific versions - e.g.
`load_pha`
and `load_image` - if appropriate
- there have been several minor syntactic clean ups to better follow the
suggestions from PEP8

When writing out code that defines a user-model, there is no attempt to make
sure that
modules used by the function are available. These will need to be added, either
directly
or imported, manually to the output.

151: Ensure AstroPy and Crates behave the same with gzipped files. Change the
behaviour of the AstroPy back end so that it matches that of Crates when given a
file name which does not exist, but a compressed version, with the suffix `.gz`
does. The Crates behavior is to read the file. This extends to PHA files whose
ancillary files - e.g. those stored in the `BACKFILE`, `ANCRFILE`, and
`RESPFILE` keywords - are given as unzipped names, but only the gzipped names
exist on disk.

As an example: if `pha.fits.gz` exists but `pha.fits` does not, then


load_pha('pha.fits')


will now load the file with either back end. If the response files are set to
`arf.fits` and `rmf.fits` (via the `ANCRFILE` and `RESPFILE` keywords), but only
the `.gz` versions exist, then they will now also be loaded by the AstroPy back
end.

153: Make comparison test in `calc_chi2datavar_errors` less stringent, so to
include the case where `sqrt(x)=0`.

155: The `get_draws` function now accepts a user-provided covariance matrix. If
no covariance matrix is provided, the covariance matrix computed by the default
implementation is used. Note that `covar()` must be invoked before invoking
`get_draws` if no covariance matrix is provided, otherwise `get_draws` will exit
with an error.

158: Fix bug that prevented region ascii files to be read in standalone
Sherpa.

165: Remove usage of deprecated numpy API.

185: Fix the problem where if the working directory contained a file called
`x` or `y` then the `sherpa.astro.ui.image_data()` function would fail with the message


DS9Err: Could not display image


187: Fix 92: a more meaningful message is given to the user when
`sherpa.astro.io` is imported directly and no fits backends are available.

188: Fix 93. The `sherpa_test` script now tries to install the test
dependencies before running the tests (but not the `sherpatest` package, which
should be installed by the user if necessary, due to its footprint). If this is
not possible, and the necessary dependencies (`pytest`) are not found, then a
meaningful message is given to the user with instructions on how to install the
dependencies. Also, the dependency on `pytest-cov` has been removed. Users can
enable coverage reports from the command line if necessary.

190: Fix 22 - The datastack package can now be used even if there is no
available plotting backend. In this case, plotting functions will not be
available, but the rest of the datastack functionality will.

Caveats

The following are known issues with the standalone 4.8.1 release

**XSpec support**: Several issues have been encountered with the optional source building with XSpec models on OSX platforms (Linux support appears unaffected). The issues include a name clash between the libcfitsio library and the astropy.io.fits Python extensions that results in XSpec failing to load fits files possibly resulting in a crash.

**SAO DS9 issue on Ubuntu 14.04**: the ds9 binaries shipped with Ubuntu and installed through apt-get install do not seem to work as expected. Binaries downloaded directly from the SAO ds9 page seem to work instead. (Note: this issue was listed in the 4.8.0 release as well).

**Wrong save_data header keywords**: when using astropy as a FITS backend to save PHA data with save_data some header keywords are incorrectly set by Sherpa. In particular, range information for certain columns may be inaccurate (see issue 203 for details).

This version of Sherpa introduces 'wstat' statistics which is an extension
of 'cstat' with Poisson background data. It also provides the ability to include
background data with the 'user statistics'.

Many changes were aimed at improving the Xspec extension by making it more
robust, intuitive, and by fixing several bugs. Sherpa 4.8.0 supports Xspec 12.9.0d
and was also tested against versions 12.8.2e, 12.8.2l, and 12.8.2q.

Most of the codebase was reviewed and cleaned up, in particular to remove the
use of deprecated functionality and to comply with Python's PEP8 standard.
More tests were added to the test suite, and
a new testing infrastructure was put in place in order to simplify writing and
running tests, and to measure the test suite code coverage.

More details below (infrastructure changes are not shown):

32: Fix segfault from CRATES update in 4.8b1. Since v4.8b1 CRATES returns
variable length arrays by default, rather than the zero-padded fixed length
ones it used to return. Sherpa manipulated the arrays so to remove the
zero-padding and obtaining variable length arrays. The change in the CRATES
API resulted in Sherpa segfaulting when trying to manipulate the data coming
from CRATES. In the patch, we use a new API offered by CRATES to get the
old-style fixed-length arrays instead of the new default ones. In the future,
we may want to update the Sherpa code to deal with the new arrays directly.

44: `save_quality` now correctly outputs 'QUALITY' as the column name,
instead of 'GROUPS'.

48: Fix up several issues seen in plot labels - titles and Y-axis labels -
for commands such as `sherpa.ui.plot_data`, `sherpa.ui.plot_fit_resid`, and
`sherpa.ui.plot_chisqr`.

59: Fix bug 38 (grouping twice gives an IndexError exception). An un-handled
corner case in one of the Sherpa internal methods (`utils.create_expr`) was
triggering an `IndexError` when two `group_counts` operations were performed
back to back. The fix handles the case so that applying `group_counts` twice
does not result in an Exception.

77: Replace `==` and `!=` comparisons to `None` with `is` and `is not`.

78: OutOfBoundErr exceptions in some sherpa.utils
functions are properly caught. There were several places where screen output
used either `print` or `sys.stderr.write`.

81: Ensure that XSPEC models which fail - for instance, because a data file
it needs is missing - return 0's for all bins, rather than random values. This
should make it more obvious that something has gone wrong (for instance if the
XSPEC chatter level is low enough not to show any error messages, as is the
case for the default setting used by Sherpa, namely 0).

82: The XSpec "spectrum number" value is now set to 1 rather than 0, as this
value is 1-based in Xspec.

83: Removed S-Lang scripts, files, and references in the code.

84: Clarified error messages in Xspec extension. Also, changed the class of
the exception from `RuntimeError` to more appropriate exception types, in
particular `LookupError`, `ValueError`, `KeyError`. This is a
backwards-incompatible change, in that code that caught the `RuntimeError`
will not catch the new error.

87: Some methods in `sherpa/fit.py` assigned mutable objects to default
arguments. This has now been fixed. More instances of this issue have been
identified (Bug 95) and will removed in the future.

90: Added background data to the UserStat class.

94: Implement wstat statistic as described at the following url:
https://heasarc.gsfc.nasa.gov/xanadu/xspec/manual/XSappendixStatistics.html

96: Remove the unused myoptfct module.

99: Correct the documentation for the `set_exposure` function.

100 Fix bug 97, whereby the save_all function would not create the necessary
`set_source()` line. This does not fix all cases, but it does the simple PHA
case such as


load_pha('src.pi')
set_source(xsphabs.gal * powlaw1d.pl)
save_all('test.out).


It also ensures that files created by `save_all`
can be run using IPython's %run directive, by explicitly importing the
`sherpa.astro.ui` module.

101: Fix handling of non-contiguous bins in Xspec - i.e. when a model is
called with both xlo and xhi arguments but the bins do not fully cover the
energy, or wavelenth, range. This fixes 62 (for XSPEC 12.8.2; switching to
XSPEC 12.9.0 should also fix it) and 56. It also fixes an (un-reported)
problem with handling of non-contiguous grids when using a table model, where
a crash was likely. When an XSPEC model is called with both low and high
values for the grid - i.e. with two arguments - then the two arrays are
checked to have the same length, and a ValueError is raised if this condition
does not hold. This is a breaking change, but the results are not guaranteed
to be correct if the two arrays are not the same length. The experimental
interface for XSPEC convolution models has changed, so that the function call
takes pars, fluxes, xlo, with optional xhi whereas before it was pars, xlo,
xhi, fluxes. This is a breaking change, but this is in the low-level API that
is not documented to users, and adds useful functionality (the ability to have
xhi be optional). The cpflux convolution model has been added. Note that these
models do not have Python classes associated with them as they are still an
experimental interface. The test suite has been updated to test the new and
changed functionality in this PR. The choice of models is made so as to avoid
known problematic models (with a version check where relevant). It is
believed that the changes in this PR fixes 42, although this is hard to prove
conclusively given the erratic nature of the bug.

110: Update the sherpa.astro.datastack module documentation to include information from the
CIAO ahelp documentation and to match the style used by the sherpa.astro.ui module.

111: Update the documentation to include more information about the pyBLoCXS code.

Caveats

These caveats are being fixed for the 4.8.1 release.

Incompatibility with matplotlib 1.5: Sherpa 4.8.0 is not compatible with matplotlib 1.5. Unfortunately, this version is currently the default package installed by conda. Users should install sherpa with matplotlib=1.4 numpy=1.9.

Test requirements are not installed automatically: sherpa_test does not work out of the box. Users should issue "pip install pytest-cov" in order for sherpa_test to run.

SAO DS9 issue on Ubuntu 14.04: the ds9 binaries shipped with Ubuntu and installed through apt-get install do not seem to work as expected. Binaries downloaded directly from the SAO ds9 page seem to work instead.

Release Notes

This standalone release is based on CIAO Sherpa v4.7.

Release notes for this baseline version can be found at the following link:
http://cxc.harvard.edu/ciao/releasenotes/ciao_4.7_release.html#Sherpa

Additionally, the present release includes some changes listed below:
- Standalone Sherpa and CIAO sherpa now look for different configuration files in the $HOME directory. For standalone this is `$HOME/.sherpa-standalone.rc`. If this file is not present, Sherpa falls back to the internal configuration file. This file has defaults better suited for the standalone mode: `pyfits` and `pylab` are set as backends, and the stack trace is not silenced. Users can still override the configuration file location by exporting the `SHERPARC` environment variable, as supported by previous versions of Sherpa and CIAO.
- All source files now have copyright and licensing information. A summary is included upfront in the repository main directory.
- Orphan code that was not actually used has been removed.
- Fixed code triggering deprecation warnings from Numpy 1.9.
- Fixed code triggering compiler warnings.
- Added documentation (README, ipython notebook).
- The version string (`sherpa.__version__`) depends on the git commit/tag, unlike in CIAO where it is fixed to 40701.
- Some classes from the template module were not exposed by `__all__`, and template models were not imported in the sherpa session. Now they are.

Known issues
- the datastack module is not imported if no plotting packages are available. Installing `matplotlib` in the same environment as Sherpa fixes the issue.

One of the tests fails in Python 3.10 because the error message has been changed.

1493 - Remove deprecated distutils.version.LooseVersion
The distutils.version.LooseVersion class is now marked as deprecated so remove its
use when building Sherpa and when importing the Sherpa XSPEC module.

1494 - Update AX_PYTHON_DEVEL.m4 from serial 17 to 25
This update covers the cleanup of distutils in the configure files for grplib (and
stklib in the case of standalone sherpa).

1495 - Note that we support Python 3.10
Note that Sherpa can be built using Python 3.10 and add two Python 3.10 CI test runs.

1496 - DS9 update from 8.2.1 to 8.3
Update to use ds9 v8.3 since v8.2.1 is no longer available

1497 - Tests: ensure matplotlib windows are closed after all tests are run
Ensure that our tests can run on CI cleanly.

Matplotlib 3.4 changes how the drawstyle argument is handled in some functions.
This change removes the use of this argument for those functions.

1126 - Tweak plot docs
Adds a new notebook that shows off a number of plots created with matplotlib and
exports the ScatterPlot, TracePlot, CDFPlot, PDFPlot, and LRHistogram classes from
sherpa.plot.

1130: Fix save_data when the output file exists
Fix problems when save_data is used with clobber=False but the output file already
exists. Fixes 1071

1132 - Update XQuartz for GH Actions workflow
Updates the xQuartz download location and version for the GitHub Actions Conda tests

1139 - Fix typo in Zenodo test that made it fail
Fixed a typo in the zenodo test which gets only run via the --runzenodo flag

1142 - Add instruction for source build on Mac
Add instruction for source building on Mac that were previously only part of the
internal release notes.

Internal code changes to improve support for NumPy 2.0.

1874 - save_all improvements for set_psf and load_xstable_model
Ensure that save_all includes set_psf calls, supports XSPEC table models,
and improve the support for data sets created with load_arrays or
dataspace1d/dataspace2d. Fixes 1873.

1877 - un-whitelist deprecation warnings
Cleaned up deprecation warning exceptions from test white list

1878 - Tests: better handle case where both astropy and crates are available
Fix up tests that would fail when both astropy and crates packages were
installed and the pyfits backend was chosen (the tested functionality was
not wrong, just they were testing the wrong things).