Python-icat

~~~~~~~~~~~~~~~~~~

New features
------------

+ `4`_: Extend :ref:`icatrestore <icatingest>` to become a generic
ingestion tool.

Rename :ref:`icatrestore <icatingest>` to :ref:`icatingest`.

Allow referencing of objects by attribute rather then by unique key
in the input file for :ref:`icatingest` (only in the XML backend).

Allow adding references to already existing objects in the input
file for :ref:`icatingest` (only in the XML backend).

Change the name of the root element in the input file for
:ref:`icatingest` (and the output of :ref:`icatdump`) from
`icatdump` to `icatdata` (only in the XML backend).

+ Implement upload of Datafiles to IDS rather then only creating the
ICAT object from :ref:`icatingest`.

+ Implement handling of duplicates in :ref:`icatingest`. The same
options (`THROW`, `IGNORE`, `CHECK`, and `OVERWRITE`) as in the
import call in the ICAT restful interface are supported.

+ `1`_: add a test suite.

+ `3`_: use Sphinx to generate the API documentation.

+ Add method :meth:`icat.client.Client.searchMatching`.

+ Add the :meth:`icat.ids.IDSClient.getIcatUrl` call introduced with
IDS 1.4.0.

Incompatible changes and deprecations
-------------------------------------

+ The Lucene calls that have been removed in ICAT 4.5.0 are also
removed from the client.

+ Deprecate the use of the `autoget` argument in
:meth:`icat.entity.Entity.getUniqueKey`.

Bug fixes and minor changes
---------------------------

+ `6`_: :class:`icat.query.Query`: adding a condition on a meta
attribute fails.

+ `10`_: client.putData: IDSInternalError is raised if
datafile.datafileCreateTime is set.

+ Ignore import errors from the backend modules in :ref:`icatingest` and
:ref:`icatdump`. This means one can use the scripts also if the
prerequisites for some backends are not fulfilled, only the
concerned backends are not available then.

+ `5`_, compatibility with ICAT 4.5: entity ids are not guaranteed to
be unique among all entities, but only for entities of the same
type.

~~~~~~~~~~~~~~~~~~

New features
------------

+ Enable verification of the SSL server certificate in HTTPS
connections. Add a new configuration variable `checkCert` to
control this. It is set to :const:`True` by default.

Note that this requires either Python 2.7.9 or 3.2 or newer. With
older Python version, this configuration option has no effect.

+ Add type conversion of configuration variables.

+ Add substituting the values of configuration variables in other
variables.

+ Add another derived configuration variable `configDir`.

+ Default search path for the configuration file: add an appropriate
path on Windows, add ``/etc/icat`` and ``~/.config/icat`` to the
path if not on Windows.

+ Add `icatexport.py` and `icatimport.py` example scripts that use the
corresponding calls to the ICAT RESTful interface to dump and
restore the ICAT content.

+ The constructor of :exc:`icat.exception.ICATError` and the
:func:`icat.exception.translateError` function are now able to
construct exceptions based on a dict such as those returned by the
ICAT RESTful interface in case of an error.

Unified handling of errors raised from the ICAT and the IDS server.

Incompatible changes
--------------------

+ As a consequence of the unified handling of errors, the exception
class hierarchy has been reviewed, with a somewhat more clear
separation of exceptions raised by other libraries, exceptions
raised by the server, and exceptions raised by python-icat
respectively.

If you put assumptions on the exception hierarchy in your code, this
might need a review. In particular,
:exc:`icat.exception.IDSResponseError` is not derived from
:exc:`icat.exception.IDSError` any more.
:exc:`icat.exception.IDSServerError` has been removed.

I.e., replace all references to :exc:`icat.exception.IDSServerError`
by :exc:`icat.exception.IDSError` in your code. Furthermore, if you
catch :exc:`icat.exception.IDSError` in your code with the intention
to catch both, errors from the IDS server and
:exc:`icat.exception.IDSResponseError` in one branch, replace::

try:
...
except IDSError:
...

by ::

try:
...
except (IDSError, IDSResponseError):
...

Bug fixes and minor changes
---------------------------

+ The :class:`icat.query.Query` class now checks the attributes
referenced in conditions and includes for validity.

+ Fix a regression introduced with version 0.7.0 that caused non-ASCII
characters in queries not to work.

+ Fix :exc:`icat.exception.ICATError` and
:exc:`icat.exception.IDSError` to gracefully deal with non-ASCII
characters in error messages. Add a common abstract base class
:exc:`icat.exception.ICATException` that cares about this.

+ Fix: the string representation operator
:meth:`icat.query.Query.__str__` should not modify the query object.

+ Cosmetic improvement in the formal representation operator
:meth:`icat.query.Query.__repr__`.


.. _changes-0_7_0:

~~~~~~~~~~~~~~~~~~

New features
------------

+ Add a module :mod:`icat.query` with a class
:class:`icat.query.Query` that can be used to build ICAT search
expressions. Instances of the class may be used in place of search
expression strings where appropriate.

Numerous examples on how to use this new class can be found in
`querytest.py` in the examples.

+ Add a class method :meth:`icat.entity.Entity.getNaturalOrder` that
returns a list of attributes suitable to be used in an ORDER BY
clause in an ICAT search expression.

+ Add a class method :meth:`icat.entity.Entity.getAttrInfo` that
queries the EntityInfo from the ICAT server and extracts the
information on an attribute.

+ Add a method :meth:`icat.client.Client.getEntityClass` that returns
the :class:`icat.entity.Entity` subclass corresponding to a name.

+ Add a warning class :exc:`icat.exception.QueryNullableOrderWarning`.

+ Add an optional argument `username` to the
:meth:`icat.ids.IDSClient.getLink` method.


.. _changes-0_6_0:

~~~~~~~~~~~~~~~~~~

New features
------------

+ Add support for ICAT 4.4.0: add new :class:`icat.entity.Entity` type
`InvestigationGroup`, `role` has been added to the constraint in
`InvestigationUser`.

+ Add new API method :meth:`icat.ids.IDSClient.getApiVersion` that
will be introduced with the upcoming version 1.3.0 of IDS. This
method may also be called with older IDS servers: if it is not
available because the server does not support it yet, the server
version is guessed from visible features in the API.

:class:`icat.ids.IDSClient` checks the API version on init.

+ Add new API methods :meth:`icat.ids.IDSClient.isReadOnly`,
:meth:`icat.ids.IDSClient.isTwoLevel`,
:meth:`icat.ids.IDSClient.getLink`, and
:meth:`icat.ids.IDSClient.getSize` introduced with IDS 1.2.0.

+ Add `no_proxy` support. The proxy configuration variables,
`http_proxy`, `https_proxy`, and `no_proxy` are set in the
environment. [Suggested by Alistair Mills]

+ Rework the dump file backend API for :ref:`icatdump` and
:ref:`icatrestore <icatingest>`. As a result, writing custom dump
or restore scripts is much cleaner and easier now.

This may cause compatibility issues for users who either wrote their
own dump file backend or for users who wrote custom dump or restore
scripts, using the XML or YAML backends. In the first case, compare
the old XML and YAML backends with the new versions and you'll
easily see what needs to get adapted. In the latter case, have a
look into the new versions of :ref:`icatdump` and :ref:`icatrestore
<icatingest>` to see how to use the new backend API.

+ Add method :meth:`icat.client.Client.searchChunked`.

+ Add method :meth:`icat.entity.Entity.getAttrType`.

Incompatible changes
--------------------

+ Move the `group` argument to method
:meth:`icat.client.Client.createRules` to the last position and make
it optional, having default :const:`None`.

In the client code, replace::

client.createRules(group, crudFlags, what)

by ::

client.createRules(crudFlags, what, group)

+ The :meth:`icat.client.Client.putData` method returns the new
Datafile object created by IDS rather then only its id.

If you depend on the old behavior in the client code, replace::

dfid = client.putData(file, datafile)

by ::

df = client.putData(file, datafile)
dfid = df.id

Minor changes and fixes
-----------------------

+ The :meth:`icat.client.Client.searchText` and
:meth:`icat.client.Client.luceneSearch` client method have been
deprecated. They are destined to be dropped from the ICAT server or
at least changed in version 4.5.0 and might get removed from
python-icat in a future release as well.

The methods now emit a deprecation warning when called. Note
however that Python by default ignores deprecation warnings, so you
won't see this unless you switch them on.

+ Fixed overly strict type checking in the constructor arguments of
:class:`icat.ids.DataSelection` and as a consequence also in the
arguments of the ICAT client methods
:meth:`icat.client.Client.getData`,
:meth:`icat.client.Client.getDataUrl`,
:meth:`icat.client.Client.prepareData`, and
:meth:`icat.client.Client.deleteData`: now, any
:class:`Sequence` of entity objects will be accepted, in particular
an :class:`icat.entity.EntityList`.

+ Change :meth:`icat.ids.IDSClient.archive` and
:meth:`icat.ids.IDSClient.restore` to not to return anything. While
formally, this might be considered an incompatible change, these
methods never returned anything meaningful in the past.

+ Slightly modified the `==` and `!=` operator for
:class:`icat.entity.Entity`. Add a
:meth:`icat.entity.Entity.__hash__` method. The latter means that
you will more likely get what you expect when you create a set of
:class:`icat.entity.Entity` objects or use them as keys in a dict.

+ The module :mod:`icat.eval` now only does its work (parsing command
line arguments and connecting to an ICAT server) when called from
the Python command line. When imported as a regular module, it will
essentially do nothing. This avoids errors to occur when imported.

+ `setup.py` raises an error with Python 2.6 if python2_6.patch has
not been applied.

+ Add missing `MANIFEST.in` in the source distribution.

+ Remove the work around the Suds datetime value bug (setting the
environment variable TZ to ``UTC``) from :mod:`icat`. Instead,
document it along with other known issues in the README.

+ Minor fixes in the sorting of entity objects.

+ Add an optional argument args to
:meth:`icat.config.Config.getconfig`. If set to a list of strings,
it replaces :attr:`sys.argv`. Mainly useful for testing.

+ Add comparison operators to class :class:`icat.listproxy.ListProxy`.


.. _changes-0_5_1:

~~~~~~~~~~~~~~~~~~

+ Add a module :mod:`icat.eval` that is intended to be run using the
``-m`` command line switch to Python. It allows to evaluate Python
expressions within an ICAT session as one liners directly from the
command line, as for example::

get all Dataset ids
$ python -m icat.eval -e 'client.search("Dataset.id")' -s root
[102284L, 102288L, 102289L, 102293L]

+ Fix an issue in the error handling in the IDS client that caused an
:exc:`urllib2.HTTPError` to be raised instead of an
:exc:`icat.exception.IDSServerError` in the case of an error from
the IDS server and thus the loss of all details about the error
reported in the reply from the server.

+ Add specific exception classes for the different error codes raised
by the IDS server.

+ Fix compatibility issue with Python 3.3 that caused the HTTP method
to be set to :const:`None` in some IDS methods, which in turn caused
an internal server error to be raised in the IDS server.

+ Fix compatibility issues with Python 3.4: some methods have been
removed from class :class:`urllib.request.Request` which caused an
:exc:`AttributeError` in the :class:`icat.ids.IDSClient`.

+ Fix: failed to connect to an ICAT server if it advertises a version
number having a trailing "-SNAPSHOT" in
:meth:`icat.client.Client.getApiVersion`. For compatibility, a
trailing "-SNAPSHOT" will be replaced by "a1" in the
client.apiversion attribute.

+ Suppress misleading context information introduced with Python 3
(PEP 3134) from the traceback in some error messages.
Unfortunately, the fix only works for Python 3.3 and newer.

+ Make example files compatible across Python versions without
modifications, such as running 2to3 on them.


.. _changes-0_5_0:

~~~~~~~~~~~~~~~~~~

+ Integrate an IDS client in the ICAT client.

+ Improved :ref:`icatdump` and :ref:`icatrestore <icatingest>`:

- Changed the logical structure of the dump file format which
significantly simplified the scripts. Note that old dump files
are not compatible with the new versions.

- Add support for XML dump files. A XML Schema Definition for the
dump file format is provided in the doc directory.

The scripts are now considered to be legitimate tools (though still
alpha) rather then mere examples. Consequently, they will be
installed into the bin directory.

+ Implicitly set a one to many relation to an empty list if it is
accessed but not present in an :class:`icat.entity.Entity` object
rather then raising an :exc:`AttributeError`. See `ICAT Issue
112`__.

+ Allow setting one to many relationship attributes and deletion of
attributes in :class:`icat.entity.Entity`. Add method
:meth:`icat.entity.Entity.truncateRelations`. Truncate dummy
relations set by the factory in newly created entity objects.

+ Cache the result from :meth:`icat.client.Client.getEntityInfo` in
the client.

+ Add a method :meth:`icat.entity.Entity.__sortkey__` that return a
key that when used as a sorting key in :meth:`list.sort` allows any
list of entity objects to have a well defined order. Sorting is
based on the Constraint attributes. Add a class variable
:attr:`icat.entity.Entity.SortAttrs` that overrides this and will be
set as a fall back for those entity classes that do not have a
suitable Constraint.

.. __: https://github.com/icatproject/icat.server/issues/112


.. _changes-0_4_0: