Aiida-core

See [CHANGELOG.md](https://github.com/aiidateam/aiida-core/blob/develop/CHANGELOG.md)

See [CHANGELOG.md](https://github.com/aiidateam/aiida-core/blob/develop/CHANGELOG.md)

This is the fifth minor version of the v1 series and comes with a couple of new features, as well as some relevant bug fixes. Support for Python 3.5 is dropped and support for Python 3.9 is added.

Process functions are now submittable

Since the changes introduced in v1.0, a process function creates a `Process` instance in the background, just as any other process. This means that, in principle, it could also be serialized and deserialized by a daemon worker from that point onwards. Here we remove the final limitations that prevented process functions from being fully submittable.

It is important to note that the usual rules for making processes submittable apply in this case as well: to be importable by the daemon, it will need to either be registered through an entry point or have its source module available through the PYTHONPATH environment variable. For more information on these requirements, you can check the specific section on [launching processes](https://aiida-core.readthedocs.io/en/latest/topics/processes/usage.html#process-launch) in the AiiDA documentation.

New REST API features

This minor version introduces two new features to the REST API interface of AiiDA. The first one is that the base URL (`http://127.0.0.1:5000/api/v4` in a local deployment, for example) will now return a list of the available endpoints. The second one is the addition of a new endpoint `full_types_count`: this returns the same branched structure that the already existing `full_types`, but with the addition of a `counter` key which contains the count of all available node instances of the type indicated.

Archive Refactor

This is part of a series of modifications that aim at making several improvements in the performance and versatility of the process of exporting and importing AiiDA archive files. These changes here should minimally affect the existing interface and might already have some performance boosts, but are an important stepping stone for more significant improvements in the upcoming versions.

This is the fourth minor version of the v1 series and comes with some important performance improvements and features, but also various bug fixes. This is the **last version to support Python 3.5**.

RabbitMQ server can now be hosted on a different machine
In previous versions, AiiDA only supported connecting to a RabbitMQ instance running on localhost. With v1.4, one can now configure AiiDA to connect to a RabbitMQ instance over TCP/IP with optional SSL enabled. Existing profiles will be automatically migrated to add the standard configuration. To see how a new profile can be configured to connect to a remote RabbitMQ instance, [please refer to the documentation](https://aiida-core.readthedocs.io/en/latest/intro/installation.html#rabbitmq-configuration).

Group extras
Groups now support setting "extras": just like nodes, you can now store arbitrary metadata on group instances, as long as it is JSON serializable. The [interface to setting, getting and deleting extras](https://aiida-core.readthedocs.io/en/latest/reference/apidoc/aiida.orm.html#aiida.orm.entities.EntityExtrasMixin) on a group is exactly the same as that for nodes. Note that, just as they are for nodes, extras are mutable. It is therefore not advised to use them to store information that is important for data provenance.

Scheduler output parsing
This version comes with a new method for the scheduler base class [`Scheduler.parse_output`](https://aiida-core.readthedocs.io/en/latest/reference/apidoc/aiida.schedulers.html#aiida.schedulers.Scheduler.parse_output) that can be implemented by scheduler plugins to parse the output that is generated by the scheduler when running a calculation job. The output consists of the detailed job info (if implemented by the scheduler plugin) and the output written to the `stdout` and `stderr` file descriptors. This information can hold important information about how the job terminated. It can be used to determine common problems, such as out-of-memory issues or out-of-walltime errors. The `parse_output` can return a corresponding exit code that is defined on the [`CalcJob`](https://aiida-core.readthedocs.io/en/latest/reference/apidoc/aiida.engine.processes.calcjobs.html#aiida.engine.processes.calcjobs.CalcJob) class. You can take a look at the [`SlurmScheduler.parse_output`](https://aiida-core.readthedocs.io/en/latest/reference/apidoc/aiida.schedulers.plugins.html#aiida.schedulers.plugins.slurm.SlurmScheduler.parse_output) implementation as an example of how it parses the OOM and OOW errors. The scheduler output parsing, if implemented, will be called by the engine before the output `Parser` is called, if any. If the scheduler parsing returns an exit code, the engine will already set it on the node but the output parser will still be called. This allows the output parser to still try to parse some data from the retrieved files, and if desired, override the exit code of the scheduler with a more specific exit code. For details on this process and the interaction between exit codes of scheduler and output parsers, [see this section of the documentation](https://aiida-core.readthedocs.io/en/latest/topics/calculations/usage.html#scheduler-errors).

This is the third minor version of the v1 series and comes with some important performance improvements and features, but also various bug fixes. In addition, [the documentation](https://aiida-core.readthedocs.io/en/latest/) has received a major overhaul, both in structure as well as appearance.

Engine performance
One of the big improvements of AiiDA v1.0 was the switch to an event-based engine that makes it a lot more responsive and performant. Running processes broadcast messages, such as when their state changes, that others, such as the daemon workers, can respond to. In this release, this mechanism is used to make the engine even more responsive. A work chain that runs subprocesses, will now start listening to their state changes, such that once the last running subprocess messages that it has finished, the work chain can resume to the next step immediately. This event-based messaging makes the work chains very responsive, however, it is possible that now and again a message slips through the cracks, in which case the work chain would be waiting indefinitely. As a backup, we keep a polling mechanism that will check the status of each subprocess every interval to see whether it has already finished. In previous releases, this was the only mechanism and it would poll every second. This would keep the computer running AiiDA unnecessarily busy. With the new broadcasting mechanism, we were able to increase the polling interval from 1 to 60 seconds. You can even customize this if you would like:

verdi config runner.poll.interval 120

Note that this polling mechanism is really a backup system and the engine will not rely on it in the vast majority of cases. Still, if you don't mind waiting a bit longer in this very unlikely case, you can always increase the interval a bit more.


For more details on this release, see the [change log](https://github.com/aiidateam/aiida-core/blob/v1.3.0/CHANGELOG.md).

Happy computing!

The AiiDA team

This is the second minor version of the v1 series and comes with mostly new features.
Especially groups have received a lot of new functionality.

Groups can now be subclassed
The `Group` class can now be subclassed to implement custom functionality, just like for example the `Data` node class. They can be registered through entry points in the new entry point category `aiida.groups`. Please refer to the [documentation](https://aiida-core.readthedocs.io/en/latest/working_with_aiida/groups.html#create-a-group-subclass) for more details.

Virtual group hierarchy through `GroupPath`
The group concept in AiiDA so far has been "flat". That is to say, group can contain nodes, but you a group cannot contain another group. Many users have expressed the wish to be able to treat groups a bit more like folders on a filesystem and add some hierarchy to them. Thanks to a contribution by Chris Sewell, the `GroupPath` utility now provides a virtual hierarchy based on the group labels. Forward slash characters `/` in group labels will be interpreted as sub groups. A very short demonstration:

In [1]: from aiida.tools.groups import GroupPath
...: Group(label='project/sub/a')
...: Group(label='project/sub/b')

In [2]: path = GroupPath('project/sub')
...: for child in path.children:
...: group = child.get_group()
...: print('Group<{}> contains {} nodes'.format(group.label, group.count()))
...:
Group<project/sub/a> contains 0 nodes
Group<project/sub/b> contains 0 nodes

Of course the `GroupPath` class provides much more than that and it is also exposed on the CLI through `verdi group path ls`. Extensive documentation of all functionality will be added to the official documentation soon.


For more details on this release, see the [change log](https://github.com/aiidateam/aiida-core/blob/v1.2.0/CHANGELOG.md).

Happy computing!

The AiiDA team