Avocado-framework

Users/Test Writers
==================

* The ``dict_variants`` plugin now allows the configuration of the
keys that will determine the variant ID.

* The legacy runner (``--test-runner=runner``) is being dismantled.
In this release, the following features have removed:

1. The ``PYTHON_UNITTEST`` test type

2. The ``--external-runner`` feature and its underlying test type.

3. The ``ROBOT`` test type

4. The ``GOLANG`` test type

* When using the Job API, test suites can be enabled or disabled.
Having a disabled test suite means it won't be executed. This eases
the creation of custom jobs where the user can choose to run a
subset of suites.

Bug Fixes
=========

* The :mod:`avocado.core.job` code was using the root logger, instead
of the logger at the ``avocado`` namespace.

* The automatic status server was very prone to failures due to
``AF_UNIX``'s limitation on the length of paths, because it was
created inside a job's result directory. Now it's created on the
base system temporary directory, which makes it very hard to exceed
the path length limit.

* The :mod:`avocado.utils.vmimage` library received the following
improvements:

1. Ubuntu's provider now properly handles the version number when it
compares versions with trailing zeroes.

2. Ubuntu and OpenSUSE providers can now fetch the best (latest)
version available when no version is given.

3. OpenSUSE provider will now use OpenStack images starting from
version 15.3, due to the other images having been discontinued.

* The ``variants.json`` file, saved at the ``jobdata`` directory
inside a job's result directory, now takes into consideration the
possible multiple suites in a job. The files are now named after
named with a numeric suffix and, if a name was given to the suite, a
name suffix as well.

* The serialization of the job configuration file, also saved in the
``jobdata`` directory, has been updated to support ``set`` data
types.

* ``avocado replay`` executions with a ``--show`` (which takes a set
of builtin loggers) now work properly due to the previous fix.

* Various fixes to the ``runnable-run`` interface behavior of all
shipped runners (detected as part of an improvement in functional
test coverage).

* When using the Job API, some code paths would still resort to using
the legacy runner.

* ``nrunner`` based jobs (the default) can now run from Python egg
based deployments (meaning zero installation steps are required).

* The ``resultsdb`` plugin is now pinned to a known working version
due to a broken release.

* Test parameters given with the command line argument ``-p`` are now
internally converted into variants values. This fixes the issue
with those parameters not being displayed in the ``results.html``
files.

Utility APIs
============

* The new method
`avocado.utils.network.interfaces.NetworkInterface.is_bond` that
allows users to check if a given interface is a bonding device.

Misc Changes
============

* A few portability improvements for macOS systems.

* Misc documentation improvements.

Internal Changes
================

* The RPM packages can now be built without the execution of tests.

* The spawner plugin interface now define two different types of
implementations, with the new one being a spawner that is capable of
deploying itself to the environments that will run the tasks.

* Avocado is now also being tested under Python 3.11.

* Various CI improvements.

Bug Fixes
=========

* Avocado can now find tests on classes that are imported using
relative ``import`` statements with multiple classes. Previously
only the first class imported in such a statement was properly
processed.

* ``avocado run`` will now create test suites without an automatic
(and usually very verbose) name, but instead without a name, given
that there will be only one suite on such jobs. This restores
the ``avocado run`` behavior users expected and are used to.

* Hint files are now being respected again, this time within the
context of test suite creation.

* Filtering by tags is now working properly when using the resolver,
that is, when using ``avocado list --resolver -t $tag --
$reference``.

* Test suites now properly respect the configuration given to them, as
opposed to using a configuration composed by the default registered
option values.

* Fixed the "elapsed time" produced by the ``avocado-instrumented``
nrunner runner (that is, ``avocado-runner-avocado-instrumented``).

* ``avocado --verbose list --resolver -- $reference`` has reinstated
the presentation of failed resolution information, which is useful
for understanding why a test reference was not resolved into a test.

* The "legacy replay plugin", that is, ``avocado run --replay``, can
now replay a subset of tests based on their status.

* The ``avocado diff`` command won't crash anymore if given
``sysinfo`` files with binary content. It will log the issue, and
not attempt to present binary differences.

* The HTML report generated by ``avocado diff`` now runs properly
and won't crash.

* The asset fetcher plugin won't crash anymore due to differences in
the AST based node attributes.

* :class:`avocado.utils.process.FDDrainer` now properly respects the
presence and absence of newlines produced when running new processes
via :func:`avocado.utils.process.run` and friends. This also fixes
tests that relied on the "output check" feature because of missing
newlines.

* The ``nrunner`` plugin will now always display test status in the
most natural order, that is, ``STARTED`` before ``PASS`` or
``FAIL``.

* The ``nrunner`` plugin will now properly set the job status in case
of test failures, resulting in the job (and ``avocado run``) exit
status to properly signal failures.

* A vast documentation review was performed, with many fixes and
improvements.

For more information, please check out the complete
Avocado changelog: https://github.com/avocado-framework/avocado/compare/81.0...82.0

Users/Test Writers
==================

* ``INSTRUMENTED`` tests using the
:func:`avocado.core.test.Test.fetch_asset` can take advantage of
plugins that will attempt to download (and cache) assets before the
test execution. This should make the overall test execution more
reliable, and give better test execution times as the download
time will be excluded. Users can also manually execute the
``avocado assets`` command to manually fetch assets from tests.

* The still experimental "N(ext) Runner" support for Avocado
Instrumented tests is more complete and supports tag filtering and
passing tags to the tests.

* A new architecture for "finding" tests has been introduced as an
alternative to the :mod:`avocado.core.loader` code. It's based
around the :mod:`avocado.core.resolver`, and it's currently used in
the still experimental "N(ext) Runner". It currently supports tests
of the following types: ``avocado-instrumented``, ``exec-test``,
``glib``, ``golang``, ``python-unittest`` and ``robot``. You can
experiment it by running ``avocado nlist``, similarly to how ``avocado
list`` is used.

* Avocado ``sysinfo`` feature file will now work out of the box on
``pip`` based installations. Previously, it would require
configuration files tweaks to adjust installation paths.

* A massive documentation overhaul, now designed around guides to
different target audiences. The "User's Guide", "Test Writer's
Guide" and "Contributor's Guide" can be easily found as first lever
sections contain curated content for those audiences.

Bug Fixes
=========

* Content supposed to be UI only could leak into TAP files, making
them invalid.

* Avocado's ``sysinfo`` feature will now run commands without a
shell, resulting in more proper captured output, without shell
related content.

* :func:`avocado.utils.process.SubProcess.send_signal` will now send a
signal to itself correctly even when using ``sudo`` mode.

Utility APIs
============

* The :mod:`avocado.utils.vmimage` library now allows a user to
define the ``qemu-img`` binary that will be used for creating
snapshot images via the :data:`avocado.utils.vmimage.QEMU_IMG`
variable.

* The :mod:`avocado.utils.configure_network` module introduced
a number of utilities, including MTU configuration support,
a method for validating network among peers, IPv6 support, etc.

* The :func:`avocado.utils.configure_network.set_ip` function now
supports different interface types through a ``interface_type``
parameter, while still defaulting to ``Ethernet``.

Internal Changes
================

* Package support for Enterprise Linux 8.

* Increased CI coverage, having tests now run on four different
hardware architectures: amd64 (x86_64), arm64 (aarch64), ppc64le and
s390x.

* Packit support adding extended CI coverage, with RPM packages being
built for Pull Requests and results shown on GitHub.

* Pylint checks for w0703 were enabled.

* Runners, such as the remote runner, vm runner, docker runner, and
the default local runner now conform to a "runner" interface and
can be seen as proper plugins with ``avocado plugins``.

* Avocado's configuration parser will now treat values with relative
paths as a special value, and evaluate their content in relation
to the Python's distribution directory where Avocado is installed.

For more information, please check out the complete
`Avocado changelog
<https://github.com/avocado-framework/avocado/compare/72.0...73.0>`_.

Users/Test Writers
==================

* Avocado can now run on systems with nothing but Python 3 (and
"quasi-standard-library" module ``setuptools``). This means that it
won't require extra packages, and should be easier to deploy on
containers, embedded systems, etc. Optional plugins may have
additional requirements.

* A new and still experimental test runner implementation, known as
"N(ext) Runner" has been introduced. It brings a number of
different concepts, increasing the decoupling between a test (and
its runner) and the job. For more information, please refer to
:ref:`the early documentation <nrunner>`.

* The new :func:`avocado.cancel_on` decorator has been added to the
Test APIs, allowing you to define the conditions for a test to be
considered canceled. See one example :ref:`here
<turning_errors_into_cancels>`.

* The ``glib`` plugin got a configuration option its safe/unsafe
operation, that is, whether it will execute binaries in an attempt to
find the whole list of tests. Look for the ``glib.conf`` shipped
with the plugin to enable the unsafe mode.

* Avocado can now use tags inside Python Unittests, and not only on
its own Instrumented tests. It's expected that other forms or
providing tags for other types of tests will also be introduced in
the near future.

* The HTML report will now show, as a handy pop-up, the contents of the
test whiteboard. If you set, say, performance metrics there, you'll
able to see straight from the report.

* The HTML report now has filtering support by test status, and can
show all records in the table.

* The :mod:`avocado.utils.runtime` module, a badly designed mechanism
for sharing Avocado runtime settings with the utility libraries, has
been removed.

* The test runner feature that would allow binaries to be run
transparently inside GDB was removed. The reason for dropping such
a feature have to do with how it limits the test runner to run one
test at a time, and the use of the :mod:`avocado.utils.runtime`
mechanism, also removed.

* Initial examples for writing custom jobs, using the so called Job
API, have been added to ``examples/jobs``. These APIs are still
non-public (under core), but they're supposed to become public
and supported soon.

* By means of a new plugin (``merge_files``, of type ``job.prepost``),
when using the :ref:`output check record <output_check_record>`
features, duplicate files created by different tests/variants will
be consolidated into unique files.

Bug Fixes
=========

* The HTML plugin now correctly shows the date for tests that were
never executed because of interrupted jobs.

* A temporarily workaround for a stack overflow problem in Python 3.7
has been addressed.

* The pict plugin (a varianter implementaion) now properly yields the
variants paths as a list.

* A Python 3 related fix to mod:`avocado.utils.software_manager`, that
was using Python 2 ``next`` on ``get_source``.

* A Python 3 related fix to the docker plugin, that wasn't caught
earlier.

Utility APIs
============

* :mod:`avocado.utils.partition` now allows ``mkfs`` and ``mount``
flags to be set.

* :func:`avocado.utils.cpu.get_cpu_vendor_name` now returns the CPU
vendor name for POWER9.

* :mod:`avocado.utils.asset` now allows a given location, as well as a
list, to be given, simplifying the most common use case.

* :meth:`avocado.utils.process.SubProcess.stop` now supports setting
a timeout. Please refer to the documentation for the important
details on its behavior.

* :mod:`avocado.utils.memory` now properly handles hugepages for POWER
platform.

Internal Changes
================

* Removal of the ``stevedore`` library dependency (previously used for
the dispatcher/plugins infrastructure).

* ``make check`` now runs selftests using the experimental N(ext)
Runner.

* Formal support for Python 3.7, which is now on our CI checks,
documentation and module information.

* The Yaml to Mux plugin now uses a safe version of the Yaml loader,
so that the execution of arbitrary Python code from Yaml input is
now no longer possible.

* Codecov coverage reports for have been enabled for Avocado, and
can be seen on every pull request.

* New tests have been added to many of the optional plugins.

* Various pylint compliance improvements, including w0231, w0235,
w0706, w0715 and w0221.

* Avocado's selftests now use tempfile.TemporaryDirectory instead of
mkdtemp and shutil.rmtree.

* :class:`avocado.core.job.Job` instantiation now takes a ``config``
dictionary parameter, instead of a ``argparse.Namespace`` instance,
and keeps it in a ``config`` attribute.

* :class:`avocado.core.job.Job` instances don't have a ``references``
attribute anymore. That information is available in the ``config``
attribute, that is, ``myjob.config['references']``.

* Basic checks for Fedora and RHEL 8 using Cirrus CI have been added,
and will be shown on every pull request.

For more information, please check out the complete
`Avocado changelog
<https://github.com/avocado-framework/avocado/compare/70.0...71.0>`_.

The major changes introduced on this version are listed below.
- Support for running tests in Docker container. Now, in addition to
running tests on a (libvirt based) Virtual Machine or on a remote host,
you can now run tests in transient Docker containers. The usage is as
simple as::

$ avocado run mytests.py --docker ldoktor/fedora-avocado

The container will be started, using `ldoktor/fedora-avocado` as
the image. This image contains a Fedora based system with Avocado
already installed, and it's provided at the official Docker hub.
- Introduction of the "Fail Fast" feature.

By running a job with the `--failfast` flag, the job will be
interrupted after the very first test failure. If your job only
makes sense if it's a complete PASS, this feature can save you a lot
of time.
- Avocado supports replaying previous jobs, selected by using their
Job IDs. Now, it's also possible to use the special keyword
`latest`, which will cause Avocado to rerun the very last job.
- Python's standard signal handling is restored for SIGPIPE, and thus
for all tests running on Avocado.

In previous releases, Avocado introduced a change that set the
default handler to SIGPIPE, which caused the application to be
terminated. This seemed to be the right approach when testing how
the Avocado app would behave on broken pipes on the command line,
but it introduced side effects to a lot of Python code. Instead of
exceptions, the affected Python code would receive the signal themselves.

This is now reverted to the Python standard, and the signal behavior
of Python based tests running on Avocado should not surprise anyone.
- The project release notes are now part of the official
documentation. That means that users can quickly find when a given
change was introduced.

Together with those changes listed, a total of 38 changes made into
this release. For more information, please check out the complete
`Avocado changelog
<https://github.com/avocado-framework/avocado/compare/38.0...39.0>`_.

Sprint Theme

After so much love that we had on the previous version, let's twist
things a bit with an antagonist title. Info on this pretty good movie
by Tarantino can be found at:

http://www.imdb.com/title/tt3460252/?ref_=nm_flmg_wr_2

https://www.youtube.com/watch?v=6_UI1GzaWv0

The story line:

In the dead of a Wyoming winter, a bounty hunter and his prisoner
find shelter in a cabin currently inhabited by a collection of
nefarious characters.

Release Meeting

The Avocado release meetings are now open to the community via
Hangouts on Air. The meetings are recorded and made available on the
`Avocado Test Framework YouTube channel
<https://www.youtube.com/channel/UC-RVZ_HFTbEztDM7wNY4NfA>`_.

For this release, you can watch the meeting on `this link
<https://www.youtube.com/watch?v=GotEH7SmHSw>`_.

Another development cycle has just finished, and our community will
receive this new release containing a nice assortment of bug fixes and
new features.
- The download of assets in tests now allow for an expiration time.
This means that tests that need to download any kind of external
asset, say a tarball, can now automatically benefit from the
download cache, but can also keep receiving new versions
automatically.

Suppose your asset uses an asset named `myproject-daily.tar.bz2`,
and that your test runs 50 times a day. By setting the expire time
to `1d` (1 day), your test will benefit from cache on most runs, but
will still fetch the new version when the the 24 hours from the
first download have passed.

For more information, please check out the documentation on the
`expire` parameter to the `fetch_asset()` method[1].
- Environment variables can be propagated into tests running on remote
systems. It's a known fact that one way to influence application behavior,
including test, is to set environment variables. A command line such as::


$ MYAPP_DEBUG=1 avocado run myapp_test.py


Will work as expected on a local system. But Avocado also allows
running tests on remote machines, and up until now, it has been
lacking a way to propagate environment variables to the remote
system.

Now, you can use::


$ MYAPP_DEBUG=1 avocado run --env-keep MYAPP_DEBUG \
--remote-host test-machine myapp_test.py

- The plugin interfaces have been moved into the
`avocado.core.plugin_interfaces` module. This means that plugin
writers now have to import the interface definitions this namespace,
example::


...
from avocado.core.plugin_interfaces import CLICmd

class MyCommand(CLICmd):
...


This is a way to keep ourselves honest, and say that there's no
difference from plugin interfaces to Avocado's core implementation,
that is, they may change at will. For greater stability, one should
be tracking the LTS releases.

Also, it effectively makes all plugins the same, whether they're
implemented and shipped as part of Avocado, or as part of external
projects.
- A contrib script for running kvm-unit-tests. As some people are
aware, Avocado has indeed a close relation to virtualization
testing. Avocado-VT is one obvious example, but there are other
virtualization related test suites can Avocado can run.

This release adds a contrib script that will fetch, download,
compile and run kvm-unit-tests using Avocado's external runner
feature. This gives results in a better granularity than the
support that exists in Avocado-VT, which gives only a single
PASS/FAIL for the entire test suite execution.

For more information, please check out the Avocado changelog[2].

Also, while we focused on Avocado, let's also not forget that
Avocado-VT maintains it's own fast pace of incoming niceties.
- s390 support: Avocado-VT is breaking into new grounds, and now has
support for the s390 architecture. Fedora 23 for s390 has been added
as a valid guest OS, and s390-virtio has been added as a new machine
type.
- Avocado-VT is now more resilient against failures to persist its
environment file, and will only give warnings instead of errors when
it fails to save it.
- An improved implementation of the "job lock" plugin, which prevents
multiple Avocado jobs with VT tests to run simultaneously. Since
there's no finer grained resource locking in Avocado-VT, this is a
global lock that will prevent issues such as image corruption when
two jobs are run at the same time.

This new implementation will now check if existing lock files are
stale, that is, they are leftovers from previous run. If the
processes associated with these files are not present, the stale
lock files are deleted, removing the need to clean them up manually.
It also outputs better debugging information when failures to
acquire lock.

The complete list of changes to Avocado-VT are available on its
changelog[3].

While not officially part of this release, this development cycle saw
the introduction of new tests on our avocado-misc-tests. Go check it
out!

Finally, since Avocado and Avocado-VT are not newly born anymore, we
decided to update information mentioning KVM-Autotest, virt-test on so
on around the web. This will hopefully redirect new users to the
Avocado community and avoid confusion.

Install avocado

Instructions are available in our documentation on how to install
either with packages or from source[4].

Updated RPM packages are be available in the project repos for EPEL 6,
EPEL 7, Fedora 22, Fedora 23 and the newly released Fedora 24.

Please note that on the next release, we'll drop support for Fedora 22
packages.

Happy hacking and testing!

---

[1] http://avocado-framework.readthedocs.io/en/38.0/WritingTests.html
[2] https://github.com/avocado-framework/avocado/compare/37.0...38.0
[3] https://github.com/avocado-framework/avocado-vt/compare/37.0...38.0
[4] http://avocado-framework.readthedocs.io/en/38.0/GetStartedGuide.html#installing-avocado