Avocado-framework

Hi guys, we're up to a new avocado release! It's basically a bugfix release, with a few usability tweaks.
- The avocado human output received some extra tweaks. Here's how it looks now:

$ avocado run passtest
JOB ID : f186c729dd234c8fdf4a46f297ff0863684e2955
JOB LOG : /home/lmr/avocado/job-results/job-2015-08-15T08.09-f186c72/job.log
TESTS : 1
(1/1) passtest.py:PassTest.test: PASS (0.00 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
JOB HTML : /home/lmr/avocado/job-results/job-2015-08-15T08.09-f186c72/html/results.html
TIME : 0.00 s
- Bugfixes. You may refer to [1] for the full list of 58 commits.

Changes in avocado-vt:
- Bugfixes. In particular, a lot of issues related to --vt-type libvirt were fixed and now that backend is fully functional.

Avocado 0.27.0 released!

Hi guys, here I am, announcing yet another avocado release! The most exciting news for this release is that our avocado-vt plugin was merged with the virt-test project. The avocado-vt plugin will be very important for QEMU/KVM/Libvirt developers, so the main avocado received updates to better support the goal of having a good quality avocado-vt.

Changes in avocado:
- The avocado human output received some tweaks and it's more compact, while still being informative. Here's an example:


JOB ID : f2f5060440bd57cba646c1f223ec8c40d03f539b
JOB LOG : /home/user/avocado/job-results/job-2015-07-27T17.13-f2f5060/job.log
JOB HTML : /home/user/avocado/job-results/job-2015-07-27T17.13-f2f5060/html/results.html
TESTS : 1
(1/1) passtest.py:PassTest.test: PASS (0.00 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TIME : 0.00 s

- The avocado test loader was refactored and behaves more consistently in different test loading scenarios.
- The `utils` API received new modules and functions:
- NEW avocado.utils.cpu: APIs related to CPU information on linux boxes [1]
- NEW avocado.utils.git: APIs to clone/update git repos [2]
- NEW avocado.utils.iso9660: Get information about ISO files [3]
- NEW avocado.utils.service: APIs to control services on linux boxes (systemv and systemd) [4]
- NEW avocado.utils.output: APIs that help avocado based CLI programs to display results to users [5]
- UPDATE avocado.utils.download: Add url_download_interactive
- UPDATE avocado.utils.download: Add new params to get_file
- Bugfixes. You may refer to [6] for the full list of 64 commits.

Changes in avocado-vt:
- Merged virt-test into avocado-vt. Basically, the virt-test core library (virttest) replaced most uses of autotest by equivalent avocado API calls, and its code was brought up to the virt-test repository [7]. This means, among other things, that you can simply install avocado-vt through RPM and enjoy all the virt tests without having to clone another repository manually to bootstrap your tests. More details about the process will be sent on an e-mail to the avocado and virt-test mailing lists. Please go to [8] for instructions on how to get started with all our new tools.

See you in a couple of weeks for our next release! Happy testing!

The avocado development team

Links

[1] http://avocado-framework.readthedocs.org/en/latest/api/utils/avocado.utils.html#module-avocado.utils.cpu
[2] http://avocado-framework.readthedocs.org/en/latest/api/utils/avocado.utils.html#module-avocado.utils.git
[3] http://avocado-framework.readthedocs.org/en/latest/api/utils/avocado.utils.html#module-avocado.utils.iso9660
[4] http://avocado-framework.readthedocs.org/en/latest/api/utils/avocado.utils.html#module-avocado.utils.service
[5] http://avocado-framework.readthedocs.org/en/latest/api/utils/avocado.utils.html#module-avocado.utils.output
[6] https://github.com/avocado-framework/avocado/compare/0.26.0...0.27.0
[7] https://github.com/avocado-framework/avocado-vt/commit/20dd39ef00db712f78419f07b10b8f8edbd19942
[8] http://avocado-vt.readthedocs.org/en/latest/GetStartedGuide.html

Avocado 0.26.0 released!

Hi guys, I'm here to announce avocado 0.26.0. This release was dedicated to
polish aspects of the avocado user experience, such as documentation and
behavior.

Changes
- Now avocado tests that raise exceptions that don't inherit from
`avocado.core.exceptions.TestBaseException` now will be marked as ERRORs.
This change was made to make avocado to have clearly defined test statuses.
A new decorator, `avocado.fail_on_error` was added to let arbitary exceptions
to raise errors, if users need a more relaxed behavior.
- The avocado.Test() utility method skip() now can only be called from inside
the setUp() method. This was made because by definition, if we get to the
test execution step, by definition it can't be skipped anymore. It's important
to keep the concepts clear and well separated if we want to give users a good
experience.
- More documentation polish and updates. Make sure you check out our documentation
website http://avocado-framework.readthedocs.org/en/latest/.
- A number of avocado command line options and help text was reviewed and
updated.
- A new, leaner and mobile friendly version of the avocado website is live.
Please check http://avocado-framework.github.io/ for more information.
- We have the first version of the avocado dashboard! avocado dashboard is the
initial version of an avocado web interface, and will serve as the frontend
to our testing database. You can check out a screenshot here
https://cloud.githubusercontent.com/assets/296807/8536678/dc5da720-242a-11e5-921c-6abd46e0f51e.png

- And the usual bugfixes. You can take a look at the full list of 68 commits here:
https://github.com/avocado-framework/avocado/compare/0.25.0...0.26.0

Avocado 0.25.0 released!

Hi guys, I'm here to announce the newest avocado release, 0.25.0. This is an
important milestone in avocado development, and we would like to invite
you to be a part of the development process, by contributing PRs, testing
and giving feedback on the test runner's usability and new plugins we came
up with.

What to expect

This is the first release aimed for general use. We did our best to deliver
a coherent and enjoyable experience, but keep in mind that it's a young
project, so please set your expectations accordingly. What is expected to work
well:
- Running avocado 'instrumented' tests
- Running arbitrary executables as tests
- Automatic test discovery and run of tests on directories
- xUnit/JSON report

Known Issues
- HTML report of test jobs with multiplexed tests has a minor naming display
issue that is scheduled to be fixed by next release.
- avocado-vt might fail to load if virt-test was not properly bootstrapped.
Make sure you always run bootstrap in the virt-test directory on any
virt-test git updates to prevent the issue. Next release will have more
mechanisms to give the user better error messages on tough to judge
situations (virt-test repo with stale or invalid config files that need
update).

Changes
- The Avocado API has been greatly streamlined. After a long discussion and
several rounds of reviews and planning, now we have a clear separation of
what is intended as functions useful for test developers and plugin/core
developers:
- avocado.core is intended for plugin/core developers. Things are more fluid
on this space, so that we can move fast with development
- avocado.utils is a generic library, with functions we found out to be
useful for a variety of tests and core code alike.
- avocado has some symbols exposed at its top level, with the test API:
- our Test() class, derived from the unittest.TestCase() class
- a main() entry point, similar to unittest.main()
- VERSION, that gives the user the avocado version (eg 0.25.0).
Those symbols and classes/APIs will be changed more carefully, and release
notes will certainly contain API update notices. In other words, we'll be
a lot more mindful of changes in this area, to reduce the maintenance cost
of writing avocado tests.

We believe this more strict separation between the available APIs will help
test developers to quickly identify what they need for test development, and
reduce following a fast moving target, what usually happens when we have a
new project that does not have clear policies behind its API design.
- There's a new plugin added to the avocado project: avocado-vt. This plugin
acts as a wrapper for the virt-test test suite
(https://github.com/autotest/virt-test), allowing people to use avocado to
list and run the tests available for that test suite. This allows people to
leverage a number of the new cool avocado features for the virt tests
themselves:
- HTML reports, a commonly asked feature for the virt-test suite. You can
see a screenshot of what the report looks like here:
https://cloud.githubusercontent.com/assets/296807/7406339/7699689e-eed7-11e4-9214-38a678c105ec.png
- You can run virt-tests on arbitrary order, and multiple instances of a
given test, something that is also currently not possible with the virt
test runner (also a commonly asked feature for the suite.
- System info collection. It's a flexible feature, you get to configure
easily what gets logged/recorded between tests.
- The avocado multiplexer (test matrix representation/generation system)
also received a lot of work and fixes during this
release. One of the most visible (and cool) features of 0.25.0 is the new,
improved --tree representation of the multiplexer file:


$ avocado multiplex examples/mux-environment.yaml -tc
┗━━ run
┣━━ hw
┃ ┣━━ cpu
┃ ┃ ╠══ intel
┃ ┃ ║ → cpu_CFLAGS: -march=core2
┃ ┃ ╠══ amd
┃ ┃ ║ → cpu_CFLAGS: -march=athlon64
┃ ┃ ╚══ arm
┃ ┃ → cpu_CFLAGS: -mabi=apcs-gnu -march=armv8-a -mtune=arm8
┃ ┗━━ disk
┃ ╠══ scsi
┃ ║ → disk_type: scsi
┃ ╚══ virtio
┃ → disk_type: virtio
┣━━ distro
┃ ╠══ fedora
┃ ║ → init: systemd
┃ ╚══ mint
┃ → init: systemv
┗━━ env
╠══ debug
║ → opt_CFLAGS: -O0 -g
╚══ prod
→ opt_CFLAGS: -O2


We hope you find the multiplexer useful and enjoyable.
- If an avocado plugin fails to load, due to factors such as missing
dependencies, environment problems and misconfiguration, in order to notify
users and make them mindful of what it takes to fix the root causes for the
loading errors, those errors are displayed in the avocado stderr stream.

However, often we can't fix the problem right now and don't need
the constant stderr nagging. If that's the case, you can set in your local
config file:


[plugins]
Suppress notification about broken plugins in the app standard error.
Add the name of each broken plugin you want to suppress the notification
in the list. The names can be easily seen from the stderr messages. Example:
avocado.core.plugins.htmlresult ImportError No module named pystache
add 'avocado.core.plugins.htmlresult' as an element of the list below.
skip_broken_plugin_notification = []

- Our documentation has received a big review, that led to a number of
improvements. Those can be seen online
(http://avocado-framework.readthedocs.org/en/latest/), but if you feel so
inclined, you can build the documentation for local viewing, provided that
you have the sphinx python package installed by executing:

$ make -C docs html

Of course, if you find places where our documentation needs
fixes/improvements, please send us a PR and we'll gladly review it.
- As one would expect, many bugs were fixed. You can take a look at the full
list of 156 commits here:
https://github.com/avocado-framework/avocado/compare/0.24.0...0.25.0

We are proud to announce the new Avocado 0.24.0 release! This time, we had made a number of API changes to reflect some good practices in API development, and changes to the multiplexer, that is making large steps towards a production ready state. Let's break down the changes:

Core/General API isolation

In order to avoid user confusion, now the `avocado` API has a clear separation with regards to what it's supposed to be used by test developers, and what is supposed to be used by plugin/core application developers:
- `avocado.core` is restricted to plugin/core app developers
- `avocado` can be used by both test developers and plugin/core app developers

Breakdown of changes:
- `avocado.core.data_dir` -> `avocado.data_dir`
- `avocado.plugins` -> `avocado.core.plugins`
- `avocado.remote` -> `avocado.core.remote`
- `avocado.job` -> `avocado.core.job`
- `avocado.job.main` -> `avocado.main`
- `avocado.loader` -> `avocado.core.loader`
- `avocado.output` -> `avocado.core.output`
- `avocado.loader` -> `avocado.core.loader`
- `avocado.parser` -> `avocado.core.parser`
- `avocado.runner` -> `avocado.core.runner`
- `avocado.sysinfo` -> `avocado.core.sysinfo`
- `avocado.external.gdbmi_parser` -> `avocado.core.gdbmi_parser`
- `avocado.external.spark` -> `avocado.core.spark`
- `avocado.cli.app` -> `avocado.core.app`
- `avocado.exceptions` -> `avocado.core.exceptions`

Multiplexer

The multiplexer has gone through changes on the parameters retrieval API. The new way of retrieving params is through the `get` API, that has the form::


get(key, path, default)


The `avocado.test.Test.params` object is no longer a dictionary, and it's not possible to refer to the values through the shortcut `avocado.test.Test.params.key` anymore.

Example of old usage::


c_file = self.get_data_path(self.params.get('source', 'doublefree.c'))


This statement has to be updated to::


c_file = self.get_data_path(self.params.get('source', default='doublefree.c'))


Otherwise the default `doublefree.c` value will be taken as the retrieval path, then c_file would be `None`.

The multiplexer file format also has a new `!mux` tag, that means to recursively multiplex all nodes below the specified tag. By default now, we don't multiplex until the `!mux` tag is specified. Let's consider the following file::


hw:
cpu:
intel:
cpu_CFLAGS: '-march=core2'
amd:
cpu_CFLAGS: '-march=athlon64'
arm:
cpu_CFLAGS: '-mabi=apcs-gnu -march=armv8-a -mtune=arm8'
disk:
scsi:
disk_type: 'scsi'
virtio:
disk_type: 'virtio'
distro:
fedora:
init: 'systemd'
mint:
init: 'systemv'
env:
debug:
opt_CFLAGS: '-O0 -g'
prod:
opt_CFLAGS: '-O2'


Here, the nodes `cpu`, `disk`, `distro` and `env` are the ones we actually want to combine, while `hw` is just for organization purposes (we don't want to combine `hw` with `distro`, really).

New version, with `!mux`:


hw:
cpu: !mux
intel:
cpu_CFLAGS: '-march=core2'
amd:
cpu_CFLAGS: '-march=athlon64'
arm:
cpu_CFLAGS: '-mabi=apcs-gnu -march=armv8-a -mtune=arm8'
disk: !mux
scsi:
disk_type: 'scsi'
virtio:
disk_type: 'virtio'
distro: !mux
fedora:
init: 'systemd'
mint:
init: 'systemv'
env: !mux
debug:
opt_CFLAGS: '-O0 -g'
prod:
opt_CFLAGS: '-O2'


This way we can keep using `hw` as a organization label, not interfering with our final purpose of combining only `cpu`, `disk`, `distro` and `env`. The variants generated with the new tag arrangement are:


Variant 1: /hw/cpu/intel, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 2: /hw/cpu/intel, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 3: /hw/cpu/intel, /hw/disk/scsi, /distro/mint, /env/debug
Variant 4: /hw/cpu/intel, /hw/disk/scsi, /distro/mint, /env/prod
Variant 5: /hw/cpu/intel, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 6: /hw/cpu/intel, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 7: /hw/cpu/intel, /hw/disk/virtio, /distro/mint, /env/debug
Variant 8: /hw/cpu/intel, /hw/disk/virtio, /distro/mint, /env/prod
Variant 9: /hw/cpu/amd, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 10: /hw/cpu/amd, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 11: /hw/cpu/amd, /hw/disk/scsi, /distro/mint, /env/debug
Variant 12: /hw/cpu/amd, /hw/disk/scsi, /distro/mint, /env/prod
Variant 13: /hw/cpu/amd, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 14: /hw/cpu/amd, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 15: /hw/cpu/amd, /hw/disk/virtio, /distro/mint, /env/debug
Variant 16: /hw/cpu/amd, /hw/disk/virtio, /distro/mint, /env/prod
Variant 17: /hw/cpu/arm, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 18: /hw/cpu/arm, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 19: /hw/cpu/arm, /hw/disk/scsi, /distro/mint, /env/debug
Variant 20: /hw/cpu/arm, /hw/disk/scsi, /distro/mint, /env/prod
Variant 21: /hw/cpu/arm, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 22: /hw/cpu/arm, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 23: /hw/cpu/arm, /hw/disk/virtio, /distro/mint, /env/debug
Variant 24: /hw/cpu/arm, /hw/disk/virtio, /distro/mint, /env/prod


The tree representation also changed to print the `!mux` points of branching, to further help the user to understand the structure of the tree::


$ scripts/avocado multiplex --tree examples/mux-environment.yaml
Config file tree structure:

/-intel
|
/cpu-<>--amd
| |
/hw \-arm
| |
| | /-scsi
| \disk-<>
| \-virtio
-|
| /-fedora
|-distro-<>
| \-mint
|
| /-debug
\env-<>
\-prod


The more detailed breakdown of the 145 commits can be seen here:

https://github.com/avocado-framework/avocado/compare/0.23.0...0.24.0

Happy testing!

Hello everyone! I'm here to announce the new Avocado 0.23.0 release! This time around, we have a few changes worth of note:
- Now avocado looks more like nose and other unittest based test runners: You can define a number of test methods prefixed with `test`, and the runner will simply run them. Also, for compatibility with the base unittest class, now `setup` and `cleanup` are `setUp` and `tearDown`.

Let's see how an old avocado test looked like:


!/usr/bin/python
from avocado import test
from avocado import job


class OldTest(test.Test):

def setup(self):
self.hello = "Hi there!"

def action(self):
self.assertEqual(self.hello, "Hi there!")

def cleanup(self):
self.log.debug('Cleanup')

if __name__ == '__main__':
job.main()


The new avocado tests look like this:


!/usr/bin/python
from avocado import test
from avocado import job


class MultipleTests(test.Test):

def setUp(self):
self.hello = "Hi there!"

def test_hello(self):
self.assertEqual(self.hello, "Hi there!")

def testIdentity(self):
self.assertTrue(1, 1)

def tearDown(self):
self.log.debug('Cleanup')

if __name__ == '__main__':
job.main()

- `avocado run`: Simplified command line options, cutting off a number of rarely used options and moving them to config file entries
- `avocado run`: Added the `--job-timeout` option, that allows users to specify the maximum time an avocado test job might take.
- `avocado run`: Consolidated the options `--remote-timeout` and `--vm-timeout` into `--job-timeout`.
- Sysinfo feature: The sysinfo feature went through refactoring and simplification - Now it is possible to specify what information from the system is to be collected through config files

See how to configure your collectibles:


[sysinfo.collectibles]
File with list of commands that will be executed and have their output collected
commands = /etc/avocado/sysinfo/commands
File with list of files that will be collected verbatim
files = /etc/avocado/sysinfo/files
File with list of commands that will run alongside the job/test
profilers = /etc/avocado/sysinfo/profilers


The default `commands` file looks like:


df -mP
dmesg -c
uname -a
lspci -vvnn
gcc --version
ld --version
mount
hostname
uptime
dmidecode
ifconfig -a
brctl show
ip link
numactl --hardware show
lscpu
fdisk -l

- Multiplexer: A new params retrieval system is in place. Now, instead of `self.params.key` you must use the `self.params.get(key, path, default)` API. A compatibility layer with the old dict structure was made, but we encourage you to migrate your param retrieval calls from `self.params.key` or `self.params['key'] to`self.params.get('key').

The full list of commits can be seen here:

https://github.com/avocado-framework/avocado/compare/0.21.0...0.23.0

Happy testing!