Benchexec

Latest version: v3.21

Safety actively analyzes 628969 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 5 of 10

2.5.1

This release does not contain any changes to BenchExec itself,
just for a script in the `contrib` directory.

2.5

This release contains only a small improvement of one tool-info module.

2.3

- A complete rewrite of the HTML tables produced by `table-generator`.
The tables are now based on [React](https://reactjs.org/), load much faster, and provide features like pagination, sorting, and more intuitive filters. More information can be found in [PR #477](https://github.com/sosy-lab/benchexec/pull/477).
Thanks bschor for this!
Note that the tables are not usable without JavaScript anymore.
The old kind of HTML tables can still be produced with the command-line flag `--static-table`, but this is deprecated and will be removed in BenchExec 3.0 in January 2020 (cf. [479](https://github.com/sosy-lab/benchexec/issues/479)).
- Recursively clean up cgroups after a run.
This enables nesting `runexec` in itself, but only if `--full-access-dir /sys/fs/cgroup` is passed to the outer `runexec`, which means that the processes in the outer container have full access to the cgroup hierarchy and could use this to circumvent resource limits.
- `benchexec` filters the tasks to execute depending on the expected verdict, if `<propertyfile expectedverdict="...">` in used the benchmark definition.
- BenchExec now stores a timestamp for the start time of each run, and timestamps for start and end of reach run set.
- `benchexec` will store arbitrary user-defined text as benchmark description together with the results if specified with `benchexec --description-file ...`.
- Support for execution on Python 3.8.
- Fix crash in `runexec` if the tool's stdout/stderr contain invalid UTF-8.
- Fix hanging `benchexec` in container mode if tool cannot be executed (e.g., if executable is missing).
- New tool-info modules and updates for SV-COMP'20 and Test-Comp'20.

2.2

This release fixes two security issues, all users are encouraged to update:

- Since BenchExec 2.1, the setup of the container for the tool-info module (which was added in BenchExec 1.20) could silently fail, for example if user namespaces are disabled on the system. In this case the tool-info module would be executed outside of the container.
Run execution was not affected.
- The kernel offers a keyring feature for storage of keys related to features like Kerberos and ecryptfs. Before Linux 5.2, there existed one keyring per user, and BenchExec did not prevent access from the tool inside the container to the kernel keyring of the user who started BenchExec.
Now such accesses are forbidden (on all kernel versions) using [seccomp](http://man7.org/linux/man-pages/man2/seccomp.2.html) if [libseccomp2](https://github.com/seccomp/libseccomp) is installed, which should the case on any standard distribution.
Note that seccomp filters do have a slight performance impact and could prevent some binaries on exotic architectures from working. In such a case please file a [bug report](https://github.com/sosy-lab/benchexec/issues/new).

2.1

Not secure
benchexec` can now partition the Level 3 cache of the CPU for parallel runs and measure cache usage and memory bandwidth, at least on some Intel CPUs and if the [pqos](https://github.com/intel/intel-cmt-cat/tree/master/pqos) and [pqos_wrapper](https://gitlab.com/sosy-lab/software/pqos-wrapper) are installed. More information is in the [documentation](https://gitlab.com/sosy-lab/software/pqos-wrapper/wikis/home).

Furthermore, some error messages for systems without container support were improved.

2.0

Not secure
This release does not add new features compared to BenchExec 1.22, but removes several deprecated features and brings several other backwards-incompatible changes to make BenchExec more consistent and user-friendly:

- Support for Python 3.2 and 3.3 is removed, the minimal Python version is now 3.4.
Additionally, `runexec`/`RunExecutor` continue to support Python 2.7 [until end of 2019](https://github.com/sosy-lab/benchexec/issues/438).
- Support for running benchmarks as a different user with `sudo` is removed (parameters `--user`/`--users`).
Use container mode as better method for isolating runs.
- [Container mode](doc/container.md) is enabled by default.
It can be disabled with `--no-container`, but this decreases reliability of benchmarking.
- If the `cpuacct` cgroup is not available, CPU-time measurements and limits are not supported.
- Either container mode or the `freezer` cgroup are required to ensure protection against fork bombs.
- [Niceness](http://man7.org/linux/man-pages/man2/nice.2.html) of benchmarked process is not changed, previously it was increased by 5.
- Changes to input of `benchexec`:
- The memory limit given to `benchexec` requires an explicitly specified unit.
- Support for `<test>` tags, `<sourcefiles>` tags, and variables named `${sourcefile_*}` removed from benchmark definitions.
Use `<rundefinition>`, `<tasks>`, and `${inputfile_*}` instead.
- Variables named `${taskdef_*}` are defined only if task-definition files are used, and variables named `${inputfile_*}` only otherwise.
- Changes to `table-generator`:
- A column named `memUsage` is automatically renamed to `memory`.
- A column named `memory` is automatically converted to Megabytes.
Both conversions are only applied if no `<column>` tags are used.
- Changes to [run-result data](doc/run-results.md):
- In case of aborted or failed runs, no dummy results (e.g., `cputime` of 0s) are present.
- The memory results of `benchexec` are named `memory`, not `memUsage`.
- Memory results have the unit `B` explicitly specified.
Furthermore, units are present in all attributes of the result XML files where they were still missing.
- Result item `exitcode` is removed, only `RunExecutor.execute_run()` still returns it, but as an object instance instead of an `int`.
Use `returnvalue` and `exitsignal` instead.
- Module `benchexec.test_tool_wrapper` is removed, use `benchexec.test_tool_info` instead.
- BenchExec (both `benchexec` and `runexec`) terminates itself cleanly after aborting all runs if it receives one of the signals `SIGTERM`, `SIGINT` (Ctrl+C), or `SIGQUIT`.

Additionally, this release adds a fix for the container that is used since BenchExec 1.20 for the tool-info module. In this container, the environment variable `HOME` did not point to `/home/benchexec` as expected but to the user's real home directory. This broke tools like Ultimate if the `/home` was configured to be hidden or read-only.

Furthermore, we declare the following features deprecated and plan on removing them for [BenchExec 3.0](https://github.com/sosy-lab/benchexec/milestone/8), which is expected to be released in January 2020:

- Support for Python 2.7 and 3.4 (cf. [438](https://github.com/sosy-lab/benchexec/issues/438))
- Support for checking correctness of run results and computing scores if task-definition files are *not* used (cf. [439](https://github.com/sosy-lab/benchexec/issues/439))

Please respond in the respective issue if one of these deprecations is a problem for you.

Page 5 of 10

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.