Skip to content

Latest commit

 

History

History
391 lines (248 loc) · 10.7 KB

bash-test-utils.md

File metadata and controls

391 lines (248 loc) · 10.7 KB

bash-test-utils

ABOUT

bash-test-utils -- support writing functional tests

It provides utility functions and facilities to write tests in baremetal and Xen/KVM contexts (host and guest support) and was polished to work with i686, x86_64, and ARM under lots of major Linux distributions (Debian, Ubuntu, RHEL, SLES, Slackware, Fedora, openSUSE).

It can be run as normal bash script or indirectly via the "prove" command, a standard Linux tool to run test scripts.

SYNOPSIS

most simple usage

  • script:
 #! /bin/bash
 . ./bash-test-utils
 # your tests here, using utility functions
 done_testing

Explanation:

  • First it imports utility functions to be used by your script
  • Then you write your tests
  • At the end it calls the actual printing of test results

Run it via 'prove' (a standard test utility)

The tool prove is a standard tool available in every Linux distribution. Use it this way:

 $ prove ./trivial-example-1.sh
 ./trivial-example-1.sh .. ok
 All tests successful.
 Files=1, Tests=5, 20 wallclock secs
 Result: PASS

Explanation:

  • "prove" is an existing standard tool
  • it prints success statistics
  • no report sending happens
  • is meant for manual developing/testing

UTILITY FUNCTIONS

Import utility functions at the beginning of the script via

 . ./bash-test-utils

Then you have several functions available.

Expressing test results

ok ARG1 "some description"

Evaluates the first argument with Shell boolean semantics (0 is true) and appends a corresponding TAP line.

See also "require_ok" below.

negate_ok ARG1 "some description"

Evaluates the first argument with Shell inverse boolean semantics (0 is false) and appends a corresponding TAP line.

append_tap "ok - some description"

Appends a complete TAP line where you have taken care for the "ok"/"not ok" at the beginning.

append_comment "foo bar baz"

Appends a complete comment line starting with "#". It appears directly after the last added TAP line so it can be used for diagnostics.

append_tapdata "key: value"

Appends a key:value line at the final tapdata YAML block. The key must start with letter and consist of only alphanum an underscore.

require_* functions

All require_* functions check for something and gracefully exit the script if the requirement is not fulfilled. Use this to allow the script to run everywhere without polluting results with false negatives.

require_ok ARG1 "some description"

Evaluates the first argument with Shell boolean semantics (0 is true) and appends a corresponding TAP line.

If it reports "not ok" the script gracefully exits.

require_amd_family_range [ MIN [ MAX ] ]

Ensures vendor is AMD and cpu family from /proc/cpuinfo is in a minimum/maximum range. If you don't specify MIN it defaults to 0. If you don't specify MAX it defaults to MIN.

require_cpufeature "foo"

Verifies that the string "foo" occurs in /proc/cpuinfo flags section.

require_kernel_config "CONFIG_FOO"

Verify that regex "^CONFIG_FOO=." occurs in /proc/config.gz or /boot/config/$(uname -r).

require_program "foo"

Verify that the program "foo" is available.

Use it to declare external programs you call, like "awk", "bc", "perl", etc.

require_l3cache

Verify that L3 cache is available (checked in /sys/devices/system/cpu/cpu0/cache/index3).

require_netcat

Verify that a variant of netcat (netcat, nc) is available.

require_root

Verify that the user executing the script is root (UID 0).

require_crit_level N

Verify that the criticality level N of the script is allowed to be run, which is controlled by environment variable CRITICALITY. See "has_crit_level" for meaning of criticality levels.

require_cpb_disabled

Enables cpufreq and disables boosting. In case of errors the calling test is skipped.

require_cpufreq_enabled

Enables cpufreq and also core boosting. In case of errors the calling test is skipped.

require_kernel_release_min

Verify that the current LK release is at least the required version number.

require_running_in_xen_guest

Verify if we are in a Xen guest.

require_running_in_kvm_guest

Verify if we are in a KVM guest.

require_running_in_virtualized_guest

Verify if we are in a virtualized guest (Xen or KVM).

request_* functions

All request_* functions try to enable something and if that fails mark it as #TODO but continue the test. It's kind of an "uncritical require_*".

request_cpb_disabled

Enables cpufreq and disables boosting. The result will be reported and returned. Errors are marked as TODO.

request_cpufreq_enabled

Enables cpufreq and also core boosting. The result will be reported and returned. Errors are marked as TODO.

Misc auxiliary functions

These are really just utilities to help you but they don't influence the behaviour like the require_* functions do.

has_l3cache

Returns 0 (shell TRUE) if L3 cache is available (checked in /sys/devices/system/cpu/cpu0/cache/index3).

has_cpufeature "foo"

Returns 0 (shell TRUE) if string "foo" occurs in /proc/cpuinfo flags section.

has_kernel_release_min

Returns 0 (shell TRUE) if the current LK release is at least the required version number.

has_kernel_config "CONFIG_FOO"

Returns 0 (shell TRUE) if regex "^CONFIG_FOO=." occurs in /proc/config.gz or /boot/config/$(uname -r).

has_program "foo"

Return 0 (shell TRUE) if the program "foo" is available.

has_crit_level N

Checks whether the criticality level N of the script is allowed to be run, which is controlled by environment variable CRITICALITY.

The criticality levels are defined as this:

  • 0: not critical
  • 1: read sysfs/debugfs/proc files
  • 2: read HW (MSRs/Northbridge)
  • 3: write sysfs/debugfs/proc files
  • 4: write HW (MSRs/Northbridge) or potentially crash the machine

get_vendor

Prints vendor "AMD" or "Intel" from /proc/cpuinfo.

vendor_intel

Returns 0 (shell TRUE) if vendor is Intel.

vendor_amd

Returns 0 (shell TRUE) if vendor is AMD.

get_random_number [ MAX ]

Prints a random integer between 0 and MAX (default 32768).

get_cpu_family

Print cpu family from /proc/cpuinfo.

get_cpu_family_hex

Print cpu family from /proc/cpuinfo in hex syntax (0x...).

cpu_family_min [ MINFAMILY ]

Returns 0 (shell TRUE) if cpu family from /proc/cpuinfo is greater or equal to MINFAMILY. Defaults to 0.

cpu_family_max [ MAXFAMILY ]

Returns 0 (shell TRUE) if cpu family from /proc/cpuinfo is less or equal to MAXFAMILY. Defaults to 999.

get_first_file [ LIST OF FILENAMES ]

Goes through all specified filenames and prints the first one that exists and is readable.

is_element_in_list WORD "SPACE SEPARATED LIST OF WORDS"

Returns 0 (shell TRUE) if WORD appears in "SPACE SEPARATED LIST OF WORDS". Remember the usual shell quoting rules.

get_kernel_release

Prints the kernel version number from uname -r.

is_running_in_xen_guest

Check if we are in a Xen guest.

is_running_in_kvm_guest

Check if we are in a KVM guest.

is_running_in_virtualized_guest

Check if we are in a virtualized guest (Xen or KVM).

upload_file

Mark a file for upload which in fact makes it part of the results.tgz file if CREATE_RESULTS_FILE is set to 1.

INFLUENCING BEHAVIOUR

You can

  • use environment variables to provide more content
  • provide commandline params that "Do What I Mean"
  • define hooks (functions) to be called

Environment Variables

These variables are expected to be set inside the script to declare meta information or influence behaviour:

  • TAP[*] - Array of TAP lines
  • TAPDATA[*] - Array of YAML lines that contain data in TAP
  • OUTPUT[*] - Array of additional output lines
  • SUITENAME - alternative suite name instead of $0
  • SUITEVERSION - alternative suite version
  • KEYWORDS - space separated keywords to influence suite name
  • OSNAME - alternative OS description
  • CHANGESET - alternative changeset
  • HOSTNAME - alternative hostname
  • TICKETURL - relevant URL in used ticket system (Bugzilla)
  • WIKIURL - relevant URL in used wiki
  • PLANNINGID - relevant task planning id (MS Project, TaskJuggler)
  • REQUIRES_GENERATE_TAP - if "1" then require_* functions generate additional "ok" line on success
  • PROVIDE_EXITCODE - test_done() will return with exitcode set to number of failed tests (max 253).
  • CREATE_RESULTS_FILE - Create a .tgz of all results instead of printing to STDOUT. The tgz contains a file with the TAP output and a file containing all files collected with upload_file() inside the test script.
  • TESTRESULTSFILE - The name of the .tgz file to be created on CREATE_RESULTS_FILE. Defaults to testresults.tgz.

External environment variables

These variables are expected to be set from outside of the script to influence behaviour:

  • EXIT_ON_SKIPALL - if "1" then on skip_all we immediately exit with 254 and do not send a report
  • CRITICALITY - Sets the allowed maximal criticality level. Scripts with higher level do skip_all

Function hooks

These are optional shell functions that you can define in your test script and that will be called in certain places.

function main_end_hook()

  • executed at the end of autoreport's main()
  • all stdout will be part of the report

Include hooks

You can have a directory bash-test-utils.hooks/FOO/ (where the first part is fix, but FOO is yours) to extend bash-test-utils with personal behaviour behaviour. This subdir is completely optional. Feel free to write your own extensions and ask for more hooks.

Activate the hooks by setting the hooks subdirectory before including the bash-test-utils:

 export TESTUTIL_HOOKS=FOO
 . ./bash-test-utils

This will then look into ./bash-test-utils.hooks/FOO/* files and include (source) them if existing.

The files are named after the place where they are called. See the example ./bash-test-utils.hooks/tapper/* which adds Tapper specific behaviour to bash-test-utils. Following hooks can be used:

file 'functions'

Included after all functions are declared. Use it to declare additional functions or patch existing ones.

file 'prepare_information'

Included at the end of the prepare_information() function. Use it to gather additional meta information.

file 'suite_meta'

Included at the end of suite_meta() function. Use it to print out additional meta information.