lnt.testing – Test Data Creation

Utilities for working with the LNT test format.

Clients can easily generate LNT test format data by creating Report objects for the runs they wish to submit, and using Report.render to convert them to JSON data suitable for submitting to the server.

class lnt.testing.Machine(name, info={}, report_version=1)

Information on the machine the test was run on.

The info dictionary can be used to describe additional information about the machine, for example the hardware resources or the operating environment.

Machines entries in the database are uniqued by their name and the entire contents of the info dictionary.

check()

Check object members are adequate to generate an LNT json report file of the version specified at construction when rendering that instance.

render()

Return info from this instance in a dictionary that respects the LNT report format in the version specified at construction when printed as json.

class lnt.testing.Report(machine, run, tests, report_version=1)

Information on a single testing run.

In the LNT test model, every test run should define exactly one machine and run, and any number of test samples.

check()

Check that object members are adequate to generate an LNT json report file of the version specified at construction when rendering that instance.

render(indent=4)

Return a LNT json report file format of the version specified at construction as a string, where each object is indented by indent spaces compared to its parent.

update_report(new_tests_samples, end_time=None)

Add extra samples to this report, and update the end time of the run.

class lnt.testing.Run(start_time=None, end_time=None, info={}, report_version=1)

Information on the particular test run.

At least one parameter must be supplied and is used as ordering among several runs. When generating a report in format 1 or earlier, both start_time and end_time are used for that effect and the current date is used if their value is None.

As with Machine, the info dictionary can be used to describe additional information on the run. This dictionary should be used to describe information on the software-under-test that is constant across the test run, for example the revision number being tested. It can also be used to describe information about the current state which could be useful in analysis, for example the current machine load.

check()

Check object members are adequate to generate an LNT json report file of the version specified at construction when rendering that instance.

render()

Return info from this instance in a dictionary that respects the LNT report format in the version specified at construction when printed as json.

update_endtime(end_time=None)

Update the end time of this run.

class lnt.testing.TestSamples(name, data, info={}, conv_f=<class 'float'>)

Information on a given test and its associated samples data.

Samples data must all relate to the same metric. When several metrics are available for a given test, the convention is to have one TestSamples per metric and to encode the metric into the name, e.g. Benchmark1.exec. The server automatically creates test database objects whenever a new test name is seen. TestSamples should only be used to generate report in version 1 or earlier of LNT JSON report file format.

Test names are intended to be a persistent, recognizable identifier for what is being executed. Currently, most formats use some form of dotted notation for the test name, and this may become enshrined in the format in the future. In general, the test names should be independent of the software-under-test and refer to some known quantity, for example the software under test. For example, ‘CINT2006.403_gcc’ is a meaningful test name.

The test info dictionary is intended to hold information on the particular permutation of the test that was run. This might include variables specific to the software-under-test . This could include, for example, the compile flags the test was built with, or the runtime parameters that were used. As a general rule, if two test samples are meaningfully and directly comparable, then they should have the same test name but different info paramaters.

The report may include an arbitrary number of samples for each test for situations where the same test is run multiple times to gather statistical data.

render()

Return info from this instance in a dictionary that respects the LNT report format in the version specified at construction when printed as json.