Importing Data

Importing Data in a Text File

The LNT importreport command will import data in a simple text file format. The command takes a space separated key value file and creates an LNT report file, which can be submitted to a LNT server. Example input file:

foo.exec 123
bar.size 456
foo/bar/baz.size 789

The format is “test-name.metric”, so exec and size are valid metrics for the test suite you are submitting to.

Example:

echo -n "foo.exec 25\nbar.score 24.2\nbar/baz.size 110.0\n" > results.txt
lnt importreport --machine=my-machine-name --order=1234 --testsuite=nts results.txt report.json
lnt submit http://mylnt.com/db_default/submitRun report.json

LNT Report File Format

The lnt importreport tool is an easy way to import data into LNTs test format. You can also create LNTs report data directly for additional flexibility.

First, make sure you’ve understood the underlying Concepts used by LNT.

{
    "format_version": "2",
    "machine": {
        "name": _String_      // machine name, mandatory
        (_String_: _String_)* // optional extra info
    },
    "run": {
        ("start_time": "%Y-%m-%dT%H:%M:%S",)? // optional, ISO8061 timestamp
        ("end_time": "%Y-%m-%dT%H:%M:%S",)?   // optional, ISO8061 timestamp, can equal start_time if not known.
        (_String_: _String_,)* // optional extra info about the run.
        // At least one of the extra fields is used as ordering and is
        // mandatory. For the 'nts' and 'Compile' schemas this is the
        // 'llvm_project_revision' field.
    },
    "tests": [
        {
            "name": _String_,   // test name mandatory
            (_String_: _Data_)* // List of metrics, _Data_ allows:
                                // number, string or list of numbers
        }+
    ]
}

A concrete small example is

{
    "format_version": "2",
    "machine": {
       "name": "LNT-AArch64-A53-O3__clang_DEV__aarch64",
       "hardware": "HAL 9000"
    },
    "run": {
       "end_time": "2017-07-18T11:28:23.991076",
       "start_time": "2017-07-18T11:28:33.00000",
       "llvm_project_revision": "265649",
       "compiler_version": "clang 4.0"
    },
    "tests": [
       {
           "name": "benchmark1",
           "execution_time": [ 0.1056, 0.1055 ],
           "hash": "49333a87d501b0aea2191830b66b5eec"
       },
       {
           "name": "benchmark2",
           "compile_time": 13.12,
           "execution_time": 0.2135,
           "hash": "c321727e7e0dfef279548efdb8ab2ea6"
       }
    ]
}

Given how simple it is to make your own results and send them to LNT, it is common to not use the LNT client application at all, and just have a custom script run your tests and submit the data to the LNT server. Details on how to do this are in lnt.testing

Default Test Suite (NTS)

The default test-suite schema is called NTS. It was originally designed for nightly test runs of the llvm test-suite. However it should fit many other benchmark suites as well. The following metrics are supported for a test:

  • execution_time: Execution time in seconds; lower is better.

  • score: Benchmarking score; higher is better.

  • compile_time: Compiling time in seconds; lower is better.

  • hash: A string with the executable hash (usually md5sum of the stripped binary)

  • mem_bytes: Memory usage in bytes during execution; lower is better.

  • code_size: Code size (usually the size of the text segment) in bytes; lower is better.

  • execution_status: A non zero value represents an execution failure.

  • compile_status: A non zero value represents a compilation failure.

  • hash_status: A non zero value represents a failure computing the executable hash.

The run information is expected to contain this:

  • llvm_project_revision: The revision or version of the compiler used for the tests. Used to sort runs.

Custom Test Suites

LNT test suite schemas define which metrics can be tracked for a test and what extra information is known about runs and machines. You can define your own test suite schemas in a yaml file. The LNT administrator has to place (or symlink) this yaml file into the servers schema directory.

Example:

format_version: '2'
name: my_suite
metrics:
- name: text_size
  bigger_is_better: false
  type: Real
- name: data_size
  bigger_is_better: false
  type: Real
- name: score
  bigger_is_better: true
  type: Real
- name: hash
  type: Hash
run_fields:
- name: llvm_project_revision
  order: true
machine_fields:
- name: hardware
- name: os

  • LNT currently supports the following metric types:

    • Real: 8-byte IEEE floating point values.

    • Hash: String values; limited to 256, sqlite is not enforcing the limit.

    • Status: StatusKind enum values (limited to ‘PASS’, ‘FAIL’, ‘XFAIL’ right now).

  • You need to mark at least 1 of the run fields as order: true so LNT knows how to sort runs.

  • Note that runs are not be limited to the fields defined in the schema for the run and machine information. The fields in the schema merely declare which keys get their own column in the database and a prefered treatment in the UI.