I'm collecting here links to some open-source benchmarks, or benchmark suites, that might add value to our test suite. https://github.com/flwende/simd_benchmarks https://github.com/tbepler/PWM-benchmarking https://github.com/pamela-project/slambench https://github.com/stream-benchmarking/firehose (http://firehose.sandia.gov/) https://github.com/hiraditya/std-benchmark https://openbenchmarking.org/suite/pts/cpu http://impact.crhc.illinois.edu/parboil/parboil.aspx https://github.com/breagen/MachSuite/ (https://breagen.github.io/MachSuite/) http://lava.cs.virginia.edu/Rodinia/download_links.htm https://bitbucket.org/eschnett/vecmathlib/wiki/Home (the library has a good regression test / benchmark)
http://parsec.cs.princeton.edu/ https://github.com/graph500/graph500/tree/v2-spec (the current reference implementation requires MPI, but the v2 branch still has serial and OpenMP versions (etc.)) https://github.com/benchmark-subsetting/NPB3.0-omp-C http://www.highproductivity.org/SSCABmks.htm (this web site does not exist any more, but there seems to be a copy of some of the benchmarks https://github.com/gtcasl/hpc-benchmarks/tree/master/SSCA2v2.2)
https://github.com/kokkos/kokkos-kernels/tree/master/perf_test https://github.com/kokkos/kokkos/tree/master/benchmarks
A proposal list has been added (docs/Proposals/TestSuite.rst) in r345074
Roman has been using his image processing suite as part of the AMD Piledriver (BDVER2) scheduler model development, he might be able to advise if this is something that would work well as a test-suite addition: https://github.com/darktable-org/rawspeed
(In reply to Simon Pilgrim from comment #4) > Roman has been using his image processing Not really processing, *decoding*. > suite as part of the AMD > Piledriver (BDVER2) scheduler model development, he might be able to advise > if this is something that would work well as a test-suite addition: > > https://github.com/darktable-org/rawspeed It totally could, both as benchmark, and a test, i would love that. I even brought it up in https://reviews.llvm.org/D46714#1095030 One caveat: it requires a test set, which is kinda heavy - 756M right now, not something you'd want to put into test-suite repo. https://raw.pixls.us/data-unique/ (+ there is rsync), we are looking into putting it into a git repo to simplify syncing.
I added it to the proposal list (http://llvm.org/docs/Proposals/TestSuite.html) in r345166. A data set of that size might also be unsuitable for compiler performance testing because a large portion of the runtime will be I/O (unless the host has a sufficient amount of RAM and we can somehow guarantee it being in the cache). Instead, could we craft smaller RAW images, or use well-compressible images (e.g. all-gray) that are decompressed on-the-fly?
(In reply to Michael Kruse from comment #6) > I added it to the proposal list > (http://llvm.org/docs/Proposals/TestSuite.html) in r345166. Thanks! > A data set of that size might also be unsuitable for compiler performance > testing because a large portion of the runtime will be I/O (unless the host > has a sufficient amount of RAM and we can somehow guarantee it being in the > cache). I'm not sure i follow. What specifically is the concern here? The disk IO itself, or that it will be counted as the time consumed by benchmark? Because the latter is absolutely avoidable by only counting the actual decoding time, not reading time. (The whole file is *already* read to memory first, and decoded from memory.) > Instead, could we craft smaller RAW images, or use well-compressible images > (e.g. all-gray) that are decompressed on-the-fly? Sadly, it's not your typical libpng / libjpeg. There aren't actual specs for these compression schemes available. And even then, one would need to implement the compressors, and then ensure that they correctly roundtrip, etc. So while it's not unfeasible, just not *too* feasible..
(In reply to Roman Lebedev from comment #7) > I'm not sure i follow. What specifically is the concern here? > The disk IO itself, or that it will be counted as the time consumed by > benchmark? The latter > Because the latter is absolutely avoidable by only counting the actual > decoding time, not reading time. > (The whole file is *already* read to memory first, and decoded from memory.) Most programs in test-suite, which the easiest to do, have their total execution time measured (timeit.c or linux perf). It's the user time (not wall-clock or kernel), I don't how much I/O has still an influence on it. I now see that rawspeed's benchmarks (https://github.com/darktable-org/rawspeed/tree/develop/bench) are already using Google Benchmark, which is understood by test-suite's microbenchmark.py.
(In reply to Michael Kruse from comment #8) > (In reply to Roman Lebedev from comment #7) > > I'm not sure i follow. What specifically is the concern here? > > The disk IO itself, or that it will be counted as the time consumed by > > benchmark? > > The latter OK, great :) > > Because the latter is absolutely avoidable by only counting the actual > > decoding time, not reading time. > > (The whole file is *already* read to memory first, and decoded from memory.) > > Most programs in test-suite, which the easiest to do, have their total > execution time measured (timeit.c or linux perf). It's the user time (not > wall-clock or kernel), I don't how much I/O has still an influence on it. > > I now see that rawspeed's benchmarks > (https://github.com/darktable-org/rawspeed/tree/develop/bench) Eh, those "benchmarks" are bad; so far, i never really looked into writing small proper benchmarks. > are already using Google Benchmark Absolutely :) The actual interesting thing is https://github.com/darktable-org/rawspeed/blob/develop/src/utilities/rsbench/main.cpp > which is understood by test-suite's microbenchmark.py.
https://gitlab.com/chriscox/CppPerformanceBenchmarks
(In reply to David Bolvansky from comment #10) > https://gitlab.com/chriscox/CppPerformanceBenchmarks Thanks for the suggestion. I added it to https://llvm.org/docs/Proposals/TestSuite.html in r347369.