| Copyright | (c) 2009-2014 Bryan O'Sullivan |
|---|---|
| License | BSD-style |
| Maintainer | bos@serpentine.com |
| Stability | experimental |
| Portability | GHC |
| Safe Haskell | None |
| Language | Haskell98 |
Criterion.Types
Contents
Description
Types for benchmarking.
The core type is Benchmarkable, which admits both pure functions
and IO actions.
For a pure function of type a -> b, the benchmarking harness
calls this function repeatedly, each time with a different Int64
argument (the number of times to run the function in a loop), and
reduces the result the function returns to weak head normal form.
For an action of type IO a, the benchmarking harness calls the
action repeatedly, but does not reduce the result.
- data Config = Config {}
- data Verbosity
- newtype Benchmarkable = Benchmarkable (Int64 -> IO ())
- data Benchmark where
- Environment :: NFData env => IO env -> (env -> Benchmark) -> Benchmark
- Benchmark :: String -> Benchmarkable -> Benchmark
- BenchGroup :: String -> [Benchmark] -> Benchmark
- data Measured = Measured {
- measTime :: !Double
- measCpuTime :: !Double
- measCycles :: !Int64
- measIters :: !Int64
- measAllocated :: !Int64
- measNumGcs :: !Int64
- measBytesCopied :: !Int64
- measMutatorWallSeconds :: !Double
- measMutatorCpuSeconds :: !Double
- measGcWallSeconds :: !Double
- measGcCpuSeconds :: !Double
- fromInt :: Int64 -> Maybe Int64
- toInt :: Maybe Int64 -> Int64
- fromDouble :: Double -> Maybe Double
- toDouble :: Maybe Double -> Double
- measureAccessors :: Map String (Measured -> Maybe Double, String)
- measureKeys :: [String]
- measure :: Unbox a => (Measured -> a) -> Vector Measured -> Vector a
- rescale :: Measured -> Measured
- env :: NFData env => IO env -> (env -> Benchmark) -> Benchmark
- bench :: String -> Benchmarkable -> Benchmark
- bgroup :: String -> [Benchmark] -> Benchmark
- addPrefix :: String -> String -> String
- benchNames :: Benchmark -> [String]
- whnf :: (a -> b) -> a -> Benchmarkable
- nf :: NFData b => (a -> b) -> a -> Benchmarkable
- nfIO :: NFData a => IO a -> Benchmarkable
- whnfIO :: IO a -> Benchmarkable
- data Outliers = Outliers {
- samplesSeen :: !Int64
- lowSevere :: !Int64
- lowMild :: !Int64
- highMild :: !Int64
- highSevere :: !Int64
- data OutlierEffect
- = Unaffected
- | Slight
- | Moderate
- | Severe
- data OutlierVariance = OutlierVariance {}
- data Regression = Regression {}
- data KDE = KDE {}
- data Report = Report {}
- data SampleAnalysis = SampleAnalysis {}
Configuration
Top-level benchmarking configuration.
Constructors
| Config | |
Fields
| |
Control the amount of information displayed.
Benchmark descriptions
newtype Benchmarkable Source
A pure function or impure action that can be benchmarked. The
Int64 parameter indicates the number of times to run the given
function or action.
Constructors
| Benchmarkable (Int64 -> IO ()) |
Specification of a collection of benchmarks and environments. A benchmark may consist of:
- An environment that creates input data for benchmarks, created
with
env. - A single
Benchmarkableitem with a name, created withbench. - A (possibly nested) group of
Benchmarks, created withbgroup.
Constructors
| Environment :: NFData env => IO env -> (env -> Benchmark) -> Benchmark | |
| Benchmark :: String -> Benchmarkable -> Benchmark | |
| BenchGroup :: String -> [Benchmark] -> Benchmark |
Measurements
A collection of measurements made while benchmarking.
Measurements related to garbage collection are tagged with GC.
They will only be available if a benchmark is run with "+RTS
-T".
Packed storage. When GC statistics cannot be collected, GC
values will be set to huge negative values. If a field is labeled
with "GC" below, use fromInt and fromDouble to safely
convert to "real" values.
Constructors
| Measured | |
Fields
| |
fromInt :: Int64 -> Maybe Int64 Source
Convert a (possibly unavailable) GC measurement to a true value.
If the measurement is a huge negative number that corresponds to
"no data", this will return Nothing.
toInt :: Maybe Int64 -> Int64 Source
Convert from a true value back to the packed representation used for GC measurements.
fromDouble :: Double -> Maybe Double Source
Convert a (possibly unavailable) GC measurement to a true value.
If the measurement is a huge negative number that corresponds to
"no data", this will return Nothing.
toDouble :: Maybe Double -> Double Source
Convert from a true value back to the packed representation used for GC measurements.
measureAccessors :: Map String (Measured -> Maybe Double, String) Source
Field names and accessors for a Measured record.
measureKeys :: [String] Source
Field names in a Measured record, in the order in which they
appear.
Benchmark construction
Arguments
| :: NFData env | |
| => IO env | Create the environment. The environment will be evaluated to normal form before being passed to the benchmark. |
| -> (env -> Benchmark) | Take the newly created environment and make it available to the given benchmarks. |
| -> Benchmark |
Run a benchmark (or collection of benchmarks) in the given environment. The purpose of an environment is to lazily create input data to pass to the functions that will be benchmarked.
A common example of environment data is input that is read from a file. Another is a large data structure constructed in-place.
Motivation. In earlier versions of criterion, all benchmark inputs were always created when a program started running. By deferring the creation of an environment when its associated benchmarks need the its, we avoid two problems that this strategy caused:
- Memory pressure distorted the results of unrelated benchmarks. If one benchmark needed e.g. a gigabyte-sized input, it would force the garbage collector to do extra work when running some other benchmark that had no use for that input. Since the data created by an environment is only available when it is in scope, it should be garbage collected before other benchmarks are run.
- The time cost of generating all needed inputs could be significant in cases where no inputs (or just a few) were really needed. This occurred often, for instance when just one out of a large suite of benchmarks was run, or when a user would list the collection of benchmarks without running any.
Creation. An environment is created right before its related
benchmarks are run. The IO action that creates the environment
is run, then the newly created environment is evaluated to normal
form (hence the NFData constraint) before being passed to the
function that receives the environment.
Complex environments. If you need to create an environment that contains multiple values, simply pack the values into a tuple.
Lazy pattern matching. In situations where a "real"
environment is not needed, e.g. if a list of benchmark names is
being generated, undefined will be passed to the function that
receives the environment. This avoids the overhead of generating
an environment that will not actually be used.
The function that receives the environment must use lazy pattern
matching to deconstruct the tuple, as use of strict pattern
matching will cause a crash if undefined is passed in.
Example. This program runs benchmarks in an environment that contains two values. The first value is the contents of a text file; the second is a string. Pay attention to the use of a lazy pattern to deconstruct the tuple in the function that returns the benchmarks to be run.
setupEnv = do
let small = replicate 1000 1
big <- readFile "/usr/dict/words"
return (small, big)
main = defaultMain [
-- notice the lazy pattern match here!
env setupEnv $ \ ~(small,big) ->
bgroup "small" [
bench "length" $ whnf length small
, bench "length . filter" $ whnf (length . filter (==1)) small
]
, bgroup "big" [
bench "length" $ whnf length big
, bench "length . filter" $ whnf (length . filter (==1)) big
]
]Discussion. The environment created in the example above is
intentionally not ideal. As Haskell's scoping rules suggest, the
variable big is in scope for the benchmarks that use only
small. It would be better to create a separate environment for
big, so that it will not be kept alive while the unrelated
benchmarks are being run.
Arguments
| :: String | A name to identify the benchmark. |
| -> Benchmarkable | An activity to be benchmarked. |
| -> Benchmark |
Create a single benchmark.
Arguments
| :: String | A name to identify the group of benchmarks. |
| -> [Benchmark] | Benchmarks to group under this name. |
| -> Benchmark |
Group several benchmarks together under a common name.
Add the given prefix to a name. If the prefix is empty, the name
is returned unmodified. Otherwise, the prefix and name are
separated by a '/' character.
benchNames :: Benchmark -> [String] Source
Retrieve the names of all benchmarks. Grouped benchmarks are prefixed with the name of the group they're in.
Evaluation control
whnf :: (a -> b) -> a -> Benchmarkable Source
Apply an argument to a function, and evaluate the result to weak head normal form (WHNF).
nf :: NFData b => (a -> b) -> a -> Benchmarkable Source
Apply an argument to a function, and evaluate the result to head normal form (NF).
nfIO :: NFData a => IO a -> Benchmarkable Source
Perform an action, then evaluate its result to head normal form.
This is particularly useful for forcing a lazy IO action to be
completely performed.
whnfIO :: IO a -> Benchmarkable Source
Perform an action, then evaluate its result to weak head normal
form (WHNF). This is useful for forcing an IO action whose result
is an expression to be evaluated down to a more useful value.
Result types
Outliers from sample data, calculated using the boxplot technique.
Constructors
| Outliers | |
Fields
| |
data OutlierEffect Source
A description of the extent to which outliers in the sample data affect the sample mean and standard deviation.
Constructors
| Unaffected | Less than 1% effect. |
| Slight | Between 1% and 10%. |
| Moderate | Between 10% and 50%. |
| Severe | Above 50% (i.e. measurements are useless). |
Instances
data OutlierVariance Source
Analysis of the extent to which outliers in a sample affect its standard deviation (and to some extent, its mean).
Constructors
| OutlierVariance | |
Fields
| |
Instances
data Regression Source
Results of a linear regression.
Constructors
| Regression | |
Fields
| |
Data for a KDE chart of performance.
Report of a sample analysis.
Constructors
| Report | |
Fields
| |
data SampleAnalysis Source
Result of a bootstrap analysis of a non-parametric sample.
Constructors
| SampleAnalysis | |
Fields
| |