criterion performance measurements

overview

want to understand this report?

SeqFrac/SeqFrac

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	1.4588797731250907	1.471862181645785	1.4851358387500113
Standard deviation	1.3071184020918045e-2	1.5135177961370412e-2	1.6762978866883643e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

ParFrac/ParFrac 1

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	1.5408751420000044	1.5978070373748778	1.6209556826247535
Standard deviation	2.551377811922377e-3	4.316138843444181e-2	5.345950626556354e-2

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

ParFrac/ParFrac 2

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	1.2780093840001427	1.3107687375625119	1.343528091124881
Standard deviation	2.815450903737189e-2	3.872764184971843e-2	4.47146064717521e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

ParFrac/ParFrac 4

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	0.9022439297500569	0.9315607583958657	0.9608775870416746
Standard deviation	2.3453656007255057e-2	3.510679828734604e-2	4.139124264535679e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

ParFrac/ParFrac 8

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	0.8822269797504987	0.9013751625833493	0.9318873297496566
Standard deviation	4.095216667339627e-4	3.0139634958818655e-2	3.822855525341981e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

ParFrac/ParFrac 16

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	0.7635821080622236	0.7916986194374734	0.8124903663749592
Standard deviation	2.9004053914257893e-3	2.7396248980941652e-2	3.39163535180602e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

ParFrac/ParFrac 32

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	0.7135004270003265	0.7480865439168459	0.7664793926668002
Standard deviation	6.2059983335227e-3	3.2953665898625244e-2	4.2476464707800705e-2

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

ParFrac/ParFrac 64

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	0.7059506792500088	0.7190559861041569	0.7327563727499182
Standard deviation	6.215318874410514e-3	1.6029226812478016e-2	2.063503247731646e-2

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

ParFrac/ParFrac 128

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	0.7136471241877871	0.7405977605208515	0.7647279069997239
Standard deviation	1.8405280093353082e-2	2.9571444461219874e-2	3.732090616729532e-2

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

• The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
• The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

• OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
• R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
• Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.