Statistical Experimental Design: $2^K$ Designs are Optimal

Readings 6.7, Rmarkdown

The claim that $2^{K}$ designs are optimal is a bold one, and before we can justify it, we need a sense of (1) the setting, i.e., what are alternative designs are up for consideration and (2) the criteria that will be used to call one design better than an another.

Setting: We have a fixed budget of $n$ samples. Without loss of generality, the factors take their values within the interval $\left[-1, 1\right]$. A candidate design is any way of gathering $n$ samples from among all the possible settings of the $K$ factors.
Criteria: We want to estimate the effects “as well as possible.” We also want to make good predictions at new factor combinations. We’ll have to make these notions precise.

We won’t give formal proofs of optimality (it’s beyond the scope of our book). But we will use simulations to get a sense of the key phenomena at work here, and we’ll create names (like $D-$, $G-$, and $I$-optimality) to describe what we see.

For simplicity, let’s consider $K = 1$. We know that we can use linear regression to estimate factor effects. Suppose that the underlying function is a true linear regression and that our sample size budget is $n = 4$. How should we distributed our four points in order to achieve a good estimate of the underlying linear regression?

Figure 1: Three candidate designs explored in our $K = 1$ simulation.

An animation illustrating the variance in estimated slopes across several designs.

To illustrate this idea, consider the animation above. Each frame is a random run from the simulation. The red line is the true regression function, the three panels correspond to the different placements of $x$, and the black dots are random $y$’s that we observe when sampling at $x$. The orange, green, and blue lines are the regression fits corresponding to those $\left(x, y\right)$ pairs After running this 1000 times, we get the figures below.

Figure 2: The original regressions overlaid (top) and a histogram of all the slopes (bottom).

When we choose x’s close to the origin, the estimates are highly variable. On the other hand, the $2^{K}$ approach, which places all samples at the extremes of the factor seems best, in two senses. First, it has the narrowest band of fitted regression lines. Second, it has the narrowest histogram of estimated slopes (and around the truth).

A design is $D$-optimal if $\left|\text{Cov}\left(\hat{\beta}\right)\right|$ is minimized. In our picture, this happens when the width of the histogram of $\hat{\beta}$ is minimized. The determinant generalizes the notion of width to higher-dimensions (specifically, it gives the volume of a high-dimensional parallelogram.)
A design is $G$-optimal if $\max_{x} \text{Var}\left(\hat{y}\left(x\right)\right)$ is minimized. In our picture, this happens when the maximum vertical spread of the prediction band is minimized.
A design is $V$-optimal if $\int_{\left[-1, 1\right]^{K}}. \text{Var}\left(\hat{y}\left(x\right)\right)dx$ is minimized. In our picture, this happens when the area of the prediction band is minimized

Figure 3: Summary of alternative optimality definitions.

While our simulation was only for $K = 1$, it turns out that according to all three criteria, the $2^{K}$-design is optimal.

Figure 4: Three candidate designs when studying 2 factors, all using 4 samples. The $2^2$ design is at the bottom.

2^K Designs are Optimal