Statistical Experimental Design: $2^{K - 1}$ Fractional Factorial Designs

The trouble with $2^{K}$ designs is that the number of samples required grows exponentially with the number of factors $K$. The differences between 4 and 7 factors doesn’t sound like a lot, but it’s the difference between 16 and 128 experimental runs. Fractional factorial designs show how to study extra factors without having to make as many runs. There is no free lunch, but the tradeoffs that it creates are often worth it.

Principles

There are two central ideas,

Sparsity of Effects: It’s often safe to assume that high-order interactions are rare.
Heredity: It is unusual for an interaction to be significant without the corresponding main effects also being significant. For example, it would be unusual to have $A \times B$ be significant without at least one of $A$ or $B$ being significant.

Initial Exploration

Suppose we want to study $A, B$ and $C$, but we only have a budget of 4 runs. We need to choose four corners of the cube to sample. Some choices are obviously bad. E.g., if we chose all the samples on one face, we wouldn’t see any variation in one factor. Here’s an idea that will lead us on the right path. Recall our old notation,

label	A	B	AB
(1)	-	-	+
a	+	-	-
b	-	+	-
ab	+	+	+

Figure 1: A bad choice of samples to subset down to, which would make it impossible to estimate effects from $C$.

Acting on the sparsity of effects principle, let’s suppose that AB is null and invent a contrast for C that exactly matches AB. We’ll update the label to match the associated corner of the cube.

label	A	B	AB	C
(c)	-	-	+	+
a	+	-	-	-
b	-	+	-	-
abc	+	+	+	+

$A more sensible fraction of samples to take.$

Figure 2: A more sensible fraction of samples to take.

This seems like a reasonable choice of 4 corners, making it possible to estimate at least all the main effects. This turns out to be a general strategy for subsetting the full design. But before we do another example, we need to define some vocabulary.

Vocabulary

$2^{K - 1}$: This denotes a fractional factorial design where we take $\frac{1}{2}$ of the samples in a $2^{K}$ design.
Aliases: Above, we assumed that AB was null. Suppose that it weren’t, though. Notice that the contrasts used for the pairs below are all equal, \[\begin{align*} \left[AB\right] &= \frac{1}{2}\left(c - a + b + abc\right) = \left[C\right] \\ \left[BC\right] &= \frac{1}{2}\left(-c + a - b + abc\right) = \left[A\right] \\ \left[AC\right] &= \frac{1}{2}\left(-c - a + b + abc\right) = \left[B\right] \end{align*}\] and we have no conclusive way of distinguishing between these aliased effects, besides appeals to the hereditary principle or domain knowledge. To denote this unidentifiability, we will use bracket notation, for example, \[\begin{align*} \left[A\right] &= A + BC. \end{align*}\]
Generators: In the example, we set $C = AB$. Multiplying both sides by $C$ and using the fact that $C^2 = I$, we find $ABC = I$. The word ABC will be called the generator for this fraction.

$An example complementary fraction.$

Figure 3: An example complementary fraction.

Complementary designs: The complementary fraction of a fractional factorial design are the corners on which we didn’t take samples. Often, if there are strong aliased effects in an initial fractional design, the complementary fraction will be run in the next experiment.
Resolution: A fractional design has resolution $R$ if no $p$-factor is aliased with an effect containing less than $R - p$ factors.

$R$	$p$	$< R - p$	Interpretation
3	1	$\leq 2$	Main effects aren’t aliased with other main effects, but could be aliased with two-way interactions
4	1	$\leq 3$	Main effects aren’t aliased with any other main effects or with any two-way interactions, but could be aliased with three-way interactions
	2	$\leq 2$	Two-way interactions aren’t aliased with main effects.
5	1	$\leq 4$	Main effects aren’t aliased with other main effects, two-way, or three way interactions.
	2	$\leq 3$	Two-way interactions aren’t aliased with with main effects or two-way interactions.

Projection: In a fractional factorial, if we decided that we don’t care about a factor after all, we automatically end up with a $2 ^ {K - 1}$ full-factorial for free.

Figure 4: Three projections of $2^3$ designs into $2^2$ designs, from the textbook.

Sequences: If we find a strong contrast for an aliased effect, it’s natural to try a follow-up experiment to resolve the ambiguity.

Code Example

Let’s suppose we only had only run half of the runs in the filtration example from week 9 [1]. The original experiment is a $2^ 4$ design; we will choose the fraction corresponding to the word¹ $I = ABCD$.
This is a general principle for $2 ^ {K - 1}$ designs: choose which configurations to run by defining the relation $K = A B C \dots J$. This turns a full $2^{K - 1}$ factorial design, which only studied $K - 1$ factors into a fractional $2^{K - 1}$ factorial that studies $K$ factors.

Let’s read the filtration dataset and pretend that we had only collected data corresponding to the fraction specified by $D = ABC$. This is exactly what the filter command does below – it removes all samples that are not consistent with the $D = ABC$ fraction.

library(readr)
library(dplyr)
code <- function(x) ifelse(x == '-', -1, 1)
filtration <- read_table2("https://uwmadison.box.com/shared/static/xxh05ngikmscnddbhg2l3v268jnu4jtc.txt") %>%
  mutate_at(vars(A:D), code) %>%
  filter(D == A * B * C)

We can perform the fit. Note that there missing values in the estimated coefficients because some effects are aliased.

fit <- lm(Rate ~ A * B * C * D, data = filtration)
summary(fit)


Call:
lm(formula = Rate ~ A * B * C * D, data = filtration)

Residuals:
ALL 8 residuals are 0: no residual degrees of freedom!

Coefficients: (8 not defined because of singularities)
            Estimate Std. Error t value Pr(>|t|)
(Intercept)    70.75        NaN     NaN      NaN
A               9.50        NaN     NaN      NaN
B               0.75        NaN     NaN      NaN
C               7.00        NaN     NaN      NaN
D               8.25        NaN     NaN      NaN
A:B            -0.50        NaN     NaN      NaN
A:C            -9.25        NaN     NaN      NaN
B:C             9.50        NaN     NaN      NaN
A:D               NA         NA      NA       NA
B:D               NA         NA      NA       NA
C:D               NA         NA      NA       NA
A:B:C             NA         NA      NA       NA
A:B:D             NA         NA      NA       NA
A:C:D             NA         NA      NA       NA
B:C:D             NA         NA      NA       NA
A:B:C:D           NA         NA      NA       NA

Residual standard error: NaN on 0 degrees of freedom
Multiple R-squared:      1, Adjusted R-squared:    NaN 
F-statistic:   NaN on 7 and 0 DF,  p-value: NA

To back out which terms are aliased, let’s look carefully at the design matrix. All TRUE elements outside of the diagonal are aliased effects. For example, the TRUE in column A on row BCD means that A is aliased with BCD.

X <- model.matrix(fit)
t(X) %*% X != 0 # TRUE on off diagonals are aliases

            (Intercept)     A     B     C     D   A:B   A:C   B:C
(Intercept)        TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
A                 FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE
B                 FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE
C                 FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE
D                 FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE
A:B               FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE
A:C               FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE
B:C               FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE
A:D               FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE
B:D               FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE
C:D               FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE
A:B:C             FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE
A:B:D             FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE
A:C:D             FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE
B:C:D             FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE
A:B:C:D            TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
              A:D   B:D   C:D A:B:C A:B:D A:C:D B:C:D A:B:C:D
(Intercept) FALSE FALSE FALSE FALSE FALSE FALSE FALSE    TRUE
A           FALSE FALSE FALSE FALSE FALSE FALSE  TRUE   FALSE
B           FALSE FALSE FALSE FALSE FALSE  TRUE FALSE   FALSE
C           FALSE FALSE FALSE FALSE  TRUE FALSE FALSE   FALSE
D           FALSE FALSE FALSE  TRUE FALSE FALSE FALSE   FALSE
A:B         FALSE FALSE  TRUE FALSE FALSE FALSE FALSE   FALSE
A:C         FALSE  TRUE FALSE FALSE FALSE FALSE FALSE   FALSE
B:C          TRUE FALSE FALSE FALSE FALSE FALSE FALSE   FALSE
A:D          TRUE FALSE FALSE FALSE FALSE FALSE FALSE   FALSE
B:D         FALSE  TRUE FALSE FALSE FALSE FALSE FALSE   FALSE
C:D         FALSE FALSE  TRUE FALSE FALSE FALSE FALSE   FALSE
A:B:C       FALSE FALSE FALSE  TRUE FALSE FALSE FALSE   FALSE
A:B:D       FALSE FALSE FALSE FALSE  TRUE FALSE FALSE   FALSE
A:C:D       FALSE FALSE FALSE FALSE FALSE  TRUE FALSE   FALSE
B:C:D       FALSE FALSE FALSE FALSE FALSE FALSE  TRUE   FALSE
A:B:C:D     FALSE FALSE FALSE FALSE FALSE FALSE FALSE    TRUE

These off-diagonal elements are automatically extracted in the function below.

library(dplyr)
library(tidyr)

aliases <- function(fit) {
  X <- model.matrix(fit)
  t(X) %*% X  %>%
    as.data.frame() %>%
    add_rownames("effect") %>%
    pivot_longer(-effect, names_to = "alias") %>%
    filter(value != 0, effect != alias) %>%
    select(-value)
}

aliases(fit)

# A tibble: 16 × 2
   effect      alias      
   <chr>       <chr>      
 1 (Intercept) A:B:C:D    
 2 A           B:C:D      
 3 B           A:C:D      
 4 C           A:B:D      
 5 D           A:B:C      
 6 A:B         C:D        
 7 A:C         B:D        
 8 B:C         A:D        
 9 A:D         B:C        
10 B:D         A:C        
11 C:D         A:B        
12 A:B:C       D          
13 A:B:D       C          
14 A:C:D       B          
15 B:C:D       A          
16 A:B:C:D     (Intercept)

So, revisiting the estimated fit, we can conclude that. \[\begin{align*} \widehat{\mu}+\widehat{A B C D} &=70.75 \\ \widehat{A}+\widehat{B C D} &=9.5 \times 2=19 \\ \widehat{B}+\widehat{A C D} &=0.75 \times 2=1.5 \\ \widehat{C}+\widehat{A B D} &=7 \times 2=14 \\ \widehat{D}+\widehat{A B C} &=8.25 \times 2=16.5 \\ \widehat{A B}+\widehat{C D} &=-0.5 \times 2=-1 \\ \widehat{A C}+\widehat{B D}=&-9.25 \times 2=-18.5 \\ \widehat{B C}+\widehat{A D} &=9.5 \times 2=19 \end{align*}\]

Let’s see which of the effects are important, and see whether we can draw any reasonable conclusions, based on the hereditary principle.

daniel_plot <- function(effects, probs = c(0.4, 0.6)) { 
  qq <- qqnorm(effects, datax = TRUE)
  qqline(effects, col = "red", probs = probs, datax = TRUE)
  text(qq$x, qq$y, names(effects), pos=1)
}

daniel_plot(2 * coef(fit)[-1], prob=c(0.25, 0.3))

The important effects look like,

$A + BCD$
$BC + AD$
$D + ABC$
$C + ABD$
$AC + BD$

It seems like we should believe in the main effects $A, D$, and $C$. For the two-way interactions, $AC$ and $AD$ are more plausible, because the main effect for $B$ was not significant.

The word $I = ABCD$ emerged from defining $D = ABC$.↩︎

\(R\)	\(p\)	\(< R - p\)	Interpretation
3	1	\(\leq 2\)	Main effects aren’t aliased with other main effects, but could be aliased with two-way interactions
4	1	\(\leq 3\)	Main effects aren’t aliased with any other main effects or with any two-way interactions, but could be aliased with three-way interactions
	2	\(\leq 2\)	Two-way interactions aren’t aliased with main effects.
5	1	\(\leq 4\)	Main effects aren’t aliased with other main effects, two-way, or three way interactions.
	2	\(\leq 3\)	Two-way interactions aren’t aliased with with main effects or two-way interactions.

2^{K - 1} Fractional Factorial Designs

Principles

Initial Exploration

Vocabulary

Code Example