design
Design matrices evaluated on the complete adata.obs data
If we don't evaluate the design matrix once on the entire alt data, we risk "transform state" (e.g., spline knots or categorical levels) which is internally defined by formulaic when setting a formula not agreeing across chunks.
Design
dataclass
Fitted formulas and (optional) spatial transform.
formulas maps each parameter key to :class:FormulaDesign.
transform appends spatial basis columns before evaluating formulas.
Source code in scdesigner/src/scdesigner/data/design.py
67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 | |
predictor_names
property
Relevant predictor columns for each formula key.
matrices(obs)
Design matrix for each key in formulas.
Source code in scdesigner/src/scdesigner/data/design.py
83 84 85 86 87 | |
FormulaCall
dataclass
A simple function(column, **literals) expression.
Source code in scdesigner/src/scdesigner/data/design.py
203 204 205 206 207 208 209 | |
FormulaDesign
dataclass
Fitted state for one formula.
Attributes:
| Name | Type | Description |
|---|---|---|
spec |
ModelSpec
|
Formula state, with |
levels |
dict of str to pandas.CategoricalDtype
|
Fit-time levels of categorical and |
predictor_names |
list of str
|
Columns retained after reference-level selection. |
Source code in scdesigner/src/scdesigner/data/design.py
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 | |
matrix(obs)
Evaluate using fit-time levels and retain the selected columns.
Source code in scdesigner/src/scdesigner/data/design.py
61 62 63 64 | |
analyze_formula(formula_text)
Collect referenced names and classify Python expressions.
This is used to extract function names from formulas when creating FormulaCalls. If the syntax is not recognized by parse_call, the associated python expression gets mapped to None.
We need this analysis so that we can compute full .obs data level summary statistics, ensuring that calls on batches are consistent with each other.
For example, bs(age, df=4) gets read into FormulaCall(function="bs", column="age", keywords={"df": 4}) which then lets us find the knots to be used in each batch. Similar operations are available for C(), to find the underlying levels.
Source code in scdesigner/src/scdesigner/data/design.py
212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 | |
categorical_levels(values)
Frequency-ordered levels of categorical columns; sorted levels otherwise.
Source code in scdesigner/src/scdesigner/data/design.py
326 327 328 329 330 | |
check_chunked(formula_text, calls)
Reject unallowed formula terms in chunked data.
Source code in scdesigner/src/scdesigner/data/design.py
287 288 289 290 291 292 293 294 295 296 297 298 299 300 | |
clip_extrapolation(transform)
Clip spline inputs beyond the training data range
By default, the formulaic splines will raise an error if they see an input value beyond the training data range. This can cause problems in our training / validation splits, where pseudotime in validation splits lies beyond the training range. This class overrides that formulaic default.
Source code in scdesigner/src/scdesigner/data/design.py
25 26 27 28 29 30 31 32 33 34 35 36 37 | |
code_levels(obs, levels)
Cast columns to fit-time levels; unseen values become missing.
Source code in scdesigner/src/scdesigner/data/design.py
333 334 335 336 337 | |
fit_design(obs, formula, *, partition_keys=frozenset(), transform=None, chunked=False)
Learn design-matrix rules from the complete observation table.
This function lets us learn categorical levels and transform state (e.g., spline knots) that can be reused across chunks and queries.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
obs
|
DataFrame
|
Full training covariate table, before any formula has been applied. |
required |
formula
|
mapping of str to str
|
Named Formulaic formulas, e.g. |
required |
partition_keys
|
set of str
|
Formula keys exempt from reference-column removal. To retain all
cell-type indicators, use |
frozenset()
|
transform
|
SpatialDesign
|
Already-fitted spatial transform, applied before each matrix evaluation. By this point, formulas must reference the already-transformed columns. |
None
|
chunked
|
bool
|
If True, learn state from full columns but build only a one-row matrix
to determine the relevant structure. This avoids creating the full
training design matrix. Allowed factors are column references,
|
False
|
Returns:
| Type | Description |
|---|---|
Design
|
Fitted rules that can be applied to new chunks.
|
Notes
String and categorical levels are ordered by decreasing frequency, with
the most common as reference. Numeric C(column) levels are sorted.
Examples:
>>> obs = pd.DataFrame({"cell_type": ["B", "B", "T"]})
>>> design = fit_design(obs, {"mean": "~ cell_type"}, chunked=True)
>>> design.predictor_names["mean"]
['Intercept', 'cell_type[T.T]']
>>> design.matrices(obs.iloc[[2, 0]])["mean"].to_numpy().tolist()
[[1.0, 1.0], [1.0, 0.0]]
Source code in scdesigner/src/scdesigner/data/design.py
90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 | |
fit_formula(obs, formula_text, *, rows, column_levels, partition=False, chunked=False)
Fit a formula and determine its predictor columns
When chunked=True, categorical levels and state are computed using the
obs data frame. Formulaic evaluates the formula on the first row to
determine column names and categorical encoding structure. Otherwise, it
evaluates the formula on every row.
Source code in scdesigner/src/scdesigner/data/design.py
164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 | |
is_stateful(function)
Whether function is a Formulaic transform that is aware of state.
Source code in scdesigner/src/scdesigner/data/design.py
303 304 305 | |
parse_call(node)
Parse terms infunction(column, keyword=value, ...).
This returns the function name, column name, and keyword values as a
FormulaCall. The function and column must be bare names, with exactly
one positional argument It returns None for any other expression.
Source code in scdesigner/src/scdesigner/data/design.py
265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 | |
reference_columns(obs, levels)
Reference columns to omit from regression designs without an intercept.
With an intercept, Formulaic already drops the reference and uses [T.]
column names. Partition designs skip this selection to keep all indicators.
Source code in scdesigner/src/scdesigner/data/design.py
340 341 342 343 344 345 346 347 348 349 | |
transform_state(obs, calls)
Formulaic state for a given formulaic call, based on a column of obs.
This uses the fact that Formulaic has a __state argument. We don't
actually use the formulaic evaluation, only the filled in transform.
Source code in scdesigner/src/scdesigner/data/design.py
308 309 310 311 312 313 314 315 316 317 318 319 | |