Skip to content

Index

Base classes for scDesigner simulation framework.

Copula

Bases: ABC

Abstract Copula Class

The scDesign3 model is built from two components: a collection of marginal models, and a copula to tie them together. This class implements an abstract version of the copula. Within this class, we may define different subclasses that implement various types of regularization or dependencies on experimental and biological conditions. Despite these differences, the overall class must always provide utilities for fitting and sampling dependent uniform variables.

Parameters:

Name Type Description Default
formula str

A string describing the dependence of the copula on experimental or biological conditions. We support predictors for categorical variables like cell type; this corresponds to estimating a different covariance for each category.

required

Attributes:

Name Type Description
loader DataLoader

A data loader object is used to estimate the covariance one batch at a time. This allows estimation of the covariance structure in a streaming way, without having to load all data into memory.

n_outcomes int

The number of features modeled by this marginal model. For example, this corresponds to the number of genes being simulated.

chunk_size int or None

Chunk request from :meth:setup_data, reused by feature summaries.

design Design

Design state for the group and marginal formulas that was adapted to all training observations in :meth:setup_data. This is needed during sampling.

parameters Dict[str, CovarianceStructure]

A dictionary of CovarianceStructure objects. Each key corresponds to a different category specified in the original formula. The covariance structure stores the relationships among genes. It can be a standard covariance matrix, but may also use more memory-efficient approximations like when using CovarianceStructure with a constraint on num_modeled_genes.

Examples:

>>> import scanpy as sc
>>> adata = sc.datasets.pbmc3k()[:, :300]
>>>
>>> class DummyCopula(Copula):
...     def fit(self):
...         pass
...     def likelihood(self):
...         pass
...     def num_params(self):
...         return 0
...     def pseudo_obs(self, x_dict):
...         return np.random.uniform(size=(x_dict["group"].shape[0], self.n_outcomes))
...
>>> model = DummyCopula({"group": "~ 1"})
>>> model.setup_data(adata, {"group": "~ 1"})
>>> model.fit()
Source code in scdesigner/src/scdesigner/base/copula.py
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
class Copula(ABC):
    """Abstract Copula Class

    The scDesign3 model is built from two components: a collection of marginal
    models, and a copula to tie them together. This class implements an abstract
    version of the copula. Within this class, we may define different subclasses
    that implement various types of regularization or dependencies on
    experimental and biological conditions. Despite these differences, the
    overall class must always provide utilities for fitting and sampling
    dependent uniform variables.

    Parameters
    ----------
    formula : str
        A string describing the dependence of the copula on experimental or
        biological conditions. We support predictors for categorical variables
        like cell type; this corresponds to estimating a different covariance
        for each category.
    Attributes
    ----------
    loader : torch.utils.data.DataLoader
        A data loader object is used to estimate the covariance one batch at a
        time. This allows estimation of the covariance structure in a streaming
        way, without having to load all data into memory.
    n_outcomes : int
        The number of features modeled by this marginal model. For example,
        this corresponds to the number of genes being simulated.
    chunk_size : int or None
        Chunk request from :meth:`setup_data`, reused by feature summaries.
    design : Design
        Design state for the group and marginal formulas that was adapted to all
        training observations in :meth:`setup_data`. This is needed during
        sampling.
    parameters : Dict[str, CovarianceStructure]
        A dictionary of CovarianceStructure objects. Each key corresponds to a
        different category specified in the original formula. The covariance
        structure stores the relationships among genes. It can be a standard
        covariance matrix, but may also use more memory-efficient approximations
        like when using CovarianceStructure with a constraint on
        num_modeled_genes.

    Examples
    --------
    >>> import scanpy as sc
    >>> adata = sc.datasets.pbmc3k()[:, :300]
    >>>
    >>> class DummyCopula(Copula):
    ...     def fit(self):
    ...         pass
    ...     def likelihood(self):
    ...         pass
    ...     def num_params(self):
    ...         return 0
    ...     def pseudo_obs(self, x_dict):
    ...         return np.random.uniform(size=(x_dict["group"].shape[0], self.n_outcomes))
    ...
    >>> model = DummyCopula({"group": "~ 1"})
    >>> model.setup_data(adata, {"group": "~ 1"})
    >>> model.fit()
    """
    def __init__(self, formula: Union[str, dict], **kwargs):
        self.formula = formula
        self.loader = None
        self.n_outcomes = None
        self.chunk_size = None
        self.design = None
        self.parameters = None # Should be a dictionary of CovarianceStructure objects

    def setup_data(
        self,
        adata: AnnData,
        marginal_formula: Dict[str, str],
        batch_size: int = 1024,
        *,
        chunk_size: Optional[int] = None,
        observation_transform=None,
        **kwargs,
    ):
        """
        Populate the .design and .loader attributes

        Parameters
        ----------
        adata : AnnData
            This is the object on which we want to estimate the simulator. This
            serves as the template for all downstream fitting.

        marginal_formula : Dict[str, str]
            A dictionary or string specifying the relationship between the columns
            of an input data frame (adata.obs, adata.var, or similar attributes) and
            the parameters of the marginal model. If only a string is provided,
            then the means are allowed to depend on the design parameters, while all
            other parameters are treated as fixed. If a dictionary is provided,
            each key should correspond to a parameter. The string values should be
            in a format that can be parsed by the formulaic package.  For example,
            '~ x' will ensure that the parameter varies linearly with X.

        chunk_size : int, optional
            Rows per chunked read; see :func:`~scdesigner.data.adata_loader`.

        observation_transform : SpatialDesign, optional
            Fitted transform that appends spatial columns to ``adata.obs``.

        Returns
        -------
        None
            This method does not return anything but populates the self.adata,
            formula, design, loader, and n_outcomes attributes based on the
            provided adata input object.
       """
        self.adata = adata
        self.chunk_size = chunk_size
        self.formula = self.formula | marginal_formula #
        self.formula["group"] = partition_formula(self.formula["group"])
        self.design = fit_design(
            adata.obs, self.formula, partition_keys=PARTITION_KEYS,
            transform=observation_transform,
            chunked=resolve_chunk_size(adata, chunk_size) is not None,
        )
        self.loader = adata_loader(
            adata, self.design, chunk_size=chunk_size, batch_size=batch_size, **kwargs
        )
        X_batch, _ = next(iter(self.loader))
        self.n_outcomes = X_batch.shape[1]

    def to(self, device):
        """Move loaded data to ``device``"""
        if self.loader is not None:
            self.loader.dataset.to(device)
        return self

    def decorrelate(self, row_pattern: str, col_pattern: str, group: Union[str, list, None] = None):
        """
        Decorrelate the covariance matrix for the given row and column patterns.

        This method can be used to generate synthetic null data where particular
        pairs of features are forced to be uncorrelated with one another. Any
        indices of the covariance that lie in the intersection of the specified
        row and column patterns will be set to zero.

        Parameters
        ----------
        row_pattern : str
            The regex pattern for the row names to match.
        col_pattern : str
            The regex pattern for the column names to match.
        group : Union[str, list, None], optional
            The group or groups to apply the transformation to. If None, the
            transformation is applied to all groups.

        Returns
        -------
        None
            This method does not return anything but modifies self parameters as
            a side effect.
        """
        if group is None:
            for g in self.groups:
                self.parameters[g].decorrelate(row_pattern, col_pattern)
        elif isinstance(group, str):
            self.parameters[group].decorrelate(row_pattern, col_pattern)
        else:
            for g in group:
                self.parameters[g].decorrelate(row_pattern, col_pattern)

    def correlate(self, factor: float, row_pattern: str, col_pattern: str, group: Union[str, list, None] = None):
        """
        Multiply selected off-diagonal entries by factor.

        To adjust the signal strength in a power analysis, we may want to
        rescale the correlation for specific entries in the covariance matrix.
        This function is used to apply a multiplicative factor to selected
        entries, allowing targeted modification of correlation strength.

        Parameters
        ----------
        factor : float
            The factor to multiply the off-diagonal entries by.
        row_pattern : str
            The regex pattern for the row names to match.
        col_pattern : str
            The regex pattern for the column names to match.
        group : Union[str, list, None], optional
            The group or groups to apply the transformation to. If None, the
            transformation is applied to all groups.

        Returns
        -------
        None
            This method does not return anything but modifies self parameters as
            a side effect.
        """
        if group is None:
            for g in self.groups:
                self.parameters[g].correlate(row_pattern, col_pattern, factor)
        elif isinstance(group, str):
            self.parameters[group].correlate(row_pattern, col_pattern, factor)
        else:
            for g in group:
                self.parameters[g].correlate(row_pattern, col_pattern, factor)

    @abstractmethod
    def fit(self, uniformizer: Callable):
        """
        Fit a Copula

        Copula models are estimated by transforming the observed data onto the
        [0, 1] space of percentiles. See the .invert() method within class
        Marginal.

        Any settings controlling the fit belong to the copula itself: they are
        given either at construction or to .setup_data().

        Parameters
        ----------
        uniformizer : Callable
            Function to transform data to uniform marginals. See .invert()
            within class Marginal for an example.
        """
        raise NotImplementedError

    @abstractmethod
    def pseudo_obs(self, x_dict: Dict):
        """
        Sample from a Copula

        Dependent uniform variables can be sampled from the copula conditional
        on a specific design matrix X (encoding biological and experimental
        covariates). For example, this will sample uniform variables with
        dependence reflecting the cell type specified by X.

        Parameters
        ----------
        x_dict : Dict
            A dictionary of tensors, with one key/value pair per parameter.
            These tensors are the conditioning information to pass to the
            .predict() function of this distribution class. They are the
            numerical design matrices implied by the initializing formulas.
        """
        raise NotImplementedError

    @abstractmethod
    def likelihood(self, uniformizer: Callable, batch: Tuple[torch.Tensor, Dict[str, torch.Tensor]]):
        """
        Parameters
        ----------
        uniformizer : Callable
            Function to transform data to uniform marginals.
        batch : Tuple[torch.Tensor, Dict[str, torch.Tensor]]
            Batch of data.
        """
        raise NotImplementedError

    @abstractmethod
    def num_params(self, **kwargs):
        """
        Covariance Parameters

        This returns the number of free parameters in the overall copula model.
        This is useful for assessing model complexity.

        Parameters
        ----------
        **kwargs
            Additional keyword arguments.
        """
        raise NotImplementedError

correlate(factor, row_pattern, col_pattern, group=None)

Multiply selected off-diagonal entries by factor.

To adjust the signal strength in a power analysis, we may want to rescale the correlation for specific entries in the covariance matrix. This function is used to apply a multiplicative factor to selected entries, allowing targeted modification of correlation strength.

Parameters:

Name Type Description Default
factor float

The factor to multiply the off-diagonal entries by.

required
row_pattern str

The regex pattern for the row names to match.

required
col_pattern str

The regex pattern for the column names to match.

required
group Union[str, list, None]

The group or groups to apply the transformation to. If None, the transformation is applied to all groups.

None

Returns:

Type Description
None

This method does not return anything but modifies self parameters as a side effect.

Source code in scdesigner/src/scdesigner/base/copula.py
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
def correlate(self, factor: float, row_pattern: str, col_pattern: str, group: Union[str, list, None] = None):
    """
    Multiply selected off-diagonal entries by factor.

    To adjust the signal strength in a power analysis, we may want to
    rescale the correlation for specific entries in the covariance matrix.
    This function is used to apply a multiplicative factor to selected
    entries, allowing targeted modification of correlation strength.

    Parameters
    ----------
    factor : float
        The factor to multiply the off-diagonal entries by.
    row_pattern : str
        The regex pattern for the row names to match.
    col_pattern : str
        The regex pattern for the column names to match.
    group : Union[str, list, None], optional
        The group or groups to apply the transformation to. If None, the
        transformation is applied to all groups.

    Returns
    -------
    None
        This method does not return anything but modifies self parameters as
        a side effect.
    """
    if group is None:
        for g in self.groups:
            self.parameters[g].correlate(row_pattern, col_pattern, factor)
    elif isinstance(group, str):
        self.parameters[group].correlate(row_pattern, col_pattern, factor)
    else:
        for g in group:
            self.parameters[g].correlate(row_pattern, col_pattern, factor)

decorrelate(row_pattern, col_pattern, group=None)

Decorrelate the covariance matrix for the given row and column patterns.

This method can be used to generate synthetic null data where particular pairs of features are forced to be uncorrelated with one another. Any indices of the covariance that lie in the intersection of the specified row and column patterns will be set to zero.

Parameters:

Name Type Description Default
row_pattern str

The regex pattern for the row names to match.

required
col_pattern str

The regex pattern for the column names to match.

required
group Union[str, list, None]

The group or groups to apply the transformation to. If None, the transformation is applied to all groups.

None

Returns:

Type Description
None

This method does not return anything but modifies self parameters as a side effect.

Source code in scdesigner/src/scdesigner/base/copula.py
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
def decorrelate(self, row_pattern: str, col_pattern: str, group: Union[str, list, None] = None):
    """
    Decorrelate the covariance matrix for the given row and column patterns.

    This method can be used to generate synthetic null data where particular
    pairs of features are forced to be uncorrelated with one another. Any
    indices of the covariance that lie in the intersection of the specified
    row and column patterns will be set to zero.

    Parameters
    ----------
    row_pattern : str
        The regex pattern for the row names to match.
    col_pattern : str
        The regex pattern for the column names to match.
    group : Union[str, list, None], optional
        The group or groups to apply the transformation to. If None, the
        transformation is applied to all groups.

    Returns
    -------
    None
        This method does not return anything but modifies self parameters as
        a side effect.
    """
    if group is None:
        for g in self.groups:
            self.parameters[g].decorrelate(row_pattern, col_pattern)
    elif isinstance(group, str):
        self.parameters[group].decorrelate(row_pattern, col_pattern)
    else:
        for g in group:
            self.parameters[g].decorrelate(row_pattern, col_pattern)

fit(uniformizer) abstractmethod

Fit a Copula

Copula models are estimated by transforming the observed data onto the [0, 1] space of percentiles. See the .invert() method within class Marginal.

Any settings controlling the fit belong to the copula itself: they are given either at construction or to .setup_data().

Parameters:

Name Type Description Default
uniformizer Callable

Function to transform data to uniform marginals. See .invert() within class Marginal for an example.

required
Source code in scdesigner/src/scdesigner/base/copula.py
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
@abstractmethod
def fit(self, uniformizer: Callable):
    """
    Fit a Copula

    Copula models are estimated by transforming the observed data onto the
    [0, 1] space of percentiles. See the .invert() method within class
    Marginal.

    Any settings controlling the fit belong to the copula itself: they are
    given either at construction or to .setup_data().

    Parameters
    ----------
    uniformizer : Callable
        Function to transform data to uniform marginals. See .invert()
        within class Marginal for an example.
    """
    raise NotImplementedError

likelihood(uniformizer, batch) abstractmethod

Parameters:

Name Type Description Default
uniformizer Callable

Function to transform data to uniform marginals.

required
batch Tuple[Tensor, Dict[str, Tensor]]

Batch of data.

required
Source code in scdesigner/src/scdesigner/base/copula.py
256
257
258
259
260
261
262
263
264
265
266
@abstractmethod
def likelihood(self, uniformizer: Callable, batch: Tuple[torch.Tensor, Dict[str, torch.Tensor]]):
    """
    Parameters
    ----------
    uniformizer : Callable
        Function to transform data to uniform marginals.
    batch : Tuple[torch.Tensor, Dict[str, torch.Tensor]]
        Batch of data.
    """
    raise NotImplementedError

num_params(**kwargs) abstractmethod

Covariance Parameters

This returns the number of free parameters in the overall copula model. This is useful for assessing model complexity.

Parameters:

Name Type Description Default
**kwargs

Additional keyword arguments.

{}
Source code in scdesigner/src/scdesigner/base/copula.py
268
269
270
271
272
273
274
275
276
277
278
279
280
281
@abstractmethod
def num_params(self, **kwargs):
    """
    Covariance Parameters

    This returns the number of free parameters in the overall copula model.
    This is useful for assessing model complexity.

    Parameters
    ----------
    **kwargs
        Additional keyword arguments.
    """
    raise NotImplementedError

pseudo_obs(x_dict) abstractmethod

Sample from a Copula

Dependent uniform variables can be sampled from the copula conditional on a specific design matrix X (encoding biological and experimental covariates). For example, this will sample uniform variables with dependence reflecting the cell type specified by X.

Parameters:

Name Type Description Default
x_dict Dict

A dictionary of tensors, with one key/value pair per parameter. These tensors are the conditioning information to pass to the .predict() function of this distribution class. They are the numerical design matrices implied by the initializing formulas.

required
Source code in scdesigner/src/scdesigner/base/copula.py
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
@abstractmethod
def pseudo_obs(self, x_dict: Dict):
    """
    Sample from a Copula

    Dependent uniform variables can be sampled from the copula conditional
    on a specific design matrix X (encoding biological and experimental
    covariates). For example, this will sample uniform variables with
    dependence reflecting the cell type specified by X.

    Parameters
    ----------
    x_dict : Dict
        A dictionary of tensors, with one key/value pair per parameter.
        These tensors are the conditioning information to pass to the
        .predict() function of this distribution class. They are the
        numerical design matrices implied by the initializing formulas.
    """
    raise NotImplementedError

setup_data(adata, marginal_formula, batch_size=1024, *, chunk_size=None, observation_transform=None, **kwargs)

Populate the .design and .loader attributes

Parameters:

Name Type Description Default
adata AnnData

This is the object on which we want to estimate the simulator. This serves as the template for all downstream fitting.

required
marginal_formula Dict[str, str]

A dictionary or string specifying the relationship between the columns of an input data frame (adata.obs, adata.var, or similar attributes) and the parameters of the marginal model. If only a string is provided, then the means are allowed to depend on the design parameters, while all other parameters are treated as fixed. If a dictionary is provided, each key should correspond to a parameter. The string values should be in a format that can be parsed by the formulaic package. For example, '~ x' will ensure that the parameter varies linearly with X.

required
chunk_size int

Rows per chunked read; see :func:~scdesigner.data.adata_loader.

None
observation_transform SpatialDesign

Fitted transform that appends spatial columns to adata.obs.

None

Returns:

Type Description
None

This method does not return anything but populates the self.adata, formula, design, loader, and n_outcomes attributes based on the provided adata input object.

Source code in scdesigner/src/scdesigner/base/copula.py
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
def setup_data(
    self,
    adata: AnnData,
    marginal_formula: Dict[str, str],
    batch_size: int = 1024,
    *,
    chunk_size: Optional[int] = None,
    observation_transform=None,
    **kwargs,
):
    """
    Populate the .design and .loader attributes

    Parameters
    ----------
    adata : AnnData
        This is the object on which we want to estimate the simulator. This
        serves as the template for all downstream fitting.

    marginal_formula : Dict[str, str]
        A dictionary or string specifying the relationship between the columns
        of an input data frame (adata.obs, adata.var, or similar attributes) and
        the parameters of the marginal model. If only a string is provided,
        then the means are allowed to depend on the design parameters, while all
        other parameters are treated as fixed. If a dictionary is provided,
        each key should correspond to a parameter. The string values should be
        in a format that can be parsed by the formulaic package.  For example,
        '~ x' will ensure that the parameter varies linearly with X.

    chunk_size : int, optional
        Rows per chunked read; see :func:`~scdesigner.data.adata_loader`.

    observation_transform : SpatialDesign, optional
        Fitted transform that appends spatial columns to ``adata.obs``.

    Returns
    -------
    None
        This method does not return anything but populates the self.adata,
        formula, design, loader, and n_outcomes attributes based on the
        provided adata input object.
   """
    self.adata = adata
    self.chunk_size = chunk_size
    self.formula = self.formula | marginal_formula #
    self.formula["group"] = partition_formula(self.formula["group"])
    self.design = fit_design(
        adata.obs, self.formula, partition_keys=PARTITION_KEYS,
        transform=observation_transform,
        chunked=resolve_chunk_size(adata, chunk_size) is not None,
    )
    self.loader = adata_loader(
        adata, self.design, chunk_size=chunk_size, batch_size=batch_size, **kwargs
    )
    X_batch, _ = next(iter(self.loader))
    self.n_outcomes = X_batch.shape[1]

to(device)

Move loaded data to device

Source code in scdesigner/src/scdesigner/base/copula.py
140
141
142
143
144
def to(self, device):
    """Move loaded data to ``device``"""
    if self.loader is not None:
        self.loader.dataset.to(device)
    return self

CovarianceStructure

Efficient storage for covariance matrices in copula-based gene expression modeling.

This class provides memory-efficient storage for covariance information by storing either a full covariance matrix or a block matrix with diagonal variances for remaining genes. This enables fast copula estimation and sampling for large gene expression datasets.

Attributes:

Name Type Description
cov DataFrame

Covariance matrix for modeled genes with gene names as index/columns

modeled_indices ndarray

Indices of modeled genes in original ordering

remaining_var Series or None

Diagonal variances for remaining genes, None if full matrix stored

remaining_indices ndarray or None

Indices of remaining genes in original ordering

num_modeled_genes int

Number of modeled genes

num_remaining_genes int

Number of remaining genes (0 if full matrix stored)

total_genes int

Total number of genes

Examples:

>>> import numpy as np
>>> import pandas as pd
>>>
>>> sigma = np.random.uniform(size=(5, 5))
>>> modeled_names = ["A", "B", "C", "D", "E"]
>>> sigma = pd.DataFrame(sigma, columns=modeled_names, index=modeled_names)
>>> covariance = CovarianceStructure(sigma, modeled_names)
Source code in scdesigner/src/scdesigner/base/copula.py
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
class CovarianceStructure:
    """
    Efficient storage for covariance matrices in copula-based gene expression modeling.

    This class provides memory-efficient storage for covariance information by storing
    either a full covariance matrix or a block matrix with diagonal variances for
    remaining genes. This enables fast copula estimation and sampling for large
    gene expression datasets.

    Attributes
    ----------
    cov : pd.DataFrame
        Covariance matrix for modeled genes with gene names as index/columns
    modeled_indices : np.ndarray
        Indices of modeled genes in original ordering
    remaining_var : pd.Series or None
        Diagonal variances for remaining genes, None if full matrix stored
    remaining_indices : np.ndarray or None
        Indices of remaining genes in original ordering
    num_modeled_genes : int
        Number of modeled genes
    num_remaining_genes : int
        Number of remaining genes (0 if full matrix stored)
    total_genes : int
        Total number of genes

    Examples
    --------
    >>> import numpy as np
    >>> import pandas as pd
    >>>
    >>> sigma = np.random.uniform(size=(5, 5))
    >>> modeled_names = ["A", "B", "C", "D", "E"]
    >>> sigma = pd.DataFrame(sigma, columns=modeled_names, index=modeled_names)
    >>> covariance = CovarianceStructure(sigma, modeled_names)
    """
    def __init__(self, cov: np.ndarray,
                 modeled_names: pd.Index,
                 modeled_indices: Optional[np.ndarray] = None,
                 remaining_var: Optional[np.ndarray] = None,
                 remaining_indices: Optional[np.ndarray] = None,
                 remaining_names: Optional[pd.Index] = None):
        """
        Initialize a CovarianceStructure object.

        Parameters
        ----------
        cov : np.ndarray
            Covariance matrix for modeled genes, shape (n_modeled_genes, n_modeled_genes)
        modeled_names : pd.Index
            Gene names for the modeled genes
        modeled_indices : Optional[np.ndarray], optional
            Indices of modeled genes in original ordering. Defaults to sequential indices.
        remaining_var : Optional[np.ndarray], optional
            Diagonal variances for remaining genes, shape (n_remaining_genes,)
        remaining_indices : Optional[np.ndarray], optional
            Indices of remaining genes in original ordering
        remaining_names : Optional[pd.Index], optional
            Gene names for remaining genes
        """
        self.cov = pd.DataFrame(cov, index=modeled_names, columns=modeled_names)

        if modeled_indices is not None:
            self.modeled_indices = modeled_indices
        else:
            self.modeled_indices = np.arange(len(modeled_names))

        if remaining_var is not None:
            self.remaining_var = pd.Series(remaining_var, index=remaining_names)
        else:
            self.remaining_var = None

        self.remaining_indices = remaining_indices
        self.num_modeled_genes = len(modeled_names)
        self.num_remaining_genes = len(remaining_indices) if remaining_indices is not None else 0
        self.total_genes = self.num_modeled_genes + self.num_remaining_genes

    def __repr__(self):
        if self.remaining_var is None:
            return self.cov.__repr__()
        else:
            return f"CovarianceStructure(modeled_genes={self.num_modeled_genes}, \
                total_genes={self.total_genes})"

    def _repr_html_(self):
        """
        Jupyter Notebook display

        Returns
        -------
        str
            HTML representation of the object.
        """
        if self.remaining_var is None:
            return self.cov._repr_html_()
        else:
            html = f"<b>CovarianceStructure:</b> {self.num_modeled_genes} modeled genes, {self.total_genes} total<br>"
            html += "<h4>Modeled Covariance Matrix</h4>" + self.cov._repr_html_()
            html += "<h4>Remaining Gene Variances</h4>" + self.remaining_var.to_frame("variance").T._repr_html_()
            return html

    def decorrelate(self, row_pattern: str, col_pattern: str):
        """
        Decorrelate the covariance matrix for the given row and column patterns.

        This method can be used to generate synthetic null data where particular
        pairs of features are forced to be uncorrelated with one another. Any
        indices of the covariance that lie in the intersection of the specified
        row and column patterns will be set to zero.

        Parameters
        ----------
        row_pattern : str
            The regex pattern for the row names to match.
        col_pattern : str
            The regex pattern for the column names to match.
        """
        from ..transform.transform import data_frame_mask
        m1 = data_frame_mask(self.cov, ".", col_pattern)
        m2 = data_frame_mask(self.cov, row_pattern, ".")
        mask = (m1 | m2)
        np.fill_diagonal(mask, False)
        self.cov.values[mask] = 0

    def correlate(self, row_pattern: str, col_pattern: str, factor: float):
        """
        Multiply selected off-diagonal entries by factor.

        To adjust the signal strength in a power analysis, we may want to
        rescale the correlation for specific entries in the covariance matrix.
        This function is used to apply a multiplicative factor to selected
        entries, allowing targeted modification of correlation strength.


        Parameters
        ----------
        row_pattern : str
            The regex pattern for the row names to match.
        col_pattern : str
            The regex pattern for the column names to match.
        factor : float
            The factor to multiply the off-diagonal entries by.
        """
        from ..transform.transform import data_frame_mask
        m1 = data_frame_mask(self.cov, ".", col_pattern)
        m2 = data_frame_mask(self.cov, row_pattern, ".")
        mask = (m1 | m2)
        np.fill_diagonal(mask, False)
        self.cov.values[mask] = self.cov.values[mask] * factor

    @property
    def shape(self):
        return (self.total_genes, self.total_genes)

    def to_full_matrix(self):
        """
        Convert to full covariance matrix for compatibility and debugging.

        Returns
        -------
        np.ndarray
            Full covariance matrix with shape (total_genes, total_genes)
        """
        if self.remaining_var is None:
            return self.cov.values
        else:
            full_cov = np.zeros((self.total_genes, self.total_genes))

            # Fill in top-k block
            ix_modeled = np.ix_(self.modeled_indices, self.modeled_indices)
            full_cov[ix_modeled] = self.cov.values

            # Fill in diagonal for remaining genes
            full_cov[self.remaining_indices, self.remaining_indices] = self.remaining_var.values

        return full_cov

__init__(cov, modeled_names, modeled_indices=None, remaining_var=None, remaining_indices=None, remaining_names=None)

Initialize a CovarianceStructure object.

Parameters:

Name Type Description Default
cov ndarray

Covariance matrix for modeled genes, shape (n_modeled_genes, n_modeled_genes)

required
modeled_names Index

Gene names for the modeled genes

required
modeled_indices Optional[ndarray]

Indices of modeled genes in original ordering. Defaults to sequential indices.

None
remaining_var Optional[ndarray]

Diagonal variances for remaining genes, shape (n_remaining_genes,)

None
remaining_indices Optional[ndarray]

Indices of remaining genes in original ordering

None
remaining_names Optional[Index]

Gene names for remaining genes

None
Source code in scdesigner/src/scdesigner/base/copula.py
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
def __init__(self, cov: np.ndarray,
             modeled_names: pd.Index,
             modeled_indices: Optional[np.ndarray] = None,
             remaining_var: Optional[np.ndarray] = None,
             remaining_indices: Optional[np.ndarray] = None,
             remaining_names: Optional[pd.Index] = None):
    """
    Initialize a CovarianceStructure object.

    Parameters
    ----------
    cov : np.ndarray
        Covariance matrix for modeled genes, shape (n_modeled_genes, n_modeled_genes)
    modeled_names : pd.Index
        Gene names for the modeled genes
    modeled_indices : Optional[np.ndarray], optional
        Indices of modeled genes in original ordering. Defaults to sequential indices.
    remaining_var : Optional[np.ndarray], optional
        Diagonal variances for remaining genes, shape (n_remaining_genes,)
    remaining_indices : Optional[np.ndarray], optional
        Indices of remaining genes in original ordering
    remaining_names : Optional[pd.Index], optional
        Gene names for remaining genes
    """
    self.cov = pd.DataFrame(cov, index=modeled_names, columns=modeled_names)

    if modeled_indices is not None:
        self.modeled_indices = modeled_indices
    else:
        self.modeled_indices = np.arange(len(modeled_names))

    if remaining_var is not None:
        self.remaining_var = pd.Series(remaining_var, index=remaining_names)
    else:
        self.remaining_var = None

    self.remaining_indices = remaining_indices
    self.num_modeled_genes = len(modeled_names)
    self.num_remaining_genes = len(remaining_indices) if remaining_indices is not None else 0
    self.total_genes = self.num_modeled_genes + self.num_remaining_genes

correlate(row_pattern, col_pattern, factor)

Multiply selected off-diagonal entries by factor.

To adjust the signal strength in a power analysis, we may want to rescale the correlation for specific entries in the covariance matrix. This function is used to apply a multiplicative factor to selected entries, allowing targeted modification of correlation strength.

Parameters:

Name Type Description Default
row_pattern str

The regex pattern for the row names to match.

required
col_pattern str

The regex pattern for the column names to match.

required
factor float

The factor to multiply the off-diagonal entries by.

required
Source code in scdesigner/src/scdesigner/base/copula.py
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
def correlate(self, row_pattern: str, col_pattern: str, factor: float):
    """
    Multiply selected off-diagonal entries by factor.

    To adjust the signal strength in a power analysis, we may want to
    rescale the correlation for specific entries in the covariance matrix.
    This function is used to apply a multiplicative factor to selected
    entries, allowing targeted modification of correlation strength.


    Parameters
    ----------
    row_pattern : str
        The regex pattern for the row names to match.
    col_pattern : str
        The regex pattern for the column names to match.
    factor : float
        The factor to multiply the off-diagonal entries by.
    """
    from ..transform.transform import data_frame_mask
    m1 = data_frame_mask(self.cov, ".", col_pattern)
    m2 = data_frame_mask(self.cov, row_pattern, ".")
    mask = (m1 | m2)
    np.fill_diagonal(mask, False)
    self.cov.values[mask] = self.cov.values[mask] * factor

decorrelate(row_pattern, col_pattern)

Decorrelate the covariance matrix for the given row and column patterns.

This method can be used to generate synthetic null data where particular pairs of features are forced to be uncorrelated with one another. Any indices of the covariance that lie in the intersection of the specified row and column patterns will be set to zero.

Parameters:

Name Type Description Default
row_pattern str

The regex pattern for the row names to match.

required
col_pattern str

The regex pattern for the column names to match.

required
Source code in scdesigner/src/scdesigner/base/copula.py
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
def decorrelate(self, row_pattern: str, col_pattern: str):
    """
    Decorrelate the covariance matrix for the given row and column patterns.

    This method can be used to generate synthetic null data where particular
    pairs of features are forced to be uncorrelated with one another. Any
    indices of the covariance that lie in the intersection of the specified
    row and column patterns will be set to zero.

    Parameters
    ----------
    row_pattern : str
        The regex pattern for the row names to match.
    col_pattern : str
        The regex pattern for the column names to match.
    """
    from ..transform.transform import data_frame_mask
    m1 = data_frame_mask(self.cov, ".", col_pattern)
    m2 = data_frame_mask(self.cov, row_pattern, ".")
    mask = (m1 | m2)
    np.fill_diagonal(mask, False)
    self.cov.values[mask] = 0

to_full_matrix()

Convert to full covariance matrix for compatibility and debugging.

Returns:

Type Description
ndarray

Full covariance matrix with shape (total_genes, total_genes)

Source code in scdesigner/src/scdesigner/base/copula.py
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
def to_full_matrix(self):
    """
    Convert to full covariance matrix for compatibility and debugging.

    Returns
    -------
    np.ndarray
        Full covariance matrix with shape (total_genes, total_genes)
    """
    if self.remaining_var is None:
        return self.cov.values
    else:
        full_cov = np.zeros((self.total_genes, self.total_genes))

        # Fill in top-k block
        ix_modeled = np.ix_(self.modeled_indices, self.modeled_indices)
        full_cov[ix_modeled] = self.cov.values

        # Fill in diagonal for remaining genes
        full_cov[self.remaining_indices, self.remaining_indices] = self.remaining_var.values

    return full_cov

Marginal

Bases: ABC

A Feature-wise Marginal Model

This is a class for handling feature-wise (e.g., gene-level) modeling that ignores any correlation between features. For example, it can be used to model the relationship between experimental design features, such as cell type or treatment, and the parameters of a collection of negative binomial models (one for each gene).

These marginals can be plugged into a copula to build the complete scDesign3 simulator. Methods are expected for estimating model parameters, evaluating likelihoods on new samples, and generating new samples given conditioning information. Since these marginal models are intended to be used within a copula, they should also provide utilities for evaluating quantiles and computing cumulative distribution functions.

Parameters:

Name Type Description Default
formula dict or str

A dictionary or string specifying the relationship between the columns of an input data frame (adata.obs, adata.var, or similar attributes) and the parameters of the marginal model. If only a string is provided, then the means are allowed to depend on the design parameters, while all other parameters are treated as fixed. If a dictionary is provided, each key should correspond to a parameter. The string values should be in a format that can be parsed by the formulaic package. For example, '~ x' will ensure that the parameter varies linearly with X.

required
penalties sequence of CoefficientPenalty

Penalties P (:mod:scdesigner.distributions.penalties) added to the training minibatch losses.

()

Attributes:

Name Type Description
formula dict or str

A dictionary or string specifying the relationship between the columns of an input data frame (adata.obs, adata.var, or similar attributes) and the parameters of the marginal model. If only a string is provided, then the means are allowed to depend on the design parameters, while all other parameters are treated as fixed. If a dictionary is provided, each key should correspond to a parameter. The string values should be in a format that can be parsed by the formulaic package. For example, '~ x' will ensure that the parameter varies linearly with X.

penalties tuple of CoefficientPenalty
chunk_size int or None

For backed anndata objects, we read continguous blocks of rows at a time. This parameter governs the size of those chunks and is used in :meth:setup_data and for feature summaries (e.g., for regularization).

design Design

Design matrix factory initially adapted to the full obs data in :meth:setup_data and then applied to prediction and sampling.

feature_dims dict

A dictionary containing the number of predictors associated with each distributional parameter. Note that this number is repeated for every feature (e.g., gene) in the marginal model. This information is often useful for computing the complexity of the estimated model.

loader DataLoader

A torch DataLoader object that returns batches of data for use during training. This loader is constructed internally within the setup_data method. Enumerating this loader returns a tuple: the first element contains a tensor of feature measurements (y), and the second element is a dictionary of tensors containing predictors to use for each parameter (x, for each parameter theta(x)). This design is useful because the design matrices may differ between parameters of the marginal model, y | x ~ F_(theta(x))(y)

train_loader DataLoader

Loader used for marginal optimization. With early stopping enabled, this is a subset of :attr:loader; otherwise it points to :attr:loader.

validation_loader DataLoader or None

Held-out loader used to monitor unpenalized validation likelihood.

fit_history list of dict

Per-epoch training and validation losses, with best/stop indicators.

_validation_split_config tuple or None

Cached (val_frac, validation_seed), where val_frac is the fraction of observations held out for validation, and validation_seed controls the deterministic random split.

n_outcomes int

The number of features modeled by this marginal model. For example, this corresponds to the number of genes being simulated.

predict Module

A torch.nn.Module storing the relationship between predictors for each parameter and the predicted feature-wise outcomes. This module is expected to take the second element of the tuple defined by each batch and then predict a tensor with the same shape as the first element of the batch tuple.

predictor_names dict of list of str

A dictionary whose keys are the parameter names associated with this marginal model. The values for each key are the names of predictors in the design matrix implied by the associated formula. Note that these names may have been expanded from the original formula specification. For example, if cell_type is included in the formula, then the predictor names will include the unique levels of cell_type as separate columns in the design matrix and therefore as separate elements in this list.

parameters dict of pandas.DataFrame

A dictionary whose keys are the parameter names associated with this marginal model. The values for each key are pandas DataFrames storing the fitted parameter values. The rows of each DataFrame are the experimental features specified by the associated formula object (the rownames are the same as those in predictor_names). The columns are features that are being predicted.

device device

The device (e.g., cpu, mps, cuda) used by the model and data loaders. You can set this when initializing the model and change it with :meth:to.

Examples:

>>> class DummyModel(Marginal):
...     def fit(self):
...         pass
...
...     def likelihood(self, batch: Tuple[torch.Tensor, Dict[str, torch.Tensor]]):
...         pass
...
...     def uniformize(self, y: torch.Tensor, x: Dict[str, torch.Tensor]):
...         pass
...
...     def invert(self, u: torch.Tensor, x: Dict[str, torch.Tensor]):
...         pass
...
...     def setup_optimizer(self):
...         pass
...
>>> model = DummyModel("~ cell_type")
>>> model.fit()
Source code in scdesigner/src/scdesigner/base/marginal.py
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
class Marginal(ABC):
    """
    A Feature-wise Marginal Model

    This is a class for handling feature-wise (e.g., gene-level) modeling that
    ignores any correlation between features. For example, it can be used to
    model the relationship between experimental design features, such as cell
    type or treatment, and the parameters of a collection of negative binomial
    models (one for each gene).

    These marginals can be plugged into a copula to build the complete scDesign3
    simulator. Methods are expected for estimating model parameters, evaluating
    likelihoods on new samples, and generating new samples given conditioning
    information. Since these marginal models are intended to be used within a
    copula, they should also provide utilities for evaluating quantiles and
    computing cumulative distribution functions.

    Parameters
    ----------
    formula : dict or str
        A dictionary or string specifying the relationship between the columns
        of an input data frame (adata.obs, adata.var, or similar attributes) and
        the parameters of the marginal model. If only a string is provided,
        then the means are allowed to depend on the design parameters, while all
        other parameters are treated as fixed. If a dictionary is provided,
        each key should correspond to a parameter. The string values should be
        in a format that can be parsed by the formulaic package.  For example,
        '~ x' will ensure that the parameter varies linearly with X.
    penalties : sequence of CoefficientPenalty
        Penalties ``P`` (:mod:`scdesigner.distributions.penalties`) added to
        the training minibatch losses.

    Attributes
    ----------
    formula : dict or str
        A dictionary or string specifying the relationship between the columns
        of an input data frame (adata.obs, adata.var, or similar attributes) and
        the parameters of the marginal model. If only a string is provided,
        then the means are allowed to depend on the design parameters, while all
        other parameters are treated as fixed. If a dictionary is provided,
        each key should correspond to a parameter. The string values should be
        in a format that can be parsed by the formulaic package.  For example,
        '~ x' will ensure that the parameter varies linearly with X.

    penalties : tuple of CoefficientPenalty

    chunk_size : int or None
        For backed anndata objects, we read continguous blocks of rows at a
        time. This parameter governs the size of those chunks and is used in
        :meth:`setup_data` and for feature summaries (e.g., for regularization).

    design : Design
        Design matrix factory initially adapted to the full `obs` data in
        :meth:`setup_data` and then applied to prediction and sampling.

    feature_dims : dict
        A dictionary containing the number of predictors associated with each
        distributional parameter.  Note that this number is repeated for every
        feature (e.g., gene) in the marginal model.  This information is often
        useful for computing the complexity of the estimated model.

    loader : torch.utils.data.DataLoader
        A torch DataLoader object that returns batches of data for use during
        training. This loader is constructed internally within the setup_data
        method. Enumerating this loader returns a tuple: the
        first element contains a tensor of feature measurements (y), and the
        second element is a dictionary of tensors containing predictors to use
        for each parameter (x, for each parameter theta(x)). This design is
        useful because the design matrices may differ between parameters of the
        marginal model, y | x ~ F_(theta(x))(y)

    train_loader : torch.utils.data.DataLoader
        Loader used for marginal optimization. With early stopping enabled,
        this is a subset of :attr:`loader`; otherwise it points to
        :attr:`loader`.

    validation_loader : torch.utils.data.DataLoader or None
        Held-out loader used to monitor unpenalized validation likelihood.

    fit_history : list of dict
        Per-epoch training and validation losses, with best/stop indicators.

    _validation_split_config : tuple or None
        Cached ``(val_frac, validation_seed)``, where ``val_frac`` is
        the fraction of observations held out for validation, and
        ``validation_seed`` controls the deterministic random split.

    n_outcomes : int
        The number of features modeled by this marginal model. For example,
        this corresponds to the number of genes being simulated.

    predict : nn.Module
        A torch.nn.Module storing the relationship between predictors for each
        parameter and the predicted feature-wise outcomes. This module is expected
        to take the second element of the tuple defined by each batch and then
        predict a tensor with the same shape as the first element of the batch
        tuple.

    predictor_names : dict of list of str
        A dictionary whose keys are the parameter names associated with this
        marginal model. The values for each key are the names of predictors in
        the design matrix implied by the associated formula. Note that these
        names may have been expanded from the original formula specification.
        For example, if cell_type is included in the formula, then the predictor
        names will include the unique levels of cell_type as separate columns in
        the design matrix and therefore as separate elements in this list.

    parameters : dict of pandas.DataFrame
        A dictionary whose keys are the parameter names associated with this
        marginal model. The values for each key are pandas DataFrames storing
        the fitted parameter values. The rows of each DataFrame are the
        experimental features specified by the associated formula object (the
        rownames are the same as those in predictor_names). The columns are
        features that are being predicted.

    device : torch.device
        The device (e.g., cpu, mps, cuda) used by the model and data loaders.
        You can set this when initializing the model and change it with
        :meth:`to`.

    Examples
    --------
    >>> class DummyModel(Marginal):
    ...     def fit(self):
    ...         pass
    ...
    ...     def likelihood(self, batch: Tuple[torch.Tensor, Dict[str, torch.Tensor]]):
    ...         pass
    ...
    ...     def uniformize(self, y: torch.Tensor, x: Dict[str, torch.Tensor]):
    ...         pass
    ...
    ...     def invert(self, u: torch.Tensor, x: Dict[str, torch.Tensor]):
    ...         pass
    ...
    ...     def setup_optimizer(self):
    ...         pass
    ...
    >>> model = DummyModel("~ cell_type")
    >>> model.fit()
    """
    def __init__(
        self,
        formula: Union[Dict, str],
        device: Optional[torch.device] = None,
        *,
        penalties: Sequence = (),
    ):
        self.formula = formula
        self.penalties = tuple(penalties)
        self._bound_penalties = ()
        self.chunk_size = None
        self.design = None
        self.feature_dims = None
        self.loader = None
        self.train_loader = None
        self.validation_loader = None
        self.fit_history = []
        self._validation_split_config = None
        self.best_epoch = None
        self.best_validation_loss = None
        self.stopped_epoch = None
        self.n_outcomes = None
        self.predict = None
        self.predictor_names = None
        self.parameters = None
        self.device = get_device(device)

    ##########################################################################
    ## data loading helpers
    ##########################################################################

    def setup_data(
        self,
        adata: AnnData,
        batch_size: int = 1024,
        *,
        chunk_size: Optional[int] = None,
        observation_transform=None,
        **kwargs,
    ):
        """Set up the dataloader for the AnnData object.

        The simulator class definition doesn’t actually require any particular
        template dataset. This is helpful for reasoning about simulators
        abstractly, but when we actually want to estimate parameters, we need a
        template. This method takes an input template dataset and adds
        attributes to the simulator object for later estimation and sampling
        steps.

        Parameters
        ----------
        adata : AnnData
            This is the object on which we want to estimate the simulator. This
            serves as the template for all downstream fitting.
        batch_size : int
            The number of sample to return on each call of the data loader.
            Defaults to 1024.
        chunk_size : int or None
            For backed anndata objects, we read continguous blocks of rows at a
            time. This parameter governs the size of those chunks and is used in
            :meth:`setup_data` and for feature summaries (e.g., for regularization).
        observation_transform : SpatialDesign, optional
            Transform that appends spatial basis columns to ``adata.obs``.
        **kwargs : Any
            Other keyword arguments passed to data loader construction. Any
            argument recognized by the PyTorch DataLoader can be passed in here.

        Returns
        -------
        None
            This method doesn't return anything but modifies the self.adata,
            self.design, self.loader, self.n_outcomes, self.feature_dims, and
            self.predictor_names attributes.
        """
        # keep a reference to the AnnData for later use (e.g., var_names)
        self.adata = adata
        self.chunk_size = chunk_size

        # learn the design matrix from the obs object, and initialize an
        # associated loader
        self.design = fit_design(
            adata.obs, self.formula, transform=observation_transform,
            chunked=resolve_chunk_size(adata, chunk_size) is not None,
        )
        kwargs["device"] = self.device
        self.loader = adata_loader(
            adata, self.design, chunk_size=chunk_size, batch_size=batch_size, **kwargs
        )

        # other attributes updated during training
        X_batch, obs_batch = next(iter(self.loader))
        self.n_outcomes = X_batch.shape[1]
        self.feature_dims = {k: v.shape[1] for k, v in obs_batch.items()}
        self.predictor_names = self.design.predictor_names
        self.train_loader = self.loader
        self.validation_loader = None
        self.fit_history = []
        self._validation_split_config = None
        self.best_epoch = None
        self.best_validation_loss = None
        self.stopped_epoch = None
        self._bound_penalties = ()

    def setup_validation_split(
        self,
        val_frac: float = 0.1,
        validation_seed: int = 0,
    ) -> None:
        """Create train and validation loaders that split the full dataset.

        The full-data loader remains available as :attr:`loader` for downstream
        workflows such as copula fitting. When validation is disabled or the
        dataset is too small to split, :attr:`train_loader` points to
        :attr:`loader` and :attr:`validation_loader` is ``None``.
        """
        if self.loader is None:
            raise RuntimeError("self.loader is not set (call setup_data first)")
        if not 0 <= val_frac < 1:
            raise ValueError("val_frac must satisfy 0 <= val_frac < 1")

        split_config = (float(val_frac), int(validation_seed))
        if (
            self._validation_split_config == split_config
            and self.train_loader is not None
        ):
            return

        n_obs = len(self.loader.dataset)
        if val_frac == 0 or n_obs < 2:
            self.train_loader = self.loader
            self.validation_loader = None
            self._validation_split_config = split_config
            return

        rng = np.random.default_rng(validation_seed)
        indices = rng.permutation(n_obs)
        n_validation = int(np.ceil(n_obs * val_frac))
        n_validation = min(max(n_validation, 1), n_obs - 1)
        validation_indices = indices[:n_validation]
        train_indices = indices[n_validation:]

        self.train_loader = self._subset_loader(train_indices, training=True)
        self.validation_loader = self._subset_loader(validation_indices, training=False)
        self._validation_split_config = split_config

    def _subset_loader(self, indices: np.ndarray, training: bool) -> DataLoader:
        # Resort rows after setup_validation_split randomizes row order. This
        # ensures backed chunk reads don't have to revisit previous chunks
        if isinstance(self.loader.dataset, AnnDataDataset):
            indices = np.sort(indices)

        sampler_is_random = isinstance(self.loader.sampler, RandomSampler)
        return DataLoader(
            Subset(self.loader.dataset, indices.tolist()),
            batch_size=self.loader.batch_size,
            shuffle=training and sampler_is_random,
            num_workers=self.loader.num_workers,
            collate_fn=self.loader.collate_fn,
            drop_last=self.loader.drop_last,
            pin_memory=self.loader.pin_memory,
            timeout=self.loader.timeout,
            worker_init_fn=self.loader.worker_init_fn,
            multiprocessing_context=self.loader.multiprocessing_context,
            generator=self.loader.generator,
            prefetch_factor=self.loader.prefetch_factor,
            persistent_workers=self.loader.persistent_workers,
        )

    ##########################################################################
    ## device managmeent
    ##########################################################################
    @staticmethod
    def _move_tensors(value, device):
        if torch.is_tensor(value):
            return value.to(device)
        if isinstance(value, dict):
            return {key: Marginal._move_tensors(item, device) for key, item in value.items()}
        if isinstance(value, list):
            return [Marginal._move_tensors(item, device) for item in value]
        if isinstance(value, tuple):
            return tuple(Marginal._move_tensors(item, device) for item in value)
        return value

    def to(self, device):
        """Move the predictor, optimizer state, and loaded data to ``device``."""
        self.device = get_device(device)
        if self.predict is not None:
            self.predict.to(self.device)
            optimizer = getattr(self.predict, "optimizer", None)
            if optimizer is not None:
                for parameter, state in optimizer.state.items():
                    optimizer.state[parameter] = self._move_tensors(state, self.device)
        if self.loader is not None:
            self.loader.dataset.to(self.device)
        return self

    def _move_batch_to_device(self, batch):
        y, x = batch
        if y.device != self.device:
            y = y.to(self.device)
            x = {k: v.to(self.device) for k, v in x.items()}
        return y, x

    ##########################################################################
    ## early stopping checks
    ##########################################################################
    def _validation_loss(self) -> Optional[float]:
        if self.validation_loader is None:
            return None

        # save current mode of the model
        was_training = self.predict.training
        self.predict.eval()
        total_loss = 0.0
        total_entries = 0
        with torch.no_grad():
            for batch in self.validation_loader:
                y, x = self._move_batch_to_device(batch)
                total_loss += -self.likelihood((y, x)).sum().item()
                total_entries += y.numel()
        # restore the original mode of the model
        self.predict.train(was_training)

        # could happen when the drop_last option of the validation loader is True
        # and the validation set is smaller than the batch size
        if total_entries == 0:
            return None
        return total_loss / total_entries

    @staticmethod
    def _validation_improved(
        val_loss: float,
        best_loss: float,
        loss_tol: float,
    ) -> bool:
        """Check whether validation loss improved by a relative tolerance."""
        if not np.isfinite(val_loss):
            return False
        if not np.isfinite(best_loss):
            return True
        required_decrease = loss_tol * max(abs(best_loss), 1e-12)
        return best_loss - val_loss >= required_decrease

    def _validate_early_stopping_args(
        self,
        min_epochs: int,
        loss_tol: float,
        patience: int,
    ) -> None:
        if min_epochs < 0:
            raise ValueError("min_epochs must be nonnegative")
        if loss_tol < 0:
            raise ValueError("loss_tol must be nonnegative")
        if patience < 0:
            raise ValueError("patience must be nonnegative")

    ##########################################################################
    ## estimate model coefficients
    ##########################################################################
    def fit(
        self,
        max_epochs: int = 2000,
        verbose: bool = True,
        val_frac: float = 0.1,
        min_epochs: int = 10,
        loss_tol: float = 1e-4,
        patience: int = 40,
        validation_seed: int = 0,
        **kwargs,
    ):
        """Fit the marginal predictor using vanilla PyTorch training loop.

        This method runs stochastic gradient optimization using the template
        dataset defined by the setup_data method. The specific optimizer used
        can be modified with the setup_optimizer method and defaults to Adam.

        Note that, unlike `fit` in class `Simulator`, this method does not allow
        the template dataset as input. This requires `.setup_data()` to be
        called first. We want to give finer-grained control over the data
        loading and optimization in this class relative to the specific
        `Simulator` implementations, which are designed to be easy to run with
        as few steps as possible.

        Parameters
        ----------
        max_epochs : int
            The maximum number of epochs. This is the number of times we feed
            through our cells in the dataset.
        verbose : bool
            Should we print intermediate training outputs?
        val_frac : float
            Fraction of observations held out for validation. Set to 0 to
            disable early stopping and train on the full dataset.
        min_epochs : int
            Minimum number of completed epochs before early stopping is allowed.
        loss_tol : float
            Required relative decrease in validation loss to reset patience.
            For example, ``1e-4`` requires a 0.01% decrease from the current
            best validation loss.
        patience : int
            Number of non-improving validation epochs allowed after the model is
            trained for ``min_epochs``.
        validation_seed : int
            Seed controlling the deterministic validation split.

        Returns
        -------
        None
            This method doesn't return anything but modifies the self.parameters
            attribute with the trained model parameters.
        """
        # Set up validation split and model optimization state.
        self._bound_penalties = ()
        self._validate_early_stopping_args(min_epochs, loss_tol, patience)
        self.setup_validation_split(val_frac=val_frac, validation_seed=validation_seed)
        self.setup_optimizer(**kwargs)
        self._initialize_parameters(**kwargs)
        self._bind_penalties()

        # Initialize validation tracking used for early stopping.
        self.fit_history = []
        self.best_epoch = None
        self.best_validation_loss = None
        self.stopped_epoch = None
        best_loss = float("inf")
        best_state = None
        wait = 0

        train_loader = self.train_loader
        for epoch in range(1, max_epochs + 1):
            epoch_loss = 0.0
            n_entries = 0

            # Main training pass.
            self.predict.train()
            for batch in train_loader:
                y, x = self._move_batch_to_device(batch)

                self.predict.optimizer.zero_grad()
                loss = self._training_loss((y, x))
                loss.backward()
                self.predict.optimizer.step()

                epoch_loss += loss.item()
                n_entries += y.numel()
            self._on_epoch_end(epoch)

            avg_loss = epoch_loss / n_entries if n_entries > 0 else float("nan")
            val_loss = self._validation_loss()
            is_best = False
            stopped = False

            # Validation pass and early-stopping decision.
            if val_loss is not None:
                if self._validation_improved(val_loss, best_loss, loss_tol):
                    best_loss = val_loss
                    wait = 0
                    best_state = copy.deepcopy(self.predict.state_dict())
                    self.best_epoch = epoch
                    self.best_validation_loss = best_loss
                    is_best = True
                else:
                    wait += 1

                if epoch >= min_epochs and wait >= patience:
                    stopped = True
                    self.stopped_epoch = epoch

            self.fit_history.append(
                {
                    "epoch": epoch,
                    "train_loss": avg_loss,
                    "val_loss": val_loss,
                    "best": is_best,
                    "stopped": stopped,
                }
            )
            if verbose:
                msg = f"Epoch {epoch}/{max_epochs}, Loss: {avg_loss:.4f}"
                if val_loss is not None:
                    msg += f", Val Loss: {val_loss:.4f}"
                print(msg, end='\r')
            if stopped:
                break
        if verbose:
            print() # Maintain the loss output

        # Restore best validation checkpoint before formatting parameters.
        if best_state is not None:
            self.predict.load_state_dict(best_state)
        self.parameters = self.format_parameters()

    def _bind_penalties(self) -> None:
        """Bind :attr:`penalties` to the current predictor's coefficients.

        Some of the penalties need to look over the gene-level means (these are
        the penalities with .uses_feature_means). We check that this is actually
        needed before computing it.
        """
        feature_means = None
        if any(spec.uses_feature_means for spec in self.penalties):
            feature_means = summarize_features(self.adata, chunk_size=self.chunk_size).mean
        self._bound_penalties = tuple(
            spec.bind(self.predictor_names, self.predict.coefs, feature_means)
            for spec in self.penalties
        )

    def _training_loss(
        self, batch: Tuple[torch.Tensor, Dict[str, torch.Tensor]]
    ) -> torch.Tensor:
        """``predict.loss_fn(batch)`` plus each bound penalty ``P``."""
        loss = self.predict.loss_fn(batch)
        for penalty in self._bound_penalties:
            loss = loss + penalty.value()
        return loss

    @property
    def fit_history_df(self) -> pd.DataFrame:
        """Return the fit history as a pandas DataFrame."""
        if len(self.fit_history) == 0:
            raise ValueError("fit_history is empty. Call fit() before accessing fit_history_df.")
        return pd.DataFrame(self.fit_history)

    ##########################################################################
    ## printing and summarizing model fits
    ##########################################################################

    def format_parameters(self):
        """Convert fitted coefficient tensors into pandas DataFrames.

        Returns:
            dict: mapping from parameter name -> pandas.DataFrame with rows
                corresponding to predictor column names (from
                `self.predictor_names[param]`) and columns corresponding to
                `self.adata.var_names` (gene names). The values are moved to
                CPU and converted to numpy floats.
        """
        var_names = list(self.adata.var_names)

        dfs = {}
        for param, tensor in self.predict.coefs.items():
            coef_np = tensor.detach().cpu().numpy()
            row_names = list(self.predictor_names[param])
            dfs[param] = pd.DataFrame(coef_np, index=row_names, columns=var_names)
        return dfs

    def num_params(self):
        """Return the number of parameters.

        Count the number of parameters in the marginal simulator. Usually this
        is just the number of predictors times the number of genes, because we
        use a linear model. However, in specific implementations, it’s possible
        to use more flexible models, in which case the number of parameters
        would increase.
        """
        if self.predict is None:
            return 0
        return sum(p.numel() for p in self.predict.parameters() if p.requires_grad)

    ##########################################################################
    ## optimizer and sampling helpers
    ##########################################################################

    @abstractmethod
    def setup_optimizer(self, **kwargs):
        raise NotImplementedError

    def _initialize_parameters(self, **kwargs):
        """Initialize fitted coefficients after train/validation loaders exist.

        Distribution subclasses can override this hook when initialization
        depends on the training data. The default GLM parameter initialization
        from :class:`GLMPredictor` is sufficient for distributions that do not
        need a data-dependent initializer.
        """
        return None

    def _on_epoch_end(self, epoch: int):
        """Run after training but validation for each epoch.

        By default, nothing happens, but for some models, we might want to have
        an estimation step after intermediate parameter updates. For example,
        this can be used to define an E-step in an EM algorithm.
        """
        return None

    @abstractmethod
    def likelihood(self, batch: Tuple[torch.Tensor, Dict[str, torch.Tensor]]) -> torch.Tensor:
        """Compute the log-likelihood for a batch.

        The likelihood is used for maximum likelihood estimation. It is also
        used when computing AIC and BIC scores, which are important when
        choosing an appropriate model complexity.

        Parameters
        ----------
        batch : tuple of (torch.Tensor, dict of str -> torch.Tensor)
            A tuple of gene expression (y) and experimental factors (x) used to
            evaluate the model likelihood. The first element of the tuple is a
            cells x genes tensor. The second is a a dictionary of tensors, with
            one key/value pair per parameter.  These tensors are the
            conditioning information to pass to the .predict() function of this
            distribution class. They are the numerical design matrices implied
            by the initializing formulas.

        Returns
        -------
        torch.Tensor
            A scalar containing the log-likelihood of the batch under the
            current model parameters.
        """
        raise NotImplementedError

    @abstractmethod
    def invert(self, u: torch.Tensor, x: Dict[str, torch.Tensor]):
        """Invert pseudoobservations.

        Return a quantile from the distribution. This handles the link between
        the marginal and the copula model. The idea is that the copula will
        generate pseudo-observations on the unit cube. All the values will be
        between zero and one. By calling invert, we transform these zero-to-one
        valued pseudo-observations into observations in the original data space;
        values between zero and one can be thought of like quantiles.

        Parameters
        ----------
        u : torch.Tensor
            Scalars between [0, 1] that specify the quantile level we want.
        x : Dict of (str -> torch.Tensor)
            A dictionary of tensors, with one key/value pair per parameter.
            These tensors are the conditioning information to pass to the
            .predict() function of this distribution class. They are the
            numerical design matrices implied by the initializing formulas.

        Returns
        -------
        z : torch.Tensor
            A tensor with dimension dim(u) x num_genes. Each row gives the
            requested quantile for the marginal distributions across all genes.
        """
        raise NotImplementedError

    @abstractmethod
    def uniformize(self, y: torch.Tensor, x: Dict[str, torch.Tensor]):
        """Uniformize using learned CDF.

        Apply a quantile/CDF transformation to observations y, accounting for
        conditioning variables x. This step is used in training the copula
        model. Since the copula needs to operate on the unit cube, we need to
        transform the original data onto the unit cube. This can be done by
        applying a CDF transformation.

        Parameters
        ----------
        y: torch.Tensor
            A cells x genes tensor with gene expression levels across all cells.
        x : Dict of (str -> torch.Tensor)
            A dictionary of tensors, with one key/value pair per parameter.
            These tensors are the conditioning information to pass to the
            .predict() function of this distribution class. They are the
            numerical design matrices implied by the initializing formulas.
        """
        raise NotImplementedError

fit_history_df property

Return the fit history as a pandas DataFrame.

fit(max_epochs=2000, verbose=True, val_frac=0.1, min_epochs=10, loss_tol=0.0001, patience=40, validation_seed=0, **kwargs)

Fit the marginal predictor using vanilla PyTorch training loop.

This method runs stochastic gradient optimization using the template dataset defined by the setup_data method. The specific optimizer used can be modified with the setup_optimizer method and defaults to Adam.

Note that, unlike fit in class Simulator, this method does not allow the template dataset as input. This requires .setup_data() to be called first. We want to give finer-grained control over the data loading and optimization in this class relative to the specific Simulator implementations, which are designed to be easy to run with as few steps as possible.

Parameters:

Name Type Description Default
max_epochs int

The maximum number of epochs. This is the number of times we feed through our cells in the dataset.

2000
verbose bool

Should we print intermediate training outputs?

True
val_frac float

Fraction of observations held out for validation. Set to 0 to disable early stopping and train on the full dataset.

0.1
min_epochs int

Minimum number of completed epochs before early stopping is allowed.

10
loss_tol float

Required relative decrease in validation loss to reset patience. For example, 1e-4 requires a 0.01% decrease from the current best validation loss.

0.0001
patience int

Number of non-improving validation epochs allowed after the model is trained for min_epochs.

40
validation_seed int

Seed controlling the deterministic validation split.

0

Returns:

Type Description
None

This method doesn't return anything but modifies the self.parameters attribute with the trained model parameters.

Source code in scdesigner/src/scdesigner/base/marginal.py
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
def fit(
    self,
    max_epochs: int = 2000,
    verbose: bool = True,
    val_frac: float = 0.1,
    min_epochs: int = 10,
    loss_tol: float = 1e-4,
    patience: int = 40,
    validation_seed: int = 0,
    **kwargs,
):
    """Fit the marginal predictor using vanilla PyTorch training loop.

    This method runs stochastic gradient optimization using the template
    dataset defined by the setup_data method. The specific optimizer used
    can be modified with the setup_optimizer method and defaults to Adam.

    Note that, unlike `fit` in class `Simulator`, this method does not allow
    the template dataset as input. This requires `.setup_data()` to be
    called first. We want to give finer-grained control over the data
    loading and optimization in this class relative to the specific
    `Simulator` implementations, which are designed to be easy to run with
    as few steps as possible.

    Parameters
    ----------
    max_epochs : int
        The maximum number of epochs. This is the number of times we feed
        through our cells in the dataset.
    verbose : bool
        Should we print intermediate training outputs?
    val_frac : float
        Fraction of observations held out for validation. Set to 0 to
        disable early stopping and train on the full dataset.
    min_epochs : int
        Minimum number of completed epochs before early stopping is allowed.
    loss_tol : float
        Required relative decrease in validation loss to reset patience.
        For example, ``1e-4`` requires a 0.01% decrease from the current
        best validation loss.
    patience : int
        Number of non-improving validation epochs allowed after the model is
        trained for ``min_epochs``.
    validation_seed : int
        Seed controlling the deterministic validation split.

    Returns
    -------
    None
        This method doesn't return anything but modifies the self.parameters
        attribute with the trained model parameters.
    """
    # Set up validation split and model optimization state.
    self._bound_penalties = ()
    self._validate_early_stopping_args(min_epochs, loss_tol, patience)
    self.setup_validation_split(val_frac=val_frac, validation_seed=validation_seed)
    self.setup_optimizer(**kwargs)
    self._initialize_parameters(**kwargs)
    self._bind_penalties()

    # Initialize validation tracking used for early stopping.
    self.fit_history = []
    self.best_epoch = None
    self.best_validation_loss = None
    self.stopped_epoch = None
    best_loss = float("inf")
    best_state = None
    wait = 0

    train_loader = self.train_loader
    for epoch in range(1, max_epochs + 1):
        epoch_loss = 0.0
        n_entries = 0

        # Main training pass.
        self.predict.train()
        for batch in train_loader:
            y, x = self._move_batch_to_device(batch)

            self.predict.optimizer.zero_grad()
            loss = self._training_loss((y, x))
            loss.backward()
            self.predict.optimizer.step()

            epoch_loss += loss.item()
            n_entries += y.numel()
        self._on_epoch_end(epoch)

        avg_loss = epoch_loss / n_entries if n_entries > 0 else float("nan")
        val_loss = self._validation_loss()
        is_best = False
        stopped = False

        # Validation pass and early-stopping decision.
        if val_loss is not None:
            if self._validation_improved(val_loss, best_loss, loss_tol):
                best_loss = val_loss
                wait = 0
                best_state = copy.deepcopy(self.predict.state_dict())
                self.best_epoch = epoch
                self.best_validation_loss = best_loss
                is_best = True
            else:
                wait += 1

            if epoch >= min_epochs and wait >= patience:
                stopped = True
                self.stopped_epoch = epoch

        self.fit_history.append(
            {
                "epoch": epoch,
                "train_loss": avg_loss,
                "val_loss": val_loss,
                "best": is_best,
                "stopped": stopped,
            }
        )
        if verbose:
            msg = f"Epoch {epoch}/{max_epochs}, Loss: {avg_loss:.4f}"
            if val_loss is not None:
                msg += f", Val Loss: {val_loss:.4f}"
            print(msg, end='\r')
        if stopped:
            break
    if verbose:
        print() # Maintain the loss output

    # Restore best validation checkpoint before formatting parameters.
    if best_state is not None:
        self.predict.load_state_dict(best_state)
    self.parameters = self.format_parameters()

format_parameters()

Convert fitted coefficient tensors into pandas DataFrames.

Returns: dict: mapping from parameter name -> pandas.DataFrame with rows corresponding to predictor column names (from self.predictor_names[param]) and columns corresponding to self.adata.var_names (gene names). The values are moved to CPU and converted to numpy floats.

Source code in scdesigner/src/scdesigner/base/marginal.py
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
def format_parameters(self):
    """Convert fitted coefficient tensors into pandas DataFrames.

    Returns:
        dict: mapping from parameter name -> pandas.DataFrame with rows
            corresponding to predictor column names (from
            `self.predictor_names[param]`) and columns corresponding to
            `self.adata.var_names` (gene names). The values are moved to
            CPU and converted to numpy floats.
    """
    var_names = list(self.adata.var_names)

    dfs = {}
    for param, tensor in self.predict.coefs.items():
        coef_np = tensor.detach().cpu().numpy()
        row_names = list(self.predictor_names[param])
        dfs[param] = pd.DataFrame(coef_np, index=row_names, columns=var_names)
    return dfs

invert(u, x) abstractmethod

Invert pseudoobservations.

Return a quantile from the distribution. This handles the link between the marginal and the copula model. The idea is that the copula will generate pseudo-observations on the unit cube. All the values will be between zero and one. By calling invert, we transform these zero-to-one valued pseudo-observations into observations in the original data space; values between zero and one can be thought of like quantiles.

Parameters:

Name Type Description Default
u Tensor

Scalars between [0, 1] that specify the quantile level we want.

required
x Dict of (str -> torch.Tensor)

A dictionary of tensors, with one key/value pair per parameter. These tensors are the conditioning information to pass to the .predict() function of this distribution class. They are the numerical design matrices implied by the initializing formulas.

required

Returns:

Name Type Description
z Tensor

A tensor with dimension dim(u) x num_genes. Each row gives the requested quantile for the marginal distributions across all genes.

Source code in scdesigner/src/scdesigner/base/marginal.py
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
@abstractmethod
def invert(self, u: torch.Tensor, x: Dict[str, torch.Tensor]):
    """Invert pseudoobservations.

    Return a quantile from the distribution. This handles the link between
    the marginal and the copula model. The idea is that the copula will
    generate pseudo-observations on the unit cube. All the values will be
    between zero and one. By calling invert, we transform these zero-to-one
    valued pseudo-observations into observations in the original data space;
    values between zero and one can be thought of like quantiles.

    Parameters
    ----------
    u : torch.Tensor
        Scalars between [0, 1] that specify the quantile level we want.
    x : Dict of (str -> torch.Tensor)
        A dictionary of tensors, with one key/value pair per parameter.
        These tensors are the conditioning information to pass to the
        .predict() function of this distribution class. They are the
        numerical design matrices implied by the initializing formulas.

    Returns
    -------
    z : torch.Tensor
        A tensor with dimension dim(u) x num_genes. Each row gives the
        requested quantile for the marginal distributions across all genes.
    """
    raise NotImplementedError

likelihood(batch) abstractmethod

Compute the log-likelihood for a batch.

The likelihood is used for maximum likelihood estimation. It is also used when computing AIC and BIC scores, which are important when choosing an appropriate model complexity.

Parameters:

Name Type Description Default
batch tuple of (torch.Tensor, dict of str -> torch.Tensor)

A tuple of gene expression (y) and experimental factors (x) used to evaluate the model likelihood. The first element of the tuple is a cells x genes tensor. The second is a a dictionary of tensors, with one key/value pair per parameter. These tensors are the conditioning information to pass to the .predict() function of this distribution class. They are the numerical design matrices implied by the initializing formulas.

required

Returns:

Type Description
Tensor

A scalar containing the log-likelihood of the batch under the current model parameters.

Source code in scdesigner/src/scdesigner/base/marginal.py
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
@abstractmethod
def likelihood(self, batch: Tuple[torch.Tensor, Dict[str, torch.Tensor]]) -> torch.Tensor:
    """Compute the log-likelihood for a batch.

    The likelihood is used for maximum likelihood estimation. It is also
    used when computing AIC and BIC scores, which are important when
    choosing an appropriate model complexity.

    Parameters
    ----------
    batch : tuple of (torch.Tensor, dict of str -> torch.Tensor)
        A tuple of gene expression (y) and experimental factors (x) used to
        evaluate the model likelihood. The first element of the tuple is a
        cells x genes tensor. The second is a a dictionary of tensors, with
        one key/value pair per parameter.  These tensors are the
        conditioning information to pass to the .predict() function of this
        distribution class. They are the numerical design matrices implied
        by the initializing formulas.

    Returns
    -------
    torch.Tensor
        A scalar containing the log-likelihood of the batch under the
        current model parameters.
    """
    raise NotImplementedError

num_params()

Return the number of parameters.

Count the number of parameters in the marginal simulator. Usually this is just the number of predictors times the number of genes, because we use a linear model. However, in specific implementations, it’s possible to use more flexible models, in which case the number of parameters would increase.

Source code in scdesigner/src/scdesigner/base/marginal.py
603
604
605
606
607
608
609
610
611
612
613
614
def num_params(self):
    """Return the number of parameters.

    Count the number of parameters in the marginal simulator. Usually this
    is just the number of predictors times the number of genes, because we
    use a linear model. However, in specific implementations, it’s possible
    to use more flexible models, in which case the number of parameters
    would increase.
    """
    if self.predict is None:
        return 0
    return sum(p.numel() for p in self.predict.parameters() if p.requires_grad)

setup_data(adata, batch_size=1024, *, chunk_size=None, observation_transform=None, **kwargs)

Set up the dataloader for the AnnData object.

The simulator class definition doesn’t actually require any particular template dataset. This is helpful for reasoning about simulators abstractly, but when we actually want to estimate parameters, we need a template. This method takes an input template dataset and adds attributes to the simulator object for later estimation and sampling steps.

Parameters:

Name Type Description Default
adata AnnData

This is the object on which we want to estimate the simulator. This serves as the template for all downstream fitting.

required
batch_size int

The number of sample to return on each call of the data loader. Defaults to 1024.

1024
chunk_size int or None

For backed anndata objects, we read continguous blocks of rows at a time. This parameter governs the size of those chunks and is used in :meth:setup_data and for feature summaries (e.g., for regularization).

None
observation_transform SpatialDesign

Transform that appends spatial basis columns to adata.obs.

None
**kwargs Any

Other keyword arguments passed to data loader construction. Any argument recognized by the PyTorch DataLoader can be passed in here.

{}

Returns:

Type Description
None

This method doesn't return anything but modifies the self.adata, self.design, self.loader, self.n_outcomes, self.feature_dims, and self.predictor_names attributes.

Source code in scdesigner/src/scdesigner/base/marginal.py
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
def setup_data(
    self,
    adata: AnnData,
    batch_size: int = 1024,
    *,
    chunk_size: Optional[int] = None,
    observation_transform=None,
    **kwargs,
):
    """Set up the dataloader for the AnnData object.

    The simulator class definition doesn’t actually require any particular
    template dataset. This is helpful for reasoning about simulators
    abstractly, but when we actually want to estimate parameters, we need a
    template. This method takes an input template dataset and adds
    attributes to the simulator object for later estimation and sampling
    steps.

    Parameters
    ----------
    adata : AnnData
        This is the object on which we want to estimate the simulator. This
        serves as the template for all downstream fitting.
    batch_size : int
        The number of sample to return on each call of the data loader.
        Defaults to 1024.
    chunk_size : int or None
        For backed anndata objects, we read continguous blocks of rows at a
        time. This parameter governs the size of those chunks and is used in
        :meth:`setup_data` and for feature summaries (e.g., for regularization).
    observation_transform : SpatialDesign, optional
        Transform that appends spatial basis columns to ``adata.obs``.
    **kwargs : Any
        Other keyword arguments passed to data loader construction. Any
        argument recognized by the PyTorch DataLoader can be passed in here.

    Returns
    -------
    None
        This method doesn't return anything but modifies the self.adata,
        self.design, self.loader, self.n_outcomes, self.feature_dims, and
        self.predictor_names attributes.
    """
    # keep a reference to the AnnData for later use (e.g., var_names)
    self.adata = adata
    self.chunk_size = chunk_size

    # learn the design matrix from the obs object, and initialize an
    # associated loader
    self.design = fit_design(
        adata.obs, self.formula, transform=observation_transform,
        chunked=resolve_chunk_size(adata, chunk_size) is not None,
    )
    kwargs["device"] = self.device
    self.loader = adata_loader(
        adata, self.design, chunk_size=chunk_size, batch_size=batch_size, **kwargs
    )

    # other attributes updated during training
    X_batch, obs_batch = next(iter(self.loader))
    self.n_outcomes = X_batch.shape[1]
    self.feature_dims = {k: v.shape[1] for k, v in obs_batch.items()}
    self.predictor_names = self.design.predictor_names
    self.train_loader = self.loader
    self.validation_loader = None
    self.fit_history = []
    self._validation_split_config = None
    self.best_epoch = None
    self.best_validation_loss = None
    self.stopped_epoch = None
    self._bound_penalties = ()

setup_validation_split(val_frac=0.1, validation_seed=0)

Create train and validation loaders that split the full dataset.

The full-data loader remains available as :attr:loader for downstream workflows such as copula fitting. When validation is disabled or the dataset is too small to split, :attr:train_loader points to :attr:loader and :attr:validation_loader is None.

Source code in scdesigner/src/scdesigner/base/marginal.py
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
def setup_validation_split(
    self,
    val_frac: float = 0.1,
    validation_seed: int = 0,
) -> None:
    """Create train and validation loaders that split the full dataset.

    The full-data loader remains available as :attr:`loader` for downstream
    workflows such as copula fitting. When validation is disabled or the
    dataset is too small to split, :attr:`train_loader` points to
    :attr:`loader` and :attr:`validation_loader` is ``None``.
    """
    if self.loader is None:
        raise RuntimeError("self.loader is not set (call setup_data first)")
    if not 0 <= val_frac < 1:
        raise ValueError("val_frac must satisfy 0 <= val_frac < 1")

    split_config = (float(val_frac), int(validation_seed))
    if (
        self._validation_split_config == split_config
        and self.train_loader is not None
    ):
        return

    n_obs = len(self.loader.dataset)
    if val_frac == 0 or n_obs < 2:
        self.train_loader = self.loader
        self.validation_loader = None
        self._validation_split_config = split_config
        return

    rng = np.random.default_rng(validation_seed)
    indices = rng.permutation(n_obs)
    n_validation = int(np.ceil(n_obs * val_frac))
    n_validation = min(max(n_validation, 1), n_obs - 1)
    validation_indices = indices[:n_validation]
    train_indices = indices[n_validation:]

    self.train_loader = self._subset_loader(train_indices, training=True)
    self.validation_loader = self._subset_loader(validation_indices, training=False)
    self._validation_split_config = split_config

to(device)

Move the predictor, optimizer state, and loaded data to device.

Source code in scdesigner/src/scdesigner/base/marginal.py
340
341
342
343
344
345
346
347
348
349
350
351
def to(self, device):
    """Move the predictor, optimizer state, and loaded data to ``device``."""
    self.device = get_device(device)
    if self.predict is not None:
        self.predict.to(self.device)
        optimizer = getattr(self.predict, "optimizer", None)
        if optimizer is not None:
            for parameter, state in optimizer.state.items():
                optimizer.state[parameter] = self._move_tensors(state, self.device)
    if self.loader is not None:
        self.loader.dataset.to(self.device)
    return self

uniformize(y, x) abstractmethod

Uniformize using learned CDF.

Apply a quantile/CDF transformation to observations y, accounting for conditioning variables x. This step is used in training the copula model. Since the copula needs to operate on the unit cube, we need to transform the original data onto the unit cube. This can be done by applying a CDF transformation.

Parameters:

Name Type Description Default
y Tensor

A cells x genes tensor with gene expression levels across all cells.

required
x Dict of (str -> torch.Tensor)

A dictionary of tensors, with one key/value pair per parameter. These tensors are the conditioning information to pass to the .predict() function of this distribution class. They are the numerical design matrices implied by the initializing formulas.

required
Source code in scdesigner/src/scdesigner/base/marginal.py
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
@abstractmethod
def uniformize(self, y: torch.Tensor, x: Dict[str, torch.Tensor]):
    """Uniformize using learned CDF.

    Apply a quantile/CDF transformation to observations y, accounting for
    conditioning variables x. This step is used in training the copula
    model. Since the copula needs to operate on the unit cube, we need to
    transform the original data onto the unit cube. This can be done by
    applying a CDF transformation.

    Parameters
    ----------
    y: torch.Tensor
        A cells x genes tensor with gene expression levels across all cells.
    x : Dict of (str -> torch.Tensor)
        A dictionary of tensors, with one key/value pair per parameter.
        These tensors are the conditioning information to pass to the
        .predict() function of this distribution class. They are the
        numerical design matrices implied by the initializing formulas.
    """
    raise NotImplementedError

Simulator

Simulation abstract class

This abstract simulator class defines the minimal methods that must be exported by every simulator in scdesigner. These methods are:

* `fit`: Given an anndata dataset object, estimate the model parameters.
* `predict`: Given experimental and biological features x in the form of
   a cell-by-features pd.DataFrame, return the parameters theta(x) of
   interest.
* `sample`: Given the same experimental/biological information x as
  `predict`, simulate hypothetical profiles associated wtih those
  samples.

Example instantiations of this class are given in the module scdesigner.base.simulators.

Examples:

>>> import scdesigner.datasets
>>>
>>> pancreas = scdesigner.datasets.pancreas()
>>> sim = Simulator()
>>> sim.parameters
>>>
>>> # this is how a subclass would run, once its fit, predict, and sample
>>> # methods are implemented.
>>> sim.fit(pancreas)
>>> sim.predict(pancreas.obs)
>>> sim.sample(pancreas.obs)
Source code in scdesigner/src/scdesigner/base/simulator.py
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
class Simulator:
    """Simulation abstract class

    This abstract simulator class defines the minimal methods that must be
    exported by every simulator in scdesigner. These methods are:

        * `fit`: Given an anndata dataset object, estimate the model parameters.
        * `predict`: Given experimental and biological features x in the form of
           a cell-by-features pd.DataFrame, return the parameters theta(x) of
           interest.
        * `sample`: Given the same experimental/biological information x as
          `predict`, simulate hypothetical profiles associated wtih those
          samples.

    Example instantiations of this class are given in the module
    `scdesigner.base.simulators`.

    Examples
    --------
    >>> import scdesigner.datasets
    >>>
    >>> pancreas = scdesigner.datasets.pancreas()
    >>> sim = Simulator()
    >>> sim.parameters
    >>>
    >>> # this is how a subclass would run, once its fit, predict, and sample
    >>> # methods are implemented.
    >>> sim.fit(pancreas) # doctest: +SKIP
    >>> sim.predict(pancreas.obs) # doctest: +SKIP
    >>> sim.sample(pancreas.obs) # doctest: +SKIP
    """
    def __init__(self):
        self.parameters = None

    @abstractmethod
    def fit(self, anndata: AnnData, **kwargs) -> None:
        """Fit the simulator

        Parameters
        ----------
        anndata : AnnData
            This is the object on which we want to estimate the simulator. This
            serves as the template for all downstream fitting.
        """
        self.template = anndata
        raise NotImplementedError

    @abstractmethod
    def predict(self, obs: DataFrame=None, **kwargs) -> Dict:
        """Predict from an obs dataframe"""
        raise NotImplementedError

    @abstractmethod
    def sample(self, obs: DataFrame=None, **kwargs) -> AnnData:
        """Generate samples."""
        raise NotImplementedError

fit(anndata, **kwargs) abstractmethod

Fit the simulator

Parameters:

Name Type Description Default
anndata AnnData

This is the object on which we want to estimate the simulator. This serves as the template for all downstream fitting.

required
Source code in scdesigner/src/scdesigner/base/simulator.py
40
41
42
43
44
45
46
47
48
49
50
51
@abstractmethod
def fit(self, anndata: AnnData, **kwargs) -> None:
    """Fit the simulator

    Parameters
    ----------
    anndata : AnnData
        This is the object on which we want to estimate the simulator. This
        serves as the template for all downstream fitting.
    """
    self.template = anndata
    raise NotImplementedError

predict(obs=None, **kwargs) abstractmethod

Predict from an obs dataframe

Source code in scdesigner/src/scdesigner/base/simulator.py
53
54
55
56
@abstractmethod
def predict(self, obs: DataFrame=None, **kwargs) -> Dict:
    """Predict from an obs dataframe"""
    raise NotImplementedError

sample(obs=None, **kwargs) abstractmethod

Generate samples.

Source code in scdesigner/src/scdesigner/base/simulator.py
58
59
60
61
@abstractmethod
def sample(self, obs: DataFrame=None, **kwargs) -> AnnData:
    """Generate samples."""
    raise NotImplementedError