Skip to content

copula

Copula

Bases: ABC

Abstract Copula Class

The scDesign3 model is built from two components: a collection of marginal models, and a copula to tie them together. This class implements an abstract version of the copula. Within this class, we may define different subclasses that implement various types of regularization or dependencies on experimental and biological conditions. Despite these differences, the overall class must always provide utilities for fitting and sampling dependent uniform variables.

Parameters:

Name Type Description Default
formula str

A string describing the dependence of the copula on experimental or biological conditions. We support predictors for categorical variables like cell type; this corresponds to estimating a different covariance for each category.

required

Attributes:

Name Type Description
loader DataLoader

A data loader object is used to estimate the covariance one batch at a time. This allows estimation of the covariance structure in a streaming way, without having to load all data into memory.

n_outcomes int

The number of features modeled by this marginal model. For example, this corresponds to the number of genes being simulated.

chunk_size int or None

Chunk request from :meth:setup_data, reused by feature summaries.

design Design

Design state for the group and marginal formulas that was adapted to all training observations in :meth:setup_data. This is needed during sampling.

parameters Dict[str, CovarianceStructure]

A dictionary of CovarianceStructure objects. Each key corresponds to a different category specified in the original formula. The covariance structure stores the relationships among genes. It can be a standard covariance matrix, but may also use more memory-efficient approximations like when using CovarianceStructure with a constraint on num_modeled_genes.

Examples:

>>> import scanpy as sc
>>> adata = sc.datasets.pbmc3k()[:, :300]
>>>
>>> class DummyCopula(Copula):
...     def fit(self):
...         pass
...     def likelihood(self):
...         pass
...     def num_params(self):
...         return 0
...     def pseudo_obs(self, x_dict):
...         return np.random.uniform(size=(x_dict["group"].shape[0], self.n_outcomes))
...
>>> model = DummyCopula({"group": "~ 1"})
>>> model.setup_data(adata, {"group": "~ 1"})
>>> model.fit()
Source code in scdesigner/src/scdesigner/base/copula.py
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
class Copula(ABC):
    """Abstract Copula Class

    The scDesign3 model is built from two components: a collection of marginal
    models, and a copula to tie them together. This class implements an abstract
    version of the copula. Within this class, we may define different subclasses
    that implement various types of regularization or dependencies on
    experimental and biological conditions. Despite these differences, the
    overall class must always provide utilities for fitting and sampling
    dependent uniform variables.

    Parameters
    ----------
    formula : str
        A string describing the dependence of the copula on experimental or
        biological conditions. We support predictors for categorical variables
        like cell type; this corresponds to estimating a different covariance
        for each category.
    Attributes
    ----------
    loader : torch.utils.data.DataLoader
        A data loader object is used to estimate the covariance one batch at a
        time. This allows estimation of the covariance structure in a streaming
        way, without having to load all data into memory.
    n_outcomes : int
        The number of features modeled by this marginal model. For example,
        this corresponds to the number of genes being simulated.
    chunk_size : int or None
        Chunk request from :meth:`setup_data`, reused by feature summaries.
    design : Design
        Design state for the group and marginal formulas that was adapted to all
        training observations in :meth:`setup_data`. This is needed during
        sampling.
    parameters : Dict[str, CovarianceStructure]
        A dictionary of CovarianceStructure objects. Each key corresponds to a
        different category specified in the original formula. The covariance
        structure stores the relationships among genes. It can be a standard
        covariance matrix, but may also use more memory-efficient approximations
        like when using CovarianceStructure with a constraint on
        num_modeled_genes.

    Examples
    --------
    >>> import scanpy as sc
    >>> adata = sc.datasets.pbmc3k()[:, :300]
    >>>
    >>> class DummyCopula(Copula):
    ...     def fit(self):
    ...         pass
    ...     def likelihood(self):
    ...         pass
    ...     def num_params(self):
    ...         return 0
    ...     def pseudo_obs(self, x_dict):
    ...         return np.random.uniform(size=(x_dict["group"].shape[0], self.n_outcomes))
    ...
    >>> model = DummyCopula({"group": "~ 1"})
    >>> model.setup_data(adata, {"group": "~ 1"})
    >>> model.fit()
    """
    def __init__(self, formula: Union[str, dict], **kwargs):
        self.formula = formula
        self.loader = None
        self.n_outcomes = None
        self.chunk_size = None
        self.design = None
        self.parameters = None # Should be a dictionary of CovarianceStructure objects

    def setup_data(
        self,
        adata: AnnData,
        marginal_formula: Dict[str, str],
        batch_size: int = 1024,
        *,
        chunk_size: Optional[int] = None,
        observation_transform=None,
        **kwargs,
    ):
        """
        Populate the .design and .loader attributes

        Parameters
        ----------
        adata : AnnData
            This is the object on which we want to estimate the simulator. This
            serves as the template for all downstream fitting.

        marginal_formula : Dict[str, str]
            A dictionary or string specifying the relationship between the columns
            of an input data frame (adata.obs, adata.var, or similar attributes) and
            the parameters of the marginal model. If only a string is provided,
            then the means are allowed to depend on the design parameters, while all
            other parameters are treated as fixed. If a dictionary is provided,
            each key should correspond to a parameter. The string values should be
            in a format that can be parsed by the formulaic package.  For example,
            '~ x' will ensure that the parameter varies linearly with X.

        chunk_size : int, optional
            Rows per chunked read; see :func:`~scdesigner.data.adata_loader`.

        observation_transform : SpatialDesign, optional
            Fitted transform that appends spatial columns to ``adata.obs``.

        Returns
        -------
        None
            This method does not return anything but populates the self.adata,
            formula, design, loader, and n_outcomes attributes based on the
            provided adata input object.
       """
        self.adata = adata
        self.chunk_size = chunk_size
        self.formula = self.formula | marginal_formula #
        self.formula["group"] = partition_formula(self.formula["group"])
        self.design = fit_design(
            adata.obs, self.formula, partition_keys=PARTITION_KEYS,
            transform=observation_transform,
            chunked=resolve_chunk_size(adata, chunk_size) is not None,
        )
        self.loader = adata_loader(
            adata, self.design, chunk_size=chunk_size, batch_size=batch_size, **kwargs
        )
        X_batch, _ = next(iter(self.loader))
        self.n_outcomes = X_batch.shape[1]

    def to(self, device):
        """Move loaded data to ``device``"""
        if self.loader is not None:
            self.loader.dataset.to(device)
        return self

    def decorrelate(self, row_pattern: str, col_pattern: str, group: Union[str, list, None] = None):
        """
        Decorrelate the covariance matrix for the given row and column patterns.

        This method can be used to generate synthetic null data where particular
        pairs of features are forced to be uncorrelated with one another. Any
        indices of the covariance that lie in the intersection of the specified
        row and column patterns will be set to zero.

        Parameters
        ----------
        row_pattern : str
            The regex pattern for the row names to match.
        col_pattern : str
            The regex pattern for the column names to match.
        group : Union[str, list, None], optional
            The group or groups to apply the transformation to. If None, the
            transformation is applied to all groups.

        Returns
        -------
        None
            This method does not return anything but modifies self parameters as
            a side effect.
        """
        if group is None:
            for g in self.groups:
                self.parameters[g].decorrelate(row_pattern, col_pattern)
        elif isinstance(group, str):
            self.parameters[group].decorrelate(row_pattern, col_pattern)
        else:
            for g in group:
                self.parameters[g].decorrelate(row_pattern, col_pattern)

    def correlate(self, factor: float, row_pattern: str, col_pattern: str, group: Union[str, list, None] = None):
        """
        Multiply selected off-diagonal entries by factor.

        To adjust the signal strength in a power analysis, we may want to
        rescale the correlation for specific entries in the covariance matrix.
        This function is used to apply a multiplicative factor to selected
        entries, allowing targeted modification of correlation strength.

        Parameters
        ----------
        factor : float
            The factor to multiply the off-diagonal entries by.
        row_pattern : str
            The regex pattern for the row names to match.
        col_pattern : str
            The regex pattern for the column names to match.
        group : Union[str, list, None], optional
            The group or groups to apply the transformation to. If None, the
            transformation is applied to all groups.

        Returns
        -------
        None
            This method does not return anything but modifies self parameters as
            a side effect.
        """
        if group is None:
            for g in self.groups:
                self.parameters[g].correlate(row_pattern, col_pattern, factor)
        elif isinstance(group, str):
            self.parameters[group].correlate(row_pattern, col_pattern, factor)
        else:
            for g in group:
                self.parameters[g].correlate(row_pattern, col_pattern, factor)

    @abstractmethod
    def fit(self, uniformizer: Callable):
        """
        Fit a Copula

        Copula models are estimated by transforming the observed data onto the
        [0, 1] space of percentiles. See the .invert() method within class
        Marginal.

        Any settings controlling the fit belong to the copula itself: they are
        given either at construction or to .setup_data().

        Parameters
        ----------
        uniformizer : Callable
            Function to transform data to uniform marginals. See .invert()
            within class Marginal for an example.
        """
        raise NotImplementedError

    @abstractmethod
    def pseudo_obs(self, x_dict: Dict):
        """
        Sample from a Copula

        Dependent uniform variables can be sampled from the copula conditional
        on a specific design matrix X (encoding biological and experimental
        covariates). For example, this will sample uniform variables with
        dependence reflecting the cell type specified by X.

        Parameters
        ----------
        x_dict : Dict
            A dictionary of tensors, with one key/value pair per parameter.
            These tensors are the conditioning information to pass to the
            .predict() function of this distribution class. They are the
            numerical design matrices implied by the initializing formulas.
        """
        raise NotImplementedError

    @abstractmethod
    def likelihood(self, uniformizer: Callable, batch: Tuple[torch.Tensor, Dict[str, torch.Tensor]]):
        """
        Parameters
        ----------
        uniformizer : Callable
            Function to transform data to uniform marginals.
        batch : Tuple[torch.Tensor, Dict[str, torch.Tensor]]
            Batch of data.
        """
        raise NotImplementedError

    @abstractmethod
    def num_params(self, **kwargs):
        """
        Covariance Parameters

        This returns the number of free parameters in the overall copula model.
        This is useful for assessing model complexity.

        Parameters
        ----------
        **kwargs
            Additional keyword arguments.
        """
        raise NotImplementedError

correlate(factor, row_pattern, col_pattern, group=None)

Multiply selected off-diagonal entries by factor.

To adjust the signal strength in a power analysis, we may want to rescale the correlation for specific entries in the covariance matrix. This function is used to apply a multiplicative factor to selected entries, allowing targeted modification of correlation strength.

Parameters:

Name Type Description Default
factor float

The factor to multiply the off-diagonal entries by.

required
row_pattern str

The regex pattern for the row names to match.

required
col_pattern str

The regex pattern for the column names to match.

required
group Union[str, list, None]

The group or groups to apply the transformation to. If None, the transformation is applied to all groups.

None

Returns:

Type Description
None

This method does not return anything but modifies self parameters as a side effect.

Source code in scdesigner/src/scdesigner/base/copula.py
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
def correlate(self, factor: float, row_pattern: str, col_pattern: str, group: Union[str, list, None] = None):
    """
    Multiply selected off-diagonal entries by factor.

    To adjust the signal strength in a power analysis, we may want to
    rescale the correlation for specific entries in the covariance matrix.
    This function is used to apply a multiplicative factor to selected
    entries, allowing targeted modification of correlation strength.

    Parameters
    ----------
    factor : float
        The factor to multiply the off-diagonal entries by.
    row_pattern : str
        The regex pattern for the row names to match.
    col_pattern : str
        The regex pattern for the column names to match.
    group : Union[str, list, None], optional
        The group or groups to apply the transformation to. If None, the
        transformation is applied to all groups.

    Returns
    -------
    None
        This method does not return anything but modifies self parameters as
        a side effect.
    """
    if group is None:
        for g in self.groups:
            self.parameters[g].correlate(row_pattern, col_pattern, factor)
    elif isinstance(group, str):
        self.parameters[group].correlate(row_pattern, col_pattern, factor)
    else:
        for g in group:
            self.parameters[g].correlate(row_pattern, col_pattern, factor)

decorrelate(row_pattern, col_pattern, group=None)

Decorrelate the covariance matrix for the given row and column patterns.

This method can be used to generate synthetic null data where particular pairs of features are forced to be uncorrelated with one another. Any indices of the covariance that lie in the intersection of the specified row and column patterns will be set to zero.

Parameters:

Name Type Description Default
row_pattern str

The regex pattern for the row names to match.

required
col_pattern str

The regex pattern for the column names to match.

required
group Union[str, list, None]

The group or groups to apply the transformation to. If None, the transformation is applied to all groups.

None

Returns:

Type Description
None

This method does not return anything but modifies self parameters as a side effect.

Source code in scdesigner/src/scdesigner/base/copula.py
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
def decorrelate(self, row_pattern: str, col_pattern: str, group: Union[str, list, None] = None):
    """
    Decorrelate the covariance matrix for the given row and column patterns.

    This method can be used to generate synthetic null data where particular
    pairs of features are forced to be uncorrelated with one another. Any
    indices of the covariance that lie in the intersection of the specified
    row and column patterns will be set to zero.

    Parameters
    ----------
    row_pattern : str
        The regex pattern for the row names to match.
    col_pattern : str
        The regex pattern for the column names to match.
    group : Union[str, list, None], optional
        The group or groups to apply the transformation to. If None, the
        transformation is applied to all groups.

    Returns
    -------
    None
        This method does not return anything but modifies self parameters as
        a side effect.
    """
    if group is None:
        for g in self.groups:
            self.parameters[g].decorrelate(row_pattern, col_pattern)
    elif isinstance(group, str):
        self.parameters[group].decorrelate(row_pattern, col_pattern)
    else:
        for g in group:
            self.parameters[g].decorrelate(row_pattern, col_pattern)

fit(uniformizer) abstractmethod

Fit a Copula

Copula models are estimated by transforming the observed data onto the [0, 1] space of percentiles. See the .invert() method within class Marginal.

Any settings controlling the fit belong to the copula itself: they are given either at construction or to .setup_data().

Parameters:

Name Type Description Default
uniformizer Callable

Function to transform data to uniform marginals. See .invert() within class Marginal for an example.

required
Source code in scdesigner/src/scdesigner/base/copula.py
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
@abstractmethod
def fit(self, uniformizer: Callable):
    """
    Fit a Copula

    Copula models are estimated by transforming the observed data onto the
    [0, 1] space of percentiles. See the .invert() method within class
    Marginal.

    Any settings controlling the fit belong to the copula itself: they are
    given either at construction or to .setup_data().

    Parameters
    ----------
    uniformizer : Callable
        Function to transform data to uniform marginals. See .invert()
        within class Marginal for an example.
    """
    raise NotImplementedError

likelihood(uniformizer, batch) abstractmethod

Parameters:

Name Type Description Default
uniformizer Callable

Function to transform data to uniform marginals.

required
batch Tuple[Tensor, Dict[str, Tensor]]

Batch of data.

required
Source code in scdesigner/src/scdesigner/base/copula.py
256
257
258
259
260
261
262
263
264
265
266
@abstractmethod
def likelihood(self, uniformizer: Callable, batch: Tuple[torch.Tensor, Dict[str, torch.Tensor]]):
    """
    Parameters
    ----------
    uniformizer : Callable
        Function to transform data to uniform marginals.
    batch : Tuple[torch.Tensor, Dict[str, torch.Tensor]]
        Batch of data.
    """
    raise NotImplementedError

num_params(**kwargs) abstractmethod

Covariance Parameters

This returns the number of free parameters in the overall copula model. This is useful for assessing model complexity.

Parameters:

Name Type Description Default
**kwargs

Additional keyword arguments.

{}
Source code in scdesigner/src/scdesigner/base/copula.py
268
269
270
271
272
273
274
275
276
277
278
279
280
281
@abstractmethod
def num_params(self, **kwargs):
    """
    Covariance Parameters

    This returns the number of free parameters in the overall copula model.
    This is useful for assessing model complexity.

    Parameters
    ----------
    **kwargs
        Additional keyword arguments.
    """
    raise NotImplementedError

pseudo_obs(x_dict) abstractmethod

Sample from a Copula

Dependent uniform variables can be sampled from the copula conditional on a specific design matrix X (encoding biological and experimental covariates). For example, this will sample uniform variables with dependence reflecting the cell type specified by X.

Parameters:

Name Type Description Default
x_dict Dict

A dictionary of tensors, with one key/value pair per parameter. These tensors are the conditioning information to pass to the .predict() function of this distribution class. They are the numerical design matrices implied by the initializing formulas.

required
Source code in scdesigner/src/scdesigner/base/copula.py
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
@abstractmethod
def pseudo_obs(self, x_dict: Dict):
    """
    Sample from a Copula

    Dependent uniform variables can be sampled from the copula conditional
    on a specific design matrix X (encoding biological and experimental
    covariates). For example, this will sample uniform variables with
    dependence reflecting the cell type specified by X.

    Parameters
    ----------
    x_dict : Dict
        A dictionary of tensors, with one key/value pair per parameter.
        These tensors are the conditioning information to pass to the
        .predict() function of this distribution class. They are the
        numerical design matrices implied by the initializing formulas.
    """
    raise NotImplementedError

setup_data(adata, marginal_formula, batch_size=1024, *, chunk_size=None, observation_transform=None, **kwargs)

Populate the .design and .loader attributes

Parameters:

Name Type Description Default
adata AnnData

This is the object on which we want to estimate the simulator. This serves as the template for all downstream fitting.

required
marginal_formula Dict[str, str]

A dictionary or string specifying the relationship between the columns of an input data frame (adata.obs, adata.var, or similar attributes) and the parameters of the marginal model. If only a string is provided, then the means are allowed to depend on the design parameters, while all other parameters are treated as fixed. If a dictionary is provided, each key should correspond to a parameter. The string values should be in a format that can be parsed by the formulaic package. For example, '~ x' will ensure that the parameter varies linearly with X.

required
chunk_size int

Rows per chunked read; see :func:~scdesigner.data.adata_loader.

None
observation_transform SpatialDesign

Fitted transform that appends spatial columns to adata.obs.

None

Returns:

Type Description
None

This method does not return anything but populates the self.adata, formula, design, loader, and n_outcomes attributes based on the provided adata input object.

Source code in scdesigner/src/scdesigner/base/copula.py
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
def setup_data(
    self,
    adata: AnnData,
    marginal_formula: Dict[str, str],
    batch_size: int = 1024,
    *,
    chunk_size: Optional[int] = None,
    observation_transform=None,
    **kwargs,
):
    """
    Populate the .design and .loader attributes

    Parameters
    ----------
    adata : AnnData
        This is the object on which we want to estimate the simulator. This
        serves as the template for all downstream fitting.

    marginal_formula : Dict[str, str]
        A dictionary or string specifying the relationship between the columns
        of an input data frame (adata.obs, adata.var, or similar attributes) and
        the parameters of the marginal model. If only a string is provided,
        then the means are allowed to depend on the design parameters, while all
        other parameters are treated as fixed. If a dictionary is provided,
        each key should correspond to a parameter. The string values should be
        in a format that can be parsed by the formulaic package.  For example,
        '~ x' will ensure that the parameter varies linearly with X.

    chunk_size : int, optional
        Rows per chunked read; see :func:`~scdesigner.data.adata_loader`.

    observation_transform : SpatialDesign, optional
        Fitted transform that appends spatial columns to ``adata.obs``.

    Returns
    -------
    None
        This method does not return anything but populates the self.adata,
        formula, design, loader, and n_outcomes attributes based on the
        provided adata input object.
   """
    self.adata = adata
    self.chunk_size = chunk_size
    self.formula = self.formula | marginal_formula #
    self.formula["group"] = partition_formula(self.formula["group"])
    self.design = fit_design(
        adata.obs, self.formula, partition_keys=PARTITION_KEYS,
        transform=observation_transform,
        chunked=resolve_chunk_size(adata, chunk_size) is not None,
    )
    self.loader = adata_loader(
        adata, self.design, chunk_size=chunk_size, batch_size=batch_size, **kwargs
    )
    X_batch, _ = next(iter(self.loader))
    self.n_outcomes = X_batch.shape[1]

to(device)

Move loaded data to device

Source code in scdesigner/src/scdesigner/base/copula.py
140
141
142
143
144
def to(self, device):
    """Move loaded data to ``device``"""
    if self.loader is not None:
        self.loader.dataset.to(device)
    return self

CovarianceStructure

Efficient storage for covariance matrices in copula-based gene expression modeling.

This class provides memory-efficient storage for covariance information by storing either a full covariance matrix or a block matrix with diagonal variances for remaining genes. This enables fast copula estimation and sampling for large gene expression datasets.

Attributes:

Name Type Description
cov DataFrame

Covariance matrix for modeled genes with gene names as index/columns

modeled_indices ndarray

Indices of modeled genes in original ordering

remaining_var Series or None

Diagonal variances for remaining genes, None if full matrix stored

remaining_indices ndarray or None

Indices of remaining genes in original ordering

num_modeled_genes int

Number of modeled genes

num_remaining_genes int

Number of remaining genes (0 if full matrix stored)

total_genes int

Total number of genes

Examples:

>>> import numpy as np
>>> import pandas as pd
>>>
>>> sigma = np.random.uniform(size=(5, 5))
>>> modeled_names = ["A", "B", "C", "D", "E"]
>>> sigma = pd.DataFrame(sigma, columns=modeled_names, index=modeled_names)
>>> covariance = CovarianceStructure(sigma, modeled_names)
Source code in scdesigner/src/scdesigner/base/copula.py
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
class CovarianceStructure:
    """
    Efficient storage for covariance matrices in copula-based gene expression modeling.

    This class provides memory-efficient storage for covariance information by storing
    either a full covariance matrix or a block matrix with diagonal variances for
    remaining genes. This enables fast copula estimation and sampling for large
    gene expression datasets.

    Attributes
    ----------
    cov : pd.DataFrame
        Covariance matrix for modeled genes with gene names as index/columns
    modeled_indices : np.ndarray
        Indices of modeled genes in original ordering
    remaining_var : pd.Series or None
        Diagonal variances for remaining genes, None if full matrix stored
    remaining_indices : np.ndarray or None
        Indices of remaining genes in original ordering
    num_modeled_genes : int
        Number of modeled genes
    num_remaining_genes : int
        Number of remaining genes (0 if full matrix stored)
    total_genes : int
        Total number of genes

    Examples
    --------
    >>> import numpy as np
    >>> import pandas as pd
    >>>
    >>> sigma = np.random.uniform(size=(5, 5))
    >>> modeled_names = ["A", "B", "C", "D", "E"]
    >>> sigma = pd.DataFrame(sigma, columns=modeled_names, index=modeled_names)
    >>> covariance = CovarianceStructure(sigma, modeled_names)
    """
    def __init__(self, cov: np.ndarray,
                 modeled_names: pd.Index,
                 modeled_indices: Optional[np.ndarray] = None,
                 remaining_var: Optional[np.ndarray] = None,
                 remaining_indices: Optional[np.ndarray] = None,
                 remaining_names: Optional[pd.Index] = None):
        """
        Initialize a CovarianceStructure object.

        Parameters
        ----------
        cov : np.ndarray
            Covariance matrix for modeled genes, shape (n_modeled_genes, n_modeled_genes)
        modeled_names : pd.Index
            Gene names for the modeled genes
        modeled_indices : Optional[np.ndarray], optional
            Indices of modeled genes in original ordering. Defaults to sequential indices.
        remaining_var : Optional[np.ndarray], optional
            Diagonal variances for remaining genes, shape (n_remaining_genes,)
        remaining_indices : Optional[np.ndarray], optional
            Indices of remaining genes in original ordering
        remaining_names : Optional[pd.Index], optional
            Gene names for remaining genes
        """
        self.cov = pd.DataFrame(cov, index=modeled_names, columns=modeled_names)

        if modeled_indices is not None:
            self.modeled_indices = modeled_indices
        else:
            self.modeled_indices = np.arange(len(modeled_names))

        if remaining_var is not None:
            self.remaining_var = pd.Series(remaining_var, index=remaining_names)
        else:
            self.remaining_var = None

        self.remaining_indices = remaining_indices
        self.num_modeled_genes = len(modeled_names)
        self.num_remaining_genes = len(remaining_indices) if remaining_indices is not None else 0
        self.total_genes = self.num_modeled_genes + self.num_remaining_genes

    def __repr__(self):
        if self.remaining_var is None:
            return self.cov.__repr__()
        else:
            return f"CovarianceStructure(modeled_genes={self.num_modeled_genes}, \
                total_genes={self.total_genes})"

    def _repr_html_(self):
        """
        Jupyter Notebook display

        Returns
        -------
        str
            HTML representation of the object.
        """
        if self.remaining_var is None:
            return self.cov._repr_html_()
        else:
            html = f"<b>CovarianceStructure:</b> {self.num_modeled_genes} modeled genes, {self.total_genes} total<br>"
            html += "<h4>Modeled Covariance Matrix</h4>" + self.cov._repr_html_()
            html += "<h4>Remaining Gene Variances</h4>" + self.remaining_var.to_frame("variance").T._repr_html_()
            return html

    def decorrelate(self, row_pattern: str, col_pattern: str):
        """
        Decorrelate the covariance matrix for the given row and column patterns.

        This method can be used to generate synthetic null data where particular
        pairs of features are forced to be uncorrelated with one another. Any
        indices of the covariance that lie in the intersection of the specified
        row and column patterns will be set to zero.

        Parameters
        ----------
        row_pattern : str
            The regex pattern for the row names to match.
        col_pattern : str
            The regex pattern for the column names to match.
        """
        from ..transform.transform import data_frame_mask
        m1 = data_frame_mask(self.cov, ".", col_pattern)
        m2 = data_frame_mask(self.cov, row_pattern, ".")
        mask = (m1 | m2)
        np.fill_diagonal(mask, False)
        self.cov.values[mask] = 0

    def correlate(self, row_pattern: str, col_pattern: str, factor: float):
        """
        Multiply selected off-diagonal entries by factor.

        To adjust the signal strength in a power analysis, we may want to
        rescale the correlation for specific entries in the covariance matrix.
        This function is used to apply a multiplicative factor to selected
        entries, allowing targeted modification of correlation strength.


        Parameters
        ----------
        row_pattern : str
            The regex pattern for the row names to match.
        col_pattern : str
            The regex pattern for the column names to match.
        factor : float
            The factor to multiply the off-diagonal entries by.
        """
        from ..transform.transform import data_frame_mask
        m1 = data_frame_mask(self.cov, ".", col_pattern)
        m2 = data_frame_mask(self.cov, row_pattern, ".")
        mask = (m1 | m2)
        np.fill_diagonal(mask, False)
        self.cov.values[mask] = self.cov.values[mask] * factor

    @property
    def shape(self):
        return (self.total_genes, self.total_genes)

    def to_full_matrix(self):
        """
        Convert to full covariance matrix for compatibility and debugging.

        Returns
        -------
        np.ndarray
            Full covariance matrix with shape (total_genes, total_genes)
        """
        if self.remaining_var is None:
            return self.cov.values
        else:
            full_cov = np.zeros((self.total_genes, self.total_genes))

            # Fill in top-k block
            ix_modeled = np.ix_(self.modeled_indices, self.modeled_indices)
            full_cov[ix_modeled] = self.cov.values

            # Fill in diagonal for remaining genes
            full_cov[self.remaining_indices, self.remaining_indices] = self.remaining_var.values

        return full_cov

__init__(cov, modeled_names, modeled_indices=None, remaining_var=None, remaining_indices=None, remaining_names=None)

Initialize a CovarianceStructure object.

Parameters:

Name Type Description Default
cov ndarray

Covariance matrix for modeled genes, shape (n_modeled_genes, n_modeled_genes)

required
modeled_names Index

Gene names for the modeled genes

required
modeled_indices Optional[ndarray]

Indices of modeled genes in original ordering. Defaults to sequential indices.

None
remaining_var Optional[ndarray]

Diagonal variances for remaining genes, shape (n_remaining_genes,)

None
remaining_indices Optional[ndarray]

Indices of remaining genes in original ordering

None
remaining_names Optional[Index]

Gene names for remaining genes

None
Source code in scdesigner/src/scdesigner/base/copula.py
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
def __init__(self, cov: np.ndarray,
             modeled_names: pd.Index,
             modeled_indices: Optional[np.ndarray] = None,
             remaining_var: Optional[np.ndarray] = None,
             remaining_indices: Optional[np.ndarray] = None,
             remaining_names: Optional[pd.Index] = None):
    """
    Initialize a CovarianceStructure object.

    Parameters
    ----------
    cov : np.ndarray
        Covariance matrix for modeled genes, shape (n_modeled_genes, n_modeled_genes)
    modeled_names : pd.Index
        Gene names for the modeled genes
    modeled_indices : Optional[np.ndarray], optional
        Indices of modeled genes in original ordering. Defaults to sequential indices.
    remaining_var : Optional[np.ndarray], optional
        Diagonal variances for remaining genes, shape (n_remaining_genes,)
    remaining_indices : Optional[np.ndarray], optional
        Indices of remaining genes in original ordering
    remaining_names : Optional[pd.Index], optional
        Gene names for remaining genes
    """
    self.cov = pd.DataFrame(cov, index=modeled_names, columns=modeled_names)

    if modeled_indices is not None:
        self.modeled_indices = modeled_indices
    else:
        self.modeled_indices = np.arange(len(modeled_names))

    if remaining_var is not None:
        self.remaining_var = pd.Series(remaining_var, index=remaining_names)
    else:
        self.remaining_var = None

    self.remaining_indices = remaining_indices
    self.num_modeled_genes = len(modeled_names)
    self.num_remaining_genes = len(remaining_indices) if remaining_indices is not None else 0
    self.total_genes = self.num_modeled_genes + self.num_remaining_genes

correlate(row_pattern, col_pattern, factor)

Multiply selected off-diagonal entries by factor.

To adjust the signal strength in a power analysis, we may want to rescale the correlation for specific entries in the covariance matrix. This function is used to apply a multiplicative factor to selected entries, allowing targeted modification of correlation strength.

Parameters:

Name Type Description Default
row_pattern str

The regex pattern for the row names to match.

required
col_pattern str

The regex pattern for the column names to match.

required
factor float

The factor to multiply the off-diagonal entries by.

required
Source code in scdesigner/src/scdesigner/base/copula.py
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
def correlate(self, row_pattern: str, col_pattern: str, factor: float):
    """
    Multiply selected off-diagonal entries by factor.

    To adjust the signal strength in a power analysis, we may want to
    rescale the correlation for specific entries in the covariance matrix.
    This function is used to apply a multiplicative factor to selected
    entries, allowing targeted modification of correlation strength.


    Parameters
    ----------
    row_pattern : str
        The regex pattern for the row names to match.
    col_pattern : str
        The regex pattern for the column names to match.
    factor : float
        The factor to multiply the off-diagonal entries by.
    """
    from ..transform.transform import data_frame_mask
    m1 = data_frame_mask(self.cov, ".", col_pattern)
    m2 = data_frame_mask(self.cov, row_pattern, ".")
    mask = (m1 | m2)
    np.fill_diagonal(mask, False)
    self.cov.values[mask] = self.cov.values[mask] * factor

decorrelate(row_pattern, col_pattern)

Decorrelate the covariance matrix for the given row and column patterns.

This method can be used to generate synthetic null data where particular pairs of features are forced to be uncorrelated with one another. Any indices of the covariance that lie in the intersection of the specified row and column patterns will be set to zero.

Parameters:

Name Type Description Default
row_pattern str

The regex pattern for the row names to match.

required
col_pattern str

The regex pattern for the column names to match.

required
Source code in scdesigner/src/scdesigner/base/copula.py
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
def decorrelate(self, row_pattern: str, col_pattern: str):
    """
    Decorrelate the covariance matrix for the given row and column patterns.

    This method can be used to generate synthetic null data where particular
    pairs of features are forced to be uncorrelated with one another. Any
    indices of the covariance that lie in the intersection of the specified
    row and column patterns will be set to zero.

    Parameters
    ----------
    row_pattern : str
        The regex pattern for the row names to match.
    col_pattern : str
        The regex pattern for the column names to match.
    """
    from ..transform.transform import data_frame_mask
    m1 = data_frame_mask(self.cov, ".", col_pattern)
    m2 = data_frame_mask(self.cov, row_pattern, ".")
    mask = (m1 | m2)
    np.fill_diagonal(mask, False)
    self.cov.values[mask] = 0

to_full_matrix()

Convert to full covariance matrix for compatibility and debugging.

Returns:

Type Description
ndarray

Full covariance matrix with shape (total_genes, total_genes)

Source code in scdesigner/src/scdesigner/base/copula.py
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
def to_full_matrix(self):
    """
    Convert to full covariance matrix for compatibility and debugging.

    Returns
    -------
    np.ndarray
        Full covariance matrix with shape (total_genes, total_genes)
    """
    if self.remaining_var is None:
        return self.cov.values
    else:
        full_cov = np.zeros((self.total_genes, self.total_genes))

        # Fill in top-k block
        ix_modeled = np.ix_(self.modeled_indices, self.modeled_indices)
        full_cov[ix_modeled] = self.cov.values

        # Fill in diagonal for remaining genes
        full_cov[self.remaining_indices, self.remaining_indices] = self.remaining_var.values

    return full_cov