latte.functional.disentanglement.mutual_info

Module Contents

Functions

get_mi_func(discrete: bool) → Callable	Get mutual information function depending on whether the attribute is discrete
latent_attr_mutual_info(z: numpy.ndarray, a: numpy.ndarray, discrete: bool = False) → numpy.ndarray	Calculate mutual information between latent vectors and a target attribute.
single_mutual_info(a: numpy.ndarray, b: numpy.ndarray, discrete: bool) → float	Calculate mutual information between two variables
entropy(a: numpy.ndarray, discrete: bool = False) → float	Calculate entropy of a variable
conditional_entropy(ai: numpy.ndarray, aj: numpy.ndarray, discrete: bool = False) → float	Calculate conditional entropy of a variable given another variable.
mig(z: numpy.ndarray, a: numpy.ndarray, reg_dim: Optional[List] = None, discrete: bool = False, fill_reg_dim: bool = False) → numpy.ndarray	Calculate Mutual Information Gap (MIG) between latent vectors and attributes.
dmig(z: numpy.ndarray, a: numpy.ndarray, reg_dim: Optional[List] = None, discrete: bool = False) → numpy.ndarray	Calculate Dependency-Aware Mutual Information Gap (DMIG) between latent vectors and attributes
dlig(z: numpy.ndarray, a: numpy.ndarray, reg_dim: Optional[List] = None, discrete: bool = False)	Calculate Dependency-Aware Latent Information Gap (DLIG) between latent vectors and attributes
xmig(z: numpy.ndarray, a: numpy.ndarray, reg_dim: Optional[List] = None, discrete: bool = False)	Calculate Dependency-Blind Mutual Information Gap (XMIG) between latent vectors and attributes

latte.functional.disentanglement.mutual_info.get_mi_func(discrete: bool) → Callable

Get mutual information function depending on whether the attribute is discrete

Parameters

discrete (bool) – whether the attribute is discrete

Returns

mutual information function handle

Return type

Callable

latte.functional.disentanglement.mutual_info.latent_attr_mutual_info(z: numpy.ndarray, a: numpy.ndarray, discrete: bool = False) → numpy.ndarray

Calculate mutual information between latent vectors and a target attribute.

Parameters

• z (np.ndarray, (n_samples, n_features)) – a batch of latent vectors

• a (np.ndarray, (n_samples,)) – a batch of one attribute

• discrete (bool, optional) – whether the attribute is discrete, by default False

Returns

mutual information between each latent vector dimension and the attribute

Return type

np.ndarray, (n_features,)

latte.functional.disentanglement.mutual_info.single_mutual_info(a: numpy.ndarray, b: numpy.ndarray, discrete: bool) → float

Calculate mutual information between two variables

Parameters

• a (np.ndarray, (n_samples,)) – a batch of a feature variable

• b (np.ndarray, (n_samples,)) – a batch of a target variable

• discrete (bool, optional) – whether the target variable is discrete, by default False

Returns

mutual information between the variables

Return type

float

latte.functional.disentanglement.mutual_info.entropy(a: numpy.ndarray, discrete: bool = False) → float

Calculate entropy of a variable

Parameters

• a (np.ndarray, (n_samples,)) – a batch of the variable

• discrete (bool, optional) – whether the variable is discrete, by default False

Returns

entropy of the variable

Return type

float

latte.functional.disentanglement.mutual_info.conditional_entropy(ai: numpy.ndarray, aj: numpy.ndarray, discrete: bool = False) → float

Calculate conditional entropy of a variable given another variable.

\[\mathcal{H}(a_i|a_j) = \mathcal{H}(a_i) - \mathcal{I}(a_i, a_j),\]

where \(\mathcal{I}(\cdot,\cdot)\) is mutual information, and \(\mathcal{H}(\cdot)\) is entropy.

Parameters

• ai (np.ndarray, (n_samples,)) – a batch of the first variable

• aj (np.ndarray, (n_samples,)) – a batch of the conditioning variable

• discrete (bool, optional) – whether the variables are discrete, by default False

Returns

conditional entropy of ai given aj.

Return type

float

latte.functional.disentanglement.mutual_info.mig(z: numpy.ndarray, a: numpy.ndarray, reg_dim: Optional[List] = None, discrete: bool = False, fill_reg_dim: bool = False) → numpy.ndarray

Calculate Mutual Information Gap (MIG) between latent vectors and attributes.

Mutual Information Gap measures the degree of disentanglement. For each attribute, MIG is calculated by difference in the mutual informations between that of the attribute and its most informative latent dimension, and that of the attribute and its second-most informative latent dimension. Mathematically, MIG is given by

\[\operatorname{MIG}(a_i, \mathbf{z}) = \dfrac{\mathcal{I}(a_i, z_j)-\mathcal{I}(a_i, z_k)}{\mathcal{H}(a_i)},\]

where \(j=\operatorname{arg}\max_n \mathcal{I}(a_i, z_n)\), \(k=\operatorname{arg}\max_{n≠j} \mathcal{I}(a_i, z_n)\), \(\mathcal{I}(\cdot,\cdot)\) is mutual information, and \(\mathcal{H}(\cdot)\) is entropy. If reg_dim is specified, \(j\) is instead overwritten to reg_dim[i], while \(k=\operatorname{arg}\max_{n≠j} \mathcal{I}(a_i, z_n)\) as usual.

MIG is best applied for independent attributes.

See also

dmig

Dependency-Aware Mutual Information Gap

xmig

Dependency-Blind Mutual Information Gap

dlig

Dependency-Aware Latent Information Gap

Parameters

• z (np.ndarray, (n_samples, n_features)) – a batch of latent vectors

• a (np.ndarray, (n_samples, n_attributes) or (n_samples,)) – a batch of attribute(s)

• reg_dim (Optional[List], optional) – regularized dimensions, by default None Attribute a[:, i] is regularized by z[:, reg_dim[i]]. If reg_dim is provided, the first mutual information is always taken between the regularized dimension and the attribute and MIG may be negative.

• discrete (bool, optional) – Whether the attributes are discrete, by default False

• fill_reg_dim (bool, optional) – Whether to automatically fill reg_dim with range(n_attributes), by default False

Returns

MIG for each attribute

Return type

np.ndarray, (n_attributes,)

References

1

17. Chen, X. Li, R. Grosse, and D. Duvenaud, “Isolating sources of disentanglement in variational autoencoders”, in Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018.

latte.functional.disentanglement.mutual_info.dmig(z: numpy.ndarray, a: numpy.ndarray, reg_dim: Optional[List] = None, discrete: bool = False) → numpy.ndarray

Calculate Dependency-Aware Mutual Information Gap (DMIG) between latent vectors and attributes

Parameters

• z (np.ndarray, (n_samples, n_features)) – a batch of latent vectors

• a (np.ndarray, (n_samples, n_attributes) or (n_samples,)) – a batch of attribute(s)

• reg_dim (Optional[List], optional) – regularized dimensions, by default None Attribute a[:, i] is regularized by z[:, reg_dim[i]]. If None, a[:, i] is assumed to be regularized by z[:, i].

• discrete (bool, optional) – Whether the attributes are discrete, by default False

Returns

DMIG for each attribute

Return type

np.ndarray, (n_attributes,)

References

1

11. 14. Watcharasupat and A. Lerch, “Evaluation of Latent Space Disentanglement in the Presence of Interdependent Attributes”, in Extended Abstracts of the Late-Breaking Demo Session of the 22nd International Society for Music Information Retrieval Conference, 2021.

2

11. 14. Watcharasupat, “Controllable Music: Supervised Learning of Disentangled Representations for Music Generation”, 2021.

latte.functional.disentanglement.mutual_info.dlig(z: numpy.ndarray, a: numpy.ndarray, reg_dim: Optional[List] = None, discrete: bool = False)

Calculate Dependency-Aware Latent Information Gap (DLIG) between latent vectors and attributes

Parameters

• z (np.ndarray, (n_samples, n_features)) – a batch of latent vectors

• a (np.ndarray, (n_samples, n_attributes)) – a batch of at least two attributes

• reg_dim (Optional[List], optional) – regularized dimensions, by default None Attribute a[:, i] is regularized by z[:, reg_dim[i]]. If None, a[:, i] is assumed to be regularized by z[:, i].

• discrete (bool, optional) – Whether the attributes are discrete, by default False

Returns

DLIG for each attribute

Return type

np.ndarray, (n_attributes,)

References

1

11. 14. Watcharasupat, “Controllable Music: Supervised Learning of Disentangled Representations for Music Generation”, 2021.

latte.functional.disentanglement.mutual_info.xmig(z: numpy.ndarray, a: numpy.ndarray, reg_dim: Optional[List] = None, discrete: bool = False)

Calculate Dependency-Blind Mutual Information Gap (XMIG) between latent vectors and attributes

Parameters

• z (np.ndarray, (n_samples, n_features)) – a batch of latent vectors

• a (np.ndarray, (n_samples, n_attributes) or (n_samples,)) – a batch of attribute(s)

• reg_dim (Optional[List], optional) – regularized dimensions, by default None Attribute a[:, i] is regularized by z[:, reg_dim[i]]. If None, a[:, i] is assumed to be regularized by z[:, i].

• discrete (bool, optional) – Whether the attributes are discrete, by default False

Returns

XMIG for each attribute

Return type

np.ndarray, (n_attributes,)

References

1

11. 14. Watcharasupat, “Controllable Music: Supervised Learning of Disentangled Representations for Music Generation”, 2021.