latte.functional.disentanglement.mutual_info
Module Contents
Functions
|
Calculate Mutual Information Gap (MIG) between latent vectors and attributes. |
|
Calculate Dependency-Aware Mutual Information Gap (DMIG) between latent vectors and attributes |
|
Calculate Dependency-Aware Latent Information Gap (DLIG) between latent vectors and attributes |
|
Calculate Dependency-Blind Mutual Information Gap (XMIG) between latent vectors and attributes |
- mig(z, a, reg_dim=None, discrete=False, fill_reg_dim=False)
Calculate Mutual Information Gap (MIG) between latent vectors and attributes.
Mutual Information Gap measures the degree of disentanglement. For each attribute, MIG is calculated by difference in the mutual informations between that of the attribute and its most informative latent dimension, and that of the attribute and its second-most informative latent dimension. Mathematically, MIG is given by
\[\operatorname{MIG}(a_i, \mathbf{z}) = \dfrac{\mathcal{I}(a_i, z_j)-\mathcal{I}(a_i, z_k)}{\mathcal{H}(a_i)},\]where \(j=\operatorname{arg}\max_n \mathcal{I}(a_i, z_n)\), \(k=\operatorname{arg}\max_{n≠j} \mathcal{I}(a_i, z_n)\), \(\mathcal{I}(\cdot,\cdot)\) is mutual information, and \(\mathcal{H}(\cdot)\) is entropy.
If reg_dim is specified, \(j\) is instead overwritten to reg_dim[i], while \(k=\operatorname{arg}\max_{n≠j} \mathcal{I}(a_i, z_n)\) as usual.
MIG is best applied for independent attributes.
- Parameters:
z (np.ndarray, (n_samples, n_features)) – a batch of latent vectors
a (np.ndarray, (n_samples, n_attributes) or (n_samples,)) – a batch of attribute(s)
reg_dim (Optional[List], optional) – regularized dimensions, by default None Attribute a[:, i] is regularized by z[:, reg_dim[i]]. If reg_dim is provided, the first mutual information is always taken between the regularized dimension and the attribute, and MIG may be negative.
discrete (bool, optional) – Whether the attributes are discrete, by default False
fill_reg_dim (bool, optional) – Whether to automatically fill reg_dim with range(n_attributes), by default False. If fill_reg_dim is True, the reg_dim behavior is the same as the dependency-aware family. This option is mainly used for compatibility with the dependency-aware family in a bundle.
- Returns:
MIG for each attribute
- Return type:
np.ndarray, (n_attributes,)
See also
References
[1]Chen, X. Li, R. Grosse, and D. Duvenaud, “Isolating sources of disentanglement in variational autoencoders”, in Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018.
- dmig(z, a, reg_dim=None, discrete=False)
Calculate Dependency-Aware Mutual Information Gap (DMIG) between latent vectors and attributes
Dependency-Aware Mutual Information Gap (DMIG) is a dependency-aware version of MIG that accounts for attribute interdependence observed in real-world data. Mathematically, DMIG is given by
\[\operatorname{DMIG}(a_i, \mathbf{z}) = \dfrac{\mathcal{I}(a_i, z_j)-\mathcal{I}(a_i, z_k)}{\mathcal{H}(a_i|a_l)},\]where \(j=\operatorname{arg}\max_n \mathcal{I}(a_i, z_n)\), \(k=\operatorname{arg}\max_{n≠j} \mathcal{I}(a_i, z_n)\), \(\mathcal{I}(\cdot,\cdot)\) is mutual information, \(\mathcal{H}(\cdot|\cdot)\) is conditional entropy, and \(a_l\) is the attribute regularized by \(z_k\). If \(z_k\) is not regularizing any attribute, DMIG reduces to the usual MIG. DMIG compensates for the reduced maximum possible value of the numerator due to attribute interdependence.
If reg_dim is specified, \(j\) is instead overwritten to reg_dim[i], while \(k=\operatorname{arg}\max_{n≠j} \mathcal{I}(a_i, z_n)\) as usual.
- Parameters:
z (np.ndarray, (n_samples, n_features)) – a batch of latent vectors
a (np.ndarray, (n_samples, n_attributes) or (n_samples,)) – a batch of attribute(s)
reg_dim (Optional[List], optional) – regularized dimensions, by default None Attribute a[:, i] is regularized by z[:, reg_dim[i]]. If None, a[:, i] is assumed to be regularized by z[:, i].
discrete (bool, optional) – Whether the attributes are discrete, by default False
- Returns:
DMIG for each attribute
- Return type:
np.ndarray, (n_attributes,)
See also
References
[1]Watcharasupat and A. Lerch, “Evaluation of Latent Space Disentanglement in the Presence of Interdependent Attributes”, in Extended Abstracts of the Late-Breaking Demo Session of the 22nd International Society for Music Information Retrieval Conference, 2021.
[2]Watcharasupat, “Controllable Music: Supervised Learning of Disentangled Representations for Music Generation”, 2021.
- dlig(z, a, reg_dim=None, discrete=False)
Calculate Dependency-Aware Latent Information Gap (DLIG) between latent vectors and attributes
Dependency-aware Latent Information Gap (DLIG) is a latent-centric counterpart to DMIG. DLIG evaluates disentanglement of a set of semantic attributes \(\{a_i\}\) with respect to a latent dimension \(z_d\) such that
\[\operatorname{DLIG}(\{a_i\}, z_d) = \dfrac{\mathcal{I}(a_j, z_d)-\mathcal{I}(a_k, z_d)}{\mathcal{H}(a_j|a_k)},\]where \(j=\operatorname{arg}\max_i \mathcal{I}(a_i, z_d)\), \(k=\operatorname{arg}\max_{i≠j} \mathcal{I}(a_i, z_d)\), \(\mathcal{I}(\cdot,\cdot)\) is mutual information, and \(\mathcal{H}(\cdot|\cdot)\) is conditional entropy.
If reg_dim is specified, \(j\) is instead overwritten to reg_dim[i], while \(k=\operatorname{arg}\max_{i≠j} \mathcal{I}(a_i, z_d)\) as usual.
- Parameters:
z (np.ndarray, (n_samples, n_features)) – a batch of latent vectors
a (np.ndarray, (n_samples, n_attributes)) – a batch of at least two attributes
reg_dim (Optional[List], optional) – regularized dimensions, by default None Attribute a[:, i] is regularized by z[:, reg_dim[i]]. If None, a[:, i] is assumed to be regularized by z[:, i].
discrete (bool, optional) – Whether the attributes are discrete, by default False
- Returns:
DLIG for each attribute-regularizing latent dimension
- Return type:
np.ndarray, (n_attributes,)
See also
References
[1]Watcharasupat, “Controllable Music: Supervised Learning of Disentangled Representations for Music Generation”, 2021.
- xmig(z, a, reg_dim=None, discrete=False)
Calculate Dependency-Blind Mutual Information Gap (XMIG) between latent vectors and attributes
Dependency-blind Mutual Information Gap (XMIG) is a complementary metric to MIG and DMIG that measures the gap in mutual information with the subtrahend restricted to dimensions which do not regularize any attribute. XMIG is given by
\[\operatorname{XMIG}(a_i, \mathbf{z}) = \dfrac{\mathcal{I}(a_i, z_j)-\mathcal{I}(a_i, z_k)}{\mathcal{H}(a_i)},\]where \(j=\operatorname{arg}\max_d \mathcal{I}(a_i, z_d)\), \(k=\operatorname{arg}\max_{d∉\mathcal{D}} \mathcal{I}(a_i, z_d)\), \(\mathcal{I}(\cdot,\cdot)\) is mutual information, \(\mathcal{H}(\cdot)\) is entropy, and \(\mathcal{D}\) is a set of latent indices which do not regularize any attribute. XMIG allows monitoring of latent disentanglement exclusively against attribute-unregularized latent dimensions.
If reg_dim is specified, \(j\) is instead overwritten to reg_dim[i], while \(k=\operatorname{arg}\max_{d∉\mathcal{D}} \mathcal{I}(a_i, z_d)\) as usual.
- Parameters:
z (np.ndarray, (n_samples, n_features)) – a batch of latent vectors
a (np.ndarray, (n_samples, n_attributes) or (n_samples,)) – a batch of attribute(s)
reg_dim (Optional[List], optional) – regularized dimensions, by default None Attribute a[:, i] is regularized by z[:, reg_dim[i]]. If None, a[:, i] is assumed to be regularized by z[:, i].
discrete (bool, optional) – Whether the attributes are discrete, by default False
- Returns:
XMIG for each attribute
- Return type:
np.ndarray, (n_attributes,)
See also
References
[1]Watcharasupat, “Controllable Music: Supervised Learning of Disentangled Representations for Music Generation”, 2021.