Deakin University
Browse
nguyen-markovinformation-2019.pdf (3.67 MB)

Markov information bottleneck to improve information flow in stochastic neural networks

Download (3.67 MB)
Version 2 2024-06-05, 11:51
Version 1 2019-11-06, 11:59
journal contribution
posted on 2024-06-05, 11:51 authored by TT Nguyen, J Choi
While rate distortion theory compresses data under a distortion constraint, information bottleneck (IB) generalizes rate distortion theory to learning problems by replacing a distortion constraint with a constraint of relevant information. In this work, we further extend IB to multiple Markov bottlenecks (i.e., latent variables that form a Markov chain), namely Markov information bottleneck (MIB), which particularly fits better in the context of stochastic neural networks (SNNs) than the original IB. We show that Markov bottlenecks cannot simultaneously achieve their information optimality in a non-collapse MIB, and thus devise an optimality compromise. With MIB, we take the novel perspective that each layer of an SNN is a bottleneck whose learning goal is to encode relevant information in a compressed form from the data. The inference from a hidden layer to the output layer is then interpreted as a variational approximation to the layer’s decoding of relevant information in the MIB. As a consequence of this perspective, the maximum likelihood estimate (MLE) principle in the context of SNNs becomes a special case of the variational MIB. We show that, compared to MLE, the variational MIB can encourage better information flow in SNNs in both principle and practice, and empirically improve performance in classification, adversarial robustness, and multi-modal learning in MNIST.

History

Journal

Entropy

Volume

21

Article number

976

Location

Basel, Switzerland

Open access

  • Yes

eISSN

1099-4300

Language

eng

Publication classification

C1 Refereed article in a scholarly journal

Copyright notice

2019, The Authors

Issue

10

Publisher

M D P I