Deakin University

File(s) under permanent embargo

Learning to remember more with less memorization

conference contribution
posted on 2019-01-01, 00:00 authored by Thai Hung Le, Truyen TranTruyen Tran, Svetha VenkateshSvetha Venkatesh
© 7th International Conference on Learning Representations, ICLR 2019. All Rights Reserved. Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize that this scheme of writing is suboptimal in memory utilization and introduces redundant computation. To validate our hypothesis, we derive a theoretical bound on the amount of information stored in a RAM-like system and formulate an optimization problem that maximizes the bound. The proposed solution dubbed Uniform Writing is proved to be optimal under the assumption of equal timestep contributions. To relax this assumption, we introduce modifications to the original solution, resulting in a solution termed Cached Uniform Writing. This method aims to balance between maximizing memorization and forgetting via overwriting mechanisms. Through an extensive set of experiments, we empirically demonstrate the advantages of our solutions over other recurrent architectures, claiming the state-of-the-arts in various sequential modeling tasks.



Learning Representations. International Conference (7th : 2019 : New Orleans, Louisiana)




New Orleans, Louisiana

Place of publication

[New Orleans, Louisiana]

Start date


End date




Publication classification

E1 Full written paper - refereed

Copyright notice

2019, 7th International Conference on Learning Representations, ICLR 2019

Title of proceedings

ICLR 2019: Proceedings of the 7th International Conference on Learning Representations

Usage metrics

    Research Publications


    No categories selected