Karol Gregor

Title:

Deep belief networks of restricted Boltzmann machines: Tutorial and simple experiments.

Abstract:

The bulk of human intelligence is located in cortex. While its various parts seem to be responsible for various aspects of our intelligence, anatomically it appears rather homogeneous: it is a neural network that (probably) uses roughly the same algorithm which is repeated in a hierarchical fashion. It is thus a worthwhile pursuit to develop a rather general algorithm that can be repeated in hierarchical structure and be used for various applications. Hinton et. al. recently discovered a fast learning algorithm for deep networks of restricted Boltzmann machines, achieving state of the part performance on several problems (e.g. handwritten digit recognition and document retrieval), opening an important avenue for building machine learning algorithms in this fashion. This talk is to a large part a tutorial: How the algorithm works, how to use it in several contexts and a review of several of the Hinton's results. However I also show some simple instructive experiments of my own. The most interesting of these are the ones dealing with temporal sequences of patterns where the algorithm extracts invariance automatically in a toy system. I give some thoughts about how to build this into multilayer networks, how this fits into the framework of the cortical computation and how to put several ingredients together to build a useful complex structure.