Ryan's wiki

Research Interests

Model Selection and Incremental Learning

Most visual recognition algorithms use an a priori model structure. A training set is then used to learn appropriate values of the model parameters. How can machines learn model structure itself from data/environment? Multiple candidate models can be evaluated according to a selection criterion. Popular model selection criteria include Minimum Description Length and BDE score (based on the Bayesian posterior probability of each candidate model given the data). (Friedman and Goldszmidt, 1998) Intuitively, model selection criteria contain a term that rewards models that fit the data and a term that penalizes model complexity. They embody Occam's razor: the simplest model that fits the data is preferred. Alternatively, cross validation can be used for model selection.

A naive approach:

• Generate a number of candidate models in parallel
• Fit each model to the training set independently
• Use model selection criterion to pick the best model.

Naturally, this can be prohibitively expensive. An alternate approach is to use an algorithm that starts with a simple model and progressively adds complexity as required by the data. As an example, Verbeek et al. (2003) developed an efficient greedy algorithm for learning Gaussian mixture models. Mixture components are inserted one at a time, and a model selection criterion can be evaluated after each step to determine the optimal number of components. Can this type of approach be extended to work with complex visual recognition models?

Incremental Learning

The above discussion is based on the assumption that a complete training set is available. Behaving organisms face a different challenge: their visual recognition systems must incorporate new representations as they are encountered in the world. It is not feasible to store the complete input pattern history as it accumulates. An incremental learning system must

"...address a learning scenario in which limited memory is available such that after a new data point is incorporated in the learning system it is discarded and cannot be re-used." (Schaal & Atkeson, 1997)

Incremental learning systems must deal with the problem of "negative interference." (Schaal & Atkeson, 1997) Newly arriving data points may cause parameter updates that compromise classification performance on previously learned examples.

Examples of Incremental Machine Learning:

• "Incremental and Decremental Support Vector Machine Learning." (Cauwenberghs & Poggio, 2001). A supervised discriminative classifier that can be trained incrementally. Addresses the negative interference problem by introducing "adiabatic" parameter updates. Could this form a building block in a recognition system that can add new classes incrementally?
• "Incremental Learning of Temporally-Coherent Gaussian Mixture Models." (Arandjelovic & Cipolla, 2005). Fits a gaussian mixture model as data points arrive one at a time. In general there is no unique solution to this problem, so the authors impose "temporal coherence" to the incoming data points; i.e. data points vary smoothly with time. At each data point arrival, the algorithm uses MDL to decide whether or not to introduce a new component or to merge components. Could a similar approach be designed for more complex generative models? The temporal coherence assumption may not be restrictive when learning from continous video streams. Interestingly, temporal coherence has been used as an objective function for learning complex cells from natural video (Kayser et al, 2001) and "view-point invariant object representations." (Einhauser et al, 2005)
• "A Constructive, Incremental-learning Network for Mixture Modeling and Classification." (Williamson, 1997).
• "Incremental Neural Learning using Adaboost." (Murphey et al, 2002).

Another challenge/opportunity in incremental learning is to take advantage of previously learned categories in order to decrease the number of training examples required to learn a new class.