Ronnie's wiki

previous life

• slides from my lab meeting talk
• OHBM poster on classifying fMRI data

current life: Face Similarity

I'm working on the Face Similarity Data.

The goal of my project is predicting similarity of faces, as defined by human observers.

The metric I use is predicting which k pictures of faces in a collection are most similar to a given query face.

Approaches

I'm trying to implement the combinatorial nearest neighbor search described above (previous life) to classify images of faces. Here are some ideas that are currently undergoing various phases of debugging:

1. Choose 1 feature at a time and determine nearest neighbors according to that feature (with a Euclidean distance metric). Compare those nearest neighbors to the nearest neighbors in psych space. Do this for every feature, and select the one that predicts psych distance best (metric specified below). Now repeat this process with that "best" feature in combination with the rest of the features. Choose the one that improves performance the most. Continue indefinitely.
2. Choose a random subset of features and determine nn according to that subset. Assign each feature in the subset a goodness score based on the performance of that subset. In subsequent iterations, bias selection toward better performing features.
3. Choose a random subset of features and add it to the list of currently-used-features. If adding these features improve performance, keep them, otherwise remove them from the list.

Results

Performance of the classifier as it selects features. Blue line for training. The green line is performance when the features selected for the training data are used to predict nearest neighbors on the second half of the dataset.

1. Choose 1 feature at a time: The graph in the first figure shows how the classifier performs as it selects features. In this case, I chose to use a neighborhood size of k = 1, so the classifier was trying to find the 1 best match for each face. (Similar results hold for k = 5.) The blue line is performance on training data, which was half of the data. The green line is performance when the features selected for the training data are used to predict nearest neighbors on the second half of the dataset. Peak generalization performance is 20%, meaning that for the 50 images in the test data, the classifer chose 10 that agreed with the human data. This strikes me as significantly above chance, which is 1%.

The most impressive example pair of faces from the final iteration (where the classifier performed at 10%). The left is the target face and the right is the face chosen by the classifier.

The second figure shows the most impressive example pair of faces from the final iteration (where the classifier performed at 10%). The left is the target face and the right is the face chosen by the classifier. One limitations of this analysis: The nearest neighbors are inferred from the psych data, but this process doesn't incorporate how the data has been partitioned into training and test data. Therefore, sometimes the nearest neighbor according to the psych data is outside the test dataset, so it is impossible for the classifer to find it. Similarly, sometimes the 'true' nearest neighbor is outside the test data, so hopefully fixing this will improve performance.

Extensions

1. Make this a gradient descent method. Choosing a random subset of voxels is the same thing as multiplying the features vector by a binary matrix A. I need to compute dE/DA, where E is error metric, such as 100-%overlapping nearest neighbors in Ax and psych space.
2. Make A continuous
3. Consider distance metrics on the nonlinear manifold determined through LLE or Isomap.
4. Use geometric features: pairwise distances between keypoint locations

future life

1. Combine with fMRI pattern analysis