Shay's wiki

Introduction

Tracking Relevant papers:

1. MCMC-Based Particle Filtering for Tracking a Variable Number of Interacting Targets / Zia Khan, Tucker Balch, and Frank Dellaert

2. Multiple Mice Tracking using a Combination of Particle Filter and K-Means / Wesley Nunes Goncalves and Joao Bosco Oliveira Monteiro and Valguima Odakura

3. Tracking multiple mice through severe occlusions / Kristin Branson, Serge Belongie, Sanjoy Dasgupta

4. A Real-Time Rodent Tracking System for Both Light and Dark Cycle Behavior Analysis / Jane Brooks Zurn*+, Drew Hohmann+, Steven I. Dworkin^, Yuichi Motai

5. Identifying semi-Invariant Features on Mouse Contours / P.A. Crook, T.C. Lukins, J.A. Heward and J.D. Armstrong

6. Qualitative Modelling and Analysis of Animal Behaviour∗ / MAJA MATETI´C SLOBODAN RIBARI´C IVO IPˇ SI´C

7. A System for Tracking Laboratory Animals Based on Optical Flow and Active Contours / Zoran KalafatiC, Slobodan RibariC, Vladimir StanisavljeviC

8. MHT, Reid 1979

Behavior Relevant papers:

Experimental setup

- Arena The arena is ?? x ?? cm

- Camera: Baslar Model A622f, Max resolution : 1024x1280. Recorded at 768x1024 at 30 fps.

4 Infrared LED (type? power?)

Two males and two females (what type?)

Tracking Pipeline

- Pre-processing (background modeling?)

Various methods were tested.

Light modeling (circular, quadratic function - all failed). Since multiple lights are used it was not possible to get a good fit using a simple analytic form.

Median filtering is a problem since mice tent to stay at a specific position for long periods. Median filtering would cause them to fade in as background.

Adaptive background modeling using a Gaussian mixture model: Problematic. How to select the number of clusters for each pixel? Was ruled out in the begining because there were a lot of shadow problems. Might work now (since lighting is a bit better).

Method currently employed: Manual sampling of mice intensity values at various points in the arena. Then, interpolation and setting the threshold surface at 30% above the interpolated surface.

- Distributed processing of long sequences Sample the long video sequence every X frames. Test whether the sample frame is reliable. A reliable frame is defined as one that has N distinct blobs with properties that are within reasonable parameters.

A job is defined as tracking mice for M frames, given the initial reliable initialization.

- Job pipeline:

Segmentation

The (non linear) assignment problem.

Segmentation is an unreliable process. In each frame, we might find evidence (observation) for mouse position, or not. In addition,

Can not use Hungarian Algorithm.

Current implementation is very similar to MHT (Multiple Hypothesis tracking). Several hypothesis are generated, and each one is given a score (say, probability or likelihood). Eventually only one hypothesis remains.

Data Association (Identity Tracking)

At the end of the tracking phase we have four trajectories. Each trajectory contains the instantaneous ellipse parameters (position, scale, orientation) and image patch that was extracted at that position. A single trajectory does not necessarily correspond to a single mouse. This is because identities are mixed up when mice interact. The goal here is to solve the data association problem by finding the correct assignment between trajectories (data) and mice (identities) in each frame. In each frame there are 4! possible assignments.

Let's look at a specific frame. Four image patches (I1,I2,I3,I4) are extracted and rectified such that the mouse is always in a horizontal orientation. HOG features are extracted from each image patch (F1,F2,F3,F4).

In a pre-process step, several mice classifiers have been trained. The classifiers are 1A1, and we have 6 in total. The classifiers are constructed using Fisher Linear Discriminant. Features are projected onto a one dimensional space that is defined by the hyperplane Wi (i=1..6). The image below shows the distribution of classifier response. For first plot, for example, shows a normalized histogram of values obtained by projecting features from a training sequence that contains 10,000 frames. The histogram was actually fitted with four Gaussians (one per mouse identity). The first plot, for example, shows the histogram of values that were obtained by projecting and F1,F2,F3,F4 from all frames using W1. We see a good separation between ID1 and ID2. Notice that even though this classifier was built to discriminate between ID1 and ID2, we still know the expected values when it will be applied on ID3 and ID3.

This knowledge can be represented using 24 Gaussians. We use the following notation G(i,j) corresponds to the gaussian that was fitted for Identity (i), under classifier (j).

Now, given F1,F2,F3,F4 that were extracted from frame K, we can apply all 6 classifiers on each of the features and obtain 24 values. V(i,j) represents the response from feature i (i=1..4) using classifier j (j=1..6).

If F1,F2,F3,F4 indeed corresponded to ID1,ID2,ID3,ID4, we would expect that V(i,j) follow G(i,j). This would correspond to the default permutation (1,2,3,4). However, if the identities are actually P=(1,4,2,3) then we would expect that V(i,j) would correspond well to G(P(i),j).

Attempts to use Viterbi algorithm have failed. Why?

The problem is in the transition matrices. If we allow transitions only during intersections, then we need to deal with shitty measurements during these intervals. On top of that, Kristin pointed out an important issue which is - given same weights results in selecting the state according to the instantaneous likelihood. It pretty much discards all history.

This lead me to think....

The following observations are true: 1. Identity swaps can only occur when ellipse intersect. 2. During the intersection, measurements are extremly not accurate. Both in terms of position, ellipse shape, and identity classification due to clutter.

A possible solution is to integrate evidence over intervals with no intersection. Once the belief crosses a threshold, we can label the entire interval until the next intersection. Furthermore, we know which are the possible states to start from in the next interval. Basically : $P\left({S_{j}|O_{1}..O_{T}}\right)={\frac {\prod \limits _{i=1}^{T}{P\left({O_{i}|S_{j}}\right)}P\left({S_{j}}\right)}{\sum \limits _{k=1}^{N}{P\left({S_{k}}\right)\prod \limits _{i=1}^{T}{P\left({O_{i}|S_{k}}\right)}}}}$

The likelihood can be computed iteratively: $P\left({S_{j}|O_{1}..O_{T}}\right)={\frac {\prod \limits _{i=1}^{T-1}{P\left({O_{i}|S_{j}}\right)}P\left({O_{T}|S_{j}}\right)P\left({S_{j}}\right)}{\sum \limits _{k=1}^{N}{P\left({S_{k}}\right)\prod \limits _{i=1}^{T-1}{P\left({O_{i}|S_{k}}\right)P\left({O_{T}|S_{k}}\right)}}}}={\frac {\alpha _{j}\left({T-1}\right)P\left({O_{T}|S_{j}}\right)P\left({S_{j}}\right)}{\sum \limits _{k=1}^{N}{P\left({S_{k}}\right)\alpha _{k}\left({T-1}\right)P\left({O_{T}|S_{k}}\right)}}}$

where we define: $\alpha _{j}\left(0\right)=1$

$\alpha _{j}\left(T\right)=\alpha _{j}\left({T-1}\right)P\left({O_{T}|S_{j}}\right)$

Classifiers for mice identity

- Hog Features

Data set Algorithms tested - SVM PCA-HOG Fisher LDA

User Interface

- Repository - Correcting tracking errors GUI - Annotation GUI

Behavior

Pietro suggestions:

1. When two mice meet, it could be that both were moving at the same time, or it could be that one was stationary and the other one came to visit him. Let's separate out the two types of events. Consider the second type first. We could ask these questions: a) Is it always mouse A visitng mouse B or could both types of interaction happen equally frequently? b) Where do these encounter happen? Anywhere? Near the tubes? Near the food? c) What is the direction of approach: does the approaching mouse come from the front or from behind, or from any direction? d) Who is the first mouse who leaves the are of the encounter: the stationary mouse or the approaching one? e) Do the patterns of encounter change as the movie goes on (e.g. initially both mice are equally likely to approach each other, but at the end it is mostly A approaching B)

2. When a mouse drinks/eats, does the other mouse tend to imitate and also go and have a drink/ eat?

3. Where do the mice spend most of their time? Are there `hot spots' where the like to sit? In what order do they visit these hot spots? Are the positions of the mice correlated in any way (e.g. they avoid each other, or they try and be in the same place).

4. Can we see any influence of the circadian rythm? Any changes in average behavior throughout the movie?

5. Are there differences in speed, activity, social behavior between the different mice in the enclosure? Do females and males behave differently?

6. Can we detect automatically when the mice are fighting?

7. Can we detect automatically when mice `freeze'?

8. Can we detect automatically any other pairwise interaction?

Results

- Comparison to GT ?

Appendices

Ellipse - ellipse intersection

Programming Technical issues - Reading compressed videos - Reading AVI files larger than 2GB

Ellipse Ellipse Intersection code in matlab

I have implemented 2D ellipse-ellipse intersection test in matlab. It returns true if two ellipse intersects, and also the intersection points. You can find it in:

shayo/public/EllipseEllipseIntersection

Reading compressed video files in linux

I have created a simple matlab wrapper that uses AviFile lib. It can read compressed AVI files under linux. I managed to read XVID files. It seems to crash on MJPEG (Moran Multimedia codec).

You can take the code from: /home/shayo/public/AviFileWrapper

The equivalent of aviread(Filename,Frame) is:

hHandle = AviFileInterfaceMex('Open',Filename);

strcInfo = AviFileInterfaceMex('Info',hHandle);

AviFileInterfaceMex('Seek',hHandle, Frame);

I=AviFileInterfaceMex('GetFrame',hHandle);

AviFileInterfaceMex('Close',hHandle);