Heiko's wiki

Clustering of -/- and w+ lines

Cluster Results incl. Wing Information without SVD, Reference Frame: Fly 1,2, 8 Clusters

Cluster Results incl. Wing Information, Reference Frame: Fly 1,2, 8 Clusters

Cluster Results incl. Wing Information, Reference Frame: Fly 1,2, 4 Clusters

Clustering of Lunge Situations

Examples for Lunges (at the end of each movie)

Clustered Lunges, Reference Frame: Fly 1,2, 8 Clusters

Clustered Lunges incl. Wing Information, Reference Frame: Fly 1,2, 8 Clusters

Some Results of EM Clustering

• taking transformed (x,y) for both flies, plus:
  • difference between head-direction of fly a and center of fly b
  • distance and its derivative between both flies

1. Reference: Fly 1

• local maxima detection in velocities of fly 1
• 64 clusters

Cluster Results, Reference Frame: Fly 1

2. Reference: Fly 2

• local maxima detection in velocities of fly 2
• 16 clusters (due to less keypoints)

Cluster Results, Reference Frame: Fly 2

3. Reference: Fly 1 & 2

• local maxima detection in mean velocities of fly 1 + fly 2
• 64 clusters

Cluster Results, Reference Frame: Fly 1,2

Clustering for all available Movies

Reference: Fly 1

• local maxima detection in velocities of fly 1
• 128 clusters --> reduced to 104 by removing low-probabilty clusters

Cluster Results, Reference Frame: Fly 1

Automated Monitoring and Analysis of the Complex Behavior of Fruit Flies

Inroduction

It is intended first to develop methods to "deconstruct" simple locomotor behavior into its "bout" structure, in populations of flies. Currently this can only be done with single flies using existing programs. Extracting bout structure from fly populations will allow us, for the first time, to determine the effect of social interactions on locomotor behavior. We will also use this approach to better dissect the effects of internal and external stimuli known to affect locomotion, such as hunger, arousal, mechanical startle, negative geotaxis, and airflow. A more ambitious goal is to develop methods that will automatically recognize components of complex behavior, such as aggression or courtship. For example courtship is composed of a stereotyped sequence of male behaviors including orientation towards the female, wing extension and vibration (courtship "song"), licking of the genitalia and finally copulation. Currently these can only be scored manually, which is both laborious and only semi-quantitative. The development of algorithms for automatic recognition of such behaviors would represent an enormous step forward, for it would allow the effects of genetic and neural circuit-level manipulations of behavior to be quantified and analyzed at a level not previously achieved. This in turn should provide important new insights into the genetic and neural control of behavior. Finally, the algorithms developed from this study could in principle be extended to other organisms closer to man, such as the mouse.

Experimental Set-Up

Fig. 1 shows our first attempt of an experimental set-up for studying aggression of Drosophila melanogaster.

Fig. 1: Experimental Setup

Fig. 2: Matrix of Fruit-Fly Chambers

Fig. 3: Recording Hard- & Software

The set-up consists of a matrix of small chambers (see Fig. 2), each with a diameter of approx. 1.1 cm and a height of approx. 2 cm, designed for experiments with two male flies in each chamber. The chambers are filled with a mixture of agar, apple juice and sugar to half of its heights. On top of it, in the center, a drop of yeast is placed to increase the aggression level of both flies. The chamber-matrix is placed above a nearly homogenous background light. Additionally the chambers are illuminated by 2 (or more) cold-light sources (approx. 5300 K). All chambers are closed by a lid after placing the fly-couples into them. To avoid condensation at the chambers walls due to the heating background light and the higher absolute air humidity lavel within the closed chambers, firstly the matrix is raised approx. 5 cm above the background light, and secondly a fan is placed beside the set-up. The video-imaging system consists of a 2 Mega pixel (1600x1280) CCD video camera, which is placed perpendicular above the chambers. The camera is connected via USB 2.0 interface to a personal computer and allows digitization of time series of optical image sequences. With a focal length of 50 mm with the actually present mount lens the minmum distance between flies and imaging sensor is approx. 1.2 m. Therefore a much larger area around the matrix is imaged. The ROI is captured and stored onto a hard disk using a common software package (Fig. 3). Using image compression techniques reasonable sizes of the data sets are reachable. The image shown in Fig. 4 belongs to a data set of 30 minutes length with a spatial resolution of 704x360 pixel, a color depth of 24 bit, and a frame rate of 27 frames/s (336 kBit/s, used codec: MS-WMV9). It is clearly visible that compression effects take place here.

The following improvements would be necessary:

- a white LED-light array for homogeneous, reflection-free illumination of all chambers (e.g. http://www.nerlite.com/Area.html, and/or http://www.nerlite.com/Ring.html)

- a compensation between chambers internal air and outside environment to avoid condensation (e.g. fine nets) or heating of the arrays lid

- a professional plexi-glas chamber-matrix with clear, scratch-free surface

- applying a high-speed CCD camera, using common interfaces like Firewire (IEEE.1394a - 400 MBit/s, IEEE.1394b/Fibre - 800 MBit/s). They are will limit the maximum frame rate. Assuming 400 MBit/s as standard interface, a resolution of 640x480 with 8 Bit color/gray-scale channels gives a maximum frame rate of approx. 170 frames/s (possible camera: http://www.imagelabs.com/components/camera/detail.php?id=26). Using Gigabit ethernet link very high data rates are possible, e.g., for setting up a camera with a data rate of up 200 frames/s (e.g. http://www.imagelabs.com/components/camera/detail.php?id=205 (B&W)). With a Data link interface (via frame-grabber card) even higher data rates would be possible (e.g. http://www.imagelabs.com/components/camera/detail.php?id=21)

- apropriate mounting lenses (higher focal length, zoom?) to use the cameras full resolution for imaging the chambers with flies

- other, more effective compression techniques, e.g. H.264, for reduction of compression-effects and a limited size of the data sets

High-Speed Video-Image Sequences

First test using a high-speed video camera (Basler A602f, B&W, http://www.imagelabs.com/components/camera/detail.php?id=27, ca. USD 1500) are currently ongoing. The camera is lent by the Dickinson lab for 1-2 weeks. Are color version is also available, which, on one hand side, might be very helpful for recognition of parts of the fly body, e.g. the red-looking eyes. On the other hand side, color images would increase the amount of data by a factor of 3.

The actual image quality using this set-up is impressive. All the important details like wings, legs, and eyes are clearly visible.

A small, single chamber with two male flies was prepared to study the aggressive behavior between them. The distance between chamber and camera is only approx. 10 cm. Using a simple, special capturing software from the Dickinson Lab (by Andrew Straw) it is possible to select a part of the whole imaging area for capturing. In this set-up frame rates of more than 400 fps are possible. Using this configuration video-image sequences at normal fram rates (30-60 fps) were recorded for typical 30 minute records. Further data sets at frame rates of 200-250 fps were recorded (data rate: approx. 5 MB/s; a 30-minutes study would cost approx. 9 GB). The data are saved in a special binary format, which can be transformed in the common AVI-format.

The high-speed data sets provide detailed information regarding the flies behavior during aggressive events like tusseling, boxing, or fighting. The legs movements can be seen quasi in slow motion. Sample images are provided very soon ... (probably during this weekend)

Ground-Truth Data

For the development of automatic pattern-recognition algorithms it is essential to have ground-truth information. In case of behavioral studies on fruit flies this is performed by manually analyzing the recorded image sequences. To reduce the efforts a captioning software is used, which makes it possible to go for- and backward within each data set, mark all visible events (start- and end frame), and classify them into four groups: fighting, chasing and lunging, boxing and tusseling, and other. All collected information are saved into a ASC file for comparison with the result retrieved by the pattern regognition algorithms. Fig. 4 shows a screenshot of the actual version. The software is available under Windows, Linux, Mac OS, and running stand-alone. Changing of files within one session is possible.

Fig. 4: Captioning Software

First Attempts on Pattern Analysis in Fruit Fly Behavior & Outlook

In a first attempt the spatio-temporal energy within the video-image sequences is analyzed. This is done by determining velocities, accelerations, deviations of accelerations etc. at active areas (where spatio-temporal changes are measurable) within the areas of investigation (chambers).

We have already labeled some better sequences, so this is a currently ongoing process. Li-Ming is analyzing more sequences with just one chamber at this moment, at normal and high-speed. Some samples using a big chamber were prepared with just many moving flies for later testing of the tracking algorithms.

We found that it is quite hard, even for an experienced biologist, to see if there is an event starting or not. E.g., sometimes the flies start kind of boxing for the fraction of a second, but then they break up - so there might be a need to subdivide a 'big' event into smaller ones. A composition of sub-events would then make the big event of interest. We think that the detailed information provided by the new set-up (Basler camera) are very usefull, maybe essential, for distinguishing between the different types aggressive behaviors. Beside the wing position also the position of the legs seem to be important.

Feature Extraction

A package is available now that is able to track the two flies within Li-Mings chamber and to extract and save features. This works pretty well with some restrictions, for example when the flies are really close together. In this case they are not separable (yet). The routine extracts numerous features per frame as follows:

• velocity & acceleration within the chamber (done by optical flow based motion estimation technique)
• change of distance
• direction / position of heads and tails
• directional change
• difference between head and moving direction
• flies orientation
• mean gray value of each fly
• velocity & acceleration of each fly
• area of each fly (distinguish between up or side view)
• flies distance to chamber's center
• flies relative distance (center of inertia, c-c)
• head-head & tail-tail distance
• change of distances (c-c, h-h, t-t)
• flies facing each other?
• flies in action?
• flies connected?
• wing extension: left, right, both
• sudden position change of flies (lunging)

Examples for important features are the head-head and tail-tail distances, as they describe behaviors like boxing or tussling, sudden position changes (lunging), or the acceleration. All the features are saved into a table file (frame by frame) to open a n-dimensional feature space.

An example time series of some features is visible in Fig. 5. A description of the plots is provided within the pdf-file. The 'jumps' of the head-direction are caused by uncertainties in the gray-value distribution of the flies body (e.g. an unfavorable position, lightning fluctuation due to AC-power), or an uncertainty of the flies movement (e.g. just rotating).

File:3 wmf example.pdf

The situation within the chamber is quite complex - the flies are most of the time walking at the chamber's walls, are sometimes visible from above, sometimes from beside; they can turn anywhere, and even walk at the ceiling or above each other. An improvement of the current setup might be possible with Susannes chamber (a member of Martin Heisenberg's lab), which David brought for Li-Ming and is now being tested. The chamber is quite different: quadratic outline with an area about twice as big as the little round chambers, its about 15 cm tall and the walls are painted with a certain chemical (Li-Ming will try to get it). This chemical avoids that the flies climb the walls. We worry that the flies might not fight that much, but maybe the slippery wall makes them tending to walk only within the chambers center-part, like I had seen at Susannes setup.

MCMC-based Particle Filtering

I got in contact with Frank Daellert and informed him about the purpose of our project and kindly asked him for the code just as tool for getting the important positioning information out of the data, especially to distinguish between the flies when they are connected. He now asked one of his undergrads to tidy up the code, test it, and send it to me. Today (or at the beginning of next week) we're gona get the MCMC-based particle filtering method of Khan, Balch, and Dellaert. The only restriction he gives us is that the code is only for use in our group. Further the code is written in a language called Caml, which is installed here. I will have a look on the algorithm and could rewrite it into Matlab code (when there is time), what I think will make more sense for future purposes.

First Version of Drosophila Behavior Recognition Software

The software shows the tracking-information of the two flies on-the- fly and writes all possible information about both flies out into a file. It detects lunging, wing extension, and chasing. Velocities and accelerations are based on six positioning information that are buffered in memory. The program can also be used to caption all interesting events on-the-fly (one can stop, go back and forward frame-by-frame and set a label with a description of the event, processing continues). So, there is one program that combines everything and it generates two spreadsheets (or tables): the first gives all determined parameters of both flies including the probable behavior detections, the second file has all ground-truth information. Finally there is a graphical output with many plots into a pdf-file. Figures 6-9 give some impressions how the processing looks like.

The software runs under, both, Linux and Windows. Under Windows a new cpp-routine for importing any kinds of movies (compressed or uncompressed) is implemented, which has been developed and changed for our needs with help of M. Richert (Salk Institute, USCD).

After an optimization process the program does currently run 4x slower than real-time. Further ROI-selection as in Fig. 6 is not necessary any more, its spatially and temporally self-calibrating, and the manual Label-selection as in Fig. 8 is also unnecessary. The user just needs to select a file, thats it. This could be extrended by selecting a whole directory for overnight-processing of many files.

Fig. 6: Selection of ROI

Fig. 7: Spatial Calibration

Fig. 8: Selection of Color-Labeled Fly

Fig. 9: Tracking, Behavior Analysis, and Ground-Truth Data

Next Tasks

1. Processing various movies by Li-Ming using 'Drosophila Behavior Recognition' Software; collecting of tracking, behavior, and ground-truth data
2. Verify that position, heading etc. measurements are correct. We need a ground truth here to tell us how often the measurements are very wrong, and how close to truth are they when they are on target. Unfortunately ground truth means hand-labeling of some sequences. I propose that you do this: have a Matlab tool that allows you to annotate by hand three frames every 100-200 frames (you would click on the head and tail of each fly). If each time you click on a triplet of nearby frames you can calculate position, velocity and acceleration. You may then correlate your ground truth with the measurements your software makes. I would suggest that your hand-labeled ground truth contains 50-100 triplets of frames on 3 different sequences. It is possible that the ground truth is noisy too: when you take 2nd derivatives, even small errors in clicking will cause large errors in your derivative estimates.
3. Think of all the possible features that make sense. Perhaps do this with Lihi so that you do not risk missing out on good ones. I suggest that you do not only focus on `instantaneous' features (computed on one frame at a time, or perhaps a triplet of frames) but also on features that depend on 10-20 frames at a time. E.g. the total displacement in 10 frames (this is just an example, talk about this with Lihi).
4. Obtain a carefully labeled set of motion sequences. I am afraid that it is not good enough to label 12 or 6 chambers simultaneously because you may miss important events in one chamber while you are paying attention to another one. You need to label 1 chamber at a time, unfortunately. Boring but necessary. Then you are sure of what you are doing.
5. Use nearest-neighbor classification to see how well can you predict fighting, tussling, chasing etc. Compare the results to the ground truth.
6. Once you have a good classifier, use a feature selection method to reduce the number of features, and a `smarter' classifier (e.g. SVM) to reduce the classification time.
7. Optimize processing speed: Once you have a fast and good classifier, go back to faster image processing methods and see if any gives you equivalent performance so that we have real-time fly screening.
8. Extend the program for a parallel analysis of fly-pairs within a matrix of chambers. After being sure the current program is robust enough one could, e.g., combine Lihis (or Frank Daellerts) program and our new one and get it working for a pre-defined number of chambers in a (i,j) matrix.