BrianKearns

WhatWhere

http://whatwhere.caltech.edu

To Do

StickyPics

Goal

MISSION: Connecting people via pictures

• Connect pictures with other pictures in an intelligent way
• Connect pictures+captions with URLs that contain similar pictures
• Allow navigation of picture database by image similarity
• Connect pictures to electronic maps and to web sites related to those pictures
• Connect people who take similar or related pictures

Tasks

• Create web site where one can easily upload a collection of pictures, or single pictures. E.g. by emailing picture, by direct file upload, by uploading zipped directory containing pictures, by emailing pictures from phones.
• Make it possible to comment the pictures, or to import comments that are already attached to pictures.
• Get Brown and Lowe code from Lihi Zelnik. Use it to find correspondences between pictures and stitch them together. Brown and Lowe code may be downloaded from David Lowe's web site as well.
• Pierre Moreels can give you efficient C code to compute the SIFT features (these are used in Brown and Lowe).
• Avoid getting tangled up into issues of `stitching' (how to make the seam betw. 2 images not be visible)
• Develop user interface to allow people to view related images (e.g. `infinite' strip)
• Develop way to take words from image comments and use those words to search the web for images (e.g. from Google Images and Flickr). Discover commonalities between query image and images returned from web search. Attach to the query image in your database the URL of pages where those matching pictures appear.
• Think of interesting ways to aggregate pictures: e.g. pictures of same object/scene, pictures that were all taken approx. at the same time, ...

Literature

• Brown and Lowe - Autostitch (place URL of PDF file here)

http://www.cs.ubc.ca/~mbrown/papers/iccv2003.pdf http://www.cs.ubc.ca/~mbrown/papers/bmvc2002.pdf http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html

• Lowe - IJCV 2004 paper on SIFT features
• Cipolla 2004(05?) paper on locating a camera via the picture it takes
• Szeliski

http://research.microsoft.com/~szeliski/publications.htm http://www.research.microsoft.com/scripts/pubs/view.asp?TR_ID=MSR-TR-2004-133 http://www.research.microsoft.com/vision/visionbasedmodeling/publications/MSR-TR-2004-92-Jan26.pdf

• Pajdla http://cmp.felk.cvut.cz/projects/benogo/
• Peleg http://www.cs.huji.ac.il/~peleg/papers/tips05-blending.pdf

Where to get pictures

• Get the web site running, so that all of us can upload our pictures
• Take pictures from

http://www.vision.caltech.edu/Image_Datasets/MtWilson/ http://www.vision.caltech.edu/Image_Datasets/Mosaics/

Demonstrations

• Upload single picture
• Upload zipped directory of pictures
• Add comments to pictures on your server (or maybe extract comments from picture's metadata)
• Extract from picture's metadata the date in which it was taken and all sorts of comments which may be useful
• Load into the server the pictures in http://www.vision.caltech.edu/Image_Datasets/Mosaics/GrandCanal and label those pictures with the tags `Grand Canal Venice' and `Canal Grande Venezia'. Use such tags to download 1000 pictures from Gooogle Images and try and find matches.
• Do same with directory `VeronaMonuments'. Add tags `Verona' and see what happens.

Related Sites

• Flickr, geotagged images
• ...