Ilya/Paper outline

From Vision Wiki
Jump to navigation Jump to search

Abstract

(Submitted to the SURF office)

Crowdsourcing services such as Amazon Mechanical Turk have allowed humans to assist computers

in difficult tasks, such as image segmentation. We present a method for collecting labels from human annotators to complement segmentation algorithms in their task. This method includes an interface that allows users to label images as well as make minor corrections to the intermediate output of a segmentation algorithm, and a back-end system to handle user labels. The interface and back-end simplify the users’ task from a difficult one, such as drawing outlines of objects, to a much simpler one: indicating a few points from each desired region. Drawing on current work, we propose a model of the annotation process for segmentations that includes estimates of annotator expertise and uncertainty in the annotations. We discuss the quality of annotations obtained from Mechanical Turk, including cost and time efficiency. Furthermore, we show that our model can predict the ground truth segmentation while

requiring fewer crowd-provided labels than naïve methods, such as majority rules.

Outline

  • Introduction (in which I basically talk about what I talked about in my first progress report)
    • Motivation 1pr
    • Related work 1pr
  • Model:
    • Assumptions, model, inference etc 2pr
    • Distance functions (show example segmentations and distances) 2pr + some easy to make figures
    • (Basically copy the exposition from my second progress report)
  • Method:
    • Interface (discuss the two iterations of it) 1pr+2pr
      • Screenshots of Interface 2pr
      • Histograms of time taken, and time taken with each worker's first two annotations removed ("experienced" workers) Make (easy)
    • Back-end (short discussion on it) 2pr
    • Description of all the parts of my system
      • A diagram Make (with diagram I drew out before)
  • Experiments:
    • Setup: Parameters from each run On here
    • Results on synthetic data? (does the model estimate parameters correctly if assumptions are followed?)
    • Turker activity (this may be combined with the section on turker competence/model results, so you can compare model parameters with activity etc):
      • Total number of turkers per set
      • Assignments/turker per set (histogram, probably a sorted bar plot with logarithmic y-axis) Make(easy)
      • Time spent per image (are there differences between workers -- some fast, some slow? Do workers get faster at the task?) Make(easy)
    • Turker annotation quality
      • Histograms for three distance functions of distances from annotation to groundtruth Make (need groundtruth)
      • Discuss means of these distributions and what it means (bad/good annotations?)
    • Results of model
      • Convergence of inference algorithm: Histograms of annotator parameters from each set after each run of the model I get to do Make (easy)
      • Histograms of distances from mean segmentations obtained from each run to groundtruth Make (need groundtruth)
      • Discuss means of these distributions and what it means (I'm actually not sure what statistics to take on these histograms to discuss how good they are)
      • Worker activity (no. of images labeled) vs. skill (see the CVPR paper for examples)
      • Time spent on task vs skill?
  • Discussion/Conclusion
  • Acknowledgments (Perona, Peter, Maire, Visipedia collaborators, SURF program)
  • References