Ilya

Post-Update 2

• Peter told me the distribution is fine
• I'll probably get the groundtruths done today and start generating histograms

Post-Update 1

• Finished most of the groundtruths -- 20 left to examine, and then about 25 to do manually.
• Then I will spend a few days trying to figure out why the model still produces a bad distribution, but if I can't figure it out, I'll leave it to post-SURF
• Then I will write my paper

9/3/10

• Spending today choosing groundtruth segmentations and doing a few run-throughs of my model (then realizing that things are possibly wrong).
• Fixed a bug where my model wasn't setting up the beta priors correctly at all.

9/2/10

• Completed the code for the EM algorithm, but need to fix some bugs so that it actually runs.
• Aaaand the bugs are fixed. I'm running it through one iteration just to see that it terminates, then I'm going to put in some code to save the information and run it through 10 with Hausdorff distance.
• Played around with multiple sets and still don't quite like the parameters. I think I'll still run it through 10 iterations overnight, but now I'm thinking with smoothed Union Intersection
• Results Notebook
• A high-level outline of my final paper

9/1/10

• Implemented most of EM algorithm.

8/31/10

• Completed mean segmentation code, placed some on visualizer (where I just gave each annotator a parameter of .7). The blur has $\sigma =3(1-a_{n})$ and the weighting is $a_{n}^{4}$ right now, with the threshold being .5.

8/30/10

• Added some nice things to the visualization to make it easier to browse and make the distances easier to use
• Began work on mean segmentation

8/29/10

• Finished the distance stuff

8/27/10

• Implementing some distance functions to find a good segmentation metric. Going to include those in the visualization display.
• Implemented the Hausdorff distance (actually found some code online to do it), and implemented the intersection over union distance.
  • Hausdorff distance:

$d_{\mathrm {H} }(X,Y)=\max\{\,\sup _{x\in X}\inf _{y\in Y}d(x,y),\,\sup _{y\in Y}\inf _{x\in X}d(x,y)\,\}{\mbox{,}}\!$

• • Intersection over union distance:

$d_{U}(X,Y)={\frac {A(X\cap Y)}{A(X\cup Y)}}$

• • What I really want is some sort of inverse sigmoid, like the following, but arctanh(x) is about equal to x for x less than .9

$arctanh(1-d_{U}(X,Y))$

• • I think I'll cheat and use something arbitrary, such as:

$500(d_{U}(X,Y))^{2}+{\frac {100(d_{U}(X,Y))^{2}}{1-d_{U}(X,Y)}}$

8/26/10

• Completed the visualization tool

8/25/10

• Visualization again

8/24/10

• Continuing to read up on Cheetah, and planning out what I want my visualizations to look like.
• Wrote some code to query the server and set up some dictionaries to make generating the visuals easier.

8/23/10

• Revised the helptext based on the new interface.
• Added a checkbox to make users at least pretend they've read the instructions
• Looked into Cheetah, a Python templating library, so I can make visualizations of my data.

8/22/10

• The UCM thing seems to be functional now.

8/20/10

• Added some more robust exception handling, which should clear up any errors I was getting.
• Implemented the front-end
• Began implementing the new UCM boundary idea

8/19/10

• Lots of UI polishing
  • Continuous painting
  • Removed boundaries tool
  • Change eraser sizes, and an erase all button
  • A 'hide all' and 'show all' button
  • (Not implemented yet, but will be quick to make) Make the get_seg function time out and give the user an error if something on my backend dies
• Talked to Peter, Michael and Marco about object segmentation datasets. I will take a look at them a little bit later.

8/18/10

• Peter and I decided to change the price point to 11 cents, and now I'm getting a lot more responses. We put up 100 randomly chosen images out of the 371 with up to 5 assignments per HIT.
• Implemented click-and-drag, and am going to implement some interpolation, so that users can draw straight 'lines' with their points.

8/17/10

• I can now get my HIT up on the MTurk sandbox and have it submit the answers
• Replaced UCM display with a (UCM + .4 * Image) display and got that all in the Visipedia API server
• Put up all the images Peter gave me as HITs with 5 assignments on MTurk.

8/16/10

• Minor tweaks to the UI
• Added instructions

8/15/10

• Added keeping track of real time it takes for a user to finish
• Some minor changes to the UI based on people's suggestions

8/13/10

• Pending a few minor code modifications and writing of help text which I will need to discuss with Peter, my stuff should be more or less ready to deploy to MTurk on Monday
• All of the UCMs are in the Visipedia API database

8/12/10

• Got the Visipedia server gateway to work
• Working on adding UCM to the Visipedia database

8/11/10

• Got Visipedia API code to work on the Visipedia server (involved installing Python 2.6 and moving some packages around) and explored the API a little.
• Worked on deploying my back-end code on the Visipedia server
• Wrote my first draft of my progress report

8/10/10

• Completed the cross-domain AJAX proxy code (which was much more non-trivial than I thought it would be).

8/9/10

• Discussed and hammered out most of the rest of my model with Peter, as well as MTurk stuff.
• Receiving an account on the Visipedia server in the lab to set up my back-end code.
• Working on setting up a proxy for cross-domain AJAX stuff.
• Going to explore the Visipedia API and MTurk API.

8/6/10

• Wrote code to compute UCM tree for segmentation dataset that Peter sent me. It is crunching on 501 as I leave the lab; should be done in a few hours.
• Thought some more about the model and implementation. I will talk to Peter about this on Monday.

8/5/10

• Completed the Sinatra handler
• Also, tested and found that my code could just be dropped in a Phusion Passenger instance and just work.
• Talked to Peter. I'm going to spend the next two days re-reading the first paper he sent me last week and trying to understand how I would implement the bounding box model for my problem and trying to come up with a good metric for segmentations
• This is a Matlab function that will find the area of image objects in a binary image.

8/4/10

• Talked to Peter about what I've done the last week or so
• Almost finished the Sinatra handler (just have to fix a few bugs)
• Need to investigate Safari bug (again).

8/3/10

• The Javascript and Ruby code now poll for the completed segmentation every 500 ms instead of waiting for it, allowing other clients to continue connecting even if there is a long segmentation going on, even on a single-threaded instance of Mongrel. I may still want to play around with moving everything to mod_rails.
• Installed a local copy of Ruby using RVM, so that I could also install myself Rubygems and then Sinatra (all on vision405). Will be using Sinatra to act as a front-end for the Matlab script.

8/2/10

• Fixed all UCM bugs I could find. If I have time later, I may improve the UCM forward/back-tracking code to connect to points that are close to being along the line but not quite
• Got Phusion Passenger (AKA mod_rails) to install on my computer. Can get the segmentation tool to load in it but not necessarily run. I'll play around with that more later, but it might be a solution to mongrel's singlethreadedness.

7/30/10

• Finished all UCM addition code minus a few bugfixes which hopefully I can get finished today
• If I have time (meaning if I figure out where these bugs are), I will be devoting today and the weekend to thinking about how to parallelize Matlab, maybe finding a lightweight ruby server for the matlab stuff, and the probabilistic model.
• From my e-mail to Peter:
  • "I've found that mongrel (the light server that comes with Rails that I've been using to run the app) can only run one controller at a time (so if one user is calling segmenter#segment, another user cannot call segmenter#index). What most people do is either use Apache as a load balancer for multiple instances of mongrel going at once, or just use something like mod_rails...Either way, my Ruby code shouldn't encounter any problems if there are multiple instances running next to each other, as there are no shared resources and they shouldn't be touching each other's database entries."
  • Waiting for Peter's response to start moving on that

7/29/10

• Fixed some bugs in the front-end UCM code I discovered
• Wrote a second draft of my abstract based on Peter's recommendations
• Finished the Ruby side of UCM handling (writing to the file)
• Working on Matlab side (reading file [done], drawing the lines using Bresenham algorithm [done], tracking back or forward [not done])

7/28/10

• Implemented all front-end code necessary for making UCM additions, will start on back-end either tonight or tomorrow
• Implemented hotkeys on the front-end
• Will write first draft of my abstract today

7/27/10

• Fixed the Safari bug in a somewhat hackish way (simply remembering the dimensions from the old image and assuming that all the overlays and stuff have the same dimensions, which is a reasonable assumption to make)
• Beginning work on UCM additions

7/26/10

• Finished reading the second paper. I understand at least the concepts in both.
• Fixed the small window bug, making the CSS much better in the process.

7/23/10

• Today I am reading the two papers Peter sent me (including this one) about crowdsourcing so that I can ask him questions.

7/22/10

• The website is now fully functional (assuming that there exists a precomputed UCM for the image), except for a WebKit bug that I should fix at some point
• I'm meeting with Peter at some point today to discuss the theoretical part, as well as hopefully the MTurk and back-end DB parts.
• Things I will think about next week (from my discussion with Peter) and order of priority:
  • How to add functionality to the front-end so that users can add lines to the UCM (1)
  • Fix the Safari bug (again) (1)
  • Small window bug (1)
  • How to setup the Matlab script to be able to parallelize with other instances of the same script (2)
  • What happens when multiple users try to do segmentations at the same time. Does it block? (2)
  • A model for combining multiple annotations (2+ in parallel)
  • How to setup a small server to run the Matlab stuff so that I don't have to sftp (3)
  • Keyboard shortcuts on the front-end (4)
  • How to judge error of a segmentation, for stats later on (4)

7/21/10

• The Matlab script works now and even produces a segmentation with the same color map as the brushes in the javascript
• Working on getting the images to download to the user

7/20/10

• Wrote a Matlab script that watches for the existence of the user seeds file and generates a segmentation from that.
• Working on getting the Rails server talking to the location where the matlab script is.
• The ssh upload is working, but the matlab script is crashing.

7/19/10

• Basically finished the segmentation back-end aside from the part that actually segments the images (including talking to the DB, periodic DB cleanup, and passing along image IDs)
• Figured out some stuff in Matlab to be able to get the segmentation code I was given to give me the segmentations I want easily, so I can probably get straight to finishing that part tomorrow morning.
• From here:
  • Next step: Finish the back-end part that actually segments images
  • Two steps from now: Make this stuff ready to be packaged as an MTurk task (incl help text in front-end)
  • Three steps from now: Get the image UCM pre-computation and storage in database stuff down
  • Four steps from now: Extensions

7/16/10

• I've decided the CSRF token isn't a problem due to the fact that I'm doing AJAX requests (protected by other means) and the fact that the token is being used as a session ID anyway.
• Figured out how to get the segmentation function talking to the DB.

7/15/10

• Completed and turned in my progress report. File:IlyanepProgRep1.pdf
• I've made the first contact in getting my front-end to talk to the back-end through HTTP POST requests.
• Before I move on any further, I'd like to figure out why my Rails instance isn't actually doing the CSRF protection it should be doing.

7/14/10

• Started working on the Rails back-end. Currently working on getting the front-end working correctly with it.
• Did some SVN cleanup and let the segmentation_client folder stop being updated. Integrated everything into the back-end (the segmentation_server folder). Pushed all of the stuff, including what I've done so far on the back-end, to the repository.

7/13/10

• Fixed support for Safari and Chrome by requiring the user to click in the canvas before starting.
• I will start looking into running a Ruby on Rails server locally to begin the back-end.
• Installed Rails locally (though I will probably need to install it on a linux box), and am reading up on Rails before beginning to build the back-end Rails application.

7/12/10

• Fixed a bug where the code was trying to loop through an empty matrix and choked.
• Additional minor bugfixes where objects were not being removed from the canvas.
• Fixed the part where Internet Explorer eats my CSS and renders incorrectly. Which leads me to...
• More testing:
  • The following browsers are now tested and definitely work equivalently: Opera 10, Firefox 3, IE8
  • The following browsers most likely work: IE7
  • The following browsers most likely do not work and will not be supported: IE6
  • The following browsers are WebKit-based and I still need to fix the stupid bug for starting the page: Safari, Chrome

7/9/10

• Mostly bugfixes:
  • (Fixed) Fixed a bug where the labels from an old round weren't actually being saved correctly
  • (Almost fixed) Working on implementing the image changing when the round is changed
  • (Fixed) Implemented preloading of images -- open to improvements later.
  • (Fixed) Changing to a different image size screwing up in Firefox on OS X, thanks to preloading.
  • (What) Still looking for how to fix the Safari bug
  • (Sort of kind of) Planning on testing in IE and Firefox on Windows.
    • Chrome for Windows exhibits similar behavior to Safari (no initial image, buttons work fine)
    • Internet Explorer 8 is just completely messed up (misparses the CSS, though JS seems to work mostly correctly, aside from the fact that the image-size-when-changing issue is three times as bad (doesn't get better after the second click!))
• Over the weekend I'll probably work on trying to fix a few of these bugs and maybe even try to get started on the back-end stuff before Monday.

7/8/10

• I'm storing all the labellings by the user in a javascript array. Each point is represented by a four-item array containing a reference to the rectangle object for that point, the brush used for that point (which bird or background), an x coordinate,and a y coordinate.
• Implemented an eraser functionality. The way it finds which points to erase is simply by searching through the whole labels array linearly and removing any such that the Euclidian distance (${\sqrt {(x_{1}-x_{2})^{2}+(y_{1}-y_{2})^{2}}}$) from the eraser click is less than eraserRadius. Right now I have that set to 10 pixels, but I might want to play around with that value to see what is most intuitive for users.
• Can now make the buttons change the image (e.g. when I switch views) and not lose all the labels. Now to just make the buttons actually switch to the correct views.
• Noticed that my page doesn't work at all on Safari. Should work on fixing that. Also if the other image it changes to is a different size it screws up the first time, but only the first time for some reason. I'll work on fixing these first thing tomorrow.
  • Fixed an eraser bug where it sometimes wouldn't remove two consecutive squares. This had to do with what I was doing with the array indeces when it popped an element out.

7/7/10

• More UI stuff.
• Added some more buttons and button functions, and implemented ability to click on the image and have the annotation show up (I foresee it being very difficult switching between image views and retaining those annotations in view, as well as storing those annotations correctly).
• Notable things left to implement:
  • Eraser functionality
  • Ability to switch between views (Image, UCM, Image/Boundaries from last round hybrid, Regions from current round) and rounds (user will get up to maxRounds rounds), and having the annotations save between the portions. This and the former point mean that I will have to store the annotations as the user adds them, instead of just grabbing them all at the end when the user submits the labelling.
  • Back-end stuff (submitting answers, submitting labelling and getting back a segmentation, etc.)
  • The help section
  • Woah, that's it?

7/6/10

• Started coding a preliminary UI using jQuery, Raphael, and some code I took from Peter's examples. Hoping to have it mostly functional by the end of tomorrow or Thursday. All the code is up on the SVN repository.

7/2/10

• Continued reading up on jQuery.

7/1/10

• Pretty sure I finally understood the label propagation code. The distance it uses is the max ucm strength crossed along a path.
• Made a flow-chart/diagram thing to plan out my project.
• Going to try starting to make a UI for the user.
  • First, I'm going to do a little bit of looking at the examples Peter sent me earlier, and some reading up on jQuery (a Javascript library for document manipulation) and Raphael (a Javascript library for drawing stuff).

6/30/10

• I installed the MatlabMPI code and Piotr's Matlab toolbox, as I am anticipating having to do some distributed evaluation.
• Discovered that performing gPb on an image of size 438x450 takes about 9 minutes, performing UCM and segmentation only takes a few seconds each.
• The gPb-OWT-UCM interactive segmentation works really well!
• Peter and I agreed that I should now start looking inside the black box of the interactive segmentation code and putting together a general plan about how I will build the system. Additionally, we discussed the possibility of later implementing some of the segmentation code clientside.
• I started working on deciphering the label/segmentation propagation code of the interactive segmentation portion.

6/29/10

• Michael explained how the GUI works for the interactive segmentation. This makes a lot more sense now. I am currently looking at the C++ and Matlab code figuring out what is going on. It's less complicated than I was thinking yesterday.
• Peter and I talked about the code a little and we are going to meet this afternoon to discuss the theoretical foundations of the code.
  • We discussed this now.
• I have gotten the gist of how the segmentation code works and the contour detection code (gPb), and also gotten all of it to work for me. Now, I need to talk to Peter tomorrow about how to proceed, as a lot of the code is actually in MATLAB. :\
• I have a file at "~/ImportantFiles" on the vision cluster where I'm keeping track of which source files are important and will be needed later on

6/28/10

• I talked to Peter to recap my project:
  • In general, I will have to write code that will pre-compute the segmentation tree (probably on the back-end), and then send it off to the client-side javascript. Then, we will ask MTurkers to annotate the images we have in some way (while giving them something presumably based on the segmentation tree). After that, info will be sent back and we will hopefully have some segmentation.
  • Something to think about: I want to have the code be general, as we might want to segment multiple objects out of an image.
  • Also: Sometimes boundaries don't "exist" in an image, but I would want a way to figure out when humans "hallucinate these".
  • Peter will send me the code for the Visipedia server, some javascript GUIs that have already been written, and the segmentation code that has already been implemented so I can get started figuring out what I need to do.
  • I will need to figure out the best way to have Turkers annotate the images as part of this.
  • Peter suggested that by tomorrow or Wednesday, I have an outline for how I want to build this system and when I want to have things completed.
  • Some subprojects/extensions:
    • Figuring out optimal GUI to get most info.
    • Developing a probabilistic model such that we know when we have enough information to be certain of our segmentation, and a way of combining users' annotations based on how good they've been in the past
    • (Probably wouldn't do this: Summer is too short a time frame) Figuring out how to detect where an object is in an image.
• Today I looked at some of Peter's example JS GUIs and understood most of those, and tried to comprehend what was going on in the segmentation code, but mostly failed. I will have to talk to Peter to figure out what is actually going on.

Before starting

• Read some papers, most notably the gPb-OWT-UCM paper and the gPb paper.
• Read parts of PRML, specifically Chapters 1,2,3.1-3.3,5.1-5.3, and 8.1-8.3.

Plan

• Week 2: (7/6/10) Have a mostly functional UI for the Turk task.
• Week 3: (7/12/10 - File:IlyanepProgRep1.pdf due 7/15 or 7/16) Complete UI and have most of the server-side back-end stuff where it accepts the user's labelling and produces a segmentation to give back complete (including setting up structure to temporarily store segmentations based on labellings and to store labellings in the DB). Also, have this code all ready to be packaged as a Turk task.
• Week 4: (7/19/10) Code a script to pre-process large sets of images for UCM and set up DB structure to store these.
• Week 5: (7/26/10) Start thinking of and implementing ways to parse multiple Turkers' labellings/segmentations into one final segmentation of an image.
• Week 6: (8/2/10 - Abstract is due 8/2) Complete week 5 stuff and start thinking of and implementing cool theoretical extensions to probabilistically be able to determine a level of confidence based on previous ability of the particular Turkers who answered our HIT.
• Week 7: (8/9/10 - Progress report due 8/13) Continue extensions.
• Week 8: (8/16/10) Continue extensions.
• Week 9: (8/23/10) Continue extensions and start thinking about/gathering some sort of statistics on the results (including: what sort of results and statistics are we going for?)
• Week 10: (8/30/10) Finish results (start on final paper, maybe start on a technical paper discussing the work I've done to put on a website).
• (Final Paper due 9/24/10)

• Milestones:
  • Functional UI: Sometime early week 3.
  • All of my code ready to be packaged as MTurk HITs: By the end of week 4 or sometime week 5.
  • Some way to interpret the MTurk HIT results: By the end of week 5.
  • Extensions: Working on it weeks 6-9.
  • Have some sort of results to show for my work, weeks 9-10.