Visual Discovery of Landmarks and their Details in Large-Scale Image Collections

Internet photo collections provide detailed coverage of the world's landmark buildings, monuments, sculptures, and paintings. This wealth of visual information can be used to construct landmark recognition engines that can automatically tag a photo of a landmark with its name and location. Landmark recognition engines rely on clustering algorithms that are able to group several millions of images by the buildings or objects they depict.

This grouping problem requires a suitable definition of landmarks, image similarity measures robust to viewpoint and lighting changes, and efficient clustering algorithms. In this thesis, we present the Iconoid Shift algorithm which fulfills these requirements: It represents each landmark by an iconic image, or Iconoid, which is the image that has the highest overlap with all other images of this landmark. We find Iconoids by performing mode search using the novel homography overlap distance measure that is robust to viewpoint and lighting changes. We propose efficient and highly parallel algorithms to perform this clustering.

The increasing density of Internet photo collections allows us to even discover sub-structures of buildings such as doors, spires, or facade details. To this end, we present the Hierarchical Iconoid Shift algorithm that produces a hierarchy of clusters that represent building sub-structures. This algorithm is based on a novel hierarchical variant of Medoid Shift that tracks the evolution of modes through scale space by continuously increasing the size of its kernel window.

Finally, we perform a large-scale evaluation of the different components of a landmark recognition system, analyzing how different choices of components and parameters affect the performance of a landmark recognition system.