The Benchmark and Evaluation

2.1 Introduction

Given the array of available applications, 3-D shape representation and matching and retrieval algorithms have been extensively investigated. These methods require standard benchmarks to evaluate their performance. Standard benchmarks are important for justifying the effectiveness of different method, and many 3-D object benchmarks have been released in recent years.

In this chapter, we review several popular 3-D benchmarks, including the Princeton Shape Benchmark (PSB) [1], the Engineering Shape Benchmark (ESB) [2], the ITI database [3], the Amsterdam Library of Object Images (ALOI) [4], the Eidgenössische Technische Hochschule (ETH) [5], and the National Taiwan University (NTU) [6] 3-D data sets. We also introduce the recent 3DOR contest, SHREC, on each year, respectively.

Evaluation methods are important to judge the performance of 3DOR. In the last part of this chapter, we introduce several widely used 3DOR retrieval performance evaluation criteria.

2.2 The Standard Benchmarks

In this section, we introduce six standard benchmarks.

• Princeton shape benchmark
PSB was collected by Princeton University. PSB contains 1814 polygonal 3-D models, which are collected from the World Wide Web from 293 different Web domains. All these models are manually classified based on functions and forms and multiple semantic labels are annotated for each 3-D model. These labels belong to a hierarchical structure, which can reflect the semantic in different levels. For example, a 3-D model with an annotation of “aircraft” can be further divided into subclasses, such as “winged_vehicle aircraft,” “balloon_vehicle aircraft,” and “helicopter aircraft.” PSB has been split into a training set and a testing set. The training set contains 907 models, and the testing set contains the other 907 models. The distribution of all the models in PSB is shown in Figures 2.1 and 2.2. Example 3-D models from PSB are shown in Figure 2.3.

• Engineering shape benchmark
ESB focuses on engineering shape representation. Computer-aided design product design is an iterative process, which is highly dependent on previous model designs. ESB aims to evaluate the shape representation to discriminate different shape forms for product design. ESB is composed of 867 3-D CAD models. For each 3-D model, there are CAD files, including two neutral formats: “.stl” and “.obj”, and one thumbnail image. These models have been classified into three higher-level categories: Flat-thin wall components, Rectangular-cubic prism, and Solids of revolution. As introduced in [2], these three higher-level categories are defined as follows:

• Flat-thin wall components are the parts with thin-walled sections and shell-like components.

• Rectangular-cubic prism are the parts whose envelopes are largely rectangular or cubic prism.

• Solids of revolution are the parts whose envelopes are largely solids of revolution.

These higher-level classes are further divided into 45 shape categories. The distribution of all the models in ESB is shown in Figure 2.4. Example 3-D models from ESB are shown in Figure 2.5.

• The ITI benchmark
The ITI data set is from the VICTORY 3-D search engineer [7]. The ITI data set contains 544 3-D models from 13 categories, including animals, spheroid objects, conventional airplanes, delta airplanes, helicopters, cars, motorcycles, tubes, couches, chairs, fish, humans, and some other models. Each 3-D model contains 5,080 vertices and 7,061 triangles on average. The distribution of all the models in the ITI data set is shown in Figure 2.6.

• Amsterdam Library of Object Images
ALOI is a color image collection containing 1,000 small objects. We note that this data set does not provide the virtual 3-D model data, but only a set of images for each object. To obtain these images, the camera view angle, the illumination angle, and the illumination color for each object are systematically varied. For the camera view angle, the frontal camera is rotated in the plane at 5° to record 72 views of each object. For the illumination angle, there are eight different light conditions employed in the image data collection procedure, and five lights are employed. Each object is recorded with only one light turned on (five kinds in total), with only the left or right lights turned on (two kinds), and with all the lights turned on (one kind). Three cameras are employed to capture the views, which leads to a total of 24 different illumination conditions for recording views. The voltage of the lamps is varied 12 times for the illumination color, which yields 12 different illumination colors. Wide-baseline stereo images for 750 different scenarios are also recorded. In total, there are 110,250 images in the ALOI data set. Example objects from the ALOI benchmark are shown in Figure 2.7.

• The Eidgenössische Technische Hochschule Zürich data set
The ETH benchmark contains 80 objects from eight categories and each category consists of 10 objects. These categories cover the following areas: fruits and vegetables, animals, small human-made objects, and large human-made objects. These eight categories include apple, pear, tomato, cow, dog, horse, cup, and car. In the ETH data set, a group of 41 views are recorded for each object. To capture these views, the object is placed on a table with blue chroma keying background and the views are obtained from directions spaced equally over the upper viewing hemisphere, subdividing an octahedron to the third recursion level. These views are recorded using a Sony DFW-X700 progressive scan digital camera at a resolution of 1024 × 768. For each view, the original color image and an associated high-quality segmentation mask are provided. Example objects from the ETH benchmark are shown in Figure 2.8.

• The National Taiwan University data set
NTU contains two parts, the NTU 3-D model benchmark and the NTU 3-D model database. The NTU 3-D model benchmark consists of 1833 3-D models and the NTU 3-D model database consists of 10,911 3-D models. All these 3-D models are downloaded from the Internet in July 2002. These models are in Wavefront file format. For each model, a thumbnail image is provided in the data set. The NTU 3-D model benchmark contains 47 categories with 549 3-D models, while another 1284 models are regarded as miscellaneous. The distribution of all models in the NTU 3-D model benchmark is shown in Figure 2.9, and example objects from the NTU 3-D model benchmark are shown in Figure 2.10.

2.3 The Shape Retrieval Contest

SHREC [8] is an annual contest that started from 2006. SHREC aims to evaluate the effectiveness of 3-D shape retrieval algorithms. Each year, different tracks are organized with various objectives and different testing benchmarks are employed or released. In this section, we briefly review the existing SHREC events, the employed benchmarks, and the challenging tasks.

• SHREC2006
SHREC2006 is the first SHREC and there is only one task: retrieving a ranking list of similar 3-D objects given one...