3D Mesh Preview Streaming

Abstract:

Sharing high-resolution 3D models over the Internet is challenging due to the models' geometric complexity. There are two typical ways to preview a 3D model online: (i) by showing a set of representative views (or "keyviews") of the 3D model; and, (ii) by showing a video of the 3D model as viewed from a moving camera along a path. This thesis proposed a third approach for mesh-based 3D models, termed as 3D mesh preview streaming, by streaming and showing parts of the mesh surfaces visible along the virtual camera path.

A preview is a sequence of 3D synthetic views (or "views") rendered by a virtual camera (or "camera") along a camera path. A good preview has the following properties: (i) the camera path should pass through representative or novel views smoothly, with the camera moving at a constant speed; and, (ii) the mesh should be refined to full resolution (view-dependently) before it is shown to the user. But the network bandwidth is limited and fluctuates in the Internet. The client may not receive sufficient data in time for mesh refinement. Therefore, this thesis addressed three issues of 3D mesh preview streaming, focusing on: (i) how the preview streaming system would best handle bandwidth variation effectively; (ii) which vertices of the mesh surfaces are important and to be sent first when the bandwidth allows; and, (iii) how to measure the novelty of a view on a camera path compared to other views already visited by the path. We considered bandwidth constraint, semantic importance of mesh surfaces, and human mental images of 3D objects, as follows.

First, we introduced the preview streaming architecture and investigated which schemes capable of adapting to bandwidth variation. A set of keyviews was given as input. Our results revealed that: (i) reducing camera speed, reducing mesh quality, and zooming out the camera are viable schemes for bandwidth adaptation; and, (ii) when the camera moves close to a keyview, it is preferable to reduce camera speed; otherwise, we reduce mesh quality if the bandwidth drops.

Second, a semantic-aware saliency-guided approach was proposed to transfer and refine perceptually important mesh vertices first. The experiment results showed the effectiveness and efficiency of our approach in prioritizing important mesh vertices during the refinement, either view-dependently or view-independently. We inferred the semantic importance of the logical parts of a mesh (e.g., "head", "wings") by requiring the user to input only a few views of the mesh that are deemed important by the user.

Third, we modeled the dissimilarity among views, and the novelty of a view was measured by its dissimilarity to other view(s). Two views are dissimilar if they have different viewing content or viewing orientations. The model evaluated perceptual cost of mentally reconstructing one view from other view(s). Higher perceptual cost means more dissimilarity. Perceptual cost was related to reaction time of rotation and completion of human mental images. Visual comparisons among dissimilarity measures suggested that our model can better determine how dissimilar one view is to other view(s).