Image-Based Representation of Clothes for Realistic Virtual Try-On

This research addresses the issue of realistic visualization of garments in real-time augmented reality environments, where virtual clothes are rendered into real video material. From a scientific point of view this scenario presents numerous challenges. The most crucial target is the realistic visualization of the virtual clothes rendered onto a moving human body while the system runs in real-time. Existing approaches focus on high-quality cloth analysis, simulation, and rendering but are far from working in real-time. Others provide a real-time visualization, but are quite limited in visual quality and plausibility. In this research project we aim at developing novel solutions that enable real-time cloth visualization with very high rendering quality. This can be achieved with new representations of clothes that combine a rough 3D model with image-based rendering techniques. The geometric information will be used for low-resolution shape adaptation while small details (e.g. fine wrinkles), as well as complex shading/reflection properties will be accounted for through a clever use of numerous images captured in an offline process. The images contain information on shading, texture distortion and silhouette at fine wrinkles. These representations will be estimated in advance from real samples of cloth captured in different poses, using sophisticated image analysis methods, thus shifting computational complexity into the training phase. For rendering, pose dependant geometry and appearance are interpolated and merged from the stored representations. This procedure will greatly reduce the computational complexity associated with interactive visualization and enable a high-end visual rendering in realtime. By separating the surface albedo (i.e. its underlying local color) from shading and estimating spatial texture distortion at small wrinkles and creases, retexturing, i.e. changing the appearance of the piece of cloth, will be possible. The use of real images leads to very natural looking results while the geometric model will allow animation of the final representation.