Relief-type cultural heritage objects, such as relief carvings or sculptures, are commonly found at historical sites worldwide and are considered to be of immense historical and cultural value. Unfortunately, many of these objects suffer from damage and deterioration over time. While modern 3D scanning techniques can digitally preserve their current form, restoring their original appearance is a challenging task that requires extensive manual labor and specialized knowledge. However, researchers have developed a novel neural network model that can reconstruct these reliefs as three-dimensional digital images from old photographs containing their pre-damage information. This innovative technology paves the way for accurate digital preservation of valuable cultural heritage objects.
Unlike 3D sculptures or 2D paintings, reliefs have a shallow depth and are meant to be viewed from the front or either side, making them suitable for 3D digital reconstruction algorithms using old photos. A multinational research team led by Professor Satoshi Tanaka from Ritsumeikan University, Japan, along with Dr. Jiao Pan from the University of Science and Technology Beijing, developed an innovative multi-task neural network for 3D reconstruction and digital preservation of reliefs using old photos. This new method enhances depth estimation, particularly along soft edges, using a novel edge-detection approach, allowing for the reconstruction of finer details such as human faces and decorations that were previously missing.
The proposed multi-task neural network performs three tasks: semantic segmentation, depth estimation, and soft-edge detection. The core strength of the network lies in its depth estimation, achieved through a novel soft-edge detector and an edge matching module. The soft-edge detector treats edge detection as a multi-classification task, determining the degree of “softness” of edges, and enhancing depth estimation. The edge matching module extracts multi-class soft-edge maps and a depth map from an input relief photo, focusing more on soft-edge regions for detailed depth estimation. The network optimizes a dynamic edge-enhanced loss function to produce clear and detailed 3D images of reliefs.
The researchers applied this innovative model to reconstruct the hidden reliefs of Borobudur Temple in Indonesia, a UNESCO World Heritage Site. The ground-level wall reliefs of the temple are covered by stone walls due to reinforcement work carried out during the Dutch colonial period and cannot be viewed. Using old photographs, the multi-task neural network successfully reconstructed these hidden sections of Borobudur’s ground-level reliefs. Through computer visualization and virtual reality, their research allows for virtual exploration of these unseen treasures, highlighting the potential impact of their work in preserving and sharing cultural heritage.
The technology developed by the researchers has vast potential for preserving and sharing cultural heritage. It not only benefits archaeologists in their research but also opens up new opportunities for immersive virtual experiences through VR and metaverse technologies. By preserving global heritage for future generations, this innovative neural network model has the potential to revolutionize the way we digitally preserve and share valuable cultural heritage objects. The findings presented by the research team at the international conference ACM Multimedia 2024 demonstrate the significant progress made in the field of digital preservation of relief-type cultural heritage objects.
