From Smart 3D Capture to Seamless XR Delivery.

AI for 3D

XRculture explores new approaches to 3D digitisation by applying artificial intelligence techniques to the processing and reconstruction of cultural heritage objects. The focus is on objects that present challenges to traditional methods - such as transparent, reflective or textureless surfaces commonly found in museum collections.

In addition to generating new models, the project will investigate how AI can improve the quality of existing 3D data. This includes improving the source material used in the reconstruction process (such as photogrammetric images) and refining the output itself - for example, by increasing texture resolution or correcting surface inconsistencies.

The team is also developing a processing workflow to optimise mesh geometry and produce lightweight 3D models for efficient visualisation and improved performance in extended reality environments.

How we work / Project phases

Defining requirements for AI-driven 3D digitisation and XR optimisation

This phase lays the groundwork for the AI-driven 3D digitisation workflow, including a comprehensive metadata assessment to determine the necessary types for AI-based 3D models and ensure alignment with digital cultural heritage standards. Collaboration with cultural heritage experts and end-users is part of this process, helping align the project with real-world needs and sector expectations.

AI-based generation of new 3D cultural heritage models

In this phase, we explore existing AI-based 3D reconstruction tools, with a focus on NeRF-based and Gaussian Splatting techniques for capturing complex museum artefacts. The selected collection includes objects with challenging surface properties: transparent, textureless, deformable, reflective, or highly detailed materials.

We produce high-quality 3D models using the realistic rendering capabilities of NeRF-based methods and the efficient memory usage of Gaussian Splatting, which makes it well suited for large datasets. These techniques are also applied to heritage at risk, enabling faster digitisation while reducing potential damage to physical objects.

Improving 3D models from museums and digital collections

This phase focuses on enhancing the quality of existing 3D models using the latest AI techniques. Activities include exploring how these tools can improve source data and texture quality, developing and testing improved workflows for optimising object geometry (meshes), and producing lightweight 3D models for XR visualisation. Particular attention will be paid to museum collections with complex artefacts that have been digitised using traditional methods, as well as lost cultural heritage objects that could benefit from further refinement.

Optimisation techniques for preparing 3D content for XR applications

This phase combines two key components to improve the interactivity and accessibility of 3D models in XR environments.

First, we focus on optimising 3D models for XR applications using open source tools. This includes standardising the optimisation process (adjusting parameters such as polygon count) and simplifying the creation of multiplayer-ready experiences in web-optimised formats (SLPK, glTF and I3S) with efficient data streaming capabilities.

Secondly, we are implementing automatic 3D to 2D video conversion for holographic projection using ARCTUR's High Performance Computing (HPC) infrastructure. This process automates the transformation of complex 3D models into high-quality 2D video content, and enables immersive holographic projections. The result is a more accessible format for cultural heritage experiences.

XRculture Middleware

The XRculture Middleware is designed to store and manage the complex data associated with multi-dimensional cultural heritage objects (3D meshes, texture maps, geospatial data, metadata, and contextual information). Content management platforms are used to enable efficient indexing, querying, retrieval, and analysis of these datasets.

To ensure interoperability and integration across platforms, the middleware relies on standardised metadata schemas and ontologies.

The aim is to develop technological solutions for the management and visualisation of 3D models that can be adopted by the Common European Data Space for Cultural Heritage. This work builds on existing and emerging infrastructures for storage, management, and visualisation. Key challenges include interoperability, scalability, long-term preservation, data security, and ethical considerations such as copyright and privacy.

How we work / Project phases

Design a middleware protocol for standardised 3D web viewers

In this phase we review the already available tools, both those running offline or on cloud infrastructures, and based on this we build the Middleware protocol. The goal is to create an abstracted communication layer that standardises how front-end 3D viewers and back-end storage systems interact. The protocol will, for example, allow a 3D viewer to declare what kind of functionality it supports and, vice-versa, a storage solution to deliver 2D or 3D structured data to the 3D viewer (along with contextual instructions for scaling or repositioning the content). This approach ensures flexible management and visualisation of complex, multidimensional models.

Implementation of the protocol in available web viewers

The XRculture Middleware protocol allows for the flexible reuse of existing management and visualisation tools. Not all tools are required to support the full range of protocol functionalities, each 3D viewer will receive different data and options depending on the capabilities it supports.

In this phase, tools already developed by the project partners (such as the INCEPTION semantic BIM viewer and the WEAVER 3D viewer) will be made available for integration into the Common European Data Space for Cultural Heritage, adopting the standard defined by the Middleware protocol.

To ensure a smoother user experience, unique features available in only one tool will be highlighted, while shared functionalities will be redesigned for consistent behaviour across both platforms. The user interfaces will also be aligned to ensure a more coherent interaction model.

Building tools to support the publishing and sharing of 3D content

This phase focuses on the remaining features needed to publish and aggregate content to the Data Space and to enable access through XR applications. It extends the use of the Share3D Dashboard to aggregate 3D models from different viewers, such as the INCEPTION 3D viewer. To support these functions, we implement connectors, converters, and micro-services that enable the use of the XRculture Middleware protocol. This includes, for example, 3D-to-2D video conversion for holographic projections within the ARCTUR viewer.