DIDYMOS-XR

Description:

The vision of DIDYMOS-XR is to enable advanced, more realistic and more dynamic extended reality (XR) applications, powered through artificial intelligence. The project thus focuses on advancing technologies for creating large-scale digital twins synchronised with the real world. DIDYMOS-XR will research and develop methods for reconstruction and mapping from heterogeneous inputs, including static and mobile sensors, AI-based data fusion, scene understanding and rendering. Digital twin-based applications also require means for accurately positioning in the environment and being aware of the dynamics of the scene, also addressed by the project.

The capture of scenes at scale, as well as using cameras and other sensor data to synchronise the digital representation, bears the risk of capturing personal and sensitive data. Hence, the technologies resulting from DIDYMOS-XR must be and will be ethical and privacy-aware by design. To demonstrate and validate the enabling technologies researched and developed in DIDYMOS-XR, the project will address use cases of XR applications in two domains that differ in their scale and characteristics. One is urban planning and smart mobility; the other is collaborating with autonomous mobile robots at an industrial production facility.

Within the project, the i2CAT Foundation leads Work Package 3 – World capture and synchronisation, which main objectives are:

• Research and develop methods to integrate information from multiple stationary and moving sensors.
• Capture, combine and transfer information obtained from scene through low-resolution sensors.
• Research methods for understanding the semantics and dynamics of the scene.
• Synchronise the digital twin from local sensor data.

Particularly, within Task 3.3 Synchronisation from moving and stationary cameras, i2CAT researchers will explore methods for compression and transmission of 3D content (i.e. point cloud), aiming at regularly updating the digital twin.

i2CAT also actively participates in Work Package 4 – Localisation, mapping and rendering, by leading Task 4.4 – Real-time rendering. Within this task, researchers will explore different alternatives and optimizations in real-time rendering for volumetric content with dynamic elements.

Finally, i2CAT contributes to Work Package 5 – XR applications and validation, oriented at integrating the project’s developed technologies into extended reality applications in the smart city and industrial production domains.

DIDYMOS-XR includes 5 use cases for demonstrating the digital twin (DT) capture and creation methods in extended reality (XR) applications:

1. Digital twin creation
2. Tourism
3. City maintenance/ inventory
4. City planning
5. Industry.

The XR applications are unique to each use case, using a mix of virtual and augmented reality (VR and AR) as well as wearable, mobile, and desktop devices.

Estimated impact:

• Strategic impact: The use of digital twins and XR applications based on them aligns with digitisation and increased automation of industrial production. These technologies are a prerequisite to keeping the competitiveness of the concerned XR industry in the EU. This not only affects jobs created by these industries but also increases the EU’s sovereignty by reducing its dependence on global logistics chains, which have turned out to pose critical vulnerabilities, such as the global COVID-19 health crisis and Russia’s aggression in Ukraine.
• Socio-environmental impact: The use of XR and digital twins for urban planning and mobility applications will support the transition of European cities to be more sustainable and inclusive. It is fully aligned with the New Associated with the new European Bauhaus initiative. The transformation of city spaces to be greener, more human-friendly and support different modes of transport relies not only on data and simulation based on digital twins but also on citizen involvement to be successful. XR applications allowing changes to the design and use of city spaces and the impact of different mobility choices are important instruments to involve citizens in planning.
• Economic and technological impact: Besides the economic and technical impacts foreseen by the call topic and destination, DIDYMOS-XR contributes to better technology for creating Urban or Local digital twins (LDTs)71, thus addressing one of the goals of the Living-In.EU initiative. The technologies researched and developed in DIDYMOS-XR will contribute to realising the vision of an AR cloud that provides a rich digital model of the real world and serves as the basis for a wide range of XR applications.

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.