Three technical challenges of scaling from social virtual reality to metaverse(s): interoperability, awareness and accessibility

1 Introduction

There are many visions for what metaverse systems will comprise: from explorations of technical affordances of next-generation communication systems (Oliveira et al., 2009; Abilkaiyrkyzy et al., 2023; Zhang et al., 2024) through to rights and principles that should underpin such systems in use (Parisi, 2021; Ball, 2022; Au, 2023). If there is a consensus, it is that we don’t quite know what the technical architecture or scope of capabilities of such as a system will be: Will it consistent of purely simulated virtual worlds or will it be grounded in mixed-reality spaces? What is the role of emergent network technologies such as peer to peer databases or blockchains? Who will control the systems and how will rights be enforced? How will the system be equitable to users with different backgrounds and access to interface devices? How will systems fulfil societal aspirations¹?

We can see the early prototypes of future metaverse systems in today’s social virtual reality (SVR) systems (e.g., see (Osborne et al., 2023; Liu and Steed, 2021)). These allow multiple users to congregate in diverse virtual places, share fantastic or realistic representations of themselves and undertake a wide variety of social activities. To some extent, a metaverse system should allow any existing SVR environment and activity to be experienced within a more unified framework where one didn’t need multiple accounts, multiple devices, multiple pieces of client software and experience with multiple interfaces. In a metaverse we would expect to be able to reach all of the virtual places supported and we would expect fluid interactions with other users. Today, while some users are already spending a lot of time in prototypical metaverse experiences, hardly any of the capital developed in one system (e.g., assets, but also social capital) are portable to other systems. Each is effectively a walled garden. Discoverability is low, with different experiences needing different software. Users are often not able to stay as a group within a single software client, never mind when switching to different software.

Today, designing a whole system that alleviates these frustrations, is an almost impossible task. Even the most evolved walled gardens of today lack significant features. Indeed some of the challenges about ownership, naming and scalability have already been identified in research going back over 30 years (e.g., see (Singhal and Zyda, 1999; Steed and Oliveira, 2009)). There is a need for more systems to be prototyped that bring us closer to understanding the infrastructure components needed. One could point at other successful distributed engineering efforts such as the Internet and the World-Wide Web and claim that system(s) will emerge by building on simple standards, and that relevant architectures will emerge from clear separation of responsibilities between clients and servers. But this belies that there are fundamentally difficult problems that client software that connects to metaverse(s) will need to solve, no matter their architecture. This essay highlights three such problems that affect the client software that a user engages with: what interoperability means; how to scale awareness levels; and how to ensure accessibility. These problems are described not because they are the most critical issues is designing future metaverses, but because they are some what overlooked by standards activities² and will have very significant impact on the experience of the user. They are derived from our own experience of teaching and developing for mixed-reality and also our experience in building platforms to support those activities.

2 Interoperability

A metaverse system would necessarily be made up of client software that interact with server systems that provide critical features. While some of communication might be peer to peer between clients, as is done with some current game systems and research prototypes (e.g., see (Steed and Oliveira, 2009; Frécon et al., 2001)), within the services of metaverses we will find the distribution of description of virtual spaces, services for identifying and rendezvousing with other users, platforms for secure transactions, etc. One aspect of interoperability is that whatever configuration of systems we end up with, users will expect to be able to port of assets between systems (Jerome, 2024). However, beyond that, it is highly likely that dense social environments will require server infrastructure for simulation and message distribution. When we talk of interoperability, we thus identify that clear interfaces must exist for describing virtual spaces (i.e., are descriptive of scenes, containing geometry, materials, animations, etc.) but also for interaction between different processes across the Internet. Previous work on inter-operation has tended to focus on the latter, plotting architectures that build upon existing web services (e.g., (Havele et al., 2022)) or proposing new server simulation models³. There is an implicit assumption in a lot of this work: there will be a single integrated client that accesses these services. That is, if only these services could be described the client would take on the complete role of creating the user experience. Systems that have been built over the past couple of decades, have thus been largely based around this model of a large client that integrates all the interfaces necessary to connect to the back-end services. Second Life illustrates the advantages and disadvantages of this approach: while originally a closed system, the client was made open along with its protocols, so servers could be implemented to complement the official servers (for an overview see (Au, 2008; Au, 2023)). Modern SVR systems are still walled gardens though their architectures will have some similarities.

While standardisation of services is certainly necessary, we highlight that almost all efforts are making this assumption that a single client is being supported. We note that with the rise of smartphones, consoles and tablets, the model of a single application taking control of the user experience has dominated. That is, while multi-tasking might be possible, certainly when it comes to immersive experiences a single client is responsible for creating the environments. Exceptions to this include the ‘holograms’ on Windows Mixed Reality/Hololens⁴ or Volumes in VisionOS⁵. Both of these are examples of code from different sources that is integrated into a shared experience for the user. Notably both of these systems are primarily used for augmented reality situations, where the analogy of multiple applications matches a conceptual model of different objects in the world having different functions.

An interesting question is thus how far inter-operation between applications can be pushed. Rather than put all the behaviour code into a client that would then have to support code extensibility or interpreted code, why are we not treating the user experience more like an operating system or window manager, where different applications are responsible for different aspects of the experience? That is, how could we assemble an immersive experience from multiple applications?

Currently when we switch between different fully immersive systems each system takes full control of the user experience and is thus responsible for everything from input device interpretation, through simulation to rendering. Of course applications rely on run-time services for device interfacing, but there is little interaction between applications. This can be contrasted with mature desktop operating systems, where we not only have multiple applications and interfaces operating at once, but the screen is composited from multiple applications. Similarly, a modern web browser provides a variety of services over the operating system, with complex application being written within distinct pages in a browser that supports multiple windows.

There are precedents for this in research prototypes. The DIVE system was an early peer to peer multi-user system that supported a variety of immersive interfaces (Frécon et al., 2001). The user interface was written in TCL/TK, so the client was dynamically extensible at run-time, but also applications could connect to a running client and insert new 3D objects as interfaces. This was an analogy of how X11 remote applications worked. A slightly different approach is the concept of scene-graph as bus, where multiple application interact using a common scene graph as a shared resource. The scene-graph acts as an interface between applications rather than using any other inter-process communication (Zeleznik et al., 2000). We can imagine a variety of ways that different spaces could be constructed from volumetrically constrained, or through the window, metaphors for layering and composition. We can also see some hints of this in recent discussions about constructing mixed reality systems that compose real and virtual scenes (e.g., VRCeption (Gruenefeld et al., 2022)).

We therefore suggest that there are more ways that metaverse system could be constructed than simply constructing a large client browser. While efforts to extend web browsers through standards such as WebXR⁶ to support larger collaborative immersive experience are very promising, e.g., see the Hubs project⁷, we believe that the key step is to re-imagine how an immersive interface is built. In Section 4 we will elaborate on one specific goal for client-side interoperability: supporting accessibility.

3 Scale

For a metaverse system to scale to the audiences envisaged, there needs to be scalable systems that support persistent data to represent spaces, users, interactivity, but also scalable systems that support real-time synchronous interaction between subsets of known users. Fundamentally, real-time interaction between N users requires O(N²) interactions between the clients supporting those users. Scalability to large numbers has long been an interest of developers from early distributed simulation days (e.g., see (Singhal and Zyda, 1999)). Early SVR systems used a variety of ways to determine whether or not to enabled communication between pairs of users or entities based on cellular partitions of spaces (Macedonia et al., 1995), reasoning about spatial overlap of users (Benford et al., 1993), by partitioning a graph of nearest neighbours (Backhaus and Krause, 2007) or fixing group size of interested players (Bharambe et al., 2008). The general area is sometimes referred to as interest management (Morse et al., 2000; Delaney et al., 2006).

Many current systems use a partitioning of users based on a shard or instance models. That is, users are partitioned up into groups of users that can be supported on a simple server. Current SVR services supports small numbers of users (<50), as do many online games. Network measurement of existing systems (Cheng et al., 2022) or open source examples (Friston et al., 2023) show that the server is a bottleneck: it has to ingest messages, simulate the world state and then distribute state changes.

There is thus a mismatch between the common vision of the metaverse as a massive online space, and the capabilities of the server and message passing infrastructure. But scalability techniques already note that user interest can be scoped down to the most proximate other users. Three levels of awareness have been identified (see also (Steed and Oliveira, 2009), chapter 12).

$•$ Primary awareness: users are co-located and are proximate to each other. We would expect detailed animation, voice and joint interaction.

$•$ Secondary awareness: awareness that a user is somehow in a reachable space, e.g., within the scope of a server or server network. This now means that rendezvous is possible and can be achieved easily, for example, by virtual travel.

$•$ Tertiary awareness: awareness of the activity of activity anywhere in the population of users with which one is connected (e.g., friend networks), scoped by privacy policies. This to facilitate coordination and rendezvous of users.

Different systems might have additional levels, but it can be a useful exercise to map this to existing systems. Tertiary awareness is more like a social network or it might be labelled ambient awareness. The facilities provided are more about finding if a user is connected and available. The secondary awareness highlights that users are only reachable for primary awareness if they are in the same system. One can imagine tertiary awareness being an intra-system service of awareness, whereas to rendezvous one might need to install more software or download specific resources. Primary awareness is the full bandwidth experience. A distinction to secondary awareness is that in constrained situations, bandwidth and capacity is prioritised to support more relevant (i.e., proximate and important) users rather than users that are more in the background (e.g., far away or not engaged in the same activity).

These distinctions of awareness also highlight that in order to support travel in an apparently large-scale open world, multi-server support will be necessary. This then returns us to the question of partitioning load across servers, but now attempting to support seamless handover between servers. Again, research prototypes exist, (Funkhouser, 1995; Iimura et al., 2004), but this is definitely an area that needs more research and development.

Finally, we will note that scalability will also have user interface aspects. Users want features to manage privacy and security, and these have to match with the practical routing of messages and distribution of assets. Users also want to travel as groups and maintain cohesion. These present significant challenges to the network infrastructure as primary awareness should reflect user preferences, especially as collaboration is necessary to support certain accessibility features. Also, as soon as users make groups, some knowledge about that group needs to persist and potentially be migrated between different systems.

4 Accessibility

Accessibility is an increasingly important topic in virtual and augmented reality ⁸ and thus it will be for metaverse systems (Dudley et al., 2023). Previous efforts have covered aspects such as supporting users with low-vision (Zhao et al., 2019) through to having one user act in support of a second user with restricted movement (Thiel and Steed, 2021). These considerations pose enormous challenges to the future developers of metaverse spaces: how to support the broadest range of users without all developers having to support all features.

We again highlight that in other domains, such as windowing or smartphone interfaces, there is a variety of support at operating system and interface management layers that help make applications more accessible: from magnifiers to screen readers. Users will want tools that allow them to modify their experiences within the metaverse. Either as plugins to tools (as is done with web browsers) or through some other interoperability mechanism. While VRML97/X3D are no longer used to describe full systems, we would note that within the specification there is a clear responsibility for the browser to provide a set of interaction techniques for certain actions (walk, fly, etc.) but without specifying how these should be implemented⁹. Thus browsers have a lot of flexibility in implementation and can support customisation. While OpenXR¹⁰ provides some functionality to abstract specific of mixed-reality devices, application developers still need to implement the full interpretation of device signals and generate a full-screen interface. Some decoupling of world description from basic interaction would allow users to use the devices that they need or prefer (Steed, 2019).

Finally, we note that some forms of accessibility can be facilitated by sharing of content. Indeed, in general with human-computer interfaces, different forms of collaboration around those interfaces is an important resource to support access (Xiao et al., 2024). In recent years there has been a significant interest in tools that share immersive experiences, either through live streaming video, or more recently through re-sharing immersive experiences (e.g., see (Thoravi Kumaravel and Wilson, 2022)). This touches on the other two themes of this essay: sharing content is an issue of supporting inter-operation between clients and sharing is one way of achieving scale by re-broadcasting. We should not think of metaverses as simple broadcasting systems, but as interfaces composed from multiple components. For example, our recent prototype AccompliceVR (Steed, 2024) adds an overlay network for sharing single user VR experiences running on SteamVR¹¹ to remote users, and having those remote users appear as avatars superimposed in the original VR application. This was inspired by Vermillion¹², which allows users to paint within other SteamVR applications. Both applications exploit features of the SteamVR/OpenXR compositors and overlay systems which were originally designed to facilitate management of the device environment, application launching and immersive system controls. Such functionality hints at a potential interoperability route.

5 Conclusion

In this essay we have highlighted three challenges for future metaverse software. While current SVR software provides many delightful and useful collaborative experiences, there is no clear route to a system that could subsume the wide variety of features and capabilities that are expected. The challenges for any one developer of client software seem almost impossible to overcome with a single client: dealing with scale, interoperating with different services that will need to be implemented by different providers, while supporting accessibility to the broadest population. Our proposal is that we should rethink the role of client software, so that developers can focus on creating small components that are integrated by users or on behalf of users. By integrating lessons learned in other areas of software engineering for distributed applications, interoperability should be possible, which then enables scale and accessibility through allowing composition of systems and interfaces.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

AS: Writing–original draft, Writing–review and editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This work was partly funded EU Horizon 2020 project Research Center on Interactive Media, Smart System and Emerging Technologies (RISE) (grant number 739578).

Conflict of interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.