Multiplayer AR with RealityKit — What Went Wrong??
Are you seeing what I’m seeing?
Augmented Reality on its own can be a great experience and a powerful tool, but it is so much better when you can share the experience with other people at the same time. If you would like to work on building a Multiplayer AR app with RealityKit, one thing you will need to check for is if the versions of RealityKit are compatible.
(If you have never worked on an AR app with Multipeer Connectivity before, you can find a nice tutorial on how to do this here. This article covers important topics which are not covered in that tutorial.)
Sharing ARAnchors
There is a difference between sharing ARAnchors and having network synchronization. I bring this up because you need to check for compatibility when using network synchronization, but not when just sharing ARAnchors. By the end of this article, I anticipate you will want to use network synchronization as well (and therefore check for compatibility). Let’s see why:
Apple defines ARAnchors as “A position and orientation of something of interest in the physical environment.” This means the anchor represents the position and orientation of something in the real-world environment which we are tracking, so that we can attach virtual content to it. When you are sharing CollaborationData in a Collaborative Session, you are sharing ARAnchors. Sharing ARAnchors means both devices know where they are in space relative to one another (with ARParticipantAnchor), and both devices agree on where any ARAnchors placed into the scene are. You cannot actually see these anchors unless you insert:
arView.debugOptions.insert(.showAnchorOrigins)
In RealityKit, “Entities” includes the 3D objects in your scene that you can see — that kind of entity is called a ModelEntity.
You can see virtual objects that you attach to the ARAnchors.
(More specifically you would make an AnchorEntity from the ARAnchor with let anchorEntity = AnchorEntity(anchor: ARAnchor) and then add the ModelEntity as a child to that AnchorEntity. You could alternatively make an AnchorEntity that targets a type of ARAnchor with let anchorEntity = AnchorEntity(.plane(.horizontal, classification: .any, minimumBounds: .zero) and add your ModelEntity as a child to that AnchorEntity with anchorEntity.addChild(modelEntity))
You can even attach entities to ARParticipantAnchors and have the entities move around as if they were attached to the other device:
Sharing ARAnchors does not mean, however, that any interaction with entities will be visible on both devices. i.e. I can add an identical-looking sphere to the same ARAnchor on both devices, and this gives the illusion of multiplayer, but these are actually two different spheres. As long as you programmatically make them both move the same way, they will appear to be the same sphere, but they are not the same sphere. This the type of “multiplayer” that is being used in this sample app from Apple, and in the tutorial I previously linked to.
As soon as you incorporate dynamic physics:
myModelEntity.physicsBody.mode = .dynamic
Or collisions, user interaction (such as dragging the object around, adjusting its scale/size, rotating it), you are likely to see the two spheres move and behave in different ways.
If you want to make a true multiplayer app, you want your entities to be able to interact with my entities. i.e. If we were making a multiplayer AR 8-Ball / Pool game, I want to be able to strike the same pool balls that you strike and have us both watch the same pool balls do the same thing.
So how do we make both devices see the same entities and see them move and behave in the same ways? When you are sharing Collaboration Data in a Collaborative Session, you are sharing ARAnchors, but we need to *also* include network synchronization on top of this.
Making Network Synchronization
You will actually notice that Apple includes synchronization components by default on all of your entities. This component is used to synchronize an entity between devices so that it looks and behaves the same on each of them.
As a side note: In their sample apps which are not multiplayer, Apple programmatically removes those synchronization components to save memory, and I suggest that you do the same on non-multiplayer apps.
Then you can call myEntity.removeSynchronizationComponents() and it will remove the Synchronization Component on that entity and all of its children (I recommend calling this method on Anchor Entities since this most easily covers all of the entities in your scene).
But for apps that are local multiplayer, you clearly would want to leave these components attached. So how do we make use of those components? Some swift packages do not set up synchronization and checking compatibility automatically for you. If yours does not, we must assign something to be our scene’s synchronizationService, like this:
Keep in mind: if you are using a swift package to set up your Multipeer Connectivity Session, you may need to edit it in order to access the session variable. To do edit the package, you will need to make it part of your main bundle.
Also, make sure your session’s configuration has isCollaborationEnabled set to true.
The version of RealityKit running on the device comes bundled with whatever version of iOS that the device is running. So having the wrong match-up of iOS versions can make this synchronization not work.
To make the comparison, you need to send your local NetworkCompatibilityToken to the other device, and handle the NetworkCompatibilityToken coming from the other device. We will encode our local token into our discoveryInfo array, which theMCNearbyServiceAdvertiser below will automatically send to the other device for us.
When you make your MCNearbyServiceAdvertiser , pass in this discovery info.
serviceAdvertiser = MCNearbyServiceAdvertiser(peer: myPeerID, discoveryInfo: discoveryInfo, serviceType: serviceType)
Now for the incoming data from the other device:
One of the best places to call this checkCompatibility() function is inside of this delegate function:
public func browser(_ browser: MCNearbyServiceBrowser, foundPeer peerID: MCPeerID, withDiscoveryInfo info: [String: String]?) {}
Here We Go!!
I hope this article helped clear things up for you! If you felt like this article was beneficial please let me know by clapping it up (members can clap up to 50 times per article) and leaving any comments you have. You might also like my article on why your multiplayer AR App is breaking which goes into further detail about asking for permissions.
