For a good chunk of my time at Meta I was working on the prototype for Assisted Scene Capture, which would launch in October 2023 with the release of the Quest 3 VR headset. It was one of the most surreal experiences of my career waking up to see Mark Zuckerberg up on stage introducing scene capture (pictured on the left) to a big crowd of people after my team and I had spent the better part of a year working tirelessly on it.
Assisted Scene Capture, or ASC, is the feature on Quest 3/Quest 3s headsets that allows you to scan the rooms in your house, which are digitized into functional geometry within virtual space. This is an exceptionally important bridge for moving from VR experiences to AR, allowing virtual content to interact with the physical space around you. You can throw a ball and see it bounce off your living room wall, or play a digital board game on top of your coffee table. You can check out way more details and examples here. This technology is all built on the foundations established by the Scene Data project I worked on prior to starting ASC. Below is a quick clip of what an ASC scan looks like.
This project was a highly collaborative effort that had a lot of very talented people across a variety of departments and specialties. Myself and one other prototyper were responsible for starting up the Unity project and building out the core structure of it. We did this closely with the Engineering team, who worked on the scanning algorithms and provided interfaces for us to implement the incoming scanning data. The Design team created concepts for the visual style and UX flow. We would continuously implement these ideas and test them out with the team, figuring out what felt good and what didn’t so we could iterate and improve the design from there. We also had some exceptionally talented shader programmers who created a dazzling variety of visuals for the scanning process.
One of the core features for ASC that I was responsible for was the wall editing system. After a scan is completed, a ‘roombox’ is generated, which is an array of planes representing the walls, floor, and ceiling of the room. The room scanning and roombox conversion can make mistakes though; walls can be offset, large furniture can get incorrectly interpreted as walls, or sometimes a user may just want to adjust their space in a particular way. We needed a way for the user to easily and intuitively make those adjustments.
I started by creating a standalone wall editing system in a blank Unity scene. Functionally I would say it was more of a ‘corner’ editing system, since it’s a lot easier to drag a corner around instead of an entire wall. On the right you can see an early version of this in action. We did try out some implementations of pushing and pulling entire walls instead of corners, but it felt very clunky and unintuitive in VR, so the corner method stayed. One particular issue that you run into when implementing a system like this is capturing raycasts reliably. First instinct may be to just raycast to the floor and ceiling, but you may end up in a situation where you move the corner outside of the floor boundary and no longer have any geometry to raycast against, or like in this animation, there is no ceiling to raycast against at all. Raycasting to the spheres that appear on the corners also causes issues because dragging too fast will outpace the sphere. Instead, a dedicated collision plane is implemented. When you begin dragging one of the spheres, the sphere itself does detect the initial raycast, but an invisible horizontal collision plane is activated and moved into position at the corner. This collision plane is the only thing that accepts raycasts while dragging, and the spheres are turned off entirely as to not obstruct the view of where you are dragging.
With basic wall editing working, next step was to read in real wall data from the room scan. The wall editor only keeps track of the floor-level corners of the walls. In the image to the left, these three connected walls have 4 floor corners (0-3). With only these 4 points (plus knowing the room height) you can recreate all 3 walls. The editor keeps track of all connected walls in the scene as arrays of corner positions like this. The incoming scene data, however, contains more literal wall data where each individual wall has position, rotation, width, and height. Additionally, the scanned walls always have some inherent noise in their properties, so the edges and corners never perfectly line up.
Some basic vector math is done to find the lower corners of each wall. This gives us a separate array for every wall with 2 corner positions each. We then compare all corner positions and look for ones that are very close together. When two corners are close enough together, they are snapped together and ‘merged’, so both walls become part of the same array of corner positions. On the right you can see the incoming wall data in red and then the converted walls in purple. Note how the gap between the walls disappears during this process. This edge snapping ended up being very useful to clean up the incoming scene data. After the scanning process is done, the user is brought to editing mode and the walls are converted to corners by the wall editor automatically. Even if the user does not make any adjustments, the walls are still converted from corners back into wall format, keeping the tightened up position data and ensuring there are no gaps in the walls which could cause physics and rendering issues later on.
Features for adding and removing walls were also implemented. Corners could be dragged onto other corners to merge large gaps between walls. Limitations had to be put in place like only allowing corners on the open ends of walls to merge, otherwise you could get strange behavior like three walls sharing the same corner.
With the whole wall editing system working, it was imported into the main scene of the project. This process went very smooth because because of my approach refining the editing system in an isolated environment and ensuring modularity. The mouse click interaction was swapped out for controller raycasting and the tool was integrated into the existing UI flow. On the left you can see me playing with the ‘add wall’ button and placing walls around my messy office room.
There were so many other things that went into prototyping ASC, but the wall editor was where I had the most fun. I was very proud of how clean and reliable it was, especially in such a fast paced and ever-changing environment.
We implemented and tested a lot of additional features and ideas, some of which didn’t work that well in practice, some of which were shelved for later and may appear in future updates to ASC. Once we had the design polished up and approved by management, we handed it off to another development team who would implement it in the OS with native code. You can check out this video to see what a full ASC scan looks like from start to end.