After the scene data collection project at Meta (read more about that here), where we used prototype Quest VR headsets to scan and label household room environments, we were left with a large data set with thousands of room scan recordings. Since this data was being used to train AI algorithms, the quality of the data needed to be pristine. This meant a human review of each scan looking for bad data. Bad data generally meant planes were misaligned with the scene, mislabeled, or just missing entirely. Below you can see an example of what these recordings looked like, or at least the part relevant for the review process. We had 3D geometry data made of individual planes, along with a video recording from the headset and the camera pose for each frame. This is just a couple seconds of a recording, but generally they were a couple minutes long.
Meta already had a tool developed to view these recordings directly from the database, which myself and a few others on the team used to look through all the scans and flag ones with issues. We did not have a way to fix any bad data though, and so I took on the responsibility to find a way.
There were a few options we discussed:
1) Extend the viewing tool to support editing. This option was ruled out pretty quickly, as the way the tool was built was extremely complex and was not built on a platform easily suitable to visual editing.
2) Build a new tool from the ground-up. This was the direction I wanted to go initially, and planned to create a standalone tool with Unity. I created a mockup for what the tool would potentially look like during the proposal, seen to the right.
3) Build a tool in Autodesk Maya. This was a bit of a last minute thought I had during the discussion, but ended up making the most sense. Maya already offered comprehensive 3D editing features and a flexible scripting system for custom tools and interfaces.
The Maya solution was going to be the quickest and most efficient route by far, so I got started. The feature list for the tool looked like this:
-Import 3D plane geometry data, with labels
-Import video frames
-Import camera pose frame data
-View plane data properly aligned with video frames
-Edit plane data
-Assign new labels to planes
-Export modified plane data
The first step was to load in all the relevant data. This was a collaborative effort between myself and one of the data scientists on the team. Data was stored in Meta’s VRS file format, built specifically to store sensor data recorded from the Quest headsets. Importing an entire VRS file is theoretically possible, however it was going to be a substantial feature to implement in a limited time frame. The VRS files also contained a ton of data, usually around the 10GB range for our recordings. The data scientist created a command line tool that would extract only the essential data directly from the database into a format that would be easy to work with. This saved a lot of development time as well as time uploading and downloading massive files. The output from this tool was a folder that contained an .obj file with the plane geometry and labels, a .json file with the camera pose frames, and a .png image sequence for the video frames. These file formats were chosen since they are all easily supported in Maya.
With the data files ready, I began work on the Maya tool using Python. Obj files are a very common 3D geometry file format, native to Maya with support for named meshes. We used the mesh names to store the plane labels. Maya also supports importing .png image sequences natively using the ‘image frames’ feature. The image frames can be locked to a camera view, meaning I only needed to adjust the camera FoV to match the video. The json data can be easily read in with Python using a json library. The camera pose data in json format was read in and applied to the camera as animation data. Apart from needing to invert the local Z axis for the camera pose, this worked perfectly.
With everything imported, the script then automatically sets up the interface in an optimal way for the workflow. The viewport panel is set to a 2-pane view, one for the recording camera perspective with the video frames, and one for the default free camera view. Below you can see a quick clip of the importing and viewport setup.
In the clip above you can also momentarily see the color-coded labeling buttons on the import window. The plane labels show up in the outliner panel on the left of the screen as GTPlane.FLOOR_0, GTPlane.WALL_3 and such. Any of the planes in the scene can be selected and renamed by pressing one of the label buttons. It assigns the new label following the appropriate naming convention.
Finally with all the data imported and the workspace set up properly, we can start fixing our planes. Anything mislabeled just needs to be selected and the appropriate label button clicked. Planes that are sized incorrectly, misaligned, or have some other geometric issue can be easily identified by scrubbing through the timeline and seeing what does or doesn’t line up properly with the video. Once a bad plane is found, it can be moved, resized, realigned, or whatever needs to be done using Maya’s comprehensive vertex modelling tools. Below you can see this being done with a slightly misshapen picture on the wall. Below that, an example of a missing plane- one of the laptop screens is missing plane data, so the nearby laptop screen plane is duplicated and moved into position. Notice how both the video perspective and the free perspective can be used to align the plane geometry. Scrubbing back and fourth on the timeline also allows viewing and adjusting the plane alignment from different angles.
Once the plane data is all fixed up, the scene can then be exported as an .obj file again using Maya’s export menu. My data scientist coworker created a second command line tool to parse the .obj file and replace the plane data in the original VRS file in the database. Only the scene geometry with labels are needed for uploading, since we never need to modify the camera path or video frames.
The tool worked exactly as needed, and I will never forget the project lead saying “It’s not often a project comes out exactly the way I envisioned it from the beginning”. I was able to create the script in a couple weeks, as opposed to what would have been at least a couple months to create a tool from scratch. Something close to 100 files needed to be fixed and this tool let us do it very rapidly. The bulk of the bad files were fixed within another week, and then as stragglers were discovered it took only minutes at a time for me to fix them.
There were only some minor things that I would improve. The first was the exporting and importing, which I wanted to be able to do entirely within the Maya script instead of using separate tools, but having Maya connect to a high-security network database to access and modify files was a tall order. Second, Maya is an inherently complex piece of software with a steep learning curve, so while I could jump in and modify plane geometry no problem, it was not so easy for others. To remedy this I made some pretty extensive internal documentation that showed how to do a lot of the vertex modeling operations that were useful for our purposes, but learning a 3D modeling program is a daunting task regardless.
This project was a unique one and a fun change of pace from my usual environment working in Unity. Even though I initially wanted to build a tool from the ground up, I’m very glad I went this route instead. I feel it’s a good reflection of my versatile skill set, and I always have fun with the opportunity to do a bit of 3D modeling work.