Tuesday, August 17, 2021

Creating in New Realities: Mixed Reality Development from a Student’s Perspective

A student trying out a HoloLensA student trying out a HoloLens

 

How would you describe yourself, what you do, and what technologies you have worked with?

 

Jing Low: I am a Software Engineering Intern at Microsoft, working under the Spatial Computing team in Cloud Advocacy. I created technical content (demo project, tutorial, and Microsoft Learn module) for the Babylon.js framework and WebXR technology. Outside of the internship, I am also an undergraduate student pursuing a Computer Science and Linguistics dual degree at the University of Florida. Prior to joining Microsoft as an intern, I mainly focused on full-stack web development and dialogue systems.

 

Gustavo Cordido: I am a Cloud Advocate at Microsoft, working as part of the Spatial Computing team. I currently specialize in creating Academic technical content focused on Mixed Reality and services such as Azure Spatial Anchors and Babylon.js. I graduated from Florida International University in December 2020 with a bachelor's degree in Computer Science where I worked as an undergraduate researcher in Natural Language Processing.

 

What was your first impression of Mixed Reality?

Jing: “Wow this is so cool!” basically sums up my first impression of both playing with and developing for Mixed Reality. The only VR game I played before is Beat Saber and it remains one of my favorite games to this day. The cool graphics and the realistic vibrations from the controllers really made me feel like I’m in a totally different world. Before the internship, I  did not own a Mixed Reality headset nor had developed anything for the virtual space. Since I started with WebXR, I was also really surprised by how accessible the technology has become.

 

A group of students analyze a 3D render using HoloLens.A group of students analyze a 3D render using HoloLens.

Gustavo: My first impression of Mixed Reality was that of a non-believer. I could not picture this technology to be real as I had not had any first-hand experience and thought that what was pictured in products such as HoloLens and VR headsets was blown out of proportion by the media. However, after giving these devices a try, my disbelief was rapidly changed. I could not fully comprehend what I was seeing nor doing. Interacting with a digital object right in front of me! From there I could not stop myself from jumping into reading more and more about the technology to try to understand how these things were made possible.

 

What language and environment did you start working in?

Jing: Since my internship focuses on WebXR, I started with Javascript/Typescript for creating Mixed Reality web applications. Specifically, I started with learning about the Babylon.js framework in general and then moved on to its WebXR support.

 

Gustavo: My first interaction with Mixed Reality development came through the use of C# in Unity. Learning the basics of digital scenes and how to interact with digital objects placed in them. From there I started delving into WebXR and learning about Babylon.js, which meant learning JavaScript as a language I had never touched upon before.

 

What resources did you access first?

Jing: I’m a huge fan of reading documentation over watching tutorial videos, so the official documentation of Babylon.js was the first resource I accessed and followed through. The “Getting Started” section is extremely helpful for learning about creating Babylon.js scenes in general, and the “WebXR” section under “Diving Deeper” taught me how to convert those scenes into VR and AR spaces.

 

Gustavo: I usually approach learning by doing, so I got into as many hands-on tutorials as I could to begin with. My first series of tutorials came directly from Unity Learn platform. However, some of the Mixed Reality fundamentals were missing and as such I did some readings on Microsoft Docs.

 

What do you like about MR the most?

Jing: Making the impossible possible. Even though MR is really useful for simulating 

real-life environments in a virtual space, I think it is the power of adding non-real-life components to the space that makes MR the most fun. For example, in the virtual space, we can make ourselves wizards by programming certain behaviors that simulate magic.  We can use voice recognition to render a magic circle or special effects when someone says a specific spell.

 

3D Render of a Dragon3D Render of a Dragon

Gustavo: For me it has to be the endless possibilities that come with Mixed Reality. The blending of both the digital and real worlds will not only benefit the industry, but also our everyday lives. I attended an anatomy class taught via HoloLens and experienced the use of Digital Twins in industry machinery, and could not believe the development that was happening. I also thoroughly enjoy the fact that we can develop MR experiences for multiple devices and using different methods, making it incredibly accessible for all developers.

 

Healthcare professionals using HoloLens to interact with a holographic render of an anatomical modelHealthcare professionals using HoloLens to interact with a holographic render of an anatomical model

What have you found most challenging about MR development?

Jing: Since I am used to designing interfaces for 2D websites, it was a little challenging for me to get used to positioning and adjusting objects in a 3D space.

 

Gustavo: I believe adapting to 3D modeling and objects coming from a traditional development background has been the biggest challenge. Not only did I have to worry about their positioning in space, but also their dimensions and their possible interactions with any other objects in their surroundings.

 

Did you find any prior school classes useful when first delving into MR development? If so, which ones?

Jing: This is not super necessary, but Geometry and Calculus 3 helped me get familiar with the 3D space in mathematical terms!

 

Gustavo: Multivariable Calculus and Physics were definitely helpful in the process of creating in a 3D space, as I was already familiar with the implications of three coordinate systems.

A 3D graph of a cubeA 3D graph of a cube

Which resources would you point a beginner towards reading or accessing to start developing in MR?

Jing: The Microsoft Docs WebXR section contains tutorials that go through the basics of Babylon.js and dives right into adding WebXR support to the web page, while the Babylon.js official documentation contains  more in-depth information about the 3D rendering framework in general as well as the specifics of WebXR support they provide. If you are not as interested in using Babylon.js to create complex meshes and scenes, the WebXR section in Microsoft Docs might be a better fit.

 

Gustavo: My first dive into Mixed Reality came through the Introduction to mixed reality Learn module. Its contents helped me familiarize with some of the general concepts in the field, differences between mixed reality, virtual reality, and augmented reality, and some of its use cases. From there I started getting familiar with C# and Unity development, as well as Windows Mixed Reality development. I highly recommend giving the Resources for Getting Started with Mixed Reality for Unity article a look!

 

If you would like to stay in touch, please follow through one of these:

Jing

Gustavo

Posted at https://sl.advdat.com/3y5XKeR