360 Degree Virtual Tours are buzzwords today that find numerous mentions in diverse technical and business literature worldwide. But is the present technological ecosystem scalable and pervasive enough? And, most importantly, does it address the most pressing problems that industries worldwide are facing today?
With the global pandemic on the rise, the reachability of physical spaces has drastically reduced, and it is proving to be a major challenge for several industries and several lines of business. Experiencing physical spaces remotely and digitally is the need of the hour. Digitization does seem to be the obvious answer to the problem, and 360 Degree Virtual Tours seem to be the prima facie solution. However, such technologies can help craft viable solutions to the problem only if they become scalable, maintainable, pervasive, and highly democratized.
Introduction
The global pandemic, coupled with other factors, has caused drastic reductions in footfalls, sales and conversion ratios across many industries. Several lines of business have either been diminished in scale or have been shut down entirely. One of the peak reasons for this is the limited reachability of physical spaces. In addition to limited physical reachability, the other major factors affecting several industries today stem from an over-dependence on people and traditional processes.
Solving such modern business problems, especially those that arise due to limited physical reachability, often involves using many cutting-edge immersive technologies in a seamless combination. This is applicable for several industries and apt across different value chains.
Immersive technologies encompass a wide range of areas, including:
- Augmented/virtual/mixed reality
- 3D computer graphics
- Computer vision and image processing
- Machine learning/deep learning
- Voice/chat AI
- IoT
As advanced immersive technologies inch closer towards the stage of democratization, the adoption of such technologies to solve complex business problems is set to go through an increasingly upward trend in the near future. Immersive technology adoption is poised to see the highest growth, especially in scenarios where physical reachability is challenging.
Current trends and pervasiveness of XR technologies
XR and other related immersive technologies have greatly evolved over the past few years through advancements in 3D computer graphics, computer vision, deep learning, and other related fields.
The adoption of XR is picking up a significant amount of pace, thanks to the recent advancements in mobile devices and wearable head-mounted devices. Such devices are now equipped with cutting-edge hardware: GPUs, processors, and co-processors that can simultaneously crunch millions of polygons and process complex neural networks.
In addition to advancements in hardware and software, the introduction of 5G comes with a promise to lean down bulky XR applications by letting them stream immersive 3D content (instead of hosting the content on-edge).
“The number of announced 5G devices continues to grow rapidly, this month exceeding the 700 mark for the first time to reach 703 announced 5G devices. This is more than 25% higher than the number of announced devices at the end of Q4 2020 and an increase of 11.9% over the last month alone. Meanwhile, the number of 5G devices understood to be commercially available has risen by more than 28% over the last quarter, now standing at 431 commercial 5G devices, which represents 61.3% of all announced 5G devices.”
“153 operators have already launched commercial 5G networks in 64 countries/territories.”
(Source: Global Mobile Suppliers Association)
However, the XR industry has numerous challenges of its own.
One of the key challenges that the XR industry has been facing for quite some time is sourcing/producing 3D content. Much of the 3D content required for immersive experiences is still mostly produced manually (except a small subset of objects that can be 3D scanned and used). This manual production of 3D content often involves artists with specialized talent and a complex workflow to craft lifelike 3D models and experiences.
The tedious nature of the people and processes involved in generating 3D content has led several organizations to look out for better and cost-effective alternatives. One of such alternatives is 360-degree views of spaces.
Challenges due to limited physical reachability
As mentioned earlier, physical reachability is one of the main challenges that industries worldwide are facing today. Let us consider a few examples.
- Real estate industry: Lesser footfalls and reduced physical property visits.
- Auto industry: Reduced footfalls in car showrooms.
- Hospitality industry: Reduction in customer confidence (due to the pandemic), resulting in reduced customer visits.
- Research labs and scientific establishments: Difficulty in providing physical tours of facilities to different stakeholders.
- Healthcare tech industry: Sales and marketing professionals finding it hard to provide physical product demos.
- Manufacturing: Difficulty in providing physical tours of facilities to different stakeholders.
Such problems primarily result in the reduction of sales and conversion ratios. Let us consider the example of the real estate industry.
Modern-day property brokers rely heavily on digital means to enlist properties for sale, lease, or rent. Such listings typically include details such as:
- Type of the property (Apartment, Villa, Studio, etc.)
- Property location
- Age of the property
- Total area of the property
- Property price
- Images, snapshots, and/or videos of the property
Images, snapshots, and/or videos alone may not be enough to instill the desired level of confidence in potential consumers to make informed decisions and finalize property deals. Virtual walkthroughs could be the solution to the problem. However, as of now, virtual walkthroughs are available for premium properties only.
As a result, interested customers usually connect with the broker separately and arrange for a physical visit to the property to experience it first-hand before they decide to close the deal finally.
However, there are multiple challenges with physical visits. To begin with, arranging a physical visit takes time, and both parties should agree to a common time slot that suits them. That becomes even more difficult if the broker and the consumer are geographically far apart from each other. Moreover, given the global COVID pandemic situation, physical visits are difficult and undesirable from a safety perspective. Additionally, most properties are almost totally closed to visitors following the pandemic. Open houses have been reduced, and physical visits have been curtailed.
With limited physical access to properties, combined with other factors, property brokers are now experiencing a decreased footfall, resulting in a gradual decline in the conversion ratio. Consequently, sellers are experiencing longer conversion times for property listings.
Circumventing the problem of physical reachability
Establishments and industries need to offer remote, immersive, and digitized tours of their physical spaces to alleviate the problems mentioned previously. Such digitized tours have the potential to:
- Transform businesses to require fewer physical visits
- Increase footfalls, sales, and conversions
- Improve customer experience and customer loyalty
However, there are multiple ways in which digitized tours can be authored, and the applicability of each method varies based on the situation and the organization’s business goals.
Not all the methods are applicable everywhere. The applicability primarily depends on the existence or availability of the physical space in question. For example, labs that can be photographed can be covered through either 2D video-based tours or 360-degree virtual tours (both video as well as static). Whereas, in the real estate industry, model flats that don’t exist physically can be imagined and toured through a fully crafted 3D experience.
Spaces with physical existence have the flexibility to be visualized through all the different modes of visualization depicted above. For such spaces, the balance between immersiveness and ease of authoring drives the choice of technology.
In this article, we specifically focus on 360-degree static panorama-based tours.
Factors affecting widespread adoption of 360-Degree Virtual Tours
360-Degree cameras have existed for quite some time now and are being used in many places. As discussed above, 360-degree tours are either video-based or static panorama-based. Although there are several 360-degree cameras available in the market, 360-degree tours are still not pervasive enough. The widespread adoption is still a big challenge.
There are several reasons for this:
- 360-degree cameras typically produce high resolution videos which are often unsuitable to stream or play on many devices. Thus, 360-degree videos are throttled down to lower resolutions to make them playable on low to mid-range devices. This causes a significant loss in the quality of the video.
- Operating such 360-degree cameras requires special experience.
- Requires complex pipeline to stitch individual photos into seamless 360-degree panoramas.
- 360-degree video/static walkthroughs do not offer much interactivity.
- 360-degree video/static walkthroughs do not offer connected experiences.
- Most 360-degree tours neither include annotations, nor waypoint-based navigation, nor any embedded rich multimedia content.
- Most 360-degree tours do not work on simple phones and typically require high end hardware to render high resolution panoramas/360-degree videos.
Making 360-Degree Virtual Tours more pervasive, immersive, usable, and scalable Several components are required to bring virtual tours (and the authoring of virtual tours) to the fingertips of end consumers. The following are some of the key differentiating factors required to make 360-degree virtual tours more pervasive, immersive, usable, and scalable:
Mobile first solution: Virtual tours should be crafted keeping mobile phones in mind. Such mobile devices have limited hardware capability but, they are some of the most pervasive devices in the world.
Should be possible to capture with different types of cameras: From sophisticated 360-degree cameras to simple mobile phones, catering to the novice and expert alike.
An annotation platform: To enable users to annotate, activate and embed rich multimedia content as well as highly interactive waypoint-based navigation.
Seamless sharing and connectivity: Consumers should be able to connect with other people and share the same setup/environment. This would enable multiple users to experience and share the same virtual space in real-time. This can be made possible through multiple channels and technologies – Twilio/Teams Integration, Video Calling, Shared VR.
A scheduler to schedule meetings: To enable users to host meetings and immersive walkthroughs in shared, virtual 360-degree spaces.
Integration with CRM and other existing backend enterprise systems: To make virtual 360-degree tours more seamless, scalable and accessible.
Recent advancements in technology and the rapid evolution of smartphones (and similar handheld devices) have made it even more feasible to realize the requirements mentioned above. Capturing 360-degree panoramas, which is one of the biggest challenges in crafting such virtual tour experiences, can be made more seamless, user-friendly, and pervasive through a combination of cutting-edge hardware and software.
Capturing 360-degree panoramas needs to be more democratized. But, do today’s handheld devices have what it takes to offer seamless capturing of 360-degree panoramas?
The technological capabilities required on handheld mobile devices to facilitate the capture of 360-degree panoramas include (but are not limited to) the following:
Audio-video/camera framework – Accessing the device camera and setting the right set of parameters for panorama capture.
Sensor-fusion: Combining readings from different device sensors, viz., the accelerometer, the magnetometer, and the gyroscope to interpret the device’s orientation real-time.
Geo-location: To determine the current location of the user.
Advanced GPU-accelerated 3D rendering: Using 3D graphics and sensor data to guide the user to capture panorama images and rendering the captured panorama images in a spherical manner in real-time.
The whole process of capturing 360-degree panoramas can be brought on to handheld devices like smartphones and tablets, allowing the common mass to capture 360-degree panoramas, which was earlier available only with expensive and sophisticated 360-degree cameras.
The possibility of democratizing 360-degree panorama capture can be attributed to modern handheld devices, which have enhanced audio/video, sensor-fusion, geo-location, and GPU-accelerated 3D rendering capabilities.
As of March 2021,
- 100% of the iPhones in use support AR (through the ARKit framework)*
- 384 Android phones support AR (through the ARCore framework) **
These devices have all the features required to enable real-time 360-degree panorama capturing. Additionally, the number and percentage of such devices are only increasing.
Moreover, it is not only about being able to capture 360-degree panoramas. It is also about being able to render them, along with additional rich multimedia, interactive content, and animations. Modern handheld devices, backed by multi-core GPU-integrated chipsets, can prove to be more than capable of rendering such integrated, immersive experiences.
*Source – Apple, iOS Device Compatibility Reference
**Source – Google, ARCore Supported Devices
The technical approach to realize 360-Degree Virtual Tours
Before taking a technological deep dive, it is essential to understand the components and the aspects that need to be addressed in the technical solution.
Now, let us examine the technical pipeline depicting the overall flow, beginning at the capture (of panoramas) and culminating in the published virtual tour.
It should be re-emphasized that the biggest hurdle to the democratization of virtual tours is the dependence on sophisticated 360-degree cameras. As the first component in the pipeline, we introduce a simple and easy-to-use mobile app that allows consumers to capture panoramic photographs of the physical space seamlessly. It uses state-of-the-art motion sensing (through a combination of accelerometer, gyroscope, and magnetometer data) to guide the user to capture photographs in a sequential-spherical manner. Such a feat has been made possible thanks to the enhanced capabilities of modern smartphones, making them viable alternatives to sophisticated 360-degree cameras.
During the capture process, multiple vertical rows of photographs need to be captured. This can be a daunting task for several users, especially if they don’t have access to tripod stands and if they have to resort to capturing all the photographs by hand. This problem can be further compounded if the user wants to capture multiple panoramas of the physical space.
Thus, the capture process should be made as seamless as possible, especially from a user experience perspective. Fast and accurate photo capture (with minimal handshaking) is the key to a good panorama capture experience. One of the ways in which the capture process can be made less tiring and more entertaining is through gamification.
Capturing the individual photos (which would be used to create the spherical panorama) is just the first step in the entire process.
Once the individual panoramic photos get captured, they can be uploaded to the cloud, where a CV-based spherical panorama stitching tool can be used to stitch the photos together into seamless 360-degree panoramas. This stitching can be achieved through multiple techniques. Description of the individual stitching techniques is beyond the scope of this paper.
After the spherical panoramas are stitched, they can be pushed to an ‘annotation tool,’ which needs to provide a simple-to-use drag-drop interface to add annotations, rich multimedia content, and waypoint-based navigation to enable users to craft immersive 360-degree virtual tours.
As mentioned earlier, it is imperative that the virtual tour be ‘shareable’, i.e., it should allow multiple users to access the same digital environment simultaneously and in real-time. Now, there are multiple methodologies to achieve such immersive, real-time sharing:
Integrating virtual tours with existing enterprise systems is of utmost importance for better adoption and future scalability. Integration should be performed without disrupting existing ecosystems.
Let’s continue with our example of the real-estate industry.
Let us suppose that a property broker/seller wants to post a property ‘listing’ on an existing property app/website with an intent to sell/lease out the particular property. A 360-degree virtual tour can then be added as part of the listing to attract potential buyers and influence their buying decision.
As mentioned earlier, listing a property requires many important pieces of information put together. A 360-degree virtual tour will be one of the missing pieces of the puzzle (as shown below).
Aspects to take into consideration while crafting Virtual Tours
Several aspects should be taken care of while crafting virtual tours. These aspects affect the democratization of virtual tours, either directly or indirectly.
- Would it cater to devices with varying processing power, screen size, aspect ratio, etc.?
- Would it run seamlessly on different browsers (and operating systems)? Virtual tours need to be optimized to run on different browsers through lazy-loading, prefetching and caching mechanisms.
- Apart from capturing panoramic photos, special attention needs to be given to the rendering of 360-degree virtual tours as well. Rendering should be fast and qualitative. It should not require high-end GPUs and should provide satisfactory results on mid to low end GPUs as well.
Benefits and how to reap them
There are several benefits that 360-degree virtual tours can offer, and comprehensive steps are required to reap the benefits.
- Better customer experience, customer retention; better sales and conversion ratios
How to achieve - Customers need to be educated and encouraged to use virtual tour experiences through targeted marketing, continuous improvement and gamification. - Faster time-to-market
How to achieve - Through meticulously crafted user experiences, 360-Virtual Tours have the potential to offer much faster time-to-market as compared to full 3D experiences. - Works on low-end as well as high-end devices
How to achieve - Through the use of lazy-loading, prefetching and caching mechanisms, virtual tours have the potential to cater to devices with varying processing powers.
Conclusion
360-Degree Virtual Tours have the potential to evolve into highly democratized, immersive, and scalable solutions. Through a meticulous and well-thought-through strategy, such tours can be realized with better market penetration, user adoption, customer experience, and retention. As devices, platforms and ecosystems continue to grow and evolve, such rich and immersive experiences are only going to become more and more pervasive in the near future.
Reference architecture
Author: Chinu Subudhi
As the lead architect for AR/VR in Mindtree, Chinu is responsible for the research, innovation, conceptualization, and design of platforms, solutions, and services leveraging cutting-edge immersive technologies. He has worked on multiple different technologies and industry verticals with 11+ years of wide-spectrum experience in the IT industry. He is also a US patent holder on Holography.
