Effect of Smooth Transition and Hybrid Reality on Virtual Realism: A Case of Virtual Art Gallery

Article history: Received: 10 February, 2021 Accepted: 21 April, 2021 Online: 23 May, 2021 Virtual reality (VR) is finding applications in a wide range of industries; however, a significant number of users find VR experience considerably different from the real-world experience. To match the real-world experience, the VR experience should look real, should be immersive, and be in line with the users' anticipation. Achieving realism in the virtual representation of objects, however, presents several technical challenges. This study presents a new approach, "Smooth Transition and Hybrid Reality (STHR)," to easily enhance VR realism. In this approach, the participants are exposed to a mix of real-world objects (RWO) and virtual objects (VO), and a smooth transitioning from the real world to the VR space is obtained by making the real-space highly relevant to the VR space and viceversa. To test the effectiveness of STHR, different experiments on spatial awareness were conducted, and finally, a virtual art gallery was created for the public. A total of 21 participants were included in the study and were randomized into experiment and control groups. The results indicated that the interaction with an RWO in the initial phase of VR significantly increases the task completion time and the attention (both, p<0.05). It was found that almost 50% of the participants relied on prior knowledge of the real space even when different visual information is delivered through VR. STHR based virtual art gallery (VAG) was created for the public, and the experience of random visitors was noted. In VAG, haptic feedback was provided by using an RWO (3D printed artwork), and the smooth transition from RW to VR was maintained. The average time spent in the VAG more than 5 minutes, and the feedback of visitors was highly positive.


Introduction
Technological advances make a significant impact on all forms of art and even create new art forms. Virtual reality (VR) and augmented reality (AR) are poised to make a remarkable shift in the way art is created, presented, and appreciated [1,2]. However, in VR, replicating the effect that real artworks make on human perception presents several technical and psychological constraints [3][4][5].
The perceived quality of artwork usually differs from person to person, depending on their previous experiences and subjective interpretation of the visual information, and involves multi-process cognition and emotional associations [6,7]. Typically, at art galleries and museums, observers view the artwork from a distance. The perception primarily relies on the processing of visual information of depth, distance, and angle. The ultimate goal of VR technology in art is to achieve digital art appreciation in the digital art space that is at least on par with its conventional counterpart. However, in the context of VR, as the viewers already know they are observing a virtual object, comprehension of "reality" gets complicated and greatly depends on the immersion and presence rendered by the overall VR experience [8,9].
The space setup and the method to transition into VR significantly influence the overall VR experience. It has been reported that creating a gentle transition from the real world into the virtual and back into the real environment could be an excellent solution to enhance the realism of VR. In a recent study, the effect of a gradual transition between the real world and VR was investigated by using a video feed from a stereo camera that gradually faded into the virtual content, creating a smooth transition, which significantly affected the participant's perception of virtual body ownership and presence [10]. The major issue with this smooth transition method is that a new 3D model must be provided for each different space. Constructing realistic 3D spaces is time-consuming, and, in several cases, the meant-to-be-shown virtual environment is entirely different from the real space. Furthermore, there can be differences in the scale of the space or in content [10][11][12][13]. However, despite all significant research in this ASTESJ ISSN: 2415-6698 area, the factors that affect VR experience quality are not yet fully established.
In this work, we introduced the concept of Smooth Transition Coupled Hybrid Reality (STHR), wherein the participants were first deliberately introduced to a physical object, which was the replica of one of the virtual objects that the participant would see during VR. This step was expected to provide haptic feedback that matched the real world and the VR [14,15]. Furthermore, the setup was designed to allow a smooth transition from the real world to the virtual world and vice-versa. This step involved making the real-world theme highly relevant to the virtual world and blocking all extraneous content to the maximum extent possible. The study also attempted to clarify the fundamental aspects of the impact of spatial awareness on the overall VR experience.

Methodology
The objective of this research was to find out approaches that can be utilized to enhance the VR experience. To that end, the concept of STHR was developed, which involved providing haptic feedback by a real-world object and enabling a smooth transition from a real-world environment to VR environment. To test the efficacy of STHR, we conducted different randomized experiments and examined anticipation, reality, immersion, and attention participants during their VR experience [8,15,16]. We have also monitored task-complication time and the number of collisions in different study groups. Finally, the validity of the STHR concept was demonstrated by creating a virtual art gallery for the common public and recording the realism experienced by random visitors [17].

Participants and questionnaire
There were 21 participants in the experiment. Before the experiments, all participants were asked to fill a questionnaire related to the basic information and previous VR experience. Each participant had to participate in all experiments. After each experiment, they were asked to fill a Presence and Reality judgment questionnaire [18]; the questionnaire had 25 questions that were to be scored on a scale of 1-10. Anticipation, reality, immersion, and attention during VR were analyzed from the questionnaire. At the end of all three experiments, we also conducted a survey and an interview with questions based on the participants' behavior.

Experiments
A total of three experiments were conducted in the same place in a specific sequence, one participant at a time. We chose a room that was unknown to all participants to eliminate any knowledge of the space. Before entering the experiment room ( Figure S1), the participants were asked to wear an eye mask, close their eyes, and keep them closed until they entered the room and put on the headmounted display (HMD). When the experiment was over, they were asked to close their eyes again until they left the room.
In the first experiment, we examined the effect of hybridreality on realism, i.e., we created the setup so that the participants interact with a real-world object (RWO) in the initial stage of the VR experiment. This approach provides haptic feedback that matched the real world and the VR gallery. The virtual space was a room with some furniture, a column offset from the center of the room creating a narrow passage, and some traffic cones to act as obstacles while walking in the room to perform the task ( Figure 1, Figure 2). The participants were divided into two groups and randomly assigned to one of the two conditions. The first condition introduced some real-world objects (RWO) (a table and chair) with similar shape, size, and VR location as in the real environment. These objects were placed near the participant's starting point, which forced a natural interaction through either deliberate or accidental. These matching objects were the only things inside the experimental space. In the control group (CG), there were no real objects. For each group, we measured the time taken to complete the task of touching the red boxes, noted their routes of movement, and observed the distance traveled by the participants to avoid the obstacles, and compared them.  In the second experiment, we examined the difference in realism and task completion time when the participant is already aware of the real space. Essentially, it was a measure of the effect of spatial awareness of the real space on VR realism. The boundaries of the virtual and physical worlds were the same. Traffic cones were scattered around the virtual space that acted as obstacles when a participant walked into the room to perform the task of touching the red boxes ( Figure 3). However, the experimental space did not have any physical object-it was empty. The experiment had two primary setup conditions: an environment where the participants can see the room set up to know it is empty (control group) and a group where the participants cannot see the actual space where the VR experiment was conducted. In this setup, a barrier (wall) blocked the participants' view of the real space. We compared the time taken to complete the tasks, the route of movement, and the distance traveled by the participants. In the last experiment, all the participants were allowed to see the room before wearing the VR headset; however, the VR setup was clearly different from the real space ( Figure 4). We aimed to examine if the participant's memory of the real world would persist during the VR experience and how this knowledge affects their decision-making while experiencing VR.
This experiment focused on the route and actions taken by the participants. There was an obstacle (table) in the real environment that did not exist in the VR space. This obstacle blocked the path needed to reach the second red box. Thus, this was a bad environmental setup for VR.  However, this answers the following question: "how will the participants treat the real-life obstacles if they do not see them in VR? Will they remember them?" Besides, after touching the second-to-last box and on the way back to the starting point where the last box was, a new obstacle (chair) was rendered, which did not exist in the real world. This will answer the question that "How will the participants handle the newly spawned virtual obstacle? Will they prioritize their knowledge of the real space, or trust the visual information provided in VR?"

Task
In the experiments, the participants were asked to perform a simple task of touching some red glowing boxes, which turned green when touched and then spawned a new red box in a new location in the virtual room. This task encouraged the participants to look and walk around the space while avoiding some room obstacles [19].

System
The system consisted of an HTC Vive headset with a Leap Motion sensor mounted on it for hand tracking and Vive motion controllers with 3D printed mounts fitted around the ankle to track the movement of feet. We tried to represent the use with a semirealistic avatar in VR to help maintain the level of immersion [20][21][22][23]. The experiment space dimensions were 4.4 m x 3.7 m, with the HTC Vive lighthouses set on two opposite corners. We run VR content on an ASUS laptop (model GL502VS) equipped with Intel Core i7-6700HQ 2.60 GHz CPU, 16 GB RAM, and GTX 1070 graphics card. The content was created using Unreal Engine 4, which showed a 3D room modeled in Autodesk 3DS Max.

Virtual Art Gallery
To verify the validity of STHR in a real-life situation, we made a VR art gallery (VAG) that was open to the public for four days ( Figure 5, Figure S2-S4). It was held in a multi-purpose space on the second floor of the designed common building of Kyushu University. The gallery showed four different 3D sculpted artworks with themes based on Bonsai plants, which are an integral part of Japanese culture. The VR gallery design was based on conventional Japanese interior design. We used elements such as Tatami (Japanese flooring), Byoubu (Japanese folding partitions), and Shoji doors. STHR was defined as delivering a smooth transition from the real world to the virtual world and providing haptic feedback by creating a mix of real-world and virtual objects. In VAG, STHR involved four main concepts.
Most important among them is the haptic feedback that matched the real world and the VAG. It was introduced at the beginning of the VR experience by using a 3D printed replica of an artwork. Additionally, to assure a smooth transition from the real-world to VR, important features of the real space were maintained in VR as well. In our case, it was the windows of the building and the ceiling. Followed are more details on the public VAG.
The system used in the gallery was similar to the previous experiments. It consisted of an HTC Vive headset and a Leap Motion Controller mounted on it for hand tracking. Vive motion controllers to track the movement of feet were not used because collisions were not anticipated. The gallery space dimensions were 4 m × 6.5 m, with the HTC Vive lighthouses set on two opposite corners. The computer was equipped with Intel Core i7-6700HQ 2.60 GHz CPU, 24 GB RAM, GTX 980Ti graphics card, and Windows 10. The content was created using Unreal Engine 4, which showed a 3D room modeled in Autodesk 3DS Max and ran at 90 fps with 6 ms latency.
Before setting up the gallery, we made a poster informing visitors that the gallery was in a traditional Japanese style ( Figure  S2A). The poster showed a space with a Japanese scroll with calligraphy of the Bonsai kanji symbol, Tatami flooring, and Japanese sliding doors. The idea behind this was to set the expectations of the visitors regarding the content of the gallery. The gallery area was covered with black cloth on all sides to ensure that the visitors did not form a spatial image of the area of the gallery space ( Figure S3A). We created a small corridor between the wall and the covered space that led to the entry point so that visitors could not peek inside the content of the covered area through the entrance. Here, we indirectly introduced them to a simple common element (museum barriers) that they would encounter and interact with within the VR space. We used the barriers to mark the entry point and to stop anyone from entering the corridor. At the starting point outside the corridor, we asked the visitors to follow three instructions. First, to take their shoes off and wear slippers because it was a traditionally styled gallery and Tatami flooring had been used. This helped us provide constantly matching haptic feedback through the feet. Second, they had to close their eyes (we guided them through the corridor) and only open them after wearing the VR headset. Third, they could touch the first artwork but were not allowed to touch the other three artworks. The "do not touch" sign was clearly visible inside the VR gallery, similar to what is commonly seen in art galleries.
As soon as a visitor opened their eyes, they saw the first artwork. This first artwork, titled "Neo Life," was 3D printed and fixed on a display table ( Figure 5C, Figure S4). Out of the four, it was the only artwork that existed in a physical form. In addition, we fixed a Vive tracking sensor on the backside of a display table to make sure that the table and the artwork are always in the correct position. The first thing the visitors did was step onto the Tatami flooring. This was the first matching and constant haptic feedback between the real world and the VR gallery throughout the whole experience. Then, we encouraged them to touch the first artwork, which provided a more detailed matching haptic feedback. A leap motion sensor was mounted on the Vive headset. It tracked and displayed the hands of the visitors while interacting with the physical objects. When they finished experiencing the first artwork, they entered the main gallery space on their right, passing between two pairs of museum barriers. We made the passage narrow to increase the chance of physical interaction with the barriers or its straps-these barriers matched with their real-world counters in shape and scale. However, there were no matching objects other than the Tatami floor; of course, the visitors did not know that the gallery was empty. The visitors freely looked at the artworks, and they were free to leave whenever they wanted to. We answered any questions and provided some verbal explanation regarding the concept of the artworks. When they finished, they went back to the starting point, closed their eyes, took off the VR headset, and were guided outside. In the end, we asked them to fill out a questionnaire to collect data about their VR gallery experience.

Statistical Analysis
Normally distributed binary data were presented as the mean and standard deviation (SD), and non-normally distributed data were shown as the median (M) and interquartile (IQR) values. The t-test was used to compare the normally distributed data, and the Mann-Whitney U test was used for comparing the data with nonnormal distribution. The significance was set at p<0.05.

STHR
There were 21 participants (6 women). The mean age of participants was 29.0±11.7 years, and there was no difference between men and women with respect to age (p=0.934). Overall, 9 (40.9%) participants had a previous VR experience. None of the participates had seen the real space before experiment 1. Nine (42.9%) participants had no video game experience, whereas 6 (28.6%) had significant video game exposure.

Effect of haptic feedback in hybrid reality on VR experience
In experiment one, there were 11 participants in the hybrid reality group (HRG) and 10 in the control group (CG) (Figure 2). There was no difference between the participants of these two groups with respect to gender, previous VR experience, or age (all p>0.05). The median time taken to complete the task was considerably longer in HRG than in CG [HRG:7.0 (6.4-7.6)vs. CG 4.0 (2.5-5.0); p=0.007]. The average number of collisions was not significantly different between the CG and HRG (p=0.888). The median value of anticipation, immersion, and reality were all slightly higher in the HRG; however, the difference was not statistically significant (all p>0.05). The median (IQR) attention score was 23.0 (22.0-27.0) in the CG and 28.0 (27.0-34.0) in the HRG (p=0.045).In the interviews, 10 out of 11 of the HRG participants, who had an initial physical interaction with the objects in the real space, reported that they thought that all the furniture, columns, and cones shown in the VR did exist real. On the other hand, participants in the CG were not sure of the realism of the objects and used phrases such as "maybe there was," "might be there," and "avoiding it just in case" while talking about the objects in VR. The participants in the CG also moved closer to the obstacles. Participants thought that not avoiding collisions with the virtual objects is part of the experiment.

Effect of complete spatial awareness on VR experience
In the second experiment, we attempted to determine the effect of an unfamiliar environment on the actions of participants ( Figure 3). There were 11 participants in the mask group (MG) and 10 in the CG. There was no difference between the participants of these two groups with respect to gender, previous virtual experience, or age (all p>0.05). The median time taken to complete the task was considerably longer in MG than in CG [MG:4.8 (4.2-5.8)Vs. CG: 43.4 (2.6-4.2); p=0.038]. The average number of collisions was not significantly different between the CG and MG (p=0.504). The anticipation, immersion reality, and attention were not different between the groups (all p>0.05). Upon observing the video capture, the completion time, and the discussion during interviews, we noticed that the people who saw the real space before wearing the headset had more freedom in their movement and were faster in completing the task. The participants in the MG were more cautious while moving around the obstacles, and 8 out of 11 used phrases such as "maybe there was," "might be there," and "avoiding it just in case" when talking about the existence of virtual objects. Two participants from this group attempted to touch and verify the presence of traffic cones.

Effect of hybrid reality on VR experience
Interestingly, 52.4% of the participants remembered the RWO (table) and avoided it without even seeing it. The participants who answered "no" had a physical interaction with the RWO. For the second question, which inquired whether the participants remembered the table at the end of the experiment, 66.7% of the participants voted "yes." However, a few people who voted "no" mentioned that they did remember the table but were unsure of its exact location. Notably, 8 of 10 participants who had a physical interaction with the table did not forget it. As for the VO (chair) present in the middle of the room, most participants tried to check if it was real. This experiment revealed that almost 50% of the participants relied on prior knowledge of the real space even when different visual information is delivered through VR. Moreover, the participants who did not remember the RWO became aware of it after the physical interaction, suggesting that physical interactions resulted in more vivid mental images. When we introduced an obstacle (a chair) at the end of experiment 3, for most, it was not as important as the table present in the real world, which was invisible in VR.

Virtual art gallery
There were 38 visitors to the VAG. The average time spent in the gallery was 5 min 57 s. As a percentage, 18.4% of the participants in this gallery participated in our previous three experiments. The age of visitors ranged17-70 years, with an average age of 31.26 years; out of the total, 58% were females. A total of 51.6% of the participants reported not having any previous experience with VR, while the rest had some prior experience with VR; however, none of them was a frequent user. Additionally, 46.4% mentioned that they did not play video games; however, most of them were interested in art. We asked the participants about the artwork they liked the most. The third artwork, titled "Distress," got the highest percentage with 47.4% of the votes. However, it received only 23% (77 s) of the view time, almost similar to the second artwork titled "Natural: Artificial" (76 s). The fourth artwork, titled "Incomplete Sculpt," recorded the longest observation time of 95 s (29%); however, it was liked by only 13% of the participants. The second and third artworks had similar sizes and viewing times. The first artwork received 78.5 s (24.5%) of the view time despite being the smallest in size. We believe this was due to the opportunity provided to visitors to touch and interact with it. We did not want to ask if the visitors thought there were physical artworks present in the VR gallery directly because we thought such a question might induce bias. Therefore, we asked them an indirect question: "Did you feel you wanted to touch the artworks?" 73.7% scored it with a five, which was the highest rating on our scale (Table S1).

Discussion
Realism in virtual reality has been a subject of great research interest. In this work, we examined whether spatial awareness of the real-world affects the overall VR experience. We have introduced a concept of STHR, wherein a mix of real-world objects and virtual objects are presented during VR experience, and a smooth transition from the RW space to VR space was made. Our results suggest that this approach significantly increases the time taken to complete the VR tasks and affects the participants' overall attention. Furthermore, in STHR based virtual art gallery made for the public, the visitors' overall feedback was highly positive.
To examine the impact of spatial awareness and haptic feedback, we first compared the VR experience of participants with and without STHR. For the STHR, the room was set up such that the participants interact with an RWO and experience haptic feedback. This interaction affected the perceived realism of VR, as was observed from increased task-completion time and change in the attention. It was also reported that by introducing minimal haptic feedback by felicitating physical matching interactions, the users' perception inside the VR environment could be altered [15]. Our results validate this hypothesis. Notably, the interviews conducted in the current work revealed that more than 90% of participants having initial physical interaction with the RWO thought that all the VO are RWO. Though in previous studies, different approaches have been tried to enhance VR realism, an analysis of different dimensions of VR realism is missing, and there is a lack of a simple and unified approach that can be used to enhance VR realism in different settings. In a notable development, a concept of substitutional reality (SR) by showing mixed footage of live scenes and recorded scenes was proposed [24]. In another work on SR [25], furniture and other objects that already existed in the physical environment were replaced with similar-sized 3D models, providing the user with a similar spatial image of both real and VR spaces, thereby resulting in a better transition to the VR content. Haptic retargeting technique to enhance VR believability was also explored to enhance the VR experience [26].
Our study provided key insights on the effect of an unfamiliar environment on VR realism. By observing the video capture, the completion time, and the discussion during interviews, we noticed that the people who saw the real space before wearing the headset had more freedom in their movement and were faster in completing the task. We believe that adding a barrier/mask increased VR realism [27][28][29][30][31]. Regarding anticipation and hybrid reality, we surveyed the participants after introducing them to a VR environment where there is a mismatch between RWO and VRO, and an additional VR object was introduced during the experiment. The experiment revealed that almost half of the participants relied on the real space's prior knowledge even when different visual information is delivered through VR. Moreover, the participants who did not remember the real space during the experiment became aware of it after the physical interaction, suggesting that physical interactions resulted in a more intense experience [14,32,33].
The experiences of the visitors to VAG, which was based on the key concepts of STHR, i.e., providing haptic feedback in the initial phase of the VR experience and maintaining a smooth transition from RW space to VR space, corroborated the abovementioned results. The average time spent in the gallery was good, suggesting a high level of engagement [34]. The feedback of visitors was generally positive, supporting the use of VAG. More than 80% of visitors rated the experience 4 or 5 on a scale of 5. Notably, there was no negative feedback. Interestingly, nine VAG visitors were curious about how the gallery looks like in real space. This part was not planned and happened naturally. Therefore, we asked them to fill the questionnaire first and then showed them how it looks. Surprisingly, all of them were shocked and did not expect that it was an empty space. Their response ranged from "I was tricked!", "It is an empty gallery," "No way!", "I fell into your plan!" to "There is nothing!" One person laughed and commented, "you are a good liar!" As mentioned before, we had used a 3D printed artwork to which visitors could interact. This 3D-printed artwork was made of plastic; however, the corresponding artwork in VR had some metal parts. The participant asked us how we made the art piece and, upon knowing that it was a 3D printed object, the visitor claimed that "…did not know you can 3D print with metal!" Notably, 12 visitors commented that it was an exciting experience to touch a sculptured artwork, which is usually not allowed in conventional galleries. These reactions suggest that the VAG successfully met its objectives. This supports our hypothesis that by enabling a smooth transition from real-space to virtual space and by providing relevant haptic feedback, realism in VR can be influenced [31,35,36].
STHR can be easily implemented in different settings; however, it is essential to understand our work's limitations before generalizing our findings. Though we have conducted controlled experiments and demonstrated the validity of STHR in VAG, more large-scale studies are needed to strengthen our conclusions' generalizability. Our approach's main limitation is that the user needs to be unfamiliar with the real space before VR. This concept, therefore, is not feasible in an already familiar space such as VR experience at home. However, it may be an effective approach for VR in exhibitions or museums.

Conclusion
By blocking out the real space and through minimal haptic interactions, we were able to improve users' experience of VR. Prior information of real space has a profound effect on the decisions made by users during VR experience. This study suggests that by using STHR, VR experience can be improved. Our results also indicate that conflicting information between the real world and VR hinders VR experience. Creating the right setup for each VR experience is challenging; however, by implementing the method proposed in this work, a semi-unified covered space for various conditions can be created. Furthermore, the matching objects in many different experiences and locations can also be reused, such as fence barriers and glass boxes present in galleries, to enable physical interactions.

Future Outlook
The technological framework presented in this work offers many possibilities for enhancing the user's immersion in VR by carefully smoothening the transition to VR environment and using a perceptual stimulus. Future research should focus on the generalizability of this approach. As demonstrated in our work by the example of a real-world VR art gallery, STHR is particularly useful in museums and art galleries. We believe that the use of STHR in VR art galleries will have a significant impact on the way patrons connect with art, as the interaction with artworks involves advanced cognitive function and substantial processing of visual information. In future studies, we believe that by deliberately exposing the VR user to specific visual or auditory information while blocking the extraneous information, some targeted emotions can be amplified. Using this approach, the believability of VR can be further manipulated. As the use of VR is going to increase in the near future, STHR could play a pivotal role in enhancing the quality of the VR experience.