Contextualization of the Augmented Reality Quality Model through Social Media Analytics

Article history: Received: 15 June, 2020 Accepted: 12 July, 2020 Online: 22 July, 2020 Augmented Reality applications are gaining popularity while maintaining novelty. Many industries are utilizing the user interface type, and use cases are becoming repeat patterns of problem solutions. Despite this rising popularity, quality has not matured nor has the technology become mainstream. Novelty must be approached as risk, and risk must be evaluated for and tested to assure adequate levels of quality. Quality itself can also be vague and have contextual definition. For these reasons, a quality model for augmented reality was created. This work analyzes over two hundred thousand tweets, collected during 2019 and 2020, relating to augmented reality technology, and contextualizes various data points to the established AR Quality Model. The education industry had the highest mentions among the tweets within the scope of this research while the tweets labeled to the transportation industry had the highest sentiment. Furthermore, the tweets were shown to illustrate the needs of testing against the characteristics within the quality model; presence, perspective, interaction, portability and persistence.


Introduction
This paper is an extension of research work producing an Augmented Reality (AR) quality model and originally presented in IEEE Ubiquitous Computing, Electronics & Mobile Communication Conference [1]. Augmented reality was first coined as a term by Caudell and Mizell in 1990 [2]. Augmented reality is accomplished through an application's user interface by merging both physical surroundings as well virtual objects, which may be models or scenes [3].
Research of AR applications covers many different use cases and industries. One study has uncovered findings such as children seem to understand how to use the augmented reality application as soon as they began to interact with the technology. Another research effort found learning English utilizing AR can help students in learning material and motivated students to learn the language. Another study proposes AR mobile applications in the household to educate parents and caregivers of potentially dangerous objects in the home that may endanger infants. Easily and quickly comprehended how innovative the space is becoming, and how useful for humans. In fact, Gartner, Inc. has predicted that "by 2021, at least one-third of enterprises will have deployed a multiexperience development platform (MXDP) to support mobile, web, conversational and augmented reality development" [4].

The Augmented Reality Application Quality Model
The importance of this technology extends into many industries. AR has been utilized in applications for factory design [5]. Military equipment maintenance is another use case where AR has been applied [6]. Yet another example is in the use of AR for teaching new electronics assembly skills [7].
The effectiveness of the applications requires them to have levels of quality. One model that identifies eight characteristics of software quality is the ISO 25010 model: Functional, Performance, Compatibility, Usability, Reliability, Security, Maintainability, Portability [8]. However, AR applications have not matured, and expectations of AR application users are not currently known [9]. Because the AR interface is a novel one, the intent of traditional usability evaluation methods which have matured over time for traditional software quality are inadequate [10]. For these reasons, a quality model was established that encapsulates the characteristics for AR applications broadly as Presence, Perspective, Interaction, Immersion, and Persistence [1].

ASTESJ ISSN: 2415-6698
Beyond the characteristics of quality is the necessity to utilize context in testing. Emerging technologies and designs such as digital twins must still begin with system needs analysis through known methods such as context diagrams and objective trees [11]. And the same is true for established technologies. According to Gartner, "your strategy for API quality needs to be built with the business and environment context in mind" [12].
These contextual concerns drive much of this extended research. Utilizing over 200,000 tweets towards AR and VR, this research contextualizes quality regarding AR and various industries using text analysis with mention, time series and sentiment analysis.

Collection
Beyond some popular and academic notes, this research methodology focuses on social media analytics. Data acquisition is completed through a small program written in R and executed daily beginning in August of 2019. The program continues to execute, as of the publication of this research, populating a MySQL database hosted in AWS RDS. The program uses the twitteR library as an API to Twitter social media platform. Figure  1 omits various keys and passwords yet implements the basics of importing the twitteR library and establishing connections to both Twitter and the MySQL database prior to searching on '#AR' and storing the search results.
The tweets are stored with other certain meta data, made available through the twitteR library and API. Fields beyond the text collected and utilized in this research include discrete data fields towards the number of times a tweet has been marked as a favorite by a Twitter user, as well as the number of times a tweet has been re-tweeted. A date field within the table indicates the date the tweet was created in the Twitter platform.

Data Preparation and Processing
Social media analytics is not without data quality and research concerns. Tweets lack standards, such as in hashtag usage. A tweet that embeds #AR may be about Arkansas, a state within the United States, and not about augmented reality technology. Prior to formulating findings and information from the collected data, much data preparation and cleaning are required. Data preparation activities may account for 80 percent of time invested in data science effort [13]. The amount of data dropped during preparation from the data set can be large. This is to be expected, as with the example of #AR as Arkansas, and not augmented reality. The search algorithm sought out technology vendors, whose tweets may have been towards other products, process, marketing or communication needs. Tweets are also fixed in length, 280 characters [14]. A short message may come from a targeted technology provider, but also fail to cite key search criteria words. Within this research set, counting only the tweets that mention specifically the words "augmented" or "virtual" (not case sensitive), the original 637,384 tweets were reduced by twothirds to 211,269 tweets. The breakdown of augmented and virtual tweet counts is illustrated in Figure 2.
The intention of data preparation and cleaning is to move the data from chaotic towards precision [13]. In this research, we create useful subsets without disrupting the raw database contents, as new hypothesis and meaning may yet to be executed and found across the raw data.
To create useful datasets, we must also create new fields, such as categorical values. Tweets have some indication to business and industry, so categorizing the tweets by industry lends toward understanding a contextual need for quality, such as HIPAA requirements and testing concerns towards health or pharmaceutical implementations of AR. A small program in R reads the tweet texts from the accumulated corpus and maps to common industries, counting the industry mentions, including if multiple industries are mentioned in a single tweet.

Results
The first finding is that among all tweets that mention either "augmented" or "virtual", augmented reality mentions (135,319) are more frequent than virtual reality mentions (80,540), illustrated below. Figure 2 Percentage of "Augmented" versus "Virtual" mention in tweets This first finding seems to contradict a 2018 survey by Gartner, where survey recipients responded towards virtual-reality apps (20%) more than augmented-reality apps (14%), in terms of apps that will have most impact on business success by year-end 2020 (shown in Figure 3). Although, the Gartner report analysts stated, "It is, however, surprising to see VR apps identified as the second most impactful type of multiexperience app (20%), as AR has more potential use cases and device support. But only 14% of the respondents thought that AR apps would be the most impactful, despite AR app development tools being more widely available. " [15]. The following chart shows where virtual-reality apps were more frequent in survey recipient response compared to augmented-reality apps for business success.

Most Favored and Most Retweeted Tweets about Augmented Reality
Tweets can be liked, or favorited, as well as re-tweeted. These mechanisms, favorite and retweet, are utilized to propagate, share, show appreciation of other user posting, and the number of favorites indicate the count of unique user accounts that like or agree with the content of the tweet [16]. Tweets frequently have meaning, and in this research, the gathered tweets are towards specific technologies.
Within scope of this research, the most favorited tweet is towards AR hardware, wearable glasses, and contains embedded video with a link to a Kickstarter site to raise investment through crowdfunding (tweet image found in Figure 4). The glasses, by TiltFive, are marketed to be the future of tabletop games. A new take on what appears similar to traditional board games. The digital approach gives the characters dimension, allows sharing the digital and augmented experience with other players, as well as the ability to save the game. These AR application quality characteristics are mentioned and defined within the quality model as presence, perspective and persistence. Further evaluation in section four applies this and subsequent text analysis to the previously established quality model. The following tweet, posted in 2017, had at the time of this research been viewed over 212,100 times and was re-tweeted 4,247 times. The tweet embeds demonstration video to augment the physical environment and makes this statement, "This is ARKit's A-ha moment". ARKit is a development technology for building AR applications. Apple released ARKit with iOS 11 in 2017 [17]. More development tool analysis is found in section 4.5. Within the scope of this research, Figure 5 illustrates the most retweeted tweet.

Mention, Sentiment and Time Series Analysis
Mention analysis identifies the frequency text occurrence. Research shows augmented reality applications being utilized across many industries; mention analysis from social media provides an additional perspective. Figure 6 indicates that Education, Entertainment and Commerce are the top three mentioned industries. This is determined by first creating a list of known industries from a source [18]. Then the algorithm reads the tweet texts and will count industry mention, including if multiple industries are mentioned in a single tweet.   Figure 8 is a tweet containing embedded AR demo video of a skeletal dinosaur walking on a physical plane as an educational and awareness instrument. This tweet was retweeted over 1,000 times and was the largest contributing factor to the November peak of tweets having mention of education industry. Tweets are short in character length but offer volume of opinion which can be mined as user-driven data and classified in terms of sentiment. Opinion mining and sentiment analysis techniques can be utilized to support or drive communication plans [19]. Sentiment analysis has also been combined with other analytic methods to produce stock market prediction models with accuracy greater than 60% [20]. Tweets are opinionated text. Opinionated text is the primary resource for shoppers when making purchases [21]. We can utilize user-driven data for measuring system quality. Indeed, the quality of the system is also important in procurement, investment and communication planning.
The highest average sentiment score within the scope of this research is Transportation. Figure 9 illustrates average sentiment scores by the aggregated tweets within industry categories. A tweet which references multiple industries would have a sentiment score towards each referenced industry.  The tweet found in Figure 11 was created near the end February of 2020 and had hundreds of retweets. This tweet is the main contributor towards the March spike of tweets categorized into the Transportation industry. The tweet references cost optimization of warehouse processing time by helping warehouse employees more quickly identify product location. Beyond cost optimization, the tweet states that AR technology can also help reduce possibility of human error within logistics. The Sentimentr library is utilized within the R code of this research to provide sentiment scores of the collected tweets. The sentimentr library utilizes valence shifters to apply more meaning and accuracy to sentiment, such as negator words (e.g. I do not like it) [22]. The highest sentiment scored tweet, score of 1.540922, is found in Figure 12 (sentimentr library scores range between -2, 2). The tweet references an easier and more fun life due to augmented reality technology and utilizes the hashtag #CX (customer experience). While application users may be customers, they may also be internal employees. This research refers towards user experience as a more generalized approach.

Application of Analysis to the Established AR Quality Model
Augmented reality applications are novel and require levels of quality to be established to be effective for the users experience. Establishing the quality of an AR application goes beyond traditional testing practices and standards. A new model has been established for test case focus and categorization, to build system confidence and inform system readiness and release decisions. These are critical decisions when dealing with novel technology.
Gartner's 2020 Digital Friction Survey found that among a survey n size of 4,582 employees surveyed, only 31% experienced High-Quality UX that was productive, empowering and easy [23]. Why might system's need to have this level of quality? The same survey found that "employees with a high-quality UX are 1.8 times more likely to have a high intent to stay as indicated by a lack of interest in leaving their current employer or actively search for a new job." The Gartner research report further explained, "Employees with a high-quality UX are 1.5 times more likely to have high discretionary effort, which includes employees' willingness to help colleagues, take on additional responsibilities, put in extra effort and find better ways of working." Augmented reality applications are a novel approach to a system's user experience. The user experience has opportunity to benefit users and organizations alike. The technology implementation requires quality for these benefits to be realized.

Contextual Approaches to Quality
The characteristics of the quality model are quickly identified in the following bulleted text. Further sections then develop the characteristics and apply the findings of the twitter text analysis to the previously established quality model.

Twitter Analysis Applied to the Characteristic of Presence
Accurately executed occlusion and collision behavior will make the AR application more realistic [1]. Together they form the characteristic of presence. Presence allows AR application users to experience both the physical and digital worlds accurately. Presence may be exemplified digital models obeying gravity or shadowing as physical objects would which meets the users' expectations. Presence must be tested for. Figure 8, a digital model of a dinosaur overlaid onto a physical landscape (the original tweet is a video of the skeletal dinosaur walking) maintains the existence of shadow beneath the digital model, an example of presence in AR.
According to Gartner, "realism, as well as usability, will be enhanced or undermined by the digital model's ability to have accurate presence within a physical landscape. There should therefore be a test for occlusion which occurs when a far object hides a near object, for example, and which also sometimes occurs when objects are floating above and closer than an original anchor point. Test for occlusion by placing multiple models or scenes and moving throughout the physical environment." [24]. Collision in AR applications exists when a digital object and a physical object appear to occupy the same physical place at the same time. Of the 211,269 tweets related to augmented and virtual reality in this research, only 70 tweets reference occlusion or collision. Average sentiment for an AR tweet is .1176; however, for tweets mentioning collision the sentiment turns negative at -0.0157, and ventures more negative when mentioning occlusion -0.0475.

Twitter Analysis Applied to the Characteristic of Perspective
Perspective frequently refers to someone's point of view. In an augmented reality, perspective must encompass the physical and digital objects, combined behavior, and how well visual distinction exists among them.
Perspective may be affected by the physical environment, the digital model's attributes, and the interaction between them. This would include the inability to scale a digital model to the correct size when embedded into the physical view. Perspective fails when rendering white font in annotations or digital models given a physical environment containing bright light or predominately white backgrounds. Perspective within AR applications should provide visual distinction and improve the quality of the user experience. Figure 12, the tweet within the scope of this research having the highest sentiment score, adequately illustrates the characteristic of perspective. This example uses a neon green annotation text color over a mostly grayscale physical scene. Quickly drawing attention to a physical device through digital annotation.

Twitter Analysis Applied to the Characteristic of Interaction
The characteristic of interaction is directly related to performance. Interaction directly affects the device, the costs, and the execution environment. The tweet text found in Figure 11 refers to AR applications used to model warehouses which may assist in finding specific freight. Warehouses can be quite large, requiring longer application runtime, as well as the rendering of many possible digital directional annotations before arriving at the correct location. If the AR application is utilizing commercial grade devices, we must concern how the interaction of such an application will consume WIFI network, the battery of the device, and whether other applications will run on the device at the same time. The greater the interaction, the more important the performance.

Twitter Analysis Applied to the Characteristic of Portability
There are many software development kits available to build AR applications. Table 2 references many examples of these technologies, and the runtime platforms they support. Not every AR application will run on your commercial or consumer system. It is not uncommon for some popular mobile AR apps to only run on iOS or Android, as example operating systems [25]. The value and availability of your application will depend upon the portability characteristic, which is a quality concern that must be considered and established during design and development. Table 1 provides search terms that were utilized to scope the collection of tweets for this research. Many of those search terms are twitter handles or the names of SDK technologies. The top three mentioned development vendors are Apple (966), Unity (713), and Google (601). The popularity found in the mention analysis is not conducive with portability, as Apple is one of the more limited SDKs when concerned with runtime environment support as indicated within Table 2.  [26] Android, iOS, UWP, Unity Spark AR [27] Android, iOS ARKit [28] iOS ARCore [29] Android, iOS, Unity, Unreal Wikitude [30] Android, iOS, Microsoft Tablet Maxst AR [31] Android, iOS, Unity Figure 13 Example augmented reality development technology ordered by mention count

Twitter Analysis Applied to the Characteristic of Persistence
Scaling augmented reality experiences across multiple users requires centralizing physical maps and virtual object anchors so that multiple users can share immersion [1]. Persistence allows an AR user experience to be shared among other application users from separate devices or application instances during runtime. Additionally, persistence allows the AR experience to be saved and resumed later by the original, or potentially other, application users. Figure 4 is the most favored tweet within the scope of this research, and it adequately exemplifies many of the characteristics of this quality model. The tweet references the future of tabletop games, a physical board game with digital models rendered above. Persistence of the digital models allows multiplayer action in the game, so that competing players may see and experience the avatars and digital scenes that are being controlled by other's play. It is possible that such a use case could leverage whichever phone device or headset might be handy to a player, which would encompass the portability characteristic. If the game is to be played for many hours, the interaction level will be high, perhaps enough to warrant recharging a device battery, or the requirement to plug an AR hardware device into a power source. Perspective is required so that the digital scenes are relevant to the location, size and position of the physical game board as seen by other's points of view. Presence of the digital models, the prevention of collision, will also provide an experience worth dedicating many hours towards.
While the previously mentioned and exemplified tweet represents the entertainment industry, certainly implementation across healthcare, manufacturing, real estate, education and many other industries would prioritize quality potential during investment in novel technology. Hence the AR quality model, which provides a basis for test case creation and categorization that promotes confidence in the application's readiness and informs decisions towards the application's release.

Conclusions
The findings reported in this research study represent essential considerations for the investment and realization of value through augmented reality applications. Augmented reality applications have characteristics that are beyond traditional interfaces. Despite a novel technology, quality still matters. For these reasons, the AR quality model has been established and extended with input from social media and in some cases exemplified through such uses.
Among all tweets that mention either "augmented" or "virtual", augmented reality mentions (135,319) are much more frequent than mentions of virtual reality (80,540). Top mentioned industries within the tweet collection scope were Education, Entertainment and Commerce. Top average tweet sentiment scores by mentioned industries within the study scope were Transportation, Hotel and Food.
AR quality model characteristics of Presence, Perspective, Interaction, Persistence and Portability all had representation among some of the highest retweeted or liked tweets, or among the tweets that drove industry spikes in the time series charts.
Still, limitations exist. Only one social media platform was utilized in this research, Twitter. Facebook, another such social media platform, is also a technology provider of AR software development. What information then may be more prevalent on their social platform? The AR quality model requires further research beyond social media and into specific teams and implementations within industry. Research towards failed AR implementations would also support the model or indicate where omissions of key characteristics exist.

Future Work
Future work on this quality model for augmented reality applications must extend into cyber security. 475 of the tweets within this research mentioned terms towards privacy or security. That is less than one quarter of one percent of tweets within the scope of this research. While the percentage is small, it is critical to consider augmented reality applications require access to the device camera. The device camera may capture personal or organizational information, potentially confidential or private in nature, that may become vulnerable to threat actors if exploited. Future work should engage in various threat and vulnerability research of AR applications and methods and technology which may help mitigate such risk or exposure.
Future works should also include case studies from many of the named industries within the paper with indication towards whether the AR quality model adequately supports the users experience of if in fact key characteristics have been omitted, such as current lack of the security characteristic.