Assessing Heutagogical Elements in Learning of Engineering Education: Instrument Validation

Article history: Received: 12 May, 2020 Accepted: 11 August, 2020 Online: 14 September, 2020 Practically level of design element (i.e., explore, sharing, connect) is an essential of heutagogical approach. The self-determined learning process can be at ease with the implementation of these elements, and the critical step is reliability to measure teaching and learning feedback. Although various instruments were proposed in the literature to assess heutagogy elements, the specific potential Rasch Measurement Model to determine the practicality levels of heutagogy element is less emphasized. This paper aimed to validate the research instrument (six constructs with 65 items). The instrument was administered to N=175 students for a pilot study. The Rasch model was conducted to examine reliability (0.93, 0.94) with α = 0.97, separation index (3.75, 4.01) for item and person, respectively. Besides, item fit (three-item dropped), polarity and standardized correlation residual (no overlapping items). The findings have shown that the instrument has high validity and reliability for use in measuring the practical level of heutagogy elements.


Introduction
Pedagogical or andragogical approach is generally related to the role of teachers and learners in the traditional education system. As the future of education has offered many new opportunities in the learning process, the continuous transfer of learning from pedagogy to andragogy and finally arriving in heutagogy shows that change must take place in the classroom compulsory [1]. With the rapid development of technology, the education system is increasingly shifting to a self-directed approach by heutagogical method [1,2,3]. From this trend, students can obtain valuable information from different sources and accessible from any platform (i.e., online or mobile system). Therefore, the heutagogy approach focuses on life-long learning and self-determined learning methods. There are multiple contexts of studies that have been discussed in the literature, such as related to necessity [4], practice [5,6,7] and teacher's training [8] of heutagogy learning approach. Most studies have found that the heutagogical approach can be an essential learning style for producing students who are well-prepared for the complexities of today's workplace.
In the context of measurement, evaluation of the heutagogical learning approach's implementation strategies in the teaching and learning process is a vital issue to be taken into account. Implementation and learning strategies need to be aligned with the design principle of the heutagogical learning environment (i.e., exploration, creation, reflection, connection, assessment and sharing) [9]. To date, a number of validation instruments, tests, experiments and studies have been developed to evaluate the effectiveness of the heutagogical learning from a different point of view [4,6,9]. The validation process consists of a systematic procedure, including a pilot study. The pilot study's main goal is to improve the quality of the item and increase confidence in the interpretation of the data [10]. While every student has a different style of preferred learning ways, a pilot study is required to determine the capability of the developed instrument to analyze the practicality of heutagogy data. During data analysis, the need to identify the element of heutagogical approach that best fits the different categories of student's profiles, especially in engineering education, arises as to the main issue Based on the above premises, an instrument was developed in this study, involving the student's reflection to determine the practical level of heutagogical elements applied in learning, based ASTESJ ISSN: 2415-6698 on the design elements proposed [9]. This research instrument was developed in the questionnaire form. It is a practical way to gather data, collect results easily, and gather information from a large number of respondents [11]. When involving a self-assessment test, the psychometric tests element must be considered during the instrument development to ensure the high reliability and validity of the instruments.
In this phase, a set of questionnaire with a Likert scale was used to obtain feedback from students. However, using only a descriptive analysis of the Likert scale data does not provide sufficient information and concrete findings [12], since the Likert scale is commonly ordinal data that categorizes the respondent's point in general (i.e., less accurate interoperation). The Rasch Measurement Model analysis was therefore conducted to improve response accuracy. The Rasch model analysis can convert the latent traits, either nominal or ordinal data into interval and ratio data by creating a logit scale. This can be useful and provide scientific measurement for this study. The objectives of the study are as follows: • to identify the validity and reliability of the heutagogy elements in learning of engineering education using Rasch Measurement Model • to identify the point measure correlation in the construct of the heutagogy elements in the learning of the engineering education which are in the acceptable range • to identify the fit items in the construction of the items heutagogy elements within the acceptable range.

Heutagogy Learning Approach
Heutagogy, or the study of self-determined learning, has gained interest in the field of engineering education aimed at 21stcentury skill requirements and industrial needs [9]. The increasing number of education institutions, for example, technical universities, have started to explore and implement a selfdetermined learning environment for their students. The main goal of heutagogy is to teach lifelong learning. Heutagogical approach differs from other traditional methods (i.e., pedagogy and andragogy). In comparison, these three learning approaches can be differentiated based on the teaching and learning environment and its implementation. Generally, figure 1 shows the progression of three teaching approaches. At the first level, pedagogy is mainly focusing on teacher-led learning, where teachers have to decide on the overall learning content and process (i.e., what students will learn and how they will learn). In the first level, students depend on instruction from the teachers. In the second level, andragogy is taking place where teachers play an important role as a mentor in andragogical environments and cultivate the students' ability to self-directed learning. At the third level, entering at heutagogy learning level 3 encourages students to fully control the learning environment, where students solve their learning problems with their own decisions. Therefore, it is important for students as well to equip themselves with problem-solving skills [13]. In this case, the teacher is only preparing the learning context in general [1].

Heutagogical Study and Instrument
Heutagogy is an extension of the pedagogy and andragogy approach which places the learner, rather than the teacher or institution or even the curriculum, at the center of the learning process [9]. The goal of heutagogy is to prepare responsible, selfcapable, proactive, competent students who are ready to face the actual workplace environment. Most heutagogical studies in engineering education, vocational education, and pre-service teacher professions have found that heutagogy is a credible response to the critical issues that learners face within the workplace and have designed their learning environments based on the approach. For example, within the vocational education, [14] the innovation of training module based on heutagogy has been developed as acceleration for increasing the pedagogical supremacy of professional education lecturers in the industrial revolution. The findings reported that heutagogy provides a learning framework that addresses the needs of vocational students, who must learn in an ever-changing environment that is both complex and helps them to become lifelong learners. Table 1 summarizes the previous relevant heutagogical studies that can be categorized into two research directions; namely the category/principle and the type of environment. The category or principle perspective, in which heutagogy is defined as the center of the teaching and learning process and becomes an active agent of all learning experiences, from planning and execution to learning assessment [2,9,15]. The main goals of these studies are to understand the perceptions, needs and experiences of learners or their characteristics or to develop a learner-centered curriculum. There are previous study that have illustrated the finding of their small-scale study on heutagogy approach, which can be one of the key elements for enhancing the curriculum, encouraging students to determine what, how, when and where they learn and enabling educators to be more flexible in delivering their curriculum [16].
Next, the research also indicated several different types of the learning environment (i.e., traditional face-to-face), online, or a combination could support self-determined learning [9]. With the availability of an online platform, implementing self-determined learning activities can be more convenient as an online platform can connect, share, create, and collect more resources, that has been highlighted in several previous studies [17]. This has already been indicated in some of the research studies [18][19][20]. Critical thinking can be developed among students [21] to solve learning problems [22] and researcher found that reflective independent and critical thinking skills can be developed [3]. Perceptions [21] Semi-structured group interviews Heutagogical needs and experiences of the learner [4] Interviews, literature review, documentation, and observation Curriculum development [15,22] Group discussion Semantics search and discussion Questionnaire survey Strategy and implementation [23] Standardized and open-ended interviews

Blended Learning
Practices and technology adaptation [8,24] Workshop Quasi-experimental design, semistructured interview in heutagogy learning program by using a blended approach. It is also noted that the affordance of mobile social media tools can facilitate student-determined learning experience (heutagogy) in authentic contexts [3].
On the other hand, interview questions were mostly used in the standard instruments, and the results encourage two-way communication. Besides, it allows participants (i.e., lecturer or students) to freely express their needs, perceptions and experiences in the implementation and practice of self-determined learning [21], the interview questions are useful in profoundly exploring the participants' attitude, experiences. For example, [19] selected 138 high school teachers to determine online learning competence through a heutagogy approach. The result indicated that all the heutagogy elements (i.e., exploration, creation, collaboration, connection, sharing and reflection of experience) were identified as important. However, one of the neglected remaining issues is on how the heutagogy approach instrument can be best validated? Therefore, this study was conducted to validate the proposed instrument used for heutagogy learning approach among engineering students. Figure 2 shows an overview of the validation process of heutagogical element in learning of engineering education. There are five main steps conducted, starting with the development of the questionnaire, data collection, data processing, data analysis, and reporting findings. First of all, the research instrument in the form of a questionnaire survey was developed. The questionnaire consists of two sections; Section A includes the students' Demographic profiles, and in Section B, a total of 65 items to gauge students' learning style.

Methodology
In section B, six main constructs/elements of heutagogical learning style have been evaluated namely, (a) explore, (b) create, (c) collaboration, (d) connect, (e) share and (f) reflection.
Furthermore, the questionnaire was distributed through the online platform and the student portal. Students have to rate items based on a scale ranging from 1 (never) to 5 (always). Students' response represents the percentage of the practice such as 1 (0 percent practice), 2 (1-20 percent practice), 3 (21 percent-50 percent practice), 4 (51 percent-80 percent practice) and 5 (>80 percent practice), respectively. Respondents were given approximately fifteen minutes to complete the questionnaire. A total of 175 students from University Tun Hussein Onn Malaysia, (UTHM) were randomly selected for research sampling. From the responses obtained, the collected data underwent a pre-processing process, such as determining the missing data and converting it to the appropriate format (i.e., convert excel to. prn text file). Next, the data analysis was performed using the Rasch measurement model, which focuses on four main analyses, including: (i) reliability and separation index; (ii) item fit and polarity. The flow of the data analysis was illustrated in figure 2.

Rasch Model in Measuring Heutagogical
Equation 1 shows the adaptation of the mathematics model underlying the Rasch measurement model [23,24,25]. This principle indicates that students have a higher probability of responding to the less difficult item and a lower probability of accurately responding to the more difficult item.
where is the ability of student n and is the difficulty of heutagogy item i. Besides, − is the probability of student's response for each heutagogy item. In this case, the { = 1} is the success probability or also known as logit value upon the interaction between the relevant students and the assessment item. Thus, this study has simply the Equation (1) into a logarithmic function, as shown in Equation (2).
The probability of students' success for each heutagogical item demonstrates the relationship between students' ability and the difficulty of given items in the heutagogy element.

Analysis Using Rasch Measurement Model
Based on the aforementioned Rasch model analysis, Table 1 shows the summary of the statistical criteria and the acceptable value for further study. The following analysis in Table 2 was examined in order to assess the validity and suitability of the item in the Rasch Model.
a. Reliability and Separation Index. Reliability refers to the consistency of the measure, which can be obtained through the Cronbach Alpha values and the internal reliability index [26]. The value of Cronbach Alpha (α) is between 0 (no internal reliability) and 1 (perfect internal reliability) with a minimum score of 0.70 [27]. While the separation index indicates the statistically distinct measurement level of an item's difficulty or a person's ability [27]. b. Item polarity. Item polarity was evaluated by using a pointmeasure correlation coefficient (PTMEA CORR). The values of the high and positive items range from 0.3 to 0.8, representing items that are working in the right direction to measure each of the developed constructs [28]. However, the negative values or zero indicated a conflicted relationship between items within the construct. An item that exceeds the 0.3-0.8 interval is recommended for removal [29]. c. Item fit. In the Rasch model, item infit and outfit mean-square fit statistics (MNSQ) were commonly used to determine the item fit assessment. The infit statistics are inlier-sensitive fit statistics, which indicate the responses of items that are close to the person's ability level [30]. Outfit statistics are referred to as outlier-sensitive fit statistics, which represent unexpected responses to items outlying from the person's ability level. The item is considered to be consistent with the Rasch measurement when MNSQ reaches the expected value of 1 and always be positive. d. The MNSQ optimal value of each item must be located within 0.5-1.5 [30]. The values of less than 0.5 or 1.5-2.0 do not bring efficiency to building measurements, but do not decline [31]. Therefore, any individual item with MNSQ more than 2.0 will be suggested for removal from the present study. The item needs to be repaired < 0. 5 The item needs to be dropped Separation [32] Value ≥ 2 Acceptable Item Validity Item Polarity [26] PTMEA CORR > 0 Acceptable Item Item Fit [26] Total Mean Square infit and outfit in range 0.6-1.4 Acceptable Total Mean Square infit and outfit in range 0.6-1.5 Acceptable e. Standardized measurement of the residual correlation value. This analysis was conducted to determine the overlapping of the items. High residual correlation for pair items indicated that item is not independent (i.e., items are the same characters or combination of the shared item). As for the pairing item's correlation, value that is greater than 0.7 indicates a high correlation value, and only one item must be maintained. In contrast, the other items should be dropped [32,33]. Table 3 presented the demographic profile of the respondents. There are three main demographic profiles collected, such as gender, year of study, and faculty members. All the information was presented in the cross-tabulation output, which is faculty versus year of study and gender versus faculty. Gender information is included in this study as it is a way of looking at how heutagogy element impacts learning experiences between different groups of male and female students.  Table 4 shows the statistical summary of the reliability and separation index of the survey instrument. The value of 0.97 obtained in this study indicates that the reliability of the instrument used was high and adequate. In item measurement, the reliability value of 0.93 indicates that items of the instrument are very sufficient to measure the learning heutagogical elements. The person's reliability indicates the probability of a person's response results of 0.94 when the same test is performed. Hence, this heutagogical learning instrument is at a high level of reliability. Meanwhile, for the separation index (SI), both item and person measurements were obtained acceptable value ≥ 2, SIitem = 3.75, and SIperson = 4.01.  Table 5 shows the mean and mean logits for each construct in this developed instrument. The higher mean score is obtained for 'explore' construct, x̄ = 4.018, which indicated that the students agreed that the explore item suggested is working well in their learning process. This is matter because the student needs to be provided with free and large opportunity to explore the variety of learning pathways, information, and resources during the learning process [9,15]. However, the value of the connect construct means obtained, x̄ = 3.804 still indicates the high level of student's agreement with the connecting element. This matters because the creation connections are comfortable with the affordance of an online platform (i.e., social media, video conferencing), which allows students to have the opportunity to share new knowledge with others who have the same interest [34].

Mean and Mean Logit
On the other hand, the measure means were obtained to determine the item agreement among the students. The negative value item indicated that students were more comfortable to agree on the item. In this case, explore (-0.24 logit) and share (-0.03 logit) constructs were more comfortable obtaining a high degree of agreement among the students.

Correlation
Standardized measurement of the residual correlation value is to determine whether there are overlapping items. High residual correlation for the two items showed that the item is not independent, either because the item has the same characteristics among each other or because it combines several different dimensions that are shared. If the correlation value of the two items above 0.7 shows a high correlation value and only one item must be maintained [31]. In contrast, the other items should be dropped. Based on Table 6, no overlapping pairs have been found, and no items have been dropped.

Item Fit and Item Polarity
Due to space limitation, Table 6 presented only the estimation of difficulty parameter and model fit estimations of misfit items. Mean square information (MNSQ) and the point measure correlation (PTMEA-CORR) aimed to determine the item's suitability for the developed instrument. For the MNSQ assessment, the list presented that MNSQ infit range is at 0.67 logit -1.90 logit. As the MNSQ infit acceptable range is 0.50 logit -1.5 logit, three items RF59 = 1.90 logit SH53, item SH44 = 1.58 logit, were found exceed the acceptable range. Based on the output presented in Table 5, an example of the ICC curve for misfitting items was generated in Figure 3. The red curve is the expected ICC if the data fitted the Rasch model. The blue curve is the observed or empirical ICC. The grey line in the outskirts of the red curve is the confidence interval constructed from an estimate and its standard error. The empirical ICC for misfitting item RF59 has a slightly large deviation from the expected ICC. This fact is reflected in RF59 with a large outfit-MNSQs value of 2.02. However, Figure 4 presented a good-fit item, SH47, in which the outfit-MNSQs value, 0.90 was obtained and is within an acceptable range. In addition, PTMEA-CORR can also be used to accept or abolish the items tested. The acceptable range depends on the purpose of the instrument. However, the negative value of the PTMEA-CORR items does not measure what should be measured and should be dropped. In this study, all tested PTMEA-CORR items were positive (0.43 logit -0.63 logit) and were therefore retained. The detail of the item fit explained in Table 7, which shows fit index item that measured in infit/outfit, and items RF59, SH83, and SH44 are more than 1.4 logit values, and this item needs to revise.

Variable Map
Based on the above findings, this list of logit measurement information for each item is useful for understanding each item's position in the variable map. Figure 5 shows the distribution of the item's difficulty. In this case, item CR12 received a high measure of logit value, +1.99 indicating the most difficult item to respond to. In contrast, item CN35 indicated the easier item to be answered as the measured value obtained, -0.82 logit.

Conclusion
An instrument for gauging a heutagogy element in learning for the engineering education area was successfully developed and validated using the Rasch measurement model. Using this model, the validity and reliability of items have been proven using the data gathered from the pilot study, to fit the engineering education students. In this study, we have provided two implications of the findings based on The Rasch model framework provides systematics procedures for validating the developed instrument (e.g., Reliability, construct validity, item fit, separation index). Besides, the Rasch measurement model provides an impact on discovering the meaning of students' measure, which heutagogy element applied in students' learning using from the context of the instrument's items. The first contribution in this context, a huge implication, is probably on the potential for individual reflection and feedback practices. Practitioners in schools and teacher educators can be referred to the Wright Map/Variable Map that explicitly uses the heutagogy element in the learning process. Teachers can prepare the best strategies and implementation process for important aspects of the heutagogy element to students. Another implication from this study identified potential strategies for curriculum development and the teaching process. A total of 175 students can generally use the heutagogical elements at least by sharing information with other engineering or non-engineering students through an online platform. In this case, students can discover more learning experiences, learn new things, and identify mutual students with similar interests. This repeating process can lead to life-long learning and future collaboration. We are currently working on a test within the different categories of students, from pre and postsecondary subjects, using the developed instrument to explain the changes in the critical heutagogical elements. The findings will be reported in the subsequent publication.