USER EVALUATION OF THE INTERFACES FOR THE REMOTE CONTROL OF MANUFACTURING SYSTEMS

This paper is a contribution to the new direction for enhancement in remote controlling functionality in manufacturing systems and presents a methodology for evaluating user interfaces for the remote control of a Ubiquitous Manufacturing System. Survey is based on the experiment that involved 68 subjects, students at the Faculty of Mechanical Engineering, the University of Belgrade, Serbia, as future users of the interface for the remote collaborative control of manufacturing systems. The results show that the user evaluation instrument consists of usability and collaboration as correlated measures. Usability comprises effectiveness, efficiency and representational fidelity measures, while collaboration includes collaboration effort, awareness/involvement and copresence.The results of this survey will help in future design decisions and measuring its quality of use.


INTRODUCTION
The process of globalization and freer movement of goods between the markets of the whole world, especially in the economic crisis that hit the world in late 2008, have shown that the classical vision of business no longer corresponds to economic reality.This fact is emphasized when it comes to production-oriented enterprises.For example, today's need for a high degree of specialization in various narrow fields of work combined with the need for a flexible manufacturing system which will adapt to a very specific, wide range of customer preferences, requires a new vision of a modern system that should unify contradictory requirements: specialization and flexibility.It would therefore be wrong to seek a solution within a classical production system and a large company.One of the foreseen solutions lies in networking and the joint use of smaller production systems that include remote operation and controlwithin a distributed and networked manufacturing systems paradigm.
The synergy of networking and communication channels is one of the essential factors in communicating and coordinating the activities of engineers in enterprises dispersed in different locations that enables the free capacities of small production systems to be used by network members/clients anywhere and anytime they are free, making the manufacturing system ubiquitous.Within this context, the issue of interface design and use, as one of the most important enabling factors for the operation and control of distributed and networked enterprises, is one of the open research questions.
The purpose of this paper is to present a methodology for evaluating user interfaces for the remote control of a Ubiquitous Manufacturing Systemand its results, which will help in future design decisions and measuring quality of use.The sets of concepts and dimensions currently available for interface evaluation vary from one author to another in terms of the number of dimensions, degree of generality or specificity, and level of precision and describe other concepts different to those described herein.Most of them, as they are not designed for this purpose, lack collaboration or representational fidelity dimensions.We started by identifying the set of variables which may be necessary to build a comprehensive evaluation framework and after that the collected data underwent second order factor analysis.The experiment involved 68 participants, students at the Faculty of Mechanical Engineering, the University of Belgrade, Serbia, as future users of the interface for the remote collaborative control of manufacturing systems.The results show that the user evaluation instrument consists of usability and collaboration measures.Usability comprises effectiveness, efficiency and representational fidelity measures, while collaboration includes collaboration effort, awareness/involvement and co-presence.

Evaluations
of human-computer interaction have traditionally been carried out by means of a number of methods, including field studies, laboratory experiments, and inspections.Each method assesses different aspects of the interfaces and places different demands on the developer, user, and evaluator.Evaluations of collaborative technology are best made through field evaluations because they can, among other things, be used to assess the social-psychological and anthropological effects of the technology, while experiments are very appealing for new and rapidly evolving technology and are potentially less expensive than field studies (Grudin, 1988).As Barnum (2002) states today the focus must be on the user, not on the product.In the case of this work and based on previous research, experiment fits best.
User satisfaction is considered one of the most importantmeasures of any information system's success (DeLone & McLean, 1992).Most of this literaturefocuses on explaining what user satisfaction is by identifyingits components, and usually suggests that user satisfaction should be a single construct.Substantive research studies use a total score obtained by summing items, thus implying that user satisfaction is a single first-order construct (Doll et al., 1996).The Ives, Olson and Baroudi instrument is the most widely used for measuring general user satisfaction on information systems (Ives et al., 1984).Several years later the importance of developing standardized instruments for measuring traditional user satisfaction was recognized (DeLone & McLean, 1992), but most studies still focused attention through exploratory studies or replications.For example, in an exploratory study, Doll and Torkzadeh (1988) proposedan end-user computing satisfaction (EUCS) measurement model consisting of content, accuracy, format, ease of use and timeliness.Two decades later Straub et al. (2002) emphasized the importance of good metrics for net enhanced organizations and stated that "the unique characteristics underlying the Web may in some cases require new metrics or at least the careful evaluation of existing ones, to facilitate the development of innovative solutions to emerging problems", which means that instruments such as the EUCS need reconstruction and revalidatation.Abdinnour-Helm et al. (2005) revised and revalidated the EUCS instrument through an experiment with a sample of 176 students who participated in a lab simulation involving a usability evaluation of the Lands ' End Website (www.landsend.com).One important contribution of the Abdinnour-Helm et al. (2005) study is the presentation of a typology with a set of four new dimensions (Competitive Environment, Marketing Environment, Usage, and Usability).Aladwani and Palvia (2002) developed new instruments and scales for web quality, directly targeted to these new interfaces and applications.In this article, an instrument with 25 items grouped in four dimensions: specific content, content quality, appearance and technical adequacy, was developed.
Most usability engineering methods were developed to design and evaluate either command-line, traditional graphical user interface systems or web quality.Therefore, they were developed to efficiently and effectively discover those usability issues that are inherent in these environments.As can be seen through the literature review, different contexts need different metrics, and that is the reason why Abdinnour-Helm et al. ( 2005) encouraged researchers to continue questioning when, and how, metrics ought to be used in new contexts.

Remote control of manufacturing systems
This survey sets out to provide answers to interface evaluation in a completely new context.Today, 50-80% of all the components manufactured by original equipment manufacturers are designed and manufactured by different groups of designers at geographically different locations (Rezayat, 2000) making it difficult for colleagues to collaborate and exchange their ideas via e-mail and phone discussions.One of the solutions is a web-based collaborative environment and systems enlargement, that has emerged in the recent crises.An example of the Wise-Shop Floor (Web-based-integrated sensor-driven e-ShopFloor) framework through amilling machine remote control in a classical production system case study is given in Wang et al (2004).
In transition from classical and traditional to modern manufacturing systems, a number of key improvements have occurred, including the ability to control a machine remotely, over the Internet.Our proposal is a new concept of horizontal inter-firm collaborations that enable the free capacities of small production systems usage by network members/clients anywhere and anytime they are free, which certainly has different user satisfaction factors to traditional information systems and web applications, that are a small part of new systems.Collaboration is very important in this new concept, since there are multiplex communication system levels, as shown in Figure 1.
One of the rare research studies in this field analyzed manufacturing grids or collaborative manufacturing networks which have a strong presence in sectors such as computer manufacturing, astronomy and bioinformatics (Liu & Shi, 2008).The paper by Pappas at al. (2006) proposed the Distributed Collaborative Design Evaluation (DiCoDEv) platform with real-time collaboration of multiple users at different sites on the same product design project and tested it on apilot case.These works analyzed how to coordinate the use of heterogeneous and independent manufacturing resources distributed through out the network, either in a traditional manufacturing grid, or in the aim of controling the product design phase.There are still many aspects to develop in this field such as the development of a comprehensive ICT prescriptive model that can manage horizontal inter-firm collaborations or evaluate the need to direct these networks (Vilana et al., 2010).

2. User evaluation and interfaces for the remote control of manufacturing systems
"One of the basic lessons we have learned in human-computer interaction (HCI) is that usability must be considered before  (Putnik at al., 2011 andShah at al., 2012) prototyping takes place" (Holzinger, 2005).Recent research on usability studies has questioned established methods and their suitability for testing more complex systems (Redish, 2007).Nevertheless, usability studies are still not considered an obligatory part of design in this domain.
Usability is defined in Part 11 of the ISO 9241 standard (1998) as "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use."Effectivenessis the accuracy and completeness with which specified users can achieve specified goals in particular environments.Efficiency is defined as the resources expended in relation to the accuracy and completeness of the goals achieved.Satisfaction is the comfort and acceptability of the work system to its users and other people affectedby its use.
On the other hand, in the field of Software Engineering (SE), the most widely accepted definition of usability is that proposed in the ISO 9126-1: 2001 "the capability of the software product to be understood, learned, operated, attractive to the user, and compliant to standards/guidelines, when used under specific conditions" (Bevis et al., 2008).
These different definitions of usability directly affect how it is evaluated, since each method or technique employed in these evaluations may focus on different aspects of the term usability.Hartson et al. (2001) added that in their opinion, given after mapping 206 research papers in the field, the usability concept has not been defined consistently in the ISO standards, which might be a problem since usability as aquality characteristic may not actually cover all the usability aspects.
On the other side, collaboration, as a very important characteristic in the remote control of manufacturing systems, is the process of multiple people working together to achieve a greater goal than is possible for any individual to accomplish alone (Todd, 1992).Although the concept is as old as human civilization, collaboration is often taken for granted, overlooked, misunderstood, and poorly accomplished in practice.Today in an extremely connected society, collaboration is ubiquitous in all professional activities (Lu et al., 2007) and requires a team of individuals to work on tasks that not only have shared resources (as in coordination) and shared outcomes (as in cooperation), but, most importantly, a shared common goal.

The interfaces for the remote control of manufacturing systems: user evaluation instrument
Modern explanations of what usability means agree that it is context dependent and shaped by the interaction between tools, problems and people (Hornbaek, 2006).Secondly, usability cannot be directly measured.Secondly, usability cannot be directly measured, but through the operationalization of the usability construct we can find aspects of usability that can be measured.In turn, the choice of such measures not only fleshes out what usability means, but also raises the question as to whether that which is measured is a valid indicator of usability.That is the reason why, based on logic, theory, previous studies, and in depth interviews with several experts in the field, the instrument structure herein is proposed.
For example, Hornbaek (2006) analyzed 587 candidate papers in the field of usability and selected 180 studies as relevant.All 180 studies found effectiveness and efficiency, and most of them satisfaction measures as the crucial ones and all of those are also contained in ISO 9241:1998.Effectiveness contains the percentage of task achievement, the accuracy of completed tasks, the number of functions learned and the number of user errors.Efficiency contains the time to complete a task, users` mental effort, communication between users and the time spent on correcting errors.User satisfaction is described with standard measures from previous instruments (EUCS) for traditional environments through Content (4 items on the second level), Traditional Accuracy (2 items on the second level), Format (2 items on the second level), Ease of use (2 items on the second level), and Timeliness (2 items on the second level), to see weather traditional measures are important in the newly proposed concept for the remote control of manufacturing systems.As stated by Hornbeak ( 2006), exactly 62,22% of the studies reviewed, included user satisfaction measures.Representational fidelity (4 items on the second level: Realistic display of environment, Smooth display of object motion, Consistency of object behavior, Embodied actions including view control, navigation and object manipulation) could be added to the satisfaction or collaboration measures, as proposed by Dalgarno (2010).As previously mentioned the collaborative surrounding should also be incorporated and measured through the measures proposed by Hrimech (2011) such as 1.Collaborative Effort ("My partner worked with me to complete the task" and "I did not help my partner very much?"), 2. Awareness/Involvement etc. ("To what extent wereyou involved in communication and the experimental task that you lost track of time or place?", "To what extent did events occurring outside the scene distract from your experience in the interface?","I was an active participant in the task" and "I enjoyed the experience in the Portuguese lab." and 3. Co-presence ("To what extent did you have a sense of being in the Portuguese lab?" and "When you continue to think back on the task, to what extent do you have a sense that you are together with the machines in the lab?").
Questions with a five-level Likert scale were mostly administered in a questionnaire, while some items were measured.Using recommendations by Grandzol and Gershon (1998) to recode 25 -50% of the questions (posed in reverse order relative to the other questions), 32% ofthe questions were recoded.Additionally, in order to carry out a check one control question was posed.Any questionnaires with no coincidence in the control question could be rejected later.

User evaluation instrument: Methodology
Traditional statistical methods normally utilize one statistical test to determine the significance of the analysis.However, Structural Equation Modeling (SEM), and Confirmatory Factor Analysis CFA specifically, relies on several statistical tests to determine the adequacy of model fit to the data.
CFA is used to study the relationships between a set of observed variables and a set of continuous latent variables.CFA is a measurement model, which implies a multivariate regression model that describes the relationships between a set of observed dependent variables and a set of continuous latent variables (Joreskog & Soborn, 1984).The observed dependent variables are referred to as factor indicators and the continuous latent variables as factors.
The methodology of confirmatory factor analysis involves the specification and estimation of factor structure models, each of which proposes a set of latent variables and its dimensions to account for covariances among a set of observed variables (Hair et al., 1998).The sets of concepts and dimensions currently available for interface evaluation vary from one author to another in terms of the number of dimensions, degree of generality or specificity, and the level of precision.As they are not designed for this purpose, most of them lack collaboration or representational fidelity dimensions.We started by identifying the set of variables which may be necessary to build a comprehensive evaluation framework and they are given in part 3.3.After that the data collected in the experiment with 68 participants underwent second order factor analysis through software Statistica 8, by StatSoft .

Experiment
The described methodology application is illustrated through the experiment undertaken during the bilateral cooperation project between Serbia and Portugal.The experiment involved 68 subjects, students at the Faculty of Mechanical Engineering, the University of Belgrade, Serbia, as future users of the interface for the remote collaborative control of manufacturing systems.The descriptive statistics on the experiment participants are shown in Table 1.
Their task was to control a CNC machine located in the laboratory at Universidade Minho, Guimarães, Portugal, from a classroom at the FME, University of Belgrade, Belgrade, Serbia, as can be seen in Figure 2.
As can be seen in Figure 2, the user interface for the remote control functionality has several key components: a) a control panel for the remote machine controls (e.g. to move the axes and the start/stop spindle, to upload and run a machine program etc.) b) the communications controls c) a panel which shows the absolute and relative positions of each axis, i.e. the feed-back information from the machine movements, Table 1.Descriptive statististics on the experiment participants and d) a communication screen for organizational level communication with a video frame for live video feeds.The "client" user interface of the distributed manufacturing system which belongs to the Ubiquitous Manufacturing System Demonstrator is described in detail in Putnik et al. (2011).This interface provides the two remotely located entities (remote user -"client", cell operator and cell machines) with remote operation ability as well as twoway multiplex communication (Putnik at al., 2011), which is absent in the traditional models.The participants` task consisted of connecting to the remote cell, starting a CNC machine, uploading the ag-code CNC program to conduct operations on the machine, remotely using the emergency stop button (which exists physically on the CNC machine), moving the axes, and assessing the status of the machine and real-time positions of the axes, as can be seen on the left side of Figure 3.

User evaluation instrument: Results
In order to identify the set of variables which are necessary to build a comprehensive evaluation framework of interfaces for the remote control of manufacturing systems, the scales were first checked for unidimensionality and statistical reliability.After that, the following validity checks were done: content validity, convergent validity, discriminant validity, and criterion-related validity.Content validity refers to whether the scale representatively measures the concept it is intended to measure.Since we derived many of the items in our questionnaire through a comprehensive study of relevant literature and existing instruments, the content validity of our instrument was established.Unidimensionality is a necessary condition for reliability analysis and construct validation.A GFI (Goodness of fit index) of 0.90 or above suggests each of the constructs is unidimensional (Hair et al., 1998).Cronbach's alpha values for the items in each question are higher than 0.70, which means high enough reliability.Convergent validity refers to the proximity of the results of different approaches to the same problem.This is examined by using Bentler & Bonett`s normed fit index (NFI), which is very close to the recommended value.
Bentler & Bonett`s NNI (Non-normed index) value not being extremly close to the threshold is influenced by the sample size in our experiment (Hooper et al., 2008).The discriminant validity of the scale refers to whether the item in one scale is distinguished from the construct of another.In order to establish discriminant validity, confirmatory factor analysis was performed on a selected scale.The results show that the user evaluation instrument consists of usability and collaboration measures.Usability comprises effectiveness, efficiency and representational fidelity measures, while collaboration includes collaboration effort, awareness/ involvement and co-presence as can be seen in Figure 4.The adequacy of the model was assessed through a large set of fit measures, which for our best model show very good corespondence to the recommended values.As can be seen in Table 2, a large class of omnibus tests exists for assessing how well the model matches the observed data and is calculated through Statistica 8 software.Here only 3 out of 9 indexes are not within the thresholds, but are very close.RMR (Root mean residual) is a residual based measure, whose higher values are acceptable when the values for CFI (Comparative fit index) or NFI (Normed fit index) are within the thresholds (Hooper et al., 2008).The SRMR (standardized RMR) is an absolute measure of fit and is defined as the standardized difference between the observed correlation and the predicted correlation.As practice shows, we obtained very good model fit results.

CONCLUSION AND DISCUSSION
In this study, we have investigated the factors important from the user side on interfaces for the remote control of manufacturing systems using confirmatory factor analysis, and explored the correlation between two latent constructs -usability and collaboration.
The results of the confirmatory factor analysis also show that effectiveness, efficiency and representational fidelity are three important indicatorsfor the usability construct, while collaboration effort, awareness/involvement and co-presence are the indicators for the collaboration construct.Most of the correlations are rather high.
We have also found that the measures from the end-user computing satisfaction (EUCS) measurement model, consisting of content, accuracy, format, ease of use and time liness, are not appropriate for the evaluation of interfaces for the remote control of manufacturing systems, since they are not part of this model.On the other side, our results demonstrate the importance of collaboration, which was rarely discussed, and is now an important part of our model.
Researchers can use this methodology when formulating models of remote control behavior, usage patterns, and overall success.These results may also be useful to remote control interface designers.Future research efforts should consider the integration of other factors, such as contextual variables influence, such as individual (gender, age, learning characteristics etc.) and group features.Marketing executives may then also use these results to discern the differences in usability across target user groups.We also expect future studies on larger sample sizes to further validate these results.

Figure 1 .
Figure 1.Multiplex communication system levels for remote control and network user communication channels(Putnik at al., 2011 andShah at al., 2012)

Figure 2 .
Figure 2. Experiment at the Faculty of Mechanical Engineering, Belgrade, Serbia

Figure 3 .
Figure 3. Remote Client Interface for Operations on Cell

209V.
Figure 4. Best measurement model with parameter estimates