How do I simulate the IEEE paper
Simulate what rarely happens. User-centered development of mobile simulators for complex maritime major disaster situations
Major damage events rarely happen, but they are associated with a considerable risk for people and property as well as for the safety of shipping routes and therefore require the special competence of all organizations involved in rescue services. At the same time, due to the complexity of the situation (including type of ship, type of cargo, number of people to be recovered, impact on the environment) and the large number of organizations involved (e.g. German Society for the Rescue of Shipwrecked People, Fire Brigade, Federal Police, Technical aid organization, air rescue) unique demands on all those involved. Events such as the fire on the Lisco Gloria ferry in October 2010 in German-Danish waters also show the importance of smooth cooperation between international aid organizations (Rosenkötter 2015). The accident of the “Costa Concordia” in January 2012 off Giglio is also an example of the dimensions of possible rescue missions at sea (Schlamp 2018), but also on inland waters - the Viking Freya collides with a bridge - and in ports - the container ship catches fire CCNI Arauco - show (Buchenau 2020).
As a rescue, spontaneous ad hoc organizational structures develop in the respective organizations involved in order to cope with any tasks that arise. Existing organizational cultures, different operational roles and a large number of other factors can lead to a different interpretation of the situation. If the interpretations differ between the organizations involved, the efficiency and effectiveness of communication is reduced and misunderstandings are likely (Salas et al. 2006; Flin et al. 2008; Zoller 2014). For this reason, regular joint training of the (potentially) participating organizations is not only elementary to improve the basic skills required for crisis management (Strohschneider 2012; Tena-Chollet et al. 2016). The training also improves an action reaction appropriate to the situation and group-specific problem solving (Weick 2010, Hofinger 2008).
This need for training with as many participating organizations as possible, often across national borders, is offset by the high financial and organizational effort involved in realistic exercises in such situations. An alternative that makes sense in terms of content and economy would be training in the simulator. Simulator exercises are far more cost-effective than real-world exercises and can be used flexibly with little expenditure of time (Wright and Madey 2008; Molka-Danielsen et al. 2018). However, isolated solutions are currently primarily available for major maritime incidents, which do not allow cross-organizational training scenarios and are therefore not suitable for simulation. There is a need for modular, networked simulators that are connected to a central instruction center (Riener 2010). In addition, there is a lack of simulators for mobile, modular and multifunctional operation stations (MOS - Mobile Operation Station), which, for example, simulate a ship's master, helicopter and control center workstation and thus can map the range of activities of the rescue forces in the event of a major maritime disaster (Lübcke et al. 2015 ).
The article presents the user-centered development and evaluation of MOS using the user-centered development process (International Organization for Standardization (ISO) 2010). On the basis of focus groups with later user groups, application-related requirements were raised, a design concept was drawn up and this was continuously developed further in three iterations. This was followed by an evaluation using user tests with quantitative and qualitative data, the results of which formed the basis for the final adjustment. At the end of the development process, there is a MOS concept that meets the requirements of all user groups and is highly usable for all three workstations. The outlook summarizes the knowledge gained for the development of such complex simulators and shows which components are of particular importance for simulating and training major disaster events. In addition, the advantages and disadvantages of the methods used are discussed.
User-centered development of the MOS
The MOS must serve three different use cases and thus be able to map the working environments of a skipper, helicopter pilot and control center employee. For this purpose, a sufficiently generic design based on user requirements must be developed that reproduces realistic workload levels in simulator training without providing a highly detailed environment (see 2.3.1). At the same time, the required mobility and modularity pose great challenges for the materials and assembly methods used. Without great effort and without specialist knowledge in the area of software and hardware installation, every potential user should be able to set up and convert the MOS and to put it into operation.
Development activities in the past have mostly concentrated on functional product development (Schlick et al. 2010), with the main focus being on financial and technological requirements when developing a new product. The user-centered development process (UCD, see Fig. 1) involves people or user groups who use the system later in the development in the early project phases and improves the usability of the developed products (ISO 2010). In order to meet the high usability and ergonomic requirements of heterogeneous user groups, the UCD is a very suitable and precise method (Bevan 2009) and is used in many different domains, e.g. also for the design of teaching content (Kahraman 2010; Ecker 2016) . The advantages also include products that can be operated safely, lower maintenance and operating costs in the long term (Ritter et al. 2014) and reduced training costs for MOS (ISO 2010; Chammas et al. 2015).
The user-centered development process is divided into five phases. After a general planning phase, developers have to go through another four phases, starting with the analysis of the context of use. In this phase, the users are characterized on the basis of their tasks, their physical and social environment, as well as their work and auxiliary equipment. In the following phase, the information about the context of use must be refined in order to derive requirements for the product to be developed. Methods that support the specification of requirements are personas or application scenarios (Maguire 2001). The creation of design solutions is the main task during the third phase. Developers create several design alternatives using methods such as paper prototyping (Bailey et al. 2008, Still and Morris 2010) or according to standardized guidelines and standards (e.g. ISO 9241-110). In this phase, developers have to consider both the technical and the requirements resulting from the context of use. In the fourth phase, users or experts evaluate the created design solutions according to the previously defined requirements, e.g. B. by means of Standard Review (Nielsen 1994, Shneiderman et al. 2016) or Cognitive Walkthrough (Blackmon 2004). Developers are thus given the opportunity to benefit from an evaluation of the design solutions in the early stages of development, to identify possible weak points and to implement improvements in further iteration loops. The following describes the user-centered development process for the mobile, modular and multifunctional operating station.
Context of use analysis
For the implementation of the first phase of the user-centered development process, the analysis of the context of use of the MOS, a mix of methods was chosen (Creswell 2009), which combined qualitative and quantitative data as a result and is usefully used in areas of application unknown by the researchers involved. In order to record the requirements for display and control elements as well as simulation parameters of all user groups (skipper, helicopter pilots and control center employees) and the resulting ergonomic and functional requirements, five focus groups were carried out for the participatory development of ideas (Möslein et al. 2010). A major advantage over online or structured personal surveys is the direct interaction with the participants, which allows a flexible discussion without focusing on a certain structure (Langford and McDonagh 2003). The moderator is able to guide the participants through the planned phases, to ask follow-up questions or to answer questions. Possible influencing factors on the quality of the results are the management of dominant group members, the quality of the discussion and the discussion of irrelevant content. The focus groups were divided into three main areas:
Objectives and contents of a maritime simulation network as well as explanation of current problems during rescue missions,
Compilation of display and control elements as well as their prioritization in the categories navigation, connecting (machine control elements) and communication (as well as other possible categories),
Paper prototyping of a MOS.
The number of focus groups was weighted taking into account the later frequency of use of the MOS. The main user group of the skippers (16) formed three focus groups, helicopter pilots and control center employees each added a further focus group to the test group. A total of 31 people took part. The group of control center employees (10) consisted of two sub-groups: control centers for emergencies on land (1) and control centers for emergencies on water (6). The focus group was supplemented by employees from offshore wind farms (3), which play an important role due to their increasing distribution in the North Sea. The acquisition of helicopter pilots turned out to be difficult, but five participants were able to provide interesting information and contributions. The breakdown of the participants according to user groups is shown in Fig. 2.
The gender distribution is very one-sided with 29 male and two female participants, whereby this gender distribution reflects the population (Konstantin 2010). The average age of all participants is 48.7 years. All test persons have a lot of experience in their domain and have already participated in various real-life exercises and simulator training, which is why they are to be classified as lead users (Bogers et al. 2010).
All five focus groups were carried out according to the same schedule. After a short welcome round, the agenda and goals of the workshop were presented. Then the topics of simulator networking and MOS were presented to ensure that all participants had the same prior knowledge. In order to initiate the group language flow, the participants were invited to introduce themselves, their organizational background and their current area of responsibility. Then the participants listed goals and content that should be trained in a simulator network. In the following discussion the results were discussed and prioritized. After visiting the existing simulator cabins in the main building of the German Shipwrecked Rescue Society in Bremen, where all focus groups took place, the participants were divided into three groups. Each group was assigned to one of three categories: Navigation, Conning, or Communication. All function, display and control elements as well as simulation parameters were developed within the group work. The results were then presented to all participants, discussed, supplemented if necessary and prioritized in a three-stage system. The gradations “must be present”, “can / should be present” and “would be nice if present” were marked with different colored dots. The focus groups were concluded with a paper prototyping slot, in which the participants could draw or draft their own design drafts. Finally, a summary of the results and an outlook on the upcoming activities was given. An audio recording was made of all focus groups. Exemplary results of the focus groups can be found in Fig. 3.
The data generated was then evaluated, with the focus at the beginning on the summary of the goals and content mentioned by the participants as well as the desired displays and operating elements. Transcriptions from all focus groups served as the basis. The total of 490 pages of text documents were examined using a qualitative content analysis. With the help of 807 text passages, a system of categories was created which reflects the requirements placed on MOS from the point of view of the categories of people, technology and organization. The result is a catalog with requirements, which is reflected in the simulated workstation, the functionality assignment (navigation, connecting, communication) and the gradations must be present, can / should be present and would be nice if available, subdivided. Tab. 1 shows an excerpt from the catalog of requirements.
The results show that helicopter pilots and skippers have a large area of overlapping requirements in the areas of communication and navigation. At the same time, the needs in terms of machine monitoring and control differ significantly, but this can be compensated for by the modular design of the MOS. The third end-user group, the control center staff, can be integrated with relatively little effort, since on the hardware side they only prefer screens for display and a standard mouse-keyboard combination as an input device to carry out their work. In Fig. 4 is an overview of the requirements for the workplaces with the criterion must be present shown. The abbreviations in brackets stand for the respective mention by the individual user groups, where (S) stands for skipper, (H) helicopter pilot and (L) for control center employees.
Determination of the degree of immersion
In the course of a literature research regarding the determination of the degree of immersion and the perception of presence by users, it was found that the determination of the degree of immersion is mostly based on expert judgments and cannot be objectively measured or traced. Only presence can be recorded as personal perception. It is defined as the experience in a virtual environment of being somewhere other than where the person is physically located. The perception of presence includes the deliberate removal of doubts about the place where the person is physically located (Slater and Usoh 1993, Slater et al. 1994).
As part of an expert workshop, this topic was taken up with six participants and discussed in a practical way. The experts found that technology is only one factor in the perception of presence. Experience from a large number of exercises carried out shows that the intrinsic motivation of the participants has a strong influence on their perception of presence. The experts stated that the acceptance of real or simulated equipment depends on the exercise goals to be trained. Furthermore, the “play instinct” in monotonous situations was identified as a distraction and thus a disturbance variable. As a result of the literature research and the expert workshop, a low level of immersion in the form of a low level of reality was specified as a requirement for the MOS prototypes (see Fig. 5). As a basis for further development work, less realistic control elements should therefore be used and work environments should be implemented.
Design of the MOS
Following the user-centered development process (ISO 2010), the design of the skipper, helicopter and control center workstations followed in various degrees of maturity. The first drafts were based on paper prototyping sketches that were created as part of the focus groups for requirements analysis. The Microsoft Visio program was chosen for quick implementation. Based on this, design drafts were made using the SketchUp software from the manufacturer Trimble. This made it possible to create three-dimensional images of the workplaces, which enabled a high level of detail.
The subsequent design process was iterative, using the prototyping solution described above as the basis. Based on the methodology for workplace planning (Kleinhenz 2011), research was first carried out on standard components for surface modules that could be used for the rough construction in the prototypes. Among other things, height-adjustable standard tables and a monitor holder were put together. In addition, there are individual tables, which allow the screens to be set at a variable angle, in all three workplaces. Then the area modules were converted into an area structure based on possible workflows. Finally, the resulting arrangement of the individual components was assessed according to their practicability and the fulfillment of the requirements, and the resulting changes were implemented. The resulting jobs formed the second iteration.
During the third iteration, the specific details of the workplaces were adjusted.For this purpose, the results of the qualitative content analysis were used in order to best meet the user requirements for the workplaces. In detail, further elements for the fulfillment of tasks, such as printers, card tables, communication units and headsets, were arranged at the workplaces. The individual software components were also assigned to the screens and the primary flight displays were arranged at the helicopter workstation. The procedure described should be clarified by comparing the iterations in Fig. 5.
In the final phase of the user-centered development process, after the prototypes had been designed, a usability test in the form of a user evaluation was carried out (Sarodnick and Brau 2006). Here, lead users were used who had already participated in the creation of the requirements in the context of the focus group workshops that had taken place beforehand. Due to the low participation rate, other people were also contacted by organizations involved in the project. Due to the limited availability of the participants and a lack of flexibility in terms of time, the usability tests were carried out with partially structured telephone interviews. A mix of qualitative and quantitative data was used. For this purpose, the participants received a 3D prototype of the respective MOS in advance of the survey, which could be opened and viewed with the Adobe Acrobat Reader. As part of the interviews, the respondents were asked to put themselves in various simulator training situations, for which a story board was used to describe a brief history of an operation (Richter and Flückiger 2013). Subsequently, elements were queried that were still missing in the 3D prototype or should be positioned differently. Furthermore, the test persons were asked questions regarding their assessment of the organizational implementation of the simulator training with the presented prototype. Finally, the User Experience Questionnaire (UEQ, Laugwitz et al. 2008) was used to evaluate usability and satisfaction. These questionnaires were sent to the participants in advance by email together with the prototype.
As part of the survey, evaluations from a total of 15 male test subjects could be recorded. The mean age was 50.8 years (SD = 9.8). Furthermore, the users had an experience of 17.7 years (SD = 12.0) for the control center area, 23.6 years for the helicopter pilot area (SD = 10.4) and an experience of 12.1 for the skipper area Years (SD = 7.8). The sample shows a high affinity for technology, which was recorded using the TA-EG questionnaire (Karrer et al. 2009) and is shown in Fig. 6.
Since a static 3D prototype was used, the subscales transparency, efficiency and predictability were dispensed with in the UEQ and only the subscales attractiveness, stimulation and originality were surveyed. As part of the evaluation, values that are below -0.8 or exceed +0.8 are to be classified as meaningful (see blue lines in Fig. 7, Laugwitz et al. 2008). The results are summarized in Fig. 7.
The results of the UEQ assessments match the comments made by the users in the semi-structured interviews. While skipper and helicopter pilot workplaces were rated particularly positively apart from their originality, the control center workplace achieved a neutral rating on average. In the evaluation of the interviews it was found that the limited space to work with pen and paper and the insufficient number of displays were viewed critically. Notes on changing the position or alignment of individual objects have been included for the other workplaces. It was also possible to include information on assembly and dismantling, such as the introduction of color coding and poka-yoke elements. Fig. 8 shows the result of the control center workstation that was created using the user-centered development process.
In contrast to the control center workstation, in addition to communication requirements (communication), there are also requirements for navigation and the command unit (conning) for the other workstations. A special feature of the design of the helicopter workstation was the contradiction between the high-end flight simulator and the mobile multifunctional simulator. The compromise reached between the development partners for an abstract and little detailed, but functional implementation was implemented with correspondingly doubled elements and as few physical control elements as possible.
Ultimately, a conning unit (module shipmaster) was specially designed for the skipper's workstation. This includes the skipper's specific controls for navigation tasks. As a separate module, it is not designed to be multifunctional, as the control elements exclude functionalities that can be used repeatedly. Fig. 8 shows a general overview for the workplace.
In addition to the hardware requirements, the arrangement of the software components is also of interest. An arrangement of the applications was proposed within the framework of the 3D PDFs, which was accepted by the users and which corresponds to the requirements. However, the wish for an individual placement of the elements was expressed several times. This should be implemented for the workplace of the control center employees and skippers. An independent arrangement of the applications is not recommended for the workstation of the helicopter pilots, as this is based on the existing systems and is therefore standardized. In addition, the number of display elements is significantly higher, which is why the dialog design criterion of error tolerance would be significantly lower compared to the other systems.
Implications, Limitations and Outlook
The contribution presented the ergonomic design and further development of the MOS with the help of the user-centered development process (ISO 2010). Based on focus groups with future users, an initial design could be drawn up that served as the basis for further development work. After the final user evaluation, the final prototype was created, which fulfilled the requirements for a compact and reduced hardware design and also demonstrated a high level of usability.
3.1. Content implications
The requirements analysis showed that the needs of the three future user groups vary widely depending on the activity to be carried out. Overlaps in all three user groups could only be determined in the area of communication. In the navigation domain, there are similar requirements between the two groups of skippers and helicopter pilots.
In particular, in the area of the displays and actuators to be integrated for control (conning), major differences have been listed (see Fig. 4 and Fig. 8). This confirmed the development goal of a modular design. By developing different modules for the central area of the MOS, it was possible to meet the various requirements.
The final evaluation showed that the prototypes in the subscales attractiveness, stimulation and originality of the UEQ provide a satisfactory result and thus represent a good basis for further developments (Laugwitz et al. 2008).
Cross-organizational training scenarios can be implemented after the presented development of a MOS with simple, modular and networked simulators, which confirms the assumption of Riener (2010). If the focus of the training is on collaboration and communication during a major maritime disaster, existing studies (Wright and Madey 2008) confirm that simulators that are tailored to a specific application can be dispensed with.
3.2 Methodological implications
The combination of quantitative and qualitative methods proved to be suitable for both test subjects and researchers and thus confirms or enriches existing knowledge (Li and Earnest 2015). The mix of methods helped both the scientists involved, some of whom had little background knowledge in the field of application, and the participants who only had rudimentary knowledge of mobile and ergonomically designed simulators.
The use of focus groups during the first phase of the user-centered development process in this application context has proven to be particularly suitable. The possibility of moderating and steering the discussion as well as direct interaction with the test subjects also allowed deep insights into the different operational domains of the MOS and their specific problem areas within complex damage situations at sea.
The final paper prototyping process provided a very broad basis for initial design solutions from MOS. This approach is recommended above all for product developers who move in a new application domain and have little knowledge of the object under investigation and who want to generate initial ideas at the same time (Töpfer and Silbermann 2008). The development of the paper prototypes for the MOS confirmed this recommendation.
The paper prototypes served as the basis for the first virtual prototypes, which could then be improved in two further development loops. The methodical combination of sending 3D prototypes, which could be viewed in advance by the test persons, and the subsequent questionnaire examination and interview application, was found to be particularly suitable for evaluations in which test persons and test management are located at a great distance or joint workshops are very difficult to carry out for organizational reasons.
3.3 Limitations and Outlook
It is also true for MOS that a product that was developed on the basis of the UCD does not guarantee “success” (Ritter et al. 2014). A high usability of the MOS does not automatically lead to high learning success because it depends on many different factors (see 2.3.1). In addition, a high degree of usability at the micro level does not ensure an exact fit with all of the display and control elements available on different ships and helicopters. In spite of the higher usability of the simulated environment, this can lead to operator errors due to the expected conformity (Baxter et al. 2005).
The final (virtual) prototypes developed in the course of this development process were converted into real MOS after completion. In order to examine their degree of ergonomics and expected handling, they should be the subject of volunteer studies. The focus of these studies should be the interaction with the MOS and its effect on people, since the usability of the display and control elements could not be examined as part of the evaluation of the digital prototypes. In addition, such a test serves to check the modularizability of the MOS in order to exclude an application-specific development and to ensure a high generalizability (McLoone et al. 2010). At the same time, checking the realism of the online prototype evaluation appears to be useful in order to validate the selection of the work tasks and situation features and to derive cross-domain statements on the use of online evaluations (Scholtz 2004). Furthermore, the online evaluation was unable to examine how efficiently the communication between the participants is simulated in the MOS. This must be done in the evaluation of the real MOS. Finally, learning success measurements should be carried out and linked to experiences from real exercises in order to further validate the effectiveness of a simulator exercise.
Bailey, B .; Biehl, J .; Cook, D .; Metcalf, H .: Adapting paper prototyping for designing user interfaces for multiple display environments. In: Personal and Ubiquitous Computing, 12 (3), 269-277, 2007
Baxter GD, Monk AF, Tan K, Dear PRF, Newell SJ (2005) Using cognitive task analysis to facilitate the integration of decision support systems into the neonatal intensive care unit. Artif Intell Med 35: 243-257
Article Google Scholar
Bevan N (2009) International standards for usability should be more widely used. J Usabil Stud 4: 106-113
Blackmon MH (2004) Cognitive walkthrough. In Bainbridge WS (Ed) Encyclopedia of Human-Computer Interaction, Vol. 1. Berkshire, Great Barrington, pp. 104-107
Bogers M, Afuah A, Bastian B (2010) Users as innovators: A review, critique, and future research directions. J Management 36 (4): 857-875
Buchenau S (2020) Fire brigade operations in ports, on and near the water - The biggest ship disasters in the new. Millennium In: Feuerwehr-magazin Ebner Media Group Gmbh Co Kg.https: //www.feuerwehrmagazin.de/wissen/die-groessten-schiffskatastrophen-im-neuen-jahrtausend-78709. Accessed: April 12, 2020
Chammas A, Quaresma M, Mont’Alvão C (2015) A Closer Look on the User Centered Design. Procedia Manuf 3: 5397-5404
Article Google Scholar
Creswell JW (2009) Research Design. Qualitative, quantitative, and mixed methods approaches, 3rd ed. SAGE, Thousand Oaks
Ecker M (2016) Usability and usability engineering for the design of learning systems - user-centered methods for the sustainable development of digital educational offers. Technical report 1/2015, Weingarten University of Education, pp. 1–38
Flin RH, O'Connor P, Crichton M (2008) Safety at the sharp end: A guide to non-technical skills. CRC, Taylor & Francis Group, Boca Raton
Hofinger G (2008) Team training for crisis management. In: Buerschaper, C .; Starke, S .: Team and leadership in critical situations. Publishing house for police science, Frankfurt am Main, pp. 190–205
International Organization for Standardization: Ergonomics of human-system interaction — Part 210: Human-centered design for interactive systems, 2010
Kahraman, Z. E. H .: Using user-centered design approach in course design. Procedia — Social and Behavioral Sciences, 2 (2), 2071-2076, 2010
Karrer K, Glaser C, Clemens C, Bruder C (2009) Assessing affinity for technology - the TA-EG questionnaire. The focus is on the human being technical. Systems 8: 196-201
Kleinhenz S (2011) The office workplace: manual for the design of workplaces in the office. Haefner, Heidelberg
Konstantin F (2010) Sea Rescue: A woman is in charge. In: Flensburger Tageblatt. https://www.shz.de/lokales/flensburger-tageblatt/seenotrettung-eine-frau-hat-das-sagen-id2442341.html. Accessed: March 20, 2019
Langford J, McDonagh D (2003) Focus Groups: Supporting effective product development. Taylor & Francis, London
Book Google Scholar
Laugwitz B, Held T, Schrepp M (2008) Construction and evaluation of a user experience questionnaire. In: Symposium of the Austrian HCI and Usability Engineering Grou. Springer, Berlin, Heidelberg, pp. 63-76
Li J, Earnest J (2015) The best of both worlds - advantages of a combination of qualitative and quantitative. Research methods In: Wzb Mitteilungen 150: 30–33
Lübcke T, Steigenberger N, Bornhorst C, Roßner P, Kemna J (2015) From Ship Handling to Maritime Incident Response Simulation — Advanced Networked Simulation. MARSIM. conference, vol. 2015. Newcastle, UK
Maguire M (2001) Methods to support human-centered design. Int J Human-computer Stud 55: 587-634
Article Google Scholar
McLoone E, Jacobson M, Hegg C, Johnson PW (2010) User-centered design and evaluation of a next generation fixed-split ergonomic keyboard. Work 37 (4): 445-456
Article Google Scholar
Molka-Danielsen, J .; Prasolova-Førland, E .; Fominykh, M .; Lamb, K .: Use of a Collaborative Virtual Reality Simulation for Multi-Professional Training in Emergency Management Communications. 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), Wollongong, NSW, 2018, 408–415, 2018
Möslein KM, Halle JBA, Bullinger AC (2010) Open Evaluation: IT for innovation management. Business informatics manag. https://doi.org/10.1007/BF03248287
Article Google Scholar
Nielsen, J .: Enhancing the explanatory power of usability heuristics. Proc. ACM CHI’94 Conf., Boston, MA, April 24-28, 152-158, 1994
Richter M, Flückiger MD (2013) Usability engineering compact: specifically developing usable products. Springer, Berlin
Book Google Scholar
Riener A (2010) Simulating On-the-road Behavior Using Driving Simulators. In: Proceedings or the 3rd International Conference on Advances in Computer-Human Interactions. IEEE Computer Society Press, Saint Maarten
Ritter FE, Baxter GD, Churchill EF (2014) Introducing User-Centered Systems Design. In: Foundations for Designing User-Centered Systems. Springer, London, pp. 3-31
Book Google Scholar
Rosenkötter S (2015) Fehmarn - The drama about the Lisco Gloria. https://www.ln-online.de/Lokales/Ostholstein/Das-Drama-um-die-Lisco-Gloria. Accessed: July 16, 2019
Salas E, Wilson KA, Burke CS, Wightman DC (2006) Does Crew Resource Management Training Work? An Update, and Extension, and Some Critical Needs. Hum Factors 48 (2): 392-412
- What is the Ophelia Complex
- Supports Shopify reminders
- How is the GATE BOOK test series
- Can you die from a tick bite?
- What is the best way to learn pranayama
- What is my self-esteem worth
- Medical equipment suppliers
- Innovative ideas for new jobs
- Is there a minus mark on NCHMCT
- How can I boot my company
- Quebec is an extremely left wing place
- What is a Kronos Developer
- What's good about Google+
- Can you actually scare someone to death
- When is the normal Zeeman effect observed?
- You should save VHS tapes
- Which entrance exams should I prepare for?
- Should I eat chickpeas before training?
- What is Euglena classified as
- What problem does this algorithm solve
- What is the decision making process in Facebook
- What is the meaning of the exam
- How is the CSU Fresno CS department
- What is the importance of conductor insulation