Evaluation of a Role Play Simulation in Political Science

Go to original draft with critical reviews.
Go to revision #1

Though simulations are often used in teaching political science they are rarely used as forms of assessment. Over the last few years Dr. Peter Shearman and Mr. Roni Linser decided to offer students in a second and third year "World Politics in Transition" course at the University of Melbourne simulations as an alternative assessment of their learning in this course. The decision to offer such an alternative arose out of a need to exemplify to students the strengths and weaknesses of International Relations theories which were the focus of the course and to enliven what might otherwise seem 'dry theory'. It was predicated on a pedagogy which asserts that long term retention and ability to use learning are better achieved through learning from experience. Prior to 1997 this course relied on the traditional approach of lectures, tutorials and individual research by students. Students were assessed on the basis of a 2,000 word essay (50%) and a 2-hour exam (50%) at the end of the course. In 1999, when students were given the alternative of doing a goal-based role play simulation in lieu of the final examination, 86 out of the 120 students in the course chose the simulation. The purpose of this paper is to describe the simulation that we used to assess student learning in this course and what students thought of this alternative after they had experienced the simulation..

This learning experience is based on four essential ingredients (Linser et. al, 1999):

goal-based learning which is acknowledged as a strong motivator of learning. Typically, goal-based learning comprises a scenario or context, which includes a trigger or a precipitating event. Our goals are "dynamic" in the sense that the students set the game goals themselves during the "role profile" write-up.
role-play, both in the sense of playing a role, playing with possibilities and alternative worlds, and playing to "have fun". By playing a role (in all these senses) students start to identify with the role which gives them a personal stake in the proceeding and thus enhances their motivation to do well. Of course, in order to play the roles effectively students need to do research. Data for this research is available via a large number of links on the role-play web site but it is also necessary for students to do traditional library research as well as attend lectures and tutorials. The provision of resources by this mechanism serves to simplify the simulation generator software in that no elaborated schema is necessary to classify the resources to provide "resources on-demand".
the Web, which houses the virtual space for the role-play, enables communication and collaboration among students, and between the students and the lecturers. The Web also enables access to "just-in-time" resources by making available to students resources (such as up-to-date news from electronic newspapers and web-sites etc.), from all over the world as and when they need them. Without this capability the content of the role-play would be significantly weaker.
traditional teaching method of face-to-face lectures and tutorials. Many experienced on-line educators (Hedberg & Harper, 1998; Brown, 1998; Durham, 1998) have emphasized the importance of including face to face interaction in teaching with the aid of Computer Mediated Communication and online teaching. The importance of incorporating these techniques into the learning architecture is not only for their utility in the presentation of facts, cases and theories. They also provide communicative events that stimulate reflection about actions undertaken and strategies pursued by comparing real world events with the simulated ones.

Students were told in the course syllabus that the purpose of the simulation was "to provide students with a hands-on approach to World Politics. By playing roles of different world leaders, organisations and media outlets, students will experience some of the dilemmas and issues involved in World Politics and will be able to test for themselves the applicability of different theoretical positions involved in the analysis of international relations."

Software Used for Simulation Generation

The simulation was powered by a Role Play Simulation Generator called Fablusi (TM). The design is based on the abstraction that human interactions are communicative events requiring information exchange (Ip et. al. 1999). Via a form-base interface, Fablusi provides a skeleton to

determine the roles in the simulation,
write up a scenario by providing information to all roles and additional specific information to some roles in order to create an information gap
design specific forums (modeled by conferences) for some of the roles to simulate real world organizations,
setup the social structure for the roles by setting the rights in the conferences, and
optionally create tasks to scaffold the roles into the simulation.

After students log into the system, each student takes on a role as specified by the creator during the simulation creation process. Fablusi supports the communication of the roles using Sim-mail and Sim-conference. A NotePad is also automatically created for each role. Sim-mail is similar to a web-based email system; however, the Sim-mail system is totally isolated from the real world such that roles can only send mail to other roles. Sim-mail also allows the lecturers to view ALL mail in order to monitor the progress of the simulation and provide support to roles. Sim conference is similar to most other generic conferences. However, the participation rights of roles in different conferences can be individually set to reflect the real life relationship of the roles. Furthermore, Sim-conference supports multiple document types in order to simulate different positions within an organization - for example a News Agency with the positions of journalists, editors and readers or an international forum with participants and a chairperson. The running of the simulation is also supported by tasks set up by the creator and the proceedings closely monitored by tutors.

Each role was played by a group of 2-4 students who together researched that role prior to the start of the simulation and then wrote up a "role profile". Fablusi's "formal writing task" was used to allow the students to write, submit and publish these profiles to the other roles automatically. In order to be able to play a role eac h team had to become familiar with objectives and strategies used by the role and those of others. Each role-profile functioned as a case-study for the evaluation of the theories studied in the course. By 'acting-out' the role, students were able to assess for themselves, by reflecting on their own actions, the extent to which different theories enable better explanations for the actions of that particular role. The roles were played in teams to enable students to 'bounce' ideas of each other... (To see the profiles, please click here and log in using username guest and password demonstration. Click the [Read Profile] button. The object of the simulation for the students was to reach the objectives they formulated in their role profile and to enhance the position of their role in the simulation. They could do so by contacting relevant players through the mail system and the Sim-conferences, utilising diplomacy, threat, coercion etc., as warranted by the specific conditions of their role (i.e. playing in character). The simulation began with a scenario to which students in different roles reacted. The scenario, written up by the lecturers in-charge, was developed in accordance with real world events adding fictional (or potentially plausible future "what if...") events. In principle it is a review by the lecturer of the current situation with a few plausible fictional developments. Students then attempt to confront these developments from the perspective, objective and strategy of their role and they may do so by creating new developments (which they assess as potentially viable given their research) to solve the initial problems. In 1999, the simulation was set around the Kosovo Crisis highlighting the role of the UN.

This was a fairly large simulation consisting of 37 team roles. The simulation was played out for three weeks (set at three weeks ahead of real time, or, in game parlance, simulation time). In these three weeks, the students sent out over 3,000 messages with an average of 7 recipients per message. Figure 1 showed the read mail screen with a sim-mail from Bill Clinton.

Student Assessment of Simulation Effectiveness

The perceived effectiveness of the simulation was found to be very high by the students. An end-of term questionnaire was administered to the students to seek feedback on their learning experience in using the role-play simulation. Thirty-eight out of the 86 students who participated in the simulation responded to the questionnaire The perceived effectiveness of the simulation in enabling students to achieve the specified learning outcomes was found to be generally high by the students as reported in Naidu et. al. [2000] Table 1 shows the questions asked and the average of the responses on a scale of 1 to 5 where 1 is "very useful" and 5 is "useless."

On this scale, the average rating for questions related to understanding, identifying facts, issues, problems and factors, was between 1.34 to 1.92 which is in the "very useful" to "useful" region.

It is interesting to note that on the question that the simulation added "active and dynamic dimensions to classroom learning processes," the average was an overwhelming, 1.08 indicating that the students agree strongly with the statement about the new innovative learning environment.

Table 2 shows how the simulation was perceived by the students as an instrument for learning in a scale of 4 where 1 is "strongly disagree" and 4 is "strongly agree". A 4-point scale is used (instead of the 5 point scale above) to force the students to make a judgment about the usefulness of the simulation.

It was again overwhelming to find the students "agree" to "strongly agree" in the usefulness of the simulation in such categories as helpful in developing understanding, identifying issues and problems gaining knowledge and evaluating strategies which they learned in the theory class. Students were asked to write a position paper in preparation for a face-to-face conference to 'roundup' the simulation experience. The evaluation of the score about writing of "the position papers" was 2.95 showing that the students thought the simulation was very effective in helping them to understand the subject matter.

The students spent a lot of time doing the simulation as shown in the following diagram. There is significant difference between those with Internet access at home (item 36) and those without. Those with Internet access at home spent significantly more time on the simulation. See table 3 .

Evaluation of Using Role Simulations in Instruction

The utility of the simulation as a whole can be gleaned from the students' responses to free text questions.

Question. Would the use of a Web-based simulation such as this influence your decision to take a course, if so why? If not why?

The students responded positively towards this question. The effectiveness of the learning experience has been reported, together with the element of "fun" and "getting to know and interact with a wider range of students within the subject". Students acknowledged that this exercise had helped them to understand and tackle complicated issues and achieve deeper learning. However, some students had reservation due to the increased amount of time they had spent on the simulation. The data from the option items indicate that some students spent over 15 hours a week during the 3 weeks of simulation. See some students responses.

Question. The mail system in the simulator is NOT a real email system. Please state the advantage(s) if any?

Unlike simulation based on generic mail system, the mail subsystem in the role play simulation generator is fully integrated into the simulation and completely separate from real world mail system and all mails are sent in the name of the role. We believe that it is important to clearly distinguish between a simulated world and the real world. A student recognised this and said,"Kept it separate from my other email so when I entered the world of the simulation, it was a distinct and separate world". See some student responses.

Question. We would like to get your comments on any aspect of the simulation that may help us in making it a better experience for students.

This question attempted to solicit the overall experience from the students. We have received mixed responses to the system. Some felt that the site was fast, others found the site intolerably slow. Sample Students responses.

Conclusion

The role play simulation reported in this paper is a proven learning architecture that has a fairly long history of use in political science and other subject domains. Students report that the recent conversion of such a learning environment into a Web-mediated environment has been a positive innovation. More work is needed to find out whether real benefit of learning outcome has been achieved. We are optimistic even if no significant improvement of learning has been achieved, the increase of "fun" and "playfulness" which enhance student motivation justifies the effort we have put in.

The simulation being used in this exercise is generated by Fablusi (TM). The software was evolving during the course of delivery of the exercise reported here. It seemed that the students did not worry about the technology and were deeply engaged in the activity.

The quality of the simulation and the effectiveness of using a simulation as a learning vehicle are the result of the creativity of the simulation creator. The role of the generator was to provide a skeleton and reduce the amount of work done by the lecturer in creating and running a simulation. However, in the exercise reported, the effort by the role play coordinator was only somewhat reduced. This outcome was mostly a result of the novelty of the technology and consequently the necessity for testing and re-testing and ironing bugs out of the current system. As a result of the feedback from this exercise, the role structure has been enhanced to allow better monitoring of the simulation activities by lecturers/tutors. A private real-time chat room has been added. During two other simulations in April and May, 2000, the performance of the system was demonstratively improved after moving the software to run on a more decent hardware. It is expected that future simulation creations will take significantly less time as the bugs in the systems are generally removed and we gain more experience in creating a more effective delivery system. We hope to see more academics taking up such an approach. Readers are welcome to contact Albert Ip to try out of Fablusi(TM) or Roni Linser about simulation design.

[Footnote -- but linked to what?] Some Web-based role play simulations

International Communication and Negotiation Simulations (ICONS) Project

This is Project at the University of Maryland which offers opportunities for college and university students to participate in web-based simulations of international relations. Their aim was to capture more of the complexity and subtlety of international political issues through the use of fairly detailed scenarios, which focused on real or plausible policy problems. The earliest trials of computer-mediated simulation exercise were performed by Robert C. Noel of the University of California at Santa Barbara. ICONS was based upon Noel's simulations, and was founded in the early 1980s at the University of Maryland by Jonathan Wilkenfeld of the Department of Government and Politics and Richard Brecht of the Department of Germanic and Slavic Languages. By 1995, the World Wide Web and related technologies made it possible to provide users with an easy-to-use interface and minimal local computer requirements. ICONSnet (the software on which ICON runs) is written in Oracle PL/SQL and runs on Oracle Database and Oracle Application Server. A scenario, which launches the exercise, outlines the state of the world based on present-day facts, and sets the stage for the interactions both within and among country-teams. The simulation includes both the asynchronous exchange of diplomatic communications and computer-assisted real-time conferences. Conferences are scheduled to focus on each of the issues in the simulation. They follow a detailed agenda and are chaired by ICONS staff. The ICON software (ICONSnet) supports both real time and asynchronous message exchange, team anonymity and possibility of language translation.

Politics and International Relations in the modern Middle East at Macquarie University, Australia

The stated aim of the simulations is threefold:

to introduce students to the facts of Middle East politics (that is, make them familiar with the major countries, leaders, groups, movements and relationships in the region)
to give them experience with the complexities of negotiation and decision-making in "real" political systems
to improve their skills in using computer technology and and the Internet as tools for the workplace.

The aim of the simulation technology has been to give students access to the Internet to facilitate their simulation communication but with minimal learning effort. They identified essentially five kinds of communication required by simulation participants:

person-to-person interactive dialogue (synchronous)
person-to-person communiques (asynchronous)
small group dialogue (synchronous)
communiques limited to a small group (asynchronous)
newspaper-style media services (asynchronous)

The technology developed for these simulations was influenced primarily by John Shepherd's experience with the Unix operating system, and by constraints on the amount of time available for software development. The system was largely a collection of appropriate Unix tool and customising that tool. The major difficulties were: the use of an editor to create email messages, and relating role names to "raw" email addresses.

References

Brown, M. (1998). New teacher for a new age: The myths and realities of the global classroom. Flexible learning: Proceedings of the Apple University consortium conference at the University of Melbourne 27-30 September 1998. [are these proceedings published? if yes, we need page numbers of article, publication city and publisher name.]

Durham, M. (1998). Working at virtual records—a simulated workplace. Flexible learning: Proceedings of the Apple University consortium conference at the University of Melbourne 27-30 September 1998. [ditto]

Hedberg, J., & Harper, B. (1998). Supporting flexible thinking with interactive multimedia. Flexible learning: Proceedings of the Apple University consortium conference at the University of Melbourne 27-30 September 1998. [ditto]

Ip, A., & Linser, R. (1999). Web-based simulation generator: Empowering teaching and learning media in political science. Paper presented at the Australasian Society for Computers in Learning in Tertiary Education conference, Brisbane, Queensland, Australia. Retrieved 25 May 2000 from the World Wide Web: http://www.roleplaysim.org/papers/rpsg.htm

Linser, R., Naidu, S., & Ip, A. (1999). Pedagogical foundations of Web-based simulations in political science. In J. Winn (Ed.), Proceedings of the Australasian Society for Computers in Learning in Tertiary Education (pp. 191-198). [Need city of publication and publisher]

Naidu, S., Ip, A., & Linser, R. (2000). Dynamic goal-based role-play simulation on the Web: A case study. Manuscript submitted for publication. Retrieved 25 May 2000 from the World Wide Web: http://www.roleplaysim.org/papers/Naidu_etal.html