PAN: Proctored Assessment Network

Several years of involvement with online education and its attendant issues lead the authors to conclude that there is no failsafe method on the horizon to ensure security when assessment is online and unproctored. The need for security will vary with the course, and some instructors believe that making the test available for a limited amount of time provides sufficient security. Constraining the amount of time students are given to complete tests and quizzes does offer a measure of security, but at the expense of disadvantaging individuals who simply take longer to complete them. High technology solutions such as iris or thumbprint recognition also seem out of reach for most instructors and students given the expense of such hardware. Even with these efforts to resolve the problem, there still is no guarantee that the person completing the test or quiz is the correct individual or the only individual sitting at that remote computer. Many lower level, high enrollment courses have security needs that are not met without a valid proctor and accurate identification of the student(s).

We recently proposed – at the Distance Education Workshop 2000, in Raleigh, North Carolina – the development of a proctored assessment network (PAN) in North Carolina to deal with this problem. Our guiding assumption has been that a “lowest common denominator” approach is essential for PAN to work. That is, it must be very simple to use and very inexpensive in terms of time, space, and financial demands.

North Carolina has 16 senior public institutions, 59 community colleges, nine Area Health Education Centers, and 100 NC Cooperative Extension Service County Extension Centers (one in each NC county). In short, there is a publicly funded facility of at least one of these types nearly within shouting distance of every North Carolina resident. In terms of organization, facilities, and personnel, we’re confident that it is quite feasible to create at least a minimal cooperative agreement wherein most-to-all of those sites could devote a facility – or at least some space within a facility – as a PAN Site for at least a couple of hours during the day and a couple of hours during the evening on one day of each week during academic terms. We would consider the PAN a success if each student living in North Carolina were able to take tests in online courses without having to drive more than about 30 minutes from home to a PAN Site.

The facility at each PAN Site should be a room – or at least an area within a room – that is reasonably quiet and, preferably, has a sufficient number of computers and reasonably fast Internet capability to accommodate online test taking. PAN Sites that are not (yet) computer and Internet equipped must be prepared to at least provide services enabling receipt, distribution, retrieval, and return of paper copy tests and quizzes. Of course, adequate and accessible parking facilities must be available for students to find and use at each PAN Site.

Students enrolled in online courses which use the PAN also have very few PAN-related roles to perform. These include:

A fifth role that would apply to some proctors – where possible – is actually monitoring students as they complete their tests. This would be feasible at some – but not all – PAN Sites.

PAN Web site and Web forms The Proctored Assessment Network we have proposed may be seen at the UNCW PAN web site. The web site includes information similar to the points made above, but also the PAN Site Finder web page, the registration web form to be used by any institution that would like to become a PAN Site, and the web form to be used by any participating PAN Site instructor to request that a test for an online course be proctored. Perusal of this site by interested readers is welcome, as are comments, observations, and, of course, applications! The PAN mailto address also is shown on the PAN Web site homepage: pan@uncwil.edu.

We understand that there are issues potentially limiting development of the PAN. These seem to center upon two questions:

The first question, when applied to institutions that deliver online courses, really asks: what's in it for me, why should my institution make it easier for your students to take online courses? The answer, of course, is that the PAN would and must function on the basis of reciprocity. It seems reasonable to assume that, reciprocity given, the lure of becoming a PAN Site should be directly related to the magnitude with which a given institution itself offers online courses. When applied to institutions that would function only as PAN Sites hosting remote testing (i.e., do not actually deliver their own online courses), reciprocity is limited to recognition of the service offered as part of its mission, or perhaps in the form of actual monetary compensation.

There also are various reasons to ask if certain institutions (and, therefore, their own online course students) should be denied PAN membership. For example, should unaccredited institutions be admitted to the PAN, should PAN services be extended to noncredit course students at member PAN Sites, should students at private institutions be given access to taxpayer underwritten facilities and equipment at public institutions, etc.?

It is, of course, difficult to anticipate the nature and extent of all factors serving to bolster or inhibit the viability of new concepts. Rather, we conclude simply by restating our certainty that the PAN service described above is integral to distance education and, therefore, growth and refinement of online courses suffer from the lack of this service. Our attention has been upon the description and promotion of the PAN concept in North Carolina, but we think that it is a globally scalable concept (PAN Maryland, PAN Canada, PAN Europe, . . . ) once a functioning model is in place.

As a piece of plain, expository writing this article works. It is clear what they want to do - complex and depending on many different entities and a lot of 'good will' (which may be scarce among competing institutions of higher ed.) They come up with some justification for use of the system that is somewhat valid, within the present paradigm.

Where is the "commentary" in this piece except perhaps obliquely commenting on dogged adherence "business as usual" using a classic Rube Goldberg? Perhaps it belongs in another section...technology reports, perhaps?

But if they are going to use this patchwork system, why are they planning to deal with shuffling papers at all? Phone lines are ubiquitous, and the scanner/fax/laser printer combinations are becoming ever more sophisticated. Why not equip each potential site with one? Whoever is responsible for shuffling and tracking paper to and from faculty examinations at an institution could then scan and fax or just fax exams to each site, and with a laser printer at the other end, the printouts could be faxed back. Drawings or diagrams could be scanned and faxed.

And what a nightmare to synchronize all the computers in a system like the one they propose - same speed processors, same amount of RAM, same software, same versions of software, same connection speeds, etc. across the entire system. That's the only way to guarantee that all students in all locations will have an equal opportunity to click on their little radio buttons and check their little boxes providing all the Internet connections are up at the same time.

At my last place of employment I witnessed the efforts necessary to standardize computers across sites that were owned by the university and staffed by university employees. Even when computers were delivered to sites they sometimes sat in their boxes for months because another entity in the University had to download and install the software, or there was no hookup to the university’s state wide LAN.

A system as complex as this will only be as strong as its weakest link - which predictably will fail, often. The same kind of effort in other settings has been accomplished by contracting with Sylvan Learning Systems.

But aren't there infinitely better ways of assessing student learning besides timed, proctored tests? Systems like this are costly among several dimensions to implement and when that money/time/effort is spent, what do you have? You have a system that locks its users into an antiquated paradigm and does nothing to encourage changes in assessment models. Back we go to point estimates of massive memorization efforts :-(

The problem addressed by this article is certainly one of concern not only for the education industry but also for commercial entities as evidenced by discussions regarding electronic signatures. However, there seem to be a few problems glossed over by the authors:

What about technical support at the PAN sites. Will the proctor automatically be trained in troubleshooting possible problems? Anything involving technology has a distressing tendency to stop working at the worst possible moment. How would institutions plan for this? The authors indicate "Administrators" at the host site would be responsible for this. Are they speaking of system administrators at the particular building/department/institution? Would technical support be available at the varied hours necessary to serve all class exam hours? How far in advance would exam schedules be known?

Where will proctors come from? The host site? Other sources? What about payment for services? The subject of possible "monetary compensation" is touched upon, but staff scheduling can be a sensitive subject at some institutions that may already be short of qualified personnel. The principal responsibility for the success of PAN rests upon the proctors but it is unclear how they will be selected.

The article seemed to suggest that a single site may offer a variety of tests at a variety of hours to a limited number of students. The suggestion that online testing can go unmonitored seems to defeat the notion of preventing students from using aids to complete the exams. Therefore an individual would have to be in the room with the student(s) being tested at all times.

That a single person may be tied up for most of a day offering exams to only two or three students per given class seems to create a drain on human resources. There is also a fair amount of coordination required by the faculty member either mailing/faxing hard copy tests to proctors (as many as 10 or more for an enrollment of 30 students?), or providing website addresses and instructions to a variety of proctors across a state or region.

Additionally, the only proctors suggested are librarians, who at most institutions can not be spared for the fairly simple supervisory task involved. On the contrary, student workers seem to be much more likely choices, both economically and for schedule flexibility, perhaps employed for the task by distance education departments at host institutions.

It seems feasible that as online course offerings expand there will be mutual interest in participating in a system such as PAN, but internal political concerns can be an issue. Administrators often delegate work to departments or divisions without prior consultation, forming an official policy that is not well handled at the point of service. This can cause dissatisfaction among the participants (students and faculty) that can cause failure of an otherwise good idea.

The point of denying PAN membership also raises interesting questions. It might be interesting to explore pros and cons in this area at greater length in the article.

One final (minor) point is that the writing is a bit wooden. This seems either like a grant proposal overview short on details, or an article to colleagues that is short of personal input.

In summary, I would recommend publication of this article, but I really feel it requires more detail of proctor selection, and the central coordination that may be necessary, as well as addressing the time factors that may be involved in setting up schedules for multiple locations.

“PAN: Proctored Assessment Network” deals with an important issue and contains useful information — so I’m confident that readers would welcome the chance to learn about the program. My only question concerning publication is the scope of the article.

Specifically, if it’s an article that’s only about PAN, then it might be reduced to a side-bar with fewer administrative details and, of course, a link to the PAN Web site. If we’re seeking a full-fledged article, there may be benefit in presenting a comparison of several methods of ensuring security for online assessment activities.

This is not a major concern, just an idea for consideration. My only other question relates to the short list of “participation issues” — which, surprisingly, did not list “costs” as a separate item.