The Role of Feedback in Interactive Learning Systems: A Comparative Analysis of Computer-Aided Assessment for Theoretical and Practical Courses

Abstract : High-fidelity e-Learning preserves the quality of the educational experience while minimizing or eliminating the need to be in a specific place at a specific time with a live instructor. The principles of this type of learning include personalization, multimodality, lean-forward learning, collaboration, accessibility, modularity, progress tracking, and balancing control. The SEI's Virtual Training Environment (VTE) is an example of a high-fidelity e-Learning success. The VTE was developed to draw upon the principles of high-fidelity e-learning to provide high-quality learning with infinite scalability. The purpose of this document is to describe the tenets of high-fidelity e-learning, to describe how VTE reflects these, and to summarize how organizations have used and are using VTE.


Computer-aided Communication and Collaboration Technologies
Three Types of Student Control [Chen & Liu 2008] Data for Comparison from ASTD Report

Executive Summary
This technical report explores high-fidelity e-learning, which we define as a set oftools and methods that delivers education and training to students anywhere, anytime, without sacrificing efficacy. High-fidelity e-learning preserves the quality ofthe educational experience while minimizing or eliminating the need to be in a specific place at a specific time with a live instructor.
The SEI's Virtual Training Environment described in this report has been developed to reflect the principles of high-fidelity e-learning for workforce development It offers the user a web-based, multimedia, interactive training experience that can be accessed anytime. This environment uses state-of-the-art software and hardware to train users with technical and managerial roles on a wide range of information assurance topics.
The intended audience for this report is responsible for building or buying e-learning programs for employees, customers, business partners, or suppliers. This report aids in evaluating alternatives and selecting solutions that need to scale and change to meet organizational awareness, training, and educational objectives. The report first describes the learning landscape, which ranges from instructor-led classroom training in a single physical location to "pure" e-Iearning where content is available anytime, anywhere, using the internet.
High-fidelity e-Iearning environments incorporate rich media technologies, immersive simulations, scenario-based instruction, and hands-on exercises that can be accessed by students on their terms and as their time permits. Drawing from extensive research, the report summarizes the principles of high-fidelity e-learning which include the following: • Personalization: using a casual, conversational style for spoken and written content • Multimodality: delivering training content using multiple sensory channels (read it, hear it, see it) • Lean-Forward Learning: using demonstrations, hands-on exercises, and self assessments to increase retention and recall (do it, master it) • Collaboration: providing technologies that allow students to engage with one another and with instructors • Accessibility: making training content broadly available to people with a range of abilities • Modularity: publishing content in manageable, self-contained chunks so that it can be easily reused by content owners and referenced by students • Progress Tracking: supporting course providers, instructors, and students with the ability to track student progress • Balancing Control: balancing control between the e-Iearning system and the student to achieve optimal learning vii I CMU/SEI- TR-2009-005 In large part, these high-fidelity e-learning principles are re~ected in the SEI' s Virtual Training Environment VTE (http:lMe.cm.org)waslaunchedin 20[14 to irrvrove the scalability oftechnical training. V IT holds over 1,800 unique modules, representing 740 hours of training, and cur-renUy delivers over 10,000 hours of training a month in ,,-,pport of U.S DoD and civilian government agencies and the general public Course material on VTE is >WJctured to allow students to progress at their Oml

Introduction
Since its inception in 1984, the Software Engineering Institute (SEI) has been leveraging distance education. Initially, the SEI recorded lectures from its software engineering curriculum to videotape and mailed them to remote students and offices. As standards for video on computers emerged, the SEI shifted to distributing CD and DVD-ROM versions of recorded lectures. As the SEI's distance and e-learning offerings evolved, best practices began to emerge. Each iteration of the e-learning environment built upon the lessons learned from organizations and students using the environment.
This introductory section describes the intended audience and desired outcomes for this technical report. It then contrasts e-leaming with other modes of instruction and sets the stage for describing the principles of high-fidelity e-learning (Section 2) and how they are reflected in the SEI's Virtual Training Environment (VTE) (Section 3). These principles can serve as criteria for evaluating candidate e-learning environments that are being considered to satisfy organizational education and training requirements. Section 3 includes a description of the VTE architecture and data on VTE use to date.
The final section ofthe report presents two perspectives for using VTE: that ofthe producer or author of new VTE content (including lectures, demonstrations, and hands-on labs) and that of a typical user pursuing an information security certification. Appendix A describes the U. S. Department of Defense's (DoD) and DISA's experiences using VTE to meet US. DoD directive requirements for information assurance workforce competence as well as a return-on-investment analysis compared to benchmark data from the American Society of Training and Development. Appendix B contains an excerpt from a VTE pilot study conducted by the U S. Secret Service.

Target Audience
The audience for this report is any individual or member of an organization charged with acquiring, developing, or deploying e-Iearning programs within and across organizational boundaries.

Desired Outcomes
After reviewing this report, readers will better understand the concept of high-fidelity e-learning as it relates to organizational training initiatives, and will be familiar with the SEI's Virtual Training Environment as an implementation of a high-fidelity e-Ieaming system. This report does not cover concepts of pedagogy or instructional design in e-Ieaming or any other medium, except as they influence the delivery model for training. That said, we do recognize that poor-quality training material can undermine a student's ability to meet objectives regardless of the delivery environment.

e-Learning, Classroom Learning, and Blended Learning
Learning models are frequently described as points on a spectrum. As depicted in Figure 1, at one extreme is instructor-led training (ILT, also referred to as classroom training or direct instruction). At the other end ofthe spectrum is a family oftechnology-based models including computer-based training (CBT) and web-based training (WE1). Collectively, models at this end are grouped into the e-Iearning category. Blended learning occurs between the two extremes, which is a combination of direct instruction and e-Iearning. Figure 1 Spectrum of Learning/Training Models 'Nhile classroom instruction as an educational method is fairly well understood, definitions for terms such as e-Iearning and blended learning are still the subject of debate. In this report, we attempt to avoid the philosophical discussion inherent in these terms and instead map training models based on their logistical requirements. Different types of training methods may require that teacher and student be in the same physical location (physical vs. virtual) and/or require they meet at the same time (synchronous vs. asynchronous). Traditional classroom instruction assumes the instructor and student are in the same location at the same time. Virtual classrooms use technology to extend the synchronous classroom experience to any location. CB T, training Iibraries, and other non-networked training materials are available only in specific locations but can be used at any time. 'Pure e-Iearning assumes that self-paced materials are readily available. Blended learning combines several ofthese approaches, using asynchronous and virtual training methods with classroom instruction as the anchor. As an expansion to the spectrum presented in Figure 1, Figure 2 plots the different training models on a grid (note that the grid excludes the categories of non-computer technology and other).

I Single location (physical)
true or-led Tra-n-9 Any location (virtual)

Dimensions
According to the American Society of Training and Development (ASlD) 2007 member survey [ ASTD 2007], self-paced asynchronous e-Iearning is the fastest growing training model, coming primarily at the expense of classroom instruction. The percentages in the grid represent the portions ofthe overall professional training market served by each type.
Another factor that varies by type of instruction is the cost to provide the education. Cost in this case is a combination of direct costs for facilities, technology, and transportation as well as opportunity costs for instructors and students. The logistical requirements of synchronous training in a specific location make it the most costly method. Synchronous virtual training and asynchronous physical training each offer some cost savings over classroom instruction, but pure e-Iearning' s anywhere, anytime availability gives it a decided cost advantage over all other forms. Additional information on the cost advantages of e-Iearning is provided in Appendix A.

High-Fidelity E-Learning
This report focuses on pure e-Ieaming. Though we recognize some situations work best in a synchronous environment, given the cost and logistical savings associated with asynchronous, selfpaced e-Iearning, this report focuses on this method of instruction. Furthermore, this report elevates the concept of e-leaming to the distinction high-jidelity e-Iearning We define high-fidelity e-leaming as a set oftools and methods that delivers education and training to students anywhere, anytime without sacrificing efficacy. High-fidelity e-leaming preserves educational quality while minimizing or eliminating logistical requirements. High-fidelity elearning begins with the assumption that instructional delivery will be primarily asynchronous, so the content and its presentation must minimize the requirement for a live instructor.
The next section describes the foundational principles that characterize a high fidelity e-leaming environment.

Principles of High Fidelity e-Learning
High-fidelity e-learning extends the traditional concepts of e-learning through the use of rich media technologies, immersive simulations, and scenario-based instruction while maintaining its anywhere, anytime foundation. It is based on the presence of a ubiquitous internet connection, representing an always-connected world. It assumes students can and will access material on their terms and that they will acquire and maintain skills while they fulfill their day-to-day responsibilities. High-fidelity e-learning supports and encourages hands-on learning, allowing students to practice instructional concepts whenever and wherever they choose.
This section describes the eight foundational principles that characterize a high fidelity e-learning environment and describe the user experience.

Personalization
As summarized by Clark and Mayer, research has shown that e-learning is enhanced when a conversational style is used for both spoken and written text It also benefits from the use of onscreen pedagogical agents. This is the basis for Clark and Mayer's Personalization principle for designing media for e-learning [Clark & Mayer 2003].
E-learning designers and developers may be tempted to incorporate precise and fonnal text onscreen and in narration read by a professional voice talent. The personalization principle counters this temptation and reminds us that a casual, personal style of communication is more effective for learning. The challenge for e-Iearning is to replicate the effective and personal communications that often happen in engaging classroom environments.
Current technologies make video segments of classroom instruction easy to integrate with elearning courseware and therefore make the experience of the classroom and the instructor available not only to present material, but also to coach, work through examples, and guide the learning process in other ways. The use of video streaming in e-Iearning can make the classroom instructor available as the pedagogical agent.

Multimodality
Multimodality is defined as "the use oftwo or more ofthe five senses for the exchange of information" [Granstrom, House, & Karlsson 2002]. Multimodality is a broad, multidisciplinary area of research that includes the study of human-to-human spoken and gestural communications, audio-visual speech perception, and the incorporation of multiple modalities into human-tosystem communication. As Granstrom et al point out, multimodality is not new, "From the early storytelling tradition, where body and facial gestures have always played a major role, through classic theater, opera, dance, film, video and multimedia, multimodal communication has been constantly present in human communication and human culture." The multimodality principle for high-fidelity e-learning is to deliver content across multiple sensory channels (visual and audible) to aid learning.
Clark and Mayer advocate the principles of multimedia and modality as guidelines for the design of e-learning. Their multimedia principle calls for including both words (written or spoken) and graphics in e-learning materials [Clark & Mayer 2003]. Reseach and cognitive learning theory indicate that people learn more deeply from words and pictures than from words alone. The modality principle recommends presenting words as speech rather than onscreen text. According to cognitive learning theory, people have different information processing channels for visual information and for audio information; presenting information simultaneously across the two channels reduces the likelihood that either channel is overloaded and therefore aids learning.
Multimodality can also be described as read it, hear it, see it learning. Classrooms offer rich opportunities for multimodality such as chalk and whiteboards, docwnent cameras, handouts, and instructor speech, gestures, facial expressions, and body language. In e-learning, the role of video can help to make the richness of gestural and body language communication available from the classroom, but it needs to be balanced with graphics and other supporting media to enable the full multimodal experience.
An example of how the e-learning environment can promote multimodality could involve the following elements: • Read it: transcripts of the instructor's presentation are available.
• Hear it: audio of the instructor maps to the video of the presentation.
• See it: presentation materials, screenshots, and diagrams appear on-screen synchronized with the instructor's words.

Lean-Forward Learning
E-Iearning is a new educational paradigm. In part, e-Iearning represents a shift away from instructor-centered learning toward student-centered learning. The web provides access to vast amounts of information anytime and anywhere, which makes information gathering and synthesis an easy part of the process of life-long learning. E-Iearning offers self-paced courses in a web environment, allowing for the pursuit of more formal learning throughout our lifetimes without the requirement to be in a particular place at a set time.
As a principle of high-fidelity e-learning, lean-forward learningrefers to the inclusion of handson, context-appropriate practices, demonstrations, exercises, and assessments to engage the student and to assist with encoding the material being learned in a way that facilitates recall in the future.

Practices, Exercises, and Labs
Based on their review of learning research and theory, Clark and Mayer advocate the incorporation of practice exercises and examples into e-learning courses. Practices, exercises, and labs are key learning activities that provide the student with the opportunity to use the knowledge being acquired to complete context-relevant problems or tasks. The practice exercises and examples should mirror the job context as closely as possible to help encode knowledge in a manner that supports its ultimate retrieval when needed to support task performance or problem solving in the workplace. They also report on evidence that practices distributed throughout the training results in better long-term retention than the same practices placed at the end of the training activity [Clark & Mayer 2003].
For IT-related content, virtual machine technologies can be used to create learning laboratories in which students can practice configuration, diagnosis, and other problem-solving skills in an environment that accurately simulates their on-the-job experience. This should lead to better retention and recall ofthe skills being trained. Virtual machine-based labs are also particularly well suited for asynchronous learning because they can be reset and allocated to a particular student on demand, without instructor intervention.
Practices, exercises, and labs extend the read it, hear it, see it model introduced in the multimodality principle to create the read it, hear it, see it, do it model for high-fidelity e-learning. This model provides students the opportunity to engage the learning topic in a multimedia and multimodal environment and provides both active as well as passive opportunities for students to learn at their own pace. Being able to replay VTE content is a key advantage over a standard classroom environment.

Formative Assessment
Assessment has long been a key part of the educational experience. There are two types of learning assessments: summative assessments are used for grading or ranking students and for confirming certain student achievements (e.g., certification exams); formative assessments, or assessments for learning, are used to provide feedback to the student to highlight areas of further study and improve performance [McAlpine & Higgison 2001]. Because this report is about self-paced e-Iearning, we focus on formative assessments.
In studies of formative computer assisted assessments (CAA) in the form of multiple choice questions, students have reported that such assessments are helpful in preparing for summative assessments [Iahad, Dafoulas, Kalaitzakis, & Macaulay 2004], useful for checking their progress in an e-learning enviromnent, and helpful for improving the understanding of content [Iahad & Dafoulas 2004].
"Formative CAA can assist in consolidation of learning, and in identifying weaknesses in assumed understanding. In addition to the specific learning effects that immediate feedback has within an online course, formative assessment may have a more general effect on students by enhancing self-assessment of understanding. This metacognitive skill (of knowing when you know something and when you don't) is considered by many to be a key feature oftertiary-level education, and it is surprising that the "lowly" multiple choice question has the potential to playa role in the development ofthis higher-order cognitive skill" [Dalziel 2001].
High-fidelity e-learning should offer students opportunities to gauge their understanding ofthe material. These formative assessments do not have to be and shouldn't be punitive or serve as gates on the student's progress through the material. They should be a chance for the student to decide whether he or she should continue to review the current module or is comfortable with the material and should proceed to the next module.
Formative assessments extend the read it, hear it, see it, do it model to create the read it, hear it, see it, do it, master it model for high-fidelity e-learning. Master it accurately describes the role of formative assessments in high-fidelity e-learning, because such assessments enable and empower the student to challenge, test, and confirm their mastery ofthe material.

Collaboration
Many research studies have shown that in conventional courses, students who work or study together learn more than those who work or study alone [Clark & Mayer 2003]. In a three-year comparative study of online and conventional courses, Benbunan-Fich and Hiltz found that "outcomes of online courses improved when professors structured them to support the growth of a learning community, by being available online to interact with students, and by using collaborative learning strategies" [Benbunan-Fich & Hiltz, 2003].
Evidence is building that collaboration among students and with an instructor improves learning from online courses. We address this principle of collaboration in two parts: 1. peer-student collaboration 2. instructor support

Peer-Student Collaboration
Research has indicated that "working in groups, instead of alone, increases motivation, perception of skill development and solution satisfaction" [Hiltz & Benbunan-Fich 1997]. This finding was based on self-reported learning, with groups working online through an asynchronous learning network reporting slightly better perceptions of learning than the groups working face-to-face. In another study, 86 percent of students in online courses reported that working in teams was helpful for learning [Lee, Magjuka, Liu, & Bonk 2006].
Many other research studies have found that collaboration among students aids learning. Benbunan-Fich and Hiltz offer two arguments to explain why participating in a group helps people to learn: 1) it mediates socio-emotional variables (e.g., increases motivation and reduces anxiety) to create a favorable climate for learning, and 2) it provides mechanisms that impact cognitive processes (e.g., resolution of disagreements through group discussions, internalization of explanations by others, and self-explanation effects). (Benbunan-Fich & Hiltz, 2003) Two major challenges to the effective use of collaboration among peer students in high-fidelity elearning involve enabling collaboration and motivating collaborative learning opportunities through appropriate instructional design or pedagogy.
Many forms of computer-aided collaboration are readily available to enable communication and collaboration in e-Ieaming. As summarized in Table 1, they comprise both synchronous and asynchronous technologies. Although this report focuses on high-fidelity e-learning, which we have defined as an asynchronous activity, we include synchronous collaboration techniques here because they can be useful in support of any online course with sufficient concurrency -a measure of the number of students actively using an online course at anyone point in time [Clark & Mayer 2003]. High concurrency can be experienced in asynchronous courses with a very large numbers of participants or in academic, for-credit courses that are temporally bound by the academic semester schedule. In online courses with extremely low concurrency, asynchronous communication and collaboration techniques may not be feasible or useful because there are not sufficient numbers of students engaged in the course material at any time.
It is clear that peer-student collaboration improves learning and should therefore be facilitated by high-fidelity e-Ieaming systems. However, the issue associated with designing and motivating effective collaboration in online learning is an instructional design issue and is beyond the scope ofthis paper.

2.4.2
Instructor Support Whether called instructor, teacher, mentor, or guide, an experienced individual delivering or facilitating access to training material adds well-established value. Self-paced, location-independent e-Iearning should provide access to an instructor when the student requests it. Two main principles apply: • The instructor's or subject matter expert's availability and contact information should be readily available to students.
• The instructor should make every attempt to minimize the time between the student's request and the instructor's offer of assistance. The goal would be to reduce the time delta to zerothe student's question is answered immediately. However, because the student can return to the material at any time, the instructor's response is still valuable to the student even if it is delayed.
Timely instructor support and interaction will be subject to course concurrency; without sufficient concurrency, active instructor support may not be feasible. Many of the same computer-mediated technologies listed in Table 1 are appropriate for instructor-student communication and collaboration.

Accessibility
The accessibility principle for high-fidelity e-learning is to deliver content in a manner that is broadly accessible to people with a wide range of abilities.
In the United States, Section 508 ofthe Rehabilitation Act "establishes requirements for electronic and information technology developed, maintained, procured, or used by the Federal government. Section 508 requires Federal electronic and information technology to be accessible to people with disabilities, including employees and members ofthe public" [U S. Department of Justice, Civil Rights Division, Disability Rights Section 2005]. Even though it does not explicitly apply to the training activities or websites of private organizations, Section 508 has become the de facto standard for making such material accessible. This section applies to all websites and is not unique to training materials or online courses. The standard requires that web content be constructed and tagged in a manner that enables the effective use ofscreen readers and other technologies that allow people with disabilities to access internet content. The commercial marketplace includes numerous design tools, checking methods, and training courses to assist website owners to comply with Section 508. § 1194.24 Video and multimedia products.
This section contains two provisions that apply explicitly to training and informational video and multimedia productions. These provisions require a) captioning ofa'1Y speech or other audio information that is necessary for the comprehension ofthe material and b) audio descriptions ofany visual information that is necessary for the comprehension ofthe material.
Some accessibility features provide benefits to all users. For example, a full transcript for audio content provides all users with a means to better understand the lecture content. This is particular-1y helpful when students are not native speakers ofthe lecture language or are new to the technology. Transcripts could be provided in multiple languages to support multi-national training programs.
Transcripts are more flexible and offer distinct advantages over captions. Captions often obscure part of the video content and are typically implemented as part of the video stream Transcripts are displayed in a separate window and do not obscure the video content. Transcripts also offer the opportunity of user-override control. With this feature, a student who is familiar with certain parts of the material can scan forward on the transcript to find the new or unfamiliar material and have the video stream re-synchronize with that section of the transcript. This is also helpful for students who want to return to the training module to review or reference the training material.

Modularity
Modularity means that the high-fidelity e-learning system should support and loosely enforce constraints on content managed and delivered by the system. These constraints may take the form of metadata, navigation rules, and schemas that encourage content creators to publish material as individual, self-contained blocks of instruction. Implementing modularity must be balanced against the constraints it may impose on the design of the learning material and outline. Modularity includes two sub-principles: 1. design for re-use 2. design for reference

Design for Re-Use
Reusability is one area where e-learning offers natural advantages over direct instruction. Once developed, e-learning content can be transferred and replayed infinitely. If properly designed, elearning content can be combined with other e-learning materials for rapid course creation with minimal redevelopment E-learning standards such as SCORM 1 and Common Cartridge' mandate reusability in conformant e-learning material. High-fidelity e-learning should support reuse balanced against the other principles described in this section.
High-fidelity e-learning systems should also encourage content authors to search existing material before creating new material. For individual modules, if existing content is suitable for the instructional needs, the content owner's problem is solved. For course development, the highfidelity e-learning system should assist in developing the outline and present candidate material for inclusion based on the course creator's searches and interests.

Design for Reference
Learning involves both skill building and skill maintenance. Postsecondary training often focuses on specialized job activities, tasks that are performed infrequently, or tasks that are specific to an organization. In these cases, it is valuable for learning content to be available as a reference source for future use to support knowledge recall or skill refreshment at the time the tasks are to be performed.
Content owners should not assume that the course for which a module was developed is the only time or place it will be used, and the high-fidelity e-learning system should offer mechanisms, such as indexing and embedding, that allow individual modules to be used in ways the content owners did not originally envision.
Making the content easy for the student to find and review long after a particular course is complete supports recall and mastery ofthe material -an important element of Read  The high-fidelity e-learning system should make content indexable for public and private search. Rich media should include a transcript searchable from outside the content Any text element should be indexed for search. ]fthe learning module is based on a timeline, the search indexing should support linking directly to the search term High-fidelity e-learning should make it easy and natural to add metadata to published content to support searching across and within modules and filtering based on characteristics of the material. The Institute of Electrical and Electronics Engineers (IEEE) Learning Object Metadata standard 3 and the Dublin Core Metadata Initiative'define sets of metadata for learning objects and a common data structure that can be implemented across disparate learning management systems. We recognize, however, that these standards can be overwhelming to a content publisher. High-fidelity e-learning systems should support these standards, but enforce the tag model found on many Web 2.0 properties. Tags allow individuals who interact with content to describe it in natural language terms that can be aggregated, searched, and filtered. High-fidelity e-learning systems should allow for content tagging by content owners and by students. In tum, searching and filtering content should take into account tags from content owners and students. These systems should also take advantage of natural metadata, such as publication date, update date, content type and creator that can be managed by the system rather than by the content owner.

2.6.2.2
Embedding High-fidelity e-learning systems should support open use and appropriate distribution of learning material. They should support embedding of material in other systems such as websites, blogs, and wikis. All content in a high-fidelity e-learning environment should have a direct link and, ideally, a code snippet to allow users to mash up the learning material with other systems. This must be balanced against permissions on the content and licensing or revenue model in place for the content.

Progress Tracking
The ability of a course provider to track a course attendee's progress is necessary to enable accurate accounting for student accomplishment and completion. For anytime, anywhere, and at-anypace courses, tracking also provides a valuable place-marker for students who access the materials in a given course intermittently over an extended period of time.
The fidelity of progress tracking is important. The learning system should monitor time spent with the learning materials to help distinguish students who are fast-forwarding through the materials from those who are spending the time to watch, read, listen, and learn. Monitoring interactivity is also valuable to help understand whether the student is actually present while the materials are being presented. The value of formative assessments, exercises, and labs has already been discussed; these interactive features offer the added benefit of enabling the system to better track student completion.
http://Itsc.ieee. orgtwg12! http://dublincore.org! Affinnative progress tracking is particularly important for training that is provided to support compliance or regulatory requirements. To support such uses, the learning system should generate auditable records of student completion.
When courses are being served to large, diffuse populations, progress tracking can help the course provider understand concurrency and evaluate the viability of peer-student interactivity in the course.

Balancing Control
Research evidence indicates that program or system control generally yields better learning results than student control [Clark & Mayer 2003]. However, users accustomed to browsing the web and using digital video recording devices expect a high level of control when interacting through a web interface or watching video content. Because students like and expect control, they are more likely to use an e-learning course if some amount of control balance has been implemented [Clark & Mayer 2003].
Chen and Liu studied three types of student control in a web-based course and correlated usage patterns and control preferences with cognitive styles. The three types of student control studied are shown in Table 2. These are typical student control options in e-learning systems [Chen & Liu 2008].
To allow students to control the selection of the contents they wish to learn Chen and Liu found that students with different cognitive styles showed measureable differences in their usage patterns, display option preferences, and navigation preferences in the web-based course [Chen & Liu 2008]. Other fonns of student control provide facilities for organizing and annotating materials and capturing notes. Vrasidas advocates student-centered systems in which students "organize information and knowledge, take control of their learning, act as autonomous individuals who plan and execute learning tasks. High-fidelity e-learning systems should provide tools that allow [students] to organize information, contribute content, and engage in learning activities" [Vrasidas 2004].

Content Control
E-Iearning systems should provide course designers and students with options for organizing, indexing, navigating, and annotating the learning materials. Doing so will improve the likelihood that students use the learning system and will provide the flexibility for tailoring to various cognitive styles.

Using the Principles
Organizations implementing technical training programs can use these principles as a broad set of evaluation criteria for selecting or implementing a training delivery system. The SEI's VTE has been designed to reflect and implement many ofthese principles.
3 Virtual Training Environment -a High-Fidelity e-Learning

System
The Software Engineering Institute's (SEI) VTEis a high-fidelity e-learning system that has been implemented by the CERT Program at the SEI. VTE was first released in 2005 and has continued to evolve and mature. VTE provides access to courses for thousands of users and offers regularly added and updated content

Background and History
The SEI has consistently developed and delivered distance education and e-learning programs. Shortly after September 11, 2001, the SEI developed an in-depth cyber security training curriculum in support ofthe Army Reserve Information Operations Command (ARIOC).' This curriculum consisted of two five-day training courses in basic and advanced principles ofinfonnation assurance, a follow-on course in cyber forensics, a capstone cyber exercise, and an assessment to measure mastery of the material. The direct instruction was heavily weighted towards hands-on training. Because the ARIOC comprised approximately 500 soldiers, the SEI selected a train-thetrainer model, certifying 30 ofthe ARIOC's best and brightest to roll out the training program. The SEI provided teaching material and instructions for building the networks needed to support the hands-on lab environments called for in the courses.
Curriculum development and training were completed in 2004, but the assessment scores did not meet expectations. Feedback from the ARIOC indicated that as reservists, soldiers did not have an effective means to review the material prior to taking the assessment exam. ARIOC instructors also reported great difficulty in replicating the hands-on lab components oftheir classes due to the lack of hardware and software as well as constraints imposed by local security policies.
Taking this feedback into account, the SEI initiated a project to address these issues. The objectives of the project were as follows: • Allow students anywhere, anytime access to SEI course material exactly as if they were in class.
• Provide on-demand instructor support to students while allowing them to progress at their own pace. The delivery mechanism should be instructor-facilitated, not instructor-led.
• Do not require installation of any software on student machines.
With these goals in mind, the SEI built an application and converted its existing library of information security training material to meet these objectives.

Implementing the Principles -VfE Features and Benef~s
From its inception, VIT was infonned by, but did not explicitly address, the principles of highfidelity e-learning desaibed in Section 2. Over the lifespan of the VTE, the SEI has made an effort to leverage these principles in V IT 's delivery interface and content authoring processes. This section introduces a range ofVIT features and indicate' each principle they "-'pport in brackets 3.2.1 Mimic Classroom Education (MuKimodal~y, Personalization) Providing an experience that rea-eates classroom inWJction helps to set a learning context for the student, v.ilo may be accessing the material in a ho".., or office settingV IT's lecture player interface indudes the S3llle elements found in an instructor-led setting such as >l.ides, visuals of the inWJctor and other students, recorded demonstrations, and document cameras. These elements reinforce the idea that the student, v.ilo is accessing the material asynchronously, is actually experiencing the dass and can benefit from questions being a>ked by other students Figure 3  VTE indudes a robust hands-on training lab capability using virtualization technology to allow students to practice concepts asynchronou>l.y Course and content Omlers create combinations 0f virtualized corrvuters in a sandboxed network and publi±! them to V IT along with instructions for completing the lab. Students provision the lab and access it retriltely through a web browser, using the instructions to guide their progress. This allows students to practice concepts using actual systems and gain practice time using software to villich they might otherwise not have access After the students corrvlete the lab, the lab resources are recyded for other students All of the tools used in the labs are open source

En courage Annotations and Takeaways (Pers onalization, lean _Forward learning)
High-fidelity e-learning is an example of Web 2.0 (or as Tim B emers-Lee calls it, the read/mite web)' Stereotypically, Web I 0 sites were static and de,;gned to present infonnation through oneway conununication (e. g., FAQs, product catalogs, and v.tJat' s newpages). Web 2.0 ,;tes are designed to encourage tmJ-way conununication. The users have OJJch more control over the content ofa Web 2.0 site. Wikis, blogs, and maeffi.bleawlications such as Google IWlps are examples of Web 2.0 properties. By allowing students to annotate the lecture material using virtual sticky notes and a highlighter, VTE encourages the >tudents' creativity and allows them to decide how best to rrark the material for future reference. Annotations are saved, so v.ilen students retum to the lecture, the notes they've rrade remain. A s the student personalizes the rraterial, the VTE in-terl3.ce begins to sinulate the annotation process >tudents use with course textbooks and class notes Because VTE training material is accessed asynchronously, students are able to progress at their Oml pace. VTE courses are typically organized as sequences of related rraterial referred to in the system as atopic. Within each topic, VTE provides written rraterial, recorded lectures, narrated demonstrations, and hands-on training labs. Students can access any item as many times as they wim until they achieved their learning obj ectives. Each type of rraterial provides the student with an opportunity to more deeply engage the topic.

Composing Recorded Lectures (Modularity)
Many commercially available products can record classroom activity, convert it to a web-based format, and immediately publish it to a website. With these products, instant gratification and ease of use come at a cost as they typically offer minimal support for editing the material once it is posted. When recording content for VTE, the course owner and an SEI instructional designer collaborate to chunk the recorded material into smaller instruction modules, which typically last 15 minutes. These individual modules are published and then re-assembled into the original course. This extra step adds time and complexity to the publishing process but significantly improves re-use and enhances the search functionality.

3.2.6
Accessing Content via Training and Library Modes (Modularity) VTE includes two complementary approaches for accessing content: a training mode designed to guide students through a linear sequence of learning material, and a library mode that allows students to scan/search available material and quickly access specific modules of interest. Training mode supports skill building, on-boarding (bringing new staff members up to speed), and other learning acquisition models. Students use library mode to maintain knowledge or quickly refresh on a specific topic.

3.2.7
Using VTE Offline (Accessibility, Balancing Control) Courses in VTE can be published to a local file system and copied to a CD or DVD. Students can then use the material on their local desktops or laptops while disconnected from the internet. This functionality is not available for the hands-on labs because they require a live environment.

3.2.8
Providing a Full Transcript (Multi modality, Accessibility) The SEI's content publishing process ensures that each piece ofVTE lecture content includes a verbatim text transcript. Transcripts provide the following benefits for VTE users: • The ability to perform deep searches (inside and across material). Students can use keyword searching to quickly find material related to a narrow topic and access the exact portion of a lecture that referenced that key word.
• The ability to print, download, and annotate course materials to improve retention.
• Access to self-paced learning. Given that most people read faster than the instructor speaks, students can move through the material at their own pace using the transcript as a guide to upcoming material. Though by default the transcript follows the audio, VIE allows the student to unlink the transcript and treat it as a standalone asset.
• Additional time to review materials. In some cases, students may not natively speak the instructor's language. Having a verbatim transcript allows them to pause the audio lecture and look up words they need to clarify without missing any ofthe material.
• Compliance with American with Disabilities Act (ADA) standards.

Content Sequencing and Tracking {Progress Tracking}
Like many learning or course management systems, VTE allows content owners to publish material in sequence and to track student progress. Since VIE heavily leverages timeline-based, synchronized media, it is possible to measure the time a student spends on a given topic and in VTE. VTE includes three interaction states between a student and a particular piece of training material: Not Started, In Process, and Completed. Each time a slide transition occurs while a student is viewing a VTE lecture, the server records the event and a timestamp. Student progress is marked In Process for the lecture until they access each slide, after which it is marked Completed.
When students have completed all the modules in a course (along with other material such as labs and assessments), they receive credit for completing the course. VTE students can generate interim progress summaries and print completion certificates through the site. Training coordinators can measure the progress of multiple students using custom reports.

Virtual Office Hours, Forums, and Breakout Rooms (Collaboration)
To provide an opportunity for live instructor-to-student interaction, VTE incorporates virtual office hours when instructors are available for personal interaction. In a university setting, office hours are a means for students to obtain additional support from the instructor or to ask questions that are not relevant to the rest ofthe class. When establishing a class in VTE, the instructor may post a list of specific hours when he or she will be available in the virtual meeting room and include a link from the course application to the virtual meeting room. Students who require additional assistance can take advantage of this opportunity.
VTE allows course publishers to automatically create and link a virtual meeting room to a course. Students and instructors may jump from the course outline directly in to office hours without requiring a separate set of credentials. While in the virtual meeting room, students and instructors can share desktops and applications, publish audio and video, and hold text chats.
These virtual meeting rooms and threaded discussion forums allow students to have synchronous collaboration with one another and with instructors to further support in-depth learning.

VTE Architecture
The VTE architecture consists of three components: 1. Core Web Application and Learning Management System. The web application layer manages the user interface and business logic. It includes a learning management system that handles course creation, enrollment, progress tracking, and reporting.

Video Management and
Delivery. This is a network of streaming video servers and cloud storage used to improve customer experience and minimize bandwidth limitations. This component is outsourced to a commercial content delivery network provider.
3. Hands-on Lab System. This component contains a lab controller and scheduler, a storage area network for managing virtual disk images (each of which can be many gigabytes), and several rackmount servers that host student lab environments. Since its launch in 20[14, VTE has seen >teady growth and adoption In 2008, VTE is on pace to deliver 120,000 hoors of training material to over 11,000 active users across the U. S. Government and general public Figure 6 shows the training hours delivered by VTE per quarter for the pa>t t"" years Neatly 20 ,DOD ,-"gistered users have acces",d VTE training rraterial ,;nce inception, and over 100,000 unique unregistered users have accessed the public CQntent in VIT that does not require registration. figure 7 shows the total VTE ,-"gistiations per quarter for the past two years, and figure 8 mows the number of unique ,-"gistered and UIlfegistered u",rs that have visited VIT over the past two years As of Au~st 2008 there are 740 unique hours oflecture, demonstration and hands~n lab training available in VIT, This material has been combined into over fifty training courses, Appendix A provide, a detailed rn,e ,tudy and ROI financial analy,;s for the DoD and DISA', u,e of VIT for infonnation ,ecurity certiflrntion training. Appendix B provide, a mtrrnary of VTE use by the U, S, Secret Service for cO"lluter forensics mining,

Comparisons Wllh Dkaet Instruction
Since VIT provide, on-demand access to recording, of direct instruction. it i' po,,;ble to compare the volume of material delivered via VTE with direct instruction. U,ing a ratio of one day of classroom training equaling six hours of online instruction (removing lunch and breaki), the 120,000 hours of instruction delivered via VIT in 2008 is equal to 20,000 person-llays of dassroom tmining,

Scale
The two most challenging cO"llonents ofVIT to acconmodate a large student demmd are the video delivety and the hands-on labs, VTE is architected to scale by adding hardware and ioflware rnpacity to each of these componenll. Because the ;ideo delivery is outsourced to a commercial content delivery netwotk (CDN). VTE can scale by purcha,;ng additional delivery rnpacity to the theoretirnllimits of the CD N. Lab' can be scaled by purcha';ng. configuring and installing an additional selVer for delivety, The lab controller can handle thousands of delivery serveri, and each server can support four to eight concurrent lab environmenti depending on ,;ze and complexity of the lab environments it hold! Re;iew of VIT usage logs shows that each lab ,erver can mpport up to 1,000 ,tudent, a, long a, they access lab' asynchronoml.y (and thus have a relatively normal di.mbution of mage) 21IC"'UISEI_TR_:>coi>OOO

Producing VTE Content
The two most typical scenarios for producing new VTE content are publishing a recorded class and creating a hands-on lab. As with any learning, education, and training initiative, getting started with VTE begins with learning objectives. Typically the process starts with an existing set of course resources such as slides, handouts, and qualified instructors. VTE can also be used to publish a simple demonstration on a specific topic.
Note that this section focuses on a content producer or author rather than an instructor. One of the design goals for VTE is that it not interfere with the instructor's normal delivery and student interaction.

Publishing a Recorded Class
This section describes requirements and resource estimates for including a recorded class as part ofVTE.

Content Requirements
In this scenario, an organization has a set of materials that represents a block of instruction, where the organization typically presents the material in a live classroom setting. The instruction is supported with slides, usually in Microsoft PowerPoint format.

Infrastructure and Environment Requirements
Once the course instructor, slides, materials, students, and time and place are ready to go, the VTE producer prepares to record the live instruction. At a minimum, both the slides and the instructor should be recorded. It is also helpful to record the students or to capture a wide shot of a live classroom experience for the eventual VIE user.
Periodically transitioning between camera views can make the material more visually interesting. The camera directed on the slides may not be a camera at all -it could be a PC add-in card designed to record slide images from VGA output. Regardless of how many cameras are used, they must record in sync with one another, from the beginning to the end. In particular, the timing of slide transitions is critical to ensure that the online student experience replicates the live class expenence.
Footage from all cameras is fed into a non-linear editor (NLE) for post production. In the NLE, a videographer can remove extraneous footage, correct color or audio deficiencies in the raw material, and convert recordings to a fonnat suitable for online display. VTE uses Adobe Flash video as the player, because it is operating system agnostic and enjoys a very high market penetration among web users. As a result, a student is unlikely to need to download and install new software to access the material.
VTE instructor video is presented at a resolution of 320 x 240, with 150 kilobits per second (kbps) of video. The slides are converted to vector (Adobe Flash) format and are presented at a resolution of 640 x 480. They are downloaded to the student's computer when the student accesses a lecture.
As part of the NLE process, the videographer produces an audio version of the material and submits that to a transcription service provider. Typical turnaround is 48 to 72 hours. The transcript is time-coded to the recorded audio file. The transcript arrives in an X1v1L format that can be easily converted to structured data for use in VIE. The structured transcript data is merged with the slide timing data to create a master event timeline. This file coordinates the events within the VTE ondemand lecture and allows the student to move through the material as they see fit 3.5.1.3

Time and Schedule Estimates for a Typical Course
Based on the SEI's experience with capturing and publishing 800 hours oftraining, it takes one week of post-production and publishing for each day of recorded material to produce a quality result from a captured course. So, if the organization captures a three-day training course, it can expect to expend three weeks to launch the course on VTE 3.5.2

Creating a Hands-On Lab
This section describes requirements and resource estimates for including a hands-on lab as part of VTE 3.5.2.1

Content Requirements
A hands-on lab should reinforce a particular learning objective. It's most helpful to think of a lab as representing a change in state -of a network, of a system, of a piece of software, etc. The student's task in the lab is to change the environment from the starting state to the finished state, so the instructor's job is to guide the student through that process. The instructor typically accomplishes this goal by creating a lab document that describes the actions the student must undertake to complete the exercise, in as much or as little detail necessary to meet the learning objective.

Infrastructure and Environment Requirements
The starting point for creating a hands-on lab environment is one or more virtual disk images, ideally in VMWare format 7 This product allows authors to create virtual machines that behave exactly like Intel-based systems -Windows, Linux, Solaris, etc.
Depending on the learning objectives for the lab, it may be possible to use preconfigured images from V1vf\Vare's Virtual Appliance Marketplace. 8 Many of the images are free for use in training.
Authors should be aware of software licensing restrictions because VTE can create hundreds of instances of a lab environment once it is built, which may violate license agreements.
After the authors have established the authoring environment and disk images, they must log in to the virtual machine(s) and install any custom software needed for the scenario. Once the starting state ofthe lab is set, the author should take a snapshot ofthe environment A snapshot is a special function of virtual environments that creates a baseline for each individual image. This snapshot represents the starting point for the lab exercise. The instructor can then perform the exercise to determine whether the lab meets training objectives. During this first dry run ofthe lab, the instructor can take screenshots and create student documentation on how to perform the lab exer-cise. When the instructor is satisfied, the snapshot can be discarded. The base images are ready to be loaded into VTE where a new lab is created to load the images on demand.

Time and Schedule Estimates for a Typical Lab
While publishing a VTE hands-on lab takes only minutes, creating the lab can be challenging for a novice author -planning and conceiving the scenario takes a lot oftime but the payoff is high. And, given that the lab environment can be replayed as often as needed, scenario reuse is high. The SEI's experience indicates that it takes about one staff-month of effort to produce a quality hands-on training lab for VTE.

Conclusion -The Future of VTE
The SEI's VTE has been developed to reflect the principles of high-fidelity e-learning. It offers the user a web-based, multimedia, interactive training experience that can be accessed anytime, anywhere. This environment uses state-of-the-art software and hardware to train users with technical and managerial roles on a wide range of information assurance topics.
The VTE learning model presents a read it, hear it, see it, do it, master it progression of course lectures, demonstrations, and hands-on labs to meet student learning objectives.
The SEI is continually enhancing the VTE platform to further support and promote the principles described in this report, and to move more training content to an anywhere, anytime model. The following are planned enhancements to VTE and the principles they support

• The Social Learning Management System [Personalization, Collaboration, Student
Control, Progress Tracking] VTE's learning management system (LMS), which includes enrollment in courses, progress tracking, reporting, and instructor support, will shift from a centralized model to one based on the decentralized model made popular by social networking sites such as MySpace, Facebook and LinkedIn. This shift is based on the principle that users and organizations are the best judges of their training needs and no centralized administrative office is able to react quickly enough to support those needs. VTE's Social LMS will empower users and organizations to build a learning structure that mirrors the real world.
Instructors and students will be able to easily create questions, assemble them into assessments, rate and discuss them, and export them to other systems. Any user will be able to create courses using combinations of VTE content and external learning resources such as Wikipedia pages or YouTube videos. Any registered user will be able to create a course; any other user can request to be enrolled in that course. Users will be able to build relationships with other users and support each others' training progress. VTE will also include built-in functions to allow users to set goals for course completion and periodically remind and encourage users to complete courses on a schedule

• Formative Assessment via hands-on labs [Lean-forward Learning]
The SEI will expand the hands-on lab capability to allow instructors to assign labs to students. Lab assignments can have completion dates and student submissions can be graded manually by the instructor or programmatically by the system. These formative hands-on assessments can be embedded in any VTE course and used to gauge progress or permit the student to enroll in other courses.

Whom to Contact; How to Proceed
Organizations that wish to discuss leveraging high-fidelity e-learning in their environment should contact VTE Support at vte-support@cert.org to discuss training initiatives and next steps.

Appendix A Defense Information Systems Agency (DISA) Case Study
Overview This appendix describes the adoption ofVTE by the DoD through a sponsorship from the DISA This appendix covers the first year (FY2007) ofthe DISA sponsorship.
The data presented in this Appendix focuses on cost comparisons and return on investment. The Appendix acknowledges the need for data that indicates the quality ofthe results achieved through VTE training such as student course completion rates or student certification pass rates. While the VTE development team has explored several possibilities for collecting measures of this type, no such data is currently available. For future research, we encourage collecting data for measuring VTE training effectiveness and will continue to explore the means to do so.
In December 2005, the DoD issued Directive 8570.1 Information Assurance Workforce Improvement Program [DoD 2005]. In summary, the directive mandates that all DoD personnel who hold elevated privileges on a DoD system or network must obtain a commercial certification within six months of obtaining the privilege. The directive identifies two different types of positions into which individuals are to be categorized: technical (IAT) and managerial (lAM) Within each type, there are three levels that describe the breadth ofthe individual's responsibility, forming a matrix of six categories of personnel. The directive and subsequent communications from the DoD map these categories to commercially available certifications in information security.
The directive was issued without specific funding guidance. It was the intent ofthe DoD that existing training budgets for the information assurance (IA) workforce be used to assist staff in preparing to sit for a certification that fits their job classification. The DoD IA community found, however, that the cost oftraining staffto meet this directive, in addition to the rest oftheir duties, significantly strained their operating budgets.
Reacting to this pressure, the DISA contracted with the SEI in November 2006 to develop preparatory courses for two of the most popular commercial certifications that met the 8570 requirements (Security+ and CISSP [Certified Information Systems Security Professional]). The SEI was to make these courses available (along with other supplemental material) to a portion ofthe DoD IA community via VTE. DoD personnel who were required to obtain a certification per the directive were able to access VTE course material to help prepare for their certification test.
In the first year ofthe program, 8,000 DoD personnel obtained VTE accounts for use in enhancing their skills. In that year, users accessed over 38,000 hours oftraining via VTE. The program was renewed for FY2008 and now averages over 10,000 hours per month, which is the equivalent of 840 person-days of classroom instruction over the year. According to George Bieber, Director, IA Workforce Improvement Program, "The SEIIVTE support to the DoD has had a significant positive impact on DoD 8570 policy implementation by providing an effective means of preparing personnel for certification testing. Training to support CISSP and Security+ training have reduced failure rates on these certifications to levels below those experienced by individuals in the private sector. "

Financial analysis and ROI of OISA contract data for 2007
Under the DISA work plan, VTE was used to deliver more than 38,000 hours oftraining in the first 10 months of calendar 2007. The average cost of a delivered training hour was $22.49 in that time period. 9 To analyze and better understand the VTE financial performance under the DISA contract, the performance and cost data from the DrSA contract have been compared to annual survey data from the American Society of Training and Development (ASTD).

THE ASTO REPORT
ASTD publishes an annual State ofthe Industry report based on a survey of its members [ASTD 2007]. The report and the survey are focused on workforce training through workplace learning and performance management initiatives, which makes the data well suited for comparison with VTE. The report contains detailed statistics about training costs reported by the members and trends in workforce training; the 2007 report contains data from 2001 through 2006. One trend discussed in the report is technology-based training methods: "Persistent usage oftechnology-based methods has dramatically shaped the field. As organizations deploy e-learning more frequently, the use of instructor-led learning has diminished. Discrete learning events in traditional classroom settings are gradually shifting to learning experiences that are occurring at the workstation and at the pace of the worker. Many organizations have adopted e-leaming platforms in recent years and are starting to see their investments bear fruit. Some positive trends from the 2006 data include an increased ratio of learning hours used to learning hours made available (also referred to as reuse ratio) and a greater ratio of employees to WLP [workplace learning and performance] staff members. These types of efficiency gains are often facilitated by technology-based solutions. • Learning Hours Available -a one-time count of the total number of hours of learning content available in all forms oftraining (classes, workshops, online, etc.).

• Learning Hours
Used -the total number of learning hours accessed or completed, which is calculated by multiplying the number of people who participate in a learning activity by the duration of that learning activity. For example, if a four-hour seminar were provided and 100 people attended, the Learning Hours Used would equal 400.
• Direct Learning Investment -direct expenditures for learning activities. The survey collects this data across a number of categories, but does not distinguish e-Iearning investments from other learning investments.
From these baseline measures collected in the survey, the ASTD calculates several measures that are provided in the report For the purposes of comparing ASTD data with VTE data, we will include the following measures: • Cost per Learning Hour Available -the report does not indicate how this measure is calculated, but based on the survey data collected, we assume that it is Learning Hours Available --:-Direct Learning Investment.
• Cost per Learning Hour Used -the report does not indicate how this measure is calculated, but based on the survey data collected, we assume that it is Learning Hours Used c-Direct Learning Investment • Reuse Ratio -the ratio of Learning Hours Used to Learning Hours Available.
The ASTD report provides the data consolidated for all survey responses, and also provides breakdowns by various member types. For the purpose of our VTE comparison, we use the consolidated data since there are no category data that are a close fit for the target audience of VTE. Table 3 presents data from the ASTD report for VTE comparison.   Cost per Learning Hour Available for VIT is higher than mOml in the ASID sUlVey data This is to be expected given that VIT is purely e-learning v.trile ASTD daB cmtains both e-learning and conventiomlllaining. In analy":s of its Oml remits, the ASID indicates that a higher co>t per learning hour available mould b e anticipated v.ilen funding the infrastructure to establim elearning solutions. The Reuse Ratio forVIT is sub>tantia1ly greater than shomlin the ASTD data This too is to be expected given that V IT is an e-learning platform. The hours available from an e-learning solution are accessible on-demand, subj ectto fewlimitations (such as bandwidth costs for a given hour of training, the VTE solution provides substantially lower cost per training hour delivered and a much higher reuse ratio for training hours than is currently being experienced in blended (e-learning plus traditional training) programs for workplace training.

Appendix B U. S. Secret Service Experiences with VTE
The appendix presents the executive summary from the Department of Homeland Security U S. Secret Service Criminal Investigative Division's "Virtual Training Environment (VTE) Pilot Study" Final Report (USSS 2005). It is reprinted with permission.
In early 2005, Carnegie-Mellon University, Software Engineering Institute, CERT, launched an initiative to create an on-line or "virtual training environment" for delivery of course materials related to its Knowledge in Depth for Defense in Depth (KD3) program.
The KD3 program employs a Virtual Training Environment (VTE) system to deliver, via the internet, program content in an always-available manner. VIE" ... allows students to train without regard to schedule, physical location, or presence of instructor in an anywhere, anytime environment 13 ." In March, informal discussions began with VTE development staff about the potential for its use by the Electronic Crimes Special Agent Program (ECSAP)14 program as a means to deliver core and specialized training, and continuing education to ECSAP personnel. VIE was described as distance learning tool that, in addition to providing material via video, audio, and text, incorporated 'virtual labs ' wherein students could safely and repeatedly practice the skills and techniques learned during instructional portions of a course.
In May, Criminal Investigative Division (CID)/Electronic Crimes Section (ECS) announced its intention to research "distance learning" for the ECSAP program. VTE was offered as a possible solution and a Pilot Study was proposed to assess VTE's strengths, weaknesses and compatibility with ECS' s vision of distance-learning.
VTE was demonstrated during the July, 2005 ECSAP conference" in San Diego, CA and 25 attendees volunteered for the Pilot Study. The Study took place over a two-month period (Oct 1<, thm November 30 th ) and consisted oftwo parts; Operational and Assessment. The operational portion began October 1<, and concluded at the end ofthe month. During this period Study participants viewed various demos, tutorials and/or courses and conducted one (or more) associated labs. The Assessment portion began November 1st and concluded November 30 th . During this time participants were asked to record their experience(s) using VTE and to rate its features, functionality and applicability to the ECSAP program. Participants recorded their observations and opinions thru either on-line or printed survey.
Overall, participants rated VTE's features and functionality between seven (7) and ten (10) on a 1O-point scale. Areas rated included access, navigation, and audio and video content. Overall, 14 of 16 16 respondents answered 'yes' when asked ifthey considered VTE of benefit to the ECSAP program and if it should be adopted for use by the Secret Service for the ECSAP program. Moreover, one participant reported that he had used VTE to assist him with resolving a technical roadblock he confronted during a recent digital examination. Source: KD3 Information Sheet "Agents assigned to ECSAP are computer investigative specialists, qualified to conduct examinations on many types of electronic evidence, including computers, personal data assistants, telecommunications devices, electronic organizers and other electronic media." [http:ltwww.ustreas.gov/usss/criminal.shtml] Presentation by Mr. Richard Nolan, VTE Development Team Manager The remaining two respondents answered 'yes, with reservations' In short, CERT's Virtual Training Environment is a high-performance, professional, anytime-anywhere distance learning program. It is capable of delivering a wide variety of content (video, audio, text) in a manner which is both intuitive and flexible. What makes VTE unique, however, is its ability to let students practice what they leam in a safe environment through its virtual labs. VTE, by design, is engineered to adapt to the changing needs and criteria of its users.