The CERT ® Approach to Cybersecurity Workforce Development

For most established organizations, developing and maintaining a competent cybersecurity workforce needs little justification and is, in fact, a central requirement for ensuring resilient operations. As a result, these organizations invest significant resources in attempts to fulfill this requirement. However, most organizations find that the rapid changes and dynamic nature of cybersecurity make keeping their workforce up to date a very challenging problem. This report describes a traditional model commonly used for addressing this challenge, explains some operational limitations associated with that model, and presents a new, continuous approach to cybersecurity workforce development.


Executive Summary
For a cybersecurity workforce to be effective, its members must possess the knowledge, skills, and experience required to perform their job duties.Proficiency and relevance are key factors in determining the effectiveness of each of these components.Proficiency refers to how well someone understands a subject matter or can apply a given skill.Relevance refers to how useful that knowledge or skill is in performing a given job duty.For example, someone could have expert-level knowledge, skill, and experience in a particular area, but those assets will have minimal bearing on performance if the person's area of expertise is not relevant to their job duties.
Organizations are faced with the ongoing challenge of ensuring that their current workforce possesses the most current knowledge, skills, and experiences-with an emphasis on proficiency and relevance.However, this issue is particularly challenging for a cybersecurity workforce because industry trends, practices, and technologies are constantly changing.For example, cyber attack vectors are constantly changing as attackers search for new ways to circumvent security controls and infiltrate systems.As a result, security practices and technologies must change accordingly to protect against new vectors of attack.To apply these new security practices and technologies successfully, cybersecurity professionals need to obtain the appropriate knowledge, skills, and, eventually, experience.
An organization must consider several factors when choosing a workforce development training program: 1.The training program needs to provide the workforce with the knowledge, skills, and experience that are most relevant to their job duties.
2. The training program needs to cultivate a high level of proficiency through maximum development of knowledge, skills, and experience.
3. Training that consumes large portions of an individual's time interferes with their job duties and leads to lost productivity.
4. The scalability of a training solution and budget limitations restrict the amount of training an organization can offer to its workforce.The more cost-effective and scalable a training solution is, the larger the audience it can reach.

Shortcomings with the Traditional Classroom Training Model
The

A New Approach f Workforce
The CERT ®1 approach to cyberse experience in a continuous cycle o on building a specific area of dev of development-with the purpos for cybersecurity professionals to and effectively perform their job for Developing the Cybersecurity ecurity workforce development builds knowledge, skills, of professional development (see Figure 1).Each phase velopment that is leveraged and supplemented by the next se of reaching some end goal.The end goal of this approa o use relevant knowledge, skills, and experience to succes duties.
Cybersecurity Workforce Development nt phases of the approach-knowledge building, skill bui evaluation phase that allows for assessment.des a solid foundation of knowledge; this is where the of a particular topic area are learned.
arning how to apply hands-on, technical skills that are ba that was learned in the previous phase.
ops the ability to adapt and successfully apply skills in ch s; individuals apply knowledge and skills in real-world s ce metrics to assess individuals' absorption of the trainin of improvement for continued professional development.The goal of the knowledge-building phase is to provide individuals with a foundation of knowledge that will facilitate the development of skills and, subsequently, the successful application of those skills.To consistently and successfully apply a skill, an individual needs to understand the basic fundamentals and concepts that are behind it.For example, when people are learning how to drive a car (the skill), they first learn how a car operates, safety practices, potential hazards, and traffic rules.All of this knowledge is learned before an individual gets into the driver's seat.If someone attempts to drive a car without learning this foundational knowledge, their chances of being able to successfully operate a vehicle are low.Similarly, this concept holds true for developing and applying cybersecurity skills.If someone does not understand the basic fundamentals and concepts of networking, they will not be able to effectively perform skillrelated activities such as reviewing intrusion detection system alerts, monitoring network resource availability, and performing basic packet capture analysis.As a result, knowledge building is a critical first step in cybersecurity workforce development because the ability to effectively apply cybersecurity skills directly affects job performance and the overall security of an organization.
Knowledge building can be performed in several different ways.Classroom training is the most traditional method for knowledge building and has been the most popular form of professional development in many fields, including cybersecurity.Another option for building knowledge is to leverage online learning solutions.These solutions may be better suited for professional development for these reasons: 1. Online training is extremely cost effective because (1) it is usually cheaper since expenses are lower for the online training providers, and (2) organizations do not have to pay travel expenses for off-site training.
2. Online training can be broken into manageable segments to accommodate work schedules and responsibilities.Because online training is performed asynchronously-meaning that participation only depends on one party (the participant)-it is possible to train while on the job, shifting between the training and job duties without them interfering with each other.
3. Online training enables individuals to train at a pace that corresponds with their ability to absorb knowledge.Some individuals may prefer a full day of training, while others may find it more beneficial to segment training into two-to three-hour blocks over several days.Furthermore, research has shown that online learning is an effective and viable option for professionals and, in some cases, yields better results than traditional face-to-face instruction. 2

Skill Building
The purpose of the skill-building phase is to develop hands-on, technical skills, based on the foundational knowledge learned in the previous phase, which will be used to effectively perform job duties.It is in this phase that individuals begin to apply the knowledge they have been learning.To extend the analogy about learning to drive, the skill-building phase is where the person gets into the car for the first time and starts to learn how to drive it.Because driving a car is a combination of various skills, the driving instructor begins by developing one skill at a time through short, simple exercises.This approach facilitates greater skill proficiency by enabling the student to concentrate on a single task.For example, the first skill someone may learn is how to operate the gas and brake pedals by slowly accelerating and then applying the brake.After the individual has mastered that skill, they progress to learning how to make turns by driving in a straight line, applying the brake to slow down, and then turning around a cone-all of which is done in an empty parking lot.Eventually, the student will have developed enough skills to combine them and begin driving around the empty parking lot similar to how someone would drive on the road.Developing cybersecurity skills uses a similar approach regardless of what the skill is related to, be it malware analysis, penetration testing, or incident response.
Skill building is an integral component of professional development for the cybersecurity workforce because sometimes it is possible for individuals to perform skill-related activities (i.e., activities where a skill needs to be applied) by using automated software tools.Usually, a person needs to possess a skill-an ability typically developed through training and experience-to perform a skill-related activity.For example, to transport a piano (the skill-related activity), a person needs to be able to drive a truck (the skill).However, automated software tools make it possible to perform skill-related activities in technical fields without possessing the skill or the foundational knowledge behind it.While the purpose of these tools is to improve efficiency by automating manuals tasks, relying on these tools can cause an erosion of skill within the cybersecurity workforce and create an unskilled population of cybersecurity professionals.For example, there are numerous digital forensic tools, such as Autopsy, EnCase, and Forensic Tookit (FTK), that automate certain skill-related activities, such as carving files from hard-drive images, identifying and recovering deleted files, and parsing file system information.These tools are valuable because manually performing those tasks is time consuming and an inefficient use of a forensic analyst's time.However, if the conditions in which these tools normally operate significantly changes, then they are rendered ineffective.In other words, if a file system is corrupted to a certain point, these tools may not be able automate activities such as carving specific files from a hard drive image.In these instances, forensic analysts need to utilize and adapt their skills to perform these tasks manually.An unskilled professional who relies on these tools will not be able to adapt to this situation.
Similar to the driving analogy, cybersecurity skills are best developed through short, narrowly focused exercises that are designed to transform knowledge into the ability to apply it.As a result, exercises should take place in controlled environments (such as the empty parking lot) so that individuals can focus on performing specific skill-building activities without being overwhelmed by complex information technology (IT) infrastructures, unpredictable variables, and external stimuli-all of which would detract from skill development and change the learning environment.For example, suppose an individual is learning how to use Wireshark to capture and subsequently analyze network traffic.In the skill-building phase, the exercise environment may involve just two systems-one to generate some simple network traffic patterns and another to capture themwith the skill-building exercise focusing on a few simple activities, such as performing a basic packet capture, applying a packet capture filter, and then identifying a transmission control protocol (TCP) handshake in the capture results.As an individual becomes more proficient with these skills, the difficulty and complexity of these exercises can be increased.However, it is important to remember that the exercises still occur in a controlled environment.Once an individual has reached a certain level of proficiency, the next step is to refine those skills by applying them in real-world scenarios-otherwise known as experience building.

Experience Building
The goal of the experience-building phase is to maximize effective job performance by exposing individuals to real-world scenarios, events, and activities that are similar to ones they will encounter in their jobs.In the previous phases, knowledge and skills were developed in controlled, focused environments.However, actual scenarios often occur in uncontrolled environments that are complex, unpredictable, and include external variables-all of which can significantly change the situation and operating environment.As a result, experience building is needed to refine knowledge and skill so that it can be successfully applied on the job in a realworld environment.
Experience is defined as active participation in events or activities that lead to the accumulation of knowledge or skill.The more activities or events an individual participates in, the more experience, and therefore more knowledge and skill, the individual gains.Experience is valuable because the knowledge and skill acquired by participating in specific events and activities can be applied when similar situations are encountered again.In terms of the driving analogy, the experience-building phase is when the student starts to learn how to drive in traffic.Until now, all knowledge and skill building has occurred in controlled environments.Once the individual has demonstrated a certain level of proficiency driving in an empty parking lot (skill building), the next logical step in development is to expose the student to live traffic (experience building).
Although it is possible to teach advanced driving skills-such as parallel parking, making threepoint turns, and navigating road hazards-in an empty parking lot, there is complexity and unpredictability that can only be experienced in live traffic situations.As a result, if someone attempts to drive in live traffic without first being exposed to it with the benefit of formal instruction, they will have difficulty adapting to the real-world environment and are more likely to have an accident.This concept holds true for refining knowledge and skill in the field of cybersecurity.Capturing and analyzing network traffic in a controlled environment is one thing; applying those skills in a real-world environment is completely different.A real-world scenario could include multiple networks, large quantities of traffic, and a wide variety of protocols.Additionally, it may also include external factors that cannot be controlled or predicted, such as attack sophistication, the accuracy of third-party information, and organizational priorities.
Being exposed to the unpredictability of real-world scenarios and environments in the experiencebuilding phase also develops the capability to successfully adapt and apply knowledge and skill in changing and unfamiliar situations.Comprehension of knowledge and proficiency with skills increases as individuals are exposed to a greater variety of situations in which they must apply them.This approach is widely practiced in the United States military, where comprehension of knowledge and proficiency with skills can be the difference between life and death.In the military, service members often participate in exercises that simulate real-world scenarios in which unpredictable and changing situations are introduced to force the participants to develop and use the ability to adapt and apply their knowledge to the given situation.
Although on-the-job training is a common method used for experience building, it is not always the most pragmatic environment for learning.First, since on-the-job training occurs in an operational work environment, efficient and effective job performance usually takes priority over training and development needs.Second, some situations happen so infrequently and are so important that they are not practical for on-the-job training, such as situations where disaster recovery and business continuity operations need to be activated (i.e., natural disasters or catastrophic failures of infrastructure).As a result, an alternative method for experience building is to simulate real-world environments and scenarios that cybersecurity professionals will encounter on the job.Until recently, creating training environments that mimicked real-world IT infrastructures was not a viable solution because doing so was a costly and resource-intensive venture that involved procuring, configuring, maintaining, and storing large amounts of equipment.However, the advent of virtualization technologies has made simulated cybersecurity training environments a cost-effective option for experience building.For example, as a result of virtualization, an entire simulated IT infrastructure, which simulates normal network traffic patterns and executes real attacks, can now exist on a single server.Additionally, virtualized cybersecurity training environments provide a much more scalable and accessible solution for experience building because they can easily be replicated and made available over the internet.

Evaluation
The goal of the evaluation phase is to enable a continuous cycle of professional development by assessing knowledge comprehension and skill proficiency and by making it possible for organizations to accurately catalog its employees' knowledge and skills.Continuous professional development is particularly important for the cybersecurity workforce because the field is constantly changing-technologies rapidly evolve, and attackers quickly adapt to circumvent the latest security practices.This constant change makes a formal evaluation mechanism critical for cybersecurity workforce development methodology because organizations need to be able to systematically ensure that their staff maintains the knowledge, skills, and experience needed to adapt to these changes.
Once an individual has gone through the three development phases of the approach-knowledge building, skill building, and experience building-it is important to assess the level of knowledge comprehension and skill proficiency that was achieved through the training.This assessment should be linked to instructional objectives, which define what an individual should know (knowledge) or be able to do (skill) after they go through a particular training course or module.As a result, instructional objectives serve as excellent metrics for assessing knowledge and skill (see Table 1).For example, one instructional objective for an incident response training course could be that, given access to network monitoring devices that capture both normal and abnormal network traffic, the student will be able to identify web server vulnerability scanning activities in at least intrusion detection system (IDS) and web server logs.In this example, the instructional objective clearly defines the skill being developed (i.e., identifying web server vulnerability scanning activities) and, as a result, skill proficiency can be measured.Behavior What do you expect the audience to be able to do?This is an overt, observable behavior, even if the actual behavior is mental in nature (i.e., comprehension of knowledge).

Condition
Under what circumstances will the learning occur?

Degree
What criteria need to be met for this objective to be achieved?For example, is the objective achieved with total mastery (100%) or when a minimum standard (70%) is met?
Assessing knowledge and skill during the evaluation phase determines whether an individual has achieved the desired level of knowledge comprehension and skill proficiency from the training.The assessment will be used to determine the next training cycle for an individual.If an individual has achieved the desired levels of knowledge and skill, they are ready to move to more advanced training or a different subject matter.However, if an individual does not achieve the desired levels of knowledge and skill, the assessment can be used to identify areas of weakness that need to be improved.In either situation, the goal is to provide individuals with a path for continued professional development.
The evaluation phase also provides a mechanism that organizations can use to obtain a better understanding of the knowledge and skills that its workforce possesses, which benefits both the organizations and its workforce.First, organizations will be able to provide more meaningful and relevant training to its workforce because the information obtained from the evaluation phase will make it easier for organizations to identify training needs.As a result, cybersecurity professionals within the organization will have more training available that furthers professional development.
Second, organizations will be able to better maximize job performance by ensuring that individuals possess the right knowledge and skills needed to effectively perform their job duties.Deficiencies in knowledge and skills will be easier to identify, and organizations can address any shortcomings.Third, organizations can use the information about its employees' knowledge and skill to help mitigate operational risk.Specifically, this information can be used to identify and avoid single points of failure with respect to critical knowledge and skills within the organization.

Case Study
The following case study is an example of how the approach can be implemented.The case study not only highlights a real-world implementation of the cybersecurity workforce development approach but also demonstrates that the approach is flexible and can be used to meet organizational training needs.The case study leveraged the Carnegie Mellon ® Software Engineering Institute's Virtual Training Environment3 (VTE) and the CERT Exercise Network4 (XNET) technologies (Table 2) to implement the different phases of the approach.

The U.S. Air Force: Unit-Level Force Development
In 2009, CERT partnered with a U.S. Air Force (USAF) Cyber Operations Squadron to develop a way to train network warfare teams, or crews, for computer network defensive operations and to measure mission readiness afterwards.One goal of this project was to develop the incident responders' knowledge, skill, and experience in basic digital forensics to reduce the workload of the forensic analysts and facilitate more effective collaboration between the incident response teams and the forensic analysts.

Segment One: Evaluation
This particular implementation of the approach to cybersecurity workforce development started with the evaluation phase, which was used to determine the kind of knowledge, skills, and experience that was to be developed in the subsequent segments of the training.During this segment, XNET was used to conduct a standard squadron technical evaluation of the real-time analysts (incident responders).Each analyst was provided with a virtualized representation of their workstation and network environment, which included the same tools they used on the job and a sampling of the Air Force Global Information Grid (GIG) infrastructure.The analysts were presented with a real-world cybersecurity scenario and were observed by unit evaluators, who injected attacks and other inputs into the scenario during the session.
After the individual technical evaluations, real-time analysts were paired up and presented with a more complex team-based incident response evaluation in XNET.The scenario encompassed the anatomy of a real attack-reconnaissance, botnet and malware staging, data exfiltration, and massive communications disruption-and involved more than 100 virtualized computers and infrastructure devices.Analysts were provided with a virtual representation of their workstations and a variety of incident detection, response, and forensic tools.Similar to the individual technical evaluations, evaluators and instructors observed the teams' performance during the scenario.

Segment Two: Knowledge and Skill Building
During segment two, VTE was used to develop the knowledge and skills that would be needed for the final capstone exercise.Knowledge building was achieved by providing the participants with lectures, slides, and technical demonstrations about various incident response and digital forensic topics.Skill building was accomplished by completing hands-on technical labs that were narrowly focused on developing specific skills.For example, all analysts were required to complete a lab that walked them through the step-by-step, forensically sound process of capturing a hard drive image from a compromised server and performing initial forensic analysis of the image.VTE proved to be an ideal environment for knowledge and skill building because it provided (1) a robust technical capability for delivering online training content and (2) a controlled hands-on learning environment, which is conducive for developing knowledge and skill.

Segment Three: Experience Building
The final segment of training culminated in a capstone exercise in XNET that required the participants to use and apply their knowledge and skill in a real-world scenario and modeled infrastructure environment.The goal of this capstone event was to further refine the participants' knowledge and skill by exposing them to an exercise that introduced complexity and unpredictability into the equation.The capstone exercise built on the team-based technical evaluation in segment one and required the five-person teams, composed of four real-time analysts and one forensic analyst, to coordinate and conduct initial incident response and forensic analysis of the cyber-attacks detected in segment one.The teams were provided with a suite of tools and were required to move outside of their regular mission scope-analyzing real time alerts-and perform digital forensics tasks, such as event inventory and correlation, live system data acquisition, and log file, hard drive, and memory image forensic analysis.XNET's automated evaluation capabilities were used to conduct staged performance-based assessments as well as collect situational awareness reporting.

Conclusion
Traditional, classroom-based training models are effective for certain types of learning but are not ideal for satisfying the developmental needs of the cybersecurity workforce.The CERT approach to cybersecurity workforce development is divided into continuous phases of development that progressively builds an individual's knowledge, skills, and experience in ways that are relevant to their job duties.The approach combines the concepts of classroom-based models with the flexibility of online platforms and adds another layer of development that is particularly beneficial for professionals in the field.Specifically, it incorporates a focus on experiential learningproviding real-world scenarios that enable participants to apply their knowledge and skills in situations and environments they may face on the job.As a result, the CERT approach to cybersecurity workforce development offers organizations a comprehensive, targeted, costeffective training option that can be tailored to their needs.

Figure 1 :
Figure 1: The CERT Approach to C There are three main developmen and experience building-and an•

2U
.S. Department of Education, Office of Planning, Evaluation, and Policy Development.Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies.Washington, DC, 2009.
most common workforce development training solution is the traditional classroom training model.While this training model is easy to implement and is widely used, there are a number of reasons why it is not adequate for providing effective, large-scale training to a technical workforce.1. Traditional classroom training is not ideal for developing skills and experience at a high level of proficiency.Rather, skills and experience are best developed in environments that closely mirror the real-world environments where they will be applied.For a cybersecurity workforce, these environments include elements such as networks, software toolsets, and user-generated traffic.
2. Traditional classroom training is time consuming.Professional training seminars and coursesoften consist of day-long sessions that can span multiple days.Attendees are unable to perform their job duties during these large training blocks, so organizations lose productivity.3.Traditional classroom training does not scale well, nor is it cost effective for large organizations that have employees who are physically distributed across different geographical areas.Specifically, classroom size-physical dimensions and student-toteacher ratio-and travel costs are both limiting factors that restrict the amount of training an organization can provide to its workforce.4.Traditional classroom training is not optimal for rapidly changing fields such as cybersecurity, where practitioners must stay abreast of the most current trends, technologies, and practices to successfully perform their job duties.Quickly disseminating new and updated training courses is a challenge because of the additional time and costs associated with printing new material and having instructors learn it.5.The limitations of the traditional classroom training model (consumption of time and lack of scalability) translate into infrequent training opportunities for cybersecurity workforce professionals.As a result, the retention and mastery of knowledge is inhibited, and exposure to the most current cybersecurity trends, technologies, and practices is limited.

Table 1 :
Four Components of an Instructional Objective

Table 2 :
CERT-Developed Training Technologies VTE Combines the components of traditional classroom training with the benefits of web-based training.Users can conveniently access VTE from their own computers and participate in training courses at their own pace.VTE contains a library of instruction and reference material about information assurance, incident response, computer forensics, and other vital cybersecurity topics.Instruction includes lectures, slides, and written material for knowledge building and hands-on exercises, lab books, and technical demonstrations for skill building.XNET Designed to address the challenges of realism and scalability of scenario-based cybersecurity exercises and simulations.XNET provides a platform for experience building by enabling instructors/trainers to create customized, full-scale cybersecurity exercises that simulate real-world scenarios and environments.Multiple instantiations of the same exercise can be deployed simultaneously to accommodate a large number of participants, and the XNET participant console is accessible via the internet to maximize the accessibility of exercises.