Competency Lifecycle Roadmap: Toward Performance Readiness (CMU/SEI-2012-TN-020

Abstract : Workforce effectiveness relies on two critical characteristics: competence and readiness. This technical note describes the Competency Lifecycle Roadmap (CLR), a preliminary roadmap for understanding and building workforce readiness developed by the Computer Security Incident Response Team (CSIRT) Development and Training team at the CERT Program, part of Carnegie Mellon University's Software Engineering Institute. This note provides an early look at the roadmap, highlights some of its uses to date, and discusses potential next steps in its development and transition.


List of Tables
1 Introduction Workforce effectiveness relies on two critical characteristics: competence and readiness.
Competence is the sufficient mastery of the knowledge, skills, and abilities-or competenciesneeded to perform a given task. 1 Competence reflects how well an individual understands a subject matter or is able to apply a given skill, and it is necessary, but not sufficient, to successfully perform a given task.Readiness is the ability to apply a set of competencies to complete a real-world task.
Consider the following scenario: A large agency recently hired several people to join its digital analytics team.The new employees' long-term job was to perform forensic evidence collection and subsequent digital media analysis in the field and back at the organization's test lab.The new hires were put through initial training to teach them how to perform these tasks, including using various forensic tools and associated analysis processes.Tools that were reviewed included applications such as EnCase2 and FTK. 3   After the new employees completed training for each technique or tool, they were tested to determine if they had acquired the knowledge, skills, and abilities needed to use it.The new team members successfully passed all of the individual tests presented to them.After they completed all of their introductory training courses, the team's leader believed that the new team members were ready for field work.However, when the new team members were presented with a compromised system to analyze in the field, none of them were able to perform the analysis adequately.They had no idea how to start the analysis and investigation process on their own.They were also unfamiliar with the media and how to collect data from it.Although they could perform parts of the process and use various tools in a classroom setting, they failed the test that mattered most.They could not adapt what they had learned to the environment of the real-world scenario.In the end, the new team members were not ready to perform their job function in the field.
In this scenario, the new team members demonstrated their competence with the suite of techniques and tools needed for their job assignments.However, they were unable to analyze a compromised system in the field.They lacked the readiness to perform their assigned tasks in real-world conditions despite having shown they possessed the required knowledge, skills, and abilities.goals.Staff must be able to show that they can apply what they have learned, and training programs must be structured to provide staff with opportunities to test that readiness before being placed in the field and confronting unexpected situations.

Competency Lifecycle Roadmap (CLR)
Researchers from the CERT ® Program at Carnegie Mellon ® University's Software Engineering Institute (SEI) created the Competency Lifecycle Roadmap (CLR), which defines a systematic approach for developing and sustaining workforce readiness over time.We designed the CLR to be domain independent so that it can be broadly applied across multiple disciplines.It takes into account a vast body of knowledge in the cognition and performance disciplines pioneered at Carnegie Mellon University, including research into the nature of expertise [Simon 1996].This roadmap comprises five core activities (assess, plan, acquire, validate, and test readiness) and two foundational elements that support those activities (criteria, environment).Section 2 of this technical note details the roadmap and its activities and foundational elements.
We envision using the CLR in many ways, including

Background
For the past several years, the CERT Program has helped client organizations improve their training and development programs.Several of these engagements focused on identifying and documenting cybersecurity competencies.However, as in the scenario, organizations began to understand that competence is not readiness.
In 2011, the CERT Program's Computer Security Incident Response Team (CSIRT) Development and Training (CDT) team chartered a project focused on building readiness within an organization's workforce.We developed the workforce readiness project based on lessons learned from identifying and documenting cybersecurity competencies for client organizations, as well as observations of work performed in related software engineering settings.The project's competency work initially focused solely on cybersecurity.However, over time the project scope broadened to include, for example, supervisors' readiness to perform their assigned leadership duties.
As we shifted the focus of our research from developing and documenting competencies to building workforce readiness, we leveraged previous SEI work in building readiness using certification programs [Behrens 2004] and cybersecurity workforce development [Hammerstein 2010].We also looked at relevant research throughout the training and development community.
In particular, we reviewed materials focused on how to build competency-based training ® CERT and Carnegie Mellon are registered marks owned by Carnegie Mellon University.
programs, evaluation and assessment methods, and methods for developing effective training and development programs.The Bibliography of this technical note lists these materials.
While the SEI has significant experience in defining and documenting cybersecurity competencies for its customers, this work is not unique to the SEI.Over the past few decades, organizations throughout the community have undertaken many similar efforts to develop and document lists of competencies.For example, both the U.S. Office of Personnel Management (OPM) and the National Initiative for Cybersecurity Education (NICE) have developed and documented competency frameworks for the cybersecurity community [OPM 2011, NICE 2011].The OPM has also developed and documented a Leadership Competency Framework that focuses on an organization's management and leadership roles [OPM 2006].

About This Technical Note
The primary audience for this technical note is managers and training officers who want to improve their organization's training and development program.Researchers focusing on training and education activities will also find this document useful.It will also benefit individuals or small working groups trying to develop their own competence and readiness to perform or improve their work.
In general, people who are interested in the following topics will find this technical note worthwhile: • competency definition and development The overarching goal of this technical note is to describe a preliminary roadmap for understanding and building workforce readiness, though the CLR is still early in its development.This note provides an early look at the roadmap, highlights some of it uses to date, and discusses potential next steps in its development and transition.Future reports will further detail the roadmap's technical aspects.
This remainder of this note comprises the following sections: The Competency Lifecycle Roadmap (CLR) is an approach for systematically building workforce competencies and maintaining them over time.The roadmap focuses on an individual's readiness to perform his or her task assignments.In this context, readiness is the ability to apply the total set of competencies (both technical and enabling) required to perform a task or set of tasks.
The roadmap comprises five core activities-assess, plan, acquire, validate, and test readinessand two foundational elements that support the activities-criteria and environment.Figure 1 illustrates the basic structure of the CLR.This section provides a conceptual overview of each roadmap activity and foundational element.

Activities
A roadmap activity is defined as a task performed to achieve a specific training and development outcome.

Assess
The first activity, assess, is an initial evaluation of key competencies and the ability to perform those competencies in a specific task.This activity should not be confused with a training assessment, which evaluates the extent to which a training course meets its objectives.In contrast, the roadmap's assessment is a performance-based test that includes measurement of an individual's current competencies.It evaluates an individual's ability to apply a stated competency, regardless of how that competency is acquired (e.g., coursework, experience, or observation).Because knowledge can be broad or specific, gradually or discretely acquired, relevant for long or short periods, or retained or lost over time, a baseline assessment of an individual's current knowledge and abilities is essential.
Assessment is important to the roadmap because this activity defines a systematic, objective, and repeatable process for establishing a baseline of strengths and weaknesses in the specific competencies needed to perform a specific task.These competencies are called identified competencies in this note. 4Assessment also provides insight into which competencies need to be maintained or improved to achieve the desired performance.In addition, as an organization assesses groupings of competencies, it gains an overall picture of an individual's relative strengths and weaknesses, which can assist that individual with professional growth opportunities.Table 1 presents the key characteristics of the assess activity.

What
• an initial evaluation of key competencies and the ability to perform them in a specific task

Why
• to identify a baseline of strengths and weaknesses in the key competencies needed to perform a specific task • to apply a systematic, objective, and repeatable process • to provide insight into how to maintain or improve the performance of identified competencies There are many different ways to perform the assessment process.Each organization will need to determine what works best in its environment.Examples of methods that might be used include the following: • Conduct a performance-based test that includes measurement of the current state of key competencies.
• Have individuals complete a skills inventory with supporting substantiation showing evidence they mastered those skills.Substantiation might be course certificates or a manager's recommendation.

Plan
The next roadmap activity, plan, defines an individual's intended course of action for maintaining or improving specific competencies that are needed to perform a specific task assignment.Table 2 presents the key characteristics of the plan activity.The plan activity is important because it specifies an attainable path that an individual can follow to maintain or improve identified competencies.Here, an individual determines which options and resources are available and relevant.Once the path, or development plan, has been developed, the individual documents it and then disseminates it to all relevant stakeholders.Planning thus lays the foundation for acquiring identified competencies, which is the next roadmap activity.
Some examples of planning methods include the following: • Map strengths and weaknesses to options and resources provided within the organization and community to develop a path for maintaining or improving identified competencies.This may often take the form of an individual development plan (IDP).
• Document and disseminate the path for maintaining or improving identified competencies.

4
The identified competencies constitute the subset of all key competencies that will be addressed.

Acquire
The acquire activity of the roadmap defines actions that will be taken to obtain the knowledge or skills required to maintain or improve identified competencies. 5Acquisition of competencies is important because it enables an individual to reinforce strengths and address weaknesses in his or her knowledge and abilities.Table 3 presents the key characteristics of the acquire activity.Validation focuses on the knowledge and abilities needed to perform a task.Its emphasis on measuring the extent to which knowledge and abilities have been acquired differs from that of the next roadmap activity, test readiness, which evaluates the application of knowledge and abilities in an actual work environment.

Test Readiness
Often overlooked or grouped with validation, the test readiness activity of the roadmap is a realworld evaluation of whether a person can perform a specific task as required.People bring a range of knowledge and experiences to any job setting or task.The initial assessment and subsequent validation will determine an individual's knowledge of and experience with certain competencies and his or her understanding of some targeted (often highly technical or organization-specific) competencies.
However, knowing an individual's current proficiency in selected competencies is insufficient for predicting that individual's overall readiness to perform a given task.An individual might have related knowledge and abilities but might not be able to apply them in a real-world setting.The ability to test an individual's readiness to perform a task is an essential component of an effective training and development program.Table 5 presents the key characteristics of testing an individual's readiness to perform assigned tasks.• interim job performance evaluations (e.g., at 30 days and 90 days after job assignment) • observation by supervisor on ability to perform as a team member or to perform a specific job function • ability to explain job tasks and concepts to newer staff More experienced individuals may assess their own readiness for new work or added responsibility by using the CLR themselves or with a supervisor or another experienced coworker who can provide objective feedback.
It is vital during this phase to understand the critical importance of real (as opposed to realistic) work.To know if someone can actually perform all of the activities related to a set of job requirements, all of the key components of that job must be made available.For example, for someone to be ready to complete a detailed report, he or she must understand more about it than how to fill in the blanks, such as how the report is communicated to others, where it is stored, and who may be using the report over what period of time.To determine that person's readiness, the entire task in its real-world context must be presented so that the staff member can demonstrate strengths and identify areas in which he or she may need additional development.
Readiness dimensions are often tailored to the job requirements and specific sets of competencies needed to both perform and excel at that job.Sometimes an individual may develop task performance over time.In other instances, depending upon the role, the task requires performance excellence, and an individual is deemed either ready or not ready to perform.

Foundational Elements
The CLR defines a foundational element as an entity that supports the execution of roadmap activities.The inclusion of foundational elements is one of the most important ways in which the CLR differs from other models and approaches to many training and development programs, whose moderate success in achieving desired outcomes may be due in part to their exclusion of such elements.These critical, enabling elements of the CLR are criteria and environment (sometimes called context).

Criteria
Criteria, the first foundational element of the roadmap, are the sets of technical and enabling competencies that define the requirements for performing tasks.Technical competencies are the subset of knowledge and abilities that directly affect the ability to perform a task.For example, a technical competency for a cybersecurity analyst is the ability to use an intrusion detection system.For a project manager, the ability to develop a schedule is a technical competency.
In contrast, enabling competencies indirectly support the completion of a task.Effective communication is an example of an enabling competency.For example, the cybersecurity analyst needs to communicate information about possible security incidents with his or her colleagues.Likewise, the project manager needs to communicate with his or her team when preparing and implementing the project's schedule of events.
Criteria establish the scope of performance requirements that define readiness to perform a task.Just as other training and competency-based programs have demonstrated, our research indicates that competencies are contextual.They work best when aligned with a role and, in particular, when the specific application for performing the role-based functions describes the competency in terms of the work that is actually done.
Table 6 defines the characteristics for the criteria element of the roadmap.
• outlined, streamlined processes for achieving assigned activities • time for management to meet with each employee to discuss career and professional development and perform a yearly assessment of the needed knowledge, skills, and abilities for performing job functions satisfactorily

Roadmap Implementation over Time
The roadmap can be used to define an individual's training development path over time.Figure 2 illustrates this process, in which the roadmap establishes an individual's progression from novice to expert for a given job assignment.It is important to note that this roadmap is not intended to be used in a linear fashion.All people have some areas of expertise, some areas that need to be developed, and perhaps some areas that are outside an individual's interest or ability.The notion of readiness may be iterative and certainly takes into account criteria at all levels, from novice to expert.While time is an important factor, it is only one indicator of the growth of competency-based readiness.Other indicators might include experience or collective abilities of a work team.

Implementation Approaches
An organization can use the CLR instrument as a guide for multiple scenarios, including This section of the technical note shows how the CLR can be used effectively for each use case.

Building a Training and Development Program
An organization can use the seven components (five activities and two foundational elements) of the CLR as a guide to building a training and development program.The CLR focuses not on completion of curricula or courses, but rather on an outcome of readiness, which may entail expansion of the training and development program to ensure its long-term success and sustainability.
To build a training and development program, an organization might perform the following steps, adapted from the work of Blank [Blank 1982] and Gott and Lesgold [Gott 2000].1. Choose criteria for competencies relevant to the roles in the organization that require training.These criteria can come from an existing set of competencies, such as the rolebased U.S. federal government's cybersecurity competencies outlined in the NICE, or they can be developed in-house.
a.If developing criteria in-house, the organization can use a role-based scenario technique that gathers information from staff on their activities and the skills and knowledge they need.
b.This data can then be synthesized into competencies or mapped to existing sets of competencies to determine the criteria.
2. Develop strategies for assessing staff knowledge and skills.A preliminary, self-administered inventory of mastered skill areas can help ensure that staff members do not pursue redundant training.It can also help management to identify projects that can benefit from particular staff skills.
3. Build a planning step into the training and development process.Individuals in the program will decide, usually based on discussions with their manager, which competencies they need to acquire, which might need to be refreshed, and which require only sustainment through professional development.Some competencies may require no additional development.
a.During the planning step, the organization should determine the most effective methods for providing knowledge and skills to the staff member.For example, some competencies, such as adjusting to organizational customer needs, may be learned only through mentoring and observation.
b.The individual and his or her manager should work together to develop a documented IDP with expected competency acquisition and completion dates.
4. Provide and support methods of competency acquisition.The organization should provide ample time for training and development activities, whether they are courses, self-paced computer-based training (CBT), mentoring, or on-the-job training.The organization can also pay for any activities that charge a fee such as formal courses or university programs.It might also need to allow trainees time away from current duties or reduce their workload until they complete the activity.All supporting information and components of this step should be added to the documented IDP established in step 3b.
5. Establish methods of validating that the completion of acquisition methods yields the desired outcome.This validation can be measured by benchmarking learning objectives, testing course content retention, or conducting scenario and role-playing activities that highlight what was learned.
6. Establish a method of readiness testing for each relevant competency or competency group.For example, in the opening scenario in Section 1: Introduction, real readiness testing would involve giving the staff member a compromised system, with no instructions, and asking him or her to determine what happened to the system and what type of malicious code it was infected or compromised by.Realistic testing would match the staff member with a more experienced partner to work out the problem as they would on a real task.It is critical to provide both the problem and the real environment or context in which the work problem is likely to occur.Without this readiness testing piece of the training and development program, organizations cannot adequately assess staff members' readiness to perform the set of competencies or the job in the field under actual working conditions.With readiness testing, organizations can assess both technical and enabling skills, including complex judgment and deductive and inductive reasoning, problem-solving, flexibility, and the ability to handle unexpected occurrences.
7. Establish the readiness testing and competency lifecycle methodology in a manner that is conducive to the culture, organizational processes, and personality of the business unit and parent company.The assessment activities need to be appropriate to the organization and must have no negative impact on day-to-day operations.CLR users can also benchmark its training and development program at a more granular, in-depth level to determine how well each component is performed.This more focused method can identify specific strengths and weaknesses and establish tailored improvement plans.For example, an organization might discover that its assessment of staff knowledge and skills did not sufficiently evaluate strengths and gaps in essential competency areas.
This more granular benchmarking can be done in numerous ways, depending on the scope of the analysis and the outcome desired:

Developing Curricula or Training Plans
Organizations can use the CLR to guide their development of training plans and curricula.The CLR can be especially useful when developing a broad curriculum, which entails many different competency acquisition methods and includes tests of skill readiness, competency, and job function level.Using the components of the CLR as a model can help build not just a set of courses, both live and virtual, but also a synergistic curriculum that focuses on the practical experience staff need to complete day-to-day tasks.
Using the CLR to help develop curricula or training plans, an organization can choose training activities that culminate in readiness testing.It also can help focus planning and acquisition methods on those requiring observation, demonstration, role-play, simulations, or shadowing a more experienced staff member.This approach not only makes the training curriculum more interesting, but it can also improve participants' understanding of readiness performance requirements.

Creating an Individual Development Plan
To create an IDP, a manager or supervisor usually works with a staff member to identify the competencies the staff member should acquire, master, refresh, or sustain based on his or her job function or role.
If possible, the staff member should then be assessed against those competencies, based on years of experience, previous job performance, certifications in the knowledge and skill domain, or similar parameters.When possible, the organization should conduct more granular assessments of current competence that employ a skill-based test and a knowledge test, much like the written and driving portions of driver's license exams.Such an assessment provides a baseline for further measurement.
To develop the IDP, the organization identifies each competency and its corresponding skills and knowledge areas.Then the organization documents appropriate, recommended training and development opportunities and determines the methods of acquisition, validation, and readiness testing.
All of this information is documented in the IDP, so both the staff member and management have an explicit understanding and agreement about competency development acquisition and measurement expectations.The organization can then use the documented IDP as an individual roadmap to track the individuals' progress toward completion and determine if and how he or she achieves readiness.

Summary and Next Steps
The CLR provides an agile, practical approach to developing and managing a competency-based staff-readiness program.We designed the CLR around the idea that staff members at all levels of expertise require periodic readiness assessments for both existing and anticipated work requirements.The CLR provides a strategy for maintaining and enhancing competence over time.
The enabling competencies are, in some ways, at the heart of this approach to long-term readiness development.While traditional training curricula may be sufficient to maintain and enhance specific skills, they are, on their own, insufficient for long-term workforce readiness.New skills must be integrated with existing skills.Organizations and their individual staff members must understand when older knowledge or skills are sufficient and when they must be enhanced or replaced with new technologies and understandings.
The CLR is still in its early stages of development.Our next steps include exploring how to describe each component in more detail.Next steps will also explore how readiness testing for more esoteric enabling competencies can be created and implemented.Specific next steps will include • building and testing assessment tools for a variety of readiness requirements at both the technical and leadership levels • piloting applications of the CLR in smaller settings with individual teams or work groups • using the CLR in a variety of benchmarking situations that might include training and curriculum design as well as mentoring and supervisory programs • documenting scenario-building techniques so they can be used for both assessment and program development • documenting the role-analysis methodology for identifying competencies for different roles • developing an assessment instrument to allow organizations to benchmark their training and development programs against the CLR so they can identify areas for improvement • exploring, in conjunction with other groups in the CERT Program, how this readiness approach can be applied to a team rather than to an individual as described in this technical note • exploring how this roadmap can be applied to cybersecurity training and development Figure 1: Competency Lifecycle Roadmap (CLR) Structure 4 Figure 2: Competency Lifecycle Roadmap (CLR): Progression over Time 10

•
building, benchmarking, and improving a training and development program • understanding personal (or team) goals for competency development and readiness improvement

•2
Section 2: The Competency Lifecycle Roadmap (CLR)-describes the five activities and two foundational elements of the CLR • Section 3: Implementation Approaches-provides a basic overview of guidelines and tools for implementing the CLR • Section 4: Summary and Next Steps-presents next steps in the development and transition of the roadmap • Bibliography-lists related publications used in creating the CLR and this technical note CMU/SEI-2012-TN-020 | 4 The Competency Lifecycle Roadmap (CLR)

•
time for staff to pursue training and development activities, even providing training as a work assignment • a centralized tracking system to allow management and staff to track training plans and accomplishments • a culture of training and education within the organization that recognizes the importance of developing and sustaining competencies and encourages such pursuits through verbal communication and dedication of time and resources • recognition by staff and management that training and development is more than just completing yearly compliance modules for ethics, security, privacy, and other such practices

•
building an initial training and development program • benchmarking its training and development program and identifying gaps and areas for improvement • developing curricula or training programs • helping an individual or team set personal goals related to a specific job or task

•
self-assessment by a training officer or training group • focus group involving a representative sample of staff, management, and human resources or training team members • survey or a series of workshops that asks staff and management how each of the five CLR activities (assess, plan, acquire, validate, and test readiness) are performed Group discussions can focus on the training and development program's competency criteria and their enablers or constraints.More open-ended discussions might generate feedback on which areas of the program need improvement.These discussions can also provide a good understanding of the staff members' general like or dislike of the existing program, which may shed light on how well they use it.

Table 1 :
Characteristics of Assess Activity 5

Table 1 :
Characteristics of Assess Activity

Table 2 :
Characteristics of Plan Activity Dimension DescriptionWhat• a course of action intended to maintain or improve identified competenciesWhy• to specify an attainable path for maintaining or improving identified competencies • to communicate the path for maintaining or improving identified competencies

Table 3 :
Characteristics of Acquire Activity • a training course or curriculum • mentoring or other on-the-job training opportunities, such as ride-alongs • shadowing management or other subject matter experts • a realistic simulation environment • targeted self-study (e.g., technical journals, online discussions, or topical blogs) • conference attendance and participation • academic coursework or degree programs2.1.4ValidateValidateistheroadmap activity that measures whether an individual's training and development actions have addressed his or her competency needs.Validation of acquired competencies is achieved by conducting a performance-based test to determine if an individual has maintained or improved identified competencies through his or her actions.It defines a structured approach to measuring knowledge and abilities that have been acquired.Table4presents the key characteristics for the validate activity.

Table 4 :
Characteristics of Validate Activity • observation of employee demonstrating what was learned 5 For further discussion of competencies and training, see Handbook for Developing Competency-Based Training Programs [Blank 1982].

Table 5 :
Characteristics of Test Readiness Activity Dimension DescriptionWhat• a real-world evaluation of whether a specific task can be performed as requiredWhy• to ensure that competencies can be appropriately applied to tasksMethods for performing readiness testing can include • real-world scenario • role-playing • capstone exercise • real-world simulation • observation of real-world task performance Readiness is best evaluated using a multidimensional, performance-based evaluation of task performance.The multiple dimensions used to test readiness comprise several inputs, including • outcomes from assessments • outcomes from evaluations • interviews with supervisor(s)

Benchmarking a Training and Development Plan or Program Organizations
8. Identify and develop enabling factors, such as time, funding, and other resources, to ensure that the training and development program supports the entire team or workforce involved.Organizations may need to establish an institutionalized process for the training program.canuse the CLR to benchmark already established training and development programs.Benchmarking allows an organization to understand its program's current strengths, weaknesses, gaps, and goals.While not prescriptive, such benchmarking can shed light on unaddressed areas, particularly those regarding competency development beyond formal training courses.An individual (training manager, supervisor, or other designated professional) or group within the organization can self-administer the CLR for benchmarking, or an external party can be involved to reduce bias.The organization can evaluate how well its current training and development program addresses each of the CLR components (five activities and two foundational elements).This benchmarking exercise would identify any gaps in the program, such as a lack of readiness testing or a lack of standard performance criteria for a work assignment.This method can also help to identify and prioritize improvement activities to address any gaps.The organization might determine which CLR components need to be incorporated into its training and development program and form plans to do so.If an organization adds a component of the CLR, it should consider the relevant steps and issues outlined in this technical note.