Report from the First CERT-RMM Users Group Workshop Series

ix


List of Tables
The purpose of this report is to describe the first CERT ® Resilience Management Model (RMM) Users Group (RUG) Workshop Series and relay experiences of members who participated in it and CERT staff who conducted it.The RUG workshop was originally conceived as a means to help CERT Resilience Management Model (CERT-RMM) users progress in their adoption of the model and get practice using it after taking the three-day Introduction to CERT-RMM course.The workshop was also intended to 1) help CERT staff members understand the requirements necessary to implement CERT-RMM and 2) develop materials that would help users put CERT-RMM practices into action on their specific improvement projects.

Structure of This Report
Section 2 provides a brief overview of first CERT-RMM Users Group Workshop Series and CERT-RMM.
Section 3 provides background about the first RUG Workshop Series and the steps RUG members took during the preparatory phases.
Sections 4−7 outline the preparation, topics, results, and next steps for the four workshops that constituted the first RUG Workshop Series; these workshops were held March 2011, May 2011, August 2011, and January/February 2012.
Section 8 lists suggestions for improving future RUG workshops.
The appendix includes the RUG panel presentation given at the Software Engineering Institute's (SEI's) 2012 Software Engineering Process Group Conference, North America.The presentation includes slides provided by four of the five RUG member organization.
® CERT is a registered trademark owned by Carnegie Mellon University.
2 Overview of the RUG Workshop Series and CERT-RMM

Overview of the First CERT-RMM Users Group (RUG) Workshop Series
The purpose of the first RUG Workshop Series is to offer RUG members an opportunity to engage in customized collaborative discussions, hands-on activities, and workshop exercises and assignments that help them to • implement a solution that meets a specific resilience improvement objective that is tied to an organizational goal • improve the effectiveness and efficiency of operational risk management activities • diagnose their current resilience activities against CERT-RMM processes and practices The RUG Workshop Series comprised four workshops, which took place between March 2011 and February 2012.RUG members were interviewed in advance of the first workshop so that the RUG Development Team (RDT) could better understand their objectives and requirements and use these to shape the RUG Workshop Series. 1 In addition, between workshops, the RDT held periodic conference calls to discuss issues and ongoing preparatory assignments.Members benefited by having access to such a diverse set of organizations that represented several of the market sectors to which CERT-RMM applies.This organizational diversity was often cited by members and CERT staff as one of the key benefits of the first RUG Workshop Series.

Overview of CERT-RMM
CERT-RMM is a capability-focused maturity model for process improvement that reflects best practices from industry and government for managing operational resilience across the domains of 1 The RDT consists of CERT staff members who support the RUG Workshop Series.
security management, business continuity management, and aspects of information technology (IT) operations management. 2 CERT-RMM defines operational resilience as the emergent property of an organization that can continue to carry out its mission in the presence of operational stress and disruption that does not exceed its limit Through CERT-RMM, these best practices are integrated into a single model that provides an organization with a transformative path from a silo-driven approach for managing operational risk to one that is focused on achieving resilience management goals and supporting the organization's strategic direction [Caralli 2011].
CERT-RMM incorporates many proven concepts and approaches from the SEI's process improvement experience in software and systems engineering, service engineering, and acquisition.Foundational concepts from Capability Maturity Model ® Integration (CMMI ® ) are integrated into CERT-RMM to elevate operational resilience management to a process approach and provide an evolutionary path for improving capability. 3Practices in the model focus on improving the organization's management of key operational resilience processes.This improvement enables highvalue services to meet their mission consistently and with high quality, particularly during times of stress and disruption [Caralli 2011].
CERT-RMM helps to ensure that the organization's important assets-people, information, technology, and facilities-effectively support business activities and services.The model serves as a foundation from which an organization can measure its current competency, set improvement targets, and establish plans and actions to close any identified gaps.As a result, the organization repositions and repurposes its security, business continuity, and IT operations activities and adopts a process improvement mindset that helps to keep services and assets productive in the long term [RUG 2011].
The context for CERT-RMM is shown in Figure 1.The RDT developed an initial architecture that described the intent of each segment of the first RUG Workshop Series.The architecture provided prospective members with an idea of what was to come and provided RDT staff with a roadmap to follow as the series progressed.The architecture was reviewed by members and updated in advance of each individual workshop.
We summarize the first RUG Workshop Series architecture, as it evolved, in the following sections.The architecture describes advance preparation for the entire workshop series and for each individual workshop, topics to be covered in sequential order, takeaways, outcomes, and expected preparation for each subsequent workshop.
As a result of conducting the first RUG Workshop Series, we developed an improved architecture, which we will use to guide future workshops.We summarize key improvements in Section 8 of this report.

Advance Preparation
Representatives from all member organizations were interviewed in advance of the first RUG Workshop Series.These interviews occurred from October 2010 to the start of Workshop 2 in May 2011, when the CMU team joined the effort.As part of each member interview, the RDT asked the questions that are listed in Table 1.We also required that members The RDT analyzed and synthesized interview data and determined that this particular group of members would be most interested in participating in a workshop that focuses on CERT-RMM model implementation and improvement, which involves leading RUG members through a CERT-RMM-based improvement cycle using member-declared improvement objectives that meet organizational goals.
At this point in the preparatory phase, none of the member organizations were interested in pursuing lead appraiser or instructor certification.Because of this, the RDT did not include this topic in the RUG Workshop Series.We subsequently learned that some members of the CMU and Lockheed Martin (LM) teams did have such interests; therefore, we took follow-up action to remedy this after Workshop 4.

Workshop 1: Planning
RUG members from the DFS, LM, the USPIS, and the CERT RDT attended this workshop, which was held March 15-16, 2011 in Pittsburgh, PA.The CMU team did not join this workshop series until Workshop 2.

Advance Preparation
In preparation for Workshop 1, we asked members to prepare slides that provide a brief introduction of themselves, their organization, their current resilience activities (if any), and the top three to five resilience concerns and issues.
We also asked members to think about improvement objectives that could be implemented in a project lasting 10-12-months.Members came to Workshop 1 prepared to discuss a small number of improvement objectives that they wished to declare and the rationale for each objective.We set expectations with members so they understood that the improvement objectives were to be business oriented and operational in nature, not specific to CERT-RMM.

Topics
Table 2 lists the topics that we presented and discussed during Workshop 1. • member feedback about Workshop 1 Prior to the workshop, members had selected candidate improvement objectives based on a range of factors including the following: • respond more effectively to high-profile, high-impact incidents • work more effectively with supply chain partners 4 For more information about the CERT Insider Threat Project, visit the CERT website (http://www.cert.org/insider_threat).
• satisfy specific compliance requirements more effectively • integrate aspects of CERT-RMM with current standards and process models • address directives from senior executives Throughout this workshop series, we regularly discussed the definition of operational resilience (presented in Section 2).We also discussed topics related to deriving specific interpretations of operational resilience for • information security, business continuity, IT operations, and software/system development • each member's specific business.We asked ourselves, "What does operational resilience mean to us and our ability to fulfill our mission and meet business objectives?" Ongoing interpretation and tailoring of the intent of CERT-RMM as applied to each member organization's improvement project occurred throughout the first RUG Workshop Series.
For organizational and model scoping, discussions included fine-grained scoping options based on CERT-RMM processes and practices of interest, the selected organizational unit(s) that will benefit from the improvement, the differences between CERT-RMM and CMMI (for those organizations that have already adopted it), and other caveats.Discussions also included whether the scope should be at the process area (PA) level or the practice level.Members were encouraged to choose their scoping granularity based on their process-improvement objectives.To facilitate this process, the RDT distributed a spreadsheet that could be used for CERT-RMM model scoping.
Workshop members spent considerable time honing their respective improvement objectives.

Outcomes
Takeaways and outcomes from Workshop 1 included the following: • considerations for refining improvement objectives − a list of the objectives that were considered and rejected with rationale • described the organizational scope (one or two slides, including an illustration of the scope on a member organizational chart) • depicted the CERT-RMM model scope by process area and practice (We provided an Excel spreadsheet to members.) • developed a sponsor presentation (i.e., a plan to garner support for the improvement project from the project sponsor) 5 Workshop 2: Improvement Objective RUG members from the CMU, DFS, LM, and USPIS project teams attended this workshop, which was held May 10-11, 2011 in Pittsburgh, PA. Between Workshops 1 and 2, the RDT decided that the first RUG Workshop Series represented an excellent opportunity to identify a CERT improvement objective and have a CERT team participate in the RUG in the same fashion as the other member teams.Thus the CERT Resilience Enterprise Management (REM) Team also attended and presented at Workshop 2.

Advance Preparation
In preparation for Workshop 2, members completed the assignments described in Section 4.4 of this report.

Topics
Table 3 lists the topics that we presented and discussed during Workshop 2.  Key discussion points included CERT-RMM variation of the IDEAL model, since this made more sense to RUG members.During Workshop 2, participants started to tease out key characteristics of effective and ineffective organizational scope and model scope.For example, workshop members noted that it is important to ensure that the gaps selected for action resulting from a diagnosis are under the control of the sponsor.This will allow organizations to make forward progress unencumbered.In addition, workshop members noted that when organizations are just getting started, it is important to select a narrow model scope-perhaps just a few specific practices (SPs).(This is an example of the proverb "walk before you run.") When performing a diagnosis, it is important to frame diagnostic questions in language that is meaningful to participants.Additionally, one member stated that when performing a diagnosis, it is helpful to have a wide range of objectives beyond the stated improvement objective.It is beneficial to ask questions such as • Did this approach work?
• How well did it work?
• If the approach did not work, how should we handle similar situations in the future?

Outcomes
Takeaways and outcomes from Workshop 2 included the following: • a greater appreciation for the thought and effort required to effectively define organizational scope and model scope so that you can make forward progress in a reasonable period of time RUG members from the CMU, DFS, LM, USPIS, and CERT REM project teams attended this workshop, which was held August 30-31, 2011 in the SEI, Arlington, VA office.

Advance Preparation
Advance preparation for Workshop 3 consisted of completing the assignments described in Section 5.4 of this report.

Topics
Table 4 lists the topics that we presented and discussed during Workshop 3. • an overview of the work of the CERT Network Situational Awareness Team 9 • an overview of the CERT work in standards-based, automated remediation 10 • member feedback about Workshop 3 Some of the criteria and success factors that members shared for determining when to commit resources to define a process include the following: The process is highly repeatable.

•
The process, while performed infrequently, needs to be completed the same way each time it is performed.(It is important that the process is executed consistently by those performing it.)

•
It is important to understand the current process before altering it.

9
For more information about the work of the CERT Network Situational Awareness Team, visit the CERT website (http://www.cert.org/netsa/).10 For more information about the Security Content Automation Protocol (SCAP), refer to the NIST SCAP website (http://scap.nist.gov).

•
Measure process performance for the purpose of improvement.
• There are risks associated with different staff members performing the process differently.

•
It is important to meet compliance obligations.
• Capture essential corporate knowledge.

•
Clearly communicate to staff members what needs to be done.
• Establish a baseline for measurement.
• Link policy to procedure to process, and help to enforce policies.
Members suggested collecting and analyzing the following measures that characterize RUG performance and success.Several of these measures will be more meaningful when collected over the course of multiple workshop series.(This list is presented in no particular order.) • number of organizations participating • number of improvements identified; number of improvements made • change in member improvement-project objectives (quantity, scope, etc.) • change in member improvement-project scope (e.g., the number of total organizations participating, number of CERT-RMM PAs, SPs, or GPs [generic practices]).(This measure and the one before it will help set member expectations for future RUGs.) • ability to accommodate member ingress and egress • reduction in barriers to becoming a member of the RUG (For example, it would be easier to participate if the RUG offering included the Introduction to RMM course.) • number of changes/improvements (to the agenda, homework assignments, architecture, etc.) in response to members' suggestions 7 Workshop 4: Improvement Progress RUG members from the CMU, DFS, LM, USPIS, and CERT REM project teams attended this workshop, which was held January 31 to February 1, 2012 in Tampa, FL. Due to the extended time between Workshops 3 and 4, the RDT scheduled a check-in call with available members on November 30, 2011.

Advance Preparation
Advance preparation for Workshop 4 consisted of completing the action items and assignments described in Section 6.4 of this report.

Topics
Table 5 lists the topics that we presented and discussed during Workshop 4. • plans for future RUG Workshop Series; members noted that the biggest stumbling block to increasing RUG participation is selling the value proposition within their organizations • an overview of the upcoming CERT research in resilience case analysis • member feedback on topics to be covered in the RUG Workshop Series report (this document) • member feedback about Workshop 4 and the first RUG Workshop Series Characteristics that may support effective socialization of CERT-RMM within organizations include the following: • Choose an organizational scope and model scope that you can control so that you are able to make forward progress and demonstrate results without having to convince other stakeholders.
• Understand your culture in terms of how best to introduce a new idea.
• Understand the business rhythm to help determine the appropriate ongoing processes and activities to attach this work to.For example, strive to capitalize on a current, hot initiative and demonstrate how to add value.

•
Use the terms and language of the organization, not those of CERT-RMM.For example, use the term "proof of concept" instead of "appraisal."Several members advocated "putting the book in the drawer."In other words, they advised their peers not to showcase or focus on CERT-RMM; instead, they should start with the organizational problem being addressed and keep the knowledge of the model within the improvement team.Frame assessment questions in terms that are meaningful to those being interviewed; avoid using "model-ese." • Build upon diagnostic methods, such as the audit and compliance processes, that are already being used within the organization; add CERT-RMM constructs to these methods.
• Select a specific and narrow topic as you begin to assess your current state.For example, ask yourself − How do we escalate incidents?
− What are the various thresholds for involving specific levels of management?
• When presenting to senior executives, keep in mind their WIIFM (What's In It For Me?) perspective.
Several RDT members who have participated in CERT-RMM appraisals have indicated that you truly do not understand how to fully apply the model until you conduct an appraisal.
Members discussed additional criteria for investing in process definition, including when the activity involves cross-functional areas or groups and when the activity is related to processes that must be done the same way even if they are done infrequently (examples include generating a VPN 11 certificate and configuring a laptop).
CERT-RMM has been used for the following purposes: diagnosis, internal improvement, policy review, intent review, strategic planning, comparison with peers within a given market sector, gap analysis, and independent validation of process improvement efforts not based on CERT-RMM.
Members provided the following feedback on the benefits of participating in the RUG Workshop Series: • They have a better understanding of CERT-RMM and how to implement it.

•
It was valuable to have practical guidance on how to integrate business continuity/disaster recovery concerns with those of information security.

•
It was beneficial to work with a knowledgeable, committed support group rather than going it alone.
• Expanding the scope of their improvement projects beyond information security was a worthwhile pursuit.
• It was useful to take a project from beginning to end.

•
It was helpful to work on bigger pain points and the highest priority issues.

•
It was helpful to have the opportunity to demonstrate success before evangelizing CERT-RMM to others.

11
VPN stands for virtual private network.

•
It was helpful to have the opportunity to tackle a problem strategically as contrasted with the more typical, more tactical technology-centered problem.
• It was valuable to 1) have diversity in member organizations and 2) hear their respective views while recognizing that it would also be useful to participate in a sector-specific RUG to address common problems.

•
It was valuable to hear about the CERT-RMM product and service lifecycle and how it has been applied-from service and solution development to operations.

•
It was beneficial to be part of the beginning of something-to be on the ground floor in starting a new community.Other recommended qualifications include finding someone who is a strong intermediary, facilitator, and integrator and who is comfortable operating in "stealth" mode.(See the comments on page 17 about putting the book in the drawer and presenting the model in terms that are meaningful to the audience.)Members need to be well respected by their co-workers and be able to command organizational attention when they recommend change.It is also useful for members to have experience with organizational change.
In terms of member qualifications, members would not recommend those who require more detailed, prescriptive guidance and those who are advocates of a "by the book" approach to process.
To increase the likelihood of success, the RDT intends to use these characteristics as criteria for evaluating future RUG members.

RUG structure
Four 2-day workshops conducted over 10-12 months; conference calls in between workshops Members identify needs and objectives for their specific workshop series in advance

Results
• Added questions to 8 SCAMPI Bs across two product lines • Were there resilience gaps on our programs?
-No obvious ones, although there were areas that could be improved • Could we incorporate elements of this model into our SCAMPI Bs?
-Quite easily, even for those B/C Team Leads that didn't have training in CERT-RMM CERT Resilience Enterprise Management Team

REM overview
CERT Resilience Enterprise Management team responsible for CERT-RMM: • development and transition • training • appraisals • users group • licensing and certification • application and tailoring of the model for customer engagements Figure 1: CERT-RMM Context 4 Figure 2: The SEI IDEAL Model [McFeeley 1996] 9 Figure 3: CERT-RMM Improvement-Project Lifecycle 11

•
complete the Introduction to CERT-RMM course • become familiar with CERT-RMM publications, webinars, podcasts, and the book CERT Resilience Management Model: A Maturity Model for Managing Operational Resilience [Caralli 2011], which describes CERT-RMM Version 1.1 • prepare a 15-20 minute presentation that addresses the following questions: − What are you doing today about resilience?− What are your top three to five issues and uncertainties in your current resilience practices?− What measures do you use to manage resilience?

• 13 6
methods and templates for determining practices and gaps in practice to meet improvement objectives based on the model scope • refinements to the organizational scope and model scope based on the planned diagnosis • a CERT-RMM improvement plan and approach based on the IDEAL model, as modified for CERT-RMM (See Figure 3.) • beneficial insights from information-sharing sessions and feedback CMU/SEI-2012-TN-008 | 12 5.4 Preparation for Workshop 3 In preparation for Workshop 3, members • updated improvement objectives, the organizational scope, and the model scope based on Workshop 2 discussions and the diagnosis of their current state • provided the results of their diagnosis (by practice, across organizational scope) and submitted the results to the RDT two weeks before Workshop 3. We asked members to include the following information: − strengths, opportunities, weaknesses, and gaps (local and systemic) − feedback on the use of the diagnostic method of choice (Members used either CERT-RMM Compass or SCAMPI.)− activities required to prepare the organizational team to conduct the diagnosis (e.g., awareness, training, ability to participate in the diagnostic process) − recommendations about diagnostic activities (e.g., what worked well, what did not work well) − prioritized actions resulting from the diagnosis − examples of useful artifacts (e.g., processes, checklists, templates, methods, and tools) • participated in teleconferences between Workshops 2 and 3 to assist with diagnosis, as needed CMU/SEI-2012-TN-008 | Workshop 3: Diagnosis , guidelines for determining organizational scope, and CERT-RMM scope, plans for the workshop series Workshop 2: Improvement objective • finalize objective, prepare for diagnosing current state, prepare for improvement Workshop 3: Diagnosis • present diagnostic results, evaluate and prioritize gaps, select 1-2 gaps for improvement, prepare for process definition and measurement Workshop 4: Improvement progress • report on actions taken, lessons learned, next steps, defined processes, defined measures • post-RUG 60-day check-in CMU/SEI-2012-TN-008 | 23 16 Lockheed Martin IS&GS © 2012 Lockheed Martin Corporation.All rights reserved.Lockheed Martin, Information Systems & Global Solutions (IS&GS)• 30,000 highly skilled professionals bringing together the full range of the corporation's information competencies in information technology solutions, management services, and advanced technology expertise.• Nearly $10 billion in sales • Leading federal services and information technology contractor with a strong heritage of delivering worldclass solutions and delivering advanced technology across a broad spectrum of civil and defense domains.

Table 2 :
Workshop 1 Topics • first RUG Workshop Series initial architecture and plan • member organization (initially DFS, LM, and the USPIS) presentations including an organizational overview, objectives for the first RUG Workshop Series, and initial improvement-project objectives • a presentation by LM Enterprise Business Resilience Services that described the company's initial experiences in − selecting CERT-RMM as its improvement model of choice − conducting CERT-RMM appraisals for improving corporate-wide business continuity, IT disaster recovery, crisis management, and pandemic-planning activities • a presentation template for gaining senior-management sponsorship • organizational scoping, applied to each member's improvement objective • CERT-RMM model scoping, applied to each member's improvement objective • an overview of the CERT Insider Threat Project 4

Table 3 :
Workshop 2 Topics Workshop 2 Topics • member organization (CMU, DFS, LM, USPIS, and CERT REM) presentations including an updated improvement-project objective, organizational scope, model scope, and sponsor presentation • project-improvement method/lifecycle based on the SEI's IDEAL model, as modified for CERT-RMM 5 (See Figure 3.) • organizational and model improvement scope, and whether the intent of the diagnosis is to evaluate an intent to improve (as described in policies and plans) or the actual implementation • the distinctions between SCAMPI class A, B, and C appraisals and CERT-RMM Compass 6, 7 • the types of evidence that are collected and reviewed during diagnosis/appraisal, their distinctions (e.g., all policy artifacts, all affirmations, or all implementation artifacts), and the value of using a consistent body of 6 SCAMPI stands for Standard CMMI Appraisal Method for Process Improvement.For more information about SCAMPI, see the 2011 handbook titled Standard CMMI Appraisal Method for Process Improvement (SCAMPI) A, Version 1.3: Method Definition Document [SCAMPI 2011].7A Compass survey is closest to a class C but does not require a plan and does not cover the generic goals (GGs) and GPs.8For more information about the work of the CERT DIID, visit the CERT website (http://www.cert.org/forensics/).

Table 5 :
Workshop 4 Topics • member organization (CMU, DFS, LM, USPIS, and CERT REM) presentations describing improvement-project results to date, defined processes, defined measures, lessons learned, effort expended, and post-RUG next steps• summaries by two member organizations that continue to define, refine, and narrow the model scope to ensure that near-term, measurable results can be produced Analyzes current and future IS&GS business directions to identify and deploy the management and engineering processes needed to enable affordable performance on programs and address our rapidly evolving customer environment.• Assists product lines with understanding and benchmarking program performance through process analytics and Lean/Six Sigma activities to ensure Performance Excellence.• Provides ISO Standards Leadership, CMMI consultation and benchmarking, ITIL consultation, Agile technical support, and Services analysis and benchmarking.Great fit for charter of Strategic Process Engineering -Identify and deploy processes to support our programs -Benchmark our programs to determine current gaps -Complements current work in CMMI® and ISO 27001 CERT® RMM Users Group • Helped us understand current usage of model constructs • Helped us understand current issues companies face in resilience management • Provided different perspectives on the various process areas -So that we more fully understood intent • Allowed us to get involved in the interchange of ideas surrounding resilience 20 © 2012 Lockheed Martin Corporation.All rights reserved.
• • Great fit for LMCO and IS&GS business -Internally and for our customers • Great fit for current IS&GS needs -Cyber Security, Cloud Computing • Example • As CMMI practitioners, we thought we had a thorough understanding of Risk Management -CERT-RMM Users Group helped us understand that more is required for resilience

•
Organizational strategy -enterprise wide initiative • Concept of "acceptable risk" • Risk tolerances and appetite for organization • Extended risk sources and categories -focused on operational risk How We Used the Model • Introduced CERT-RMM-focused questions into our CMMI-DEV SCAMPI B Interviews