CMMI High Maturity Measurement and Analysis Workshop Report : March 2008

Abstract : Organizations are increasingly looking for guidance on what it takes to implement Capability Maturity Model Integration (CMMI) high maturity practices and how to sustain their momentum for improvement. As high maturity organizations work to improve their use of measurement and analysis, they often look to examples of successful implementations for guidance. In response to the need for clarification and guidance on implementing measurement and analysis in the context of high maturity processes, members of the SEI s Software Engineering Measurement and Analysis (SEMA) initiative organized a workshop at the 2008 SEPG North America conference to bring leaders in the field together at a forum on the topic. Other workshops will be held as part of an ongoing series to allow high maturity organizations to share best practices and case studies.

More and more organizations are striving for and reaching high maturity status, yet there is still an insufficient shared understanding of which measurement and analysis related practices are appropriate for high maturity organizations.Although Capability Maturity Model ® Integration (CMMI ® ) provides high-level guidance, some organizations struggle to find an effective path to high maturity, and those that have reached it must persist in evolving their efforts in the spirit of continuous improvement.As a result, organizations are increasingly looking for guidance on what it takes to reach CMMI high maturity status and how to keep improving once they get there.

Overcoming Barriers to High Maturity
The Software Engineering and Measurement and Analysis (SEMA) initiative at the Software Engineering Institute (SEI) works with organizations to develop, evolve, and evaluate measurement and analysis practices. 1 SEMA researchers have identified several barriers to CMMI high maturity during their work in the field.In this section, these barriers are described and solutions are presented.IPPSS and DPPSS courses have reaffirmed that the domain knowledge used to identify the proper set of factors (y's and x's) remains the greatest challenge, not statistical knowledge.The courses include job aids that minimize the memorization of statistics and statistical theory.Using these job aids, students are almost unanimous in the assessment that domain knowledge remains the greatest challenge.Students still gain a sufficient understanding of statistics to recognize and avoid common misuses and know when to ask for help from coaches.They are also asked to seek out mentors in the workplace who possess the expertise to solve real-time problems and drive compelling business improvements.This coaching and mentoring structure is taken from the Six Sigma realm in which people in a hierarchy of "belts" coach one another.Coaching is the single aspect that has best enabled the successful use of Six Sigma over the past 20 years.

Adoption of Process Performance Models Needs to Be Accelerated
SEMA researchers realized community adoption of process performance models needed to be accelerated to meet immediate business needs and show return on investment for CMMI business improvement.Thus, instead of waiting the projected five to seven years for statistically based process performance models to become more widely adopted, SEMA aims to help the community achieve significant adoption in the next two to three years.An accelerated schedule will also be in keeping with planned CMMI model changes and the rollout of additional CMMI constellations.
Of special note, discrete event simulation, an effective modeling approach, has already been widely adopted by the services community to predict things like cycle time, workflow bottlenecks, wait times, and queue lengths. 3

Experienced Coaches and Information Sharing Are Needed
Compared to the size of the CMMI Lead Appraiser community, the size of the CMMI high maturity coaching and mentoring community is very small.During 2007 and 2008, it became apparent that a number of SEI clients who wanted to pursue CMMI high maturity needed hands-on coaching related to the development of process performance models.The need for coaches and mentors knowledgeable about CMMI high maturity topics could rapidly exceed the need for appraisal services, and SEI Partners offering high maturity coaching could be in much greater demand.
The SEI is establishing an SEI CMMI-Six Sigma Belt certification program that will provide Black Belt and Master Black Belt coaches via an SEI Partner list.This approach will provide a venue for small-to medium-sized organizations to network and learn from others that are also developing process performance models.4

Lead Appraisers Need Experience Evaluating High Maturity Measurement Activities
The CMMI High Maturity Lead Appraiser Oral Exam is an opportunity for appraisers to demonstrate their knowledge of required topics, discuss professional experiences, and show their under-
For further information, see Moving Up the CMMI Capability and Maturity Levels Using Simulation [Raffo 2008].
standing of the roles and responsibilities of a High Maturity Lead Appraiser.During the first year the exam was given, it became apparent that many Lead Appraisers were disadvantaged in evaluating evidence during CMMI High Maturity SCAMPIs.Many lacked direct experience in conducting process performance modeling and had not observed high maturity organizations performing process performance modeling.As a result, they had little frame of reference to evaluate evidence of process performance modeling during the SCAMPI A's.From this perspective, workshops would illuminate an entire landscape of modeling that participants could use to further their professional development and assist clients seeking additional guidance on process performance modeling best practices.

High Maturity Practices Workshop Series
A series of twice-yearly SEI workshops has been planned to address the challenges and community needs described in Section 1.1 by encouraging organizations to share their experiences throughout the wider community.The workshop format was selected to allow organizations to share lessons learned in deployment, adoption, and institutionalization of CMMI process performance baselines and models with the goal of improving the practice of and value added by measurement and analysis in high maturity organizations.

Workshop Participants and Goals
The focus of the first workshop was building and using CMMI process performance models.Participation was limited to a small group of organizations who were early adopters of process performance models and baselines and was by invitation only.Representatives from Hill Air Logistics Center, Lockheed-Martin, Northrop Grumman, and Raytheon attended.
The main goals of the workshop were to • allow CMMI high maturity organizations to share best practices and case studies • identify ways to develop CMMI high maturity measurement and analysis practices and accelerate their adoption • enable networking among practitioners

Workshop Structure
The workshop was scheduled for two days and was held in conjunction with the SEPG North America conference March 17-20, 2008, in Tampa, Florida.The workshop began with an SEI presentation summarizing the interpretation of process performance models and baselines and an overview of the SEMA CMMI high maturity project.This presentation is available at http://www.sei.cmu.edu/sema/presentations/hmworkshop.pdf.
Each organization gave a 20-minute presentation summarizing its past experiences and future plans related to the following topics: • barriers faced • lessons learned in the deployment, training, adoption, and institutionalization of CMMI process performance baselines and models • best practices and examples of valid, practical methods for implementing process performance models and baselines • data quality and integrity issues • plans for modeling over the next three to six months, including the nature of the performance outcomes and drivers most likely to be investigated • suggestions for subject matter to include in future SEI state-of-the-practice studies

Barriers Faced
The organizations noted that they faced challenges in the following areas: 1. establishing the value of developing and using process performance models (PPMs) and baselines (PPBs) 2. convincing project managers to collect new measures to be used for their PPMs and PPBs 3. retaining consistent operational definitions as the scope of their measurement and analysis activities expand The participants expressed a strong consensus about these barriers.As with any new initiative or tool adoption requiring significant investment, the business value of statistically based process performance models must be communicated.Without immediate help in this area, many participants felt that their efforts to convince management and the organization to collect additional measures, with consistent operational definitions, would be an uphill battle.Although participants agreed that domain knowledge was the greatest challenge, there remains a moderate degree of effort to create models.Several participants recounted that the effort to create individual models took several weeks or months.For some, this was unexpected as they incorrectly believed that modeling was a one-pass approach warranting only several hours of effort.

Lessons Learned
Among the lessons learned, the workshop attendees noted that • useful PPMs and PPBs require domain and statistical knowledge.Neither alone is sufficient.
• coaching and mentoring are critical elements of the adoption strategy when developing and using the PPMs.This includes guidance on possible decisions and actions associated with results from PPMs and PPBs.
Participants echoed the need for domain knowledge in developing models.Some even shared experiences in which statistical experts lacking sufficient domain knowledge created models that had little value to the organization and its projects.For this reason, many participants noted that their organizations were striving to involve a variety of domain experts in developing models.
A number of organizations noted that their modeling experts were located centrally in the organization rather than at the project level.This unfortunately created challenges in domain relevancy and made it difficult to have a project-level focus in the models to aid actual project execution.
All of the organizations represented in the workshop made use of some form of structured coaching and mentoring.Although this manifested predominantly in the form of Six Sigma belts, several of the organizations used other methods of coaching and mentoring for modeling.When questioned by the SEI group, there was little recognition of the possible need for upward coaching and mentoring.The SEI group shared the notion of upward mentoring as a possible improvement in guiding middle and upper management in CMMI high maturity practices, specifically in the use of process performance modeling to manage the organization and projects.For mentoring to be successful, process and behavioral changes are needed from both project personnel and management.For example, upper management may benefit from coaching to enable them to correctly interpret and use at the organizational level the results of the analyses conducted at the project level.
• measurement system evaluations to identify the degree of noise in data due to the measurement process • discrete event simulation to facilitate Lean Six Sigma improvements Again, the SEI team was surprised by the variety of modeling techniques participating organizations planned to use.This demonstrates that the community is not fearful of statistical or modeling techniques.What may be the challenge, as discussed earlier, is that the community needs to learn how to more effectively apply process performance modeling in support of project execution.In this manner, the organization will have institutional learning and the ability to affect real-time execution of projects towards successful outcomes.

Subject Matter for State-of-the-Practice Studies
Workshop participants suggested the following topics of interest for future SEI state-of-themeasurement-practice studies: • adoption and use of measurement and analysis related to high maturity practices, particularly the use of PPMs and PPBs • balancing statistical and domain expertise to develop and sustain the value and use of PPMs and PPBs • bases for choosing critical subprocesses to place under statistical control • ways in which to develop a collection of useful PPMs • data archeology (i.e., creating baselines from paper records for previously unmeasured attributes) • data quality and integrity The SEI team reconfirmed topics for the 2008 State of the Measurement Practice survey using the suggestions of workshop participants.Due to the need for an unusually detailed survey of high maturity organizations, the SEI decided to conduct two annual surveys on the state of the practice in measurement: 1) a survey for the general community, and 2) a survey targeting CMMI high maturity organizations.In this manner, each subpopulation could be given information pertinent to their perspectives and needs.

Misconceptions About Process Performance Models (PPMs) Need to Be Dispelled
Are NeededSEMA launched two new measurement training offerings in 2007 and 2008: Improving Process Performance Using Six Sigma (IPPSS) and Designing Products and Processes Using Six Sigma (DPPSS).2Theintent of the courses is to expand the use of statistical modeling, including various forms of regression, simulation, and probabilistic modeling.These courses teach the use of logistic and dummy variable regression in addition to traditional simple linear multiple regression so practitioners can use modeling techniques that support both continuous and discrete data types.Students attending the courses sought a wider array of industry examples showing how different kinds of process performance models could lead to better performance outcomes.If a wider set of examples is collected, future course updates could include example modules that are more closely related to the domain and frame of reference of the students.Also in recent years, a number of CMMI high maturity consultants, Lead Appraisers, and sponsors have questioned the business value of statistically based process performance models.