An Investigation of the Role of System Effectiveness in the Acquisition and Sustainment of US Defense Systems:1958 to 2021

—This paper highlights the System Effectiveness methodology and its role in acquiring and sustaining U.S. military weapons systems from 1958 to 2021. Given the long period covered by this study and the many changes to the acquisition process, it would be reasonable to expect that the methodology would change and adapt, and the study supports this assumption. The study used three qualitative methods: a structured review of the literature related to System Effectiveness, a grounded theory analysis of the structured literature review, and a historiography of the initial grounded theory results. The research identiﬁed three epochs, the ﬁrst two lasting approxi- mately 22 years each. The conclusions are fourfold. First, the role of System Effectiveness today is vastly diminished from its original purpose because original material was not widely accessible to the community of interest during the formative years. Second, analysis of source documents provides insight into how to correct the misconceptions of the past and incorporate System Effectiveness into modern system engineering. Third, the models developed in epoch one may have relevance for today’s problems. The last conclusion is that an integrated research methodology is a valuable tool for making sense of a mass of conﬂicting information spread out over time. This the most comprehensive research on systems effectiveness ever published. Based on over 500 documents, it is rigorous and innovative and provides the basis for a structured and integrated approach for utilizing System Effectiveness in emerging U.S. defense system acquisition and sustainment challenges.


I. INTRODUCTION
T HIS paper presents the outcome of an "Investigation into the role of System Effectiveness in the acquisition and sustainment of U.S. defense systems from 1958 to 2021." It describes the inception of System Effectiveness, the attempts to apply the concept, and ultimately, using an approach that combines the Structured Literature Review and Grounded Theory Analysis methods, a theory as to why System Effectiveness fell into disfavor.
Given the long time span covered by this study and the many changes to the acquisition process that occurred during the period of interest, it would be reasonable to expect a change in the role of System Effectiveness. The literature supports several changes throughout the time-frame [20]. Still, changes were not necessarily driven by the acquisition process itself but by the underlying methodologies for developing systems J.M. Green is a Senior Lecturer with the Department of Systems Engineering, Naval Postgraduate School, Monterey, CA, 93943 USA e-mail: (jmgreen@nps.edu).
J. Stracener is a Senior Research Associate with the AT&T Center for Virtualization, Southern Methodist University.
Manuscript received that were also changing during this time. The analysis shows a dynamic tension between the diverse communities involved in developing System Effectiveness as a concept that eventually led to the demise of developing the idea as a methodology. However, it still exists as a concept in many systems engineering texts such as Wasson's System Engineering Analysis, Design, and Development [18].

A. Background
World War II highlighted the need for a concept by which the military could assess the effectiveness of weapons systems. Complex problems faced the Department of Defense. Stateof-the-art solutions were required, as were methods by which to evaluate them. In the 50s and 60s military systems were pushing the state of the art. Post-war systems were even more complex, encompassing programs such as the B-52 bomber and the Polaris missile program. Moreover, given the nature of their missions, they had to be reliable and effective. As a result, Secretary of Defense McNamara introduced Systems Analysis into the defense acquisition process to address the quantification of cost and the effectiveness of weapons systems [19] . The initial response to McNamara's challenge came from the Army, Navy, Air Force, and analysis organizations within the Department of Defense. As a result, throughout the 1960s, there was a flurry of activity by all three military services as they tried to incorporate McNamara's ideas into their vision of the acquisition process [1] .
A review of the literature related to System Effectiveness showed inconsistency in the concept from its first uses in the early 1960s through today. Earlier work by the reliability community started in the late 1950s. It served as the basis for developing the concept in the 1960s. While the literature shows little academic interest in the topic, a substantial body of work is available produced by the Department of Defense and defense contractors. There was a serious effort by the Department of Defense to develop System Effectiveness as a discipline highlighted by the development of the WSEIAC methodology [22] to predict and measure System Effectiveness. 1 By the early 1980s, the concept had all but disappeared from the literature. As System Effectiveness faded to a definition in the Defense Acquisition Guidebook, substantial interest in the topic of measures of effectiveness began to dominate the defense-related literature and is the most common concept currently in use. There is a subtle question that underlies this transition: why measures of effectiveness over System Effectiveness? Accordingly, it is the purpose of this paper to examine this fundamental question, provide context, and provide an answer as to why the shift occurred. Further,the approach used is a combination of a structured literature review and grounded theory analysis. The literature review is the database for the application of grounded theory techniques.
B. Defining System Effectiveness: What is System Effectiveness, and why is it important?
Cost-effectiveness, System Effectiveness, integrated logistics support, and maintainability comprise the acquisition and sustainment process [2]. Of these four components, System Effectiveness is the linchpin. System Effectiveness is the starting point for deriving the other three components. As Blanchard notes, "The ultimate goal of any system or equipment is to fulfill a particular mission for which it was designed. The degree of fulfillment is often referred to as System Effectiveness. " [1].
System Effectiveness was developed as a concept by which to focus management attention on the overall effectiveness of a system. Further, System Effectiveness provides a framework for analytic methods to predict and measure the overall results of the analysis while placing the contributing characteristics in their proper perspective relative to the desired outcome of the system performing the mission. Thus, System Effectiveness provides a basis for developing needs and requirements during project definition and for evaluating accomplishments during the acquisition and operation phases.
As developed by the Department of Defense, System Effectiveness combined elements of reliability theory and system analysis. It was an outgrowth of work started in the 1950s by the Reliability community and the system analysis work done by the Rand Corporation. Dordick noted in 1965 that it could be an uncomfortable relationship because the two groups viewed the problem from different perspectives [3].
The official definition of System Effectiveness is [1]: "a measure of the degree to which an item can be expected to achieve a set of specific mission requirements, and which may be expressed as a function of availability, dependability, and capability" . However, the official definition does not indicate the scope and scale of System Effectiveness as initially envisioned. As previously noted, it was an engineering management process concerned with the description, control, and measurement of system performance in practice. System Effectiveness is one of the two elements of cost-effectiveness (the other being cost). Together, System Effectiveness and cost-effectiveness represent the key elements of a 15-step management approach formulated to deal with the cost and complexity of modern military systems. It provides a framework for system development through the four phases (conceptual, definition, acquisition, and operational) of system development described by the Air Force Systems Engineering Management Procedures Manual [16].
Restating the definition of System Effectiveness per Blanchard [1], [2], the management goal is to establish the probability that a system can successfully meet the operational demand within a given time when operated under specified conditions. This goal is the probability of success for the system. Accordingly, the framework focuses on evaluating or predicting the degree of effectiveness for any system configuration (existing or proposed). This degree of effectiveness has a cost associated with it that is the value used in the cost-effectiveness (CE) equation [2] : where: SE = System Effectiveness IC = initial cost SC = sustainment cost System Effectiveness has three elements that determine both cost and the probability of success. This paper refers to them as the pillars upon which the System Effectiveness concept rests. These pillars are: 1) Availability -is the system ready to perform its function? 2) Dependability -how well will the system perform during a mission? 3) Capability -will the system produce the desired effects?
The first pillar is commonly referred to as operational availability or readiness and the second pillar is commonly called mission reliability. Some sources equate the third pillar, capability, with design adequacy, i.e., is the design adequate for its intended mission?
The intent was to use the System Effectiveness concept as a vehicle to proceed from predicted values in the conceptual phase of acquisition to empirical values as the system design matured, became operational and sustainment costs become paramount.

C. Statement of the Problem
Current literature referencing the System Effectiveness concept (and, by extension, measures of effectiveness) describes it in an ad hoc manner, based more on tribal lore than primary sources. Consequently, references such as the Navy's Operational Availability Handbook [7] contain erroneous discussions and descriptions of System Effectiveness. Further, System Effectiveness is more than a definition in the Defense Acquisition Guidebook. It is a framework. The separation of the three pillars into stand-alone definitions may lead to misunderstandings about the relative importance of the individual elements within the context of the framework. The loss of the big picture may result in inappropriate attempts to substitute measures of effectiveness for without context.

D. Specific Contribution of the Research
The study upon which this paper is based thoroughly explores the subject of System Effectiveness. This paper reports on the methodology used in that study and expands upon the paper of Tillman, Hwang, and Kuo [11]. the specific contribution of this paper is that it extends knowledge in the domain of System Effectiveness related to acquisition and sustainment 2 . It is a significant contribution because it presents a more current, thorough, and detailed analysis of a topic of interest to the acquisition and sustainment communities as well as supporting disciplines such as system engineering and reliability engineering. It is unique because it uses several research techniques in a triangulated approach not normally applied to research in this area. It combines a Structured Literature Review with Grounded Theory Analysis and Historiography to develop a more complete understanding of System Effectiveness based upon a comprehensive database of relevant papers gleaned from current and archival sources.

II. RELATED WORK
Structured Literature Reviews and Grounded Theory have their roots in the social sciences. However, applications of grounded theory can extend beyond the social sciences. For example, Johnson recently published a doctoral dissertation titled "Complex Adaptive Systems of Systems: A Grounded Theory Approach" [4], [5]. In addition, Structured Literature Reviews and Grounded Theory are being used in Software Engineering [6], [13], [14].

III. RESEARCH METHODOLOGY
There are four essential elements to developing a structured literature review and grounded theory analysis: Step 1: The research question.
Step 2: The structured literature review.
Step 3: The domain of inquiry.
Step 4: Critical elements in findings The research question is the starting point, and the structured literature review represents the timeline and the data that supports answering the research question. Finally, the domain of inquiry is the examination of the literature in the context of the timeline using grounded theory. The outcome is in the form of themes and patterns that emerge from the literature analysis with time.

A. The Research Question(s)
The goal is an in depth understanding of System Effectiveness from its origins to 2021. The aim of the research is to assess how System Effectiveness evolved with changes in the U.S. Department of Defense acquisition and sustainment processes. Table I presents the questions that served as drivers for the study. 2 Sustainment is the appropriate term. Sustainability and sustainable have taken on specific meanings within the environmental community What themes began to emerge with the changing role? Q3: Were there noticeable patterns of change involved? Q4: Were causal reasons identifiable?

B. The Structured Literature Review
The structured literature review served two purposes in this study. First, the use of a search protocol identified material related to System Effectiveness. Second, the structured literature review served as the first filter in identifying potential patterns for the grounded theory analysis. The scoping study of Figure 1 identified possible sources to search. Table III presents the list of sources used. The scoping study also helped to bound the keywords to be used in the search. Table IV lists prospective keywords developed from several sources, the primary source being the paper written by Tillman, Hwang, and Kuo [11].   [8]. Because the development of System Effectiveness was primarily a government effort, the majority of the literature retrieved fell into the Grey category.
The literature retrieval process used two steps. The first step was the use of a focused search string: TITLE-ABS (("System Effectiveness" AND (("keyword")) The search was conducted over 42 days during early 2020 and resulted in over 500 pieces of literature. Figure 2 shows the basics of the Structured Literature Review.The filtering process was performed manually.
The second step was a "snowball" search [9] using the reference section of the selected papers. This produced another 52 papers. The search string was also employed in a general web search which resulted in three conference proceeding found in Google Books that were unavailable from other sources.
The selection of sources to search (Table III)

C. Grounded Theory and Coding the Data
McCall and Edwards [15] have identified three methodologies associated with grounded theory: the classic grounded theory of Glaser and Strauss (1967); the pragmatic grounded theory of Strauss and Corbin (1990, 1998); and constructivist grounded theory espoused by Charmez (2006, 2014). The discussion of the differences among these methodologies is beyond the scope of this paper. However, the study that this paper is reporting on used the pragmatic grounded theory approach. The following reasons are the basis for the selection of this approach: First, it recognizes the literature as the phenomena to be studied. Second, it takes an interpretive approach that allows the development of a more profound understanding of the literature and the evolution of an abstract theory. Resultant  theories are the researcher's interpretations of causal mechanisms. Third, the role of the researcher is that of an interpreter. However, this approach recognizes the researcher's personal experience and knowledge as a factor. The data sampling process is a back-and-forth effort that results in substantial memo writing and diagramming to identify and incorporate the data into manageable sets. The technique employs three distinct methods: open coding, axial coating, and selective coding. These processes are sequential and take the researcher through a series of steps that develop the patterns in the data (open coding) and examining the derived patterns for causality (axial coding). Axial coding confirms relationships between categories or bounds their applicability. Selective coding is about determining which category embodies the characteristics of the previously derived patterns. This category becomes the core category and represents the resulting theory. The overall procedure is recursive and proceeds until the sequence results in a candidate theory.

IV. ANALYSIS AND SYNTHESIS A. The Data
The Grounded Theory developed during this study is based on over 500 papers which cover approximately 70 years (1950 to 2021). The details are too extensive to present here. Rather, the scope of this article is to present an overview of how the research questions were addressed. Thus, to that end, an outline of the core concepts and their analysis is presented.   2) Changes with policy: Changes with policy identify the major policy changes that occurred to the acquisition structure with time from 1958 to the present. Sample factors include the cycles of acquisition reform, the type of cycle, and the form of the changes .

B. Step 1: The Analysis of the Data
3) Changes with DoD structure: Factors in this pattern include reorganization of research labs, changes in responsibility for System Effectiveness within the DOD structure, and a lack of central authority.

4) Changes with technology:
The factors in this pattern refer to the emphasis of reliability over complexity. For example, the user community originally favored systems that demonstrated mission reliability over capability 5) Changes with knowledge: This could also read "changes with lack of knowledge". Factors include loss of experienced analysts, inexperienced analysts, lack of reference material, and lack of example reports . The latter two are problem areas because early material was archived as microfiche. Retrieval rates for the study was four of eight documents requested. 6) Disparate technical disciplines: This pattern is distinguished by a lack of common background or education. 7) Tension among technical disciplines: Factors in this pattern include a failure by some disciplines to see the big picture. This is better known as "if your only tool is a hammer, you tend to see problems as a nail". 8) Inconsistent models of System Effectiveness: The factors in this pattern center around differences in what comprised effectiveness as well as similar terms that had different meanings among the various models of System Effectiveness. 9) Following fads: This pattern contains the factors that describe misguided attempts to redefine System Effectiveness to accommodate the management of the fad du jour. An example is equating System Effectiveness to quality at the expense of capability.
10) Lack of participation by industry: This pattern is found throughout the literature. Factors include proprietary methods that are time tested and no financial incentives to change.
11) Lack of participation by academia: Factors in this pattern address the lack of research and publication by the academic community.
12) Misuse of the concept: This is a common issue in the literature. Factors include failure to understand the purpose of the System Effectiveness concept and misrepresentation of the concept as solely a reliability model. 13) Lack of a consistent language: Currently there is a lack of common and consistent terms for use when discussing System Effectiveness. Factors include no ontology and/or taxonomy for System Effectiveness and cost effectiveness. The lexicon developed in the 1960s has been corrupted through lack of knowledge of System Effectiveness. This has led to a confusion between what is System Effectiveness and what is a measure of effectiveness.
The use of historiography [10] with Grounded Theory enhanced the ability to develop patterns over time.

D. Step 3: Results of the Axial Coding
Axial coding is about causation. Again, this coding step is recursive and the stopping point is when patterns have been grouped with their causal effects. Table VII presents the distillation of the 13 patterns into a list of candidates for the selective coding step. The analysis of these five effects over the timeline leads a sense of disarray. For example, The tension between disciplines was identified by Dordick early on [3]. The sense of disarray is heightened by the continued immaturity of the System Effectiveness concept. Too many people confuse the concept with only reliability and maintainability (RAM) modeling because the various models shown in the Blanchard papers [1], [2] were not fully developed nor integrated into one consolidated model. Three points stand out in the literature. First, Aziz [19] points out the confusion in terminology and the lack of organized progress, particularly in the area of performance analysis. Second, Coppola [20] considered System Effectiveness to be a transient idea and notes that System Effectiveness gave way to life cycle cost as the emphasis. Third, Coppola's point is supported by the advent of MIL-STD-721C [21] which removed all reference to System Effectiveness.

V. THE THEORY
It would be easy to say, given the evidence, that System Effectiveness is a failed concept, that the theory is one of failure. However, [12] presents a strong case to the contrary. The book presents three applications: hardware system evaluation, organizational development and evaluation, and conflict analysis. A search on [12] led to new material in Asia, specifically in China where System Effectiveness is referred to as the ADC model. While they are beyond the scope of the study A. The Theory of Immaturity?
How can a concept that is in its 60s be considered to be immature? Simple. What may be signs of failure can also be signs that the idea was not developed to it full potential. That is the contention here. The literature shows that System Effectiveness may have been a victim of a short attention span within the U.S. Department of Defense environment. It is clear that The Chinese have discovered the value of the concept. Based on this realization, the theory is more importantly about the potential of System Effectiveness.

B. Threats to the Validity of the Study
In a study of this type where the result is subjective, the question of validity must be addressed. There are two factors that must be discussed: the literature review and the coding. The challenge of the literature review is building a comprehensive data base. there were negatives such as the impact of COVID-19 isolation restricting access to physical materials.The study was also limited in that it was a one person effort. Positives were the search string strategy with focused key word for pass one and the use of the snowball search. The challenges associated with the coding were the one person effort though a strength was that person's long experience in the subject area. The coding was confirmed by a researcher experienced in Grounded Theory. In research of this type, imprecision and subjective interpretation may occur. For that reason, the database is available for verification of the analyses.

C. Answering the Research Questions
Table VIII restates the research questions that this paper set out to answer. Q1 can be addressed by noting that the critical factor was the people involved. They practiced different disciplines and brought different perspectives and levels of experience to System Effectiveness. Coppola was a reliability person, and his comment about System Effectiveness meshes with Dordick's perspective that it would be hard for different disciplines to set aside their differences. The answer to Q2 has three answers. The first theme was the emphasis on reliability and maintainability (RAM) at the expense of capability. The second theme was life cycle cost which incorporated the cost of RAM. Again, capability was not in the picture. The third theme was the focus on sustainment which encapsulated the first two themes. It became more about a sustainable system than a capable system. The answer to Q3 is yes. There was a shift in focus to life cycle cost and how to accomplish analyses. First came the Cost and Operational Effectiveness Analysis (COEA), different from a Systems Effectiveness study. System Effectiveness focused on the three pillars, whereas the COEA followed a rigid, prescribed approach. Next came the analysis of alternatives concept, an analytical comparison of alternative material solutions that satisfy an established capability. The answer to Q4 is yes. They show up in the three stages of coding as well as the discussion above. What factors were connected to the change in the role of System Effectiveness? Q2: What themes began to emerge with the changing role? Q3: Were there noticeable patterns of change involved? Q4: Were causal reasons identifiable?
The unasked question is: should the System Effectiveness concept have gone away? The answer is that it has not gone away. On the contrary, the idea of System Effectiveness is always lurking in the background. However, there are weaknesses in the concept. There is an issue with both a lexicon and a taxonomy. Thus, there is a need for an ontology to provide structure and organization. Resolution of these issues and needs would remove System Effectiveness from "tribal lore" to established fact.

A. Conclusions
The selected research method(s) served to clarify how System Effectiveness came about, the attempts to make it viable, and how it meandered from the original concept. The research methods also clarified future research directions and highlighted issues and ideas that can improve the understanding and usage. The System Effectiveness concept has application to a wide variety of systems engineering problems including system of systems architecture and cost-effective modeling with tools such as COPLIMO (Constructive Product Line Investment Model).

B. Future Work
"Theory informs practice." Using the knowledge gained in the study, the next steps will be to develop an ontology and taxonomy that will provide a defined foundation to inform application of System Effectiveness and its methods. A second focus will be on the development of case studies to illustrate application and clarify the lexicon.