Application of Operational Effectiveness Models in Naval Ship concept exploration and design

Resumen Aplicación del modelo de efectividad operacional en el concepto de exploración y diseño de buques navales Date Received: January 16th, 2013 Fecha de recepción: 16 de Enero de 2013 Date Accepted: March 1st, 2013 Fecha de aceptación: 1 de Marzo de 2013 Application of Operational Effectiveness Models in Naval Ship concept exploration and design 1 NAVSEA Professor; Virginia Polytechnic and State University, Blacksburg, VA, USA. e-mail: albrown5@vt.edu Ship Science & Technology Vol. 7 n.° 13 (9-21) July 2013 Cartagena (Colombia) 10 In the US Navy, continuous early mission operational and capabilities analysis is typically performed by naval or multi-service officers and civilians in what is (today) called the Joint Capabilities Integration and Development System, JCIDS (DoD, 2008; DoD, 2003). Working from strategic guidance and after a significant effort of operational analysis, including a concept of operations, functional needs analysis, capabilitiesbased assessment, and consideration of high-level material and non-material solutions, an Initial Capabilities Document (ICD) is developed and passed to the acquisition community. Among other things, ICDs provide the following: • Identification of a mission need. • Capabilities required and their associated operational characteristics and attributes. • Capability gaps and associated operational risks. • Assessment of the viability of non-material solutions. • Recommendations on type of solutions (transformational, evolutionary, or information technology) to be pursued. • Next, the acquisition community develops Analysis of Alternatives (AoA) guidance and an AoA is performed. The AoA brings ship designers and engineers into the process typically to identify potential technologies and assess a matrix of ship design alternatives identified by the AoA study team, acquisition executives, and operational community. An AoA begins by establishing Key Performance Parameters (KPPs) for each alternative. The KPPs help compare the operational effectiveness, suitability, and life cycle costs of alternatives to satisfy the military need. The AoA and its associated documentation is required before any major investment decision and before each decision milestone and is, therefore, one of the most important steps in the US military acquisition process; however, the process often lacks a complete and effective systems engineering approach. Over the last 15 years at Virginia Tech and MIT, we have called this design phase Concept and Requirements Exploration (C&RE). During C&RE we use a total systems approach, including an efficient search of the design space for nondominated designs based on the multi-objective considerations of effectiveness, cost, and risk (Brown and Thomas, 1998; Shahak, 1998; Brown and Salcedo, 2003; Strock and Brown, 2008; Brown, 2010). Our current method of calculating an Overall Measure of Effectiveness (OMOE) for the C&RE uses expert opinion and pairwise comparison. Given that we have developed this process, systems engineering has also evolved to include new Model-Based Systems Engineering (MBSE) approaches (Buede, 2011), enterprise architectures including new Department of Defense architectures (DoD, 2011), and naval systems analysis and effectiveness-modeling methods (Fox, 2011; Gomez-Torres, 2010). We have concluded that an early structured search of the design space through the synthesis and assessment of hundreds or thousands of alternative concepts is essential for sufficient understanding of the relationship among cost, effectiveness, and risk. This is a difficult problem and requires significant upfront effort, but without this understanding, responsible specification of requirements and subsequent responsible acquisition decisions cannot be made. Others have come to similar conclusions. Andrews (2012) calls his early C&RE process “Requirements Elucidation”. He proposes that requirements engineering, as often practiced without material solutions, is not appropriate for warships and that requirements engineering without material solutions is bad systems engineering. He calls his Requirements Elucidation problem the “Wicked Problem”. We heartily concur. Jons (2012) presents ship design as illustrated in Fig. 1. Here, concept formulation is shown as a complex spiral interaction of operational needs and technical inputs, another “wicked problem” which becomes more wicked when performed in a large organization. Jons also points out that the ultimate success of any design must be effectiveness, not Introduction Ship Science & Technology Vol. 7 n.° 13 (9-21) July 2013 Cartagena (Colombia) Brown

In the US Navy, continuous early mission operational and capabilities analysis is typically performed by naval or multi-service officers and civilians in what is (today) called the Joint Capabilities Integration and Development System, JCIDS (DoD, 2008;DoD, 2003).Working from strategic guidance and after a significant effort of operational analysis, including a concept of operations, functional needs analysis, capabilitiesbased assessment, and consideration of high-level material and non-material solutions, an Initial Capabilities Document (ICD) is developed and passed to the acquisition community.Among other things, ICDs provide the following: • Identification of a mission need.
• Capabilities required and their associated operational characteristics and attributes.• Capability gaps and associated operational risks.
• Assessment of the viability of non-material solutions.• Recommendations on type of solutions (transformational, evolutionary, or information technology) to be pursued.• Next, the acquisition community develops Analysis of Alternatives (AoA) guidance and an AoA is performed.The AoA brings ship designers and engineers into the process typically to identify potential technologies and assess a matrix of ship design alternatives identified by the AoA study team, acquisition executives, and operational community.An AoA begins by establishing Key Performance Parameters (KPPs) for each alternative.The KPPs help compare the operational effectiveness, suitability, and life cycle costs of alternatives to satisfy the military need.
The AoA and its associated documentation is required before any major investment decision and before each decision milestone and is, therefore, one of the most important steps in the US military acquisition process; however, the process often lacks a complete and effective systems engineering approach.
Over the last 15 years at Virginia Tech and MIT, we have called this design phase Concept and Requirements Exploration (C&RE).During C&RE we use a total systems approach, including an efficient search of the design space for nondominated designs based on the multi-objective considerations of effectiveness, cost, and risk (Brown and Thomas, 1998;Shahak, 1998;Brown and Salcedo, 2003;Strock and Brown, 2008;Brown, 2010).Our current method of calculating an Overall Measure of Effectiveness (OMOE) for the C&RE uses expert opinion and pairwise comparison.Given that we have developed this process, systems engineering has also evolved to include new Model-Based Systems Engineering (MBSE) approaches (Buede, 2011), enterprise architectures including new Department of Defense architectures (DoD, 2011), and naval systems analysis and effectiveness-modeling methods (Fox, 2011;Gomez-Torres, 2010).We have concluded that an early structured search of the design space through the synthesis and assessment of hundreds or thousands of alternative concepts is essential for sufficient understanding of the relationship among cost, effectiveness, and risk.This is a difficult problem and requires significant upfront effort, but without this understanding, responsible specification of requirements and subsequent responsible acquisition decisions cannot be made.
Others have come to similar conclusions.Andrews (2012) calls his early C&RE process "Requirements Elucidation".He proposes that requirements engineering, as often practiced without material solutions, is not appropriate for warships and that requirements engineering without material solutions is bad systems engineering.He calls his Requirements Elucidation problem the "Wicked Problem".We heartily concur.Jons (2012) presents ship design as illustrated in Fig. 1.Here, concept formulation is shown as a complex spiral interaction of operational needs and technical inputs, another "wicked problem" which becomes more wicked when performed in a large organization.Jons also points out that the ultimate success of any design must be effectiveness, not performance.This is illustrated in Fig. 2. Once the ship is designed and built, the "means" are determined.The "environment" comes with the threat and the "ways" are determined by the operator.Collectively, they determine effectiveness, but the "means" are locked in with the designmostly, during the earliest stages of design.
It should also be noted that the human element ("means") in determining effectiveness is critical and very difficult to capture in a model.When we first developed our C&RE process, we developed an OMOE metric using expert opinion and pairwise comparison (Demko, 2005) because of this difficulty.We are now developing a methodology using Operational Effectiveness Models (OEMs) as an alternative to an expert opinion-based OMOE, and working to integrate this methodology into our current multi-objective optimization method.In this new methodology, a Design Reference Mission (DRM) composed of multiple Operational Situations (OpSits) with conditions and measures is used to develop the OEMs.An MBSE approach and a Total-Ship System Architecture are used to define and understand the relationships among various aspects of the ship design and their relationships to operational effectiveness (Kerns, 2011).
In our search for the design space, a non-dominated solution for cost, risk, and effectiveness is a feasible solution for which no other feasible solution  exists, which is better in one objective attribute and at least as good in all others.Non-dominated concepts are typically presented in a multidimensional plot of cost, risk, and effectiveness where each point in the plot represents a feasible ship design.Fig. 3 is a typical 2D representation of non-dominated (Pareto) results with the color/ shape of each design point representing the risk, with cost on the x axis, and effectiveness on the y axis.All the designs represented in this plot are feasible and non-dominated.The preferred design should always be a non-dominated design.The non-dominated surface with a full range of costrisk-effectiveness possibilities can be presented to decision-makers, "knees in the curve" can be seen graphically, trade-off decisions can be made, and specific design concepts can be chosen for further analysis.
Our new C&RE process is shown in Fig. 4. The first steps in this process must develop a clear and precise mission definition including mission essential tasks, Design Reference Mission with Operational Situations, Operational Effectiveness Models, Required Operational Capabilities, and ultimately an Overall Effectiveness Model.The development of these important system elements is the primary focus of this paper.
Systems Engineering (SE) addresses both engineering and management processes.It begins with a clear statement and understanding of the problem, must resolve competing trade-offs, must identify system boundaries, and most importantly, it must manage complexity.A critical tool to a successful SE approach is an effective system architecture and data model (Vitech, 2010).A systems model serves as a "repository" to document important characteristics, data, and relationships   from customer requirements to function to system architecture to validation and verification.Complex Systems-of-Systems (SoS) are too large to manage without a system model.Unfortunately, the process of managing an acquisition through a large government organization with many stakeholders is often characterized by a series of "throw-it-overthe-wall" events where early assumptions, models, analyses, and results may get lost.These details may be very significant and frequently must be "reinvented" later in the process.This requires redundant work, but a greater concern is that different parts of the SoS may be inconsistent, even incompatible with one another, and not reflect the original foundation of strategic guidance and the original mission operational and capabilities analysis.Chaos ensues, the design never does what it was conceived to do and program cost skyrockets in attempts to band-aid and fix.The system model database consists of elements modified by attributes and related to other elements.This structure uses an object-oriented approach.Elements are represented as objects with the attributes stored as data within the objects.The relationships then define the interaction between objects.Tasks, etc.Each class contains elements of that class.Thereby, the ship would be a component and then it would be broken down into its many physical parts with each being captured as a component system or subsystem.The Operational Architecture Domain provides necessary classes, attributes, and relationships to capture the initial operational requirements, guidance, mission, and required capabilities.The Capability class defines the qualities, abilities, features, etc., of the entire architecture that can be used or developed to achieve action goals.The Mission element includes the mission(s) that the overall architecture was designed to achieve.The Operational Task element is an action to be performed in support of a mission.An Operational Activity is an action or process needed to fulfill a mission, task, or role.The Operational Item element class is the data or physical entity required for the flow among operational activities and, thereby, among the performers.
The Ship System Architecture Domain includes Component elements and the system or system of systems with their interfaces.It includes Function elements and ultimately Requirement elements, standards, and specifications.The Components are the physical or logical elements that perform a specific function or functions.The Function elements define the functions of the components.The Requirement elements can be either an originating requirement extracted from a source document, a refinement of a higher-level requirement, a derived characteristic of the system or one of its subcomponents, or a design decision (Vitech, 2011).
An OMOE model or function is an essential prerequisite for optimization and design tradeoff.Our prior work with mission effectiveness   (Belton, 1986;Saaty, 1986;Demko, 2005).The benefit and efficiency of this approach has been demonstrated in a study revisiting the DDG-51 design (Stepanchick and Brown, 2007) and in a simple experimental study comparing results obtained by using a commercial war-gaming tool for expert opinion results (Demko, 2005).
Despite the results obtained using expert opinion, more direct physics-based OEMs starting with a detailed DRM may provide greater confidence in the validity of the results and results that are more unbiased and rational.
Design Reference Mission(s) define the specific projected threat and operating environment baseline for a given force element, which may range from a single-purpose weapon system to a multi-mission platform to a multi-system, multi-platform system of systems.They are primarily an engineering design tool to support systems engineering activities by identifying significant design-driving operational elements and characterizing them to the level of detail necessary to assess design impact.
A DRM also includes detailed characterizations of the threat, background traffic, weather, and other factors required to assess system performance and overall platform effectiveness.OpSits are developed as part of the DRM to feature selected operational characteristics, or combinations thereof, in operationally viable combat environments and situations (Skolnick et.al., 2000).
Tasks that are enabled by the required capabilities are the basic elements for accomplishing the mission or mission objective.Fig. 7 illustrates important relationships of these components.
The Universal Navy Task List (UNTL) is a combined Navy, Marine Corps, and Coast Guard document that includes tasks from the UJTL and NTTL.The UNTL also includes measures for each task and a chapter of possible conditions to assign to task.The conditions list is thorough but not comprehensive as commanders may make and assign new conditions as appropriate (NWDC, 2000).
In our process, Navy Mission Essential Tasks (NMETs) for a particular ship mission or ship design are selected from the UNTL with associated measures of task performance and conditions under which the task could be accomplished.
The final collection of tasks is called a Naval Mission Essential Task List (NMETL), tailored for a particular design.The NMETs, properly sequenced, form a scenario that includes its own measures and conditions.The scenarios built from NMETs become the OpSits that make up the DRM.
These OpSits can be translated into a discreteevent simulation that considers the conditions and uses the identified measures of task performance to calculate a specific measure of effectiveness for a ship design in that OpSit.The family of OpSit simulations that fully encompass the mission set of the ship can be combined to calculate an OMOE  The C&RE Process was presented in Figure 4. Given the JCIDS input, ICD, ADM, and AoA Guidance, the process begins by building or adapting a totalship system enterprise architecture.This is followed by further development of the Design Reference Mission with Operational Situations, Operational Effectiveness Models, Required Operational Capabilities, and, finally, an Overall Effectiveness Model.Nothing should have to start from scratch.
The system architecture and data repository must be built from the start of the JCIDS process, but even this nucleus may be adapted from earlier similar system architectures and repositories.This is the knowledge base!It should not depend on people, but stand on its own and support program continuity and consistency as people change.
Once the NMETL has been developed, it is used to assemble the DRM with the OpSits and the NMETL tasks.Here, we use a notional OPV to illustrate this process.The missions of the OPV have been defined by extracting them from the ICD.The mission objectives are described on each mission element property sheet as the element description (Vitech, 2011).Fig. 10 shows a Port, Waterway and Coastal Security (PWCS) mission element property sheet with the mission description.
The next step in the process is to identify the tasks that must be completed to accomplish each OpSit for each mission.Our notional OPV DRM is defined using 11 OpSits, one for each mission including the sub-missions under Defense Readiness.Once the necessary NMETs are selected, OpSit Functional Flow Block Diagrams (FFBDs) are built.When complete, OpSit elements can be expanded to show the sequenced NMETs that make up the OpSit.
A DRUG Interdiction (DRUG) OpSit FFBD is  This paper describes a framework for performing operational-effectiveness-based ship concept and requirements exploration.Important to this framework are a total ship design architecture and a structured approach to ship effectiveness using a Design Reference Mission, Operational Situations and standard Naval Mission Essential Tasks to guide the integration of Operational Effectiveness Models and calculate total ship effectiveness given the mission need, capabilities, and threat specified in an Initial Capabilities Document.
Our system architecture development focused on the Operational Architecture Domain to rationally define a DRM and its OpSits.The ability of the architecture to act as a single source repository for all data, guidance, design characteristics, functions, processes, cost, risk, effectiveness, and capture all of the relationships among these aspects makes it a potentially powerful tool.
The DRM provides rational measures of effectiveness (MOEs) based on realistic operational situations and the NMETL.The DRM is developed as an integral part of the Total-Ship System Architecture.By defining a DRM for a given ship design, the foundation is laid for using OEMs in the effectiveness model.If the Total-Ship System Architecture approach is adopted as a standard requirement for the ship design process, then OEMs become a natural choice to measure effectiveness and evolve directly from the architecture.
It takes significant effort to build the OEM simulations.Integrating the OEMs into an OMOE may also require some expert opinion.Research is continuing to determine the best balance of expert opinion and OEM-based methods.This research will include comparing the results of both methods to determine which is better or more valid for a given level of effort.

Fig. 5
Fig. 5 illustrates the MBSE process and architecture as a series of layers.Each layer involves the four activities of Requirements Analysis, Behavior or Functional Analysis, Synthesis/Architecture, and Validation and Verification.Concept and Requirements Exploration deals significantly with Requirements, defining requirements in terms of missions, operational tasks, and operational scenarios and translating them into cost-effective capabilities, system functions, components and interfaces.This is represented by Layer 1.
Our ship system design architecture includes three domains: Operational Architecture, Ship System Architecture, and Program/Engineering Management Domains with classes and relationships, as shown in Fig. 6.The individual parts of the architecture are classes including Capabilities, Functions, Components, Operational Ship Science & Technology -Vol.7 -n.° 13 -(9-21) July 2013 -Cartagena (Colombia) Application of Operational Effectiveness Models in Naval Ship concept exploration and design