Director, Operational Test and Evaluation
FY 2013 Annual Report
January 2014 This report satisfies the provisions of Title 10, United States Code, Section 139. The report summarizes the operational test and evaluation activities (including live fire testing activities) of the Department of Defense during the preceding fiscal year.
J. Michael Gilmore
Director
Introduction
FY 2013 Annual Report The United States’ Department of Defense (DoD) acquires some of the most complex systems known. Because of this complexity, they often require many years of development and testing; and if not tested properly, we run the very serious risk of delivering poorly performing equipment to the warfighter. Our Airmen, Sailors, Marines, and Soldiers rely on these systems to be effective, suitable, survivable, and lethal. Because in many respects their lives depend on weapons systems that work, it is essential that adequate testing is done to fully characterize those systems’ capabilities and shortcomings across all of the relevant operational conditions in which the system is anticipated to be employed. Such characterization is needed in part so that well-informed acquisition and development decisions can be made, but also so the men and women in combat understand what these systems can and cannot do. As a nation, we cannot afford to field weapons systems that do not work, do not provide a clear improvement over existing systems, or are not militarily useful; nor can we afford to make these important fielding decisions without knowledge of the systems’ operational effectiveness. Time and again I have found that without adequate operational testing, we would not have understood the specific conditions in which a system is effective and suitable; my reporting continues to be focused on this characterization, since no system can provide perfect performance under all operational conditions or against all relevant threats. Provided the information gained from operational testing is used, characterization of performance as a function of operational conditions and threats enables developers to understand and fix problems quickly. Early testing (both developmental test events and operational assessments) can and should inform the development process and enable the early identification of major problems. The requirement for adequate operational testing is part of the natural and healthy tension between the testing, acquisition, and requirements communities. This year, I have found several cases where the testing I determined to be adequate was beyond the narrow definitions in the requirements document(s) established by the Services and Joint Staff. I have observed two distinct limitations in requirements definitions: • Requirements stated in terms of technical parameters that are not mission-oriented • Requirements that are narrowly defined to specific conditions, when the Services will certainly employ the system in other conditions I provided a specific example of the former case to the Vice Chairman of the Joint Chiefs of Staff. I found that the P-8A Multi-Mission Maritime Patrol Aircraft could be fully compliant with all Key Performance Parameter (KPP) and Key System Attribute (KSA) threshold requirements, and nonetheless possess significant shortfalls in mission effectiveness. The P-8 requirements define supporting system characteristics or attributes that are necessary, but not nearly sufficient, to ensure mission effectiveness. In an extreme case, the contractor could deliver an aircraft that meets all the KPPs but has no mission capability whatsoever. Such an airplane would only have to be designed to be reliable, equipped with self-protection features and radios, and capable of transporting weapons and sonobuoys across the specified distances, but would not actually have to have the ability to successfully find and sink threat submarines in an Anti-Submarine Warfare mission (its primary mission). The lack of KPPs/KSAs related directly to mission effectiveness will inevitably create a disconnect between the determination of operational effectiveness in test reports and the KPP and KSA compliance assessments that typically drive program reviews throughout development. The Department could therefore be making early acquisition decisions on the basis of standards that are useful, but do not capture the primary reason for procuring these systems: to provide a warfighting capability. For the second case mentioned above, where requirements are too narrowly defined, I remain committed to conducting adequate testing in all the relevant operational conditions in which men and women in combat will employ the system. Requirements may be too narrowly defined because there is a common concern that failing to specify a certain, limited set of conditions could lead to an unwieldy or excessive test. The need to test neither too much nor too little is a key reason DOT&E is using Design of Experiments (DOE) methods to plan testing that efficiently spans the operational envelope. The DOE method is rooted in a structured and disciplined determination of the operational envelope. In some cases, a clear understanding of the operational envelope reveals the need to conduct testing in conditions not specified in the requirements documents, and such testing does indeed require funding for additional test events or test resources. Such investments are essential, and in my view, must be done to ensure prompt delivery of effective, suitable, and survivable warfighting
i
Introduction capabilities. Test costs represent a small fraction of the cost of the program, and operational testing of conditions that are outside the scope of the requirements documents is usually the only venue by which system performance can be determined under those conditions. The Department cannot shrink from the need to conduct adequate testing. As an important example of the above principle, I mention briefly the need for conducting testing of our Navy’s current and future combat systems on destroyers, cruisers, and other “big-deck” surface ships. We currently use an unmanned, remotely controlled ship, called the Self-Defense Test Ship (SDTS), with the actual radars, weapons, and combat systems employed on some (not all) of these ships to examine the ability of these systems to protect against incoming anti-ship cruise missiles. The use of an unmanned, remotely controlled ship is essential, since conducting most engagements in the self-defense (close-in) region is not possible on manned ships due to safety considerations. Furthermore, modeling and simulation efforts, while useful, have not been able to reproduce the results of many of these tests. For the future radar and combat system now in development for the DDG 51 Flight III ships, we must conduct adequate testing under all relevant operational conditions. These conditions include examining end-to-end combat system performance against multiple simultaneous threat missiles within the self-defense zone of the ship, where manned testing is impossible. An SDTS is therefore essential for an adequate operational test. Previous testing has revealed for the combat systems of amphibious assault ships and carriers that without the use of an SDTS, critical problems in defending against certain threats would not have been found. Now, because of that test resource, many of those combat system problems have been corrected, and our Sailors are safer from harm. We cannot afford to not test future DDG combat systems and radars under stressing conditions in the self-defense zone, particularly since the DDGs themselves provide the defensive shield for the battlegroup. Our nation needs to pursue the testing and resources necessary to ensure system performance is understood in all regions of the operational envelope. In the remainder of this Introduction, I briefly describe the other areas of focus for my office. These include: • My continued emphasis on the need for statistical rigor in both the planning of operational tests and in the analysis of data from testing. • My continued emphasis on the need to improve reliability of all weapons systems – here I include an assessment of new policies on reliability growth and tracking, as well as how the Department is progressing in improving reliability of weapons systems. • My observations of software-intensive system development and testing, including the vulnerability of business systems. • Other areas of interest, including cybersecurity testing and test protocols for personal protective equipment. My assessment of critical test resources is also a focus area, but discussed in a separate section of this report. • An assessment of problem discovery during testing – this section of the report was added in 2011 based on concerns from Congress that significant problems in acquisition programs are being discovered during operational testing that arguably should have been discovered in developmental testing (page 13 in the DOT&E Activity and Oversight section).
CONTINUED EFFORTS TO ENSURE RIGOROUS, DEFENSIBLE, AND EFFICIENT TESTING At my confirmation hearing in September 2009, I pledged to work to “assure that all systems undergo rigorous operational test and evaluation in order to determine whether they are operationally effective, suitable, and survivable.” A rigorous operational test characterizes a system’s end-to-end mission effectiveness across the operational envelope and quantifies the risk in such assessments. Statistical methods, including DOE, provide a defensible methodology for ensuring the adequacy of any test. These methods encapsulate the need to “do more without more,” especially in light of a highly constrained fiscal environment. They provide a methodology for optimizing scarce test resources, ensuring that each test point provides the maximum information for my evaluation. They provide sound rationale for the level of testing prescribed, ensuring that we avoid either over-testing or under-testing weapons systems. Finally, they ensure we gather the data needed to provide men and women in combat confidence in evaluations of the performance of those weapons systems. In October 2010, I communicated to the Operational Test Agencies (OTAs) and Service T&E Executives my expectations regarding the use of DOE for developing rigorous, adequate, and defensible test programs and for evaluating their results. The statistical methods that I have made key to my assessment of test adequacy constitute well-established best practices in both industry and government at large. The pharmaceutical, automotive, agriculture, and chemical and process industries, where many of these techniques were originally developed, all use the same statistical methods for test design and analysis that I advocate. Furthermore, other government agencies such as the Food and Drug Administration, Census Bureau, the National Laboratories that ensure the integrity of our nation’s nuclear stockpile, as well as the National Aeronautics and Space Administration, which also engage in the testing of large and/or complex systems (similar to the DoD), all rely on the use of these methods.
ii
Introduction There has been much progress in increasing the statistical rigor of test plans since 2009. Over the past several years, all of the OTAs have implemented DOE practices to varying degrees and have offered training to their staff on the statistical principles of DOE. Additionally, the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) endorses these methods and advocates them through his Scientific Test and Analysis Techniques (STAT) implementation plan. That office has also stood up a STAT Test and Evaluation Center of Excellence, which employs qualified statistics experts to aid acquisition program managers in applying advanced statistical techniques to the design of developmental tests and analysis of resulting data. However, there is still variability in the application of these tools across the Services’ T&E communities. To that end, my office has recently completed a “Roadmap” to institutionalize test science and statistical rigor in T&E (the published version can be found here: http://www.dote.osd.mil/pub/reports/20130711TestScienceRoadmapReport.pdf). Additionally, I continue to provide guidance on best practices on the employment of these methods in OT&E. This year, I provided two additional guidance memos that address misconceptions and highlight best practices for employing DOE in OT&E. Below, I provide a summary of this most recent guidance on the use of DOE in operational testing. I also discuss the major advances in the application of these tools to T&E in several key focus areas, highlighting resources available to the T&E community. Finally, I conclude with a look to the future and how we can further improve our capabilities to take advantage of state-of-the-art methodologies. Working with the operational and developmental test communities, I will continue to employ advanced statistical methods, and continue to improve our acumen in this area, as it can only benefit the Department and ultimately, our men and women in combat, in the end. 2013 DOT&E Guidance Memos In my review of Test and Evaluation Master Plans (TEMPs) and in discussions within the test community, I have learned that misunderstandings persist of what DOT&E advocates regarding the use of DOE when designing operational tests. In 2013, I provided two additional guidance memos; key points in those memos are highlighted below. 1. Clear Test Goals The most essential element of any test design is clearly defined test goals. Operational testing should seek to characterize a system’s end-to-end mission effectiveness across the operational envelope. Such characterization of performance informs the system operators, as well as strategic and tactical planners, of its capabilities and limitations in the various conditions that will be encountered during combat operations. The goal of operational testing is not solely to verify that a threshold requirement has been met in a single or static set of conditions. Using DOE enables test programs (including integrated testing, where appropriate) to determine the effect of factors on a comprehensive set of operational mission- and capability‑focused quantitative response variables. The determination of whether requirements have been met is also a test goal, but is a subset of this larger and much more important goal. 2. Mission-Oriented Metrics OT&E metrics must provide a measure of mission accomplishment (not technical performance for a single subsystem), be continuous rather than discrete so as to support good test design, and address the reasons for procuring the system. Good measures in OT&E often reflect the timely and accurate accomplishment of a combat mission. 3. Consideration of all Operational Factors and Strategic Control of them in the Test Plan The users often employ the system in conditions that are different from those identified for system development and specification compliance. Operational testing must enable the evaluation of a system across the conditions under which it will actually be employed. By selecting test factors (the variables that define the test conditions across the operational envelope) and forcing purposeful control of those factors, we can ensure that the operational test covers those conditions, which the system will encounter once fielded. The test factors must be varied in a purposeful way, yet not overly constrain the operational realism of the test. This balance must be obtained while ensuring that the test will generate adequate information for my evaluation. Uncontrolled “free play” is not a defensible test methodology. Operational testing should consist of deliberate control of the conditions while still allowing operators and simulated opposing forces to react as they would in a true operational scenario. Factors should be varied in a way enabling diagnosis of the root cause of changes in performance across the operational envelope. Eliminating factors or specific conditions is usually the first tactic in reducing test costs, but this is a false economy. Time and money are saved by examining the operating envelope as early as possible and mitigating risks through rigorous testing across all phases of the acquisition life cycle.
iii
Introduction 4. Avoidance of Single-Hypothesis Tests Single-hypothesis statistical tests and their corresponding statistical power calculations are generally inappropriate for designing operational tests because they do not provide the ability to characterize performance across the operational envelope. Nor do they provide insights on the placement of test points within the operational envelope. 5. Statistical Assessment of Test Designs Statistical confidence and power continue to be essential tools in my assessment of test designs. When used correctly in the context of the goal of the test (which is to say, provided the test variables and factors have been well selected to address mission needs, as discussed above), these quantitative measures provide great insight into the adequacy of the test design. In an experimental design, power not only describes the risk in concluding a factor is not important when it really is, but also directly relates to the precision we will achieve in making quantitative estimates of system performance. The latter is key in my determination of test adequacy; without a measure of the expected precision we expect to obtain in the analysis of test data, we have no way of determining if the test will accurately characterize system performance across the operational envelope. A test that has low power to detect factor effects might fail to detect true system flaws; if that occurs, we have failed in our duty as testers. It is also essential that we consider additional criteria in the evaluation of the statistical design. Other criteria that are important to consider are the prediction variance across the operational envelope and correlation between factors. I provided these criteria and others in a recent memorandum to the T&E community, the use of which will enable all of the Services to prepare good test designs.1 Current Focus Areas In an effort to institutionalize the use of scientific/statistical approaches to T&E, DOT&E has focused on several key areas including: developing the workforce, updating policy and guidance, developing case study best practices, and advancing state-of-the-art methodologies to address challenges unique to the T&E community. In June 2013, my Science Advisor published the DOT&E Test Science Roadmap Report, which captures the progress in each of these areas. Workforce Development The Test Science Roadmap Report indicates clearly that all of the OTAs could benefit by increasing the number of civilian employees with scientific, technology, engineering, and mathematics (STEM) backgrounds in their workforce. Additionally, the Commanders of each OTA would benefit from having a senior technical advisor who is well versed in the science of experimental design and data analysis and is responsible for ensuring technical rigor across the entire Command. Education and training are essential in the development of our T&E workforce. At DOT&E, I ensure that my staff receives regular training on important topics such as experimental design, reliability growth and analysis, and survey design. I welcome members of the broader test community in these training opportunities, especially the OTAs. Additionally, there are many excellent training and education programs available to the T&E community (details can be found in the Roadmap Report). Policy and Guidance Updates Policy and guidance updates that are currently underway will support the institutionalization of a scientific approach to T&E. These updates include the Defense acquisition policy, the DoD Instruction 5000.02, and the Defense Acquisition Guidebook. In addition to these broader policy documents, DOT&E has published a TEMP Guidebook, which provides an up-to-date resource for the T&E community. I continue to update the guidebook as new best practices and lessons learned are captured. The guidebook highlights the substantive content DOT&E is looking for in TEMPs. The TEMP Guidebook is available on the DOT&E public website (http://www.dote.osd.mil/temp-guidebook) and provides guidance on many test science topics, including: • Design of Experiments • Mission-oriented metrics • Reliability growth • Modeling and Simulation • Information Assurance • Software-intensive systems
Memorandum dated July 23, 2013, “Best Practices for Assessing the Statistical Adequacy of Experimental Designs Used in Operational Test and Evaluation.”
1
iv
Introduction Case Studies, Best Practices, and Lessons Learned In recent years, DOT&E, the Service OTAs, as well as the broader T&E community have captured many case studies that highlight best practices and lessons learned. These case studies are available in the Test Science Roadmap Report. Additionally, many of the best practices are captured in my most recent guidance memos. Best practices I advocate include: • Provide clear justification for all designs – every design requires the quantification of acceptable risks and a determination of what changes in performance (effect size) need to be captured by the test design. These elements need to be clearly described and justified by the operational context. • Use existing system and developmental test data. Operational test designs have the greatest chance of succeeding if they leverage all existing data on the system and its intended employment. • Use continuous metrics where possible, since they provide the maximum information from a given test size; furthermore, they enable at least a 50 percent (and likely greater) reduction in test size over comparable pass/fail metrics for similar test goals. • Ensure that power calculations are consistent with test goals and avoid single hypothesis tests. Additionally, use power curves to show trade-offs in resources and risk. • Include all relevant factors (cast as continuous where possible) in design; mitigate risks by leveraging data and information from developmental testing. • Do not limit test goals to verifying requirements under limited set of conditions; focus on characterizing performance across the operational space. • Use statistical measures of merit to evaluate the trade-space in the test design. Test Science Research Consortium In conjunction with the Department’s Test Resource Management Center, DOT&E continues to fund a multi-year research consortium to address the unique needs of the T&E community. This consortium funds several graduate-level research projects on advanced statistical techniques. By doing so, it not only enables these projects to be focused on topics of benefit to the Department’s T&E needs, but also creates a pool of professionals with strong technical skills who can contribute to solving the many problems the Department confronts in improving its ability to acquire and field complex weapons systems. Scientific Test and Analysis Techniques Test and Evaluation Center of Excellence (STAT T&E COE) The STAT T&E COE, stood up by DASD(DT&E), provides direct T&E support to the program offices of Major Defense Acquisition Programs (MDAPs). The STAT experts are assigned to the program’s T&E leads and work directly with the larger teams to assist by injecting more statistical rigor into defensible test planning, design, execution, and assessment processes. In 2013, the COE supported a total of 25 major programs, as well as various ad hoc requests. STAT experts have created and delivered multiple two-day STAT courses for various program test teams. These courses educate and inform testers and program office personnel on the value and implementation of a rigorous test methodology. Looking to the Future While significant progress has been made in recent years, there is still work to be done in ensuring that the scientific community’s full toolset is available to support T&E. All programs need to employ best practices identified over the past several years. In addition to implementing these best practices, I have noted further areas for improvement that I will emphasize in the upcoming year. These specific areas for improvement include: • Conducting data analysis commensurate with DOE design. Although most in the T&E community are now using statistical rigor to develop test designs, they are not always following up with the same rigor in the analysis of the data. The worst case of this occurs when a test is designed to cover the important operational conditions efficiently through DOE techniques, yet the data analysis is limited to reporting a single average (mean) across the test conditions. A more comprehensive statistical analysis is needed to fully realize the efficiencies and increased information provided by a rigorous experimental design. We must employ standard statistical tools, such as regression analysis techniques, that utilize all of the factors that affect system performance (meaning the “recordable variables” that were not controlled in the test design, as well as the factors that were). Additionally, we must improve our capabilities to verify these empirical statistical models to ensure they accurately reflect the data. • Employing advanced methods. Many tests are complicated by data that require more than the “standard” or “simple” analysis methods. In these cases, we should embrace the opportunity to employ advanced methods. I plan to continue efforts to employ these advanced statistical tools where appropriate, and will continue to encourage the use of and train the community on these methods. Some examples include-- Bayesian approaches (especially in a reliability context) allow us to leverage information from multiple phases of test while ensuring the results still reflect the operational reliability.
v
Introduction - Censored data analysis allows us to incorporate information from continuous measures in cases where traditional pass/fail metrics would have been the only option. - Generalized linear models and mixed models allow flexible analysis methodologies that truly reflect the character of the data. • Improving the use of surveys in OT&E. Surveys provide essential information for the evaluation of systems. However, I have observed that their use in OT&E often does not reflect best practices of the survey community. The result is data that have limited utility in my evaluations. In the upcoming year, I will provide guidance on the appropriate use of surveys in OT&E.
RELIABILITY ANALYSIS, PLANNING, TRACKING, AND REPORTING I, and other Department leaders, have placed emphasis on improving the reliability of DoD systems via several reliability improvement initiatives, and I continue to emphasize the importance of reliability in my assessments of operational suitability.2 Test results from the last few decades indicate that the DoD has not yet realized significant statistical improvements in the reliability of many systems. However, there is evidence that those systems that implemented a comprehensive reliability growth program are more likely to meet their development goals. While always important, it is especially important in the current fiscal climate that system reliability is emphasized early in the acquisition process. Reliable systems cost less overall (because they require less maintenance and fewer spare parts), are more likely to be available when called upon, and enable a longer system lifespan. Reliability is more effectively and efficiently designed‑in early (design for reliability) vice being tested‑in late. While more upfront effort is Figure 1: (a) Reduction in support costs due to reliability improvements for a vehicle electronics system required to build reliable systems, (b) Life cycle cost reduction due to investment in reliability for select programs the future savings potential is too great to ignore. The Department has recognized these potential cost savings. Figures 1a and 1b are examples from two studies that illustrate how investments in reliability lead to reduced life cycle costs. Programs that invest in reliability improvements early in their life cycle, such as the C-17 in Figure 1b, are expected to get the greatest return on investment and concomitant reduction in life cycle costs. Evidence of Continuing Reliability Problems Despite the implementation of the previously cited policies intended to encourage development of more reliable systems, the fraction of DoD systems assessed as reliable during operational testing has not improved. From FY97 to FY13, 56 percent (75 of 135) of the systems that conducted an operational test met or exceeded their reliability threshold requirements as compared to nearly 64 percent between FY85 and FY96. Figure 2 shows performance by Service.
Figure 2: Fraction of DOT&E oversight Programs Meeting Reliability Thresholds at IOT&E by Service (FY97-FY13)
e.g., Reliability Growth section of the DOT&E TEMP Guidebook version 2.1; USD(AT&L) policies including July 21, 2008, “Reliability, Availability, and Maintainability Policy” and March 21, 2011, Directive-Type Memorandum (DTM) 11-003 – “Reliability Analysis, Planning, Tracking, and Reporting.”
2
vi
Introduction Figure 3 shows the relationship between previous and current policies for reliability as ratios of achieved reliability to threshold requirements between FY85 and FY13. The yellow highlight with two-year lag is the period of the prescriptive policy described in MIL-STD-785B; the green highlight also with two-year lag is the period of non-prescriptive, commercial best-practices; and the red is the current policy with emphasis on design for reliability and reliability test planning and growth. All data points greater than or equal to 1 indicate the system demonstrated reliability at or above its threshold requirement. Data points below 1 indicate the system failed to demonstrate its reliability threshold in operational testing. A linear Figure 3: Reliability Trends versus Policy Periods for Years FY85 to FY13 fit to the data suggests there has been no improvement in program reliability over time. The boxplots in Figure 3 show that the three groupings have similar median values, but a larger fraction of data in the first grouping (FY85 to FY98) is concentrated at somewhat higher values compared to the latter two groupings. Although the plots suggest a decreasing trend in reliability, the trend is not statistically significant. Nonetheless, the data are conclusive that the reliability of DoD systems has not significantly improved over time. The Department has acknowledged this poor track record of meeting system reliability requirements in March 2011 when USD(AT&L) issued a Directive Type Memorandum (DTM 11-003) on “Reliability, Analysis, Planning, Tracking, and Reporting.” The DTM requires program managers to formulate a comprehensive reliability and maintainability program that is part of the systems engineering process, assess the reliability growth required for the system to achieve its reliability threshold during IOT&E, and report the results of that assessment to the Milestone Decision Authority at Milestone C. To instantiate reliability reporting in support of Defense Acquisition Executive Summary (DAES) reviews, DOT&E has worked with DoD Systems Engineering to implement a systematic process of tracking MDAP reliability status. MDAPs in system-level developmental testing with a documented reliability growth curve in the Systems Engineering Plan and TEMP will be required to report reliability data on a quarterly basis. The data will be used to inform the DAES selection process, review MDAP reliability performance-to-plan, and support reliability growth planning for future programs. At the direction of Acquisition Resource and Analysis, MDAPs that meet the criteria for reporting will submit their reliability data starting in FY14. Evidence of Some Success To better understand these trends, I have conducted a survey of programs under DOT&E oversight in each of the past five years to determine the extent to which reliability-focused policy guidance is being implemented and to assess whether it is leading to improved reliability. The most recent survey focused on 90 programs that submitted either a Test and Evaluation Strategy (TES) or TEMP to DOT&E, and/or had an operational test in FY12. The survey results indicate, not surprisingly, that systems with a comprehensive reliability growth program are more likely to reach reliability goals compared to those that do not employ a growth program. In particular, the results show the importance of establishing and meeting operational test Reliability, Availability, and Maintainability (RAM) entrance criteria before proceeding to operational test. While many programs did not establish or meet operational test RAM entrance criteria, those that did were far more likely to demonstrate reliability at or above the required value during operational test. There is also evidence that having intermediate goals linked to the reliability growth curve improves the chance of meeting RAM entrance criteria.
vii
Introduction The survey results indicate that programs are increasingly incorporating reliability-focused policy guidance. In FY12: • 92 percent of programs had a reliability growth strategy, with 90 percent documenting it in the TEMP. • Likewise, 78 percent of programs incorporated reliability growth curves into the TEMP. • 59 percent of programs used a reliability growth curve to develop intermediate goals. • 87 percent of programs used reliability metrics to ensure that growth was on track to achieve requirements. • 49 percent of programs had a process for calculating growth potential. Despite these policy implementation improvements, many programs still fail to reach reliability goals. In other words, the policy has not yet proven effective at changing the trends displayed in Figure 3. The reasons programs fail to reach reliability goals include inadequate requirements, unrealistic assumptions, lack of a design for reliability effort prior to Milestone B, and failure to employ a comprehensive reliability growth process. For example, the reliability thresholds for some programs were unachievably high or disconnected from what was really needed for the mission. Other unrealistic assumptions include choosing an initial reliability value for their reliability growth curve that was significantly higher than comparable systems have been able to achieve, or choosing an optimistic initial value for the growth curve without an adequate design-for‑reliability effort (which should occur prior to the growth program) to achieve that initial value. In some cases, a program’s reliability growth goal, while documented in a TEMP or Systems Engineering Plan, was not supported by contractual obligations or funding. As a result, a larger fraction of surveyed programs met their reliability thresholds after fielding during FOT&E (72 percent) rather than before fielding during IOT&E (50 percent). I conclude from this study that although we are in a period of new policy that emphasizes good reliability growth principles, without a consistent implementation of those principles, the reliability trend will remain flat. Recommendations for the Future In the future, programs need to do a better job incorporating a robust design and reliability growth program from the beginning that includes the design for reliability tenets described in the ANSI/GEIA-STD-0009, “Reliability Program Standard for Systems Design, Development, and Manufacturing.” Programs that follow this practice are more likely to be reliable. There should be a greater emphasis on ensuring that reliability requirements are achievable, and reliability expectations during each phase of development are supported by realistic assumptions that are linked with systems engineering activities. I recommend that all programs establish operational test entrance criteria and ensure these criteria are met prior to proceeding to the next test phase. Examples of effective RAM entrance criteria include (1) demonstrating in the last developmental test event prior to the IOT&E a reliability point estimate that is consistent with the reliability growth curve, and (2) for automated information systems and software-intensive sensor and weapons systems, ensuring that there are no open Category 1 or 2 deficiency reports prior to operational test. I also reemphasize USD(AT&L) policy described in DTM 11-003, “Reliability Analysis, Planning, Tracking, and Reporting” that reliability growth curves/programs should be constructed with a series of intermediate goals, with time allowed in the program schedule for test-fix-test activities to support achieving those goals. System reliability should be tracked through system-level T&E events until the reliability threshold is achieved. Second, when sufficient evidence exists to determine that a program’s demonstrated reliability is significantly below the growth curve, I recommend that the program develop a path forward to address shortfalls and brief their corrective action plan to the acquisition executive. Such efforts might include a reexamination of the requirements and updates to the assumptions made in the growth curve, and may reveal the need for the program to perform lower level re-design work to get back on course. This will help encourage sound development processes, including the use of design-for-reliability efforts, and allow the growth curve to be a much more useful tool for decision makers. Based on findings from surveys, reliability trend analysis, and other lessons learned, I continue to update and refine the reliability growth guidance section of DOT&E’s TEMP Guidebook. The latest edition, updated July 12, 2013, provides specific reliability growth planning guidance for different types of systems, including hardware-only systems; hybrid systems containing a combination of software, hardware, and human interfaces; and software-intensive systems. The Guidebook also provides an overview of the key systems engineering and design activities that constitute a comprehensive reliability growth program and requires the TEMP to include a description of these activities for each of the three system types, with emphasis on the latter two. For hybrid systems (e.g., weapons systems composed of both hardware and software, such as radars), the TEMP requires plans for categorizing hardware failures verses software failures, for tracking software failures, and for regression testing software failure fixes. Software-intensive systems, starting in the design phase, should describe a plan to track software reliability to include defined entrance and exit criteria for system reliability at critical decision points. Finally, the latest Guidebook illustrates how to use operating characteristic curves to quantify allowable test risks (consumer’s and producer’s risk) and develop the reliability growth goal.
viii
Introduction TESTING OF SOFTWARE-INTENSIVE SYSTEMS Over the last several decades, the Department’s reliance on and procurement of software-intensive systems has significantly increased. These Major Automated Information Systems (MAIS) provide key capabilities to the Department, including financial and budgetary management functions, command and control, medical records management, and logistics and inventory management. Furthermore, nearly every military system is based upon software to provide functionality and capability. Because of the importance of the issue, and because many capability shortfalls are directly related to software failures and poor software maintenance capabilities, I have increased my involvement in testing these systems. I note four areas are of interest in testing of software-intensive systems. First, I continue to observe that many MAIS programs do not create adequate software maintenance capabilities early enough to support deployment. Second, software requirements continue to be poorly stated. Third, as a new area of interest within the last several years, I am focusing on testing the financial vulnerabilities of systems that have direct accounting or logistics functions. Finally, as the Department begins to examine how its test processes can and should be adjusted to accommodate the Agile software development model, I provide three distinct models of how Agile concepts can be applied to operational testing. Software Maintenance Current Department acquisition practices categorize software maintenance as a sustainment activity – something that begins after software is deployed. This is problematic as it sets our programs up for failure. Disciplined software maintenance (by which I mean configuration control, defect tracking and prioritization, maintenance of a high fidelity test environment, and automated testing within that environment) must begin as soon as there is any software to maintain. Software that is deployed in the absence of a robust maintenance capability typically has poor operational results, and the reliability of such software can grow steadily worse with each new upgrade or patch to the software. Illustrative examples of late development of software maintenance capabilities include the DoD Automated Biometric Identification System (ABIS), Defense Enterprise Accounting and Management System (DEAMS), and Navy Enterprise Resource Planning (Navy ERP). • DoD ABIS. A key action item for the program manager from the stakeholder meeting following the fourth failed deployment attempt of ABIS 1.2 (see “Problem Discovery Affecting OT&E” in the Activity and Oversight section of this report) was to determine and document what functionality ABIS 1.0 provides to its users. How DoD ABIS can be developed and maintained without comprehensive knowledge of the capability it currently provides is a key question. • DEAMS. DEAMS had 2 operational assessments in 2 years, each identifying 200+ defects. DEAMS appears to be improving after the program manager implemented improved configuration control and defect tracking, as well as rudimentary regression testing. • Navy ERP. The Navy ERP system demonstrated significant reliability shortfalls due to software maintenance in early testing. After developing an improved software maintenance capability, the program is now operationally effective and operationally suitable. The program has a functioning software configuration control board and defect management process that is expeditiously correcting new deficiencies, particularly high-severity ones. The regression testing process is efficient, being almost entirely automated. Between the 2008 IOT&E of Release 1.0 and the 2013 FOT&E of Release 1.1 (which is to say, the five years following initial deployment), the Program Office instituted disciplined software management practices. It probably would not have taken so long to reach the full deployment decision if the software had been better managed early on. For example, during the Release 1.1 IOT&E in 2010, the discovery rate for new system defects was 125 per month with a backlog of nearly 500 defects remaining at the conclusion of testing. After the 2010 IOT&E, the Program Office improved the defect management process, which included reviewing outstanding defects more frequently and increasing the emphasis on maintaining accurate status on all defects. Navy ERP is now the Department’s second successfully deployed ERP system. To promote earlier attention to software maintenance, I have begun enforcing the following test automation policy, which was put into effect recently in the interim (November 26, 2013) Defense acquisition policy, the DoD Instruction 5000.02: For software in any system, the evaluation of operational suitability will include a demonstrated capability to maintain the software. Program managers must sustain an operationally realistic maintenance test environment in which software patches can be developed and upgrades of all kinds (developed or commercial) can be tested. (1) IOT&E or a prior test event will include an end-to-end demonstration of regression test, preferably automated, in the maintenance test environment from requirements to test scripts to defect tracing. (2) IOT&E or a prior test event will include a demonstration of processes used to update the maintenance test environment so as to replicate deficiencies first found in the operational environment.
ix
Introduction I have also worked in the last year to help programs make the transition to the use of automation for regression testing. My staff has initiated a Test Automation Center of Excellence (TACE), which is now helping to automate the third of their target list of seven highly similar MAIS programs. In the last year, by working closely with the Defense Logistics Agency (DLA) sustainment staff and support contractors (for the Department’s first successfully deployed ERP, the DLA’s Enterprise Business System), the TACE has trained 38 DLA staff in the use of automation; 6 in the development of automation; and transitioned 12 validated scripts to operational use. These scripts (and associated setup) take 18 human-at-keyboard minutes on average to execute as compared to 142 minutes on average for the corresponding manual scripts. Five scripts were executed in November 2013 as part of normal operations, including two that were developed by the DLA staff. DLA has made substantial progress in one year (and I expect another year will be needed to make DLA fully self-sufficient) at a direct cost of $500,000, as opposed to the $11.5 Million over 5 years originally quoted to DLA by a leading market analysis group. The Services have begun making efforts to include planning for software regression testing and automation. Seventeen of the 63 unclassified TEMPs, TESs, or Operational Test Plans that I signed out between December 1, 2012, and December 1, 2013, included detailed discussion of software regression testing methods and/or test automation. Finally, the importance of these software testing efforts is amplified by the push to deploy the Joint Information Environment (JIE). The JIE is envisioned to be a shared and upgraded information technology infrastructure that will, amongst other things, consolidate existing net-centric systems into a reduced number of data centers and operations centers using a common computing model for virtualization. This means that for each existing net-centric system there should at some point be two copies: the current system and the new virtualized, JIE version of the system. No existing system should be shut off until the JIE version is shown to perform at least as well, and that testing should be automated. That automated validation would then ideally be reused for subsequent regression testing. Software Requirements In most cases, it will be possible to develop software that automatically provides performance metrics. If operational testers cannot answer reasonable questions about software system performance from data that the system owners are already gathering, then the system owners also, clearly, do not fully understand how well their system is performing. This is operationally important for the same reason as software maintenance: the software will change over time. In order to maintain and improve system performance, parameters that are key to the capability should ideally be automatically measured and monitored by the Program Office vice being checked manually during operational tests. The bias and presumption in operational software testing should be toward independent review of automatically gathered performance metrics. Interactions between testers and users often provide helpful insights; however, human execution of highly repetitive, precise operations is an unnecessary expense and a missed opportunity. In the latter case, operational testing should verify that automated performance metrics exist and that the Program Office is organized to utilize those metrics in its ongoing software maintenance. I would not want nor expect a Program Office to optimize software around a performance metric that was not relevant to mission accomplishment. Unfortunately, software KPPs and their associated measures are often uninformative with respect to mission accomplishment. The measures can be seen to carry a bias and presumption toward testing that consists of human review of compliance checklists. Human review is open-loop. Program Office use of automated metrics is closed-loop, which will be better. The F-22A program, Theater Medical Information Program – Joint (TMIP-J), and Air Operations Center – Weapon System (AOC-WS) programs provide examples of open-loop and closed-loop review processes. • F-22A. The Net-Ready KPP in the F-22A TEMP (January 2013) is geared toward paperwork compliance instead of mission-relevant, automated performance measures. The KPP is: “Key Interface Profiles will be satisfied to the requirements of the specific joint integrated architecture products and information assurance accreditation.” This KPP is stated in terms of documents and accreditation, and was translated in the TEMP into various measures of compliance (for example, one measure requires all “policy enforcement controls designated as enterprise level or critical in the joint integrated architecture”). In the future, I will require that TEMPs and test plans evaluate this KPP using mission-oriented measures collected using monitoring of the operational network. In particular, the KPP should be evaluated using continuous observation of measures, including time to detect protocol deviations and error tolerance levels. • TMIP-J. The TMIP-J Increment 2 TEMP (May 2013) has a Critical Operational Issue (COI) for Supportability which translates into nine different surveys and subject matter expert evaluations. The COI “Are TMIP-J Increment 2 features, training plans, characteristics, processes, procedures, and resources adequate to sustain its intended operations?” is clearly mission-critical; the TMIP-J operators certainly need to know if and when the system becomes inadequate. However, the COI would better lend itself to appropriate automation and use by the Program Office if it were phrased or interpreted as: “Does TMIP-J Increment 2 provide reporting on its features, training, characteristics, processes, procedures, and
x
Introduction resources sufficient to determine that it is fulfilling its intended operations?” As in the previous example, the COI should be understood in terms of continuous monitoring rather than occasional compliance-checking via surveys. • AOC-WS. Conversely, the AOC-WS TEMP (October 2013) has a good measure for its Data Accuracy capability: “Percent of missions flown linked to Air Operations Directive tactical tasks.” This measure indicates that all targets must be “linked” to their desired effects. The linkage requires the AOC-WS machine-assisted capability to maintain a connection to the planned operational assessment results throughout the development of all AOC products. The connection links actions to effects and traces effects to the expected data sources. This measure of accuracy can be achieved through automation, and it will help AOC commanders evolve tasking orders during engagements by ensuring that the software can always trace planned actions to desired effects and then trace observed effects back to their associated actions, which must then be repeated or updated in subsequent tasking orders. It is important to the mission that this metric be satisfied, and it can assist in software maintenance by automatically identifying mission areas where the linkage is not working properly. With few exceptions, software KPPs should support ongoing software management by requiring automated measurement and reporting (for system managers) of help desk use, interface throughput, system productivity/utilization, training adequacy, reliability metrics, and other (less generic) mission critical performance parameters. Such reports would also answer most software OT&E questions. To promote improved requirements, I have begun enforcing the following polices, which were put into effect recently in the interim (November 26, 2013) Defense acquisition policy, the DoD Instruction 5000.02: Beginning at Milestone A, every TEMP will include an annex containing the Component’s rationale for the requirements in the draft Capability Development Document (CDD) or equivalent requirements document. Program managers for software acquisitions will provide plans at Milestone B indicating how system logs and system status records will interface with operational command and control. At IOT&E or a prior test event, program managers for software acquisitions will demonstrate performance monitoring of operational metrics to manage and operate each system capability (or the whole system, as appropriate). Financial Vulnerabilities I have 13 accounting or logistics systems on oversight, and all will be required to undergo operational testing geared to their unique vulnerabilities.3 These systems are typically being acquired so as to achieve full auditability by 2017 in accordance with the National Defense Authorization Act (NDAA) for FY10. They will homogenize the sometimes obscure or conflicting legacy accounting practices within the Department, but in the process they may also expose the Department to new or expanded vulnerabilities to theft, fraud, or nation state manipulation. Losses due to fraud in the commercial sector are estimated at 5 percent of revenues each year.4 Common fraud controls – such as those required by the Government Accountability Office Federal Information System Controls Audit Manual – should result in significant reductions in both the amount lost and the undetected time span of fraudulent activities. The Defense Intelligence Agency has not yet evaluated the potential threat to U.S. supply lines and/or U.S. markets through manipulation of the Department’s accounting and logistics systems, and there is currently no guidance for mitigating these risks. This year, the Navy ERP program conducted the first fraud vulnerability test. The test identified 1,799 user accounts that had multiple segregated roles (and who could therefore potentially commit fraud without assistance). The Navy ERP Program Office was not aware if any of those user accounts had in fact been used fraudulently. Accordingly, subsequent financial vulnerability scans and assessments will include forensic accounting activities so as to provide immediate information on the extent to which identified vulnerabilities have been exploited. The Navy ERP test was also similar to a “Blue Team” Information Assurance vulnerability scan (as opposed to a “Red Team” penetration test). The second fraud vulnerability test (for DEAMS) will complete in early 2014. DEAMS data from the last year have been provided to forensic accountants for analysis. A certified and accredited Red Team paired with trained accountants will conduct the penetration test. If the Red Team is able to penetrate the system cyber defenses, then the accountants will assess the potential operational effects that they will be able to cause. These assessments will occur in four threat scenarios that include insider threat and nation state threat scenarios. Air Force Integrated Personnel and Pay System (AF-IPPS); Defense Agency Initiative (DAI); Defense Enterprise Accounting and Management System (DEAMS); Expeditionary Combat Support System (ECSS); EProcurement; Future Pay and Personnel Management Solution (FPPS); General Fund Enterprise Business System (GFEBS); Global Combat Support System – Joint (GCSS-J); Global Combat Support System – Marine Corps (GCSS-MC); Global Combat Support System – Army (GCSS-Army); Integrated Personnel and Pay System – Army (IPPS-A); Logistics Modernization Program (LMP); Navy Enterprise Resource Planning (ERP) 4 According to the 2012 Report to the Nations of the Association of Certified Fraud Examiners, http://www.acfe.com/rttn-highlights.aspx 3
xi
Introduction Finally, I have directed my staff to develop a new Enterprise Cyber Range Environment (ECRE) to mimic the software stack supporting U.S. Transportation Command. This ECRE will enable observation of the undetected duration and magnitude of the operational effects of nation state cyber attacks that might be launched to disrupt U.S. supply lines. Agile Operational Testing of Software This year, I have approved three operational assessments that provide three distinct models of Agile operational testing. • For the Integrated Electronic Health Record (iEHR) program, I established that the responsible OTA, the Army Test and Evaluation Command (ATEC), would observe all tests (including developmental testing) and send me a report or synopsis. An ATEC tester is now embedded with the iEHR program. • For DEAMS, I approved a two-stage test. The first stage took less than one month from execution to reporting. In the first test phase, my staff interviewed DEAMS managers and users following deployment of the new (Release 2) software to existing users. The interviews were sufficient to determine that the DEAMS software management had improved, that deploying Release 2 did not disrupt operations, and that I could support the decision to deploy Release 2 to new users. The second test phase will provide me with data to evaluate the Release 2 capabilities. In this model of Agile OT&E, a rapid check on the gross features of an initial software deployment to existing users is followed by a risk-appropriate level of test of the system within a new group of users and the existing users. • For the Integrated Personnel and Pay System – Army (IPPS-A ) Increment 1, I have approved an operational test concept that will largely utilize data gathered organically by IPPS-A. The program manager and ATEC were able to implement an inexpensive email dialogue and survey process. This process will continuously track for all IPPS-A users whether their Soldier Record Brief (SRB) data are correct and, if not, what data are incorrect, and, later, whether the user has been able to successfully use the instructions for correcting their data. The survey process will also assess the usability of the IPPS-A system. Once the data have been corrected in the legacy systems (which remain the authoritative data sources in Increment 1), the final automated user survey will ask the user to review their SRB and verify whether the corrections are now displayed in their SRB. As discussed in the Software Requirements section above, this process will provide IPPS-A system owners with valuable ongoing self-monitoring information relevant to the system’s customer service needs, and it also predominantly meets operational test needs for Increment 1. With these working models of Agile operational testing in hand, I expect to be able to craft appropriate test approaches for subsequent Agile acquisitions.
OTHER AREAS OF INTEREST Electronic Warfare Test Infrastructure In February 2012, I identified significant shortfalls in the test resources required to test mission systems electronic warfare capabilities under operationally realistic conditions. The Department programmed for an Electronic Warfare Infrastructure Improvement Program starting in FY13 to add both closed-loop and open-loop emitter resources for testing on the open-air ranges, to make at least one government anechoic chamber capable of providing a representative threat environment for electronic warfare testing, and to upgrade the electronic warfare programming laboratory that will produce threat data files. These test capabilities are essential to many programs, including F-35 Joint Strike Fighter (JSF), F-22 Increment 3.2 A/B, B-2 Defensive Management System, Long-Range Strike Bomber, Next Generation Jammer for the EA-18G, Integrated Defensive Electronic Countermeasures upgrades, as well as several other programs. However, progress in selecting sources and beginning development of the test resources has been slower than needed to assure these resources are available in time for the JSF Block 3 IOT&E in 2018. Without these resources, the JSF IOT&E of Block 3 capability will not be adequate to determine the system’s effectiveness in existing operationally realistic threat environments. Aegis-Capable Self-Defense Test Ship (SDTS) As mentioned above, the test community currently relies on an unmanned, remotely controlled ship, called the SDTS, with the actual radars, weapons, and combat systems employed on some (not all) of the Navy’s currently deployed ships to examine the ability of these systems to protect against incoming anti-ship cruise missiles. Navy range safety restrictions prohibit close-in testing on manned ships because the targets and debris from successful intercepts will pose an unacceptable risk to the ship and personnel at the ranges where these self-defense engagements take place. The importance of this testing and the need for such a test resource is underscored by the recent incident in November 2013, where two Sailors were injured when an aerial target struck the USS Chancellorsville (CG-62) during what was considered to be a low-risk test of its combat
xii
Introduction system. The Navy employs a high-fidelity modeling and simulation capability that relies heavily on data collected from testing with the SDTS, as well as data from manned ship testing, so that a full assessment of ship self-defense capabilities of non-Aegis ships can be completely and affordably conducted. While the Navy recognizes the capability as integral to the test programs for certain weapons systems (the Ship Self-Defense System, Rolling Airframe Missile Block 2, and the Evolved Sea-Sparrow Missile Block 1) and ship classes (LPD-17, LHA-6, Littoral Combat Ship, DDG 100, and CVN-78), the Navy has not made a similar investment in an Aegis-capable SDTS for adequate operational testing of the DDG 51 Flight III Destroyer (with Aegis Advanced Capability Build “Next” Combat System and Air and Missile Defense Radar (AMDR)) capabilities. The current SDTS lacks the appropriate sensors and other combat system elements to test these capabilities. I continue to strongly advocate for the development of an Aegis-capable SDTS to test ship self-defense systems’ performance in the final seconds of the close-in battle and to acquire sufficient data to accredit ship self-defense modeling and simulation test beds. Other methods that are being examined and desired in lieu of an STDS, in my estimation, are wholly inadequate to fully examine the complex, close-in battlespace where multiple components of the combat system must work simultaneously to orchestrate shooting down multiple incoming highly-capable anti-ship cruise missiles, all within an engagement timeline of tens of seconds. The estimated cost for development and acquisition of an SDTS capability over the Future Years Defense Program is approximately $284 Million. Of that, $228 Million would be recouped after the test program completes by installing the hardware in a future DDG 51 Flight III hull. I have disapproved the Milestone B AMDR TEMP because, contrary to its predecessor AMDR TES, the TEMP did not provide for the resources needed to equip an SDTS. Similarly, I will disapprove the DDG 51 Flight III TEMP if it omits the resources needed to equip an SDTS. Cybersecurity Testing DOT&E continues to focus cybersecurity testing for all systems subject to information systems certifications and exposure to information networks. A review of the existing cybersecurity T&E procedures is underway in anticipation of the coming updates to the processes by which the Department certifies and accredits systems to operate on DoD networks (a shift from the DoD Information Assurance Certification and Accreditation Process to the National Institute of Standards and Technology “Risk Management Framework” in use by other federal agencies). A review of testing over the past several years continues to indicate the need to discover and resolve information system security vulnerabilities as early as possible in program development. The majority of system vulnerabilities discovered in operational testing over the last two years could and probably should have been identified and resolved prior to these tests. These challenges are also discussed in the Information Assurance and Interoperability Assessment section of this report. Testing of Personal Protective Equipment I continue to exercise oversight over personal protective equipment. The Services and the U.S. Special Operations Command (USSOCOM) continue to implement rigorous, statistically-principled testing protocols approved by DOT&E for hard body armor inserts and military combat helmets. In partnership with the Services and USSOCOM, I am developing a soft armor vest testing protocol that will standardize testing of soft armor vests and require them to meet rigorous statistical measures of performance. In its final report, the National Academy of Sciences’ Committee to Review the Testing of Body Armor supported the use of statistically-based protocols that allow decision makers to explicitly address the necessary and unavoidable risk trade-offs that must be faced in body armor testing. As a result of Congressional interest, the Department’s Inspector General completed a Technical Assessment of the Advanced Combat Helmet (ACH) in May 2013. The assessment found that the DOT&E test protocol for the ACH adopts a statistically principled approach and represents an improvement from the legacy test protocol with regard to increased sample size. In response to a recommendation in this assessment, I will conduct characterization testing of new combat helmet designs that are being procured: specifically, the lightweight ACH, the Enhanced Combat Helmet, and the Soldier Protective System Integrated Head Protection System. Based on these data, I will determine whether the relevant test protocols should be updated to be more consistent with the products’ demonstrated performance. Additionally, we developed a specific statistical procedure that provides increased confidence that combat helmets meet minimum performance standards for all helmet sizes and test environments. I asked the National Research Council to conduct an independent review of the helmet testing protocols. Their report is anticipated to be released in FY14 and I will act on its findings. As noted by the National Research Council of the National Academy of Sciences in their final report on the Testing of Body Armor and in my report to Congress on the Live Fire Test and Evaluation of the Enhanced Combat Helmet, medically validated injury criteria for behind-armor and behind-helmet blunt trauma do not exist. This is a serious limitation for the T&E of all personal protective equipment. Body armor and helmets made from modern materials deform rapidly during a bullet or fragment impact. The blunt force of the impact to the torso or of the impact of the deforming helmet shell on the
xiii
Introduction head might cause injury or death even if the threat does not penetrate. The current acceptance criteria for helmets are based on the ability to withstand penetration and on acceptable levels of deformation in the event a bullet impacts but does not penetrate. The requirements for the latter were not established using medical data nor were they informed by how much deformation would be acceptable to prevent serious injury from bullet impact. Therefore, using Joint Live Fire funds, I have funded an effort to establish injury risk criteria for one type of injury due to behind-helmet blunt trauma. My office is also monitoring a multi-year Army program to investigate behind-helmet blunt trauma, determine injury mechanisms and risks, and develop an injury criterion that can be used for helmet testing. The results of such testing have the potential of changing the way we evaluate helmets, and the protocols for testing these helmets may need to change. My office is also overseeing and participating in an Army effort to improve helmet test mount headforms by developing multiple-sized headforms to replace the single-sized headform currently used to test all helmet sizes (a recognized limitation to the current test method). Finally, I have provided funding to help characterize new potential ballistic clay formulations for use in the testing of personal protective equipment. The Army is pursuing a ballistic clay formulation with a more consistent dynamic response; these efforts have the potential to reduce the variability in the clay’s response to an impact, thereby providing a better measure of the true performance of the tested equipment. I continue to work with the Services and USSOCOM to incorporate improved test procedures as they are developed and to update personal protective equipment test standards based on the results of these studies. Warrior Injury Assessment Manikin (WIAMan) In 2010, I brought to the Department’s attention the lack of validated medical criteria and adequate instrumentation by which to assess occupant injuries in underbody blast Live Fire tests conducted against ground combat and tactical wheeled vehicles. This is a serious limitation to the T&E of all ground combat and tactical wheeled vehicles. In 2011, the Deputy Secretary of Defense directed the Army, with OSD oversight, to execute a project to conduct medical research to develop underbody blast-specific injury criteria, as well as an anthropomorphic test device (ATD) designed specifically for the underbody blast environment. The WIAMan project made significant progress in 2013 after I directed a major restructuring to address delays in medical research planning and execution. The WIAMan Project Office now resides at the U.S. Army Research Laboratory, and under this new management has begun to execute medical research, as well as ATD development. The university research performers on the WIAMan project are some of the premier injury biomechanics researchers in the country and provide the project with the requisite experience and laboratory capabilities. The first phase of medical research is well underway, and the results from that research, as well as from anthropometric studies, are informing the concept for the initial ATD prototype. The project has also provided insights into the shortcomings of the current ATDs used in Live Fire Test and Evaluation. By using a unique, purpose-built test device that is able to expose ATDs and other test subjects to a controlled, blast-driven, vertical accelerative load environment, the research revealed the lack of biofidelity of the currently-used ATD when compared to the human response. These results further reinforce the need to continue this important work. To this end, I have provided Joint Live Fire funds to support the Army’s efforts on this project and will continue to work with the Army to update underbody blast test standards and procedures to incorporate the results of this project. Fifth-Generation Aerial Target With the advent of fifth-generation aerial threats, to include low observability, low probability of intercept sensors, and embedded electronic attack, the feasibility of completing operationally realistic testing will decline significantly without developing adequate test capabilities that will assure U.S. air superiority in future conflicts. Over the past seven years, my staff has developed an alternative, low-cost fifth-generation aircraft design that will enable end-to-end testing to evaluate U.S. weapons systems effectiveness, from post-launch acquisition to end-game fusing, against fifth-generation fighter threats in Anti-Access/Area Denial missions. The Department, in partnership with the Canadian government, is considering funding a three-year, $80 Million critical design, prototyping, and flight test effort that could provide an essential developmental and operational T&E capability.
xiv
Introduction CONCLUSION Since my first report to you in 2009, we have made progress increasing the scientific and statistical rigor of operational test and evaluation; there is much work to be done, however, to improve and consistently apply these techniques. Additionally, we have focused attention on reliability management, design and growth testing, and the improvement in testing software‑intensive systems. Operational testing continues to be essential to characterize system effectiveness in combat so that well-informed acquisition and development decisions can be made, and men and women in combat understand what their equipment and weapons systems can and cannot do. I submit this report, as required by law, summarizing the operational and live fire test and evaluation activities of the Department of Defense during fiscal year 2013.
J. Michael Gilmore Director
xv
Introduction
xvi
D O T & E TAacbtl iev iotfy Caonndt eonvtesr s i g h t DOT&E Activity and Oversight FY13 Activity Summary Program Oversight Problem Discovery Affecting OT&E
1 7 13
DoD Programs Defense Readiness Reporting System (DRRS) F-35 Joint Strike Fighter (JSF) Global Command and Control System – Joint (GCCS-J) Integrated Electronic Health Record (iEHR) Joint Biological Tactical Detection System (JBTDS) Joint Information Environment (JIE) Joint Warning and Reporting Network (JWARN) Key Management Infrastructure (KMI) Mine Resistant Ambush Protected (MRAP) Family of Vehicles (FoV) Next Generation Diagnostics System (NGDS) Public Key Infrastructure (PKI) Theater Medical Information Program – Joint (TMIP-J)
31 33 53 57 61 63 65 67 69 73 75 79
Army Programs Network Integration Evaluation (NIE) Armored Tactical Vehicles Bradley Engineering Change Proposal (ECP) CH-47F (Chinook) – Improved Cargo Helicopter Distributed Common Ground System – Army (DCGS-A) DoD Automated Biometric Information System (ABIS) Global Combat Support System – Army (GCSS-Army) Handheld, Manpack, and Small Form Fit (HMS) Manpack Radio Handheld, Manpack, and Small Form Fit (HMS) Rifleman Radio Joint Battle Command – Platform (JBC-P) Joint Land Attack Cruise Missile Defense Elevated Netted Sensor System (JLENS) Joint Light Tactical Vehicle (JLTV) Family of Vehicles (FoV) M109 Family of Vehicles (FoV) Paladin Integrated Management (PIM) Mk248 Mod 0 Sniper Round Nett Warrior Patriot Advanced Capability-3 (PAC-3) Precision Guidance Kit (PGK) Q-53 Counterfire Target Acquisition Radar System Spider XM7 Network Command Munition Stryker Mobile Gun System (MGS) Warfighter Information Network – Tactical (WIN-T)
81 83 87 89 93 95 99 101 103 105 107 111 113 117 119 121 125 127 131 133 135
Navy Programs Acoustic Rapid Commercial Off‑the‑Shelf (COTS) Insertion (A-RCI) and AN/BYG‑1 Combat Control System AGM-88E Advanced Anti-Radiation Guided Missile (AARGM) Program AIM-9X Air-to-Air Missile Upgrade AN/BLQ-10 Submarine Electronic Warfare Support System AN/SQQ-89A(V)15 Integrated Undersea Warfare (USW) Combat System Suite Cobra Judy Replacement (CJR) Consolidated Afloat Networks and Enterprise Services (CANES)
137 141 143 145 147 149 151
xvii
D O T & E TAacbtl iev iotfy Caonndt eonvtesr s i g h t Cooperative Engagement Capability (CEC) CVN-78 Gerald R. Ford Class Nuclear Aircraft Carrier DDG 51 Flight III Destroyer/Air and Missile Defense Radar (AMDR)/Aegis Modernization DDG 1000 – Zumwalt Class Destroyer Distributed Common Ground System – Navy (DCGS-N) E-2D Advanced Hawkeye Enhanced Combat Helmet (ECH) F/A-18E/F Super Hornet and EA-18G Growler Ground/Air Task Oriented Radar (G/ATOR) H-1 Upgrades – U.S. Marine Corps Upgrade to AH-1Z Attack Helicopter and UH-1Y Utility Helicopter Integrated Defensive Electronic Countermeasures (IDECM) Joint High Speed Vessel (JHSV) LHA-6 Light Armored Vehicle (LAV) Upgrade Littoral Combat Ship (LCS) LPD-17 San Antonio Class Amphibious Transport Dock Mark XIIA Identification Friend or Foe (IFF) Mode 5 MH-60R Multi-Mission Helicopter MH-60S Multi-Mission Combat Support Helicopter Mk 48 Advanced Capability (ADCAP) Torpedo Modifications Mk 54 Lightweight Torpedo MQ-4C Triton Unmanned Aircraft System Multi-Static Active Coherent (MAC) System Navy Enterprise Resource Planning (ERP) P-8A Poseidon Multi-Mission Maritime Aircraft Remote Minehunting System (RMS) Ship Self-Defense Small Tactical Unmanned Aerial System (STUAS) Tier II SSBN Ohio Class Replacement Program SSN 774 Virginia Class Submarine Standard Missile-6 (SM-6) Surface Ship Torpedo Defense (SSTD): Torpedo Warning System and Countermeasure Anti- torpedo Torpedo Surveillance Towed Array Sensor System (SURTASS) and Compact Low Frequency Active (CLFA) Tomahawk Missile and Weapon System Vertical Take-Off and Landing Unmanned Aerial Vehicle (VTUAV) (Fire Scout)
153 155 161 165 167 169 171 173 177 181 183 185 189 193 195 201 203 205 207 209 211 213 215 217 221 227 231 235 237 239 243 245 249 251 253
Air Force Programs AC-130J Ghostrider Advanced Extremely High Frequency (AEHF) Satellite Communications System AIM-120 Advanced Medium-Range Air-to-Air Missile (AMRAAM) Air Operations Center – Weapon System (AOC-WS) Battle Control System – Fixed (BCS-F) CV-22 Osprey Defense Enterprise Accounting and Management System (DEAMS) F-15E Radar Modernization Program (RMP) F-22A Advanced Tactical Fighter Global Broadcast System (GBS) HC/MC-130J Joint Air-to-Surface Standoff Missile – Extended Range (JASSM-ER) Joint Space Operations Center (JSpOC) Mission System (JMS) KC-46A
255 257 259 261 265 267 269 271 273 275 277 279 281 283
xviii
D O T & E TAacbtl iev iotfy Caonndt eonvtesr s i g h t Massive Ordnance Penetrator (MOP) Miniature Air Launched Decoy (MALD) and MALD-Jammer (MALD-J) Mission Planning System (MPS)/Joint Mission Planning System – Air Force (JMPS-AF) MQ-9 Reaper Armed Unmanned Aircraft System (UAS) RQ-4B Global Hawk High-Altitude Long-Endurance Unmanned Aerial System (UAS) Space-Based Infrared System (SBIRS)
285 287 289 291 293 297
Ballistic Missile Defense Programs Ballistic Missile Defense System (BMDS) Aegis Ballistic Missile Defense (Aegis BMD) Command and Control, Battle Management, and Communications (C2BMC) System Ground-Based Midcourse Defense (GMD) Sensors Terminal High-Altitude Area Defense (THAAD)
299 303 307 311 313 317
Live Fire Test and Evaluation (LFT&E) Program
319
Information Assurance (IA) and Interoperability (IOP)
329
Test and Evaluation Resources
337
Joint Test and Evaluation (JT&E)
345
Center for Countermeasures
351
Index of Programs
357
xix
D O T & E TAacbtl iev iotfy Caonndt eonvtesr s i g h t
xx
DOT&E Activity and Oversight
DOT&E Activity and Oversight
DOT&E Activity and oversight
FY13 Activity Summary DOT&E activity for FY13 involved oversight of 312 programs, including 46 Major Automated Information Systems (MAIS). Oversight activity begins with the early acquisition milestones, continues through approval for full-rate production and, in some instances, during full production until deleted from the DOT&E oversight list.
(DAB) principals for consideration in DAB deliberations. Additional FY13 DOT&E reports that did not go to Congress included: 9 Operational Assessment Reports, 3 LFT&E reports, 3 MAIS reports, 3 Limited User Test reports, 5 FOT&E reports, 4 Operational Utility Evaluation reports, and 4 special reports.
Our review of test planning activities for FY13 included approval of 44 Test and Evaluation Master Plans (TEMPs) and 2 Test and Evaluation Strategies, as well as 63 Operational Test Plans, and 5 Live Fire Test and Evaluation (LFT&E) Strategies and Management Plans, and disapproval of 2 TEMPs (Air and Missile Defense Radar (AMDR) and AEGIS Cruiser and Destroyer Program TEMP CNO Project No. 1669 Revision 1) and 1 Test Plan (Integrated Electronic Health Record (iEHR) Increment I Operational Assessment).
During FY13, DOT&E met with Service operational test agencies, program officials, private sector organizations, and academia; monitored test activities; and provided information to the DAB committees as well as the DAB principals, the Secretary and Deputy Secretary of Defense, the Under Secretary of Defense (Acquisition, Technology and Logistics), the Service Secretaries, and Congress. Active on-site participation in, and observation of, tests and test-related activities are a primary source of information for DOT&E evaluations. In addition to on-site participation and local travel within the National Capital Region, approximately 701 trips supported the DOT&E mission.
In FY13, DOT&E prepared for the Secretary of Defense and Congress: 9 Initial Operational Test and Evaluation (IOT&E) Reports, 3 Early Fielding Reports, 11 Follow-on Operational Test and Evaluation (FOT&E) Reports, 6 LFT&E reports, 5 special reports, as well as the Ballistic Missile Defense (BMD) Programs FY12 Annual Report. DOT&E also prepared and submitted numerous reports to the Defense Acquisition Board
Security considerations preclude identifying classified programs in this report. The objective, however, is to ensure operational effectiveness and suitability do not suffer due to extraordinary security constraints imposed on those programs.
TEST AND EVALUATION MASTER PLANS / STRATEGIES APPROVED AAC-130J TEMP AN/BQQ-10 Sonar System (Revision D) TEMP for Post Milestone C FOT&E AN/BYG-1 Combat Control System (Revision 7) TEMP for Post Milestone C FOT&E AN/SLQ-32(V) Block Upgrade Surface Electronic Warfare Improvement Program (SEWIP) Block 2 TEMP Prepared for Milestone C Decision Army Logistics Modernization Program (LMP) Increment 2 Milestone B TEMP and Test and Evaluation Paragraph of the Business Case
Global Combat Support System – Army (GCSS-Army) Post-Full Deployment Decision TEMP Guided Multiple Launch Rocket System with Alternative Warhead (GMLRS-AW) Change 1 to the Milestone B TEMP HC/MC-130 Recapitalization TEMP Joint Air to Surface Missile – Extended Range (JASSM-ER) Full-Rate Production Decision (FRPD) TEMP Joint Precision Approach and Landing System (JPALS) Increment 1A TEMP
B61 Life Extension Program (LEP) Ail Kit Assembly (TKA) Program Milestone B TEMP
Joint Space Operations Center (JSPOC) Mission System (JMS) TEMP
Battle Control System – Fixed (BCS-F) Increment 3, Release 3.2 (R3.2) TEMP
Joint Warning and Reporting Network (JWARN) TEMP Annex
CARTRIDGE, 7.62 MM: BALL, M80A1 TEMP Cobra Judy Replacement (CJR) TEMP No. 1655, Revision B Command and Control Air Operations Suite/Command and Control Information Services (C2AOS/C2IS) Increment 1 TEMP Defense Readiness Reporting System Strategic (DRRS-S) TEMP E-2D Advanced Hawkeye (AHE) TEMP F/A-18E/F Flight Plan TEMP 1787 Change 1 F-22 Increment 3.2A TEMP F-22A Increment 3.2B TEMP Fleet Replenishment Oiler T-AO(X) Test and Evaluation Strategy (TES)
Joint Strike Fighter (JSF) TEMP Revision 4 (Conditional Approval) KC-46 TEMP Littoral Combat Ship (LCS) Draft TEMP (Conditional Approval) Littoral Combat Ship (LCS) Program TEMP Revision A M109 Family of Vehicles Paladin Integrated Management (PIM) Self‑Propelled Howitzer (SPH) and Carrier, Ammunition, Tracked (CAT) Vehicle TEMP Update M982E1 Excalibur Increment Ib Milestone C TEMP Mk 54 Torpedo with Block Upgrade (BUG) Software TEMP Ml09 Family of Vehicles Paladin Integrated Management (PIM) TEMP MQ-1C Gray Eagle Unmanned Aircraft System (UAS) TEMP
Activity
1
DOT&E Activity and oversight Navy Enterprise Resource Planning (ERP) TEMP Update
SCS 25X TEMP
Next Generation Enterprise Network (NGEN) Program TEMP
Space Fence TEMP
Next Generation Jammer (NGJ) Test and Evaluation Strategy
Standard Missile-6 (SM-6) TEMP Revision A
Precision Guidance Kit (PGK) TEMP Presidential Helicopter Replacement Program (VXX) TEMP
Submarine Electronic Warfare Support (ES) System (AN/BLQ-10) TEMP with changes
Public Key Infrastructure (PKI) TEMP Addendum and Future Operational Testing
Theater Medical Information Program – Joint (TMIP-J) TEMP XM7 Spider TEMP Update Rev2.0 Version
QF-16 Full Scale Aerial Target (FSAT) TEMP
XM7A1 SPIDER Increment 1A TEMP
Operational Test Plans APPROVED AC-130J LFT&E Alternative Test Plan Amphibious Assault Ship Replacement (LHA-6) Test Plan for CNO Project No 1697 Operational Assessment (OT-B2) AN/SQQ-89A(V) 14 Surface Ship Undersea (USW) Combat System Program Test Plan for CNO Project No. 0802-02, Operational Assessment AN/SQQ-89A(V)15 Surface Ship Undersea Warfare Combat System Program Test Plan for CNO Project No. 0802-02, IOT&E AN/TPQ-53 Limited User Test Operational Test Agency Test Plan Ballistic Missile Defense System (BMDS) Flight Test Operational -01 (FTO-01) Test Plan and Classified Annex Battle Control System – Fixed (BCS-F) FOT&E Plan C-17 Increased Gross Weight and Formation Spacing Reduction Follow‑On Operational Test Agency Test Plan CH-47F Cargo On/Off Loading System Test Plan COBRA JUDY Replacement (CJR) Multiservice Operational Test and Evaluation (MOT&E) Plan Combat Control System (AN/BYG-1(V)/Acoustic-Rapid Commercial Off-The-Shelf-Insertion (AN/BQQ-10(V)) System Advanced Processing Build 11 Combined FOT&E Test Plan Defense Enterprise Accounting and Management System (DEAMS) Increment 1 Release 1 Operational Assessment 2 Plan Defense Readiness Reporting System (DRRS) Version 4.6 IOT&E Plan Distributed Common Ground System – Navy (DCGS-N) Increment 1, Block 2 Operational Assessment Test Plan E-2D Advanced Hawkeye Data Management and Analysis Plan (DMAP) for Verification of Correction of Deficiencies (VCD) E-2D Test Assessment Plan Electronic Protection Improvement Program (EPIP) Test Plan Approval Enhanced Polar System (EPS) Early Operational Assessment Operational Test Agency Test Plan F/A-18A+/A++/C/D/E/F System Configuration Set (SCS) 25X FOT&E Test Plan F-15E Radar Modernization Program (RMP) Operational Test Agency IOT&E Plan F-35 Radar Cross Section Initial Look Test Operational Utility Evaluation Test Plan F-35 Radar Cross Section Initial Look Test Operational Utility Evaluation Test Plan Amendment Family of Advanced Beyond Line-of-Sight Terminals (FAB-T) Increment 1 Operational Assessment Plan
2
Activity
Flight II DDG-51 Aegis Weapon System 7.1R and Cooperative Engagement Capability USG-2A FOT&E Test Plan Change Transmittal 1 Global Broadcast System (GBS) Defense Computing Center Force Development Evaluation Operational Test Agency Test Plan Ground/Air Task Oriented Radar (G/ATOR) Operational Assessment Test Plan and Strategy Headquarters Central Command (CENTCOM) Information Assurance Assessment Plan Joint Battle Command – Platform (JBC-P) IOT&E Operational Test Agency Test Plan Joint High Speed Vessel (JHSV) IOT&E Test Plan Joint High Speed Vessel (JHSV) Change Transmittal 2 to IOT&E Test Plan Joint Land Attack Cruise Missile Defense Elevated Netted Sensor System (JLENS) Early User Test Plan Joint Land Attack Cruise Missile Defense Elevated Netted Sensor System (JLENS) Early User Test (EUT) Operational Test Agency Test Plan Test Change Proposal Joint Precision Approach and Landing System (JPALS) Increment 1A IT-B2 Test Plan Joint Warning and Reporting Network (JWARN) Operational Test Agency Test Plan KC-46A Operational Assessment-1 Plan M109 Family of Vehicles Limited User Test (LUT) Operational Test Agency Test Plan Mk 54 MOD 0 Lightweight Torpedo with Block Upgrade Software Change 2 Operational Test Plan Mk 54 Set-NOT-to-Hit Test Plan Annex for Cape Cod events in Sept 13 Mk 54 Set-to-Hit Test Plan Annex for Cape Cod events in Sept 13 Mode 5 Identification Friend or Foe (IFF) Joint Operational Test Approach 2 (JOTA 2) Test Plan MQ-9 GBU-38 Joint Direct Attack Munition (JDAM) Integration Combined Developmental Test Support/Force Development Evaluation (FDE) Test Plan Multi-static Active Coherent (MAC) Increment I Phase I on P-3 Aircraft IOT&E Test Plan MV-22B FOT&E Test Plan Navy Enterprise Resource Planning (ERP) System Release 1.1 FOT&E Plan and FOT&E PLan Change 1 Nett Warrior Limited User Test Operational Test Agency Test Plan
DOT&E Activity and oversight Nett Warrior Limited User Test Operational Test Agency Test Plan Test Change Proposal
Submarine Electronic Warfare Support (ES) System (AN/BLQ-10) Test Plan
Precision Guided Kit (PGK) Early User Assessment Operational Test Agency Test Plan
Theater Medical Information Program – Joint Increment 2 Release 2 Multi‑Service Operational Test and Evaluation Operational Test Agency Test Plan
Public Key Infrastructure (PKI) Increment 2, Spiral 3 FOT&E 2/Non-Person Entity (NPE) Operational Assesment Plan
U.S. Africa Command (USAFRICOM) Vulnerability Assessment FY13 (VA13) Final Assessment Plan
QF-16 Operational Assessment Plan
U.S. Army Warfighter Exercise 13-4 Assessment Plan
Rolling Airframe Missile (RAM) Block 2 Program Test Plan for Enterprise Test (ET-05) Phase 1 Warfare/Ship Self-Defense (AW/SSD) Enterprise, FOT&E (OT-IIIF)
U.S. European Command (EUCOM) Theater Cyber Readiness Campaign FY13 Information Assurance Assessment Plan
RQ-21A Small Tactical Unmanned Aircraft System, Project No. 1719‑OT‑B2 Operational Assessment Test Plan RQ-4B Block 40 Early Operational Capability (EOC) Operational Utility Evaluation (OUE) Test Plan Ship-to-Shore Connector (SSC) Integrated Evaluation Framework (IEF) Endorsement
U.S. Special Operations Command Information Assurance and Interoperability Assessment Plan for EMERALD WARRIOR 13 USTRANSCOM Real World 13 Final Assessment Plan (FAP) Virginia Class Submarine FOT&E Test Plan Warfighter Information Network – Tactical (WIN-T) Increment 2 FOT&E Operational Test Agency Test Plan
Spider SM7, Dispensing Set, Munition, Network Command Follow-on Operational Test 3 Operational Test Agency Test Plan
Live Fire Test and Evaluation Strategies/Management Plans AC-130J LFT&E Alternate Test Plan
Littoral Combat Ship 30 mm LFT&E Management Plan
CH-47F Improved Cargo Helicopter Post-Production Improvement Alternate LFT&E Strategy and Operational Test Agency Test Plan
Presidential Helicopter LFT&E Concept
Joint Air-to-Surface Standoff Missile (JASSM) Electronic Safe and Arm Fuze LFT&E Strategy
Activity
3
DOT&E Activity and oversight FY13 Reports to Congress Program
Date IOT&E Reports
E-3 Airborne Warning and Control System (AWACS) Block 40/45
October 2012
Distributed Common Ground System – Army (DCGS-A) Software Baseline (DSB) 1.0
October 2012
ALR-69A Radar Warning Receiver
October 2012
B-2 Extremely High Frequency (EHF) Satellite Communications (SATCOM) and Computer Upgrade Increment 1
December 2012
MQ-1C Gray Eagle Unmanned Aircraft System (with classified annex)
January 2013
E-2D Advanced Hawkeye (AHE)
February 2013
HC/MC-130J (with classified annex)
April 2013
AGM-158B Joint Air-to-Surface Standoff Missile – Extended Range (JASSM-ER)
May 2013
Standard Missile-6 (SM-6)
May 2013 Early Fielding Reports
20 mm Fixed Forward Firing Weapons (FFFW) for the MH-60 Armed Helicopter Weapon System (AHWS)
January 2013
Massive Ordnance Penetrator (MOP) Phase 2
January 2013
RQ-4B Global Hawk Block 40
August 2013 FOT&E Reports
Joint Warning and Reporting Network (JWARN)
October 2012
Virginia Class Submarine, AN/BQQ-10 Acoustic Rapid Commercial Off-the-Shelf (A-RCI) Sonar System Advanced Processor Build 2009 (APB-09) and AN/BYG-1 Combat Control System (APB-09) Consolidated
November 2012
LPD-17 Chemical, Biological, and Radiological (CBR) Defense
November 2012
F-22A Increment 3.1 (classified report in SCIF)
December 2012
Virginia Class Submarine Arctic Operations and Susceptibility to Passive Acoustic Sensors
May 2013
Mk 48 Heavyweight Torpedo with APB Spiral 4 Tactical Software
May 2013
USNS Lewis & Clark (T-AKE) Class of Dry Cargo and Ammunition Ships
May 2013
H-1 Upgrades
July 2013
AN/BLQ-10 Submarine Electronic Warfare Support System with the Technical Insertion 2008 (TI-08) Upgrade and the Multifunction Modular Mast (MMM)
September 2013
USG-3B Cooperative Engagement Capability (CEC)
September 2013
Warfighter Information Network – Tactical (WIN-T) Increment 2 (with classified annex)
September 2013
LFT&E Reports Bradley Family of Vehicles (BFoVs) Engineering Change Proposal (ECP)
November 2012
United States Marine Corps Light Armored Vehicle (LAV) with A2 Upgrades Mine-Resistant Ambush-Protected (MRAP) All-Terrain Vehicle (M-ATV) Underbody Improvement Kit (UIK) Final Assessment Mk 248 Mod 0 .300 Caliber Cartridge (U)
January 2013 March 2013 June 2013
Enhanced Combat Helmet (ECH) Follow-On Report
June 2013
Stryker Double-V Hull (DVH) Summary Report
August 2013 Special Reports
Ship Self-Defense (SSD) Operational Mission Capability Assessment
November 2012
Space-Based Infrared System (SBIRS) Effectivity 5 Operational Utility Evaluation (OUE) Report
December 2012
Mobile User Objective System (MUOS) Multi-Service Operational Test and Evaluation Report
January 2013
Assessment of Department of Defense (DoD) Cybersecurity during Major Combatant Command Service Exercises (FY12)
April 2013
Patriot Post-Deployment Build-7 (PDB-7) Limited User Test (LUT) Assessment Report
April 2013
BMD Reports FY12 Assessment of the Ballistic Missile Defense Systems (includes Classified Appendices A, B, C, D)
4
Activity
February 2013
DOT&E Activity and oversight Other FY13 Reports (Not Sent to Congress) Program
Date
Operational Assessment Reports Joint Tactical Radio System (JTRS) Handheld, Manpack, Small Form Fit (HMS) Manpack Radio and Joint Enterprise Network Manager (JENM) (with classified annex) Consolidated Afloat Network and Enterprise Services (CANES)
October 2012 December 2012
XM1156 Precision Guidance Kit (PGK)
March 2013
Integrated Defensive Electronic Countermeasures (IDECM) Block 4
April 2013
Small Tactical Unmanned Aerial System (STUAS)
April 2013
MH-60S Block 2A Airborne Mine Countermeasures (AMCM) Helicopter and the AN/AES-1 Airborne Laser Mine Detection System (ALMDS)
May 2013
KC-46A
June 2013
CH-47F Cargo On-Off Loading System (COOLS) (Integrated Test and Live Fire Test) QF-16 Full Scale Aerial Target Program
July 2013 September 2013
MAIS IOT&E Reports Integrated Strategic Planning and Analysis Network (ISPAN) Increment 2 (with classified annex) Key Management Infrastructure (KMI) Capability Increment 2, Spiral 1
October 2012 October 2012
Battle Control System – Fixed (BCS-F) Increment 3, Release 3.2 (with classified annex)
November 2012
Limited User Test Reports Nett Warrior Operational Assessment
March 2013
Operational Assessment of the AN/TPQ-53 Counterfire Target Acquisition Radar (Q-53) Paladin PIM Limited User Test Operational Assessment
April 2013 August 2013
FOT&E Reports C-130J Station Keeping Equipment (SKE) Software Enhancement (SSE)
October 2012
C-130J Data Transfer and Diagnostic System (DTADS)
October 2012
Spider XM7 Network Command Munition
February 2013
Key Management Infrastructure (KMI) Capability Increment (CI)-2 Spiral 1
April 2013
Public Key Infrastructure (PKI) Increment 2
May 2013 LFT&E Reports
Stryker Double-V Hull (DVH) Configuration of the Anti-Tank Guided Missile Vehicle (ATVV) and the Driver's Station Enchancement II (DSE II)
October 2012
CH-47F Cargo On-Off Loading System (COOLS) Live Fire Test and Evaluation Assessment
August 2013
Live Fire Test and Evaluation Report for the M109 Family of Vehicles, Paladin Integrated Management (PIM)
September 2013
Operational Utility Evaluation Reports Information Transport System Increment 2 (ITS2)
February 2013
F-35A Joint Strike Fighter (JSF) Ready for Training
February 2013
Joint Space Operations Center (JSpOC) Mission System (JMS) (includes classified annex)
February 2013
Advanced Extremely High Frequency (AEHF) Mission Control Segment Increment 5
June 2013
Special Reports C-5M Testing for Operational Flight Program (OFP) 3.5 and Thrust Reverser Modifications Force Development Evaluation
October 2012
Assessment of the U.S. Central Command (USCENTCOM) Ballistic Missile Defense Architecture (Fast Eagle exercise)
February 2013
Hellfire Romeo Missile Interim Lethality Assessment
May 2013
Global Combat Support System – Army (GCSS-A) Lead Site Verification Test Assessment
June 2013
Activity
5
DOT&E Activity and oversight
6
DOT&E Activity and oversight
Program Oversight DOT&E is responsible for approving the adequacy of plans for operational test and evaluation and for reporting the operational test results for all major defense acquisition programs to the Secretary of Defense, Under Secretary of Defense (Acquisition, Technology and Logistics), Service Secretaries, and Congress. For DOT&E oversight purposes, major defense acquisition programs were defined in the law to mean those programs meeting the criteria for reporting under Section 2430, Title 10, United States Code (USC) (Selected Acquisition Reports (SARs)). The law (sec.139(a)(2)(B)) also stipulates that DOT&E may designate any other programs for the purpose of oversight, review, and reporting. With the addition of such “non-major” programs, DOT&E was responsible for oversight of a total of 312 acquisition programs during FY13.
• The program has a close relationship to or is a key component of a major program. • The program is an existing system undergoing major modification. • The program was previously a SAR program and operational testing is not yet complete.
Non-major programs are selected for DOT&E oversight after careful consideration of the relative importance of the individual program. In determining non-SAR systems for oversight, consideration is given to one or more of the following essential elements: • Congress or OSD agencies have expressed a high-level of interest in the program. • Congress has directed that DOT&E assess or report on the program as a condition for progress or production. • The program requires joint or multi-Service testing (the law (sec. 139(b)(4)) requires DOT&E to coordinate “testing conducted jointly by more than one military department or defense agency”). • The program exceeds or has the potential to exceed the dollar threshold definition of a major program according to DoD 5000.1, but does not appear on the current SAR list (e.g., highly-classified systems).
A covered system, for the purpose of oversight for LFT&E, has been determined by DOT&E to meet one or more of the following criteria: • A major system, within the meaning of that term in 10 USC 2302(5), that is: - User-occupied and designed to provide some degree of protection to the system or its occupants in combat - A conventional munitions program or missile program • A conventional munitions program for which more than 1,000,000 rounds are planned to be acquired. • A modification to a covered system that is likely to affect significantly the survivability or lethality of such a system.
This office is also responsible for the oversight of LFT&E programs, in accordance with 10 USC 139. DoD regulation uses the term “covered system” to include all categories of systems or programs identified in 10 USC 2366 as requiring LFT&E. In addition, systems or programs that do not have acquisition points referenced in 10 USC 2366, but otherwise meet the statutory criteria, are considered “covered systems” for the purpose of DOT&E oversight.
DOT&E was responsible for the oversight of 117 LFT&E acquisition programs during FY13.
Programs Under DOT&E Oversight Fiscal Year 2013
(As taken from the September 2013 DOT&E Oversight List) DoD PROGRAMS AC-130J
Defense Agency Initiative (DAI)
BMDS – Ballistic Missile Defense System Program BMTC – Ballistic Missile Technical Collection
Defense Enterprise Accounting and Management System – Increment 1 (DEAMS - Inc. 1)
CHEM DEMIL-ACWA – Chemical Demilitarization Program – Assembled Chemical Weapons Alternatives
Defense Readiness Reporting System – Strategic Defense Security Assistance Management System (DSAMS) – Block 3
CHEM DEMIL-CMA – Chemical Demilitarization (Chem Demil) - Chemical Materals Agency (Army Executing Agent)
EProcurement
Common Analytical Laboratory System
Global Command & Control System – Joint (GCCS-J)
Conventional Prompt Global Strike
Global Combat Support System – Joint (GCSS-J)
Program Oversight
7
DOT&E Activity and oversight DoD PROGRAMS (continued) integrated Electronic Health Record (iEHR)
Modernized Intelligence Database (MIDB)
JLTV -–Joint Light Tactical Vehicle
Multi-Functional Information Distribution System (includes integration into USAF & USN aircraft)
Joint Aerial Layer Network
Next Generation Chemical Detector
Joint Biological Tactical Detection System Joint Chemical Agent Detector (JCAD)
Next Generation Diagnostic System
Joint Command and Control Capabilities (JC2C) [Encompasses GCCS-FoS (GCCS-J, GCCS-A, GCCS-M, TBMCS-FL, DCAPES, GCCS-AF, USMC JTCW, USMC TCO]
Public Key Infrastructure (PKI) Incr 2 SOCOM Dry Combat Submersible Medium (DCSM)
Joint Information Environment
Teleport, Generation III
Joint Tactical Radio System (JTRS) Enterprise Network Manager (JENM) Joint Warning and Reporting Network (JWARN)
SOCOM Next Generation Dry Deck Shelter Theater Medical Information Program – Joint (TMIP-J) Block 2 Virtual Interactive Processing System (VIPS)
Key Management Infrastructure (KMI) Increment 2 Mid-Tier Networking Vehicle Radio
ARMY PROGRAMS AN/TPQ-53 Radar System (Q-53)
EXCALIBUR – Family of Precision, 155mm Projectiles
ABRAMS TANK MODERNIZATION – Abrams Tank Modernization (M1E3)
FBCB2 – Force XXI Battle Command Brigade and Below Program
Abrams Tank Upgrade (M1A1 SA / M1A2 SEP)
FBCB2 – Joint Capability Release (FBCB2 - JCR)
AH-64E Apache
FMTV – Family of Medium Tactical Vehicles
AMF JTRS – Joint Tactical Radio System Airborne & Maritime/Fixed Station
General Fund Enterprise Business System (GFEBS)
AN/PRC-117G Radio
Global Combat Support System – Army (GCSS-Army)
Armed Aerial Scout (previously named ARH Armed Recon Helicopter)
Ground Combat Vehicle (GCV) and the lethality of the 30mm ammunition
Armored Multipurpose Vehicle (AMPV)
Guided Multiple Launch Rocket System – Unitary (GMLRS Unitary)
Armored Truck – Heavy Dump Truck (HDT)
Guided Multiple Launch Rocket System Alternate Warhead (GMLRS AW)
Armored Truck – Heavy Equipment Transporter (HET)
HELLFIRE Romeo
Armored Truck – Heavy Expanded Mobility Tactical Truck (HEMTT)
High Mobility Multipurpose Wheeled Vehicle (HMMWV)
Armored Truck – M915A5 Line Hauler
HIMARS – High Mobility Artillery Rocket System
Armored Truck – M939 General Purpose Truck
Hostile Fire Detection System
Armored Truck – Palletized Loading System (PLS)
Identification Friend or Foe Mark XIIA Mode 5 (all development and integration programs)
Army Vertical Unmanned Aircraft System Biometrics Enabling Capability (BEC) Increment 1 Biometrics Enabling Capability Increment 0 Black HAWK (UH-60L) – Utility Helicopter Program Black HAWK (UH-60M) – Utility Helicopter Program BRADLEY MODERNIZATION – Bradley Modernization (M2A3 V2)
Improved Turbine Engine Program Indirect Fire Protection Capability Increment 2 – Intercept Individual Carbine Integrated Air & Missile Defense (IAMD) Integrated Personnel and Pay System – Army (Army IPPS)
BRADLEY UPGRADE – Bradley Fighting Vehicle System Upgrade
Interceptor Body Armor
C-17 Increase Gross Weight (IGW) and reduced Formation Spacing Requirements (FSR) with T-11 parachute
Joint Air-to-Ground Missile
Cartridge, 7.62mm, M80A1
Joint Assault Bridge
CH-47F – Cargo Helicopter
Joint Battle Command Platform (JBC-P)
Common Infrared Countermeasures (CIRCM)
Joint Future Theater Lift Concept (JFTLC)
Common Remotely Operated Weapons System III
Joint Land Attack Cruise Missile Defense Elevated Netted Sensor System
Department of Defense Automated Biometric Information System
Joint Personnel Identification (JPIv2)
Distributed Common Ground System – Army (DCGS-A)
Joint Tactical Networks (JTN)
8
Program Oversight
Javelin Antitank Missile System – Medium
DOT&E Activity and oversight ARMY PROGRAMS (continued) Kiowa Warrior, OH-58F Cockpit and Sensor Upgrade Program (CASUP) Logistics Modernization Program (LMP)
Stryker M1132 Engineer Squad Vehicle Including the Double V-Hull Variant Stryker M1133 Medical Evacuation Vehicle Including the Double V-Hull Variant
M1200 Knight Targeting Under Armor (TUA) M270A1 Multiple Launch Rocket System (MLRS)
Stryker M1134 ATGM Vehicle Including the Double V-Hull Variant
M829E4 Modernized Expanded Capacity Vehicle (MECV) – Survivability Project MQ-1C Unmanned Aircraft System Gray Eagle
Stryker M1135 NBC Reconnaissance Vehicle (NBCRV) STRYKER MOD – STRYKER Modernization Program Tactical Mission Command
Nett Warrior (GSS)
Tactical Radio System Manpack
One System Remote Video Terminal
Tactical Radio System Rifleman Radio
Paladin/FASSV Integrated Management (PIM) PATRIOT PAC-3 – Patriot Advanced Capability 3 (Missile only) PATRIOT/MEADS – Patriot/Medium Extended Air Defense System RQ-11B Raven – Small Unmanned Aircraft System
UH-72A Lakota Light Utility Helicopter WIN-T INCREMENT 1 – Warfighter Information Network – Tactical Increment 1 WIN-T INCREMENT 2 – Warfighter Information Network – Tactical Increment 2
RQ-7B SHADOW – Tactical Unmanned Aircraft System Soldier Protection System
WIN-T INCREMENT 3 – Warfighter Information Network – Tactical Increment 3
Spider XM7 Network Command Munition Stryker M1126 Infantry Carrier Vehicle including Double V-Hull Variant Stryker M1127 Reconnaissance Vehicle
WIN-T INCREMENT 4 – Warfighter Information Network – Tactical Increment 4 XM1156 Precision Guidance Kit (PGK)
Stryker M1128 Mobile Gun System Stryker M1129 Mortar Carrier including the Double V-Hull Variant Stryker M1130 Commander’s Vehicle including the Double V-Hull Variant
XM25, Counter Defilade Target Engagement (CDTE) System XM395 Accelerated Precision Mortar Initiative (APMI)
Stryker M1131 Fire Support Vehicle Including the Double V-Hull Variant
NAVY PROGRAMS Acoustic Rapid COTS Insertion for SONAR
AN/SQQ-89A(V) Integrated USW Combat Systems Suite
Advanced Airborne Sensor
Anti-Torpedo Torpedo Defense System including all associated programs (Countermeasure Anti-Torpedo (CAT), Torpedo Warning System (TWS), and SLQ-25X (NIXIE))
Advanced Extremely High Frequency Navy Multiband Terminal Satellite Program (NMT) AEGIS Modernization AGM-88E Advanced Anti-Radiation Guided Missile AH-1Z AIM-9X – Air-to-Air Missile Upgrade Block II Air and Missile Defense Radar (AMDR) Air Warfare Ship Self Defense Enterprise Airborne Laser Mine Detection System (AN/AES-1) (ALMDS) Airborne Mine Neutralization System (AN/ASW-235) (AMNS) Airborne Resupply/Logistics for Seabasing Amphibious Assault Vehicle Upgrade Amphibious Combat Vehicle (ACV) AN/APR-39 Radar Warning Receiver AN/AQS-20A Minehunting Sonar An/BLQ-10 Submarine Electronics Support Measures AN/BVY-1 Integrated Submarine Imaging System
Assault Breaching System Coastal Battlefield Reconnaissance and Analysis System Block I Assault Breaching System Coastal Battlefield Reconnaissance and Analysis System Block II BYG-1 Fire Control (Weapon Control & TMA) CANES – Consolidated Afloat Networks and Enterprise Services CH-53K – Heavy Lift Replacement Program Close-In Weapon System (CIWS) including SEARAM COBRA JUDY REPLACEMENT – Ship-based radar system Common Aviation Command and Control System (CAC2S) Cooperative Engagement Capability (CEC) Countermeasure Anti-Torpedo CVN-78 – Electro-Magnetic Aircraft Launching System CVN-78 – Gerald R. Ford Class Nuclear Aircraft Carrier DDG 1000 – Zumwalt Class Destroyer – includes all supporting PARMs and the lethality of the LRLAP and 30mm ammunition
Program Oversight
9
DOT&E Activity and oversight NAVY PROGRAMS (continued) DDG 51 – Arleigh Burke Class Guided Missile Destroyer – includes all supporting PARMs
LPD 17 – San Antonio Class – Amphibious Transport Dock Ship – includes all supporting PARMs and 30mm lethality
DDG 51 Flight III – Arleigh Burke Class Guided Missile Destroyer – includes all supporting PARMs
Marine Personnel Carrier
Dept of Navy Large Aircraft Infrared Countermeasures Program
Medium Tactical Vehicle Replacement Program (USMC) (MTVR)
Distributed Common Ground System – Navy (DCGS-N) Distributed Common Ground System – Marine Corps (DCGS-MC) E-2D Advanced Hawkeye
Maritime Prepositioning Force (Future) Mobile Landing Platform MH-60R Multi-Mission Helicopter Upgrade MH-60S Multi-Mission Combat Support Helicopter
EA-18G – Airborne Electronic Attack variant of the F/A-18 aircraft
Mk 54 torpedo/MK 54 VLA/MK 54 Upgrades Including High Altitude ASW Weapon Capability (HAAWC)
Enhanced Combat Helmet
MK-48 CBASS Torpedo
Evolved Sea Sparrow Missile (ESSM)
MK-48 Torpedo Mods
Evolved Sea Sparrow Missile Block 2
Mobile Landing Platform (MLP) Core Capability Set (CCS) Variant and MLP Afloat Forward Staging Base (AFSB) Variant
F/A-18E/F – SUPER HORNET Naval Strike Fighter Future Pay and Personnel Management Solution (FPPS) Global Combat Support System – Marine Corps (GCSS-MC) Global Command and Control System – Maritime (GCCS-M) Griffin Interim Surface to Surface Missile (LCS SSM) Ground/Air Task Oriented Radar (G/ATOR) Identification Friend or Foe Mark XIIA Mode 5 (all development and integration programs) Infrared Search and Track System Integrated Defensive Electronic Countermeasures (All Blocks) JATAS – Joint and Allied Threat Awareness System Joint Expeditionary Fires Joint High Speed Vessel (JHSV) JOINT MRAP – Joint Mine Resistant Ambush Protected Vehicles FOV – including SOCOM vehicles Joint Precision Approach and Landing System Increment 1 (Ship system) Joint Precision Approach and Landing System Increment 2 (Land system) Joint Stand-Off Weapon C-1 variant (JSOW C-1) KC-130J with Harvest Hawk Landing Ship Dock Replacement (LX(R)) LHA-6 – America Class – Amphibious Assault Ship – includes all supporting PARMs LHD 8 Amphibious Assault Ship
Mobile User Objective System (MUOS) MQ-4C Triton MQ-8 – Vertical Takeoff and Land Tactical Unmanned Air Vehicle VTUAV (Fire Scout) Multi-static Active Coherent (MAC) System CNO project 1758 Naval Integrated Fire Control – Counter Air (NIFC-CA) From the Air Naval Integrated Fire Control – Counter Air (NIFC-CA) Navy Enterprise Resource Planning (ERP) Next Generation Enterprise Network (NGEN) Next Generation Jammer Offensive Anti-Surface Warfare Ohio Replacement Program (Sea-based Strategic Deterrence) – including all supporting PARMs OSPREY MV-22 – Joint Advanced Vertical Lift Aircraft P-8A Poseidon Program Remote Minehunting System (RMS) Replacement Oiler Rolling Airframe Missile (RAM) including RAM Block 1A Helicopter Aircraft Surface (HAS) and RAM Block 2 Programs Ship Self-Defense System (SSDS) Ship to Shore Connector Small Tactical Unmanned Aerial System (STUAS) – UAS Tier II
Light Armored Vehicle
SSN 774 Virginia Class Submarine
Light Weight Tow Torpedo Countermeasure (part of LCS ASW Mission Module)
SSN 784 Virginia Class Block III Submarine Standard Missile 2 (SM-2) including all mods
Littoral Combat Ship (LCS) – includes all supporting PARMs, and 57mm lethality
Standard Missile-6 (SM-6)
Littoral Combat Ship Mission Modules including 30mm and missile lethality Littoral Combat Ship Surface-to-Surface Missile Module (follow on to the interim Griffin Missile) Littoral Combat Ship Veriable Depth Sonar (LCS VDS) Logistics Vehicle System Replacement
10
Program Oversight
Submarine Torpedo Defense System (Sub TDS) including countermeasures and Next Generation Countermeasure System (NGCM) Surface Electronic Warfare Improvement Program (SEWIP) Block 2 Surface Electronic Warfare Improvement Program (SEWIP) Block 3 Surface Electronic Warfare Improvement Program Block 4 Surface Mine Countermeasures Unmanned Undersea Vehicle (also called Knifefish UUV) (SMCM UUV)
DOT&E Activity and oversight NAVY PROGRAMS (continued) Surveillance Towed Array Sonar System/Low Frequency Active (SURTASS/ LFA) including Compact LFA (CLFA) Torpedo Warning System (Previously included with Surface Ship Torpedo Defense System) including all sensors and decision tools TRIDENT II MISSILE – Sea Launched Ballistic Missile
Unmanned Carrier Launched Airborne Surveillance and Strike System Unmanned Influence Sweep System (UISS) include Unmanned Surface Vessel (USV) and Unmanned Surface Sweep System (US3) VXX – Presidential Helicopter Fleet Replacement Program
UH-1Y
AIR FORCE PROGRAMS 20 mm PGU-28/B Replacement Combat Round
F-15E Radar Modernization Program
Advanced Pilot Trainer
F-16 Combat Avionics Programmed Extension Suite
AEHF – Advanced Extremely High Frequency (AEHF) Satellite Program
F-22 – RAPTOR Advanced Tactical Fighter
AFNet Modernization capabilities (Bitlocker, Data at Rest (DaR), Situational Awareness Modernization (SAMP))
F-35 - Lightning II Joint Strike Fighter (JSF) Program
AFNET Vulnerability Management (AFVM) – Assured Compliance Assessment Solution (ACAS)
Full Scale Aerial Target
AIM-120 Advanced Medium-Range Air-to-Air Missile Air Force – Distributed Common Ground System (AF-DCGS) Air Force – Integrated Personnel and Pay System (AF-IPPS) Air Operations Center – Weapon System (AOC-WS) initiatives including 10.0 and 10.1
FAB-T – Family of beyond Line-of-Sight Terminals GBS – Global Broadcast Service Global Broadcast System (GBS) Defense Enterprise Computing Center (DECC) GLOBAL HAWK (RQ-4B) Block 30 – High Altitude Endurance Unmanned Aircraft System
Air Operations Center – Weapon System (AOC-WS) initiative 10.2
GLOBAL HAWK (RQ-4B) Block 40 – High Altitude Endurance Unmanned Aircraft System
Airborne Signals Intelligence Payload (ASIP) Family of Sensors
GPS OCX – Global Positioning Satellite Next Generation Control Segment
Airborne Warning and Control System Block 40/45 Computer and Display Upgrade
GPS-IIIA – Global Positioning Satellite III
ALR-69A Radar Warning Receiver
Identification Friend or Foe Mark XIIA Mode 5 (all development and integration programs)
B-2 Defensive Management System Modernization (DMS)
HC/MC-130 Recapitalization
B-2 EHF SATCOM AND COMPUTER INCREMENT I – B-2 Advanced Extremely High Frequency SatCom Capability
Integrated Strategic Planning and Analysis Network (ISPAN) Increment 2
B-2 EHF SATCOM AND COMPUTER INCREMENT II – B-2 Advanced Extremely High Frequency SatCom and Computer Capability
Joint Air-to-Surface Standoff Missile – Extended Range
B61 Mod 12 Life Extension Program Base Information Transport Infrastructure (BITI) – Wireless Battle Control System – Fixed (BCS-F) 3.2 C-130J – HERCULES Cargo Aircraft Program C-5 Aircraft Reliability Enhancement and Re-engining Program C-5 Core Mission Computer and Weather Radar Replacement Cobra Judy Replacement Mission Planning Tool Combat Rescue Helicopter (CRH) Command and Control Air Operations Suite (C2AOS)/Command and Control Information Services (C2IS) (Follow-on to Theater Battle Management Core Systems) Defense Enterprise Accounting and Management System – Air Force (DEAMS – AF) ECSS – Expeditionary Combat Support system Enclave Control Node (ECN) EPS – Enhanced Polar System F-15 Eagle Passive Active Warning Survivability System
Integrated Strategic Planning and Analysis Network (ISPAN) Increment 4 Joint Space Operations Center Mission System (JMS) KC-46 – Tanker Replacement Program Large Aircraft Infrared Countermeasures Long Range Stand Off (LRSO) Weapon Long Range Strike Bomber Massive Ordnance Penetrator (MOP) Military GPS User Equipment (GPS MGUE) Miniature Air Launched Decoy – Jammer (MALD-J) MQ-9 REAPER – Unmanned Aircraft System MQ-X Multi-Platform Radar Technology Insertion Program NAVSTAR Global Positioning System (GPS) (Includes Satellites, Control and User Equipment) OSPREY CV-22 - Joint Advanced Vertical Lift Aircraft Presidential Aircraft Recapitalization SBIRS HIGH – Space-Based Infrared System Program, High Component SBSS B10 Follow-on – Space-Based Space Surveillance Block 10 Follow-on
Program Oversight
11
DOT&E Activity and oversight AIR FORCE PROGRAMS (continued) SF – Space Fence
Three-Dimensional Expeditionary Long-Range Radar
SIPRNET Modernization
Weather Satellite Follow-on (WSF)
Small Diameter Bomb, Increment II
12
Program Oversight
DOT&E Activity and oversight
Problem Discovery Affecting OT&E Adequate developmental and operational testing are essential for determining whether systems provide an effective, suitable, and survivable warfighting capability to our Soldiers, Sailors, Airmen, and Marines. Developmental testing, in particular, serves as an early means to identify problems in the performance of weapon systems. The later a performance problem is discovered in a program’s development timeline, the more costly and more difficult it is to correct it. Provided it is done adequately and rigorously, developmental testing also serves to determine if a program is ready for operational testing. Furthermore, discovery in operational testing has the potential to delay fielding while problems are corrected, or in the worst case, reveal a fatal flaw; neither of which is desirable. Background In 2010, Congress expressed concern that significant problems with acquisition programs are being discovered during operational testing that: (1) should have been discovered in development testing and (2) should have been corrected prior to operational testing. In response to this congressional request, I added this section to my annual report as a means to survey, across all DOT&E oversight programs, the extent of problem discovery occurring late in program development. Unfortunately, each year, operational testing continues to reveal performance problems for a significant number of programs that should have been discovered in developmental testing. Evaluation of Problem Discovery My evaluation of this issue falls into several cases, which are illustrated in Figure 1: • Case 1. In the worst case (illustrated in red), problems were discovered solely in operational testing. The implication is that developmental testing (DT) was not conducted or was not adequate to uncover the problem prior to operational testing (OT). These cases illustrate that when decision makers focus too much on budget and schedule and not enough on the outcomes of testing (and the need to conduct adequate developmental testing), there is an increased likelihood of observing problems in operational testing. • Case 2. A second case (illustrated in orange) includes those programs where problems were observed in operational testing that were also observed in developmental testing prior to the operational test period. Here, the implication is that the program chose to proceed to operational testing and accept the risk of potentially experiencing a poor
operational testing outcome. Unfortunately, the problems were observed again and had an adverse effect on the determination of operational effectiveness, suitability, and/or survivability: a situation that is entirely avoidable. • Cases 3 and 4. Two additional cases, illustrated at the bottom of Figure 1, show the desired paradigm: early testing is conducted; problems with system performance are uncovered and recognized for their potential effect on the upcoming determination of effectiveness, suitability, and survivability; and the program has the opportunity to resolve problems before entering operational testing. - In Case 3, programs made the decision to correct the problem(s) identified in early testing, which is laudable in light of the fact that it delayed the program and its entry into operational testing. - In Case 4, early testing uncovered problems, and the program has an opportunity to correct the problems. For this case, I recommend the program take action to address the issue before proceeding to the IOT&E/FOT&E period. It is noteworthy that many of the problems identified early were discovered during an operational assessment or limited user test; this reveals the value of conducting such early operationally realistic test events. I have expanded this section of the report over previous years, with specific details provided to enable programs to take action. My discussion below identifies programs applicable to each of these cases and includes the reasons (if known) specific to each program.
Figure 1. Illustration of Problem Discovery Cases Observed in Oversight Programs
Problem Discovery Affecting OT&E
13
DOT&E Activity and oversight Conclusions Some of the cases discussed below reveal that problem discovery only could have occurred in operational testing because that is when the operational implications of a performance deficiency become clear. This again reflects the value of operational testing – without such testing, the problems would have been discovered by the Services during operational use, and in the worst case, during actual conflict. There will always be a need for operational testing; nonetheless, in most of the cases below, the discovery of problems in operational testing was entirely avoidable. Several solutions exist to curb the trends observed here: • Programs should generate and execute schedules that allow adequate time for thorough developmental testing, and time to troubleshoot and resolve deficiencies. The results of testing should be used to guide program development decisions, including the need to extend developmental testing (and potentially delay operational testing until problems are corrected), and to ensure the system will meet its intended operational use. • Programs should conduct developmental testing with a focus on the mission. In some cases, this will require developmental testing to go beyond specification compliance testing to demonstrate the desired system performance in an operational context. • Services should develop concepts of operations and concepts of employment earlier so that developers can better understand how the system will be used in the field and can inform both system design and developmental test design. • The requirements and acquisition communities need to work closely to develop requirement documents that ensure specification requirements are written to incentivize contractors and program managers to focus on demonstrating mission capabilities. These requirements should also clearly define performance expectations across the conditions the system is intended to be used, not just for a narrowly defined set of conditions. • Often, effectiveness shortfalls and/or suitability shortfalls found in operational testing are discovered because operational use profiles (how the Soldier uses the equipment) reveal failure modes (reliability) or performance shortfalls that are unique to the operational test environment; such shortfalls would not have been revealed under the more structured, controlled, and benign conditions common to development testing. Development testing is often limited to verifying narrowly-defined requirements regardless of the operational relevance of those specifications. When the user takes the system to more operationally realistic conditions (more difficult threats; more difficult, but still relevant, operational environments), these performance failures are discovered. The Deputy Assistant Secretary of Defense (DASD) Developmental Test and Evaluation (DT&E) is implementing
14
Problem Discovery Affecting OT&E
initiatives consistent with these solutions that will be discussed in that office’s upcoming report. If requirements are set in a manner to ensure high performance under benign conditions, then developmental testing will likely only examine performance in those specified conditions. Therefore, well-defined requirements, especially the contractual specifications that are derived from the system’s concept of employment, can help drive the developmental testing to examine performance under the conditions expected in the field. Furthermore, the early test events should also provide information to the requirements and resource sponsors for the system to ensure that the documented requirements are still relevant and feasible. Operational testing, by definition, must examine performance across the expected operational envelope. Summary In 2013, 44 programs had significant problem discovery affecting OT&E. Of these, 12 are considered to be Case 1, meaning problems were discovered solely in operational testing (IOT&E or FOT&E). Ten programs fall into the Case 2 category, where problems that were identified in developmental testing were re-identified in operational testing. Six programs are considered to be Case 3, where problems were discovered in early testing and the program delayed operational testing to correct the problem. For these cases, I consider the developmental test and evaluation process to have been successful and the program to have responded appropriately. The remaining 16 programs fall under Case 4, where early testing has identified problems that need to be corrected. The value of this early identification of programs cannot be overstated. The benefit is lost, however, if these deficiencies are not corrected prior to IOT&E. I have also included an assessment of cybersecurity vulnerabilities discovered during operational testing. I categorize these discoveries under Case 1, as they should have been discovered earlier in the systems’ development. Operational testing of 33 programs in FY12 and FY13 revealed over 400 cybersecurity vulnerabilities, about 90 percent of which could have been found and corrected earlier in the systems’ development. I also provide updates to the problem discovery cases listed in my FY12 Annual Report. Last year, I documented 23 systems with significant discovery during testing: 6 of those systems had discovery in early testing, of which 5 implemented fixes that were verified by successful OT&E, are currently in OT&E, or are planning OT&E. Of the 17 programs that discovered significant issues during their IOT&E in 2011-2012, 10 have implemented fixes that were either verified in successful OT&E or are planning additional operational test periods; 2 of the remaining 7 programs were cancelled. Thus, while significant issues are being discovered late in the acquisition cycle, most programs are addressing the discoveries and verifying fixes in follow-on operational testing.
DOT&E Activity and oversight CASE 1: PROBLEMS DISCOVERED IN 2013 DURING OPERATIONAL TESTING THAT SHOULD HAVE BEEN DISCOVERED DURING DEVELOPMENTAL TESTING
IOT&Es in FY13 with Discovery AIM-9X Air-to-Air Missile Upgrade
OTs (other than IOT&E) in FY13 with Discovery Acoustic Rapid Commercial Off-the-Shelf Insertion (A-RCI) AN/BQQ‑10 (V) Submarine Sonar System
AIM-120 Advanced Medium-Range Air-to-Air Missile (AMRAAM)
Defense Enterprise Accounting and Management System (DEAMS)
Joint Battle Command – Platform (JBC-P)
DoD Automated Biometric Identification System (ABIS)
Miniature Air Launched Decoy (MALD) and MALD-Jammer (MALD-J)
Mk 54 Lightweight Torpedo
Multi-Static Active Coherent (MAC) System
Public Key Infrastructure (PKI) Increment 2
Surveillance Towed Array Sensor System (SURTASS) and Compact Low Frequency Active (CLFA)
Warfighter Information Network – Tactical (WIN-T)
All Programs Tested in FY12-13: Discovery of Cybersecurity Vulnerabilities
Acoustic Rapid Commercial Off-the-Shelf Insertion (A-RCI) AN/BQQ-10 (V) Submarine Sonar System A-RCI is composed of the computer processors and displays that process the data collected from submarines’ acoustic arrays. It encompasses the primary components of U.S. submarines’ combat systems and enables submarines to conduct all missions. The active operating mode of the Low Cost Conformal Array (LCCA), the mode in which the sonar pings and listens for the echoes, was unable to be evaluated due to a flaw in system software. Due to coding problems, the sonar was incapable of functioning in high reverberation environments, making detection of ships nearly impossible.
Early testing did not catch the problem because the software issue was not apparent in the more benign environmental conditions of the early developmental testing. The problem was discovered just hours before the commencement of the operational test of the system. Because of the late discovery, operational testing of the remaining components of the sonar system proceeded without examining the active operating mode capability. Subsequent to the operational test, the Navy developed a software update to correct this issue and verified proper functionality with in-lab testing, including playback and analysis of recorded at-sea data. Operational testing of the active operating mode of the LCCA with this software update is still required and has not yet been conducted.
AIM-9X Air-to-Air Missile Upgrade AIM-9X is the latest generation short-range, heat-seeking, air-to-air missile. IOT&E of the AIM-9X Block II missile was paused in April 2013 after multiple flight test failures. Two hardware reliability failures were traced to poor manufacturing. Additionally, IOT&E revealed problems with missile guidance. Missiles made porpoise-like maneuvers that contributed to misses when combined with inertial measurement units that showed errors occurring after launch shock. This launch shock problem occurred once during developmental testing, but the missile guided successfully to target. Currently the Program Office is pursuing root cause investigation with poor inertial measurement hardware units and guidance, navigation, and control (GNC) software as possible causes. AIM-120 Advanced Medium-Range Air-to-Air Missile (AMRAAM) The AIM-120 AMRAAM is a radar-guided air to-air missile with capability in both the beyond-visual-range and within-visual-range arenas. A single launch aircraft can engage multiple targets with multiple missiles simultaneously when using AMRAAM. Problems affecting missile performance and suitability were discovered in IOT&E in FY12, and the IOT&E was suspended until the problems were resolved. Specific details are classified. IOT&E resumed in May 2013, but the program continues to experience delays, and IOT&E is not projected to be complete until FY14.
Problem Discovery Affecting OT&E
15
DOT&E Activity and oversight Defense Enterprise Accounting and Management System (DEAMS) DEAMS replaces legacy systems using an enterprise architecture with commercial off-the-shelf (COTS)-based financial accounting software (such as general ledger, accounts payable, accounts receivable, financial reporting, and billing). An initial operational assessment (OA-1) occurred in 2012, commensurate with the initial limited deployment of the system. The Air Force Operational Test and Evaluation Center began a second operational assessment (OA-2) of DEAMS Release 2.2 in August 2013, with the intent to determine if the issues discovered during OA-1 were remedied, and that processes and procedures had been put in place to allow for continued operational use. Although the OA was not a formal IOT&E, it was conducted on a live and fielded system; many of the problems discovered could have been found earlier had adequate developmental testing been conducted. Results of OA-1 and initial deployment indicated numerous software defects (over 200) and showed that there was essentially no method or process for adequate configuration control. Furthermore, the live system was used to troubleshoot and fix severe deficiencies instead of employing a robust developmental regression testing process. A degree of regression testing automation is being employed that should reduce developmental test time and allow for greater depth of testing in future code development. DoD Automated Biometric Identification System (ABIS) The DoD ABIS is the result of a Joint Urgent Operational Need request and consists of information technology components and biometric examiner experts that receive, process, and store biometrics from collection assets across the world, match new biometrics against previously stored assets, and update stored records with new biometrics and contextual data to positively identify and verify actual or potential adversaries. While operational as ABIS 1.0, the system has not had any formal OT&E in its over 10-year existence, with only limited testing done by the Program Management Office and users to support new software releases, specifically ABIS 1.2. Since 2010, there have been four failed attempts to deploy the ABIS upgrade, with the latest failed attempt in August 2013. The upgrade disabled critical interfaces with ABIS customers, preventing high-priority customers from receiving timely, accurate match results while maintaining compliance with established sharing agreements. The Director, Defense Forensics and Biometrics Agency recommended that the legacy ABIS 1.0 be restored after customers reported significant operational impacts to missions. Issues discovered during these deployment attempts should have been found beforehand through developmental test and evaluation. Joint Battle Command – Platform (JBC-P) JBC-P is a multi-Service situational awareness and mission command tool that automatically propagates the position of friendly forces, allows friendly forces to manually place allied and threat elements, and allows units to send preformatted and
16
Problem Discovery Affecting OT&E
free-text messages across echelons from individual vehicles to Corps headquarters. The JBC-P system exhibited problems in operational testing that were not identified in developmental testing, including spontaneous computer reboots, software unpredictability, and message management problems (duplicate entries and message format changes during transmission). Reliability failure modes were observed in the IOT&E that had not been observed in previous developmental testing, which indicates that the system’s software development was immature. Miniature Air Launched Decoy (MALD) and MALD-Jammer (MALD-J) MALD is a small, low-cost, expendable, air-launched vehicle that replicates how fighter, attack, and bomber aircraft appear to enemy radar operators. The Air Force designed the MALD-J as an expendable, close-in jammer to degrade and deny an early warning or acquisition radar’s ability to establish a track on strike aircraft while maintaining the ability to fulfill the MALD decoy mission. MALD-J IOT&E was conducted throughout FY13. The MALD and its follow-on MALD-J variant have been extensively tested over a number of years. However, the MALD-J variant poses significant potential for self-interference and is particularly reliant on accurate navigation to remain effective. All MALD-J vehicles launched during developmental testing performed within the navigational accuracy requirements. During IOT&E at an open-air flight test range (a more challenging operationally representative environment), several MALD-J vehicles experienced unexpected navigational accuracy issues. There were several different causes of the navigational errors, all classified, but all arose from technical performance issues that should have been uncovered during developmental testing. Mk 54 Lightweight Torpedo The Mk 54 Lightweight Torpedo is the primary Anti-Submarine Warfare (ASW) weapon used by U.S. surface ships, fixed-wing aircraft, and helicopters. In May 2013, for one phase of operational testing of the Mk 54 torpedo with Block Upgrade software, the Navy planned to launch the weapons from MH-60R helicopters against a stationary submarine surrogate target off the coast of California. The plans called for the use of specific torpedo tactical presets that had been optimized for this scenario. This preset had not been examined in developmental testing. Discussions between fleet aviation personnel, Navy testers, and torpedo developers revealed that the MH-60R could not execute the desired presets and that published tactical guidance and documentation were inaccurate. This incident led to a broader Navy investigation that identified gaps in communication and coordination between the undersea warfare community, which manages the torpedo programs, and the Naval aviation community, which is responsible for airborne fire control systems and tactical development.
DOT&E Activity and oversight Multi-Static Active Coherent (MAC) System The MAC system is an active sonar system composed of two types of buoys (source and receiver) and an acoustic processing software suite. It is employed by the Navy’s maritime patrol aircraft (P-3Cs and eventually P-8As) to search for and locate threat submarines in a variety of ocean conditions. During operational testing of the MAC sonobuoys system, P-3C maritime patrol aircraft deployed and monitored large fields of these sonar sensors in order to search for target submarines. As per approved test plans, the Navy conducted the tests at various sites in order to evaluate MAC detection capability in a variety of acoustic environments. Relevant conditions include sound speed profile, ambient noise, bathymetric profile, and bottom composition. Testing revealed that the presentation of a valid target to the operator can vary significantly between environments and likely target types, making operator training and recognition of target-specific characteristics critical to performance. These differences were not identified in developmental testing, since all developmental testing was restricted to an environment where these effects could not have been studied. Data from a May 2013 test had to be invalidated because of the discovery of the phenomenon during the operational testing. Based on the data collected in operational testing, the Navy revised the employment concept and conducted additional training for the crews, and then repeated the operational test in October 2013. Public Key Infrastructure (PKI) Increment 2 PKI Increment 2 provides authenticated identity management via password-protected Secret Internet Protocol Routing Network (SIPRNet) tokens to enable DoD members and others to access the SIPRNet securely, and encrypt and digitally sign e-mail. The Joint Interoperability Test Command conducted a combined FOT&E I and II of the PKI Increment 2 from January 8 through February 1, 2013, to verify correction of system deficiencies discovered during the IOT&E in 2011 for Spirals 1 and 2, and to evaluate preliminary Spiral 3 enhancements, respectively. The FOT&Es were originally scheduled to be completed in FY12, but were postponed due to system development delays. Furthermore, a stop-test in December 2012 resulted from systemic configuration management problems and lack of coordinated test-preparation. Delays in delivering the Integrated Logistics System (ILS) capability for token ordering and shipping contributed to delays in the delivery of several key Spiral 3 capabilities, including an Alternate Token Capability to support system administrator roles on the SIPRNet. The FOT&E identified problems with blacklisting and token reuse in the token management system, and the operational testing exposed usability and auditing problems in ILS; none of these areas were adequately examined during developmental testing. The ILS was not effective for tracking tokens returned for reuse, was cumbersome to use, and did not provide the necessary functions to replace existing spreadsheet tracking
mechanisms. More operationally relevant use cases should have been executed during developmental testing to avoid discovering these problems in the operational test. System user involvement in developmental testing likely would have identified ILS inadequacies early in the system design and development. Surveillance Towed Array Sensor System (SURTASS) and Compact Low Frequency Active (CLFA) SURTASS/CLFA is a low frequency, passive and active acoustic surveillance system installed on tactical auxiliary general ocean surveillance ships as a component of the Integrated Undersea Surveillance System. The Navy conducted the first phase of IOT&E in the Western Pacific in September 2012 to evaluate the ability of SURTASS/CLFA to detect submarine targets at long ranges as part of a large area search. The test revealed that the system is prone to detecting surface ships and presenting them as valid submarine targets, creating a false alarm problem. Although similar results were seen in developmental testing, the significance of the problem was only made clear when the system was put in an operationally realistic war time scenario. Warfighter Information Network – Tactical (WIN-T) WIN-T is a three-tiered communications architecture (space, terrestrial, and airborne) serving as the Army’s high-speed and high-capacity tactical communications network. Testing of the WIN-T vehicle kits, specifically the Soldier Network Extension and the Point of Presence, during the WIN-T IOT&E in May 2012 and the WIN-T FOT&E in May 2013 showed that the systems were too complex for Soldier operation and troubleshooting. Additionally, mission command applications were sluggish. These key problems were not identified in the Risk Reduction Events (conducted at contractor facilities using engineers as operators) held prior to the operational tests. Discovery of Cybersecurity Vulnerabilities Where appropriate, programs that conducted operational testing in FY13 included a cybersecurity assessment – suitably scoped for the system under test – as part of the operational test program. DOT&E assessed 33 of these programs from FY12 and FY13 whose operational tests included cybersecurity assessments. Over 400 Information Assurance (cybersecurity) vulnerabilities were uncovered during the vulnerability assessment and/or the penetration testing that occurred during the operational test period. Of those, approximately half were serious (Category 1) vulnerabilities that could allow debilitating compromise to a system, and approximately three-quarters of the systems reviewed had one or more serious vulnerabilities. The three most common Category 1 vulnerabilities were: (1) out‑of‑date/unpatched software, (2) configurations that included known code vulnerabilities, and (3) the use of default passwords in fielded systems. All of the problem discoveries could have and should have been identified prior to operational testing.
Problem Discovery Affecting OT&E
17
DOT&E Activity and oversight An assessment of the problems found reveals that only about 11 percent of those 400 vulnerabilities required an operational environment/operational test to uncover; 89 percent of the 400 vulnerabilities found in FY12 and FY13 could have been found in developmental testing. The review did not demonstrate whether these vulnerabilities were discovered in developmental testing but not remediated (Case 2 below), or if they were uniquely discovered in operational testing due to an inadequate developmental test process. However, the fact
18
Problem Discovery Affecting OT&E
that so many vulnerabilities are being found late in a program’s acquisition cycle is one of the main reasons why DOT&E and USD(AT&L) are collaborating on a revised cybersecurity policy. There is general agreement that systems must be assessed for cybersecurity earlier in a system’s development. Testing over the past several years has indicated the need to move the discovery and resolution of system vulnerabilities earlier in program development, and the revised cybersecurity T&E process addresses this need.
DOT&E Activity and oversight CASE 2: PROBLEMS IDENTIFIED IN DT&E THAT WERE RE-IDENTIFIED IN OT&E Beginning this year I am reporting findings for oversight programs for which problems were identified in DT&E and then were re-identified in OT&E (10 programs). This is illustrated as the second type of undesirable problem discovery, since it could have been avoided.
PROBLEMS IDENTIFIED IN DT&E THAT WERE RE-IDENTIFIED IN OT&E AIM-120 Advanced Medium-Range Air-to-Air Missile (AMRAAM)
Global Command and Control System – Joint (GCCS -J)
Cooperative Engagement Capability (CEC)
H-1 Upgrades – U.S. Marine Corps Upgrade to AH-1Z Attack Helicopter and UH-1Y Utility Helicopter
E-2D Advanced Hawkeye
Handheld, Manpack, and Small Form Fit (HMS) Manpack Radio
F-15E Radar Modernization Program (RMP)
Mission Planning System (MPS)/Joint Mission Planning System – Air Force (JMPS-AF)
Global Broadcast System (GBS)
P-8A Poseidon Multi-Mission Maritime Aircraft (MMA)
AIM-120 Advanced Medium-Range Air-to-Air Missile (AMRAAM) AIM-120 AMRAAM is a radar-guided air to-air missile with • Because CEC software deficiencies that caused the CEC capability in both the beyond-visual-range and within‑visual‑range system to create multiple tracks for the same contact were arenas. IOT&E began in 2012. Problems that had been identified still occurring at the start of the E-2D IOT&E, CEC testing in DT&E reoccurred, which caused a pause in the IOT&E until was decoupled from the E-2D IOT&E. The multiple track May 2013. Specific details are classified. problem remained during the CEC FOT&E that occurred immediately after the E-2D IOT&E. Cooperative Engagement Capability (CEC) • Radar track re-labeling was observed in developmental The CEC is a system of hardware and software that allows the testing, but the full magnitude of the problem only sharing of radar and weapons systems data on air targets among manifested itself under the conditions of IOT&E. U.S. Navy ships, U.S. Navy aircraft, and some U.S. Marine Corps • Poor radar reliability and availability were seen in units. Developmental testing of the USG-3B CEC variant developmental testing and persisted into IOT&E. installed on the E-2D Advanced Hawkeye, conducted in FY12, revealed problems with the system’s determination of relative F-15E Radar Modernization Program (RMP) sensor alignment, problems related to the system’s capability to The F-15E is a twin engine, tandem seat, fixed-wing, all maintain a consistent air contacts picture on other CEC platforms weather, multi-role fighter aircraft. The RMP replaces the (such as CEC-equipped ships and E-2Ds), and reliability F-15E legacy APG-70 mechanically scanned radar with problems. These problems were re-discovered during FOT&Es an active electronically scanned array system designated conducted in FY13. the APG 82(V)1, and is designed to retain functionality of the legacy radar system while providing expanded mission E-2D Advanced Hawkeye employment capabilities. F-15E RMP developmental flight The E-2D Advanced Hawkeye is a carrier-based Airborne testing began in January 2011. IOT&E started in April 2013 Early Warning and Command and Control aircraft. The Navy and completed in September 2013. The program experienced conducted the E-2D IOT&E from February to September 2012. software maturation challenges during developmental test. Four major deficiencies, found during developmental testing, Radar software maturity anomalies resulted in multiple were also observed during the IOT&E: unplanned software releases requiring additional regression • Accuracy issues found in developmental testing still existed testing to mature the radar functionality. in IOT&E.
Problem Discovery Affecting OT&E
19
DOT&E Activity and oversight The program originally intended that later operational flight program releases would focus on software stability/Mean Time between Software Anomaly (MTBSA) fixes without additional functionality and performance changes. Due to challenges in maturing performance and functionality, the program exhausted its developmental schedule and funding before achieving the user’s MTBSA requirement. Preliminary results from operational testing show software stability performance did not meet the 30-hour MTBSA goal, as predicted in the FY12 Annual Report.
Global Broadcast System (GBS) The GBS is a one-way satellite communications system that works in a manner similar to satellite television. The Defense Enterprise Computing Center (DECC) upgrade consolidates several Navy ground sites into a single facility that creates broadcasts and provides technical support to users. The Air Force conducted a Force Development Evaluation of the GBS DECC upgrade from July through September 2013 at the Oklahoma City, Oklahoma DECC site; Mechanicsburg, Pennsylvania, DECC site; and Schriever AFB, Colorado. Problems were discovered in developmental testing when users attempted to reauthorize receive suites to participate in the network. The program took corrective actions, but because of cost and schedule constraints, chose not to conduct additional developmental testing to verify these corrective actions were sufficient to provide system restoral capability. During operational testing, the same problems were seen. The inexperience of personnel, poor operating procedures, and technical shortcomings were noted in previous developmental testing. Operational testing found similar deficiencies. Training and documentation for the GBS Operations Center personnel were not suitable for troubleshooting GBS user problems. Operations Center personnel needed to call contractor support to resolve more than half of the technical help desk tickets submitted during the operational test. Also, while transitioning from the main site at Oklahoma City to the backup site at Mechanicsburg, the absence of automated processes for reauthorizing users contributed to the extended time it took to restore service to all GBS users. The program knowingly entered operational testing with these immature procedures in place. Global Command and Control System – Joint (GCCS-J) GCCS-J is a command and control system utilizing communications, computers, and intelligence capabilities. The system consists of hardware, software (commercial and government off-the-shelf), procedures, standards, and interfaces that provide an integrated near real-time picture of the battlespace necessary to conduct joint and multi-national operations. Operational testing of GCCS-J version 4.3 Global was originally planned for May 2013; however, because of system immaturity, the program decided to conduct additional developmental testing to allow more time to find and fix deficiencies. Operational testing was conducted in August 2013,
20
Problem Discovery Affecting OT&E
and while not adequate, was sufficient to determine that the system is not effective and not suitable. While laudable that the program delayed operational testing to conduct additional developmental testing, several significant deficiencies were identified again during the second developmental test period, and the program did not again delay entry into operational testing, where the deficiencies were found again. Deficiencies included: • Target lists that have been created and locked in GCCS-J 4.3 cannot be opened as read only using legacy versions of GCCS-J. • The fielded version of the Generic Area Limitation Environment used to process electronic intelligence data could not pass processed data to the GCCS-J Common Operational Picture. • Target lists take too long to replicate between GCCS-J 4.3 and legacy versions of GCCS-J. This issue was also seen during developmental testing, and must be retested using an operationally relevant test server. • When large target lists are being synchronized across multiple versions of GCCS-J, the list is marked “validated” or “approved” before the synchronization process has completed. This will require a change to the synchronization process, followed by retesting using an operationally relevant test server. • The process of upgrading the target folders in the new database structure resulted in incorrect security classification markings being used. At a minimum, the target folder should reflect the highest classification level of any information contained in the target folder. H-1 Upgrades – U.S. Marine Corps Upgrade to AH-1Z Attack Helicopter and UH-1Y Utility Helicopter This program upgrades the AH-1W attack helicopter to AH-1Z and the UH-1N utility helicopter to the UH-1Y. In 2010, the Navy began full-rate production and fielding of the AH-1Z aircraft following successful completion of Phase III IOT&E. Since 2010, the Navy has continued to develop software to correct previously noted deficiencies and provide new capabilities. By 2012, Software Configuration Set (SCS) version 6.0 had become mature enough to warrant FOT&E before fielding the new version. The Navy requested that Commander, Operational Test and Evaluation Force conduct FOT&E (OT‑IIIB) of the new version of software. Effectiveness, suitability, and survivability of H-1 Upgrades aircraft with SCS 6.0 are degraded by occasional software blanking of the electronic warfare display. If SCS 6.0 detects any failure (actual or false) in the aircraft survivability equipment (APR-39 and AAR-47), SCS 6.0 causes the electronic warfare display to go blank. Manual deployment of chaff and flares remains possible. Although detected during developmental testing, the operational implications of this loss of electronic warfare situational awareness were not apparent until operational testing.
DOT&E Activity and oversight Handheld, Manpack, and Small Form Fit (HMS) Manpack Radio The HMS program evolved from the Joint Tactical Radio System program and provides software-programmable digital radios to support tactical communications requirements. The Manpack radio is a two-channel radio with military GPS. The Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) stated in 2012 that the Manpack radio was not sufficiently mature to enter Multi‑Service Operational Test and Evaluation (MOT&E). Waveform performance, particularly for the Single Channel Ground and Airborne Radio System (SINCGARS) was poor, and reliability was very low. However, the Army proceeded to conduct the MOT&E. DOT&E assessed the Manpack as not operationally effective and not operationally suitable, primarily because of SINCGARS performance and low reliability. The Army has not conducted operational testing since the May 2012 MOT&E to demonstrate improvements to Manpack. There have been multiple low-rate initial production procurements totaling 5,326 radios, and the Army has fielded the system to the 101st Airborne Division. Mission Planning System (MPS)/Joint Mission Planning System – Air Force (JMPS-AF) MPS is a package of common and platform-unique mission planning applications. The IOT&E for the JMPS Mission Planning Environment version 1.3 for the E-8 Joint Surveillance Target Attack Radar System began in 2011. During this initial phase, incorrect magnetic variation computations and unreliability of the process to transfer mission planning data to the aircraft were uncovered; these problems had also been observed in developmental testing prior to IOT&E. The operational test was paused and restarted more than a year later to ensure that these deficiencies had been corrected. The program went back into testing in 1QFY13, demonstrating that these two deficiencies were corrected. Other problems observed during developmental testing and found again during the first phase of the IOT&E include: • The system’s inability to automatically calculate flight plans with orbits based on user inputs • Problems calculating take-off and landing data • Failures in the implementation of vector vertical obstruction data P-8A Poseidon Multi-Mission Maritime Aircraft (MMA) The P-8A Poseidon MMA is a fixed-wing aircraft that will replace the P-3C Orion; its primary mission is to detect, identify, track, and destroy submarine targets (ASW), but it also is intended to conduct Anti-Surface Ship Warfare and Intelligence, Surveillance, and Reconnaissance (ISR). The Navy conducted IOT&E of the P-8A Increment 1 from September 2012
through March 2013. Nearly all of the major deficiencies that were identified during the developmental test period were re‑discovered during the IOT&E; many of these deficiencies led to DOT&E determining that P-8A is not effective for the ISR mission and is unable to execute the full range of ASW Concept of Operations at its Initial Operational Capability (IOC). Prior to IOT&E, DOT&E sent two memoranda to the Navy emphasizing the potential operational impact of critical performance deficiencies identified during developmental testing. • Synthetic Aperture Radar imagery collection capabilities were severely limited due to radar stability problems, target cueing errors, and image quality problems, which severely degraded ISR mission performance. • Communication and data transfer system interoperability problems limited receipt of tactical intelligence updates and transmission of P-8A imagery intelligence products to operational users. • Electronic Support Measures deficiencies limited threat detection and localization, seriously degrading capabilities and aircraft survivability across all major missions. • Developmental testing identified significant maritime surface target tracking errors while operating in the radar track‑while‑scan mode. Operational testing confirmed and further quantified these errors, which degrade operator capabilities to maintain an accurate surface operational picture while executing mission operations. Detailed DOT&E analysis of developmental test results indicated that the P-8 radar was not meeting detection requirements for some types of critical surface targets. Operational testing confirmed these results and characterized the operational impact of the performance limitations on the ASW mission. Additional details are classified and can be found in DOT&E’s October 2013 IOT&E report. Although the P-8A Increment 1 system provides an effective small area, cued ASW search, localization, and attack mission capability, similar to the legacy P-3C system, the Navy’s decision to cancel plans to integrate the Improved Extended Echo Ranging capability into P-8A ensured that the aircraft would have no wide‑area ASW search capability at IOC. Additionally, fundamental limitations with the P-8A’s current sensor technology restrict search capabilities against more stressing adversary targets, making the P-8A not effective at ASW in some mission scenarios. The Navy intends to use the Multi-static Active Coherent (MAC) sonobuoy system to address these shortfalls, and will test the capability in the P-8A Increment 2 program. The Navy plans to conduct additional developmental testing after the IOT&E to verify the correction of some of the system deficiencies identified during IOT&E.
Problem Discovery Affecting OT&E
21
DOT&E Activity and oversight CASE 3: PROBLEMS DISCOVERED IN EARLY TESTING AND THE PROGRAM WAS DELAYED TO CORRECT THE PROBLEM These cases could be considered instances in which the developmental test and evaluation process was successful and the program responded appropriately. Early testing can be both early developmental testing as well as operational assessments conducted prior to Milestone C. The latter have proven to be essential for identifying problems early.
PROBLEMS IDENTIFIED IN DT&E THAT DELAYED OT&E Air Operations Center – Weapons System (AOC-WS)
Ground/Air Task Oriented Radar (G/ATOR)
Battle Control System – Fixed (BCS-F)
Small Tactical Unmanned Aerial System (STUAS) Tier II
F/A-18E/F Super Hornet and EA-18G Growler
Vertical Take-Off and Landing Unmanned Aerial Vehicle (VTUAV) (Fire Scout)
Air Operations Center – Weapons System (AOC-WS) The AOC-WS is the senior command and control element of the U.S. Air Force’s Theater Air Control System and provides operational-level command and control of air, space, and cyberspace operations, as well as joint and combined air, space, and cyberspace operations. The Air Force originally planned to conduct both developmental and operational testing of AOC‑WS 10.1 Recurring Event (RE)12 in December 2012. The AOC‑WS 10.1 RE12 test article and associated documentation that entered operational testing in August 2013 was the direct output of a thorough developmental test-fix-test cycle. Extended developmental test and evaluation efforts ensured that this test article successfully passed operational test Phase II without any significant deficiencies. The RE12 test article in December 2012 was built on top of a flawed RE11 test baseline. The developmental test process recommended a clean rebuild of the RE11 baseline, followed by a rebuild of the RE12 test article. This was consistent with the plan for fielding to operational sites. Developmental testing in December 2012 identified 2 known significant deficiencies that had not been fixed and 10 new significant deficiencies. The developmental test-fix-test cycle continued until all significant deficiencies were verified fixed. Battle Control System – Fixed (BCS-F) The BCS-F is a tactical air battle management command and control system that provides the two continental U.S. North American Aerospace Defense Command (NORAD) air defense sectors, as well as the Hawaii and Alaska Regional Air Operation Centers, with COTS hardware using an open‑architecture
22
Problem Discovery Affecting OT&E
software configuration. The system operates within the NORAD air defense architecture and is employed by the U.S. and Canada. During developmental testing, several problems were found with the hardware and software configurations of the servers, firewalls, intrusion detection systems, and system guards that generated vulnerabilities in the system’s defenses. The start of IOT&E was delayed while the contractor and Program Office corrected the deficiencies and tested the corrections to ensure the deficiencies were fixed. A key problem underlying many of the deficiencies was that the documentation was insufficient, which contributed to problems with software installation and configuration. F/A-18E/F Super Hornet and EA-18G Growler The Super Hornet is the Navy’s premier strike-fighter aircraft that replaces earlier F/A-18 variants in carrier air wings. The F/A-18E/F software is being incrementally upgraded. The most recent software version is known as Software Configuration Set (SCS) H8E. Phase 1 of operational testing for SCS H8E took place from June 2012 to May 2013 after a delay of six months, because the Navy discovered problems during developmental testing in 6 of the 14 new SCS H8E capabilities. Ultimately these problematic capabilities were deferred to a later operational test and SCS H8E (Phase 1) proceeded with the remaining planned capabilities. Several of these deferrals resulted from the Navy’s difficulty in integrating electronics support on the Super Hornet while others would have allowed the aircraft to detect the position of an
DOT&E Activity and oversight emitter using onboard sensors only, integrate the latest version of a self-protection jammer, and navigate through civilian airspace using GPS navigation instead of the traditional Tactical Air Navigation (TACAN) system. Ground/Air Task Oriented Radar (G/ATOR) G/ATOR is a three-dimensional short- to medium-range tactical radar designed to detect, identify, and track low-level cruise missiles, manned aircraft, and unmanned aerial vehicles as well as rockets, mortars, and artillery fire. The Marine Corps’ G/ATOR program conducted three developmental test periods beginning in July 2012 and continuing until April 2013. An operational assessment was to be conducted in April 2013, but because reliability problems primarily related to software deficiencies were identified during the preceding developmental test periods, the operational assessment was postponed and a Field Users Evaluation was conducted instead.
G/ATOR reliability-related software deficiencies have continued and have kept the radar from meeting its Mean Time Between Operational Mission Failure (MTBOMF) requirements. After allowing additional time for the software to further mature prior to the program’s Milestone C decision (scheduled for 1QFY14), the program added a fourth developmental test period to assess improvement. While laudable, the program’s reliability growth plan has not been fully defined; it remains unclear if G/ATOR will meet key reliability metrics by the start of IOT&E (scheduled for 3QFY17). Small Tactical Unmanned Aerial System (STUAS) Tier II The STUAS consists of five RQ-21A unmanned air vehicles, surface components, and assorted government-provided equipment; it is intended to provide units ashore with a dedicated persistent battlefield intelligence, surveillance, and reconnaissance capability. During integrated testing, developmental testers identified an issue with the STUAS sensor payload. Frequently during flight, the imagery provided by the payload would freeze, flicker, and drift, or the operators would lose payload control. The remedial action was to conduct a “soft” reset similar to rebooting a computer. If the soft reset (or multiple soft resets) did not restore payload functionality,
the operator would conduct a “hard reset,” which consisted of powering off and then powering on the payload. Developmental testers did not see the 1 to 4 minutes required to restore functionality as a detriment to system effectiveness. During the operational assessment in support of Milestone C, the frequency of payload resets, along with the time required to restore functionality, caused operators to lose track of targets or interrupted ongoing missions; this caused operational testers to conclude that the payload reset issue had the potential to render the system not effective during IOT&E. Detailed analyses identified issues with the payload to air vehicle interface (electrical and software). After Milestone C, the Program Office inserted an additional integrated test period before IOT&E and implemented modifications to the air vehicle, which contributed to a three‑month delay in the IOT&E. The last integrated test period demonstrated that the payload reset problem has been corrected and that changes to the recovery procedures have resulted in less damage on recovery. As a result, these two are not expected to be issues for the IOT&E. Vertical Take-Off and Landing Unmanned Aerial Vehicle (VTUAV) (Fire Scout) The Fire Scout is a helicopter-based tactical unmanned aerial system comprised of up to three MQ-8 air vehicles with payloads, a shipboard integrated Ground Control Station with associated Tactical Common Data Link, and the UAV Common Automatic Recovery System. In 2009, the Navy produced a draft VTUAV Developmental Test to Operational Test Transition Report, which assessed the system’s readiness to enter IOT&E using the MQ-8B air vehicle. The draft report stated: “The VTUAV system is not recommended to proceed to IOT&E based on the high risk of an OPEVAL [operational evaluation] determination of not operationally suitable.” Because of this draft recommendation, VTUAV did not enter IOT&E as scheduled in early 2010. Since that time, the Navy decided not to proceed with full-rate production of the MQ-8B, and will delay the VTUAV IOT&E until the MQ-8C replaces the MQ-8B at some future date.
Problem Discovery Affecting OT&E
23
DOT&E Activity and oversight CASE 4: PROBLEMS DISCOVERED DURING EARLY TESTING, THAT IF NOT CORRECTED, COULD ADVERSELY AFFECT MY ASSESSMENT OF OPERATIONAL EFFECTIVENESS, SUITABILITY, AND SURVIVABILITY DURING INITIAL OPERATIONAL TEST AND EVALUATION I include this section of the report to identify early in a program’s development problems that need to be corrected to improve the potential for a successful IOT&E. The list includes programs that conducted either early developmental testing or an operational assessment that was conducted prior to Milestone C. The latter have proven to be essential for identifying problems early and clearly continue to reveal their value to the acquisition process. Most of these entries identify problem discoveries in early testing that need to be corrected soon, as their IOT&E or FOT&E periods are approaching within the next two or three years.
DISCOVERIES IN EARLY TESTING in FY13 THAT SHOULD BE CORRECTED PRIOR TO IOT&E CVN-78 Gerald R Ford Class Nuclear Aircraft Carrier
LHA-6 Amphibious Assault Ship
Defense Enterprise Accounting and Management System (DEAMS)
(Includes Seaframes and Mine-Countermeasures Mission Package with the Remote Minehunting System (RMS) and Airborne Mine Neutralization System (AMNS))
DoD Automated Biometric Identification System (ABIS)
M109 Family of Vehicles (FoV) Paladin Integrated Management (PIM)
Handheld, Manpack, and Small Form Fit (HMS) Manpack Radio
Next Generation Diagnostic System (NGDS)
Handheld, Manpack, and Small Form Fit (HMS) Rifleman Radio and Nett Warrior
Public Key Infrastructure (PKI) Increment 2
Integrated Defensive Electronic Countermeasures (IDECM)
Q-53 Counterfire Target Acquisition Radar System
Integrated Electronic Health Record (iEHR)
RQ-4B Global Hawk High-Altitude Long-Endurance Unmanned Aerial System (UAS)
Joint Warning and Reporting Network (JWARN)
Surface Ship Torpedo Defense (SSTD) System: Torpedo Warning System (TWS) and Countermeasure Anti-torpedo Torpedo (CAT)
CVN-78 Gerald R Ford Class Nuclear Aircraft Carrier The CVN-78 Gerald R. Ford class of aircraft carriers is the first new aircraft carrier design in more than 30 years and will replace the CVN-68 Nimitz class. Compared to the Nimitz class, CVN‑78 has design features intended to enhance its ability to launch, recover, and service aircraft, such as a slightly larger flight deck, dedicated weapons handling areas, and increased aircraft refueling stations. In FY13, the Navy completed an operational assessment for CVN-78 that examined design documentation and data from developmental testing. The CVN-78 test schedule is aggressive, leaving little time to fix problems discovered in developmental testing before IOT&E begins. Based on past comments that CVN-78 had inadequate developmental testing, the Program Office has been working to
24
Problem Discovery Affecting OT&E
Littoral Combat Ship (LCS)
incorporate additional developmental test events into the test program. Nonetheless, major developmental test events are still scheduled to occur after IOT&E begins. DOT&E concludes this aggressive schedule increases the likelihood that problems will be discovered during CVN-78’s IOT&E, which could inhibit the successful completion of testing. There are concerns with the reliability of key systems that support sortie generation on CVN-78. These systems include the new catapults, arresting gear, dual-band radar, and weapons elevators. These systems are critical to CVN-78 operations and will be tested for the first time in their shipboard configurations after they have been installed in CVN-78. To date, the Navy has conducted limited reliability testing of these systems. They
DOT&E Activity and oversight have either poor or unknown reliability. Poor reliability of these critical systems could cause a cascading series of delays during flight operations that would affect CVN-78’s ability to generate sorties, make the ship more vulnerable to attack, or create limitations during routine operations. DOT&E assesses the poor or unknown reliability of these critical subsystems will be the most significant risk to CVN-78’s successful completion of IOT&E. If reliability problems with these systems drive CVN‑78’s sortie generation rate well below Nimitz performance, the result could be significant to strategic planners. Due to known problems with aircraft carrier combat systems, there is a high risk that CVN-78 will not achieve its self-defense requirements. Although the CVN-78 design incorporates several combat system improvements over the Nimitz class, these improvements are unlikely to address all of the known shortfalls CVN-78 cannot support multiple Common Data Links (CDLs) and this fact limits the carrier’s ability to communicate with current and future systems, including MH-60 helicopters, P-3 and P-8 aircraft, unmanned aerial vehicles, and other assets. DOT&E concludes the lack of CDL coverage on CVN-78 will limit its operational effectiveness and pose a risk to successful completion of IOT&E. Two common problems with the first ship of a new class is that training and documentation for new systems are provided too late to train the crew before the start of IOT&E; current CVN-78 plans indicate that these problems will affect CVN-78’s IOT&E as well. The CVN-78 Master Integrated Schedule for Logistics shows the production status of required technical documentation. Based on that schedule, Integrated Logistics Support documentation for training, operation, and maintenance of many unique CVN-78 systems are likely to be delivered late. Defense Enterprise Accounting and Management System (DEAMS) DEAMS replaces legacy systems using an enterprise architecture with commercial off-the-shelf (COTS)-based financial accounting software (such as general ledger, accounts payable, accounts receivable, financial reporting, and billing). The Air Force began a second operational assessment (OA-2) of DEAMS Release 2.2 in August 2013. The intent of OA-2, to be completed in February 2014, is to determine if the issues discovered during a previous operational assessment (OA-1) in 2012 were remedied, and that processes and procedures have been put in place to allow for continued operational use. The DOT&E assessment from OA-1 cast doubts on the ability of the system to support financial management for the Air Force. In contrast, the current system has the potential to be both operationally effective and suitable. The problems below, some of which were mentioned in Case 1 above, have the potential to affect a future determination of effectiveness and suitability if not addressed. • Feedback from new users at McConnell AFB, where DEAMS was deployed in October 2012, indicated that the training they had received was inadequate. They noted that it focused
on navigating DEAMS but did not provide them with a real understanding of the system and its application to their day-to-day work process. McConnell users also stated that they need more on-site technical support during DEAMS implementation. • Effective workarounds for existing software defects have been well documented at the Defense Finance and Accounting Service in Limestone, Maine, but workarounds have not been documented within the Air Force. • Although configuration management has improved, a large number of defects remain open and several currently required capabilities and enhancements are still being developed and are not planned for implementation until 2014. • The percent of subsidiary accounts reconciled to general ledger accounts does not meet the 95 percent threshold requirement. This could significantly affect the ability of DEAMS to attain an unqualified financial audit by FY17 as required. DoD Automated Biometric Identification System (ABIS) The DoD ABIS is the result of a Joint Urgent Operational Need request and consists of information technology components and biometric examiner experts that receive, process, and store biometrics from collection assets across the globe, match new biometrics against previously stored assets, and update stored records with new biometrics and contextual data to positively identify and verify actual or potential adversaries. While operational as ABIS 1.0, the system has not had any formal operational testing in its over 10-year existence, and the follow‑on release, ABIS 1.2 has failed to demonstrate adequate maturity during four unsuccessful demonstrations since 2010. Several ABIS 1.2 deficiencies have been identified during early testing including lack of approved requirements, lack of a baseline system against which to make comparisons, lack of configuration management plans and processes to support tracking of fixes and new requests, and lack of a standards conformance program to enable interoperability certification. Unless all of the above prerequisites to a successful IOT&E are addressed, DoD ABIS 1.2 will likely be found not operationally effective nor operationally suitable in the IOT&E scheduled for 3QFY14. Handheld, Manpack, and Small Form Fit (HMS) Manpack Radio The HMS program evolved from the Joint Tactical Radio System program and provides software-programmable digital radios to support tactical communications requirements. The Manpack radio is a two-channel radio with military GPS. In September 2012, the Army conducted a Government Development Test (GDT) 3 to demonstrate improvements in deficiencies identified in the 2012 MOT&E. During GDT 3, the Manpack radio demonstrated improved waveform performance but poor reliability. If reliability is not improved, it could adversely affect the performance during the next operational test.
Problem Discovery Affecting OT&E
25
DOT&E Activity and oversight Additionally, a number of key Manpack required capabilities, such as the ability to pass data and voice between different radio networks, have not yet been fully tested. The Army plans to test these requirements during GDT 4 in January 2014. Conducting operational testing without proving these capabilities in a developmental test will increase the likelihood of Manpack demonstrating poor performance during operational testing. Handheld, Manpack, and Small Form Fit (HMS) Rifleman Radio and Nett Warrior Nett Warrior is an integrated, dismounted Soldier situational awareness system for use by leaders during combat operations. The Rifleman Radio, AN/PRC-154A, is a component of the Nett Warrior system. Nett Warrior is designed to facilitate command, control, and sharing of battlefield information and to integrate each leader into the digitized battlefield. The Army intends to use Nett Warrior to provide mission command and position location information down to the team leader level. In the Nett Warrior Limited User Test during Network Integration Evaluation 13.2, the AN/PRC-154A classified radio did not support the mission of the test unit.
The radio provided inconsistent digital communications, and the majority of the unit leaders indicated that voice quality was degraded beyond 500 meters. The radio experienced delays in re-joining the network, and experienced problems with battery over-heating and rapid battery depletion. If the problems with the radio are not fixed, the effectiveness of the Nett Warrior to provide situational awareness will be severely limited, and future operational effectiveness and operational suitability assessments of the radio will be adversely affected. Integrated Defensive Electronic Countermeasures (IDECM) The IDECM system is a radio frequency, self-protection electronic countermeasure suite on F/A-18 aircraft. The system is comprised of onboard and off-board components. The onboard components receive and process radar signals and can employ onboard and/or off-board jamming components in response to an identified threat. IDECM Block 4 program completed an operational assessment in FY13. The operational assessment was originally planned to consist of flight testing and three laboratory tests with hardware-in-the-loop. One of those laboratory tests was postponed because the system was insufficiently mature, and a second was rescheduled because of a higher priority program. Partially because the system was immature at the time of the test, and partially by design, very little developmental flight testing had occurred prior to the operational assessment.
As a result of poor record-keeping, some aspects of suitability could not be assessed for the analysis of the operational assessment; however, sufficient information was available to determine that reliability was extremely low. The primary contributors to these failures were system instability and resets. While the Navy in general was aware of the problems – its system anomaly database had over 100 open anomalies at the
26
Problem Discovery Affecting OT&E
time of the operational assessment – the Service had focused on tracking each mode of failure rather than their frequency. If reliability does not significantly improve prior to accomplishing FOT&E, it is likely the system will be assessed as both not effective and not suitable because IDECM’s poor reliability will preclude effective use in combat. In addition to these documented shortfalls, the Navy must collect complete and comprehensive suitability data to enable the assessment of availability, maintainability, and built-in test. The Navy needs to improve interoperability between IDECM Block 4 and the radar warning receiver and fire control radar. Since the operational assessment, the prime contractor has released several updates to the system software and further laboratory and flight testing have been accomplished in preparation for the FOT&E, currently scheduled for early CY14. It is not yet clear whether these efforts have been sufficient to address all the shortfalls noted above. Integrated Electronic Health Record (iEHR) The DoD and Veterans Affairs (VA) will use the iEHR program to implement an electronic health record that both organizations can use to meet the healthcare needs of their beneficiaries and the clinicians providing healthcare. Increment 1 will provide a Single Sign-on (SSO) capability for multiple applications via the users’ Common Access Card, and a Context Management (CM) capability to allow fast user switching between applications while keeping the patient and associated clinical information in context. The Interagency Program Office designed and developed SSO‑CM using the capabilities of COTS products. The U.S. Army Medical Department Board planned to conduct an SSO-CM operational assessment in November 2012, but testing was delayed due to system defects and site configuration problems. • Four developmental test events identified a total of 32 defects: 14 in the initial test, 7 in the first System Integration Test (SIT‑1), 7 in SIT-2, and 4 in SIT-3. At the end of SIT-3, 13 defects remained open. At the completion of SIT-3, the program manager further delayed the operational assessment. • DOT&E rejected the operational assessment plan because it did not demonstrate that the SSO-CM systems would work with, and not interfere with, the Interagency Program Office’s primary deliverables, which are the DoD and VA iEHR accelerators. • The Program Executive Officer for the DoD Healthcare Management Systems should work with DOT&E to develop an adequate plan for an operational assessment of the SSO-CM functionality and the impact on Health Data Sharing and Interoperability. Joint Warning and Reporting Network (JWARN) JWARN is a chemical, biological, radiological, and nuclear (CBRN) warning and reporting software application intended to provide men and women in combat with an integrated analysis and response capability to minimize the effects of hostile CBRN
DOT&E Activity and oversight attacks. The Army Test and Evaluation Command conducted the JWARN Increment 1 Modernization operational assessment in a laboratory setting at the Central Technical Support Facility (CTSF) at Fort Hood, Texas, from July 25–31, 2013. During the operational assessment, the immaturity of Army Command Web and network instability diminished the capability of JWARN web application operators to provide timely warnings to units at risk. Since there is no other developmental test venue for the Army network other than the CTSF, these problems could not be predicted or knowable by the program manager prior to the operational assessment. The Army should schedule a developmental test event in the CTSF with a goal of achieving a stable network prior to operational testing.
•
Configuration problems with the command and control infrastructure virtual machine software, which supports lower‑level tactical messaging, prevented Variable Message Format warning messages from being exchanged between battalions using JWARN and company units using Joint Battle Command – Platform (JBC-P) in both unicast and multicast modes. This limitation precluded an end-to-end evaluation of battalion-to-company or company-to-battalion hazard warning using JWARN. LHA-6 Amphibious Assault Ship LHA-6 is a large-deck amphibious ship designed to support a notional mix of fixed- and rotary-wing aircraft. Completed testing of the Ship Self-Defense System (SSDS) Mk 2-based combat system on the CVN-68 class carrier indicates that it is not likely that LHA-6’s nearly equivalent SSDS Mk 2-based combat system will meet the ship’s Probability of Raid Annihilation requirement against all classes of anti‑ship cruise missiles (ASCMs). Additionally, LFT&E analysis completed to date identified potential problems in susceptibility and vulnerability that would likely result in the LHA-6 being unable to maintain or recover mission capability following a hit by some threat weapons. Littoral Combat Ship (LCS)
(Includes Seaframes and Mine-Countermeasures Mission Package with the Remote Minehunting System (RMS) and Airborne Mine Neutralization System (AMNS))
The LCS is the Navy’s newly-designed surface ship intended to accommodate a variety of individual warfare systems (mission modules) assembled and integrated into interchangeable mission packages. Testing conducted in FY13 and analysis of data from FY12 testing continued to identify deficiencies in the LCS seaframes and essential mission systems: • Analysis of equipment casualty reports filed by LCS 1, LCS 2, and LCS 3 showed that the reliability of both seaframe variants has been degraded by frequent critical system failures during early operations and testing. Failures of the LCS 1 seaframe’s diesel-powered generators, air compressors, and propulsion drive train components have degraded the seaframe’s reliability during developmental testing and early
•
•
•
operations. The operational reliability of the LCS 2 variant’s seaframe has been degraded by equipment failures, including problems with operator consoles, power generation equipment, components of the Total Ship Computing Environment and the ship’s internal networks, propulsion drive train components, communications systems, and mission package support systems. The Remote Multi-Mission Vehicle (RMMV), which is a component of the Mine Countermeasures (MCM) mission package, has a history of poor reliability that if not corrected would affect the assessment of LCS’s operational suitability in conducting MCM operations. Following a second phase of vehicle improvements and reliability growth testing, the Navy reported that RMMV reliability was meeting Navy requirements. However, DOT&E’s review showed that the Navy’s assessment excluded some critical failures and was based on failure definitions and scoring criteria that were inconsistent with those used during the program’s Nunn‑McCurdy review; the estimates also do not reflect the expected reliability in more operationally realistic mission scenarios where vehicle usage is more stressed. An upcoming shore-based operational assessment will provide another opportunity to evaluate the system’s reliability. The MCM mission package performance during developmental testing has been degraded by immature mission systems, low sensor detection performance in some operational conditions, high false alarm rates, unproven tactics, and low operator proficiency. The Navy completed developmental testing to assess Multi‑Vehicle Communications System (MVCS) upgrades and improvements to the launch, handling, and recovery systems for the RMMV. Following testing, the Navy reported that additional efforts are required to retire risks associated with RMMV launch and recovery. Sailors also reported that communications between an RMMV equipped with MVCS upgrades and LCS 2 were unreliable throughout the test. DOT&E completed analysis of data from an FY12 shore‑based operational assessment of the MH-60S helicopter and Airborne Laser Mine Detection System (ALMDS) and found that ALMDS detection depth does not meet the Navy’s requirement. This deficiency will make it necessary to extend the detection envelope of the AN/AQS-20A Sonar Mine Detecting Set to restore the desired overlap with the demonstrated ALMDS envelope. The Navy conducted additional developmental testing of the AN/AQS-20A using a surface craft to tow the sensor and expert operators to evaluate the AN/AQS-20A capability to detect and classify near-surface mines during post-mission analysis. While this has the potential to ameliorate the deficiency, the Navy has not yet completed an operational test of this capability with the RMMV, controlled by fleet operators, towing the sensor and fleet personnel performing the post‑mission analysis of the sonar data. Additional testing will be required in other environments as well to fully characterize the capability.
Problem Discovery Affecting OT&E
27
DOT&E Activity and oversight • The Navy completed developmental testing to evaluate the performance of the Airborne Mine Neutralization System (AMNS) when it is operated in high current and reported problems with compass corrections and fiber-optic communications losses. These failures have the potential of making AMNS not effective since even minor currents are expected in many operational environments. Additional testing is needed to determine the maximum current in which the system is still operable, and determine the operational impact of the performance deficiency. • The Navy’s Quick Reaction Assessment uncovered classified deficiencies in LCS 1’s capability to protect the security of information. M109 Family of Vehicles (FoV) Paladin Integrated Management (PIM) The PIM program is a sustainability and survivability upgrade of the currently fielded Paladin M109A6 self-propelled howitzer and companion M992A2 resupply vehicle. The Army conducted the PIM Limited User Test (LUT) in November 2012 to support the program’s Milestone C decision. The PIM LUT Pilot Test and collective live firing events revealed issues with the M82 primer when firing M232A1 Modular Artillery Charge System (MACS) Charge 5 propellant. The M82 primer deforms and jams in the cannon firing mechanism due to higher breech pressures when firing MACS Charge 5 propellant. This problem had been observed in developmental testing, but the scope of the problem and operational implications were not widely understood until the LUT Pilot Test. There were no plans to address the issue. Problems encountered during training and the pilot test prompted replacement of MACS 5 with another propellant during the LUT. The Program Executive Officer, Ground Combat Systems and Program Executive Officer, Ammunition established a special research team to identify solution options involving modification of the propellant, redesign of the breech and firing mechanisms, development of alternative ignition systems, and/or restriction of the use of MACS propellant to no more than four increments. If the issue is not resolved before the FY16 IOT&E, it is unlikely the test unit will be responsive when firing missions with MACS 5 propellant. Next Generation Diagnostic System (NGDS) The NGDS is a U.S. Food and Drug Administration (FDA)‑cleared reusable, portable biological pathogen diagnostic and identification system capable of rapidly analyzing clinical and environmental samples. The U.S. Army Medical Department Board conducted an early operational assessment of three candidate NGDS systems in 3QFY13. The three candidates were commercial off-the-shelf medical diagnostic devices. One of the vendor systems encountered major reliability problems during testing, resulting in systems having to be
28
Problem Discovery Affecting OT&E
replaced. Other vendor systems experienced minor hardware problems, such as loose wiring connections, that could also affect suitability. One vendor system used complex operating procedures that at times proved difficult for operators to follow correctly and often resulted in invalid results. Ensuring protocols are clear and operators are appropriately trained to operate the system will be key as the program moves to MOT&E. Public Key Infrastructure (PKI) Increment 2 PKI Increment 2 provides authenticated identity management via password-protected SIPRNet tokens to enable DoD members and others to access the SIPRNet securely and to encrypt and digitally sign e-mail. The program continues to add capability through spiral development, and these spirals will undergo testing in the future. Limited and poorly designed developmental testing was directly attributable to the problems observed in previous operational testing. While the Program Management Office has made some initial attempts to correct the configuration management issues, adequate Configuration Control Board structure and overall repeatable processes for defect identification and resolution still do not exist. Unless the program can fix the configuration management processes for prioritizing needed capabilities and improve configuration control processes for ensuring deployments can be sustained without impacting availability and reliability, DOT&E may once again assess the PKI as not operationally effective and suitable for current and future Spirals. Q-53 Counterfire Target Acquisition Radar System The Q-53 radar is designed to detect, classify, and locate projectiles fired from mortar, artillery, and rocket systems using a 90-degree or continuous 360-degree sector search. Early developmental testing indicates the Q-53’s probability of detection and location accuracy against volley-fired weapons is worse than the performance demonstrated against single-fired weapons. Volley-fire is the technique of firing multiple weapons from the same location at a single target. Although the Army has not identified a volley-fire requirement for the Q-53 radar, volley-fire is a standard threat technique and will be used as a threat tactic in the FY14 Q-53 IOT&E. Developmental testing was conducted under conditions that do not match all expected threat employment profiles; therefore, IOT&E results have the potential of being different than observed in developmental testing. If corrections are not made and the IOT&E results reveal the same performance deficiencies observed in developmental testing, then DOT&E’s assessment of operational effectiveness could be affected. RQ-4B Global Hawk High-Altitude Long-Endurance Unmanned Aerial System (UAS) The RQ-4 Global Hawk is a remotely-piloted, high-altitude, long-endurance airborne Intelligence, Surveillance, and Reconnaissance system that includes the Global Hawk unmanned
DOT&E Activity and oversight air vehicle, various intelligence and communications relay mission payloads, and supporting command and control ground stations. In March 2013, the Air Force conducted an Operational Utility Evaluation (OUE) of the RQ-4B Global Hawk UAS. The OUE discovered previously unidentified shortfalls in synthetic aperture radar stationary target imagery capabilities. These capabilities do not currently meet established operational requirement thresholds for image resolution. Multi-Platform Radar Technology Insertion Program (MP-RTIP) operator displays and control interfaces are also immature, which significantly increases operator workload during target‑intense operations. During OUE missions, frequent MP-RTIP sensor faults required sensor operators to halt intelligence collection operations to reset or restart the system. Resulting sensor downtime reduced on‑station intelligence collection time by 23 percent. Additionally, contactor maintenance and supply support was required to compensate for immature system-level reliability, maintenance training, documentation, and logistics support systems.
Surface Ship Torpedo Defense (SSTD) System: Torpedo Warning System (TWS) and Countermeasure Anti-torpedo Torpedo (CAT) The SSTD is a system-of-systems that includes two new sub-programs: the TWS program (an Acquisition Category III program) and CAT (not an acquisition program until FY16). TWS is being built as an early warning system to alert on and localize incoming threat torpedoes. While TWS was designed to employ both active and passive sonar to detect incoming threat torpedoes, hardware reliability failures forced the Navy to delay development of the active component. During early testing from March through August 2013, using the purely passive detection approach, the Navy observed that TWS was subject to false alarms and poor detection performance. The Navy temporarily addressed this problem by assigning a civilian contractor acoustics specialist to monitor and report indications of threat detections using displays not normally available to the ship’s crew. Contractors provided this service during the November 2013 Quick Reaction Assessment aboard USS George H. W. Bush (CVN-77), and are expected to deploy with the ship in FY14.
Problem Discovery Affecting OT&E
29
DOT&E Activity and oversight PROGRESS UPDATES ON DISCOVERIES REPORTED IN THE FY12 DOT&E ANNUAL REPORT FY12 Discoveries in IOT&E that should have been Resolved prior to Operational Test In FY12, I identified 17 systems that had significant issues in IOT&E that should have been discovered and resolved prior to commencement of operational testing. Two of the 17 programs were cancelled: Mine Resistant Ambush Protected (MRAP) Dash Ambulance and MRAP Caiman Multi-Terrain Vehicle (CMTV). For the ALR-69A Radar Warning Receiver, the Program Office has implemented a fix for the program, but operational testing will not be completed until a future aircraft program integrates the system. The Standard Missile-6 (SM-6) Program Office is studying potential fixes. The following updates the status of the remaining 13 systems. Fixes Implemented and Demonstrated in FOT&E • Key Management Infrastructure (KMI) Increment 2 • Mission Planning System (MPS)/Joint Mission Planning System – Air Force (JMPS-AF) Fixes Implemented but New Issues Discovered • Distributed Common Ground System – Army (DCGS-A) Fixes Implemented; Currently in OT or Planning Additional OT • AGM-88E Advanced Anti-Radiation Guided Missile (AARGM) Program • Battle Control System – Fixed (BCS-F) • E-2D Advanced Hawkeye • E-3 Airborne Warning and Control System (AWACS) Block 40/45 Upgrade • Handheld, Manpack, and Small Form Fit (HMS) Manpack Radio • Handheld, Manpack, and Small Form Fit (HMS) Rifleman Radio • Miniature Air Launched Decoy (MALD) and MALD – Jammer (MALD-J) • MV-22 • SSN 774 Virginia Class Submarine • Warfighter Information Network-Tactical (WIN-T) No Fixes Planned • None
30
Problem Discovery Affecting OT&E
FY12 Discoveries in Early Testing that should be Corrected prior to IOT&E In FY12, I identified six systems that had significant issues in early testing that should be corrected before IOT&E. The following provides an update on the progress those systems made in implementing fixes to those problems. Five of the six programs have or are implementing corrective actions that will be tested and assessed in either LFT&E or OT&E. Fixes Implemented and Demonstrated in OT or LFT&E • Bradley Engineering Change Proposal (ECP) Fixes Implemented, but Effect is Unknown; Currently in OT or Planning OT • F-15E Radar Modernization Program (RMP) • Joint Standoff Weapon (JSOW) C-1 • Littoral Combat Ship (LCS) • Patriot Advanced Capability-3 (PAC-3) Some Fixes Implemented; Testing Constrained Pending Future Acquisition Decisions • None No Fixes Planned or Plans not Determined • Multi-Static Active Coherent (MAC) System
DoD Programs
DoD Programs
DOD PROGRAMS
Defense Readiness Reporting System (DRRS) Executive Summary • The Joint Interoperability Test Command (JITC) conducted an operational assessment (OA) of the Defense Readiness Reporting System – Strategic (DRRS-S) from April 1 through May 31, 2013. Additionally, JITC and the Navy Information Operations Command (NIOC) conducted an Information Assurance (IA) assessment from November 19, 2012, through April 12, 2013. Based upon the system deficiencies and lack of functionality demonstrated during the OA, the system is not ready to proceed to IOT&E. • Defense Readiness Reporting System – Army (DRRS-A), DRRS – Marine Corps (DRRS-MC), and DRRS – Navy (DRRS-N) mission readiness and unit status data exchanges with DRRS-S were successful. However, DRRS-N mission readiness data exchanges with DRRS-S were not fully assessed because the Navy Mission Essential Task List data were not following the data route described in the requirements documents and the JITC test instrumentation was configured to capture data from the documented route. The actual data route was not identified until after the test was complete. • Air Force and joint reportable units were able to input, manage, and assess mission readiness data using the DRRS-S system. Air Force users successfully entered unit status data, including the Commander’s and Personnel ratings, within DRRS-S using the Air Force Input Tool (AF-IT). However, additional AF-IT development is required to allow input of the Resource, Supply, and Training ratings. • DRRS-S adequately supports the Joint Force Readiness Review and Quarterly Readiness Report to Congress. • JITC partially assessed the DRRS to Joint Operational Planning and Execution System (JOPES) interface, critical to Global Status of Resources and Training System (GSORTS) retirement, showing that exchanged data were accurate and complete. However, the DRRS-S to Global Combat Support System – Joint (GCSS-J) interface, also critical to GSORTS retirement, was not available during the OA. • DRRS-S met reliability, availability, and maintainability thresholds. The DRRS-S help desk effectively supported both the system under test and production system. Review of help desk logs showed that the system employed effective patch management and that the software was mature. Users were satisfied with system training and documentation. • JITC discovered a number of IA vulnerabilities during IA testing. The DRRS program manager must resolve or mitigate all vulnerabilities prior to IOT&E.
System • DRRS-S is a Secret Internet Protocol Router Network (SIPRNET)-accessible web application designed to replace the GSORTS, a Force Readiness component of GCCS-J. • The Fleet Forces Command hosts DRRS-S on commercial off‑the-shelf (COTS) hardware consisting of a server enclave of application and database servers using Microsoft Windows® operating systems. • DRRS-S receives and processes readiness reports and data from Service-specific increments of the larger DRRS enterprise including DRRS-A, DRRS-MC, and DRRS-N. Combatant Commanders (CCDRs) (and subordinates they direct), DoD agencies, and Air Force units report directly within DRRS-S. Mission • CCDRs, Military Services, Joint Chiefs of Staff, Combat Support Agencies (CSAs), and other key DoD users (e.g., SECDEF and National Guard) use the DRRS collaborative environment to evaluate the readiness and capability of U.S. Armed Forces to carry out assigned and potential tasks. • Reporting organizations input both mission readiness and unit (i.e., GSORTS) readiness data into DRRS-S and use DRRS-S to make mission readiness assessments against standardized missions and tasks. Major Contractor InnovaSystems International, LLC – San Diego, California
DRRS
31
DOD PROGRAMS Activity • JITC conducted an OA on DRRS-S from April 1 through May 31, 2013. JITC and the NIOC conducted an IA assessment from November 19, 2012, through April 12, 2013. • In October 2013, DOT&E submitted an OA report on DRRS-S to the Under Secretary of Defense on Personnel and Readiness. • JITC conducted all testing in accordance with the DOT&E‑approved Test and Evaluation Master Plan and OA plan. Assessment • Service DRRS-A and DRRS-MC mission readiness data exchanges with DRRS-S were accurate, timely, and complete. Air Force and joint reportable units were able to input, manage, and assess mission readiness data using the DRRS-S system. Following the OA, JITC discovered the DRRS-N was exchanging mission readiness data in the legacy format and not in accordance with the data route described in the requirements documents. JITC test instrumentation was configured to capture data from the documented route and as a result, JITC was not able to fully assess Navy mission readiness data exchanges. • Service DRRS-A, DRRS-MC, and DRRS-N unit status data exchanges with DRRS-S were accurate, timely, and complete. Air Force users successfully entered unit status data, including the Commander’s and Personnel ratings, within DRRS-S using the AF-IT. Additional AF-IT development is required to allow input of the Resource, Supply, and Training ratings. During testing, an Air Force policy changed the personnel readiness rating calculation method, creating uncertainty about calculation accuracy among users, invalidating satisfaction ratings. The incomplete AF-IT and calculation method changes prevented a full evaluation. • JITC verified unit registration during developmental testing. However, no new units completed unit registration during the operational test window in either GSORTS or DRRS-S, preventing a full evaluation. • DRRS-S adequately supports the Joint Force Readiness Review and Quarterly Readiness Report to Congress. • DRRS-S business intelligence capabilities allow users to aggregate, filter, and display data to support mission assessments. Features such as data views, watch lists, and groups were effective. However, the user success rate with quick look reports was 61 percent and user satisfaction with the query tool was 46 percent.
32 DRRS
• The DRRS-S historical data warehousing capability, which allows users to query archived readiness data to evaluate trends, was used once during operational testing and could not be characterized. • JITC partially assessed the DRRS to JOPES interface, critical to GSORTS retirement, during testing. Results show that exchanged data were accurate and complete. However, JITC did not evaluate timeliness. The Defense Information Systems Agency is developing capabilities to check and restore synchronization between DRRS and JOPES. Additional testing is required to fully evaluate the DRRS to JOPES interface. • The DRRS-S to GCSS-J interface, also critical to GSORTS retirement, was not available during the test. The interface is currently under development and will be available for test in 1QFY14. • DRRS-S met reliability, availability, and maintainability thresholds. The DRRS-S help desk effectively supported both the system under test and production system during the operational test window. A review of help desk logs showed that the system employed effective patch management and that the software was mature. Users were satisfied with system training and documentation. • JITC discovered a number of IA vulnerabilities during IA testing. The DRRS program manager must resolve or mitigate all vulnerabilities prior to IOT&E. • Based upon the system deficiencies and lack of functionality, the system is not ready to proceed to IOT&E. Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. 1. Once DRRS-S includes all capabilities required for GSORTS retirement, JITC should conduct IOT&E to determine the operational effectiveness, suitability, and survivability. 2. A certified Red Team must conduct penetration and exploitation testing to verify correction of IA findings and evaluate the DRRS-S ability to protect, detect, react, and restore against an operationally relevant cyber-security threat.
DOD PROGRAMS
F-35 Joint Strike Fighter (JSF) Executive Summary • Flight test teams operating the 18 test aircraft assigned to the developmental flight test centers nearly matched or exceeded flight test sortie goals through October 2013. This occurred despite loss of several government employee work days due to furloughs and sequestration, and two fleet-wide grounding instances. Flight sciences testing made the planned progress in envelope expansion and handling qualities for the year; however, mission systems and weapons integration testing made little progress and continued to fall behind test point execution goals driven by upcoming fleet release and Services’ Initial Operational Capability plans. • Mission systems development and test teams focused on getting Block 2B capability into flight test, which began several months later than planned in the integrated master schedule. Block 2B capability is the next major increment planned to be released to the fleet of production aircraft, and the first planned to have combat capability. A considerable amount of testing was necessarily devoted to completing development of prior-block capabilities, attempting to complete fixes to known problems, and regression testing of new versions of software. As a result, through October 2013, little progress was made in completing flight testing required by the baseline Block 2B joint test plan. This creates significant pressure on development and flight test of the remaining increments of Block 2B, with approximately 12 months remaining on the program timeline before final preparations are planned to begin for an operational utility evaluation of the combat effectiveness and suitability of Block 2B. • Weapons integration, which includes both flight sciences and mission systems test events, did not make the planned progress in CY13. Weapons integration is recognized by the program as a critical path to both Block 2B completion and the end of Block 3F development. • Flight operations of production aircraft and upcoming operational testing of Block 2B capability depend on the functionality of the Autonomic Logistics Information System (ALIS), which has been fielded with significant deficiencies. The current ALIS capability forces maintenance operations into numerous workarounds and causes delays in determining aircraft status and conducting maintenance. The program expects improvements in the next ALIS version, scheduled in time for the release of Block 2B capability to the fleet, but there is no margin in the development and test schedule. • F-35B flight test aircraft completed 10 days of testing aboard USS Wasp as planned in August 2013. Testing included evaluating changes to control laws, expanding the operational flight envelope, and flight operations at night. • Overall suitability performance continues to be immature, and relies heavily on contractor support and workarounds unacceptable for combat operations. Aircraft availability
•
•
•
•
and measures of reliability and maintainability are all below program target values for the current stage of development. The program is now at significant risk of failing to mature the Verification Simulation (VSim) and failing to adequately verify and validate that it will faithfully represent the performance of the F-35 in the mission scenarios for which the simulation is to be used in operational testing. The program completed F135 engine vulnerability test series that demonstrated: - The engine can tolerate a range of fuel leak rates ingested through the inlet to simulate and assess ballistically induced fuel tank damage effects. System-level live fire tests using a structural F-35C test article with an operating engine will determine the engine tolerance to the fuel quantity ingested as a result of actual ballistic damage. - The engine is tolerant of mechanical component damage from single-missile fragments, while fluid-filled engine components are vulnerable to fire. Results from two tests demonstrated engine vulnerabilities against more severe threats and were consistent with results from prior legacy engine tests. The program examined the F-35 vulnerability to ballistically induced damage to the F-35 gun ammunition. Missile fragment ballistic testing on single PGU-32 rounds demonstrated that a propellant explosive reaction and sympathetic reaction of adjacent rounds in multiple round tests were unlikely. The F-35 is, however, vulnerable to ballistically-induced propellant fire from all combat threats. The vulnerability of the F-35 to electrical system ballistic damage remains an open question. Based on the F-35A aircraft (AA:0001) in-flight incident in 2007, electrical arcing
F-35 JSF 33
DOD PROGRAMS tests in 2009, and the flight-critical system-level test events in 2012, DOT&E recommended that the program conduct additional analyses to address the likelihood and consequence of arcing from the 270-volt to 28-volt system. The Lockheed Martin electrical power system team is currently working on a response to these concerns. • The program provided no update on the decision to reinstate the Polyalphaolefin (PAO) shut-off valve, a 2-pound vulnerability reduction system that could reduce crew
casualties and the overall F-35 vulnerability by approximately 12 percent, averaged across all threats and F-35 variants. • The program redesigned the On-Board Inert Gas Generation System (OBIGGS) to meet vulnerability reduction and lightning requirements. The program is currently planning the tests for FY14 to ensure that the system is able to maintain fuel tank inerting throughout all mission profiles. The system should protect the F-35 from threat-induced or lightning‑induced fuel tank explosions.
Actual versus Planned Test Metrics through October 2013 Test Flights All Testing
Flight Sciences
All Variants
F-35B Only
F-35A Only
F-35C Only
Mission Systems
993
284
226
181
302
2013 Actual 2013 Planned
985
287
241
171
286
Difference from Planned
+0.8%
-1.0%
-6.2%
+5.8%
+5.6%
Cumulative Actual
3,601
1,269
963
612
757
Cumulative Planned
3,284
1,127
910
584
663
Difference from Planned
+9.7%
+12.6%
+5.8%
+4.8%
+14.2%
Test Points All Testing
Flight Sciences
Mission Systems
All Variants
F-35B Only
F-35A Only
F-35C Only
Block 1*
Block 2A
Block 2B
Block 3
Other
2013 Baseline Accomplished
5,464
1,418
1,713
1,032
326
168
461
0
346
2013 Baseline Planned
7,180
1,701
1,836
1,165
1,755
0
723
Difference from Planned
-23.9%
-16.6%
-6.7%
-11.4%
-45.6%
Added Points
1,776
178
193
211
1,194
-52.1% 0
0
Points from Future Year Plans
720
320
0
400
0
0
0
Total Points Accomplished**
7,960
1,916
1,906
1,643
2,149
0
346
Cumulative SDD Baseline Actual
26,689
9,356
7,636
5,859
0
1,198
Cumulative SDD Baseline Planned
27,075
9,256
7,735
5,564
2,663
0
1,857
Difference from Planned
-1.4%
+1.1%
-1.3%
+5.3%
-0.9%
0.0%
-35.5%
Program Office Estimated Test Points Remaining
31,218
9,726
6,057
7,493
1,739
2,021
1,166
350
614
606
860
3,226
* Includes Block 0.5 and Block 1 quantities ** Total Points Accomplished = 2013 Baseline Accomplished + Added Points SDD = System Development and Demonstration
System • The F-35 Joint Strike Fighter (JSF) program is a tri-Service, multi-national, single seat, single-engine family of strike aircraft consisting of three variants: - F-35A Conventional Take-Off and Landing (CTOL) - F-35B Short Take-Off/Vertical-Landing (STOVL) - F-35C Aircraft Carrier Variant (CV) • It is designed to survive in an advanced threat (year 2012 and beyond) environment using numerous advanced capabilities. It is also designed to have improved lethality in this environment compared to legacy multi-role aircraft. • Using an Active Electronically Scanned Array radar and other sensors, the F-35 is intended to employ precision-guided
34 F-35 JSF
bombs such as the Joint Direct Attack Munition (JDAM) and Joint Standoff Weapon, AIM-120C radar-guided Advanced Medium-Range Air-to-Air Missile, and AIM-9 infrared-guided short-range air-to-air missile. • The program provides mission capability in three increments: - Block 1 (initial training) - Block 2 (advanced training and initial combat) - Block 3 (full combat) • The F-35 is under development by a partnership of countries: the United States, Great Britain, Italy, the Netherlands, Turkey, Canada, Australia, Denmark, and Norway.
DOD PROGRAMS Mission • A force equipped with F-35 units should permit the Combatant Commander to attack targets day or night, in all weather, and in highly-defended areas of joint operations. • F-35 will be used to attack fixed and mobile land targets, enemy surface units at-sea, and air threats, including advanced cruise missiles.
Test Strategy, Planning, and Resourcing • The JSF Program Office, in coordination with the Services and the operational test agencies, submitted Revision 4 of the Test and Evaluation Master Plan (TEMP) for approval in late CY12. - DOT&E approved the TEMP in March 2013, under the condition that the schedule in the TEMP be revised such that no overlap exists between the final preparation period for IOT&E and the certification period required for the Services’ airworthiness authorities to issue flight clearances. - DOT&E required that the final preparation for the IOT&E could not begin any earlier than the Operational Test Readiness Review, a point in time when the JSF Program Executive Officer certifies the system ready for IOT&E. • This report reviews the program by analyzing the progress of testing and the capability delivered as a function of test results. The program plans a specific set of test points (discrete measurements of performance under specific test conditions) for accomplishment in a given calendar year. In this report, test points planned for a given calendar year are referred to as baseline test points. In addition to baseline test points, the program accomplishes test points added for discovery, regression of new software, and regression of fixes to deficiencies identified in flight test. Cumulative System Development and Demonstration (SDD) test point data refer to the total progress towards completing development at the end of SDD. F-35A Flight Sciences Flight Test Activity with AF-1, AF-2, and AF-4 Test Aircraft • F-35A flight sciences testing focused on: -- Accomplishing clean-wing (no external stores or weapons) flutter testing of the full Block 2B flight envelope with weapons bay doors closed and open -- Evaluating flying qualities with internal stores (GBU-31 JDAM, GBU-12 laser-guided Bomb, and AIM-120 Advanced Medium-Range Air-to-Air Missile) and external stores (AIM-9X short-range missile) -- Characterizing the subsonic and supersonic weapons bay door and environment -- High angle-of-attack (above 20 degrees) testing in clean configuration and in landing configuration • F-35A flight testing was affected by two directives to halt testing in early CY13.
Major Contractor Lockheed Martin, Aeronautics Division – Fort Worth, Texas
•
• •
•
-- T he entire F-35 fleet was grounded on February 21, 2013, after a crack was discovered on February 19, 2013, in one of the third-stage, low-pressure turbine blades in the engine of AF-2, a flight sciences test aircraft at Edwards AFB, California. The cause of the crack was determined to be a rupture due to thermal creep, a condition in which deformation of material forms from the accumulated exposure to elevated temperatures at high-stress conditions. The stop order was lifted one week later, on February 28, 2013, with the requirement for additional inspections of the engines to ensure the effects of creep, if they occur, are within tolerances. -- Discovery of excessive wear on the rudder hinge attachments on AF-2 in early March 2013 also affected availability of test aircraft. As a result, the test fleet was grounded for inspections and maintenance actions, including replacing part of the hinge on AF-2 and adding wear-preventing washers to the hinges of the rest of the test fleet. -- In total, AF-2 was down for six weeks for replacement of the engine and rudder hinge repair. The test team completed supersonic clean wing flutter testing with the weapons bay doors open and closed, clearing the F-35A Block 2B envelope to 1.6 Mach/700 knots calibrated airspeed. The team began testing F-35A controllability at high angles of attack and high yaw rates, including the first intentional departures from controlled flight with external stores. The test team completed all weapons safe-separation events of GBU-31, JDAM, and AIM-120 weapons for the Block 2B envelope by the end of August. These tests precede end‑to‑end weapons delivery accuracy test events performed with mission systems test aircraft. The program tested two aircraft modified with new horizontal tail surface coatings and instrumented with temperature sensors to monitor heating from conditions of extended afterburner use. Damage to horizontal tail coatings was previously discovered during flight tests on all three variants involving extended use of the afterburner not expected to be representative of operational use, but which was necessary to achieve certain test points. Non‑instrumented test aircraft continue to operate with restrictions to the flight envelope and use of the afterburner.
F-35 JSF 35
DOD PROGRAMS Flight Sciences Assessment • Through the end of October, the F-35A flight sciences test team lagged in completing the planned flights for the year, having accomplished 226 sorties against the plan of 241. Productivity in baseline test points also lagged by 6.7 percent, as the team accomplished 1,713 baseline points against a plan of 1,836. • The amount of added work from new discoveries or from regression of new versions of air vehicle software (i.e., control laws governing performance and handling qualities) has been less than expected through the end of October. The team allocated 311 points for growth, but accumulated only 193 growth test points by the end of October. • The test team accomplished test points for clearing the flight envelopes for Blocks 2B and 3F. -- Progress through the Block 2B test points was accomplished according to the plan, with 1,089 Block 2B points accomplished compared to 1,083 planned. -- The team also accomplished test points needed to clear the Block 3F flight envelope, but did so at a rate behind the plan. Through the end of October, the team accomplished 624 Block 3F envelope test points against the plan of 753 points, or 83 percent of the plan. The work accomplished for the Block 3F envelope included points with weapons bay doors open and with external air-to-air weapon load-outs. • Weight management of the F-35A variant is important for meeting air vehicle performance requirements. Monthly aircraft weight status reports produced by the program compute a sum of measured weights of components or subassemblies, calculated weights from approved design drawings released for build, and engineering weight estimates of remaining components. -- According to these reports, the weight estimates for the F-35A decreased by 72 pounds from January to October 2013. The latest October 2013 F-35A weight status report showed the estimated weight of 29,030 pounds to be within 341 pounds of the projected maximum weight needed to meet the technical performance required per contract specifications in January 2015. -- Although the weight management of the F-35A has demonstrated a positive trend over the past year, this small margin allows for only 1.16 percent weight growth over the next year to meet contract specification requirements in January 2015. The program will need to continue rigorous weight management beyond the contract specification timeline endpoint in January 2015 and through the end of SDD to avoid performance degradation and operational impacts. • F-35A discoveries included: -- During early high angle-of-attack testing, problems with the air data computer algorithms were discovered, requiring an adjustment to the control laws in the air vehicle software and delaying a portion of the testing until the updated software was delivered to flight test in
36 F-35 JSF
September. High angle-of-attack testing resumed, and is required to support the full flight envelope and weapons employment capabilities planned for Block 2B. -- Buffet and transonic roll-off (TRO) continue to be a concern to achieving operational capability for all variants. The program changed the flight control laws to reduce buffet and TRO in the F-35A. No further changes to the control laws are being considered, as further changes will potentially adversely affect combat maneuverability or unacceptably increase accelerative loading on the aircraft’s structure. The program plans to assess the operational effect of the remaining TRO and the effect of buffet on helmet-mounted display utility by conducting test missions with operational scenarios in late CY13 and early CY14. F-35B Flight Sciences Flight Test Activity with BF-1, BF-2, BF-3, BF-4, and BF-5 Test Aircraft • F-35B flight sciences focused on: -- Continued expansion of the Block 2B flight envelope -- Expansion of the envelope for vertical-lift and short take-off operations, including operations with external stores and the gun pod (mounted on the centerline station) -- Flight clearance requirements for the second set of ship trials on the USS Wasp -- Block 2B weapons separation testing (for GBU-12, GBU-32, and the AIM-120 missile) -- Fuel dump operations with a redesigned dump valve and flap seals -- Initiating high angle-of-attack testing -- Completing tanker air refueling with strategic tankers, i.e., KC-135 and KC-10 aircraft -- Regression testing of new vehicle systems software • The F-35B fleet was grounded after the first British production aircraft, BK-1, experienced a fueldraulic line failure in the STOVL-unique swivel nozzle at Eglin AFB, Florida, on January 16, 2013. The cause was determined to be a poor manufacturing process used for the hoses, leading to crimping dimensions being out of specification; the stop order was lifted nearly four weeks later on February 11, 2013, allowing all F-35B flights to resume. • The program modified one F-35B test aircraft with new coatings on the horizontal tail to address deficiencies seen in bonding of the skin under high-temperature and high- airspeed conditions. These conditions involve extended use of the afterburner not expected to be representative of operational use but which was necessary to achieve certain test points. The new bonded coating failed during flight test and experienced dis-bonding and peeling. The program continues to investigate the effects of afterburner use on the horizontal tails and plans to modify two F-35B test aircraft with new coatings and temperature sensing instrumentation to collect more data. Non-instrumented test aircraft continue to operate with restrictions to the flight envelope and use of the afterburner.
DOD PROGRAMS Flight Sciences Assessment • Through the end of October, the F-35B flight sciences test team accomplished 284 of 287 planned flights, a shortfall of 1 percent. Completion of baseline test points was short by nearly 17 percent, as the team accomplished 1,418 of 1,701 planned baseline points. Similar to the F-35A flight science testing, the amount of added points due to growth was lower than expected, as the team flew only 178 growth points through the end of October, below the 287 points planned. • Completed workup and second set of ship trials (referred to as DT-2) on time. The primary objective of the test period was to collect data for providing a ship-based flight envelope for vertical landings and short take-offs to support Block 2B fleet release and Marine Corps Initial Operational Capability. Flight activity included night operations and inert internal weapons stores.
• Progress through weapons safe-separation testing was behind the planned schedule, as only 12 of the planned 22 separations had been accomplished. • Progress through the work needed to release the Block 2B flight envelope also lagged the plan, with completion of 1,247 of the 1,530 baseline points. Some weapons-related points were blocked earlier in the year when a problem with the GBU-12 lanyard was discovered, requiring a new lanyard and procedures to be developed. The test team was able to accomplish additional points in the Block 3F envelope – similar to the work being done in the F-35A flight sciences – completing 491 points against the plan of 171, pulling forward 320 points from future Block 3F test plans. • The following table, first displayed in the FY11 Annual Report, describes the observed door and propulsion problems by component and identifies the production cut-in, if known.
F-35B Door and Propulsion Problems Category
Design Fix and Test Status
Production Cut-In
Inadequate life on door locks, excessive wear and fatigue due to the buffet environment, inadequate seal design.
New designed doors are being installed on low-rate initial production (LRIP) aircraft as part of the ongoing modification plan; five completed through the end of September. Fatigue testing started in November 2012 and has completed just over 6 percent of the planned two lifetimes of testing as of end of September.
BF-38 LRIP 6 2014
Lift fan drive shaft undergoing a second redesign. Original design was inadequate due to shaft stretch requirements to accommodate thermal growth, tolerances, and maneuver deflections. First redesign failed qualification testing.
New design of the drive shaft will begin qualification testing in December. Full envelope requirements are currently being met on production aircraft with an interim design solution using spacers to lengthen the early production drive shaft.
BF-50 LRIP 8 2016
Lift fan clutch has experienced higher than expected drag heating during conventional (up and away) flight during early testing.
New clutch plate design, with more heat-tolerant material, is complete. Clutch plates are being thinned on LRIP 5 and 6 aircraft, at the expense of reduced life (engagements) to the clutch, to prevent drag heating.
BF-44 LRIP 7 2015
Roll Post Nozzle Actuator
Roll post nozzle bay temperatures exceed current actuator capability; insulation is needed to prevent possible actuator failure during vertical lift operations.
Insulation between the roll post nozzle bay and the actuators is being installed in pre-LRIP 7 aircraft to allow unrestricted operations; however, the actuators must be replaced at 1,000-hour intervals. New actuators will be installed in LRIP 7 aircraft and beyond, removing the requirements for the insulation and extending the service life to 4,000 hours.
BF-44 LRIP 7 2015
Propulsion
Bleed Air Leak Detectors
Nuisance overheat warnings to the pilot are generated because of poor temperature sensitivity of the sensors; overheats are designed to be triggered at 460 degrees F, but have been annunciated as low as 340 degrees F.
More stringent acceptance test procedures are in place, requiring the sensors to be more accurate. Maintenance personnel are checking the detectors on pre-LRIP 5 aircraft, and replacing them in accordance with directives, if necessary.
BF-35 LRIP 5 2014
Propulsion
Aux Air Inlet Door Aft down-lock seal doors (aka 'saloon doors')
Doors are spring-loaded to the closed position and designed as overlapping doors with a 0.5- inch gap. The gap induces air flow disturbance and make the doors prone to damage and out-of-sequence closing. Damage observed on flight test aircraft.
Springs are being limited to 4,000 hours or half the planned lifetime. Program continues to investigate whether a new design to the doors is required.
Structure
Component
Auxiliary Air Inlet Door (AAID)
Propulsion
Drive Shaft
Propulsion
Clutch
Propulsion
Problem
• Weight management of the F-35B aircraft is critical to meeting the Key Performance Parameters (KPPs) in the Operational Requirements Document (ORD), including the vertical lift bring-back requirement. This KPP requires the F-35B to be able to fly an operationally representative profile
TBD
and recover to the ship with the necessary fuel and balance of unexpended weapons (two 1,000-pound bombs and two AIM-120 missiles) to safely conduct a vertical landing. -- Weight reports for the F-35B have varied little in 2013, increasing 36 pounds from either changes in the
F-35 JSF 37
DOD PROGRAMS manufacturing processes or more fidelity in the weight estimates from January through October 2013. Current estimates are within 202 pounds of the not-to-exceed weight of 32,577 pounds – the target weight of the aircraft in January 2015 to meet specification requirements and ORD mission performance requirements for vertical lift bring back. The small difference between the current weight estimate and the not-to-exceed weight allows for weight growth of 0.62 percent over the next year to meet technical specifications in January 2015. -- Managing weight growth with such small margins will continue to be a significant program challenge. Since the program will conduct the technical performance measurement of the aircraft in January 2015, well before the completion of SDD, continued weight growth through the balance of SDD will affect the ability of the F-35B to meet the STOVL mission performance KPP during IOT&E. • Other F-35B discoveries included: -- Wet runway testing, required to assess braking performance with a new brake control unit in both conventional and slow landing operations, has been delayed due to the inability to create the properly degraded friction conditions on the runways at the Patuxent River Naval Air Station, Maryland. The program plans to complete this testing in early CY14. Fielded F-35B aircraft at Eglin and at Yuma are operating under restricted landing conditions until the wet runway testing is complete. -- Buffet and TRO continue to be a concern to achieving operational capability for all variants. The program made changes to the flight control laws to reduce buffet and TRO in the F-35B in CY13. No further changes to the control laws are being considered, as further changes will potentially adversely affect combat maneuverability or unacceptably increase accelerative loading on the aircraft’s structure. The program plans to assess the operational effect of the remaining TRO and the effect of buffet on helmet-mounted display utility by conducting test missions with operational scenarios in late CY13 and early CY14. F-35C Flight Sciences Flight Test Activity with CF-1, CF-2, and CF-3 Test Aircraft • F-35C flight sciences focused on: -- Block 2B envelope expansion for weapons bay doors open and closed -- Completing electromagnetic environmental effects testing to support shipboard operations -- Surveying handling qualities in the transonic flight regimes -- Regression testing of new air vehicle systems software -- High angle-of-attack testing, which began in August -- C arrier suitability testing in preparation for the first set of ship trials scheduled for mid-CY14. The program configured aircraft CF-3 with a modified and instrumented nose landing gear system to begin initial catapult testing in August 2013. The test team modified CF-3 with the new
38 F-35 JSF
arresting hook system and began on-aircraft testing with rolling engagements in late CY13. • The test team completed three weapon safe-separation events by the end of October. • The program modified one F-35C with new coatings on the horizontal tail, and similar to what was experienced in the F-35B and the F-35A, the coatings bubbled and peeled after experiencing high-temperature and high-airspeed conditions. These conditions involve extended use of the afterburner not expected to be representative of operational use, but which was necessary to achieve certain test points. The program plans to modify all three F-35C flight sciences aircraft with new tail coatings and temperature-sensing instrumentation to collect data to characterize conditions and determine what, if any, material solutions will be required. Non-instrumented test aircraft continue to operate with restrictions to the flight envelope and use of the afterburner. Flight Sciences Assessment • F-35C flight sciences test flights accomplished were ahead of the plan through the end of October, with 181 sorties completed compared to 171 planned. • The test team lagged by 11 percent in completing the planned baseline test points through the end of October, accomplishing 1,032 points against the plan of 1,165 points. Progress through the Block 2B flight envelope lagged by 12 percent, as 947 of 1,080 points were accomplished. The test team was able to accomplish more test points in the Block 3F envelope than planned – completing 485 points, compared to 85 planned, pulling 400 points projected for completion in 2014 back into 2013. • Weight management is important for meeting air vehicle performance requirements. The aircraft weight is computed monthly, and adjusted for known corrections from engineering estimates and production modifications. -- The program added 139 pounds to the F-35C weight status in May 2013 to account for the redesigned arresting hook system. The latest weight status report from October 2013 showed the estimated weight of 34,593 pounds to be within 275 pounds (0.79 percent) of the projected maximum weight needed to meet technical performance requirements in January 2016. -- This margin allows for 0.35 percent weight growth per year. The program will need to continue rigorous weight management through the end of SDD to avoid performance degradation and operational impacts. • F-35C discoveries included: -- Buffet and TRO continue to be a concern to achieving operational combat capability for all variants. Control laws have been changed to reduce buffet and TRO in the F-35A and F-35B with some success; however, both problems persist in regions of the flight envelope, and are most severe in the F-35C. -- Characterization testing of buffet and TRO in the F-35C with the current control laws and without the use of
DOD PROGRAMS leading edge spoilers is ongoing. Unlike the other two variants, the program has the option to conduct flight testing with leading edge spoilers to reduce buffet and the onset of TRO with two of the F-35C flight test aircraft if trade-offs made in control laws are not sufficient to manage the negative impact of these effects. Mission Systems Flight Test Activity with AF-3, AF-6, AF-7, BF-17, BF-18, and CF-8 Test Aircraft and Software Development Progress • Mission systems are developed and fielded in incremental blocks of capability. -- Block 1. The program designated Block 1 for initial training capability and allocated two increments: Block 1A for Lot 2 (12 aircraft) and Block 1B for Lot 3 aircraft (17 aircraft). No combat capability is available in either Block 1 increment. -- Block 2A. The program designated Block 2A for advanced training capability and designated this block for delivery of aircraft in production Lots 4 and 5. No combat capability is available in Block 2A. -- Block 2B. The program designated Block 2B for initial, limited combat capability with internal weapons (AIM‑120C, GBU-32/31, and GBU-12). This block is not associated with the delivery of any production aircraft. Block 2B software, once complete with development and certification, will be retrofitted onto earlier production aircraft. -- Block 3i. The program designated Block 3i for delivery of aircraft in production Lots 6 through 8, as these aircraft will be built with an improved integrated core processor and other upgrades collectively known as “Technology Refresh 2”, or TR2. No new capability beyond Block 2B is introduced in Block 3i. -- Block 3F. The program designated Block 3F as the full SDD capability for production Lot 9 and later. • The Edwards test site accepted the fifth F-35C test aircraft, designated as CF-8, in September 2013; it is a mission systems flight test aircraft. • The six mission systems flight test aircraft assigned to the Edwards AFB test center flew 302 test sorties against a plan of 286 though October, exceeding the plan by 5.6 percent. • However, the test team accomplished only 54 percent of the planned 2013 baseline mission systems test points from test plans for Blocks 1, 2A, and 2B by the end of October (955 baseline test points accomplished, 1,755 planned). The team also accomplished an additional 1,194 test points for regression testing of new revisions of Block 2A and 2B software and other testing the program found necessary to add to the test plans. The team also lagged in completing planned radar signature testing, completing 346 of 723 planned test points, or 48 percent, by the end of October. • The program initiated a Block Review Board process in late 2012 to manage the increments of mission systems software development, monitor maturity of capability, and release to flight test.
Mission Systems Assessment • Despite flying the mission systems test flights planned for CY13, the program did not make the planned progress in developing and testing mission systems capabilities. Software development, integration in the contractor labs, and delivery of mature capability to flight test continued to be behind schedule. Testing of Block 2A training capability (no planned combat capability) was completed in 2013. The first increment of Block 2B software, version 2BS1, was delivered to flight test in February 2013, four months later than indicated in the integrated master schedule. • The program completed testing on the Block 2A software needed for delivery of the Lot 4 and Lot 5 production aircraft. This production version of software, designated 2AS3, was designed to provide enhanced training capabilities to the Integrated Training Center at Eglin AFB, Florida, and to the first operational units – the F-35B unit at Yuma Marine Corps Air Station, Arizona, and the F-35A unit at Nellis AFB, Nevada. -- However, the teams at both test centers (Edwards and Patuxent River) determined the initial version of 2AS3 to be deficient in providing the necessary capabilities for unmonitored flight operations under night and instrument meteorological conditions (IMC). In order to finalize Block 2A capability so that it could eventually be certified in production aircraft for flight at night and in IMC, the program made adjustments to plans for the following increment, Block 2B, to accommodate the need for another, final version of Block 2A software, designated 2AS3.1. The test centers completed testing of Block 2AS3.1 in June; however, the certification to allow F-35A and F-35B production aircraft to fly at night or in IMC had not been released as of the time of this report. -- Additionally, the test teams also noted Block 2A deficiencies in the aircraft sensor operations, particularly the Electro-Optical Targeting System (EOTS), aircraft communications capabilities, pilot electronic interfaces, and the aircraft Caution, Advisory, and Warning System. Although the software was intended to provide more mission systems capability, poor sensor performance and stability, excessive nuisance warnings, and disproportionate pilot workload required for workarounds and system resets made the software of limited utility for training. In any type of operational mission scenario, the performance of the software would be unacceptable. -- The program delivered 10 F-35A aircraft to the U.S. Air Force, 12 F-35B aircraft to the U.S. Marine Corps, and 2 F-35C aircraft to the U.S. Navy from production Lot 4 through the end of October. These aircraft were delivered in the Block 2A configuration, but with less capability than defined by the production contract. Specifically, 22 of 47 (47 percent) of the capabilities defined in the production contract were not complete when the aircraft were delivered. The program began checkout and delivery of F-35A, F-35B, and F-35C aircraft from production Lot 5, and these aircraft were similarly delivered with less
F-35 JSF 39
DOD PROGRAMS than planned capabilities. Fifty percent (27 of 54) of the capabilities required by the contract were not complete when these aircraft were delivered to the Services. • The initial Block 2B software increment began flight testing in February 2013. Though four months later than the 2012 integrated master schedule, this timing was in accordance with the expectations set by the program’s new Block Review Board process, which was initiated in late 2012. As it was the initial Block 2B increment, no new capability was mature enough for verification. In October 2013, a new increment of Block 2B, intended to provide a significant increase in verifiable capability, including many fixes to previously identified deficiencies, began flight testing. Initial results with the new increment of Block 2B software indicate deficiencies still exist in fusion, radar, electronic warfare, navigation, EOTS, Distributed Aperture System (DAS), Helmet‑Mounted Display System (HMDS), and datalink. These deficiencies block the ability of the test team to complete baseline Block 2B test points, including weapons integration. The program’s plan is to gradually increase maturity of the software and reduce these obstacles to test progress over three more increments of software in CY14. The degree to which the maturity of the capability has improved and the test teams can verify performance against planned criteria will determine how long it will take to complete Block 2B development and flight test. • The program began implementing plans for testing Block 3i capability, which will be used to deliver production aircraft in Lots 6 through 8, all of which will have an upgraded core processor and other mission systems processor improvements. The program plans Block 3i to include no new capability beyond Block 2B, as it is intended to only encompass rehosting of Block 2B capability on the new TR2 hardware. -- O ne F-35A mission systems test aircraft was temporarily modified with the TR2 hardware in November 2013 to conduct risk reduction testing of an early version of 3i software. Testing was attempted on an F-35C test aircraft in October, which was temporarily modified with the TR2 hardware, but the software did not load properly and the ground testing could not be conducted. -- One mission systems test aircraft of each variant will be modified in early CY14 to begin the start of flight testing of the 3i software. -- All production aircraft from Lot 6 and beyond will have the TR2 hardware and will only be able to operate mission and vehicle systems software that is compatible with this hardware configuration. • Shortfalls in the test resources required to test mission systems electronic warfare capabilities under operationally realistic conditions were identified by DOT&E in February 2012. The DoD programmed for an Electronic Warfare Infrastructure Improvement Program starting in FY13 to add both closed-loop and open-loop emitter resources for testing on the open-air ranges, to make at least
40 F-35 JSF
one government anechoic chamber capable of providing a representative threat environment for electronic warfare testing, and to upgrade the electronic warfare programming laboratory that will produce threat data files. However, progress has been slower than needed to assure these resources are available in time for Block 3 IOT&E in 2018. JSF IOT&E will not be adequate and will be delayed unless this test capability is available. • Deficiencies in the HMDS added testing at both the Edwards and Patuxent River test sites in late CY12 and in CY13. The program dedicated 42 flights to investigating and addressing deficiencies in the HMDS. Seven aircraft from all three variants flew test missions from October 2012 through May 2013 to investigate jitter in the helmet display, night vision camera acuity, latency in the DAS projection, and light leakage onto the helmet display under low-light conditions. Although some progress has been achieved, results of these tests have been mixed. -- Filters for reducing the effects of jitter have been helpful, but have introduced instability, or “swimming,” of the projected symbology. -- Night vision acuity was assessed as not acceptable with the current night vision camera, but may be improved with a new camera planned for inclusion in the next version of the helmet (referred to as the Gen III helmet) being considered by the program. -- Latency with the DAS projection has improved from earlier versions of software, but has not yet been tested in operationally representative scenarios. -- Light leakage onto the helmet display may be addressed with fine-tuning adjustments of the symbology brightness—a process pilots will have to accomplish as ambient and background levels of light change, adding to their workload. -- Although not an objective of the dedicated testing, alignment and “double vision” problems have also been identified by pilots and were noted in the DOT&E report on the F-35A Ready for Training Operational Utility Evaluation. -- Developmental testing has yet to be accomplished in the full operational flight envelope evaluating mission- related tasks, as the full combat flight envelope is not yet available. Use of the HMDS in the full envelope under operational conditions is needed to verify effectiveness of the HMDS. This might not occur until the Block 2B operational utility evaluation, currently planned for late 2015. • Three factors create a significant challenge for completing developmental testing of Block 2B mission systems as planned before the end of October 2014: completing tests of prior blocks of mission systems capability, managing growth in testing, and constraints on test resources. -- The test centers continue to accomplish a significant amount of test points originally designated for completion in prior blocks of mission systems capability. As of the
DOD PROGRAMS end of October, 34 percent of the baseline mission system test points accomplished in CY13 (326 of 955) were for capabilities in Block 1; 18 percent (168 of 955) were for capabilities in Block 2A, and 48 percent (461 of 955) were for Block 2B capabilities. The program intends to complete or delete the test points planned in these previous blocks by the time Block 2B capability completes development in late CY14. All program plans and schedules for the subsequent blocks of mission systems software (Block 3i and Block 3F) depend on this occurring so that the development laboratories and test venues can be converted and devoted to testing the Block 3 hardware configuration. -- The program continues to have significant growth in mission systems testing. Beyond the testing accomplished in late CY12 and CY13 for the helmet, additional testing has been required for regression testing of seven software loads delivered to flight test in CY13 through October, and for deficiencies in the EOTS, the radar, night flying qualities, and navigation systems. Dedicated testing added for the purpose of identifying problems with the helmet accounted for only 22 percent of the total mission systems growth in CY13 by the end of October; the remaining growth executed by the program exceeded the planning factors for added testing by over 40 percent. The program plans to complete Block 2B flight testing in October 2014; however, there is no margin for additional growth to meet that date. Projections based on the planned growth rate show that Block 2B developmental testing will complete in May 2015, approximately 7 months later than planned. Projections for completing Block 2B flight testing using the historical rate of continued growth (excluding the growth associated with the HMDS) show that Block 2B developmental testing will complete about 13 months later, in November 2015, and delay the associated fleet release to July of 2016. -- Mission systems SDD flight test aircraft available to support Block 2B developmental testing will be reduced in CY14, as the program will need to modify aircraft with the TR2 processors to achieve the Block 3i configuration. Aircraft from production Lot 6, which are scheduled to be delivered in mid-CY14, cannot be operated with Block 2B software; they must have certified Block 3i software. The program plans to modify one mission systems aircraft of each variant to begin flight testing of the first increment of Block 3i software in early CY14. The reduction of mission systems aircraft to support Block 2B developmental testing, created by the need to test software to support the production and delivery of Lot 6 and later aircraft, will add to the challenges of completing Block 2B development on schedule. • Mission systems discoveries included: -- A lthough improving, stability of the mission systems software continues to fall short of objectives. The program tracks mission systems software stability
--
--
--
--
--
by analyzing the number of anomalies observed as a function of flight time. The program objective for time between resets for the integrated core processor and the Communications/ Navigation/Identification Friend or Foe suite is a minimum of 15 hours between reset events. October reports for the latest Block 2B mission systems software increment in flight test show a rate of 11.4 hours between anomalies, based on 79.5 hours of flight test. Subsystems, such as the radar, EOTS, DAS, and the navigation solution often require component resets as well, but these are not tracked in the stability metric. The EOTS fails to meet target recognition ranges, exhibits track instability in portions of its field-of-view, and has large line-of-sight angle and azimuth errors when computing target locations. These deficiencies are being investigated and addressed by the program with software fixes. The program continues to monitor loading of the aircraft core processors in the laboratories as more functionality is added in software increments. Projections of the loads expected on all processors for the Block 3 capabilities estimate that three processors, which support landing systems, weapons employment, multi-aircraft datalinks, and earth spatial modeling, will be tasked between 160 and 170 percent of capacity. The program intends to shift the distribution of processing loads with each incremental build of mission systems software; however, margin is limited and the efficiencies gained by the changes need to be assessed under actual, sensor-stressing, flight conditions. The DAS has displayed a high false alarm rate for missile detections during ownship and formation flare testing. The inability of the DAS to distinguish between flares and threat missiles makes the warning system ineffective and reduces pilot situational awareness. The onboard navigation solution – referred to as the ownship kinematic model – has shown excessive position and velocity errors when not receiving updates from the GPS satellite constellation. These errors prevent accurate targeting solutions for weapons employment in a GPS‑denied environment. The program is addressing these errors in the next iteration of software and further flight testing will be required. The radar mapping function does not provide adequate target location accuracy.
Weapons Integration • Weapons integration involves flight sciences testing, mission systems testing, and ground crew support. Testing includes measuring the environment around the weapon during carriage (internal and external), handling characteristics of the aircraft, safe-separation of the weapon from the aircraft, communications between the aircraft sensors and the weapons, and weapons delivery accuracy events. The program has identified lethality, the product of weapons
F-35 JSF 41
DOD PROGRAMS integration test and evaluation, as the critical path to completing development of Block 2B and Block 3F. The Block 2B weapons are the GBU-12 laser-guided bomb, the GBU-31/32 JDAM, and the AIM-120 air-to-air missile. The Block 3F weapons add Small Diameter Bomb Increment I (SDB-I), AIM-9X air-to-air missile, Joint Standoff Weapon, gun (internal for F-35A and external gun pod for F-35B and F-35C), and the United Kingdom’s Paveway IV bomb. • As of the end of October, weapons integration was near the planned progress scheduled for the year on the F-35A. The test teams had completed 567 of 589 planned environmental test points and all 19 planned weapons separation events. Progress on the other variants, however, was behind the plan. On the F-35B, the team had completed 285 of the 455 planned environmental test points and 12 of the 24 planned separation events. On the F-35C, the team began environmental testing late in the year and had completed 176 of 181 planned test points but only 2 of 10 planned separation events. • Progress in testing of mission systems capability to enable end-to-end weapon delivery events was behind schedule for all Block 2B weapons. Weapons integration has been slowed by discoveries of deficiencies requiring software fixes and additional testing. -- Problems with the lanyard on the laser-guided bomb required a new lanyard and routing procedure -- Inaccuracies in the data transfer of position and velocity from the aircraft to the JDAM, which spatially align the bomb with the target, required a fix in the mission systems software -- Problems involving integration of the AIM-120 medium- range missile have been difficult to replicate in lab and ground testing -- Poor target track quality displayed to the pilot from the radar, or from fusion of the aircraft sensors, prevented targeting solutions for simulated weapons engagements -- Poor performance of the EOTS in image quality, tracking stability, and targeting accuracy required software fixes to allow weapons integration testing of the air-to-ground munitions to proceed -- Erroneous target coordinates were derived from the synthetic aperture radar mapping function • The integrated test team continued to rework weapons integration scheduling in 2013 to account for discoveries of deficiencies and the slower than expected delivery of capability needed to conduct weapons delivery accuracy (WDA) events. The team conducted the first WDA test event with a laser-guided bomb on October 29, followed two days later by the first launch of the AIM-120 air-to‑air missile. The second launch of an AIM-120 missile occurred on November 15. Data analyses of the missile launches was ongoing at the time of this report. The team accomplished the first WDA test event with a JDAM bomb (GBU-32) on December 6; data analysis was ongoing at the time of this report. These early WDA events have included non-operationally relevant workarounds to mission systems
42 F-35 JSF
deficiencies that will not be tolerable in operational testing or combat employment. Completion of all Block 2B weapons testing by the end of October 2014 is dependent on: -- The ability of the test team to accomplish a successful weapons-related test mission at a consistently high rate -- The Block 2B version of mission systems software delivered in October 2013 adequately correcting deficiencies and permitting WDA events to proceed in an operationally relevant manner -- Reliable instrumentation and priority from range support assets -- Maintaining the test aircraft used for weapons testing in the Block 2B configuration while the program manages the requirement to start testing mission systems aircraft in the Block 3i configuration • Current program schedules indicate weapons integration testing to be complete by the end of October 2014 and August 2016 for Blocks 2B and 3F, respectively. To meet the schedule for Block 2B, the test team planned to have completed 8 of 15 total Block 2B WDA events by the beginning of December; however, only 4 have been accomplished. WDA events beyond these first four have been blocked from completion due to lack of adequate mission systems performance in radar, fusion, and EOTS. Corrections to the known deficiencies and fix verification are planned to be delivered in the 2BS4.2 and 2BS5 versions of software, the first of which is scheduled to begin weapons flight testing in March 2014. The result of this blocking of subsequent WDA events is a 4- to 6-month delay in the completion of Block 2B weapons integration, which will likely be done between February and April 2015. Detailed planning of the Block 3F weapons integration schedule to complete in August 2016 is under development. However, given historical performance and reasonable planning factors, it is more likely that the final Block 3F weapons events will not be completed within the current SDD schedule. Static Structural and Durability Testing • Durability testing and analysis on the ground test articles of all three variants continued in 2013; progress is measured in aircraft lifetimes. An aircraft lifetime is defined as 8,000 Equivalent Flight Hours (EFH), which is a composite of time under different test conditions (i.e., maneuver and buffet for durability testing). In accordance with the SDD contract, all three variants will complete two full lifetimes, or 16,000 EFH of durability testing. The completion dates for the second aircraft lifetimes are late 2014 for the F-35B and early 2015 for the F-35A and F-35C. The program made plans in 2013 to add a third lifetime of durability testing on the test articles of all three variants. • The F-35A ground test article, AJ-1, completed the first aircraft lifetime in August 2012, as planned. For most of 2013, AJ-1 underwent detailed inspections and repairs on cracks revealed after the first lifetime of testing, including repairs to the wing forward root rib and to a bulkhead stiffener. The
DOD PROGRAMS
•
•
•
•
•
second lifetime of durability testing is planned to begin in December 2013. F-35B durability testing on BH-1 completed the first lifetime of 8,000 EFH on February 9, 2013, then underwent detailed inspection and repairs prior to starting the second lifetime of testing on July 22. The program completed the first block of 1,000 EFH (9,000 EFH total) on August 19, approximately 1 month ahead of schedule. Further testing was halted in September when cracks were discovered in two of the bulkheads, requiring repair. The F-35C fatigue test article restarted testing on January 9, 2013, after previously completing 4,000 hours of testing and associated inspections. It completed 8,000 EFH of testing, or the first lifetime, on September 28. Testing is behind schedule, as cracks discovered in the floor of the avionics bay in February caused a two-month pause while interim repairs were completed. Cracks discovered in fuselage station 402 and the surrounding structure caused a stop test after 7,620 EFH of testing to complete repairs. These cracks were not predicted by prior analysis. Detailed inspections from the first lifetime were ongoing as of this report. Component durability testing for two lifetimes of the vertical tails was completed for the F-35A and F-35B during 2012. Vertical tail testing started in August 2012 for the F-35C and completed 12,901 EFH as of the end of October 2013. Component testing of the horizontal tail for the F-35A and F-35C began third-lifetime testing, completing 23,000 EFH and 21,000 EFH, respectively, as of the end of August. The redesigned F-35B auxiliary air inlet doors, required for STOVL operations, are undergoing ground tests on the F-35B static loads test article (BG-1). Static load testing was completed late in CY12 and durability testing had completed just over 3,000 cycles (approximately 8 percent) of the planned testing as of the end of August. Modifications of the auxiliary air inlet doors on production aircraft have already begun. Discoveries from durability testing included significant findings in both the F-35A and F-35B ground test articles. - Discoveries this year on the F-35A test article include cracks in the engine thrust mount shear webs (designed to carry some of the fore and aft engine load) on both sides of the aircraft, and a crack in the frame of the web stiffener located at fuselage station 402. The program has redesigned the thrust mounts for production cut-in with Lot 6, and retrofits to be completed on earlier aircraft during depot modification periods. Root cause, corrective action, and modification plans for the frame crack are to be determined. - In the F-35B, the program halted testing in December 2012 after multiple cracks were found in a bulkhead (FS472) flange on the underside of the fuselage during the 7,000‑hour inspection. Root cause analysis, correlation to previous model predictions, and corrective action planning are ongoing. ▪▪ Discoveries during detailed inspections following the first lifetime of testing include cracks on the left and right hand sides of the wing aft spar lower flanges and
cracking in the frame of the jack point stiffener, a portion of the support frame outboard of the main fuselage above the main landing gear designed to support load bearing of the aircraft during jacking operations. Redesign, modification, and retrofit plans for these discoveries have not yet been determined by the program. As of August 5, 2013, two redesigns of the part were being evaluated for potential replacement. ▪▪ During its 8,000-hour detailed inspection period between February and July, cracks were found on both the right and left rear spar lower flanges near bulkhead FS556. This particular spar was already on the list of limited life parts, but not for the location of concern. ▪▪ Also during its 8,000-hour inspections, cracks were found in the lower arch of the FS496 bulkhead, but were below limits which would cause a break in planned testing, which restarted at the end of July. At the 9,000‑hour inspection in September, the cracks had grown, but were not deemed sufficient to stop testing, but required increased inspection intervals. The cracks continued to grow during subsequent testing, until at 9,056 EFH, at the end of September, the bulkhead severed and transferred loads which caused cracking in the adjacent FS518 bulkhead. Analysis and corrective action were ongoing at the time of this report. ▪▪ All of these discoveries will require mitigation plans and may include redesigning parts and additional weight. Also, the repairs to the jack point stiffeners – accomplished after the first lifetime of testing – were not adequate, requiring the program to design a new repair concept. - Discoveries in the F-35C test article include cracks in the floor of the avionics bay and, similar to the F-35B, cracking in the frame of the jack point stiffener. Cracks were also found in the bay floor of the power distribution center; repair, retrofit, and production impacts are to be determined. Modeling and Simulation Verification Simulation (VSim) • VSim is a man-in-the-loop, mission software-in-the-loop simulation developed to meet the operational test agencies’ requirements for the Block 2B operational utility evaluation and Block 3F IOT&E. • The program is now at significant risk of failing to (1) mature the VSim and (2) adequately verify and validate that it will faithfully represent the performance of the F-35 in the mission scenarios for which the simulation is to be used in operational testing. Key concerns are: -- VSim development, and verification and validation activities may not be completed in time to support the Block 2B operational utility evaluation, beginning in late CY15. In particular, long lead items such as threat mission data files are at risk of being delivered too late for integration into VSim in time to support the planned Block 2B operational utility evaluation timeline.
F-35 JSF 43
DOD PROGRAMS Additionally, the current VSim schedule has validation and accreditation documentation production activities scheduled until September 2015, months late to support the initial accreditation report required by the Operational Test Readiness Review for the Block 2B operational utility evaluation, scheduled for May 2015. -- The current VSim validation plan does not provide the detail or rigor needed to be able to anticipate accreditation of VSim for use in mission-level evaluation in operational testing. Shortfalls identified include: lack of detail in validation plans for VSim component models; lack of a clear path from component model validation to F-35 system validation to mission-level validation; absence of planned validation for government-furnished threat and weapons models that require significant additional validation after the modifications made to them during integration into VSim; and lack of a plan for structured regression testing after model modifications have been made. As of November 2013, the JSF Operational Test Team, the JSF Program Office, and Lockheed Martin are in the midst of a series of intensive VSim validation meetings aimed at overcoming these shortfalls. -- V Sim may not adequately replicate the installed system performance (i.e., the performance of all F-35 systems and subsystems as installed in the aircraft) in the mission scenarios for which the simulation is planned to be used in the Block 2B operational utility evaluation. There may not be adequate validation data to support accreditation of the simulation for operational testing. -- N o dedicated testing is planned by the program to validate F-35 installed performance in the VSim. The program currently expects validation data to come from planned developmental mission systems and weapons integration testing. However, developmental testing seeks only to acquire verification of contract specification criteria, and does not span the set of conditions over which mission effectiveness will be assessed using VSim in both developmental and operational testing. This creates a significant gap for the program in being able to validate VSim for both developmental and operational testing. • In addition to the risks cited above, DOT&E has highlighted shortfalls in the test resources needed to gather key elements of data required for validation of the VSim for IOT&E, in particular for electronic warfare performance in the presence of advanced threats. These shortfalls are a function of limitations in the test assets currently available to represent threat systems. DOT&E has made formal recommendations to address the shortfalls and is pursuing solutions to make the assets available in time to prepare for IOT&E in a realistic threat environment. • The JSF Program Office and Lockheed Martin have begun to try to address these concerns. Important recent activities have included technical interchange meetings with threat model developers in the intelligence community to address the modeling of electronic attack capabilities, a series of
44 F-35 JSF
intensive validation planning meetings currently underway to provide detailed validation data requirements, and a summer 2013 VSim risk reduction event using the simulation in an F‑35 Block 2A configuration. Other Models and Corporate Labs Activity • At the beginning of 2013, the Program Office had accredited 7 of the 25 models and simulations currently planned to support verification of the F-35. No additional models and simulations planned to support verification of F-35 requirements were accredited in 2013; so, the total number accredited remains at seven. • As of the end of 2012, the program had planned to accredit six models and simulations intended for use in the requirements verification plan in 2013. Of the 18 remaining models and simulations listed in Program Office documentation as requiring accreditation for use in verification, the program characterizes 12 as on-track for accreditation. The progress of the remaining six is characterized as either off-track with mitigation efforts in place or as on-track but with significant execution risk. Training System • In late 2012, the program completed a Ready For Training Operational Utility Evaluation (OUE) to support the Air Force’s Air Education and Training Command’s decision to begin student training at Eglin AFB, Florida. The OUE evaluated the capability of both the F-35A air vehicle and the training system to train an experienced initial cadre of pilots in the equivalent of the familiarization phase of a fighter aircraft transition syllabus. It also evaluated the ability of the F-35A maintenance and Autonomic Logistics Information System (ALIS) to sustain a sortie generation rate for the Block 1A syllabus. • Restrictions on the aircraft operating limits prevented instruction in most high performance maneuvering and flight through instrument meteorological conditions (i.e., clouds). However, pilots were adequately trained in the basic operation of the aircraft. Mission systems were still immature, but generally unnecessary for this phase of training since no combat training could be performed. Even at this reduced level of activity, the radar, the HMDS, and the cockpit interfaces caused increased workload or had deficiencies. Aircraft availability was low during the OUE, but was adequate to meet the training sortie requirements with extensive workarounds. • Pilot training classes continued throughout 2013. Although aircraft availability and reliability at the training center remains below expectations, the shortened syllabus allowed pilot production to remain at planned levels. Eglin originally planned to produce 68 pilots during the 2013 period of performance, but the Services reduced their need to 66 pilots. All students completed planned training (of the reduced syllabus) on schedule.
DOD PROGRAMS • There are currently two configurations of aircraft at the training center, Block 1B and 2A. Six Lot 4 (Block 2A) aircraft were delivered in 2013 and several Lot 5 aircraft are in various stages of delivery. The first two F-35C aircraft were delivered to Eglin AFB in June. Pilot training using the syllabus for the Block 2A configuration starts in early 2014 after a small group rehearsal. • The training center continued to conduct maintenance training for experienced maintenance personnel for both the F-35A and F-35B during 2013. As of the end of September, 978 personnel had completed training in one or more of the maintenance courses to support fielded maintenance operations. Live Fire Test and Evaluation F135 Engine F135 engine vulnerability testing consisted of two test series: (1) fuel ingestion tests to examine the vulnerability of the F135 engine caused by fuel leakage from ballistically damaged fuel tanks adjacent to the engine inlets, and (2) ballistic tests to determine the damage tolerance of engine components, including fluid-filled components, sensors, actuators, and rotating components. • The fuel ingestion tests demonstrated the engine can tolerate a range of inlet fuel flows. These fuel flow rates simulated quantities representative of missile fragment-induced damage to fuel tanks adjacent to the engine. System-level ballistic test events planned for FY15, using a structural F-35C test article with an operating engine, will quantify the exact relationship of the simulated leak rates to those expected in an actual threat encounter. Further analysis will assess the vulnerability to multiple fragment impacts, which are probable in missile encounters. • The fuel ingestion tests did not simulate engagements by ground-based or aircraft gun systems that are possible during low-altitude close-air support missions and within‑visual‑range air-to-air combat. A Concept Demonstrator Aircraft engine test in 2005 showed the engine could not tolerate fuel ingestion events representative of such conditions (i.e., low-altitude, high-speed, high-engine thrust, and higher leak rates). The program made no design changes in response to those earlier test results and this vulnerability remains in the final production engine design. A ballistic liner in the fuel tank could mitigate this vulnerability, but the program removed this feature during its weight-reduction efforts, saving 48 pounds. • Tests using single missile fragments showed that the F135 rotating components were tolerant to these threats, with little or no effect on engine performance or component survival. However, three of four tests against fuel-filled external components resulted in massive fuel leaks, and one produced a sustained fire. The F-35C system-level tests in FY15 will evaluate whether installation effects, resulting in leaked fuel interacting with the engine exhaust, would increase the risk of fire. Engine vulnerability to high-explosive incendiary (HEI) and armor-piercing incendiary (API) threats was not
confirmed in this test series since historical data on similar engines already demonstrated that these threats can penetrate the engine core and create cascading damage resulting in engine failure and fires. F-35B Lift System • Ballistic tests on an F-35B STOVL propulsion system showed that single fragment damage to the lift fan did not degrade propulsion system performance. Analyses showed that fragment-induced damage could result in the release of more than 25 percent of a single lift fan blade, resulting in a catastrophic STOVL system failure. In order to preserve the test article for the remainder of the series, these engagement conditions were not tested. More severe threats, encountered at low-altitude or in air-to-air gun engagements, will likely cause catastrophic damage. • Ballistic tests of the lift fan shaft demonstrated that the design changes from the earlier Concept Demonstration Aircraft article improved its survivability against all threats, including the more severe API threat. • The F-35 has no sensors to warn the pilot of lift fan damage prior to conversion to STOVL flight upon return for landing. Conversion to STOVL flight puts high loads on the quickly accelerating system components that can result in catastrophic failure before the pilot can react and return the aircraft to wing-borne flight, or can create uncontained damage that cascades into other critical system failures. Prognostics and Health Management sensors that monitor component health and system degradation for maintenance purposes, could provide some warning, but the relevant software and hardware would have to be improved to provide reliable information to the pilot to support critical survivability decisions. On-Board Inert Gas Generation System (OBIGGS) • An OBIGGS/lightning protection Critical Design Review in February 2013 reviewed a system design capable of providing fuel tank inerting that would prevent fuel tank ullage explosion due to ballistic threat encounters or lightning strikes. The program is currently planning the F-35B fuel system simulator testing and ground tests on all three variants. Tests will include a spectrum of mission profiles, including high descent-rate dives to evaluate the improved OBIGGS ability to provide fuel tank inerting without compromising fuel tank and wing structure integrity. • In-flight inerting does not protect the aircraft against damage to the airframe resulting from lightning-induced currents. Most line-replaceable units (e.g., actuators and components of the electrical power system) have passed lightning tolerance qualification testing, but the existing F-35 airframe fasteners, selected to satisfy weight reduction criteria, are not lightning tolerant. The program still needs to complete lightning tolerance qualification testing for remaining components and current injection tests, before lifting current restrictions preventing aircraft operations within 25 miles of known lightning.
F-35 JSF 45
DOD PROGRAMS Polyalphaolefin (PAO) Shut-Off Valve • A live fire test in 2012 demonstrated crew and aircraft vulnerabilities to avionics coolant (PAO) system fires. The threat ruptured the PAO pressure line in the area just below the cockpit, causing a sustained PAO based fire with a leak rate of 2.2 gallons per minute (gpm). These results showed that a PAO shut-off valve that could detect and react to a 2 gpm, low leak rate could mitigate this vulnerability. Designing a system with this criterion poses some technical challenges, given a potential for excessive false alarms at these detection rates. • DOT&E repeatedly recommended redesigning and reinstalling a PAO shut-off valve after the program decided on removal for weight reduction. The program has been reconsidering the reinstatement of the PAO shut-off valve and has tasked Lockheed Martin to develop a technical solution to meet the criteria demonstrated in live fire tests. The program has not provided any updates on the operational feasibility and effectiveness of the design, or an official decision to reinstate this vulnerability reduction feature. Fueldraulic Fuses • The fueldraulic system is a fuel-based hydraulic system used to control the F-35B engine exhaust nozzle. It introduces a significant amount of fuel plumbing to the aft end of the engine and, consequently, an increased potential for fire. A live fire test in 2012 demonstrated the fueldraulics system is vulnerable to missile fragments, resulting in potential fire and loss of aircraft. Engine ballistic tests in FY13 also showed that the fueldraulics system is vulnerable and that a shut-off for a damaged system could mitigate much of the vulnerability. • A fueldraulic shut-off feature could also provide safety‑related protection. In 2013, prior to a routine flight test, testers discovered an F-35B fueldraulics line failure due to an improperly manufactured hose that could have led to an engine nacelle fire. An effective fueldraulic shut-off would prevent such an outcome. Electrical System • The F-35 includes several technologies used for the first time in a fighter aircraft that represent advancement of the more electric aircraft topology. The advances also provide a potential source of unique F-35 vulnerabilities. • All flight control electronic units and the electrical power system electrical distribution units have two voltage levels (270 and 28 volts DC) in internal circuits. An in-flight incident in 2007, electrical arcing tests in 2009, and the flight-critical system-level test events in 2012 showed that the vulnerability of the F-35 electrical power system requires further analyses to address the likelihood and significance of ballistically induced arcing between the 270-volt and 28-volt electrical systems. • Lockheed Martin also confirmed that all three F-35 variants include up to 28 wire harnesses that contain both 28- and 270-volt wires, but the contractor is still working
46 F-35 JSF
on providing the comprehensive extent and locations of these harness runs. Lockheed Martin should conduct a vulnerability analysis as soon as possible to determine the likelihood of ballistically- or lightning-induced arcing from the 270-volt on a 28-volt system and to determine whether the resulting damage effects would be catastrophic to the airplane. DOT&E will review these analyses to provide a comprehensive assessment of the F-35 vulnerability to ballistic damage to the electrical power system. Chemical/Biological Vulnerability The program continues to make progress in the development of the decontamination system in preparation for the full-up system-level test planned for FY17. • The F-35 Chemical Biological Warfare Survivability Integrated Product Team oversaw design and construction of a full-scale shelter liner and associated portable process containment shelter for chemical and biological decontamination operations. The contractor will set up the initial demonstration of shelter and liner for a form, fit, and function demonstration in 1QFY14 in conjunction with the Tactical, Cargo, and Rotary-Wing Aircraft Decontamination device. A full-scale setup at Edwards AFB in FY14 will demonstrate performance of the integrated liner, shelter, and decontamination system in preparation for the FY17 full-up system-level test of the apparatus with F-35 test article BF-4. • The Integrated Product Team is coordinating closely with the Joint Program Executive Office for Chemical and Biological Defense in developing the F-35 Joint Strike Fighter variant of the Joint Service Aircrew Mask. The mask, scheduled to undergo a Critical Design Review in 1QFY14, has high-schedule risk because its development is contingent on mask integration with the F-35 HMDS. The Mask Program Manager expects an LRIP version of the mask to be available in 3QFY14 in preparation for Mask/ HMDS flight qualification in 1QFY15. Gun Ammunition Lethality and Vulnerability • The F-35 program, the Air Force, Navy, Marines, and their international partners are conducting lethality live fire testing and evaluation of three different 25 mm gun ammunition types. -- PGU-48 frangible tungsten armor piercing design for the F-35A -- PGU-32 semi-armor piercing HEI ammunition for the F-35B and F-35C -- PGU-47 armor-piercing explosive ammunition for the partner F-35A variant and, depending on the overall cost and final lethality and reliability assessment results, possibly for the U.S. F-35B and F-35C variants • Each ammunition is specialized against different target sets particular to each Service, including personnel, small boats, ground structures, trucks, light armor, and fixed-/rotary-wing aircraft.
DOD PROGRAMS • Fracture characterization tests of the PGU-48 showed the tungsten to be much more frangible than other tungsten materials tested previously, which should increase predicted damage against targets employing widely-spaced materials. Characterization of all three ammunitions will continue in FY14 with terminal ballistics tests against multi-plate structures (representing vehicle materials) as well as building wall materials. FY15 tests will include ground-based and flight testing against representative targets. • The program assessed the vulnerability of the F-35 aircraft to ballistic threats while carrying these ammunitions in FY13. Ballistic tests against a single F-35 ammunition type (PGU‑32) showed that propellant explosive reaction was highly unlikely, while a propellant fire was probable. No propellant fire generated by ballistic impact triggered a propellant explosion. There was no evidence of sympathetic reactions in multiple round tests. Issues Affecting Operational Suitability Overall suitability performance continues to be immature, and relies heavily on contractor support and workarounds unacceptable for combat operations. Aircraft availability and measures of reliability and maintainability are all below program target values for the current stage of development. F-35 Fleet Availability • Average F-35 availability rates for operational units are below established threshold values. (Availability is not a meaningful metric for aircraft dedicated to test, and thus SDD aircraft are not included in this section.) -- The program established an availability threshold rate of 50 percent and an objective rate of 75 percent to track fleet performance for Performance Based Logistics agreements. -- Aircraft availability rates by operating location from November 2012 through October 2013 are summarized in the following table. The first column indicates the average availability achieved for the whole period, while the maximum and minimum columns represent the range of monthly availabilities reported over the period. F-35 Availability from November 2012 through October 2013* Operational Site
Average
Maximum
Minimum
Whole Fleet
37%
46%
26%
Eglin F-35A
38%
51%
24%
Eglin F-35B
39%
54%
22%
Eglin F-35C **
32%
61%
13%
Yuma F-35B
29%
45%
6%
Edwards F-35A
29%
41%
14%
Nellis F-35A
37%
63%
14%
* Data do not include SDD aircraft ** Eglin F-35C data began in August 2013
• Overall fleet availability has averaged 37 percent and showed a gradual decline in the latter half of the period reported in the table, with the last five months of the period all below
the average for the year. Late in the reporting period, the program began increasing the number of aircraft undergoing modifications and depot-level repairs, which contributed to the decline in fleet availability. While some operating sites did achieve threshold availability for a month or more, overall fleet availability never reached the threshold of 50 percent and was as low as 26 percent in February. • Unavailable aircraft are considered Not Mission Capable (NMC) because they are undergoing maintenance (NMC-M) for systems necessary for safe flight or are awaiting parts from supply (NMC-S). -- From November 2012 through August 2013, the NMC-M rate averaged 35 percent and was generally stable, but rose afterward and peaked at 47 percent in October. This observed NMC-M rate is well above the target rate of 6 percent established by the program for Performance Based Logistics evaluation. ▪▪ A significant portion of the aircraft down time has been the result of field maintenance organizations waiting for technical dispositions or guidance from the contractor on how to address a maintenance issue that has grounded an aircraft. These Action Requests (ARs) are a result of incomplete or inadequate technical data in the field, and waiting for their resolution accounts for 25 to 30 percent of the aircraft downtime. Recent trends have shown an increasing number of ARs per aircraft each month. Reducing the rate of ARs, or decreasing the response time to the ARs, should improve NMC-M rates. ▪▪ The requirement for modifications will continue to increase on the fleet and will likely adversely affect NMC-M rates for the next two years. Analysis of current modification plans show that up to 13 percent of the fielded fleet would be unavailable due to depot work alone in the late 2014 timeframe. -- Over the same period, the NMC-S rate averaged 27 percent, peaking at just over 30 percent in July 2013 and then gradually declining. The target value established by the Program Office is an NMC-S rate of 20 percent or less. According to the Program Office, lower than expected performance in NMC-S rates has been due to late contracting of the necessary spares for recent production lots. They expect that improved contracting performance and increasing maturity of the supply system will result in improved parts support by late 2014. F-35 Fleet Reliability • The F-35 program uses reliability growth curves that project expected reliability for each variant throughout the development period based on accumulated flight hours. -- These growth curves are established to compare observed reliability with a target to meet the Mean Flight Hours Between Critical Failure (MFHBCF) threshold requirement by 75,000 flight hours for the F-35A and F-35B, and by 50,000 flight hours for the F-35C.
F-35 JSF 47
DOD PROGRAMS -- Currently, none of the variants are achieving their predicted reliability based on flight hours accumulated as of the end of August 2013, as shown in the following table. F-35 Reliability as of August 31, 2013 – MFHBCF, Hours Requirement
Observed MFHBCF as of May 2012
Observed MFHBCF
Current Total Flight Hours
Objective MFHBCF from Growth Curve
Observed as % of Objective Value
75,000
4.5
4,204
13.5
33%
5.9
12
75,000
3.0
3,286
7.7
39%
4.2
14
50,000
2.7
903
9.0
30%
6.7
Threshold MFHBCF
Threshold Flight Hour Target
F-35A
20
F-35B F-35C
Variant
Current Values
• Though month-to-month reliability rates vary significantly, in part due to the small fleet size, the F-35B showed slight improvement over the reporting period, while F-35A reliability appears to be relatively flat. The program has fielded too few F-35C aircraft to assess reliability trends. • Statistical analysis of the 90-day rolling averages for Mean Flight Hours Between Critical Failure – Design Controllable (MFHBCFDC) through the end of July 2013 show flat trend lines for the F-35A and F-35B with most data points below the threshold growth curve, meaning the observed reliability is not within the desired envelope for design controllable failures. Design controllable failures are those that can be attributed to deficiencies in component design, but considered by the Program Office to be fixable by design modification. -- While some design improvements will be incorporated in production of the Lot 5 aircraft, most of the remaining planned improvements are being incorporated in Lots 6 and 7. The next opportunity to expect improvement in the fleet reliability performance is likely to be in 2015. However, some design improvements planned to be cut-in with these production lots are for structural fatigue life and increased mission capability which will not necessarily improve reliability. -- Through November 2013, all F-35 test and production aircraft combined had achieved 11,500 total flight hours, 6 percent of the flight hour total (200,000 hours) at which the ORD reliability goal is to be achieved. However, the design is becoming more stable and opportunities for reliability growth are decreasing. While the relatively low number of flight hours shows there is still time for program reliability to improve, this is not likely to occur without a focused, aggressive, and well-resourced effort. • A number of components have demonstrated reliability much lower than predicted by engineering analysis, which has driven down the overall system reliability. High driver components affecting low availability and reliability include the following, grouped by components common to all variants as well as by components failing more frequently on a particular variant or completely unique to it, as shown in the following table.
48 F-35 JSF
High Driver Components Affecting Low Availability & Reliability Specific to Variant
Common to All Variants
F-35A
• Data transfer cartridge • Position/strobe light lens assembly
F-35B
• Upper lift fan door actuator • Main landing gear wheel/ tire assembly
F-35C
• Ejection seat portion assembly • Data security module
• • • • • • • • •
270 Volt Direct Current battery Fiber channel switch Avionics processor Power and thermal management system Landing gear and tire assembly Display management computer/ helmet On-Board Oxygen Generating System Crew escape and safety system 80kW Inverter/Converter/Controller
Maintainability • The amount of time required to repair failures for all variants exceeds that required for mature aircraft, and has increased over the past year. The table below compares the Mean Corrective Maintenance Time for Critical Failure (MCMTCF) and Mean Time To Repair (MTTR) for all unscheduled maintenance for each variant as of August 31, 2013, to the threshold requirement from the ORD and the same value reported in the FY12 Annual Report. F-35 Maintainability as of August 31, 2013 - MCMTCF (Hours) Variant
Threshold
Observed
% of Threshold
FY12 Annual Report
F-35A
4.0
12.1
303%
9.3
F-35B
4.5
15.5
344%
8.0
F-35C
4.0
9.6
241%
6.6
F-35 Maintainability as of August 31, 2013 - MTTR (unscheduled) Variant
Threshold
Observed
% of Threshold
FY12 Annual Report
F-35A
2.5
9.2
366%
4.2
F-35B
3.0
8.9
294%
5.3
F-35C
2.5
7.7
307%
4.0
• Maintenance times reported by the Program Office have increased (worsened) compared to those reported a year ago. -- The causes of this increase are not clear from the available data, which are derived from a fleet that has only early mission systems functionality, but has grown to include three new operating locations this year. It is too early to determine if the increase in maintenance times is from immaturity of sustainment operations in the field (i.e., incomplete technical data and low experience of newly-trained maintenance personnel) or from underlying maintainability and aircraft design issues, such as poor component reliability and maintenance actions requiring excessive time to complete. -- Cure time to restore low-observable (LO) characteristics following maintenance behind panels not designed for frequent access might be a factor in the increased maintenance time, but the Program Office has not tracked
DOD PROGRAMS LO maintenance times separately. The Program Office should include LO and non-LO repair times in their monthly performance metrics to help understand the root cause of these increases and take corrective actions. Further, LO repair should be broken down into repair times for inherent LO failures, and LO repairs required to facilitate other maintenance. The proportion of all LO repairs that are required to facilitate other maintenance should be reported. Autonomic Logistics Information System (ALIS) • The Program Office continues to develop and field ALIS in incremental capabilities similar to the mission systems capability in the air vehicle. Overall, the ALIS is immature and behind schedule, which adversely affects maintainability and sortie generation. Shortfalls in functionality and data quality integrity require workarounds and manual intervention. • ALIS version 1.0.3, required for the Services to accept production Lot 4 aircraft at Eglin AFB, Florida, Nellis AFB, Nevada, and Yuma Marine Corps Air Station, Arizona, underwent initial testing at the Edwards test center in late 2012 and began fielding in early 2013. -- During initial testing in 2012, the Edwards test team found shortcomings in the systems integration of ALIS applications and a lack of maturity in handling data elements. The team identified four critical (Category I) deficiencies, which required correction before fielding, and 54 severe (Category II) deficiencies, which required significant workarounds. -- The contractor developed an updated version of the ALIS 1.0.3 software to address some of the deficiencies identified during initial testing and the Edwards test team retested the software in December 2012. The program subsequently started fielding this version of ALIS 1.0.3 in early 2013. -- The Patuxent River test team reported on the performance of the updated version of ALIS 1.0.3 in May 2013, and indicated that at least three of the four Category I deficiencies identified during initial testing remained open. • Prior to the start of the Block 2B operational utility evaluation, the program must correct deficiencies in ALIS 1.0.3, finish development of ALIS 2.0, and integrate the propulsion module in ALIS 2.0.1, which is required for Marine Corps Initial Operational Capability (IOC). The Edwards test center plans to begin testing of ALIS 2.0 in April 2014 and ALIS 2.0.1 in September 2014. Delays in the release of ALIS 2.0 or 2.0.1 will add schedule risk to the Block 2B fleet release planned for mid-2015. • The current Squadron Operating Unit (SOU) used by ALIS failed to meet the deployability requirement in the ORD due to the size, bulk, and weight of the current SOU design. To address the requirement, the program is developing a deployable version of the SOU, deemed SOU V2. It will support aircraft in the Block 2B, 3i, and 3F configuration, and is a critical delivery item for meeting Service IOC dates.
The Program Office has divided the SOU V2 development into multiple increments. -- The first increment includes the capability to deploy and support the requirements for Marine Corps IOC. This increment will align hardware (SOU V2) and software (ALIS 2.0.1) releases to allow testing to begin at the Edwards flight test center in January 2015. -- The second increment, currently unfunded, will address U.S. Air Force requirements for sub-squadron reporting capabilities and inter-squadron unit connectivity. -- A third increment, also unfunded, plans to add decentralized maintenance capability, which will allow personnel to manage tasks with or without connectivity to the main SOU. • To date, diagnostic system performance has failed to meet basic functional requirements, including fault detection, fault isolation, and false alarm rates. Due to the failure to meet these requirements, the program has discontinued the development of enhanced diagnostics (model-based reasoning) for the remainder of SDD. The program has initiated manual workarounds in the field, such as maintainer-initiated built-in tests and reliance on contractor support personnel, for more accurate diagnostics of system faults. Joint Technical Data • Development of Joint Technical Data (JTD) modules for the F-35A and F-35B is largely complete. Verification naturally lags behind development, but is progressing toward completion. Verification of modules requiring extensive intrusion into the aircraft is planned to be completed during depot-level modifications or opportunistic maintenance. The F-35C lags behind the other variants, but is proceeding quickly because of variant similarities. The chart below shows the status of JTD development and verification for each variant, propulsion, support equipment, and sustainable low observable (SLO) maintenance. Results exclude JTD for pilot flight equipment and JTD unique to LRIP aircraft (such as structural field repairs) that will not be needed for full-rate production aircraft. From October 2012 to October 2013, the Program Office verified 2,581 aircraft and 822 propulsion modules. Early in 2014, the primary focus in JTD verification will be weapons and stores. Data Modules Identified
Data Modules Completed
% Data Modules Completed
% Data Modules Verified
1
F-35A
4,404
4,045
91.9%
81%
F-35B1
5,314
4,766
89.7%
76%
(as of Oct 2013)
F-35C
4,514
3,357
74.4%
55%
Propulsion
2,892
2,861
98.9%
94%
1
SE
2,241
489
21.8%
13%
SLO
1,362
291
21.3%
3%
Total
20,727
15,809
76.3%
64%
Note: 1. Includes field and depot-level JTD for Operations and Maintenance (O&M) for air vehicle only
F-35 JSF 49
DOD PROGRAMS • As stated earlier in the F-35 fleet availability section, aircraft maintenance personnel submit ARs to Lockheed Martin when the needed JTD is not available to troubleshoot or resolve a problem with an aircraft. The time maintenance personnel wait for resolution of these ARs contribute to aircraft non-availability (25-30 percent of the reported NMC time has been due to AR wait time). -- Lockheed Martin prioritizes and responds to ARs through the Lightning Support Team, which is composed of Service and contractor personnel. The support has been fairly successful in responding to the most critical ARs with at least an interim solution in a timely manner, but because of manpower limitations, has been unable to handle the backlog of less severe ARs. -- A s of August 2013, 231 critical ARs remained open, while over 200 severe ARs were open. A critical AR addresses a deficiency which may cause major loss or damage to a system, or severe injury or possible death to personnel if not corrected. A severe AR addresses a deficiency which adversely affects operational safety, suitability, or effectiveness; however, a workaround is permitted. F-35B Air-Ship Integration and Ship Suitability Testing • The Navy deployed two F-35Bs to LHD-1 (USS Wasp) for two weeks in August 2013 to continue assessing shipboard suitability and integration. The Navy is continuing to analyze data from this deployment. Permanent modifications to the Wasp to prepare for JSF integration included: -- A ddition of transverse stiffeners to the underside of the flight deck for the two landing spots used by the F-35B and application of thermal non-skid material to the topside of the flight deck for one landing location. The Marine Corps applied the non-skid material to the other landing location before an earlier detachment to the Wasp. -- Deck edge modifications, including the removal, replacement, relocation, and shielding of communications systems. -- Added fire detection and alarming systems for the lithium‑ion battery charging and storage area. • Temporary alterations for the Wasp for this detachment include: -- Lithium-ion battery charging and storage areas. The Marine Corps has not determined the final design of these areas. -- Short take-off rotation line lights. Analysis of results will determine the precise location of these lights. -- Addition of test equipment. • The deployment met the primary objective of collecting data to support the development of a Block 2B operational flight envelope for take-offs and landings. The test team expanded the range of aircraft weight and center of gravity compared to that developed from the first deployment in 2011 and conducted operations in both day and night conditions. The test team completed 95 short take-offs and vertical landings, including forward and aft facing landings, and 17 night take-offs and landings during the deployment.
50 F-35 JSF
• The Marine Corps is developing solutions to a number of challenges in integrating the F-35B onto L-class ships: -- Large-scale application of a thermal non-skid material to the flight deck in F-35B landing locations. -- Modification of the flight deck structure to eliminate excess stress, which includes transverse panel breakers installed on the underside of the existing flight deck structure. -- Design of separate charging and storage lockers for the lithium-ion batteries required for the JSF and new storage locker for pilot flight equipment, as the JSF helmet is larger and more fragile than legacy helmets. -- New firefighting procedures in the event of a fire on the flight deck near aircraft carrying internal ordnance. -- Understanding requirements for gun pod storage. -- Conducting feasibility studies on the resupply of F-35B engines while underway, which could include a greater space allocation for engine storage aboard ship or through underway replenishment using a Navy system currently installed on one supply ship and scheduled for installation on CVN-78. -- The Marine Corps has determined that new active noise reduction personal hearing protection is necessary for on-deck personnel because of the high level of engine noise. Noise damping materials and/or personal hearing protection may also be needed for below-deck personnel. F-35C Air-Ship Integration and Ship Suitability Testing • Although a number of air-ship integration issues are common to both CVN and L-class ships, such as lithium-ion battery storage, pilot flight equipment storage, need for new shipboard firefighting procedures, and high noise levels, some issues and their solutions are particular to aircraft carriers. The Navy has made progress in addressing some of these integration issues, but several challenges remain. -- The program began testing its redesigned arresting hook system on a flight test aircraft in late CY13. The redesign was necessary after the original system failed to engage the cable and demonstrate sufficient load-carrying capacity. The arresting hook system remains an integration risk as the JSF development schedule leaves no time for new discoveries. Other risks include the potential for gouging of the flight deck after a missed cable engagement (due to an increase in weight of 139 pounds) and the potential for sparking from the tail hook across the flight deck because of the increased weight and sharper geometry of the redesigned hook. -- The Navy is redesigning the cooling system in the Jet Blast Deflectors, which deflect engine exhaust during catapult launches, to handle JSF engine exhaust. The redesign will include improvements in side-cooling panels. -- CVN-78 will receive the new Heavy underway replenishment (UNREP) system along with one resupply ship, but the Navy has delayed this system for eight years on other ships. This new UNREP system is the
DOD PROGRAMS
--
--
---
only system capable of transporting the JSF engine and container while the carrier is underway. The JSF engine container was unable to sustain the required sudden drop of 18 inches (4.5 g’s) without damage to the power module during shock testing. The Navy is redesigning the container to better protect this engine, but this is likely to result in an increase in container size and weight. The Navy estimates new container availability in late 2016. Engine noise is a potential risk to personnel on the flight deck and one level below the flight deck. The Navy has decided to procure active noise reduction personal hearing protection for on-deck personnel. Projected noise levels one level below the flight deck (03 level) will require at least single hearing protection. On most carriers this is a berthing area, but on CVN-78 this is a mission planning space; personnel wearing hearing protection in mission- planning areas will find it difficult to perform their duties. The Navy previously tested acoustic damping material in 2012 and is developing a model to optimize material placement. Storage of the JSF engine is limited to the hangar bay, which will affect hangar bay maintenance operations. The impact on the JSF logistics footprint is not yet known. Lightning protection of JSF aircraft while on the flight deck will require the Navy to modify nitrogen carts to increase capacity. Nitrogen is used to fill fuel tank cavities and inert aircraft at specified intervals while on deck.
Progress in Plans for Modification of LRIP Aircraft • The Program Office and Services continued planning for modification of early LRIP aircraft to attain planned service life and the final SDD Block 3 capability. -- Planning has focused on modifying aircraft in preparation for the Block 2B operational utility evaluation and Marine Corps IOC, both planned to occur in 2015. -- Because operational test aircraft are to be production‑representative, the Program Office must coordinate verification and approval of all modifications, the availability of docks at the aircraft depots as they open for operation, and the availability of long-lead aircraft parts needed for modifications with inputs from the Services on modification priority. • The Program Office developed a modification and retrofit database that contains information for each entry on Service prioritization, when the modification will become part of the production line, which aircraft will require modification, whether unmodified aircraft are limited in performance envelope and service life or will require additional inspections, and operational test requirements and concerns. • Modifications that do not require depot induction will be performed by depot field teams (who will travel to aircraft operating locations or to depots to work alongside depot teams) or by unit-level maintainers. The Program Office and Services adjudicate the location of all Block 2B modifications.
• Modifications to support the operational utility evaluation of Block 2B capability include: -- Missions systems modifications, including those for Block 2B capability -- Structural life limited parts, referred to as Group 1 modifications -- STOVL Mode 4 operations modifications, which include a modification to the Three Bearing Swivel Module, which is required to allow STOVL aircraft to conduct unrestricted Mode 4 operations -- Lightning certification, which includes OBIGGS modification (the lightning qualification of line-replaceable components and development of a system-level test still need to be completed before the aircraft modifications can proceed) -- Support/training systems, which include the ALIS and pilot training device to support operational test aircraft -- Other modifications, including those to vehicle systems, airframes, aircraft operating limitations, and weapons. • The concurrency of production with development created the need for an extensive modification plan to ensure aircraft are available and production-representative for operational testing. The current modification schedule contains no margin and puts at risk the likelihood that operationally representative aircraft will be available for the Block 2B operational utility evaluation when it is currently planned by the Program Office to occur in 2015. Recommendations • Status of Previous Recommendations. The program and Services are satisfactorily addressing three of ten previous recommendations. The remaining recommendations concerning correction of the schedule in the TEMP, end-to‑end ALIS testing, VSim validation, alignment of weapons test schedules with the Integrated Master Schedule, test of the redesigned OBIGGS system, reinstatement of the PAO shut-off valve, reinstatement of the dry-bay fire extinguisher system, and provision of a higher resolution estimate of time remaining for controlled flight after a ballistic damage event are outstanding. • FY13 Recommendations. The program should: 1. Ensure flight test timeline estimates for remaining SDD flight testing faithfully account for the historical growth in JSF testing, in particular for mission systems and weapons integration. 2. Plan realistic rates of accomplishment for remaining weapons integration events; assure the events are adequately resourced from the planning phase through data analysis. 3. Resource and plan SDD flight test to acquire the needed validation data for VSim. 4. Track and publish metrics on overall software stability in flight test. The stability metrics should be “mission focused” and account for any instability event in core
F-35 JSF 51
DOD PROGRAMS or sensor processors, navigation, communication, radar, EOTS, DAS, or fusion display to the pilot. 5. Design and reinstate an effective fueldraulic shut-off system to protect the aircraft from fuel-induced fires. Recent testing has shown that this feature could protect the aircraft from threat-induced fire; this is also a critical flight safety feature. 6. Determine the vulnerability potential of putting 270-volt power on a 28-volt signal bus. Due to the unique electrical nature of the F-35 flight control system, the Program Office should thoroughly examine and understand this vulnerability before this aircraft becomes operational. The Program Office should successfully incorporate the wire harness design and the associated vulnerabilities in the F-35 vulnerability analysis tools. 7. Develop a plan to improve the Integrated Caution and Warning system to provide the pilot with necessary vulnerability information. The vehicle system should have
52 F-35 JSF
the capability of detecting and reporting to the pilot any component ballistic damage (e.g., lift fan shaft) that could lead to catastrophic failure (e.g., upon attempt to convert to STOVL flight). 8. Track LO and non-LO repair times across the fleet and report them separately in monthly performance metrics. Separately track LO repairs due to inherent LO failures and due to facilitating other maintenance actions, and note the proportion of all LO repairs that are caused by facilitating other maintenance actions. 9. Plan to conduct the operational utility evaluation of Block 2B using comparative testing of the capabilities Block 2B provides relative to the capabilities provided by legacy aircraft. This approach was used to test the F-22, and is particularly critical for Block 2B operational testing because no detailed formal requirements for Block 2B performance exist.
DOD PROGRAMS
Global Command and Control System – Joint (GCCS-J) Executive Summary • Defense Information Systems Agency (DISA) development focused on implementing high-priority capability enhancements, infrastructure improvements, and software defect corrections to both the Global Command and Control System – Joint (GCCS-J) Global (referred to as Global) and Joint Operation Planning and Execution System (JOPES). Global • DISA developed Global v4.2.0.9 Update 2 and Update 2 Emergency Release to correct remaining defects discovered during Air Operations Center – Weapons System (AOC- WS) Recurring Event (RE)12 testing. -- The AOC-WS RE12-1 operational testing, which included fixes to defects from RE12, concluded that all Category I deficiencies had been adequately resolved, except for a Category I deficiency with the software build and upgrade process. After corrective action, the Air Force executed the RE12-1 build and upgrade process with acceptable levels of interaction with the Tier II help desk and the documentation showed improved maturity. • DISA developed Global v4.2.0.10 to implement the initial phase of Pedigree Security Data Tagging, which identifies the originator and classification level of each track, allowing releasable tracks to be passed between U.S. and coalition partners. -- Global v4.2.0.10 testing confirmed the system is operationally effective, suitable, and secure. • DISA developed Global Lite v1.0 to provide a desktop computer-based software platform for situational awareness and intelligence capabilities. Global Lite v1.0 includes select features of situational awareness, intelligence, and cross-functional/infrastructure capabilities provided in fielded versions of Global. Global Lite v1.0 also supports the Public Key Enabling hardware token requirements for Public Key Infrastructure authentication and single sign-on capabilities. -- Global Lite v1.0 testing confirmed the system is operationally effective, suitable, and secure. • DISA developed Global v4.3 to implement changes to the infrastructure of the Global products and move the baseline towards a more flexible and service-oriented architecture. The release also provides high-priority updates to the Integrated Command, Control, Communications, Computers, and Intelligence System Framework; Joint Targeting Toolbox (JTT); and Modernized Integrated Database (MIDB). -- The Global v4.3 operational test was not adequate to stress the system, but demonstrated that the system would not be effective in an operational environment involving current and legacy versions of Global and
associated versions of the MIDB requiring the use of the T-Sync system to achieve MIDB data synchronization. DOT&E concluded that critical defects, primarily with the MIDB, would substantially degrade U.S. capabilities with respect to situational awareness, targeting, weaponeering, and intelligence information. DOT&E recommended follow-on operational testing of Global v4.3 with MIDB synchronization using realistic operational loads in an environment using current and legacy versions of Global. This testing should include an operationally representative T-Sync system. JOPES • DISA developed JOPES v4.2.0.2 Update 1 to implement Transportation Tracking Account Number (TTAN) capabilities in the Joint Forces Requirements Generator (JFRG) II, enabling commanders to track personnel and equipment through the planning and deployment process. This release also provides upgrades to JOPES Data Network Services (JDNETS) software supporting Deliberate Crisis Action Planning and Execution Segments (DCAPES) v5.0.0.1 ability to pull Unit Type Codes (UTCs). -- JOPES v4.2.0.2 Update 1 testing showed TTAN capabilities were successfully implemented within JFRG II and all Category I problems were resolved. While changes to the JDNETS web services did not support the ability of DCAPES v4.2.2.2 to pull UTCs, these changes did not degrade current operations. The results of the OT&E demonstrated that JOPES v4.2.0.2 Update 1 remains operationally effective, suitable, and secure.
GCCS-J 53
DOD PROGRAMS • DISA developed JOPES v4.2.0.3 to implement the required framework for interoperability and synchronization between JOPES and Defense Readiness Reporting System – Strategic (DRRS-S). This release also contains the infrastructure needed for the Joint Planning and Execution System Framework to begin connecting to JOPES using web services, rather than via a direct database connection. -- The Joint Interoperability Test Command (JITC), in conjunction with DISA, was scheduled to conduct a combined System Acceptance Test (SAT) and operational test of the JOPES v4.2.0.3 release from September 25 through October 24, 2013. Testing has been delayed due to the shutdown of the Federal Government and the lack of a Defense Appropriation. System • GCCS-J consists of hardware, software (commercial offthe- shelf and government off-the-shelf), procedures, standards, and interfaces that provide an integrated near real-time picture of the battlespace necessary to conduct joint and multi-national operations. GCCS-J consists of a client/server architecture using open systems standards, government-developed military planning software, and an increasing use of World Wide Web technology. • GCCS-J consists of two components: - Global v4.3 (Force Protection, Situational Awareness, Intelligence applications) to include the Global Lite v1.0 variant (Situational Awareness, Intelligence, and Cross‑functional applications) - JOPES v4.2.0.3 (Force Employment, Projection, Planning, and Deployment/Redeployment applications) Mission • Joint Commanders utilize the GCCS-J to accomplish command and control. Global • Commanders use Global:
Activity Global • DISA developed Global v4.2.0.9 Update 2 and Update 2 Emergency Release to correct remaining defects discovered during AOC-WS RE12 testing. • DISA developed Global v4.2.0.10 to implement the initial phase of Pedigree Security Data Tagging, which identifies the originator and classification level of each track, allowing releasable tracks to be passed between U.S. and coalition partners. • DISA developed Global Lite v1.0 to provide a desktop computer-based software platform for situational awareness and intelligence capabilities. Global Lite v1.0 includes select features of situational awareness, intelligence, and
54 GCCS-J
-- T o link the National Command Authority to the Joint Task Force, Component Commanders, and Service- unique systems at lower levels of command -- To process, correlate, and display geographic track information integrated with available intelligence and environmental information to provide the user a fused battlespace picture -- To provide Integrated Imagery and Intelligence capabilities, which integrate imagery and other relevant intelligence into the common operational picture and allow Commanders to manage and produce target data using the Joint Targeting Toolbox -- To provide a missile warning and tracking capability • The AOC uses Global: -- To build the air picture portion of the common operational picture and maintain its accuracy -- To correlate or merge raw track data from multiple sources -- To associate raw Electronics Intelligence data with track data -- To perform targeting operations JOPES • Commanders use JOPES: -- To translate policy decisions into operation plans to meet U.S. requirements for the employment of military forces -- To support force deployment, redeployment, retrograde, and reposturing -- To conduct contingency and crisis action planning Major Contractors • Government Integrator: DISA • Software Developers: - Northrop Grumman – Arlington, Virginia - SAIC – Arlington, Virginia - Pragmatics – Arlington, Virginia
cross-functional/infrastructure capabilities provided in fielded versions of Global. Global Lite v1.0 also supports the Public Key Enabling hardware token requirements for Public Key Infrastructure authentication and single sign-on capabilities. • DISA developed Global v4.3 to implement changes to the infrastructure of the Global products and move the baseline toward a more flexible and service-oriented architecture. The infrastructure changes include migration to Solaris 10 08/11, Windows 7®, and Windows Server 2008® R2 operating systems. Global v4.3 will automate software deployment and installation to support both full installation and update without dedicated install teams. The
DOD PROGRAMS release also provides high-priority updates to the Integrated Command, Control, Communications, Computers, and Intelligence System Framework; JTT; and MIDB. • JITC and the Air Force conducted Global testing at multiple echelons. JITC led testing at the higher Combatant Command echelon to support DISA Global fielding decisions. The Air Force led testing at the lower echelon to support AOC-WS fielding decisions. • Combatant Command-level testing included the following events: -- JITC, in conjunction with DISA, conducted a combined SAT and operational test of Global v4.2.0.10 from October 22, 2012, through March 27, 2013. -- JITC, in conjunction with DISA, conducted an operational test of Global Lite v1.0 from March 13 through April 2, 2013. -- JITC conducted the Global v4.3 Operational Assessment from June 5 – 13, 2013. -- JITC conducted a Global v4.3 operational test from August 7 – 16, 2013. • AOC-level testing included the following events: -- The Air Force performed developmental testing of AOC‑WS RE12 from December 3 – 8, 2012, at Langley AFB, Virginia. A significant portion of the testing involved additional testing of the Global v4.2.0.9 Update 2. -- The Air Force performed regression testing of RE12-1 from July 8 – 19, 2013, at Langley AFB, Virginia. A significant portion of RE12-1 involved additional testing of the Global v4.2.0.9 Update 2 Emergency Release. -- The Air Force performed operational testing of RE12-1 in August 2013 at Langley AFB, Virginia. JOPES • DISA developed JOPES v4.2.0.2 Update 1 to implement TTAN capabilities in the JFRG II, enabling commanders to track personnel and equipment through the planning and deployment process. This release also provides upgrades to JDNETS software supporting DCAPES v5.0.0.1 ability to pull UTCs. • JITC, in conjunction with DISA, conducted an operational test of the JOPES v4.2.0.2 Update 1 from October 22, 2012, through March 27, 2013. • DISA developed JOPES v4.2.0.3 to implement the required framework for interoperability and synchronization between JOPES and DRRS-S. DRRS-S is intended to replace the Status of Resources and Training System as the readiness reporting system of record. JOPES v4.2.0.3 also contains the infrastructure needed for the Joint Planning and Execution System Framework to begin connecting to JOPES using web services, rather than via a direct database connection. • JITC, in conjunction with DISA, were scheduled to conduct a combined SAT and operational test of JOPES v4.2.0.3 from September 25 through October 24, 2013. Testing has been delayed due to the shutdown of the Federal Government and the lack of a Defense Appropriation.
• Global and JOPES operational testing did not require DOT&E-approved test plans due to their limited scope. This is in accordance with the DOT&E risk assessment policy, which JITC exercised. Operational test plans were approved by their respective test organizations and coordinated with DOT&E. JITC deviated from the Global v4.3 operational test plan due to T-Sync problems and limited user participation. Assessment Global • Results from the Combatant Command-level testing include the following: -- Global v4.2.0.10 testing confirmed the system is operationally effective and suitable, interoperable with conditions, and secure. -- Global Lite v1.0 testing confirmed the system is operationally effective and suitable, interoperable with conditions, and secure. -- The Global v4.3 Operational Assessment was conducted to characterize defects discovered during DISA-led developmental testing and assess readiness to proceed into operational testing. Multiple critical defects with the JTT, Generic Area Limitation Environment Interface, Defense Intelligence Agency’s MIDB, and the T-Sync system (used to perform MIDB synchronization) remained open at the conclusion of testing. -- Global v4.3 operational testing was not adequate because it did not include operationally representative stress levels. Specifically, it employed too few operational users to place operationally representative loading on the Global v4.3 applications, insufficient loading on the Global 4.3 MIDB test servers to stress the system, and a significantly undersized T-Sync system. Operational testing showed that Global v4.3 would not be effective in an operational environment involving current and legacy versions of Global and associated versions of the MIDB, thus requiring the use of the T-Sync system. Additionally, the remaining critical defects, primarily with the MIDB, would substantially degrade U.S. capabilities with respect to situational awareness, targeting, weaponeering, and intelligence information. DOT&E recommended follow-on operational testing of Global v4.3 with MIDB synchronization using realistic operational loads in an environment using current and legacy versions of Global. This testing should include an operationally representative T-Sync system. -- DOT&E recommended to DISA that GCCS-J 4.3 not be fielded to any site until the critical defects that were found in recent testing are fixed and verified through additional testing. • Results from the AOC-level testing include the following: -- The AOC-WS RE12-1 developmental testing identified several software deficiencies in Global v4.2.0.9 Update 2, but concluded the system is a significant improvement from previous versions.
GCCS-J 55
DOD PROGRAMS -- T he AOC-WS RE12-1 regression testing concluded that Global 4.2.0.9 Update 2, with fixes contained in Update 2 Emergency Release, provides the AOC-WS improved functionality over previously delivered versions of Global, with existing deficiencies, and is ready to proceed to operational testing. -- The AOC-WS RE12-1 operational testing concluded that all deficiencies against Global v4.2.0.9 Update 2 had been adequately resolved, except for a Category 1 deficiency with the software build and upgrade process. After corrective action, the RE12-1 build and upgrade was executed with acceptable levels of interaction with the Tier II help desk and the documentation showed improved maturity. JOPES • JOPES v4.2.0.2 Update 1 testing showed TTAN capabilities were successfully implemented within JFRG II and all Category I problems were resolved. While changes to the JDNETS web services did not support the ability of DCAPES v4.2.2.2 to pull UTCs, these changes did not
56 GCCS-J
degrade current operations. The results of the OT&E demonstrate that JOPES v4.2.0.2 Update 1 remains operationally effective, suitable, and secure. Recommendations • Status of Previous Recommendations. DISA partially addressed the previous recommendation for the Combatant Command and AOC communities to test Global v4.3. However, Global v4.3 operational testing did not include operationally representative stress levels. • FY13 Recommendations. 1. JITC should conduct adequate operational testing of Global v4.3 in accordance with a DOT&E-approved test plan to clearly demonstrate that the MIDB supports data synchronization in an environment with current and legacy versions of Global at the rates essential to the conduct of major combat operations. 2. The Defense Intelligence Agency should use the follow-on testing as an opportunity to establish a standing test bed for subsequent releases of the MIDB synchronization software.
DOD PROGRAMS
Integrated Electronic Health Record (iEHR) Executive Summary • The Department of Defense (DoD) and Veterans Affairs (VA) will use the Integrated Electronic Health Record (iEHR) program to implement an EHR that both organizations can use to meet the healthcare needs of their beneficiaries and the clinicians providing the healthcare. iEHR Accelerators • The iEHR Program Manager is developing the accelerator programs in multiple phases. Only the first phase of each accelerator is to be tested and deployed to meet the December 2013 deadline established in the June 21, 2013, USD(AT&L) Acquisition Decision Memorandum. Subsequent phases of the accelerators will be consolidated under one program for completion by September 30, 2014. -- DOT&E observed developmental testing (DT), which began November 11, 2013, and is scheduled to complete in December 2013. The Data Federation accelerator is designed to achieve data interoperability within the DoD and VA healthcare systems. -- The Tricare Online (TOL) Blue Button application Phase I, which converts patient data into a standard format, was completed April 26, 2013. Blue Button Phase II DT, which allows sharing patient data with medical providers, began on November 19, 2013, and is scheduled to complete in December 2013. -- The Medical Community of Interest (Med-COI) accelerator DT was successfully completed in October 2013. It is intended to create a medical network that meets both DoD and VA security requirements. The Med-COI Interagency Program Office (IPO) recently decoupled Med-COI from other iEHR accelerators and it is not clear how the capability will fit into the larger iEHR architecture. • An operational assessment (OA) of the iEHR accelerators is scheduled for January 6 – 17, 2014. A second OA will be conducted later in FY14, once the accelerators have completed all phases of development. iEHR Increment 1 and Other Development • The Single Sign-on and Context Management (SSO‑CM) solution was designed to provide a virtual clinical workspace within which doctors and clinicians can seamlessly transact across multiple applications. SSO-CM underwent development and testing in FY13, but testing revealed a significant number of defects that persisted through multiple DTs. DOT&E rejected the OA plan in July 2013 because it did not demonstrate that the SSO-CM systems would work with, and not interfere with, the IPO’s primary deliverables, which are the DoD and VA iEHR accelerators. SSO-CM development will continue with a new completion date of September 2014. DOT&E
will oversee and report on testing of SSO-CM prior to its deployment. DOT&E views it as being essential to perform operational evaluations of all iEHR and DoD Healthcare Management System Modernization (DHMSM) capabilities as they are deployed for use. • DOT&E has added the DHMSM program to the test and evaluation oversight list to test the full, end-to-end capabilities of the new “core” capability with iEHR architecture, accelerators, and SSO-CM capabilities. DOT&E will ensure adequate plans are developed and integrated between the two programs to assess the required interfaces and interaction between the systems as part of the overall effort to modernize the DoD and VA healthcare systems. • Development Test Center/Environment (DTC/DTE) was to be used in support of accelerator development and testing; however, technical problems have prevented its use. The IPO anticipates the DTC/DTE will be fully operational in December 2013. • Service-Oriented Architecture (SOA) Suite/Enterprise Service Bus (ESB) provides the transport for message exchange among the DoD Military Health System (MHS), the VA EHRs, and associated information management systems. The SOA Suite/ESB-combined DT was successfully conducted in FY13 using test tools to simulate operational traffic. SOA Suite/ESB will use adapters to connect to external systems; however, no adapters were planned as part of the initial deployment. Operational testing will be conducted once adapters are available to allow external applications, services, and consumers to connect to the SOA Suite/ESB. The Program Executive Officer (PEO) for the DoD Healthcare Management
iEHR 57
DOD PROGRAMS Systems (DHMS) is currently uncertain if the SOA Suite/ESB will be part of the objective Data Federation accelerator design. System • The iEHR program represents the collective DoD and VA effort to implement an EHR system that both organizations can use to meet the healthcare needs of their beneficiaries. The iEHR program is developing accelerators and plans to deliver capabilities in phases. iEHR Accelerators • The DoD and VA established the iEHR Data Federation accelerators to execute federation of clinical health data to improve efficiency and accuracy of communication between both departments. -- The Data Federation accelerator is designed to achieve data interoperability within the DoD and VA healthcare systems and present patient data to doctors and clinicians using an enhanced Joint Legacy Viewer (JLV) web presentation system to retrieve information from disparate healthcare systems in real time for presentation in a web browser. The iEHR program has developed threshold and objective designs for data federation in parallel. ▪▪ The threshold solution utilizes the jMeadows web service and a Common Information Infrastructure Framework (CIIF) Terminology Service to map DoD and VA terminology to a common set of terms. Doctors and clinicians, using JLV, will view aggregated DoD and VA patient data presented with common terms. The threshold solution is designed for a limited set of users and provides the initial capability for data interoperability. ▪▪ The objective solution will provide enterprise-scalable Data Management Services (DMS) of which jMeadows will be a part, a data caching system to improve CIIF performance, and enhancements to existing access and identity management services. -- TOL Blue Button application uses the TOL enterprise architecture, which is comprised of a web-based application and Oracle database server system. Blue Button will enable authorized beneficiaries to download their DoD medical record in Healthcare Information
Activity iEHR Accelerators • The IPO has defined a threshold and objective Data Federation accelerator architecture to be delivered in December 2013 and June 2014, respectively. -- For the threshold architecture, the iEHR program plans to deliver seven normalized data domains (medication, laboratory, immunization, vitals, documentation/notes, allergies, and problem lists) via JLV, using jMeadows, in December 2013.
58 iEHR
Technology Standard Panel (HITSP) C32 format to a device of their choosing, such as a thumb drive or mobile phone. -- The IPO is developing a Med-COI accelerator to create a medical network that meets both DoD and VA security requirements. Med-COI will permit connected facilities to simultaneously connect to both the medical enclave and to external sites using a secure virtual private network over the Non-secure Internet Protocol Network. iEHR Increment 1 and Other Development • SSO-CM is intended to provide a Single Sign-on capability for multiple applications via the users’ Common Access Card and allow fast user switching between applications while keeping the patient and associated clinical information in context. The IPO designed and developed SSO-CM using the capabilities of the following commercial off-the-shelf (COTS) products: -- Citrix Password Manager for SSO -- Carefx Fusionfx for CM • DTC/DTE is intended to provide a federated testing environment to support software development and testing. Firewalls are utilized to create required separation between the .com and .mil environments. Both environments will be connected to external systems through gateways. • SOA Suite/ESB implements the International Business Machines WebSphere Message Broker, a COTS product, which provides SOA enabling capabilities. It is intended to provide the transport for message exchange among the DoD MHS, the VA EHR, and associated information management systems. Mission The DoD and VA will use the iEHR program to implement an EHR that both organizations can use to meet the healthcare needs of their beneficiaries and the clinicians providing the healthcare. Major Contractors • Hawaii Resource Group – Honolulu, Hawaii • Harris – Leesburg, Virginia • General Dynamics Information Technology – Fairfax, Virginia • Technatomy – Fairfax, Virginia • MITRE – McLean, Virginia • Deloitte – Alexandria, Virginia
-- The objective architecture will add the DMS and implement the remaining translation service capabilities. As more data domains are normalized using industry standards, they will be moved out of jMeadows and into the DMS. The IPO is also enhancing the JLV to provide users a graphic user interface to view normalized data provided by the Data Federation accelerator. -- DT began on November 11 and is scheduled to complete in December 2013.
DOD PROGRAMS • The iEHR Program Office conducted Blue Button Phase I DT in April 2013. Blue Button Phase II DT began on November 19 and is scheduled to be completed in December 2013 • Med-COI DT completed in October 2013. • An OA of the iEHR accelerators is currently scheduled for January 6 – 17, 2014. A second OA will be conducted later in FY14, once the accelerators have completed all phases of development. iEHR Increment 1 and Other Development • The U.S. Army Medical Department Board planned to conduct an SSO-CM OA in November 2012, but testing was delayed due to system defects and site configuration problems. -- The Joint Medical Information Systems DT team conducted four DT events to verify the correction of defects found in November 2012. -- SSO-CM development will continue with a new completion date of September 2014. • DOT&E has added DHMSM to the test and evaluation oversight list to test the full, end-to-end capabilities of the new “core” capability with iEHR architecture, accelerators, and SSO-CM capabilities. • The IPO conducted an SOA Suite/ESB combined contractor/government DT from February 21 through March 8, 2013. • The SOA Suite/ESB was deployed to San Antonio and Hampton Roads areas and a regional deployment at the Defense Enterprise Computing Center Montgomery. Assessment iEHR Accelerators • Blue Button, Data Federation, and Med-COI are being developed in multiple phases. Only the first phase of each accelerator is to be tested and deployed to meet the December 2013 deadline. Subsequent phases of the accelerators will be consolidated under one program for completion by September 30, 2014. • Data Federation accelerator DT of the threshold architecture identified a number of problems with terminology mapping and data normalization, which are in the process of being corrected and patched. The OA planned for January 2014 will assess these areas with doctors and clinicians in an operational environment. • Blue Button Phase I successfully completed testing in April 2013 and was deployed. Blue Button Phase II DT results were not available to include in this report. • Med-COI DT was successfully completed in October 2013. The Program Executive Officer for the DoD Healthcare
Management System recently decoupled Med-COI from other iEHR accelerators and it is not clear how this capability will fit into the larger iEHR architecture. iEHR Increment 1 and Other Development • SSO-CM underwent development and testing in FY13, but testing revealed a significant number of defects that persisted over the reporting period. An OA was attempted in November 2012; however, during site setup, the program manager delayed testing citing numerous network challenges, clinical application problems, incompatible virtual architectures, and content management defects as the cause of the delay. -- Four DT events identified a total of 32 defects: 14 in the initial test, 7 in the first System Integration Test (SIT-1), 7 in SIT-2, and 4 in SIT-3. At the end of SIT-3, 13 defects remained open. Following SIT-3, the program manager further delayed the OA. -- DOT&E rejected the OA plan because it did not demonstrate that the SSO-CM systems would work with, and not interfere with, the IPO's primary deliverables, which are the DoD and VA iEHR accelerators. • The DTC/DTE was to be used in support of accelerator development and testing; however, technical problems with IP addresses, ports, and external interfaces have prevented its use. The IPO plans to continue DTC/DTE development until full deployment in December 2013. • SOA Suite/ESB combined DT was successfully conducted using test tools to simulate operational traffic. SOA Suite/ ESB will use adapters to connect to external systems; however, no adapters were planned as part of the initial deployment. Operational testing will be conducted once adapters are available to allow external applications, services, and consumers to connect to the SOA Suite/ESB. Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. 1. The January 2014 OA should include VA, DoD, and Private Doctor and Clinician ratings of normalized patient data in cases for which VA and DoD use different terms. 2. The Program Executive Officer for the DoD Healthcare Management System should work with DOT&E to develop an adequate plan for an operational assessment of the SSO‑CM functionality and the impact on Health Data Sharing and Interoperability.
iEHR 59
DOD PROGRAMS
60
DOD PROGRAMS
Joint Biological Tactical Detection System (JBTDS) Executive Summary • The Joint Biological Tactical Detection System (JBTDS) program conducted developmental testing from January 2012 to September 2013 on three prototype systems to assess the technical maturity of the prototype systems and to identify risk in meeting operational requirements. • Based on demonstrated performance and modeling, the Program Office extrapolated that two of the three JBTDS prototype systems are expected to detect, collect, and identify some threat representative releases of biological warfare agents estimated to cause high casualty rates if the dissemination point is close to the detector. • The program faces significant challenges in meeting Service requirements for: - One false alarm a week for a networked array of detectors - Networking an array of JBTDS to support remote operations in a tactical environment • Developmental testing of JBTDS prototype systems indicates that reliability growth will be required to meet the Service reliability requirements for the system. System • The JBTDS is designed to be a man-portable, battery-operated system comprised of a detector that alarms to the presence of a biological agent threat cloud, a collector that takes an air sample, and an identifier to analyze the sample. The Marine Corps intends to employ the collector and identifier without the detector. • The Army, Navy, and Air Force intend to deploy the detector and collector in an array around the area of operations to maximize the probability of encountering a biological warfare cloud. JBTDS detectors will have a local alarm and be networked to an operational command center. • JBTDS is intended to augment existing biological detection systems, such as the Joint Biological Point Detection System, when networked.
decisions, enable medical planning and treatment, and mitigate the consequences of biological attacks. • Units will employ the system during periods of increased biological threat, and during routine biological surveillance operations when integrated in the protection capabilities for fixed sites and forward operating bases. Major Contractors • Battelle Memorial Institute ‒ Columbus, Ohio • Camber Corporation ‒ Edgewood, Maryland • ITT Corporation ‒ Abingdon, Maryland
Mission • Chemical, biological, radiological, and nuclear personnel will use JBTDS to support time-sensitive force protection
Activity • The JBTDS program conducted developmental testing from January 2012 to September 2013 on three prototype systems to assess the technical maturity of the prototype systems and to identify risk in meeting operational requirements. Army personnel at the Edgewood Chemical and Biological Center conducted developmental testing of the prototype identifiers
using liquid biological agent from June to December 2012. Army personnel at the Aberdeen Test Center, Maryland, conducted electromagnetic environmental effects testing from June to September 2012. • Army personnel at Dugway Proving Ground, Utah, conducted chamber testing of the prototype systems to characterize the
JBTDS 61
DOD PROGRAMS performance of the integrated detector, collector, and identifier against four classes of biological warfare agents from January to May 2013. Army personnel conducted detector testing to determine if the presence of common battlefield interferents would cause the detectors to false alarm. • The Program Office tested the prototype systems’ capability to be remotely operated over a network and the propensity of the detectors to alarm when no biological warfare agent threat is present. Testing was conducted at Edgewood, Maryland, from April 17 through May 14, 2012. Aberdeen Test Center personnel conducted tests on the capability of the prototype systems to operate in an extreme operating environment June 18 through September 7, 2012. • The Program Office funded the Institute for Defense Analyses to use demonstrated performance data to model the ability of a unit equipped with the JBTDS to mitigate casualties from a range of biological warfare agent attacks. Assessment • Based on demonstrated performance and modeling, the Program Office extrapolated that two of the three JBTDS prototype systems are expected to detect, collect, and identify some threat representative releases of biological warfare agents estimated to cause high casualty rates if the dissemination point is close to the detector. • The JBTDS prototype systems demonstrated the required 90 percent probability to detect 3 of 4 biological warfare agent classes and 5 of 8 agent preparations at concentrations expected to cause significant casualties. • The JBTDS prototype collection technology is mature and in operational use today. • Prototype identification technologies demonstrated the capability to identify three of the four agent classes at concentrations that are estimated to result in significant casualties. For one of the three agent classes, the capability was dependent upon how the agent was prepared. • The program faces a significant challenge in meeting the Service-defined requirement of one false alarm a week for a
62 JBTDS
networked array of detectors. This equates to 168 hours mean time between detector false alarms for a networked array of multiple JBTDSs. JBTDS prototype systems demonstrated a mean time between detector false alarms between 30 and 97 hours for a single system. • The program faces a substantial challenge in networking an array of JBTDSs to support remote operations in a tactical environment. Two of the JBTDS prototype systems demonstrated basic capability to send alert notifications to a base station and remotely trigger collection of an aerosol sample using a commercial wireless network during testing in Edgewood, Maryland. • Developmental testing of JBTDS prototype systems indicates that reliability growth will be required to meet the Service operational reliability requirements for the system. Combined detector and collector prototype reliability ranged from 176 to 531 hours at the 80 percent lower confidence bound in comparison to the operational requirement of 480 hours. Identifier prototype reliability ranged from 109 to 202 hours at the 80 percent lower confidence bound. The operational requirement is 150 hours. Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. The program should: 1. Invest in technology development to improve detector component sensitivity and false alarm rate. 2. Develop or leverage development of more sensitive identification technologies for two of the three agent classes. 3. Begin operational network capability development and testing early in the Engineering and Manufacturing Development phase of the program. 4. Institute a reliability growth program after award of an Engineering and Manufacturing Development contract.
DOD PROGRAMS
Joint Information Environment (JIE) Executive Summary • The Chairman, Joint Chiefs of Staff (CJCS) published a White Paper on the Joint Information Environment (JIE) in January 2013 and the Deputy Secretary of Defense published implementation guidance for JIE in May 2013. • DOT&E subsequently placed the JIE framework on test and evaluation oversight in August 2013. • In September 2013, the DoD Chief Information Officer (CIO) published implementation guidance for JIE to realign the structure and operations of DoD Information Technology systems and services. • To date, no documented testing of JIE infrastructure, components, or operational concepts has been conducted or provided for DOT&E review. Capability and Attributes • The JIE is envisioned to be a shared and upgraded information technology infrastructure, enterprise services, and security architecture intended to achieve full-spectrum superiority, improve mission effectiveness, increase security, and realize efficiencies. The CJCS White Paper lists the enabling characteristics of JIE to include: - Transition from Network Centric to Data Centric solutions - Rapid delivery and use of integrated cloud services - Interdependent information environment providing real-time cyber situational awareness - Scalability and flexibility - Secure, resilient, and consolidated framework - Common standards and operational tactics, techniques, and procedures - Improved and dynamic identity and access management tools • The DoD intends to achieve these goals via several initiatives including: - Implementing a Single Security Architecture across a federated network structure, standardized access management, and enterprise services such as e-mail - Consolidating common services and applications into centralized data centers both regionally and globally, which will use a common computing model for virtualization and security services - Using or upgrading existing infrastructure to support the improved functionality • The DoD intends to achieve reductions in data centers, operations centers, timelines for procurement of services and equipment, and manning requirements. • JIE is intended to provide DoD information and network services to fixed, deployed, and mobile users. The overarching concept is to develop a network architecture with flexibility to support existing and future capabilities
and allow the introduction of improved technologies, such as Multi‑Protocol Label Switching. • JIE-related infrastructure is to be repurposed or acquired from a variety of sources, both government and commercial. The government integrator is the Defense Information Systems Agency (DISA). Current plans are to implement a first increment in the European Theater, building on the network consolidation efforts already underway. This will be followed by subsequent capability upgrades leading to second and third increments across the DoD. • JIE is not a program of record. Activity • CJCS published a White Paper on JIE in January 2013. • The Deputy Secretary of Defense published implementation guidance for JIE in May 2013. • In August 2013, DOT&E placed the JIE framework on test and evaluation oversight. • In a September 2013 letter to the DoD leadership, the DoD CIO published JIE implementation guidance to fundamentally realign and restructure how the Department’s Information Technology networks, systems, and services are constructed, operated, and defended. • DOT&E met with senior DISA leadership to discuss test and evaluation of JIE and establish expectations for oversight. DISA described plans to test the smaller components and devices to standard, but plans to evaluate the overall system are unclear. Planning is in progress for an operational review in March 2014 of JIE Increment 1 (European). • To date, DOT&E has not received any formal test documentation, and available test strategy documents are high-level and non-specific to the events currently planned. DOT&E has requested that DISA: - Provide a test plan for the March 2014 event for DOT&E review and approval - Include DOT&E in the weekly JIE updates to the CIO to enable collaboration and test planning - Prepare a long-range test strategy for test and evaluation of JIE for DOT&E review and approval • DOT&E will assist DISA with any documents, lessons learned, or templates developed during Information Assurance and Interoperability assessments during exercises (reported separately). Assessment • No test data are available at this point. Areas of interest for upcoming assessments will include: - Validation of component performance where new technologies or designs are implemented
JIE 63
DOD PROGRAMS - Evaluation of services provided (including service-level agreements, where appropriate) - Effectiveness of the framework to securely provision information services to key missions and tasks - Validation of re-hosted, virtualized DoD applications - Integration with cyber-range nodes and other networked test capabilities Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. The DoD CIO and Director of DISA should:
64 JIE
1. Prepare and provide test schedules and plans for DOT&E review at the earliest opportunity, and plan for appropriate implementation milestones to allow for fielding decisions based on review and correction of any issues identified during operational test events. 2. Provide a test plan for the March 2014 evaluation event of JIE Increment 1 to DOT&E for review and approval no later than early February 2014. 3. Provide a long-range test strategy for events to occur after March 2014 to DOT&E for review and concurrence. 4. Develop a test and evaluation strategy for end-to-end operational test and evaluation of JIE infrastructure.
DOD PROGRAMS
Joint Warning and Reporting Network (JWARN) Executive Summary • The Army Test and Evaluation Command conducted the Joint Warning and Reporting Network (JWARN) Increment 1 Modernization Operational Assessment test event in a laboratory setting at the Central Technical Support Facility at Fort Hood, Texas, from July 25 – 31, 2013. • During the operational assessment, DOT&E observed the following: - Network instability and the immaturity of Army Command Web, on which the JWARN Increment 1 Modernization application resides, adversely affected the ability of JWARN operators to warn at-risk company-level units. - JWARN operators were able to send warning reports to notional at-risk battalions and the brigade in time to take protective action when the units were more than 10 kilometers downwind from the hazard release location. System • JWARN is a joint automated chemical, biological, radiological, and nuclear (CBRN) warning, reporting, and analysis software tool that resides on joint and Service command and control systems including the Global Command and Control System (GCCS) – Army, GCCS – Joint, GCCS – Maritime, and Command and Control Personal Computer/ Joint Tactical Common Workstation. • JWARN Increment 1 Modernization is a web-application on the Army Command Web as part of the Command Post Computing Environment. There is also a JWARN version that operates on a stand-alone computer. • JWARN software automates the NATO CBRN warning and reporting process to increase the speed and accuracy of information sharing to support force protection decision making and situational awareness.
Activity • DOT&E approved the JWARN Increment 1 Test and Evaluation Master Plan (TEMP) Annex on July 10, 2013. • DOT&E approved the test plan for the JWARN Increment 1 Modernization Operational Assessment on July 9, 2013. • The Army Test and Evaluation Command conducted the JWARN Increment 1 Modernization Operational Assessment test event in a laboratory setting at the Central Technical Support Facility at Fort Hood, Texas, from July 25 – 31, 2013, in accordance with the DOT&E-approved test plan. Assessment • During the laboratory-based operational assessment, the immaturity of Army Command Web and network instability adversely affected the JWARN web application operators’
• JWARN uses the common operating picture map of the host command and control system or computing environment to display the location of CBRN events and the predicted or actual location of hazards to support the Commanders’ situational awareness and response capability. Mission JWARN operators in command cells provide CBRN force protection, battlefield management, and operational planning by predicting chemical, biological, and nuclear hazard areas based on sensor and observer reports, identifying affected units and operating areas, and transmitting warning reports. Major Contractor Northrop Grumman Mission Systems – Orlando, Florida
ability to provide timely warnings to notional at-risk company- level units, which rely on graphical depictions of the warning on the common operating picture display. • The JWARN web application operators demonstrated the ability to provide timely warning to notional at-risk units located more than 10 kilometers downwind from the initial hazard. For notional units closer than 10 kilometers, the JWARN web application operators provided timely warnings to 42 percent of the at-risk units. This performance is consistent with prior versions of JWARN employed on Service command and control systems. • The JWARN web application reliability failures during the operational assessment, to include intermittent unresponsiveness, delayed transmission of e-mail messages,
JWARN 65
DOD PROGRAMS and intermittent ability to generate high-fidelity hazard prediction plumes using the Joint Effects Model, did not prevent the timely warning of at-risk units. • Training provided to operators by the Army was not adequate for operators to consistently provide accurate situational awareness during multiple simultaneous attacks and when information was received from more than one observer report. Recommendations • Status of Previous Recommendations. The Navy and Program Office have addressed DOT&E’s previous recommendations.
66 JWARN
However, the program manager still needs to validate and field to the Services the computer-based training for JWARN on GCCS–Joint and GCCS–Maritime. • FY13 Recommendation. 1. The Program Office should develop, validate, and field computer-based training for JWARN on GCCS – Army and the JWARN web application on the Army Command Web that includes basic to advanced scenario exercises to increase operator skills and provide sustainment training.
DOD PROGRAMS
Key Management Infrastructure (KMI) Executive Summary • The Key Management Infrastructure (KMI) Program Management Office (PMO) and Joint Interoperability Test Command (JITC) conducted an FOT&E in accordance with a DOT&E-approved test plan from January 14 through February 1, 2013, which included infrastructure and 13 separate Service and agency locations across the United States. • In April 2013, DOT&E reported that KMI significantly improved from the IOT&E and is now operationally effective, suitable, secure, and remains interoperable; however, the FOT&E demonstrated continued problems with token reliability and revealed some minor shortfalls in system availability and sustainment. Transition procedures improved but need further refinement. • Subsequent to the DOT&E report, the DoD Chief Information Officer published the KMI Acquisition Decision Memorandum on June 19, 2013, approving full-rate production and deployment of Spiral 1 to DoD Services and agencies. System • KMI is intended to replace the legacy Electronic Key Management System (EKMS) to provide a means for securely ordering, generating, producing, distributing, managing, and auditing cryptographic products (e.g., asymmetric keys, symmetric keys, manual cryptographic systems, and cryptographic applications). • KMI Spiral 1 consists of core nodes that provide web operations at a single site operated by the National Security Agency, as well as individual client nodes distributed globally to provide secure key and software provisioning services for the DoD, intelligence community, and agencies. Spiral 2 will provide improved capability through software enhancements to the Spiral 1 baseline. • KMI combines substantial custom software and hardware development with commercial off-the-shelf computer components. The custom hardware includes an Advanced Key Processor for autonomous cryptographic key generation and a Type 1 user token for role-based user authentication. The commercial off-the-shelf components providing user operations include a client host computer, High Assurance Internet Protocol Encryptor (KG-250), monitor, keyboard, mouse, printer, and barcode scanner.
Activity • The KMI PMO and JITC conducted an FOT&E in accordance with a DOT&E-approved test plan January 14 through February 1, 2013, which included
Mission • Combatant Commands, Services, DoD agencies, other Federal Government agencies, coalition partners, and allies will use KMI to provide secure and interoperable cryptographic key generation, distribution, and management capabilities to support mission-critical systems, the Global Information Grid, and initiatives such as Cryptographic Modernization. • Service members will use KMI cryptographic products and services to enable security services (confidentiality, non‑repudiation, authentication, and source authentication) for diverse systems such as Identification Friend or Foe, GPS, Advanced Extremely High Frequency Satellite System, and Warfighter Information Network – Tactical. Major Contractors • Leidos (formerly SAIC) – Columbia, Maryland (Spiral 2 Prime) • General Dynamics Information Assurance Division – Needham, Massachusetts (Spiral 1 Prime) • BAE Systems – Linthicum, Maryland • L3 Systems – Camden, New Jersey • SafeNet – Belcamp, Maryland • Praxis Engineering – Annapolis Junction, Maryland
infrastructure and 13 separate Service and agency locations across the United States. • DOT&E issued a classified FOT&E report in April 2013.
KMI 67
DOD PROGRAMS • Subsequent to the DOT&E report, the DoD Chief Information Officer published the KMI Acquisition Decision Memorandum on June 19, 2013, approving full-rate production and deployment of Spiral 1 to DoD Services and agencies. • The PMO and Operations Manager completed the facility Uninterruptable Power Supply (UPS) expansion in July 2013 to support the resiliency of KMI Storefront (which provides backend processing for generation of cryptographic products; also called core nodes) and redundant systems. • The PMO and JITC are updating the Spiral 2 Test and Evaluation Master Plan (expected in March 2014) that will describe the test and evaluation strategy to support planned program activities to support a Full Deployment Decision by April 2017. Assessment • KMI is operationally effective. The PMO and Networking Tiger Team corrected EKMS-to-KMI transition problems previously encountered in the 2012 IOT&E. Once accounts transitioned, KMI supported required operational tasks with no difficulties in product key ordering and account management, and Service and agency user feedback was positive regarding KMI’s effectiveness versus the legacy EKMS. • KMI is operationally suitable; however, the FOT&E demonstrated continued problems with token reliability and revealed some minor shortfalls in system availability and sustainment. Transition procedures improved but still need further refinement. - While the PMO conducted extensive analysis to determine the underlying token failure modes, the KMI tokens redesigned to correct the problems were not available for the FOT&E. - The program’s custom-designed Advanced Key Processor performed well and continued to meet reliability expectations. - The facility UPS was inadequate to support the KMI Storefront and redundant systems, contributing to availability problems observed during the FOT&E that the PMO subsequently resolved in July 2013.
68 KMI
- KMI and Service-level help desk support was adequate in providing required user support during transition, routine activities, and subsequent mission operations. - Configuration management procedures matured significantly and are now adequate for operations. - The Configuration Control Board efficiently prioritized discrepancy reports logged against the system and approved build changes. • KMI is secure. The detailed Information Assurance assessment results are classified and can be found in the annex to the April 2013 DOT&E report. • The discussion of continuity of operations planning and facility preparations is classified and can be found in the April 2013 DOT&E report. • KMI remains interoperable. The system continued to successfully exchange critical information with all external interfaces (fill devices, end cryptographic units, and EKMS) accurately and without failure during the FOT&E. Recommendations • Status of Previous Recommendations. The KMI PMO satisfactorily addressed the five previous recommendations. • FY13 Recommendations. The KMI PMO should: 1. Verify increased KMI token reliability through a combination of laboratory and operational testing with automated data collection from system logs for accurate reliability and usage analysis. 2. Stress test the facility’s UPS for the Storefront systems to include pertinent nodes and execute routine planned failover tests periodically to ensure necessary data synchronization between redundant equipment. 3. Complete the Spiral 2 Test and Evaluation Master Plan update to support future operational testing by March 2014. 4. Follow the recommendations for the KMI continuity of operations plan listed in the classified April 2013 DOT&E report.
DOD PROGRAMS
Mine Resistant Ambush Protected (MRAP) Family of Vehicles (FoV) Executive Summary • DOT&E delivered the classified Live Fire Test and Evaluation Assessment of the Mine Resistant Ambush Protected (MRAP) All-Terrain Vehicle (M-ATV) with Underbody Improvement Kit (UIK) to Congress in March 2013. The UIK-equipped M-ATV provided protection beyond its required level and is a significant improvement over the baseline M-ATV. • The Services will retain approximately 43 percent (12,092) of the 27,701 MRAP Family of Vehicles (FoV) produced. • The Special Operations Forces (SOF) M-ATV User Demonstration focused on verifying fixes to deficiencies identified in the SOF M-ATV IOT&E. The results from the User Demonstration indicate that the most significant deficiencies were not resolved. - The crews operating the SOF M-ATV continued to possess poor situational awareness due to the small rear windows and the limited field-of-view of the Common Remotely Operated Weapon Station II (CROWS II). - No improvements were made to the limited field-of-view of the CROWS II for target acquisition. - The crews operating the CROWS experienced the same weapon-firing and ammunition jamming failures identified during the IOT&E, which degraded the vehicle’s reliability. • The SOF M-ATV had improved vehicle acceleration while maneuvering over primary, secondary, and cross-country terrain during the User Demonstration. The addition of a muffler has reduced the loud aural signature. System • The MRAP program is a FoV designed to provide increased crew protection and vehicle survivability against current battlefield threats, such as IEDs, mines, and small arms. The MRAPs are employed by units in current combat operations in the execution of missions previously accomplished with the High Mobility Multi-purpose Wheeled Vehicle. This report covers four MRAP variants: - M-ATV Capability Set 13 (CS-13) Point of Presence (PoP) with UIK - M-ATV CS-13 Soldier Network Equipment (SNE) with UIK - SOF M-ATV with UIK - NAVISTAR Dash with MaxxPro Survivability Upgrade (MSU) • The M-ATV with UIK is designed to provide improved underbody blast protection. • The CS-13 M-ATV PoP vehicle is integrated with the Warfighter Information Network – Tactical (WIN-T) Increment 2 communications networking equipment and
mission command applications. The vehicle is designed to provide command and control on-the-move capability at division, brigade, and battalion levels.
MRAP FoV 69
DOD PROGRAMS • The CS-13 M-ATV SNE vehicle is integrated with the WIN-T Increment 2 communications networking equipment and mission command applications. The vehicle is designed to provide command and control on-the-move capability down to the company level. • United States Special Operations Command required modifications to the Army M-ATV to support SOF missions. The modifications included five passenger positions including a gunner, protection for the cargo area, and rear area access. • The Dash variant with MSU is designed to provide improved underbody blast protection.
Activity MRAP FoV • In anticipation of the end of major hostilities in the Afghanistan theater, the Services determined their enduring force requirements and divestment plans for the MRAP FoVs. M-ATV • The program developed, procured, and integrated the Army CS-13 network equipment and mission command applications onto M-ATV vehicles to support brigades deploying to Afghanistan. • The M-ATV CS-13 PoP and SNE vehicles participated in the WIN-T Increment 2 FOT&E during Network Integration Evaluation (NIE) 13.2 at Fort Bliss, Texas, in May 2013. • The Army conducted a blast test on the CS-13 equipped M-ATV to assess what impact the mission equipment had on the vulnerability mitigation features of the M-ATV. SOF M-ATV • United States Special Operations Command completed a User Demonstration of the SOF M-ATV at Yuma Proving Ground, Arizona, in June 2013 to verify fixes to deficiencies found during the SOF M-ATV IOT&E. • The program conducted a design review of an M-ATV Tube‑Launched, Optically-Tracked, Wire-Guided (TOW) missile variant and contracted for two engineering prototype M-ATV TOW variants to be developed and tested. Dash • The Army completed live fire testing of the Dash equipped with the MSU kit. Assessment MRAP FoV • The Services will retain approximately 43 percent (12,092) of the 27,701 MRAP FoV produced. M-ATV • DOT&E delivered the classified Live Fire Test and Evaluation Assessment of the M-ATV with UIK to Congress in March 2013. The UIK-equipped M-ATV
70 MRAP FoV
Mission Multi-service and special operations units equipped with the MRAP FoV conduct mounted patrols, convoy patrols, convoy protection, reconnaissance, and communications, as well as command and control missions to support combat and stability operations in highly restricted rural, mountainous, and urban terrain. Major Contractors • Oshkosh Corporation – Oshkosh, Wisconsin • Navistar Defense – Warrenville, Illinois
provided protection beyond its required level, and is a significant improvement over the baseline M-ATV. • Based on the WIN-T Increment 2 FOT&E, the CS-13 M-ATV PoP and SNE vehicles provide an increased operational capability over the WIN-T NIE-configured M-ATV. -- The addition of the Smart Display Unit and rear‑mounted Multi-Domain Atlas platform contributed to increased situational awareness between commander and crew. -- The Multi-Domain Atlas and integrated bridge software allowed the commander to distribute tasks to the crew reducing his workload. • During the WIN-T Increment 2 FOT&E, the NIE‑configured M-ATV experienced numerous air conditioner, water pump, and water pump belt failures due to the vehicles running continuously during operations to provide power to WIN-T and other communications equipment. • The integration of the CS-13 mission equipment onto the UIK‑equipped M-ATV does not adversely affect the performance of the vulnerability reduction features of the M-ATV during an underbody blast event. SOF M-ATV • The results from the SOF M-ATV User Demonstration indicate that the most significant deficiencies were not resolved. The crews operating the SOF M-ATV continued to possess poor situational awareness due to the small rear windows and limited field-of-view of CROWS II. The program did not make improvements to the limited field- of- view of the CROWS II for target acquisition. The crews operating the CROWS experienced the same weapon-firing and ammunition jamming failures identified during the IOT&E, which degraded the vehicle’s reliability. • The SOF M-ATV had improved vehicle acceleration while maneuvering over primary, secondary, and cross-country terrain during the SOF M-ATV User Demonstration. The addition of a muffler reduced the loud aural signature.
DOD PROGRAMS Dash • The MSU-equipped Dash provides increased occupant protection over the baseline Dash. LFT&E of the MSU‑equipped Dash revealed problems with kit integration that the program will address during reset of the vehicles. Testing and evaluation of solutions to address these problems are ongoing. Recommendations • Status of Previous Recommendations. The program is making progress implementing the previous recommendations.
• FY13 Recommendations. 1. The CROWS Program Office should investigate and determine the cause of CROWS weapon-firing failures and ammunition jamming problems and conduct additional operational testing of CROWS on tactical vehicles to verify fixes. 2. The program should improve the visibility of the SOF passenger by installing larger rear windows in SOF M-ATV as previously recommended.
MRAP FoV 71
DOD PROGRAMS
72
DOD PROGRAMS
Next Generation Diagnostics System (NGDS) Executive Summary • DOT&E approved the Next Generation Diagnostics System (NGDS) Increment 1 Deployable Component Test and Evaluation Strategy (TES) on December 12, 2012. The TES addresses the strategy to support selecting a single vendor to procure common pathogen diagnostic and identification systems and the development of clinical and environmental biological warfare agent diagnostic and identifications assays. • The program conducted an early operational assessment and developmental testing from April to September 2013 on three commercial off-the-shelf (COTS) systems from three vendors: BioFire Diagnostic Incorporated, Focus Diagnostics Incorporated, and IQuum, Incorporated. • Early operational testing demonstrated that the commercial systems have the capability to support rapid analysis of clinical samples to support diagnostic and medical treatment decisions in a field environment. System • The NGDS Increment 1 Deployable Component will be an analytical system capable of detecting and identifying the presence of nucleic acids of biological warfare agents and infectious diseases. It will be comprised of: - A liquid sample analytical instrument with an internal or external computer - Software - Consumable assays and reagents - Sample preparation protocols and equipment - A shipping container - Power management equipment - Operator-level spares - Preventive maintenance tools, training, and manuals • The Services intend to use the NGDS Increment 1 Deployable Component in existing microbiology laboratories equipped with common laboratory support equipment such as Class II Bio Safety Cabinet, refrigerator, freezer, level
Activity • The Joint Program Executive Officer for Chemical and Biological Defense (JPEO-CBD) approved the NGDS Increment 1 Deployable Component Milestone A on March 27, 2012. • On November 5, 2012, the JPEO-CBD approved the NGDS Increment 1 Deployable Component program strategy to lead an integration effort with the Common Analytical Laboratory System and the Joint Biological Tactical Detection System
work surfaces, line power sources, lighting, and appropriately trained personnel. Mission Trained laboratory personnel will use the NGDS Increment 1 Deployable Component to identify biological warfare agents and infectious diseases in clinical specimens (e.g., blood, sputum, stool, urine, nasopharyngeal swabs, and environmental samples) to provide information to: • Support clinical diagnosis • Mitigate the impact of biological warfare attacks and endemic infectious disease • Support Force Health Protection decision making • Augment situational awareness Major Contractors • BioFire Diagnostics Incorporated – Salt Lake City, Utah • Focus Diagnostics Incorporated – Cypress, California • IQuum, Incorporated – Marlborough, Massachusetts
program to find a common materiel solution for pathogen diagnostics and pathogen identification. • DOT&E approved the NGDS Increment 1 Deployable Component TES on December 12, 2012. • The program awarded three contracts to provide COTS systems for competitive prototyping on February 28, 2013. • DOT&E approved the NGDS Increment 1 Deployable Component early operational test plan on April 1, 2013.
NGDS 73
DOD PROGRAMS • The U.S. Army Medical Department Board and the Air Force Medical Evaluation Support Activity conducted an early operational test April 16 – 30, 2013, at Camp Bullis, Texas, in accordance with the DOT&E-approved operational test plan. • The program conducted competitive prototype developmental testing of the candidate COTS systems from May through September 2013 to support selection of a single vendor in January 2014. Assessment • Early operational testing demonstrated that COTS systems have the capability to support rapid analysis of clinical samples to enable diagnostic and medical decisions and treatment in a field environment. • The COTS systems demonstrated varying levels of automation, complexity, and time to prepare and analyze
74 NGDS
clinical samples during testing. Each system will require development of biological warfare agent assays and sample preparation processes for use in a field environment. • The COTS systems demonstrated operational reliability ranging from 40 to 243 mean runs between operational mission failure (OMF) at the 80 percent confidence level when operated by representative Soldiers, Sailors, and Airmen in a realistic field laboratory. The Services require a 94.4 percent probability of completing 5 analytical runs without experiencing an OMF, which translates to 86 mean runs between OMF. Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. None.
DOD PROGRAMS
Public Key Infrastructure (PKI) Executive Summary • DoD Public Key Infrastructure (PKI) Increment 2 provides a cryptographic capability for DoD members and others to access the Secret Internet Protocol Router Network (SIPRNet) securely and to encrypt and digitally sign e-mail. Increment 1, which provided the Non-secure Internet Protocol Router Network (NIPRNet) PKI infrastructure with controlled access using Common Access Cards (CACs), is complete. The PKI infrastructure provides a personal identification number-protected SIPRNet token for electronically identifying individuals and managing access to resources over globally dispersed SIPRNet nodes. Full implementation will enable authorized users and Non-Person Entity (NPE)-enabled devices (e.g., servers and workstations) to access restricted websites and enroll in online services. • The Joint Interoperability Test Command (JITC) conducted a combined FOT&E I and II in January 2013 on the SIPRNet environment to address suitability shortcomings discovered during the 2011 IOT&E and to evaluate preliminary Increment 2 Spiral 3 enhancements. The major suitability concerns cited in the IOT&E were not addressed in the FOT&Es and new findings were discovered including increased token failures in the field and inefficiencies in token management. However, the PKI Program Management Office (PMO) has taken steps to address these problems including changes to improve system stability. No completed operational testing to date confirms resolution of the effectiveness and suitability problems. • An Inventory Logistics System (ILS) for managing SIPRNet token stock at each issuance site was not effective for tracking tokens returned for reuse, was cumbersome to use, and does not provide the necessary functions to replace existing spreadsheet tracking mechanisms. The capability to track reused tokens requires significant redesign and development investments as well as adoption of taxing procedures currently not required for NIPRNet CACs, which are not reusable. Given budget constraints, the Services and agencies opted to rely on workarounds to track returned tokens and requested that remaining Increment 2 resources be reserved for higher priority capabilities, such as group and role-based tokens. The ILS is not part of the original PKI baseline and was developed to support the end-to-end logistics of token distribution and tracking since no common system across the Services and agencies exists on the SIPRNet. • The DoD Chief Information Officer directive requiring all SIPRNet users to be issued tokens was met for the initial target population. However, select user groups, including some DoD contractors, intelligence personnel, and users supporting tactical operations, have not yet received SIPRNet tokens. • Increment 2 was originally intended to provide infrastructure upgrades to support DoD’s transition to Internet Protocol
version 6 (IPv6), migration to stronger PKI algorithms, and to provide the flexibility needed to expand PKI usage in tactical environments. Due to lack of infrastructure readiness across the DoD networks, these areas will not be tested and evaluated as part of Increment 2. • The National Security Agency (NSA) Senior Acquisition Executive declared a PKI program significant change in September 2013 and a critical change in October 2013. System • DoD PKI is a critical enabling technology for Information Assurance. It supports the secure flow of information across the Global Information Grid as well as secure local storage of information. • DoD PKI provides for the generation, production, distribution, control, revocation, recovery, and tracking of public key certificates and their corresponding private keys. The private keys are encoded on a token, which is a credit-card sized smartcard embedded with a microchip. • DoD PKI is comprised of commercial off-the-shelf hardware and software and other applications developed by the NSA. - The Defense Enrollment Eligibility Reporting System (DEERS) and Secret DEERS provide the personnel data for certificates imprinted on NIPRNet CACs and SIPRNet tokens, respectively. - DoD PKI Certification Authorities for the NIPRNet and SIPRNet tokens reside in the Defense Information Systems Agency (DISA) Enterprise Service Centers in Oklahoma City, Oklahoma, and Mechanicsburg, Pennsylvania. • DISA and NSA are jointly developing DoD PKI in multiple increments. Increment 1 is complete and deployed on the NIPRNet. Increment 2 is being developed and deployed
PKI 75
DOD PROGRAMS in three spirals on the SIPRNet and NIPRNet to deliver the infrastructure, PKI services and products, and logistical support. Mission • Military operators, communities of interest, and other authorized users will use DoD PKI to securely access, process, store, transport, and use information, applications, and networks regardless of technology, organization, or location. • Commanders at all levels will use DoD PKI to provide authenticated identity management via personal identification number-protected CACs or SIPRNet tokens to enable DoD
Activity • DOT&E approved an Operational Assessment plan for the NPE capability in November 2012. However, the NPE technical solution has since evolved to support changes in operating constraints such as the need to support virtual web servers hosting multiple web sites. Furthermore, the PMO delayed the test indefinitely due to the lack of DoD policy defining the types of devices requiring DoD enterprise medium assurance certificates. • The PKI PMO and JITC, in accordance with a DOT&E- approved test plan, conducted a combined FOT&E I and II of the PKI Increment 2 from January 8 through February 1, 2013, to verify correction of system deficiencies discovered during the IOT&E in 2011 for Spirals 1 and 2, and to evaluate preliminary Spiral 3 enhancements, respectively. The FOT&Es were originally scheduled for 3QFY12 but were postponed due to system development delays. Furthermore, a stop-test in December 2012 resulted from systemic configuration management problems. • Delays in delivering the ILS capability for token ordering and shipping diverted resources and indirectly contributed to delays in the delivery of several key Spiral 3 capabilities, including the NPE and alternate token capabilities to support system administrator roles on the SIPRNet and NIPRNet. • In June 2013, JITC conducted a Level II user test to assess improvements to Certificate Authority user management functions. • In 4QFY13, DISA moved the PKI primary site from Chambersburg to Mechanicsburg, Pennsylvania, to address previous Information Assurance operational test findings. • The NSA Senior Acquisition Executive declared a PKI program significant change in September 2013 and a critical change in October 2013. Assessment • The DOT&E report in May 2013 found PKI’s Token Management System (TMS) and the ILS to be not operationally effective and not suitable. • PKI Increment 2, Spiral 3 is not operationally effective. The Spiral 3 enhancements assessed during the FOT&E I and II degraded existing capabilities and lowered efficiency by
76 PKI
members, coalition partners, and others to access restricted websites, enroll in online services, and encrypt and digitally sign e-mail. • Military network operators will use NPE certificates to create fully identified network domains, which will facilitate intrusion protection and detection. Major Contractors • General Dynamics Information Technology – Needham, Massachusetts (Prime) • 90Meter – Newport Beach, California • SafeNet – Belcamp, Maryland
increasing Service and agency workload. The initial Spiral 3 deployment of capabilities was intended to provide the following upgrades: (1) blacklisting of tokens, (2) auto-key recovery of private encryption keys escrowed by the core system, and (3) tracing of tokens to the original issuing Service or Local Registration Authority. Specific deficiencies include the following: - Blacklisting of tokens successfully identified tokens that should not be allowed reentry into the TMS but had the unintended consequence of lengthening the time to reformat valid user tokens because field operators lost the ability to reformat tokens returned for reuse. - The auto-key recovery capability allows end-users to recover private encryption keys through two methods: a self-service web-based capability and a third-party web‑based capability requiring Key Recovery Agent approval before granting access to encryption keys. However, a system limitation in the underlying commercial off-the-shelf product prevented users from recovering encryption keys to a token and subsequently using those keys to retrieve encrypted messages. - Users were not able to view all potential encryption certificates they have the ability to self-recover or request Key Recovery Agent assistance to recover on their behalf. The failure to deliver needed upgrades while maintaining critical operational functionality underscores immature configuration management problems and a need for processes that incorporate user feedback into capability design, development, test, and deployment. - The users expected the ILS to ease the burden of tracking and accounting for tokens but it added more steps without providing significant benefit. A verification of deficiencies test in May 2013, however, confirmed three ILS deficiencies were corrected to improve warehouse managers’ ability to leverage the ILS. - In summary, the Spiral 3 enhancements assessed during the FOT&E I and II were minor and instead of providing needed capability and enhancement, degraded existing capabilities, and lowered efficiency by increasing Service and agency workload.
DOD PROGRAMS • PKI Increment 2 is not operationally suitable. The end‑to‑end logistics processes continue to rely on manual, Service- and agency-specific methods for procuring, distributing, accounting, and tracking of tokens. Although over 45 bulk token formatters deployed across the DoD have helped increase token issuance rates, token reliability is not accurately tracked or reported and does not reflect user reports of growing failure rates in the field (as much as 15 percent). - The ILS was not designed to address logistics shortfalls identified in the IOT&E including token failure tracking and token statistics reporting, such as reporting of token issuance numbers by geographic region and Service affiliation. - The ILS has the potential to track shipments but was not effective for tracking tokens returned for reuse. It does not provide necessary functions such as the ability to ship between issuance sites and the ability to terminate bad tokens in a stack. - ILS procedures were cumbersome and confusing, and documentation and training were not adequate to improve usability. • Critical capabilities including the capabilities to generate group and role-based certificates and NPE device certificates (on both SIPRNet and NIPRNet) have been delayed. Sustainment plans for ILS after calendar year 2014 are uncertain further hampering the development of long-term Service and agency logistics processes for token ordering and shipping. Hosting the logistics and token management systems on the same network should improve manpower and usability concerns. However, due to budget constraints, the ILS development schedule has been suspended. • System reliability, availability, and maintainability of the core PKI infrastructure degraded since the IOT&E with two long unplanned downtimes (4 and 6 hours, respectively) and 12 days of system degradation as reported by users in the field. Configuration management problems persist, causing unannounced system degradations. The PMO has implemented changes to improve overall system reliability; however, these changes have not been independently verified through operational testing. • Increment 2 also included a requirement to support interoperability with coalition PKI. The SIPRNet PKI infrastructure uses a common root Certificate Authority to ease certificate validation path processing; however, partner nations must stand up their own certificate issuance capabilities in order to make interoperability a reality. These efforts are ongoing, but no operational testing on the SIPRNet has been conducted to date. • With continual changes to planned Spiral 3 capabilities, configuration management still lacks adequate processes for inserting user-prioritized capabilities and fixes into the field. Since the FOT&E I and II, the PMO has established a Configuration Control Board to address this issue; however, the process is still maturing. • Based on the results of the June 2013 user test, trusted agents can now perform pin resets in the field, thereby shifting a significant burden off of the registration authorities’ workflow.
•
•
•
•
While this assessment was largely positive, the new release again caused unwanted changes to existing capabilities: unanticipated changes in the user interface hampered registration authorities from viewing the full history of transactions performed on each card that underwent a pin reset. More rigorous developmental testing is required to identify problems so user workflow is not negatively affected by capability releases. The NPE development efforts have been halted to allow time for a thorough assessment of current mission requirements and changes in technology. Until a requirements review is conducted, no further development or testing is planned for Increment 2. A transition plan to support post-2014 operations and maintenance is still undefined between NSA, DISA, and the Services and agencies. Given the inability to address IOT&E and FOT&E I and II suitability shortcomings, the initial PKI Spiral 3 deployment remains not operationally suitable. The developmental test program processes and procedures directed in both the Test and Evaluation Master Plan and System Engineering Plan were not implemented, which has resulted in limited visibility into actual performance of the system prior to OT&E. Further testing will be necessary of the recently moved PKI primary site in Mechanicsburg, Pennsylvania, to assess improvements in Information Assurance, operational availability, system health and monitoring, and continuity of operations plans.
Recommendations • Status of Previous Recommendations. The PKI PMO satisfactorily addressed three of four recommendations from the FY12 Annual Report for Increment 2, Spirals 1 and 2. The recommendation for the PMO to establish a more realistic schedule for PKI development, delivery, and testing remains. • FY13 Recommendations. The PKI PMO should: 1. Address and independently verify fixes to operational effectiveness and operational suitability shortcomings in follow-on operational test activities. In particular, improve configuration management practices to ensure patches and releases do not impact critical mission functions and improve token failure tracking to more accurately reflect user experience. 2. Update the Test and Evaluation Master Plan in accordance with the redefined PKI Increment 2 acquisition strategy to prepare stakeholders for the remaining deliveries, resource commitments, and test and evaluation goals. 3. Create a transition plan defining roles and responsibilities for stakeholders once the program enters sustainment to support a smooth transition and ensure minimal impact to PKI operations. 4. Conduct a follow-on operational test of the new Mechanicsburg, Pennsylvania, PKI hosting site to assess improvements in Information Assurance, operational availability, system health and monitoring, and continuity of operations plans.
PKI 77
DOD PROGRAMS
78
DOD PROGRAMS
Theater Medical Information Program – Joint (TMIP-J) Executive Summary • The Army Test and Evaluation Command (ATEC) led a Multi‑Service Operational Test and Evaluation (MOT&E) of Theater Medical Information Program – Joint (TMIP-J) Increment 2 Release 2 (I2R2) from May 20 through June 13, 2013. All four Service Operational Test Agencies (OTAs) participated, as did the Army Medical Department Board, the Air Force Medical Evaluation Support Activity, Army Research Laboratory (ARL), Navy Information Operations Command (NIOC), and the Joint Interoperability Test Command (JITC). • TMIP-J I2R2 is operationally effective and operationally suitable for all four Services. TMIP-J I2R2 is survivable for the Army, Air Force, and Marine Corps, but not for the Navy. An Information Assurance (IA) defect related to the backup and restoration of the Maritime Medical Modules (MMM) application must be corrected before introducing TMIP-J to the Navy fleet. • Joint concerns that require prompt action include IA vulnerabilities; a logistics defect that can cause incorrect units of purchase, training on manual procedures for allergy entries, and testing of joint interfaces in the production environment once TMIP-J I2R2 is fielded. System • TMIP-J is a Major Automated Information System that integrates software from sustaining base medical applications into a multi-Service system for use by deployed forces. Examples of integrated applications include the theater versions of the Armed Forces Health Longitudinal Technology Application (AHLTA), Composite Health Care System, and Defense Medical Logistics Standard Support. • TMIP-J provides the following medical capabilities: - Electronic Health Record (EHR) - Medical command and control - Medical logistics - Patient movement and tracking - Patient data to populate the Theater Medical Data Store (theater database) and the Clinical Data Repository (Continental U.S. database)
Activity • ATEC led an MOT&E of TMIP-J I2R2 from May 20 through June 13, 2013, in accordance with the DOT&E-approved Test and Evaluation Master Plan and the detailed OTA test plan. The OTAs of all four Services participated, as did the Army Medical Department Board, the Air Force Medical Evaluation Support Activity, ARL, NIOC, and JITC.
• The Services provide their own infrastructure (networks and communications) and computer hardware to host the TMIP-J software. • TMIP-J consists of two increments. Increment 1 was fielded in 2003. Increment 2 is being developed in multiple incremental releases. Release 1 was fielded in 2009. I2R2 was the system under test during 2013. Mission • Combatant Commanders, Joint Task Force commanders, and their medical staff equipped with TMIP-J can make informed and timely decisions regarding the planning and delivery of health care services in the theater. • Military health care providers equipped with TMIP-J can electronically document medical care provided to deployed forces to support the continuum of medical care from the theater to the sustaining base. Major Contractors • SAIC – Falls Church, Virginia • Northrop Grumman – Chantilly, Virginia • Akimeka LLC, Kihei – Maui, Hawaii
• The MOT&E evaluated production-representative software at simulated deployment sites at Fort Detrick, Maryland (Army and Air Force); Camp Pendleton, California (Marine Corps); and aboard the USS Ronald Reagan (Navy).
TMIP-J 79
DOD PROGRAMS • ARL, the Army’s Threat System Management Office, and NIOC performed Red Team penetration testing to evaluate IA vulnerabilities. Assessment • TMIP-J I2R2 is operationally effective and operationally suitable for all four Services. TMIP-J I2R2 is survivable for the Army, Air Force, and Marine Corps, but not for the Navy. A defect related to the backup and restoration of the MMM application must be corrected before introducing TMIP-J I2R2 to the fleet. • Red Team penetration testing revealed that the system has a strong security posture when faced with cyber security threats from outside the network but is vulnerable to threats originating from “insiders” with direct access to TMIP-J applications and from “nearsiders” who have network but not application access. A password discrepancy that facilitated this was corrected and retested by ARL with satisfactory results. • One major deficiency was noted in TMIP-J’s core mission area of medical logistics that produced incorrect item quantities in some cases. A viable workaround was developed that adequately mitigates this problem until a material fix can be applied in the next software release. The temporary workaround was agreed to by user representatives of the Service logistics communities and sanctioned by the OTAs. • JITC successfully tested 8 joint interfaces in the test environment, but 38 other interfaces had no test bed and must await interoperability certification in the production environment in order to achieve net-ready compliance. • Although training was adequate overall, several minor deficiencies could be traced to insufficient training. One major deficiency revealed that special training is needed for manually inputting allergy information. • TMIP is the EHR system for deployed military forces. The private health care sector is currently conforming to EHR
80 TMIP-J
standards for medical nomenclature and a national health information infrastructure, as defined by Health and Human Services health information technology standards. In the future, medication reconciliation and real-time sharing of medical records across DoD, Veterans Affairs, and private health care EHR systems will be necessary as military personnel transfer to and from the private and public segments. Future testing will need to demonstrate that TMIP-J conforms to appropriate standards to maintain EHR interoperability with other medical systems as required. Recommendations • Status of Previous Recommendations. The program has satisfactorily addressed all previous recommendations. • FY13 Recommendations. 1. The Deployment and Readiness Systems and TMIP Maritime Program Offices must investigate and correct the major defect regarding restoration of MMM. Restoration must be successfully retested by NIOC and validated by the Commander, Operational Test and Evaluation Force and ATEC prior to introduction of TMIP-J I2R2 to the Navy fleet. 2. The joint and Service TMIP program managers should address remaining cyber security vulnerabilities and the Service OTAs should verify corrective action. 3. The Deployment and Readiness Systems Program Office should ensure that the next software release of the logistics module includes a fix to the defect regarding incorrect units of purchase. 4. JITC needs to test all joint interfaces in the production environment and certify interoperability once I2R2 is fielded. 5. The Service Program Offices should ensure that TMIP-J training for new personnel is more robust and includes manual procedures for allergy entries where applicable.
Army Programs
Army Programs
ARMY PROGRAMS
Network Integration Evaluation (NIE) In FY13, the Army executed two Network Integration Evaluations (NIEs) at Fort Bliss, Texas, and White Sands Missile Range, New Mexico. NIE 13.1 was conducted October through November 2012 and NIE 13.2 was conducted April through May 2013. The purpose of the NIEs is to provide a venue for operational testing of Army acquisition programs, with a particular focus on the integrated testing of tactical mission command networks. The Army intends the NIEs to serve as a venue for evaluating emerging capabilities that are not formal acquisition programs. These systems, termed by the Army as “systems under evaluation” (SUEs), are not acquisition programs of record, but rather systems that may offer value for future development. The Army’s intended objective of the NIE to test and evaluate network components in a combined event is sound. The NIE events should allow for a more comprehensive evaluation of an integrated mission command network, instead of piecemeal evaluations of individual network components. Conducting NIEs two times a year creates an opportunity for event-driven operational testing as opposed to schedule-driven testing. For example, if a system were not ready to enter operational testing at one NIE event, it would have the opportunity to enter testing in a subsequent NIE event. The Army intends to conduct NIE events approximately every six months for the foreseeable future. NIE 13.1 During NIE 13.1, the Army executed a Limited User Test for the Nett Warrior, an FOT&E for the Spider Network Command
Munition, and assessments of 22 SUEs. Individual articles providing assessments of Nett Warrior and Spider can be found later in this Annual Report. NIE 13.2 During NIE 13.2, the Army conducted an IOT&E for the Joint Battle Command – Platform, an FOT&E for Warfighter Information Network – Tactical Increment 2, and a Limited User Test for the Nett Warrior. Individual articles on these programs are provided later in this Annual Report. The Army also conducted assessments of three SUEs during NIE 13.2.
NIE Assessment NIE 13.1 and 13.2 were the fourth and fifth such events conducted to date. The Army’s execution of the NIEs has shown steady improvement over time. The Army has developed a systematic approach to preparing for and conducting NIEs, applying lessons learned from previous events. Overall, NIEs have been a satisfactory venue for conducting operational tests of individual network acquisition programs.
To date, NIEs have focused primarily on scenarios that reflect Iraq/Afghanistan experiences, with combat predominately against dispersed irregular forces. Future NIEs should include more challenging and stressful combined arms maneuver against regular conventional forces. Such scenarios would place greater stress on the tactical network and elicit a more complete assessment of that network.
Operational Scenarios and Test Design. The Brigade Modernization Command, in conjunction with the Army Test and Evaluation Command’s Operational Test Command, continues to develop realistic, well-designed operational scenarios for use during NIEs. Additionally, the 2d Brigade, 1st Armored Division, as a dedicated NIE test unit, is a valuable resource for the conduct of NIEs.
Threat Operations. An aggressive, adaptive threat intent on winning the battle is an essential component of good operational testing. The Army continues to improve threat operations during NIEs, particularly with respect to threat information operations, such as electronic warfare and computer network operations. Future NIEs should incorporate a larger, more challenging regular force threat. This threat should include a sizeable armored force and significant indirect fire capabilities, both of which have been absent in past NIEs. Furthermore, efforts
The challenge for future NIEs will be to develop new and more taxing operational scenarios to reflect future combat operations.
NIE
81
Army PROGRAMS should be made to integrate appropriate unmanned aerial vehicles into the threat forces. Logistics. The Army should place greater emphasis during NIEs on satisfactorily replicating realistic battlefield maintenance and logistical support operations for systems under test. Field Service Representative (FSR) support plans, maintenance and repair parts stockage, and the quantity and management of system spares do not accurately reflect what a unit will observe upon fielding. Easy access to and over-reliance on FSR support results in the test unit not having to realistically execute its field-level maintenance actions. Failure to accurately replicate “real world” maintenance and logistics support cause operational availability rates and ease of maintenance to be overestimated in NIEs. Real-Time Casualty Assessment (RTCA) Instrumentation. An essential component of good force-on-force operational testing, such as that conducted at NIEs, is RTCA instrumentation, which adequately simulates direct and indirect fire effects for both friendly and threat forces. Other key components of functional RTCA instrumentation, in addition to realistic weapons
engagements, include accurate time and position location tracking for all individuals and vehicles on the battlefield and a capability to centrally collect and store in real time weapons engagements, engagement outcomes, and position locations. This battle data collection and storage capability enables analysts to replay battles when conducting evaluations of system performance. The Army has long recognized the need for adequate RTCA to support training, as exemplified by the use of RTCA to support its training venues such as the National Training Center. However, to date, the Army Test and Evaluation Command (ATEC) has used a fraction of the full capability of the RTCA instrumentation that it currently possesses to support operational testing at the NIEs. For instance, ATEC has not used the capabilities to replicate indirect fire effects and to centrally collect battlefield data in real time, despite the existence of a capacity to do so. ATEC should use its full RTCA capabilities for future operational tests in the NIE and initiate efforts to enhance RTCA instrumentation for future use.
Network performance observations Unit Task Reorganization. Operational units frequently change The following are general observations of tactical network task organizations to tailor for tactical missions. The process to performance during NIEs. These observations focus on network update the network to accommodate a new unit task organization performance deficiencies that the Army should address as it remains excessively lengthy and complex. moves forward with integrated network development. Complexity of Use. Network components, both mission command systems and elements of the transport layer, are excessively complex to use. The current capability of an integrated network to enhance mission command is diminished due to pervasive task complexity. It is challenging to achieve and maintain user proficiency. For example, what should be relatively simple tasks of starting up and shutting down systems require a complex series of actions by the operator. Common Operating Picture (COP). Joint Publication 3-0, (Joint Operations) defines a COP as “a single identical display of relevant information shared by more than one command that facilitates collaborative planning and assists all echelons to achieve situational awareness.” With current mission command systems, units have multiple individual COPs (e.g., for maneuver, intelligence, and logistics) based upon the corresponding mission command systems, instead of a single COP that is accessible on one system. The Army is seeking to resolve this problem and these efforts should continue. Network Configuration. The process for planning and loading a Soldier Radio Waveform network is cumbersome and time consuming. For example, during the Handheld, Manpack, and Small Form Fit – Manpack radio operational test in NIE 12.2, it took two Soldiers 2 to 3 days to set up and load all 46 Manpack radios and 96 Rifleman Radios in the test company. A single Manpack radio required up to 25 minutes to load the network plan, download cryptographic keys, and perform a communications check.
82
NIE
Armored Brigade Combat Team Integration. The challenge of integrating network components into tracked combat vehicles remains unresolved. Due to vehicle space and power constraints, the Army has yet to successfully integrate desired network capabilities into Abrams tanks and Bradley infantry fighting vehicles. It is not clear how the desired tactical network will be incorporated into heavy brigades. Signal Soldier Manpower. The Army has added a large number of new network components without a corresponding increase in signal Soldiers to manage and maintain these components. This has considerably increased the demands upon the signal Soldiers who are available. There are currently insufficient signal Soldiers assigned to the brigade to effectively operate and maintain the increased number of network components. The Army should evaluate the force structure implications of adding a large amount of new communications equipment into tactical units without a corresponding increase in support personnel. Dependence on FSRs. Units remain overly dependent upon civilian FSRs to establish and maintain the integrated network. This dependency corresponds directly to the excessive complexity of use of network components. Survivability. An integrated tactical network introduces new vulnerabilities to threat countermeasures, such as threat computer network attacks and the ability of a threat to covertly track friendly operations. The Army should continue to improve its capabilities to secure and defend its tactical network.
ARMY PROGRAMS
Armored Tactical Vehicles Executive Summary • In FY13, the Army developed survivability upgrades intended to improve force protection compared to the existing Family of Medium Tactical Vehicles (FMTV) with the Long Term Armor Strategy (LTAS) B-kit armor. These upgrades include armor for the underbody of the crew cab (designated as a C-kit), improved blast mats on the crew cab floor, and blast mitigation seats. DOT&E’s preliminary assessment is that the survivability upgrades improve force protection compared to the existing FMTV LTAS with B-kit armor. • From February to December 2013, four High Mobility Multi-purpose Wheeled Vehicle (HMMWV) sustainment modification initiative (SMI) concept demonstrators underwent performance, endurance, and transportability testing at the Nevada Automotive Test Center, Nevada. The program will use the results from testing to select the best concept to develop the HMMWV SMI system design specification. • In May 2013, the Army awarded contracts to AM General (partnered with Hardwire LLC), AM General (partnered with Plasan Sasa), Ceradyne Inc., and Textron Land & Marine Systems (partnered with Granite Tactical Vehicles) to conduct ballistic testing of their Modernized Expanded Capacity Vehicle (MECV) designs. The MECV HMMWV is a research and development effort that the Army does not intend to transition to a formal acquisition program. System FMTV • The FMTV re-procurement is the Army’s fourth contract used for FMTV purchase. The FMTV is a series of trucks based on a common chassis that vary by payload and mission. These vehicles consist of the following light and medium variants that operate on- and off-road: -- The Light Medium Tactical Vehicle (LMTV) transports a 5,000-pound payload and a 12,000-pound towed load. -- The Medium Tactical Vehicle (MTV) transports a 10,000-pound payload and a 21,000-pound towed load. HMMWV • The HMMWV is a general-purpose tactical wheeled vehicle with light and heavy variants. -- The Light Variant includes the light utility, weapons carrier, and two-litter ambulance with a required minimum payload of 2,600 pounds. -- The Heavy Variant includes the heavy shelter carrier and four-litter ambulance with a required minimum payload of 4,550 pounds. • The Marine Corps expects the HMMWV SMI program to restore performance, reliability, and sustainment capabilities of the HMMWV ECV lost due to up-armoring.
• The Army expects the MECV HMMWV effort to identify improved underbody crew protection. Mission FMTV • The Army employs the FMTV to provide multi-purpose transportation in maneuver, maneuver support, and sustainment units. HMMWV • The Army and Marine Corps employ this vehicle throughout the battlefield to provide highly-mobile, light tactical wheeled transport for command and control, troops and light cargo, medical evacuation, and weapon platforms to division and below units. The HMMWV operates in off-road and cross-country environments.
Armored Tactical Vehicles
83
Army PROGRAMS Major Contractors FMTV • Oshkosh Corporation – Oshkosh, Wisconsin HMMWV • AM General – South Bend, Indiana HMMWV SMI • To be determined HMMWV MECV • AM General (partnered with Hardwire LLC) – South Bend, Indiana
Activity FMTV • In FY13, the Army developed survivability upgrades intended to improve force protection compared to the existing FMTV with the LTAS B-kit armor. These upgrades include armor for the underbody of the crew cab (designated as a C-kit), improved blast mats on the crew cab floor, and blast mitigation seats. • In July 2013, DOT&E approved the Army’s LFT&E plans for the survivability upgrades. The program will use the results from the LFT&E to evaluate if the survivability upgrades improve force protection. • From July to September 2013, the Army conducted two underbody blast tests against realistic threats at Aberdeen Test Center, Maryland. The Army will conduct three additional underbody blast test events in FY14. • The program may issue a Full Material Release for the survivability upgrades in FY14. HMMWV SMI • From February to December 2013, four HMMWV SMI concept demonstrators underwent performance, endurance, and transportability testing at the Nevada Automotive Test Center. Results from the testing will be used to select the best concept to develop the HMMWV SMI system design specification. • The Program Office began drafting the initial HMMWV SMI Test and Evaluation Master Plan (TEMP) in March 2013 that outlines the HMMWV SMI developmental, operational, and live fire test and evaluation plans and resources for the Engineering and Manufacturing Development and production phases of the program. • In November 2013, the Marines Corps expects to release a HMMWV Operational Requirement Document (ORD) clarification letter to define the HMMWV SMI requirement. The clarification letter will identify updated requirements to: -- Restore safe operations over the expeditionary mission profile -- Retain reliability, availability, and maintainability to ORD threshold values over the expeditionary mission profile -- Retain or improve transportability
84
Armored Tactical Vehicles
• AM General (partnered with Plasan Sasa) – South Bend, Indiana • Ceradyne Inc. – Casa Mesa, California • Textron Land & Marine Systems (partnered with Granite Tactical Vehicles) – Slidell, Louisiana
-- Restore payload to ORD values -- Reduce operations and maintenance costs -- Retain or improve protection features • The HMMWV SMI program plans to release a Request for Proposals in 2QFY14 for a competitive contract awarded to two vendors to produce prototype vehicles and participate in the Engineering and Manufacturing Development phase. • The HMMWV SMI Milestone B decision is planned for May 2014. HMMWV MECV • In May 2013, the Army awarded contracts to AM General (partnered with Hardwire LLC), AM General (partnered with Plasan Sasa), Ceradyne Inc., and Textron Land & Marine Systems (partnered with Granite Tactical Vehicles) to conduct ballistic testing of their MECV designs. • In July 2013, DOT&E approved the Army’s test plan for the live fire test of the MECV designs. The Army will use the test results to characterize the industry’s ability to improve the underbody crew protection of the existing armored HMMWV. • From August to September 2013 at Aberdeen Test Center, Maryland, the Army conducted two underbody blast tests on each of the MECV designs, plus one underbody blast test on an Army developed design and one underbody blast test on an existing armored HMMWV. This completes the test series. Due to sequestration, Federal Government shutdown, and restrictions in the Pay Our Military Act, the Army analysis has been delayed. DOT&E will provide a report to Congress with the test results compared to existing light tactical vehicles in 2QFY14. DOT&E will provide a second report to Congress with the MECV test results compared to test results from the Joint Light Tactical Vehicle program in 2QFY15. Assessment FMTV • Analysis of the FMTV survivability upgrades underbody test data is ongoing. DOT&E’s preliminary assessment is that the survivability upgrades improve force protection compared to the existing FMTV LTAS with B-kit armor.
Army PROGRAMS HMMWV SMI • The draft HMMWI SMI TEMP being proposed by the Marines requires additional details on reliability growth, developmental, and operational testing prior to DOT&E approval. • The HMMWV SMI program intendeds to procure approximately 6,000 HMMWVs. HMMVW MECV • Analysis of the MECV underbody test data is ongoing. • The MECV is a research and development effort that the Army does not intend to transition to a formal acquisition
program. If the Army decides to transition the MECV to an acquisition program, adequate developmental, operational, and live fire testing will be required. Recommendations • Status of Previous Recommendations. The Army has addressed all previous recommendations. • FY13 Recommendations. None.
Armored Tactical Vehicles
85
Army PROGRAMS
86
ARMY PROGRAMS
Bradley Engineering Change Proposal (ECP) Executive Summary • In June 2013, the Army conducted an underbody blast test of an M2A2 Operation Desert Storm Bradley modified in the squad area to represent an M2A3 Infantry Fighting Vehicle with Engineering Change Proposal 1 (ECP1) components. The vehicle also included proposed modifications to the underbody add-on armor, squad area’s floor, and ammunition stowage plan. • The blast test revealed that significant improvements to the Bradley Fighting Vehicle Systems (BFVS) level of force protection and vulnerability are feasible. Additional testing is required to further refine and evaluate the proposed modifications. System • The Army expects the Bradley ECP1 to restore ground clearance with upgrades to the suspension and track. ECP2 will integrate network technologies as they become available for three variants of the BFVS: - M2A3 Infantry Fighting Vehicle - M3A3 Cavalry Fighting Vehicle - Bradley Fire Support Team with Fire Support Sensor System • The program designed the Bradley Urban Survivability Kit I, II, and III and add-on armor kit to improve vehicle and crew survivability. These kits were urgently fielded for Operation Iraqi Freedom and are now part of the M2A3 configuration.
Activity • The Army is developing additional survivability upgrades outside of the ECP efforts to improve force protection and decrease vulnerabilities identified in FY12. • In June 2013, DOT&E approved the Detailed Test Plan Addendum for a third Bradley ECP underbody blast event. The objective of the test was to determine if the additional proposed survivability upgrades could improve force protection. Additionally, DOT&E directed that the Program Office develop a Test and Evaluation Master Plan for the ECP2 test and evaluation. • In June 2013, the Army conducted an underbody blast test at the Aberdeen Test Center, Maryland, of an M2A2 Operation Desert Storm Bradley modified in the squad area to represent an M2A3 Infantry Fighting Vehicle with ECP1 components, along with proposed survivability upgrades to the underbody add-on armor, squad area floor, and ammunition stowage plan.
Mission Combatant Commanders employ BFVS-equipped Armor Brigade Combat Teams to provide protected transport of Soldiers; provide overwatching fires to support dismounted infantry and suppress an enemy; and perform missions to disrupt or destroy enemy military forces and control land areas. Major Contractor BAE Systems Land and Armaments – Sterling Heights, Michigan
now part of the M2A3 configuration, was not adequate. Furthermore, testing in FY12 revealed severe vehicle and occupant vulnerabilities. • Results from the third underbody blast test in June 2013 revealed that significant improvements to the BFVS’s force protection and vulnerability are feasible. Additional testing is required to further refine and evaluate the proposed survivability modifications. • Results from the third underbody blast test also demonstrate that the Armored Multi-purpose Vehicle survivability requirement is achievable with a Bradley-like platform. • The underbody blast tests with realistic threats (as opposed to outdated underbody requirements) conducted to-date alone are not sufficient to address all of the critical BFVS survivability concerns. The Army will need to develop a comprehensive LFT&E strategy once the design of the improvement kit is fixed.
Assessment • The Army’s previous vulnerability testing of the Bradley Urban Survivability Kit I, II, and III and add-on-armor kit,
Bradley ECP
87
Army PROGRAMS Recommendations • Status of Previous Recommendations. The Army began addressing the two previous recommendations concerning the need for a comprehensive live fire strategy and examination of vulnerabilities identified during early testing; however, these recommendations remain open and will be addressed in FY14. • FY13 Recommendation. 1. The Army should conduct adequate technical testing of proposed survivability improvement kits and modifications to optimize the design prior to conducting formal live fire testing.
88
Bradley ECP
ARMY PROGRAMS
CH-47F (Chinook) – Improved Cargo Helicopter Executive Summary • DOT&E published a combined OT&E/LFT&E report in June 2007 and found that the CH-47F is operationally effective, suitable, and survivable. • As of September 2013, Boeing has delivered 256 of the planned 464 CH-47F aircraft. • Commanders in combat and homeland support report that the CH-47F is much more capable than the CH-47D. • The Army continues to improve the CH-47F by incorporating product improvements to address operational test findings and respond to emerging operational needs. Key product improvements since 2007 have enhanced mission capabilities and increased aircraft survivability. • Issues identified during testing in 2012 and 2013 that require resolution: - The CH-47F Cargo On-Off Loading System (COOLS) enhances the operational effectiveness of CH‑47F‑equipped units and is an improvement over the legacy cargo handling system for supporting combat operations. Ramp rollers do not provide adequate clearance to accommodate the forklift tines on the Army’s primary tactical forklift and unrestrained cargo movement poses a danger to crews. The Army should modify the COOLS design or installation to provide sufficient ramp clearance for forklift tines and reinforce the need to heed published warnings to avoid crewmember injury and equipment damage when operating the system. - The COOLS under-floor Ballistic Protection System (BPS) provides some ballistic protection to the crew and passengers, but not to the same level expected from earlier qualification testing. The Army should conduct additional ballistic testing of the BPS to understand the varying performance noted in testing and determine if the new version of the BPS meets the Army’s requirements. System • The CH-47F is a twin-turbine, tandem-rotor, heavy-lift transport helicopter that enables the Army to support the rapid response capability necessary for forcible and early entry contingency missions, as well as linear and nonlinear, and simultaneous or sequential operations. • The CH-47F is used in General Support Aviation Battalions assigned to Combat Aviation Brigades. Each Battalion has 12 CH-47F helicopters authorized. • The CH-47F is designed to transport artillery and light equipment, up to 16,000 pounds, or 31 combat troops. • The CH-47F aircraft capability improvements include: - A CAAS cockpit to increase crew situational awareness and increase cockpit commonality with other Army aircraft
- A Digital Automatic Flight Control System to improve handling qualities and decrease pilot workload - Engine upgrades for increased power - Fuselage stiffening, corrosion protection, and a new monolithic airframe structure to reduce cockpit vibration and increase airframe durability - The Common Missile Warning System, an Infrared Suppression System, and an Advanced Tactical Infrared Countermeasures system for increased aircraft survivability • The Army acquisition objective is to procure 464 CH-47F aircraft. Mission The General Support Aviation Battalion assigned to the Combat Aviation Brigade employs the CH-47F to conduct the following types of missions: • Air Assault operations to transport ground forces and equipment • Air Movement operations to move passengers, fuel, ammunition, and equipment • Casualty evacuation operations • Disaster relief, fire-fighting, and rescue operations Major Contractors • Aircraft: The Boeing Helicopter Company – Ridley Park, Pennsylvania • Engine: Honeywell – Phoenix, Arizona • Software development: Rockwell Collins – Cedar Rapids, Iowa
CH-47F
89
Army PROGRAMS Activity • As of September 2013, Boeing has delivered 256 of the planned 464 CH-47F aircraft. • The Army completed testing, fielding, and deployment of the following product improvements between 2007 and 2012: - Enhanced coupled flight director to reduce pilot workload in the cruise/en-route flight mode (functionality that links flight plan/navigation guidance to the flight control system allowing it to generate corresponding flight control inputs) - Integrated communications system upgrade to enhance voice, data, and navigation capabilities - Additional Common Missile Warning System sensor to increase missile warning effectiveness - Infrared Suppression System to reduce aircraft infrared signature - Advanced Tactical Infrared Countermeasure with an active infrared jammer for missile defense • The Army completed integrated testing of the CH-47F COOLS from November through December 2012 where operational crews completed four end-to-end internal cargo missions using a COOLS-equipped CH-47F. Aircrews loaded and unloaded 20 cargo pallets, and on 2 missions reconfigured the COOLS in flight; this was a task that crews could not have completed using the legacy Helicopter Internal Cargo Handling System (HICHS). • The Army Research Laboratory/Survivability/Lethality Analysis Directorate (ARL/SLAD) conducted live fire testing from May through June 2013 to evaluate the ballistic performance of the under-floor BPS against a variety of expected small-arms projectiles. Production representative panels were installed in the same configuration for the test as they are on the aircraft. ARL/SLAD also performed a force protection analysis to assess the level of protection afforded to the crew and passengers. • The Army conducted integrated testing and live fire testing of the COOLS in accordance with the DOT&E-approved test plans. • The Army conducted flight testing of CAAS v9.2 from August 2011 to February 2012. The CAAS v9.2 is designed to improve situational awareness, expand Identification Friend or Foe Mode 5 capability, update the flight performance modules, and provide the highest level of navigation certification: Required Navigational Performance/Area Navigation. • DOT&E published the CH-47F COOLS Integrated Test report in July 2013 and the CH-47F COOLS BPS LF&E report in August 2013. Assessment • Reports from theater indicate that the CH-47F is much more capable than the CH-47D. Commanders in Afghanistan commend the CH-47F for its superior navigation, enhanced voice and digital communications, ability to operate in a high‑altitude and hot-temperature environment, high- operational tempo, and overall system reliability. • The Army continues to improve the CH-47F by incorporating product improvements to address operational test findings
90
CH-47F
•
•
•
•
and respond to emerging operational needs. Key product improvements since 2007 have enhanced mission capabilities and increased aircraft survivability. CAAS v9.2 performed satisfactorily and the aircraft achieved the Required Navigation Performance/Area Navigation certification. - The handling qualities and flight characteristics of the CH-47F have not changed and the aircraft continues to be capable of performing its mission. - Identification Friend or Foe Mode 5 capability and flight performance modules performed satisfactorily. - Workload when using CAAS v9.2 was satisfactory. The COOLS enhances the operational effectiveness of CH-47F-equipped units and is an improvement over the legacy cargo handling system for supporting combat operations. Testing revealed problems with ramp roller clearance and the dangers of unrestrained cargo movement. - The COOLS is easily reconfigured for carrying troops or cargo, adds flexibility for CH-47F mission tasks, and increases cargo carrying capacity by 543 pounds when combat-configured. Soldiers can easily accomplish loading, handling, securing, and unloading palletized cargo with the COOLS at airfields and field sites. - The COOLS is not compatible with all fielded Army tactical forklifts. COOLS ramp rollers do not provide adequate clearance to accommodate the forklift tines on the Army’s primary tactical forklift. - Unrestrained cargo movement on COOLS rollers is dangerous. The risk is mitigated through New Equipment Training and published notes, cautions, and warnings in operator and technical manuals. The COOLS under-floor BPS provides improved coverage over the legacy BPS. The COOLS BPS does not provide the level of ballistic protection expected from previous material qualification testing, requiring additional distance between the weapon and the aircraft to have equivalent protection. Live fire testing also revealed some anomalies indicating that the ballistic performance varies with impact angle. - The COOLS BPS is semi-permanently installed below the cabin floor and provides coverage for the entire cabin floor and ramp areas. - The legacy BPS cannot be reconfigured in flight to accommodate a change of mission from cargo to passenger transfer and also is significantly heavier, so the complete legacy BPS is often not used in theater. - The COOLS BPS, being lighter and stowed away, provides a greater area of coverage under a wider variety of missions. The results of the force protection analysis indicated that the level of protection offered by the COOLS BPS is moderate for one of the two projectiles analyzed, and relatively low for the other higher caliber projectile. The results presented were for impacts representative of the weapon’s muzzle velocity and three additional velocities representative of typical
Army PROGRAMS increasingly greater combat standoff ranges (i.e., lower impact velocities for distances of 100, 200, and 300 meters). Recommendations • Status of Previous Recommendations. The Army addressed 9 of 10 FY07 recommendations. The Army should improve the APR-39 radar warning receiver performance to increase threat reporting accuracy for the aircrew or install a more accurate alternative radar warning receiver. • FY13 Recommendations. The Army should:
1. Modify the COOLS design or installation to provide sufficient ramp clearance for forklift tines. 2. Reinforce in training the need to heed published warnings to avoid crewmember injury and equipment damage when operating the COOLS. 3. Perform additional testing of the COOLS BPS armor to understand the varying performance with regard to angles of impact and determine if it still meets the Army’s requirements.
CH-47F
91
Army PROGRAMS
92
ARMY PROGRAMS
Distributed Common Ground System – Army (DCGS-A) Executive Summary • From May through June 2012, the Army Test and Evaluation Command conducted an IOT&E of the Distributed Common Ground System – Army (DCGS-A) Software Baseline (DSB) 1.0 system in an operationally representative field configuration. DOT&E evaluated the DSB 1.0 to be not effective, not suitable, and not survivable. • The Army reconfigured the DSB 1.0 without the Top Secret (TS)/Sensitive Compartmented Information (SCI) enclave to mitigate the effectiveness and suitability shortfalls identified in DOT&E’s IOT&E report, and demonstrated fixes to the critical Information Assurance shortfalls. The reconfigured package is called Release 1. • DOT&E released a memorandum in November 2012 that stated Release 1 will provide users with capabilities at least as good as those provided by the current systems. OSD approved the full deployment for Increment 1 in December 2012. • DOT&E and the Army are preparing for developmental and operational testing of Release 2, which includes SCI capability at the brigade level. • DOT&E published a report on October 21, 2013, in response to the House Armed Services Committee request to report on DCGS-A’s database interoperability. System • DCGS-A allows users to collect, process, fuse, and display intelligence information. • DCGS-A is the information- and intelligence-processing centerpiece of the Army Intelligence, Surveillance, and Reconnaissance framework and is the enabler for all intelligence functions at the Division, Brigade Combat Team, Maneuver Battalion, and Company levels. • The DSB 1.0 configuration established the architecture that will provided an organic net-centric Intelligence, Surveillance, and Reconnaissance capability by combining 16 stove-piped legacy applications into one comprehensive network, and providing an integrated TS/SCI capability at the brigade level. • After the IOT&E report, the Army reconfigured the system as Release 1 with only the Secret enclave components. OSD approved the full deployment of this configuration. • The Army is developing Release 2 to fulfill the capabilities that did not work effectively with DSB 1.0. Release 2 is intended to provide enhanced capabilities to include: - TS/SCI capability - Workflows that are based on how an intelligence section would employ the system
- Methods for transferring entity data within the system and between systems that are more efficient than DSB 1.0 - New entity database structure - Enhanced fusion software for correlation of entity data - New geospatial intelligence configuration - New materiel solution for transfer of information across security domains • The Army is developing Release 3 to include a cloud computing capability to support worldwide intelligence analysis and database synchronization including in disconnected or low-bandwidth environments. Mission Army intelligence analysts use DCGS-A Release 1 to support six Mission Command Capabilities: • Display and share relevant information • Provide a standard and shareable geospatial foundation • Collaborate in voice, text, data, and video modes • Execute running estimates of enemy force progress • Interoperate across the joint, interagency, intergovernmental, and multinational forces Major Contractors • Lead System Integrator – Intelligence and Information Warfare Directorate, U.S. Army Communications-Electronics Research, Development, and Engineering Center – Aberdeen, Maryland • Northrop Grumman Electronic Systems – Linthicum, Maryland
DCGS-A
93
Army PROGRAMS Activity • From May through June 2012, the Army Test and Evaluation Command conducted an IOT&E of the DCGS-A DSB 1.0 system in an operationally representative field configuration. In October 2012, DOT&E provided an IOT&E report on the results of testing to the Milestone Decision Authority and Congress. • The Army reconfigured the DSB 1.0 system without the TS/ SCI enclave to mitigate the effectiveness and suitability shortfalls in the DOT&E IOT&E report and demonstrated fixes to the critical Information Assurance shortfalls. The reconfigured system is called Release 1. DOT&E provided an evaluation of Release 1 in a November 2012 memorandum. • In December 2012, the Defense Acquisition Board approved full deployment of Release 1 and discussed the need for a comparative test of link analysis tools. • In March 2013, the DCGS-A program manager ceased development of the initial cloud capability in favor of a new architectural approach developed by the intelligence community. • DOT&E is working with the Army to comprehensively test and evaluate DCGS-A’s capabilities compared to other commercially available tools. DOT&E continues to work with the Army to define and execute adequate comparative test; however, agreement on the content of that testing has not yet been reached. • DOT&E published a report on October 21, 2013, in response to the House Armed Services Committee request to report on DCGS-A’s database interoperability.
94
DCGS-A
Assessment • DOT&E evaluated the DSB 1.0 to be not effective and not suitable in the October 2012 report of the IOT&E. • DOT&E evaluated the Release 1 configuration, without the TS/SCI enclave, to be at least as good as those provided by the current systems. • As of November 2013, Release 2 software is still in development and preparing for developmental testing. • There are insufficient test data to assess fully the worldwide synchronization of Army intelligence databases including operations in degraded communication environments. Full assessment of worldwide synchronization, including the cloud edge node, will be needed when the Army develops a new cloud functionality as part of Release 3. Recommendations • Status of Previous Recommendations. The Army is complying with the recommendation to conduct operational testing of all releases of DCGS-A Increment 1. • FY13 Recommendations. The Army: 1. Needs to continue to plan for and conduct Release 2 operational testing. 2. Must plan for, resource, and execute an adequate comparative test of DCGS-A and other commercially available products.
ARMY PROGRAMS
DoD Automated Biometric Information System (ABIS) Executive Summary • The DoD Automated Biometric Identification System (ABIS) 1.0 was fielded to the Biometrics Identity Management Agency (BIMA) in January 2009 as a quick reaction capability to support storing, matching, and sharing of collected biometric data primarily obtained during Operation Iraqi Freedom and Operation Enduring Freedom. • The Army stood up a Program Management Office (PMO) in 2009 to foster the establishment of ABIS as a formal program of record to be known as the Biometrics Enabling Capability (BEC) Increment 0. • In January 2011, USD(AT&L) issued an Acquisition Decision Memorandum establishing ABIS 1.2 as the baseline for the BEC 0 upon completion of a Full Deployment Decision (originally scheduled for FY11). • Since 2010, there have been four failed attempts to deploy ABIS 1.2, all resulting in roll-back decisions. - In the most recent deployment attempt (August 2013), BIMA did not operate ABIS 1.0 and 1.2 in parallel. All users were forced to use only ABIS 1.2, which did not meet user needs. - Customers immediately reported significant operational impacts, but the roll-back to ABIS 1.0 did not occur until the tenth day of operations with ABIS 1.2. - U.S. Special Operations Command (USSOCOM) documented 31 high-priority deficiencies and U.S. Central Command (USCENTCOM) documented 11 additional high-priority deficiencies that affected mission accomplishment due to deficiencies in the ABIS 1.2 baseline. - The Director, Defense Forensics and Biometrics Agency (DFBA), the Executive Manager for Biometrics decided to restore ABIS 1.0 as the system of record. DBFA recommended that no further upgrade attempts be made until (1) requirements are refined and validated by the joint community and (2) all high-priority findings are addressed and fixes are acceptable to ABIS customers. • ATEC performed a limited capabilities and limitations assessment on ABIS 1.0; however, no independent ATEC testing has been conducted on ABIS 1.2. • Developmental tests to date have, for the most part, not been operationally realistic. The program needs to address deficiencies identified in testing to date and verify correction of those deficiencies in an operationally relevant environment before proceeding to IOT&E. System • DoD ABIS is an authoritative database that uses software applications to: - Process and store biometrics modalities (i.e., fingerprints, palm prints, iris scans, and facial recognition data) from collection assets across the globe
•
•
•
• •
- Update stored records with new biometrics data - Produce biometrics match results (against stored data) - Share responses among approved DoD, interagency, and multinational partners, in accordance with applicable law and policy For biometric submissions that are unable to produce a match using automated processes, biometric examiners (subject matter experts) use ABIS workstations with specialized software to attempt to manually match submissions. ABIS interfaces with global biometrics data collectors and users, as well as outside databases. - Military Services and Combatant Commands collect biometrics data (fingerprint, palm print, iris scans, and facial scans) from persons of interest in the field using hand-held devices and submit that data to ABIS. - Intelligence analysts analyze and fuse biometrics information via the Biometric Identity Intelligence Resources (BI2R), an automated database outside the ABIS, and provide information back to the users in the field. Custom components include: - A transaction manager for managing customer submission workflows - A portal allowing authorized operators to perform user management, system configuration, real-time system monitoring, submission tracking, and report generation The U.S. Army BIMA currently operates ABIS on the Non‑secure Internet Protocol Router Network (NIPRNet). The PMO is developing ABIS 1.2 as an enhancement to the current ABIS 1.0. The new system is intended to address hardware and software scalability limitations in ABIS 1.0. The PMO intends ABIS 1.2 to be an architecture that will enable increased throughput of biometric submissions.
DoD ABIS
95
Army PROGRAMS Mission • Military Services and U.S. Combatant Commands rely on ABIS to provide timely, accurate, and complete responses indicating whether persons of interest in the field have a prior history of criminal activity, to assist in identifying potential threats to U.S. forces and facilities. • The Federal Bureau of Investigation, the Department of Homeland Security, and the National Ground Intelligence Center interface with ABIS to identify biometrics matches in support of U.S. criminal cases, border control, and intelligence watchlists, respectively.
Activity • ABIS was first developed as a prototype in 2004 in response to a Joint Urgent Operational Need Statement (JUONS). ABIS 1.0 was deployed to BIMA in January 2009 as a prototype system to provide multi-modal and multifunctional biometric capabilities to assist in the Global War on Terrorism and subsequently in Overseas Contingency Operations. • Since 2004, DOT&E has designated all biometrics programs be placed on the test and evaluation oversight list as pre‑Major Automated Information Systems. As such, although not a formal program of record, ABIS is included on DOT&E oversight. • In July 2009, USD(AT&L) designated the Army as the Executive Agent for Biometrics with responsibility for development, acquisition, and fielding of common biometrics enterprise systems to support joint DoD requirements. • The Army, as the executive agent and the lead materiel developer for ABIS, stood up a PMO in 2009 to foster the establishment of ABIS as a formal program of record to be known as the BEC Increment 0. • In 2009, ATEC conducted a limited capabilities and limitations assessment on ABIS 1.0. • In January 2011, USD(AT&L) issued an ADM establishing ABIS 1.2 as the baseline for BEC 0 upon completion of a Full Deployment Decision (originally scheduled for FY11). • Concurrent with attempts to transition BEC 0 to a program of record, the Army has been developing joint requirements for the BEC Increment 1 (BEC 1) Capabilities Development Document. However, requirements approval for the document is under further review due to concerns about affordability. • The PMO attempted to deploy ABIS 1.2 three times from 2010 through 2012. In each case, the attempt was unsuccessful and the system was rolled back to the DoD ABIS 1.0 baseline. - After the first failed deployment attempt (November 2010), the PMO and BIMA executed two developmental tests (January and June 2011). The second developmental test was intended to verify fixes to defects discovered in the first developmental test. The PMO and BIMA said the tests were conducted in an operationally representative environment (using an operationally representative
96
DoD ABIS
Major Contractor Northrop Grumman, Information Technology (NGIT) – Fairmont, West Virginia
backup facility, operational biometric data, operationally representative use cases, and actual BIMA operators). - The PMO attempted second deployment of ABIS 1.2 in August 2011. The program had intended to use ABIS 1.2 as the primary operational capability, and to monitor interoperability and operational availability, reliability, and maintainability during deployment; then address and fix deficiencies as they were identified. - In August 2012, the PMO deployed ABIS 1.2 using a parallel operations test construct to compare capabilities of ABIS 1.0 against ABIS 1.2. The program intended to execute a broad range of operational use cases on both systems and compare results at each stage of the process; capabilities evaluated were the ability to match, store, share, reference, analyze, and manage biometric data in a timely, accurate, reliable, and usable manner. This third deployment attempt resulted in failure to meet exit criteria outlined in the Memorandum of Agreement between the PMO and the Deputy Director for Operations for BIMA conducting the DoD ABIS Parallel Operations Assessment. • In December 2012, the PMO conducted a “customer” (developmental) test to determine if ABIS 1.2 enabled the operators to access the functions they needed to perform their duties and that the system would react with consistent, accurate, and useful reports, displays, or other responses. • The PMO conducted a subsequent customer test April 29 through May 1, 2013, and a third customer test June 25‑27, 2013, to verify correction of Severity 1 and 2 findings from the initial customer test. - The tests were conducted in a laboratory-like (not operationally representative) enclave with no external connectivity that would allow submissions to be received or responses sent out to the warfighter. - Independent subject matter experts were unavailable to support testing, and were not used to assess the validity of the use cases, test data, and results. There was also no attempt to perform regression testing. - The second test suggested that the defects from the first test had been fixed, but there was no testing performed to
Army PROGRAMS determine if new defects had been introduced by the fixes to the program. • In August 2013, the PMO deployed ABIS 1.2, which again resulted in a roll-back decision. During this deployment attempt, ABIS 1.2 was activated as the system of record directly supporting real-world operations. - In the previous deployment attempt, BIMA operated ABIS 1.0 and ABIS 1.2 in parallel. In the August 2013 deployment attempt, all users were forced to use only ABIS 1.2, which did not meet user needs. - Customers immediately reported significant operational impacts, but the roll-back to ABIS 1.0 did not occur until the tenth day of operations with ABIS 1.2. - USSOCOM documented 31 high-priority deficiencies and USCENTCOM documented 11 high-priority deficiencies that affected mission accomplishment due to deficiencies in the ABIS 1.2 baseline. - Users noted suitability deficiencies, including the inability to locate data and an increase in the percentage of files requiring human review. • As a result of the latest roll-back decision, DBFA recommended that no further upgrade attempts be made until: - Requirements are refined and validated by the joint community - All high-priority findings are addressed and fixes are acceptable to ABIS customers Assessment • The results of the Army’s 2009 capabilities and limitations assessment indicated that ABIS 1.0 successfully met BIMA performance and suitability requirements (subject to test limitations including lack of external connectivity preventing an assessment of operational interfaces and lack of independently administered surveys). • The program established a pattern of “fixing” problems found in developmental testing, conducting developmental testing in a non-operational environment, and then failing in operational deployment. This pattern indicates an undisciplined software development, deployment, and maintenance process. The lack of requirements approved at the joint level, lack of a measured baseline system against which comparisons of the ABIS 1.2 can be made, and lack of mature configuration management processes have further exacerbated test adequacy and reporting concerns. • The November 2010 deployment attempt showed failures to import the Biometrics Enabled Watchlist, failure to properly migrate all the data, and failure to migrate all identities. A prior Joint Interoperability Test Command assessment (September 2009) found that lack of a formal standards conformance program for the DoD Biometric enterprise resulted in a multitude of interoperability deficiencies. Two deficiencies – incorrect cross-linking of distinct identities and substantial errors during data ingestion creating a backlog in workload – were potentially related to the November 2010 results.
• The August 2011 deployment attempt demonstrated operational shortfalls including system instability, inconsistent processing times, system congestion, transaction errors, and a 48-hour outage. Prior to that deployment attempt, a developmental test in January 2011 showed critical function failures, inadequately trained operators, and incomplete documentation. These defects were deemed fixed in a June 2011 developmental test, once again reflecting the lack of adequate developmental testing and lack of an operationally relevant test environment. • The August 2012 deployment attempt revealed 10 Severity 2 and 35 Severity 3 deficiencies; however, the PMO focused on only 2 of the Severity 2 deficiencies when conducting the customer tests leading to the August 2013 deployment. Consequently, many of the same deficiencies arose during the August 2013 deployment, including degradation in match accuracy, unexplained discrepancies in results between the two versions, system congestion under routine load, transaction errors from incoming/outgoing external interfaces, and (once again) problems migrating identities from the ABIS 1.0 database into the new ABIS 1.2 database. • During the August 2013 deployment, testing revealed that the interfaces between the current 1.0 system and its customers are not fully defined and documented. Interfaces have been created and sustained on an ad-hoc basis by BIMA in support of mission needs. Documentation of the interfaces and services provided by ABIS 1.0 and re-creation of those services in ABIS 1.2 will require close collaboration between operators and the system engineers responsible for the 1.0 and 1.2 systems. • In addition to demonstrating significant shortfalls with ABIS 1.2, testing during the August 2013 deployment also revealed problems with ABIS supportability. - ABIS 1.0 authority to operate will be at risk due to its dependence on Windows XP®, which is being phased out. Significant Information Assurance vulnerabilities will not be patched if support for the Windows XP® product is curtailed by the vendor. Accordingly, ABIS 1.0 should be modernized to eliminate its dependency on Windows XP® and other hardware or software reaching end-of-life. - Should ABIS 1.0 be baselined, an assessment is needed of the ABIS 1.0 system, to include the definition of external interfaces to the current system and customers. Any script utilities that exist to enable end-to-end exchange should be defined, documented, and consolidated to the extent possible so that each interface can be maintained even as changes continue to be made to the core system. - The ABIS 1.2 system is also dependent upon Windows XP® and utilizes the same search core employed by the ABIS 1.0 system. For ABIS 1.2, both the Windows® operating system and the search core are planned to be updated starting in 1QFY14. • Based on nearly five years of development and testing, ABIS 1.2 continues to experience a significant number of high-priority defects, which have not yet been addressed and
DoD ABIS
97
Army PROGRAMS verified to be fixed in an operationally realistic environment. Progression of ABIS into the BEC Program of Record will require resolution of problems with ABIS 1.2 deficiencies. Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. The PMO should: 1. Capture and report baseline performance measures of the ABIS 1.0 before attempting any further upgrades. 2. Prior to IOT&E of ABIS 1.2, fix deficiencies identified in prior developmental testing and verify correction of fixes to include regression testing to assure that all critical functionalities are adequately tested in an operationally relevant environment. Additionally, testing should assess end-to-end exchanges with ABIS customers under various load and threat conditions; demonstrate adequate reporting tools, processes, and procedures for providing system health
98
DoD ABIS
and monitoring metrics; and demonstrate suitable software maintenance procedures in an operationally realistic environment. 3. Conduct an IOT&E of ABIS 1.2 to confirm operational effectiveness, operational suitability, and survivability. 4. If ABIS 1.2 is not likely to successfully complete IOT&E by May 2014: -- Consider utilizing ABIS 1.0 as the baseline system for BEC 0 to become a program of record -- Modernize ABIS 1.0 to eliminate its dependency on Windows XP® and other hardware or software reaching end-of-life -- Conduct a baseline assessment, to include the definition of external interfaces to the current system and customers 5. Institutionalize a formal standards conformance program, listing external systems that have been independently verified to be interoperable with the biometrics enterprise.
ARMY PROGRAMS
Global Combat Support System – Army (GCSS-Army) Executive Summary • The Army conducted a Lead Site Verification Test (LSVT) for Global Combat Support System – Army (GCSS-Army) with the Army National Guard, Army Reserves, and Directorate of Logistics during November 2012 through March 2013. • DOT&E reported on the results of the LSVT in June 2013 and evaluated the system to be effective for users in the Army National Guard, Army Reserves, and Directorate of Logistics performing retail supply missions and associated financial transactions. The system is suitable but needs improvements in training and system usability. • GCSS-Army continues to work on achieving financial auditability no later than 2017 in accordance with the 2010 National Defense Authorization Act (NDAA) requirements. System • GCSS-Army is an information technology system made up of commercial off-the-shelf and government off-the-shelf software and hardware. • The core functionality of GCSS-Army comes from the adaptation of a commercially-available Enterprise Resource Planning (ERP) system. The ERP system integrates internal and external management information across an entire organization, including finance/accounting, manufacturing, sales and service, and customer relationship management. The ERP software centralizes and standardizes these activities, and it provides automation to assist users with common tasks (such as reporting). • The hardware component of GCSS-Army is limited to the production server in Redstone, Alabama, and the Continuity of Operation server in Radford, Virginia. • The GCSS-Army program includes the Army Enterprise Systems Integration Program that provides the enterprise hub services, centralized master data management, and cross-functional business intelligence and analytics for the Army ERP solutions, including the General Fund Enterprise Business System and Logistics Modernization Program.
Activity • The Army Test and Evaluation Command (ATEC) tested Wave 1 functions during the LSVT in accordance with the DOT&E-approved Test and Evaluation Master Plan. However, per DOT&E’s guidance for Business and Information systems, LSVT was a level 1 test and did not require DOT&E’s approval of the test plan. • From November 2012 through March 2013, users from the National Guard, Army Reserves, and the Directorate
• Based on the revised acquisition strategy, the Army will field GCSS-Army in two waves. Wave 1 only deploys the retail supply function with the associated financial functions. Wave 2 will field the rest of the functionalities to all users. • GCSS-Army executes finance actions and thus is subject to the 2010 NDAA requirements to be auditable by 2017. Mission Army logisticians will use this system to access information and exchange operational logistics data related to tactical maintenance, materiel management, property accountability, tactical financial management, and logistics planning. Major Contractors • ERP Solution Component: Northrop Grumman Space and Mission Systems Division – McLean, Virginia • Army Enterprise Systems Integration Program Component: Computer Sciences Corporation – Falls Church, Virginia
of Logistics participated in the LSVT during their normal operations, which allowed ATEC to collect data. • In accordance with the approved acquisition strategy, the Program Office will release Wave 2 functions in multiple production releases. DOT&E will review ATEC’s recommendations regarding the scope of operational test for each production release. The recommendations are based on the risk assessment as documented in the DOT&E Guidelines
GCSS-Army
99
Army PROGRAMS for Operational Test and Evaluation of Information and Business Systems. • DOT&E agreed with ATEC’s recommendation to conduct a test for Production Release 8, the first of the production releases for Wave 2. Production Release 8 fixed four software problems observed during the LSVT and did not add any new functionality. ATEC reported on Production Release 8 in an August 2013 assessment. Assessment • DOT&E reported GCSS-Army to be operationally effective and suitable after the IOT&E in June 2012. • The LSVT augmented the IOT&E results with data from five units that represented the Army National Guard, Army Reserves, and Directorate of Logistics. These units were not fielded at the time of the IOT&E. In a report released in June 2013, DOT&E evaluated GCSS-Army to be operationally effective for units performing Wave 1 functions but documented software shortfalls. These software problems did not prevent mission accomplishment but caused users to spend additional time and effort. The Program Office keeps track of problems and recommendations for software improvement, and implements fixes in new releases based on the priorities and available resources. GCSS-Army was suitable and needed improvement on training and system usability. • Production Release 8 was effective in fixing four of the software problems found during the LSVT.
100
GCSS-Army
• The 2010 NDAA requires financial audibility by 2017. GCSS-Army has not achieved this requirement at the time of this report. DOT&E will work with the Program Office to implement a financial Red Team test of system vulnerability to cybercrime. Recommendations • Status of Previous Recommendations. The Army began addressing the three previous recommendations but further action is required. The Army still should: 1. Take steps to achieve financial auditability no later than 2017. 2. Continue to collect data for computational (server capacity, storage, and bandwidth) and human factors (help desk responsiveness, overhead labor and communication costs, and data noise) impacts of an increased user base. Use such data to establish a pattern of demand on the system, so that future demand can be adequately anticipated and resourced as more users come online. 3. Conduct test and evaluation when the software is developed for Army Reserves and National Guard units in accordance with the September 2010 DOT&E-published Guidelines for Operational Test and Evaluation of Information and Business Systems memorandum. • FY13 Recommendations. None.
ARMY PROGRAMS
Handheld, Manpack, and Small Form Fit (HMS) Manpack Radio Executive Summary • In May 2012, the Army Test and Evaluation Command conducted the Manpack radio Multi-Service Operational Test and Evaluation (MOT&E) as a part of its Network Integration Evaluation 12.2. - DOT&E assessed the Manpack radio as not operationally effective due to the poor performance of the Single Channel Ground and Airborne Radio System (SINCGARS) waveform and not operationally suitable due to a failure to meet reliability or availability requirements. - In September 2012, the Army conducted Government Development Test (GDT) 3 to demonstrate improvements in MOT&E deficiencies. During GDT 3, the Manpack’s SINCGARS performance improved but it continued to exhibit poor reliability. • In October 2012, the Defense Acquisition Executive (DAE) approved a second low-rate initial production (LRIP) decision for 3,726 Manpack radios, increasing the total LRIP procurement to 3,826 radios. The DAE directed the Army to conduct a full and open competition for future Manpack radio procurements. • During July and August 2013, the Army began fielding Manpack radios. • In December 2013, the DAE approved an additional LRIP lot of 1,500 Manpack radios, increasing the total LRIP procurement to 5,326 radios. • The Army continues preparation for a Manpack radio FOT&E in 2014. The Army is planning to conduct an IOT&E to support the Full-Rate Production Decision Review for the Manpack radio that will be chosen as a part of the full and open competition. System • The Handheld, Manpack, and Small Form Fit (HMS) program evolved from the Joint Tactical Radio System program and provides software-programmable digital radios to support tactical communications requirements.
Activity • In May 2012, the Army conducted the Manpack radio MOT&E as part of the Network Integration Evaluation 12.2 at White Sands Missile Range, New Mexico, in accordance with a DOT&E-approved test plan. • The Army conducted Manpack radio GDT 3 in September 2012 to verify fixes to reliability and performance deficiencies found during the MOT&E and previous GDTs.
• The Manpack radio is a two-channel radio with military GPS that: - Is capable of operating at various transmission frequencies using the Soldier Radio Waveform (SRW), the legacy SINCGARS waveform, and current military satellite communications waveforms - Allows Soldiers to participate in voice and data communications networks and transmit Position Location Information - Hosts the Mobile User Objective Satellite waveform - Operates up to 20 watts at maximum power output Mission Army commanders use Manpack radios to: • Provide networked communications for host vehicles and dismounted Soldiers during all aspects of military operations • Communicate and create networks to exchange voice, video, and data using legacy waveforms or the SRW • Share voice and data between two different communications networks Major Contractors • General Dynamics, C4 Systems – Scottsdale, Arizona • Rockwell Collins – Cedar Rapids, Iowa
• On October 11, 2012, the DAE approved a second LRIP for Manpack radios, increasing the total LRIP quantity to 3,826. The DAE also directed the Army to conduct a full and open competition for future Manpack radio procurements. On December 12, 2013, the DAE approved an additional LRIP lot of 1,500 Manpack radios. The 5,326 radios procured through LRIP are 7 percent of the acquisition objective.
HMS Manpack
101
Army PROGRAMS • The Manpack radio will be re-competed in a full and open competition. The chosen Manpack will need to be operationally tested prior to the Full-Rate Production decision. • The Army is developing an HMS Acquisition Strategy and an HMS Manpack Test and Evaluation Master Plan, required for future developmental and operational testing. • The Army continues preparation for a Manpack radio FOT&E to operationally test fixes to deficiencies noted during the MOT&E and capabilities that have not yet been tested. • In July and August 2013, the Army fielded Manpack radios to the 101st Airborne Division, despite the radio’s demonstrated deficiencies. Assessment • During the Manpack radio MOT&E: - The Manpack radio was not operationally effective due to the poor voice quality and limited range of the SINCGARS waveform compared to legacy SINCGARS radios. - The SRW performance was good and the Soldiers were able to employ the Manpack radio for intra-company voice and data communications. - The Manpack radio was not operationally suitable and demonstrated poor reliability and poor availability. - The Army’s integration of the radios into Mine Resistant Ambush Protected vehicles was poor and reduced the radio’s performance. • During GDT 3 in late FY12 (intended to verify fixes to reliability and performance deficiencies found during the MOT&E and previous GDTs), the Manpack radio demonstrated improved SINGCARS performance under benign conditions. Reliability shortfalls continued. The SRW waveform Mean Time Between Essential Function Failure was 177 hours compared to the Manpack requirement of 477 hours.
102
HMS Manpack
This results in a 66 percent chance of completing a 72-hour mission compared to a requirement of 86 percent. • The Manpack radio has not yet demonstrated improvements in a realistic operational test environment. • The Army has fielded Manpack radios as part of a schedule‑driven plan without apparent concern about performance deficiencies. Units are receiving Manpack radios that may have performance deficiencies. Recommendations • Status of Previous Recommendations. The HMS program addressed the previous recommendation to perform a reliability growth analysis to assess Manpack radio maturity, but has not yet provided a detailed plan for achieving required reliability. The Army is addressing the previous recommendation to complete necessary Manpack radio documentation to support future developmental and operational testing by developing an acquisition strategy and Test and Evaluation Master Plan. • FY13 Recommendations. The Army should: 1. Ensure units currently equipped with Manpack radios understand the radio’s effectiveness and suitability limitations. 2. Correct deficiencies noted during the May 2012 Manpack radio MOT&E and conduct an operational test as soon as possible. 3. Ensure that adequate developmental testing is performed prior to future operational tests. 4. Use the reliability growth analysis assessing the Manpack radio maturity to develop a detailed plan for achieving required reliability. 5. Complete necessary Manpack radio documentation to support future developmental and operational testing.
ARMY PROGRAMS
Handheld, Manpack, and Small Form Fit (HMS) Rifleman Radio Executive Summary • In November 2011, the Army conducted an IOT&E for AN/ PRC-154 Rifleman Radio intended to support a Full- Rate Production Decision. DOT&E assessed the AN/ PRC-154 to be operationally effective with poor reliability. • From February through April 2012, the Army conducted Governmental Developmental Test (GDT) 2.3 and GDT 2.3a to complete developmental testing normally completed prior to IOT&E. • In May 2012, the Defense Acquisition Executive (DAE) directed the Army to change the Rifleman Radio acquisition strategy to conduct a full and open competition. The DAE approved a second low-rate initial production (LRIP) purchase of 13,077 AN/PRC-154 radios. • In 3QFY13, the Army approved an engineering change proposal to the Rifleman Radio LRIP contract to purchase AN/ PRC-154A radios instead of AN/PRC-154 radios. The AN/PRC-154A Rifleman Radios have encryption capabilities to enable Secret and below communications. The AN/ PRC- 154 radios are not capable of Secret-level encryption. • The Rifleman Radio program is schedule-driven. The Army did not complete developmental testing prior to IOT&E, fielded the AN/PRC-154 without verifying that problems discovered during IOT&E were fixed, and is planning to field the AN/PRC-154A in early FY14 prior to completing dedicated operational testing. System • The Army’s Handheld, Manpack, and Small Form Fit program evolved from the Joint Tactical Radio System program and provides software-programmable digital radios to support the Army’s tactical communications requirements. • The Rifleman Radio is a handheld, networking radio. The AN/PRC-154 variant of the Rifleman Radio was designed with National Security Agency Type 2 encryption suitable for unclassified communications and data transfer. In 2013, the Army approved an engineering change proposal to the Rifleman Radio contract to begin procuring AN/PRC-154A variants with National Security Agency Type 1 encryption suitable for Secret communications and data. In addition to
Activity • On May 23, 2012, the DAE directed the Army to change the Rifleman Radio acquisition strategy to require a full and open
functioning as a stand-alone, handheld radio, the Army intends the AN/PRC-154A variant to be the radio used as part of the Nett Warrior program. • Both the Secret and unclassified variants of the Rifleman Radio are single-channel radios with a commercial GPS receiver that: - Operate at various transmission frequencies using the Soldier Radio Waveform (SRW), which enables the radios to form an ad-hoc data and voice communications network with other SRW-capable radios - Provide 5 watts maximum power output - Allow Soldiers to transmit Position Location Information across the SRW network Mission Army leaders and Soldiers use Rifleman Radios to communicate and create networks to exchange voice, video, and data using the SRW during all aspects of military operations. Major Contractors • General Dynamics, C4 Systems – Scottsdale, Arizona • Thales Communications, Inc. – Clarksburg, Maryland
competition and approved a second LRIP for AN/PRC-154 Rifleman Radios.
HMS Rifleman
103
Army PROGRAMS • The AN/PRC-154 Rifleman Radio does not have the encryption required to handle Secret data. In 3QFY13, the Army approved an engineering change proposal to the Rifleman Radio LRIP contract in order to buy radios capable of transmitting encrypted Secret data. - The Army stopped buying AN/PRC-154 radios and began acquiring Secret-capable AN/PRC-154A radios. - The AN/PRC-154As have both hardware and software upgrades from the AN/PRC-154, and have not had a dedicated operational test. The AN/PRC-154A was tested as part of the Nett Warrior Limited User Test (LUT) in May 2013 during Network Integration Evaluation 13.2 in accordance with a DOT&E-approved test plan. • The Army currently plans to test the AN/PRC-154A in a GDT followed by an IOT&E during FY14. • The Army plans to field the AN/PRC-154A in early FY14, prior to the planned IOT&E. • The Army continues preparation for a future Rifleman Radio operational test that will be conducted on the Rifleman Radio chosen during the full and open competition required by the DAE. Assessment • During Network Integration Evaluation 13.2 in May 2013, the AN/PRC-154A radio was part of the Nett Warrior system. As employed during the Nett Warrior LUT, the AN/PRC-154A: - Provided situational awareness and communications to leaders equipped with Nett Warrior - Demonstrated voice degradation at ranges greater than 500 meters - Did not support the full mission of a Cavalry Troop due to inconsistent communications and insufficient range for their operations - Demonstrated numerous suitability issues that contributed to Soldiers concluding that this radio was not yet acceptable for combat in its current Nett Warrior configuration-▪▪ Spontaneous rebooting ▪▪ Taking excessive time to rejoin the radio network ▪▪ Lack of a display screen for radio status ▪▪ Battery overheating and rapid battery depletion
104
HMS Rifleman
• The Army plans to field Rifleman Radio as part of a schedule- driven capability set. As a result: - The Army did not perform the necessary developmental testing required to ensure performance was known prior to the AN/PRC-154 Rifleman Radio IOT&E conducted in FY11. The first developmental test event was conducted prior to IOT&E. The Army conducted the remaining two planned developmental test events several months after the operational test. - The Army conducted the Nett Warrior LUT using the AN/ PRC-154A radio with insufficient developmental testing. - The Army has fielded the AN/PRC-154 radio without an operational test demonstrating fixes to the problems discovered during the FY11 IOT&E. - The Army plans to field the AN/PRC-154A in early FY14, prior to any dedicated operational testing, and despite the performance and suitability deficiencies the radio demonstrated during the Nett Warrior LUT. Recommendations • Status of Previous Recommendations. The Army is addressing the previous recommendation to complete necessary Rifleman Radio documentation to support future developmental and operational testing by developing an acquisition strategy and Test and Evaluation Master Plan. • FY13 Recommendations. The Army should: 1. Ensure the AN/PRC-154A performance and suitability problems experienced during the Nett Warrior LUT are addressed prior to fielding the radio. 2. Conduct dedicated operational testing of the AN/PRC-154A as soon as possible to characterize the performance of the radio to be fielded. 3. Ensure that adequate developmental testing is performed prior to future operational tests. 4. Complete necessary Rifleman Radio documentation, including a Test and Evaluation Master Plan, to support future developmental and operational testing.
ARMY PROGRAMS
Joint Battle Command – Platform (JBC-P) Executive Summary • In 2012, the Army approved a Joint Battle Command – Platform (JBC-P) Milestone C based upon contractor testing, developmental testing, and a May 2012 customer test. • Based upon an October through November 2012 customer test, the Army Test and Evaluation Command assessed JBC-P software build 4 as: - Not effective due to message completion rate deficiencies and problems sending and receiving orders - Not suitable due to poor reliability - Not survivable due to cyber security vulnerabilities • In May 2013, the Army conducted a JBC-P IOT&E to support a planned 1QFY14 Full-Rate Production (FRP) decision. The IOT&E assessed JBC-P software build 5 as employed by the 2nd Brigade, 1st Armored Division executing a variety of missions under operationally realistic conditions. • Based on results from the 2013 IOT&E, DOT&E released a JBC-P IOT&E report in November 2013, which assessed JBC-P as: - Operationally effective in supporting Army commanders and Soldiers with situational awareness, command and control (C2) messages, and chat when operating from Tactical Operational Centers (TOCs) and on-the-move in tactical vehicles. JBC-P served as the Soldiers’ primary tool for C2 when on-the-move. - Not operationally suitable due to poor reliability (less than the Army’s reduced requirement) and deficiencies in training provided to Soldiers. - Not survivable due to Information Assurance vulnerabilities. • During October through November 2013, the Army conducted a JBC-P software build 5.1 customer test to demonstrate correction of IOT&E deficiencies to support an intended 1QFY14 FRP decision. • The Army and Marine Corps plan to conduct a JBC-P software build 6 Multi-Service Operational Test and Evaluation (MOT&E) from May through June 2014. System • JBC-P is a networked battle command information system that enables units to share near real-time friendly and enemy situational awareness information, operational maps and graphics, and C2 messages. The Army and Marine Corps intend JBC-P to achieve platform-level interoperability for ground vehicles, dismounted Soldiers/Marines, and
aviation assets operating in land/littoral and joint operational environments. • JBC-P is an upgrade from Force XXI Battle Command Brigade and Below Joint Capabilities Release and provides the following improvements: - Tactical chat combined with chat room capability, providing enhanced collaboration for commanders - Improved mission command applications for planning and execution - A more intuitive graphical user interface with improved display of maps and images - Enhanced blue force situational awareness between mobile platforms, TOCs, and dismounted Soldiers equipped with Nett Warrior • JBC-P is fielded in both mobile and command post versions. JBC-P communications is supported by Blue Force Tracker 2 satellite for mobile operations, and the Tactical Internet for command post operations Mission Army, Marine Corps, and Special Operations Forces commanders use JBC-P to provide integrated, on-the-move, near real-time battle command information and situational awareness from brigade to maneuver platform to dismounted Soldiers/Marines. Major Contractor Software Engineering Directorate, U.S. Army Aviation & Missile Research, Development & Engineering Center – Huntsville, Alabama
JBC-P
105
Army PROGRAMS Activity • In July 2012, the Program Executive Office, Command Control Communications Tactical, as the JBC-P Milestone Decision Authority, approved the program’s Milestone C. This decision was based upon developmental and contractor testing of JBC-P software build 3. • During the October through November 2012 Network Integration Evaluation (NIE) 13.1, the Army conducted a JBC-P software build 4 customer test. • During the May 2013 NIE 13.2, the Army conducted a JBC-P IOT&E in accordance with a DOT&E-approved test plan to support a planned 1QFY14 FRP decision. The Army tested JBC-P software build 5 as employed by the 2nd Brigade, 1st Armored Division executing a variety of missions under operationally realistic conditions. • From July through September 2013, the Army conducted a lab‑based Risk Reduction Event 13 to demonstrate correction of IOT&E deficiencies and to prepare JBC-P software build 5.1 for NIE 14.1. • During the October through November 2013 NIE 14.1, the Army conducted a JBC-P software build 5.1 customer test to demonstrate correction of IOT&E deficiencies to support an intended 1QFY14 FRP decision. • DOT&E released an IOT&E report in November 2013. • The Army is planning a JBC-P MOT&E during the May through June 2014 NIE 14.2. The Army and Marine Corps intend to participate in the MOT&E and use the results to support JBC-P software build 6 fielding decisions. Assessment • During the 2012 NIE 13.1 customer test, the Army Test and Evaluation Command assessed JBC-P build 4 as not effective, not suitable, and not survivable. During the test, JBC-P: - Exceeded requirements for shared blue (friendly) situational awareness - Did not meet its survivability entity data (battlefield hazards) requirements - Demonstrated Network Operations Center reliability and interoperability problems - Did not meet its requirement to send and receive operations orders, fragmentary orders, and graphics - Did not meet its reliability requirement, demonstrating a Mean Time Between Essential Function Failure of 88 hours for the Joint Platform Tablet and 389 hours for Joint Version 5 (JV-5) Block I and II computers against the Army’s requirement of 477 hours - Demonstrated significant survivability risks from threat computer network operations • Based on the May 2013 IOT&E, DOT&E assessed the JBC-P as operationally effective for combat operations. During IOT&E, JBC-P:
106
JBC-P
- Exceeded message completion and timeliness requirements for situational awareness and survivability data - Demonstrated the ability to pass C2 messages - Provided effective chat communications between all echelons of the brigade - Was used by Soldiers as their primary tool for maintaining extended communications while on-the-move - Cluttered its map display with numerous battlefield hazard icons, which confused Soldiers • Based upon IOT&E, DOT&E assessed JBC-P as not operationally suitable and highlighted the following deficiencies: - JBC-P did not meet its reliability requirement, demonstrating a Mean Time Between Essential Function Failure of 74 hours for Joint Platform Tablet, and 59 and 82 hours respectively for JV-5 Block I and II computers against an Army-reduced requirement of 290 hours. - Seventy-eight percent of Essential Function Failures were due to software problems, with the largest share due to KGV-72 Encryption Device communications authentication failures. - Training provided to Soldiers did not prepare the unit to use all the capabilities of JBC-P. Soldiers required more hands-on training, extended leader training, and improved training on the KGV-72 Encryption Device. - Spontaneous JBC-P reboots reduced the Soldiers’ confidence in the system. • The JBC-P IOT&E demonstrated the system as not survivable against threat computer network operations. While improved from NIE 13.1, the Army needs to improve JBC-P’s cyber security. • The Army completed JBC-P Risk Reduction Event 13 and demonstrated many corrections of IOT&E deficiencies under benign lab-based conditions. Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. The Army should: 1. Improve JBC-P reliability to include improving the ability of the JBC-P software to maintain encryption device synchronization and correcting the spontaneous reboot deficiency. 2. Improve JBC-P leader and Soldier training. 3. Improve the JBC-P map icon display to provide relevant and uncluttered information on battlefield hazards. 4. Correct cyber security survivability deficiencies demonstrated during the JBC-P IOT&E.
ARMY PROGRAMS
Joint Land Attack Cruise Missile Defense Elevated Netted Sensor System (JLENS) Executive Summary • The Joint Land Attack Cruise Missile Defense Elevated Netted Sensor System (JLENS) program experienced a critical Nunn-McCurdy cost breach due to an FY12 budget decision to eliminate procurement of all production systems. USD(AT&L) directed the Army to continue with a limited development program through 2013, using the two existing JLENS orbits. The Army executed a reduced JLENS test program to evaluate JLENS technologies and capabilities as directed by USD(AT&L). • In 1QFY13 and 3QFY13, the Army Test and Evaluation Command (ATEC) conducted two Early User Tests (EUTs) of JLENS operational capabilities. The tests assessed JLENS orbit-level performance during missions involving fighters, rotary-wing aircraft, unmanned aerial vehicles, drones (cruise missile surrogates), and land-based surface moving targets. Soldier operators conducted missions with extensive contractor support. • To date, there have been four developmental test events that demonstrated a potential capability to perform Integrated Fire Control (IFC). - In April 2012, JLENS supported an IFC live missile flight test that resulted in a successful intercept of a fixed-wing target drone aircraft in a controlled test environment. - In September 2012, the Army and Navy conducted a joint JLENS Navy Integrated Fire Control-Counter Air (NIFC-CA) missile flight test event. JLENS provided IFC targeting information to a Navy Aegis-based missile system using Cooperative Engagement Capability datalinks in a controlled test environment. - In July 2013, JLENS supported an IFC test with an Air Force F-15 targeting a drone in a controlled test environment. - In August 2013, JLENS supported a Weapons System Evaluation Program event. JLENS provided IFC targeting information to Air Force fighters and operational Army air defense missile systems during multiple live missile flight tests in a controlled test environment. • Testing has been limited in scope, leading to a restricted demonstration of JLENS’s capabilities. Interoperability has been minimally tested and demonstrated. Test range restrictions (to include target profile limits due to safety and Federal Aviation Administration restrictions) have limited the target tracking/detection range demonstration and target profiles, resulting in an incomplete demonstration of requirements and performance in an operational environment. All testing occurred in remote, mid-country locations; therefore, no testing occurred in an operationally
•
•
•
•
realistic over-water environment or in its intended Electronic Environmental Effects environment. Testing showed that the fire control radar consistently provided fire control quality tracking data that were sufficient to support air defense missile engagements. The system also demonstrated a limited target identification capability that partially met requirements and basic interoperability with other air defense systems. Based on data collected during developmental testing, JLENS system-level reliability is not meeting program reliability growth goals. Both software and hardware reliability problems contribute to low system reliability. Based on limited data collected during two EUTs, JLENS has not demonstrated the ability to survive in the domains of Electronic Environment Effects; Information Assurance; electronic attack; and chemical, biological, radiological, and nuclear effects. The Joint Staff directed the JLENS program to deploy an orbit to Aberdeen Proving Grounds, Maryland, to participate in a three-year Northern Command (NORTHCOM) homeland defense exercise from FY15-FY17.
System • A JLENS orbit consists of surveillance and fire control radar systems. Each radar system is mounted separately on 74-meter tethered aerostat balloons that operate at altitudes up to 10,000 feet above mean sea level. • An 180,000-pound mobile mooring station and tether system is used to launch, recover, and secure each aerostat system. The aerostat tether system provides radar control and data transfer links to supporting ground control and data processing
JLENS
107
Army PROGRAMS stations. JLENS is deployable to pre-planned operational sites that have been prepared to support mobile mooring station operations. Five days are required to transition between fully operational status and a transportable configuration. Operators control the radar, process data, and transmit radar track information from mobile communication and control stations co-located with the mobile mooring stations. A mobile power generation and distribution system and associated ground equipment support each JLENS orbit. Mission • Army air defense units equipped with the JLENS provide persistent air and missile threat warning to friendly forces,
Activity • The JLENS program experienced a critical Nunn-McCurdy cost breach due to an FY12 budget decision to eliminate procurement of all production systems. Following a Nunn-McCurdy review, USD(AT&L) rescinded the JLENS Acquisition Program Baseline and directed the Army to continue with a reduced JLENS test program using the two existing JLENS developmental orbits. The focus of the reduced test program is to improve airborne and surface moving target capabilities in advance of JLENS participation in an FY15-17 NORTHCOM Combatant Commander exercise. USD(AT&L) did not authorize the program to complete the previously planned system development program or to proceed to a Milestone C or production decision. • Based on the USD(AT&L) direction, the Army revised the JLENS operational test strategy, transitioning from the previously planned Milestone C operational assessment to two EUTs (EUT1 and EUT2) in FY13. The purpose of the EUT events was to assess JLENS operational capabilities and limitations in advance of the FY15 Combatant Commander exercise. During these limited assessments, Soldier operators conducted missions with significant contractor support. Contractor personnel provided the vast majority of system maintenance support. • The timing of the second EUT was delayed three months due to difficulties integrating the JLENS Stimulator (JSTM). JSTM was a modeling and simulation tool that was planned to enable testing of robust threat scenarios that could not be replicated through live target testing. Ultimately, the Program Office was not able to successfully integrate JSTM, so EUT2 proceeded without these scenarios. • ATEC conducted EUT1 and EUT2 (occurring in 1QFY13 and 3QFY13, respectively) at the Utah Test and Training Range. ATEC assessed JLENS orbit-level performance during missions involving fighters, rotary-wing aircraft, unmanned aerial vehicles, drones, and land-based surface moving targets. Test range restrictions (to include target profile limits due to safety and Federal Aviation Administration restrictions) limited identification friend or foe testing, tracking/detection
108
JLENS
target identification, target cueing for airborne interceptor aircraft, and precision targeting information to ground-based air defense weapons systems. • Primary JLENS air-breathing targets include all fixed- and rotary-wing aircraft, unmanned aerial vehicles, and land attack cruise missiles. Secondary targets include surface moving targets, large rockets, and tactical ballistic missiles. Major Contractor Raytheon Integrated Defense Systems – Andover, Massachusetts
ranges, and target flight profiles. These test restrictions limited both operational realism and requirements demonstration. • There have been four live missile flight tests in which JLENS has provided integrated fire support to different platforms. - During the developmental IFC test phase in April 2012, an integrated JLENS orbit supported a series of simulated missile flight test engagements of airborne targets with an operational Army air defense missile system. This phase concluded with an IFC live missile flight test that resulted in a successful intercept of a fixed-wing target drone aircraft in a controlled test environment. - The Army and Navy conducted a joint JLENS Navy NIFC-CA missile flight test event at White Sands Missile Range, New Mexico, in late September 2012. The JLENS provided IFC targeting information to a Navy Aegis-based missile system using Cooperative Engagement Capability datalinks to engage and destroy a surrogate cruise missile aerial target. - During July 2013, JLENS supported an IFC test with an Air Force F-15 targeting a drone. - In August 2013, JLENS supported a Weapons System Evaluation Program event. JLENS provided IFC targeting information to Air Force fighters and operational Army air defense missile systems during multiple live missile flight tests in a controlled test environment. Since no further JLENS production is currently authorized, a JLENS IOT&E event will not be required. However, DOT&E will remain involved throughout the Combatant Commander exercise (FY15-FY17) in order to facilitate the NORTHCOM assessment of JLENS. Assessment • Since the Nunn-McCurdy breach, testing has been limited in scope, leading to restricted demonstration of JLENS’s capabilities. Interoperability has been minimally tested and demonstrated. Test range restrictions have limited the target tracking/detection range demonstration and target profiles, resulting in an incomplete demonstration of requirements
Army PROGRAMS and performance in an operational environment. All testing occurred in remote, mid-country locations; therefore, no testing occurred in an operationally realistic over-water environment or in its intended Electronic Environmental Effects testing environment. • Testing showed that the fire control radar consistently provided fire control quality tracking data that were sufficient to support air defense missile engagements. The system also demonstrated a limited target identification capability that partially met requirements and basic interoperability with other air defense systems. Testing included a successful demonstration of the fully-deployed aerostat tether system, including power and fiber-optic data transmission paths. Testing also identified critical performance areas for improvement to include: non-cooperative target recognition, friendly aircraft identification capabilities, and target track consistency. • During four flight tests, JLENS demonstrated a potential capability to perform over-land IFC. The four demonstrations occurred during developmental testing and involved target flight-path restrictions and an operationally unrealistic test environment. Test equipment that is not part of the JLENS system was required during the IFC demonstrations. • Based on data collected during developmental testing, JLENS system-level reliability is not meeting program growth goals. The system does not meet the requirements for Operational
Availability, Mean Time to Repair, or Mean Time Between System Abort. Both software and hardware reliability problems contribute to low system reliability and availability. • While JLENS is intended to provide 24-hour coverage, weather can limit system availability and performance. Poor weather may reduce radar detection performance or require the aerostats be returned to the surface. • JLENS did not demonstrate the ability to survive in its intended operational environment. Electronic Environmental Effects testing was limited and revealed several anomalies affecting mission critical systems. Information Assurance testing was limited to a vulnerability assessment and resulted in the system being deemed not ready for a Red Team assessment. Testing against electronic attack was very limited and did not include most techniques the system would likely encounter. Recommendations • Status of Previous Recommendations. The Army has addressed one of the two previous recommendations. However, while system-level reliability has improved, it does not meet system requirements; therefore, the Army should still develop a reliability improvement plan. • FY13 Recommendation. 1. The Army should ensure Soldiers are properly trained to support the NORTHCOM Homeland Defense Exercise.
JLENS
109
Army PROGRAMS
110
ARMY PROGRAMS
Joint Light Tactical Vehicle (JLTV) Family of Vehicles (FoV) Executive Summary • From November 2012 through August 2013, the Army conducted early ballistic testing of the Joint Light Tactical Vehicle (JLTV) designs. - These tests identified vulnerabilities in crew protection, which the contractors are addressing. - Early tests indicate that some of the threshold-level force protection Key Performance Parameters (KPPs) may be achievable but a full assessment of the results is not yet complete. • In August 2013, the contractors delivered 22 full-up prototypes per contractor for developmental, live fire, and operational testing. The program plans to begin live fire testing in November 2013 and developmental/operational testing in April 2014. • The government began automotive and Reliability, Availability, and Maintenance (RAM) testing in October 2013 at Aberdeen Test Center (ATC), Maryland, and Yuma Proving Ground, Arizona. The objective is to uncover failure modes, implement corrective actions, and assess whether the JLTV vehicles can meet the Mean Miles Between Operational Mission Failure requirement prior to the Milestone C decision. This testing will continue until June 2014. System • The JLTV Family of Vehicles (FoV) is the Marine Corps and Army partial replacement for the High Mobility Multi-purpose Wheeled Vehicle (HMMWV). The Services intend the JLTV to provide increased crew protection against IED and underbody attacks, improved mobility, and higher reliability than the HMMWV. • The JLTV FoV consists of two vehicle categories: the JLTV Combat Tactical Vehicle, designed to seat four passengers; and the JLTV Combat Support Vehicle, designed to seat two passengers. • The JLTV Combat Tactical Vehicle has a 3,500-pound payload and three mission package configurations: - Close Combat Weapons Carrier Vehicle - General Purpose Vehicle - Heavy Guns Carrier Vehicle • The JLTV Combat Support Vehicle has a 5,100-pound payload and two mission package configurations: - Utility Prime Mover - Shelter Carrier • The JLTV program is using a competitive prototype acquisition strategy. During the Engineering and Manufacturing Development phase, the program will test three contractors’ FoVs.
Mission • Military units will employ JLTV as a light tactical wheeled vehicle to support all types of military operations. JLTVs will be used by airborne, air assault, light, Stryker, and heavy forces as reconnaissance, maneuver, and maneuver sustainment platforms. • Small ground combat units will employ JLTV in combat patrols, raids, long-range reconnaissance, and convoy escort.
JLTV FoV
111
Army PROGRAMS Major Contractors • AM General – South Bend, Indiana • Lockheed Martin Corporation – Dallas, Texas • Oshkosh Corporation – Oshkosh, Wisconsin
Activity • The program conducted contractor Design Understanding Reviews (DURs) from December 2012 through January 2013. The DURs appraised each contractor’s progress toward achieving threshold JLTV requirements and served as a means to identify technical challenges. • In 2QFY13, the JLTV contractors conducted user reviews that provided JLTV contractors with Soldiers and Marines input and recommended modifications to the designs. • In October 2012, the contractors delivered armor samples for live fire testing. The Army conducted testing of the armor samples from November 2012 through January 2013 at ATC. Due to sequestration, some armor sample testing will be postponed until the Production and Deployment phase. • In January 2013, the contractors delivered ballistic cabs for live fire testing. A ballistic cab is an armored crew compartment mounted on a representative vehicle chassis intended to provide early insights into ballistic vulnerabilities. The Army conducted testing of the ballistic cabs from March through August 2013 at ATC. • In August 2013, the contractors delivered 22 full-up prototypes per contractor for developmental, live fire, and operational testing. • JLTV prototypes completed 500 miles of break-in RAM testing and 1,000 miles of shakedown testing. The break-in testing is performed by the contractor to verify basic vehicle functionality. The shakedown testing is intended to ensure workmanship and infant-mortality problems are discovered and addressed. The contractors used the results of these tests to correct vehicle build and quality concerns prior to government testing. • The government began automotive testing on the JLTV vendor vehicles in October 2013 at ATC. The Army Test and Evaluation Command planned to start RAM testing of vendor vehicles in October 2013 at ATC and Yuma Proving Ground. This testing was delayed due to the Federal Government shutdown. The objective of the RAM testing is to uncover failure modes, implement corrective actions, and assess whether the vendor’s vehicles can meet the Mean Miles Between Operational Mission Failure requirement prior to the
112
JLTV FoV
Milestone C decision. This testing is planned to continue until June 2014. • The program began system-level live fire testing in November 2013. Eighteen ballistic test events will occur per contractor prior to the Milestone C decision. Due to sequestration, testing of the Automatic Fire Extinguisher Systems will be postponed until the Production and Deployment phase. • The developmental/operational testing is planned to begin in April 2014. Assessment • Based on the DUR and user reviews, all three contractors will have challenges satisfying the payload requirements to carry vehicle occupants with mission essential equipment, weapons, and sustainment loads. Visibility from the crew compartment is limited for all vendor vehicles due to small rear windows, positioning of window panels, and seating arrangements. • The planned reliability growth testing and corrective action periods provide limited time to identify and resolve failure modes prior to the Limited User Testing (LUT) planned for August 2014. • Early live fire testing of the armor samples and ballistic cabs identified vulnerabilities in crew protection. Contractors made design changes as they deemed appropriate. The program will re-test all survivability design changes during system-level testing and the design changes will be incorporated on the prototype vehicles for the developmental/operational testing and the LUT. • Early live fire test results indicate that the small arms, side- and underbody-detonated IED threshold-level force protection KPPs may be achievable. The system-level testing is required to make a final assessment of all threshold-level force protection KPPs. Recommendations • Status of Previous Recommendations. The Army has addressed all previous recommendations. • FY13 Recommendations. None.
ARMY PROGRAMS
M109 Family of Vehicles (FoV) Paladin Integrated Management (PIM) Executive Summary • In November 2012, the Army conducted the Paladin Integrated Management (PIM) Limited User Test (LUT) at Yuma Proving Ground (YPG), Arizona. Soldier crews operating 2 PIM Self-propelled Howitzers (SPHs) and 2 Carrier, Ammunition, Tracked (CAT) resupply vehicles fired 1,247 projectiles and drove 864 miles during 6 days of operational testing. - During the LUT, the delivery accuracy of the PIM-equipped unit firing unguided high-explosive projectiles was comparable to the accuracy of the Field Artillery unit that conducted the Paladin M109A6 FOT&E. - The LUT test unit was not as timely as units equipped with the current Paladin M109A6 Howitzer and did not achieve accuracy requirements when firing the M795 and M107 high-explosive projectiles at short ranges. - Performance of the SPH and CAT during the LUT indicates the program is meeting reliability growth estimates necessary for achieving operational suitability in the July 2016 IOT&E. • During post-LUT developmental testing, Soldier crews operating a PIM SPH with hardware and software changes intended to fix problems identified during the LUT demonstrated improved fire mission timeliness and rate-of-fire and emergency fire mission time standards. • Compatibility testing of a PIM SPH firing the Precision Guidance Kit (PGK) and the Excalibur precision munition demonstrated increased delivery accuracy. • As part of the LFT&E program, armor performance testing demonstrated PIM armor configurations provide required levels of threat protection. Ballistic Hull and Turret (BH&T) exploitation testing revealed vulnerable areas in the SPH and CAT. Underbody blast testing will not be accomplished until the 5A prototype and full-up systems are available.
System • The M109 Family of Vehicles (FoV) PIM consists of two vehicles, the SPH and CAT. • The M109 FoV SPH is a tracked, self-propelled 155 mm howitzer designed to improve sustainability over the legacy M109A6 howitzer fleet. The full-rate production howitzers will have a newly designed hull, modified Bradley Fighting Vehicle power train and suspension, and a high-voltage electrical system. The SPH is operated by a crew of 4 and can engage targets at ranges of 22 kilometers using standard projectiles and 30 kilometers using rocket-assisted projectiles. • The M109 FoV CAT supplies the SPH with ammunition. The full-rate production ammunition carriers will have a
•
•
•
•
•
common chassis with the SPH. The ammunition carriers are designed to carry 12,000 pounds of ammunition in various configurations and a crew of 4 Soldiers. The Army will equip the SPH and CAT with two armor configurations to meet two threshold requirements for force protection and survivability – Threshold 1 (T1) and Threshold 2 (T2). The base T1 armor configuration is integral to the SPH and CAT. The T2 configuration is intended to meet protection requirements beyond the T1 threshold with add-on armor kits. The Army plans to employ PIM vehicles in the T1 configuration during normal operations and will equip the SPH and CAT with T2 add-on armor kits during combat operations. The M109 FoV SPH can fire the PGK and the Excalibur precision munition to increase delivery accuracy. The Army developed the PGK to reduce the dispersion of unguided projectiles and the Excalibur precision munition to provide Field Artillery units a precision engagement capability. The Army intends to employ the M109 FoV as part of a Fires Battalion in the Armored Brigade Combat Team and Artillery Fires Brigades with the capability to support any Brigade Combat Team. The Army plans to field up to 557 sets of the M109 FoV with full-rate production planned for FY17.
Mission Field Artillery units employ the M109 FoV to destroy, defeat, or disrupt the enemy by providing integrated, massed, and precision indirect fire effects in support of maneuver units conducting unified land operations.
M109 PIM
113
Army PROGRAMS Major Contractor BAE Systems – York, Pennsylvania
Activity • In November 2012, the Army conducted the PIM LUT, in accordance with a DOT&E-approved test plan, at YPG, Arizona. The test unit executed two 72-hour scenarios at an operational tempo consistent with Armored Brigade Combat Team Operational Mode Summary/Mission Profile. Soldier crews operating 2 PIM SPHs and 2 CATs fired 1,247 projectiles and drove 864 miles during 6 days of operational testing. • In January 2013, the program began installing and testing a series of Corrective Actions, Producibility, and Obsolescence (CPO) changes for the SPH and CAT. CPO changes are the program’s systematic approach to address failures identified in operational and developmental testing, improvements in survivability and force protection, maintainability of obsolete government-furnished SPH and CAT components, and integration of Bradley common components. To mitigate risk, the program plans to design, procure, and test CPO changes on prototype vehicles prior to installation on low-rate initial production (LRIP) test articles. • During compatibility and Lot Acceptance Testing in February 2013 at YPG, Arizona, the program fired 6 Excalibur Increment Ia-2 projectiles from a PIM SPH at targets positioned 9, 25, and 35 kilometers from the howitzer. The radial miss distances for all 6 Excalibur projectiles were less than 10 meters. Previous compatibility testing with PIM SPH firing 3 high-explosive projectiles fuzed with the PGK at a target 15 kilometers from the howitzer resulted in an average radial miss distance of 24 meters. • In March 2013, post-LUT developmental testing for PIM accuracy, SPH crews fired 36 groups of 6 projectiles at targets positioned 4, 6, and 21.5 kilometers from the howitzer. Projectile groups fired at 21.5 kilometers met PIM accuracy requirements but groups fired at 4 and 6 kilometers did not meet requirements. • In early 3QFY13, the Army delayed the PIM Milestone C decision from June to October 2013 and revised the PIM LRIP schedule due to LRIP contract delays and reduction in program funding. The revised schedule reduces the program’s planned LRIP quantity of 72 sets to 66 sets and LRIP procurement period from 4 years to 3 years. The Army continues to plan for a Full-Rate Production decision with the first PIM unit equipped in January 2017. • DOT&E approved an updated PIM Test and Evaluation Master Plan (TEMP) on July 24, 2013. The TEMP update revised the test and evaluation strategy to support production and deployment phase testing. • Planned BH&T exploitation testing to characterize PIM protection provided by current armor configurations is ongoing. The program is addressing vulnerabilities identified
114
M109 PIM
in BH&T testing and developing plans to validate proposed corrective actions. Underbody blast testing will occur when the 5A prototype and full-up systems are available, currently planned for FY15-16. • The Army is designing and testing a separate underbelly kit (not a component of the T1 and T2 armor configurations) to determine the potential protection an SPH and CAT provide against equivalent IEDs encountered in Iraq and Afghanistan. Assessment • During the November 2012 LUT, the delivery accuracy of the PIM-equipped unit firing unguided high-explosive projectiles was comparable to the accuracy of the Field Artillery unit that conducted the Paladin M109A6 FOT&E. • The LUT test unit was not as timely as units equipped with the current Paladin M109A6 Howitzer, meeting less than 20 percent of conventional fire mission time standards. Rate-of-fire and emergency fire mission timeliness requirements were not achieved. Beginning in 2014, the program plans to make hardware and software improvements to increase rammer movement responsiveness and simultaneous movement with cannon tube elevation to improve PIM fire mission timeliness. • In 1QFY13 developmental testing using a PIM SPH with hardware and software changes intended to fix problems identified during the LUT, Soldier crews that participated in the LUT demonstrated improved fire mission timeliness and demonstrated rate-of-fire and emergency fire mission time standards. • During the LUT, the PIM SPH did not achieve accuracy requirements when firing the M795 and M107 high‑explosive projectiles at short ranges and demonstrated greater variance in accuracy at night when engaging short‑ and medium-range targets. • PIM SPH compatibility with the PGK and the Excalibur precision munition increases delivery accuracy. • The demonstrated reliability of the SPH and CAT indicates the program is meeting reliability growth estimates necessary for achieving operational suitability in the July 2016 IOT&E. • The program has identified seven critical path SPH and CAT CPO changes necessary to meet system performance requirements. Verification testing of the CPO modifications on prototype vehicles will reduce the risk of discovering new CPO-related failure modes on LRIP vehicles. • In LFT&E, armor performance testing demonstrated PIM armor configurations provide required levels of threat protection against fragmenting and other ballistic threats. BH&T testing revealed vulnerable areas in the SPH and CAT.
Army PROGRAMS Recommendations • Status of Previous Recommendations. The Army has satisfactorily addressed all previous recommendations. • FY13 Recommendations. The Army should: 1. Implement planned hardware and software improvements to increase rammer movement responsiveness and simultaneous movement with cannon tube elevation to improve PIM fire mission timeliness. 2. Continue testing upgraded suspension and transmission components to characterize impacts of increased weight from the add-on armor and underbelly kits.
3. Determine if the PIM Capabilities Production Document short-range accuracy requirements remain valid and continue efforts to improve PIM SPH accuracy. 4. Implement and validate planned armor configuration changes for LRIP vehicles prior to full-up system-level testing.
M109 PIM
115
Army PROGRAMS
116
ARMY PROGRAMS
Mk248 Mod 0 Sniper Round Executive Summary • Snipers will use the Mk248 Mod 0 cartridge in conjunction with the M2010 Enhanced Sniper Rifle (ESR) to defeat specified targets at greater ranges and with improved accuracy compared to current sniper systems. • DOT&E assessed the Mk248 Mod 0 as lethal. System • Army snipers use the Mk248 Mod 0 cartridge in conjunction with the M2010 ESR to engage enemy targets at extended ranges. • During Operation Enduring Freedom, the Army identified the need for an upgraded sniper rifle capable of firing at longer ranges and with improved accuracy than currently fielded sniper weapons. The Army determined the M2010 ESR, a reconfigured M24 Sniper Weapon System modified to fire a .300 caliber Winchester Magnum cartridge, was the preferred solution. • The Mk248 Mod 0 .300 caliber cartridge, fired from the M2010 ESR, will replace the use of the 7.62 mm M118LR cartridge fired from the M24 Sniper Weapon System.
Major Contractor Alliant Techsystems Inc. Federal Cartridge Company – Anoka, Minnesota
Mission Snipers firing the Mk 248 Mod 0 cartridge with the M2010 ESR will engage designated enemy targets in accordance with applicable tactics, techniques, and procedures.
Activity • The Army successfully completed live fire testing of the Mk248 Mod 0 in March 2013. Testing was conducted in accordance with the DOT&E-approved live fire strategy and test plans. • The Army used gelatin targets to obtain data as inputs for complex computer modeling of Mk248 Mod 0 performance. Testing also included shots against light material barriers and other targets to determine the projectile’s ability to perforate the target. • DOT&E published a classified lethality report for the Mk248 Mod 0 in June 2013.
• The June 2013 classified lethality report contains additional assessment details. Recommendation • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendation. 1. The Army should continue to improve the complex computer models it uses to model small caliber ammunition performance.
Assessment • The Mk248 Mod 0 demonstrated adequate performance and lethality when fired from the M2010 ESR.
Mk248
117
Army PROGRAMS
118
118
ARMY PROGRAMS
Nett Warrior Executive Summary • In July 2013, the Assistant Secretary of the Army for Acquisition, Logistics, and Technology authorized an additional low-rate initial production (LRIP) for 2,052 Nett Warrior systems, which allows the program manager to buy 21 percent of the Approved Acquisition Objective. • The Army Test and Evaluation Command (ATEC) conducted two Limited User Tests (LUTs) on Nett Warrior at the Network Integration Evaluation (NIE) 13.1 in November 2012 and NIE 13.2 in May 2013. DOT&E published Operational Assessment reports on both Nett Warrior LUTs. • Nett Warrior demonstrated a capability for providing situational awareness and communications to each equipped leader in directing forces under his control. System • The Nett Warrior is a dismounted leader situational awareness system for use during combat operations. Nett Warrior consists of the following: - End User Device (EUD), a commercial off-the-shelf Samsung Note smartphone - Government-furnished Rifleman Radio (AN/PRC-154A) - Conformal battery - Connecting cables • Frequent Nett Warrior enhancements integrate improved commercial EUD technologies. • The Nett Warrior graphically displays the location of an individual leader, other leaders, friendly vehicles, battlefield messages, and enemy activity on a digital geo-referenced map image. The Nett Warrior is connected through a secure radio to the Soldier Radio Waveform network to communicate
Activity • ATEC executed two LUT events on Nett Warrior in FY13 during NIE 13.1 and 13.2 at Fort Bliss, Texas, and White Sands Missile Range, New Mexico. • DOT&E published Operational Assessment reports on both Nett Warrior LUTs. • In July 2013, ATEC conducted a ballistic test on the conformal battery at Aberdeen Proving Ground, Maryland. • In July 2013, the Assistant Secretary of the Army for Acquisition, Logistics, and Technology authorized an additional LRIP for 2,052 Nett Warrior systems. These additional systems are intended to provide additional test assets to reduce program risk, provide Soldier feedback, and establish the production line at Tobyhanna Army Depot, Pennsylvania. This additional LRIP allows the program
among different echelons using voice, data, and position location information messages. Mission • Leaders within the Brigade Combat Team will use Nett Warrior to provide improved situational awareness, command and control, and enhanced communications. • Combatant Commanders employ Nett Warrior-equipped infantry and cavalry dismounted leaders as part of a Brigade Combat Team to conduct operations (offensive, defensive, stability, and defense support of civil authorities) against conventional or unconventional enemy forces in all types of terrain and climate conditions. Major Contractors • EUD: Samsung – Seoul, South Korea • Rifleman Radio: - General Dynamics C4 Systems – Scottsdale, Arizona - Thales Communications Inc. – Clarksburg, Maryland
manager to buy 21 percent of the Approved Acquisition Objective. • The Army conducted all testing in accordance with DOT&E- approved test plans. Assessment • During the LUT events, Nett Warrior demonstrated a capability to provide situational awareness and communications to each equipped leader in directing forces under his control. Nett Warrior enhanced such tasks as issuing orders, land navigation, message reporting, and command and control. • Testing revealed interoperability problems between Nett Warrior and the Joint Battle Command – Platform (JBC-P).
Nett Warrior
119
Army PROGRAMS
•
•
•
•
The systems could not consistently transmit and receive survivability and command and control data messages. The Nett Warrior software could send and receive only 5 of 25 variable message formats used with JBC-P. The Army could better assess operational effectiveness with a more robust real-time casualty assessment tool than the Multiple Integrated Laser Engagement System, which provides end-of-battle casualty results without the benefit of GPS tracking of users, real-time user engagement feedback, and video playback. Nett Warrior met material reliability and availability requirements during the NIE 13.1 LUT. Nett Warrior demonstrated a reliability point estimate of 342 operating hours exceeding the objective requirement of 291 hours. Nett Warrior demonstrated 95 percent availability, with the requirement being 90 percent. The reliability and availability values demonstrated during the NIE 13.2 LUT were just below the Army-specified requirements. Demonstrated reliability did not affect mission accomplishment. During operational testing, Soldiers indicated the value of Nett Warrior but found that the Rifleman Radio suitability
120
Nett Warrior
shortfalls reduce the suitability of the Nett Warrior. For further information, see the FY13 Annual Report on Rifleman Radio. • At night, screen brightness can inadvertently disclose the user’s location to the enemy. • The Army conducted ballistic testing, shooting the conformal battery in one of the battery’s 16 cells. The conformal battery continued to provide power. There was no “spalling” or expelling of battery materials. There appears to be no degradation in protective characteristics of the Improved Outer Tactical Vest when worn with the conformal battery. Recommendations • Status of Previous Recommendations. The Army addressed the previous recommendations. • FY13 Recommendations. The Army should: 1. Increase the number of Variable Message Format messages Nett Warrior can receive so that it is more interoperable with JBC-P. 2. Use the ATEC Personnel and Equipment Tracking System real-time casualty assessment tool during future operational testing to better assess the operational effectiveness of Nett Warrior.
ARMY PROGRAMS
Patriot Advanced Capability-3 (PAC-3) Executive Summary • The Army began the Post Deployment Build-7 (PDB-7) Limited User Test (LUT) in May 2012 at White Sands Missile Range (WSMR), New Mexico. The PDB-7 LUT ended in January 2013 with the completion of Regression Test 2. • The Army conducted four developmental Patriot flight test missions and one Medium Extended Air Defense System (MEADS) Flight Test (FT) in FY13. • The Missile Defense Agency conducted an integrated flight test of the Ballistic Missile Defense System (BMDS) in October 2012, during which Patriot engaged and killed a cruise missile target and a tactical ballistic missile target in the debris field caused by another BMDS intercept. • In the five Patriot flight tests conducted in FY13, Patriot achieved successful intercepts of four short-range ballistic missile targets and three cruise missile targets using Missile Segment Enhancement (MSE) and Patriot Advanced Capability-3 (PAC-3) missiles. The final MSE engagement demonstrated performance in the extended battlespace. System • Patriot is a mobile air and missile defense system that counters missile and aircraft threats. The system includes the following: - C-band phased-array radars for detecting, tracking, classifying, identifying, and discriminating targets - Battalion and battery battle management elements - Communications Relay Groups and Antenna Mast Groups for communicating between battery and battalion assets - A mix of PAC-3 hit-to-kill missiles and PAC-2 blast fragmentation warhead missiles for negating missile and aircraft threats • The newest version of the PAC-3 missile is the Cost Reduction Initiative (CRI) missile. In addition, the Army is developing the PAC-3 MSE missile with increased battlespace defense capabilities and improved lethality. • Earlier versions of Patriot missiles include the Patriot Standard missile, the PAC-2 Anti-Tactical Missile, and the Guidance Enhanced Missile (GEM) family (includes the GEM-T and
Activity • The Army began the PDB-7 LUT in May 2012 at WSMR, New Mexico. The PDB-7 LUT ended in January 2013 with the completion of Regression Test 2. The Army conducted the testing in accordance with the DOT&E-approved Test and Evaluation Master Plan and test plan. • During Flight Test Integrated-01 (FTI-01) in October 2012 at the Reagan Test Site (on Kwajalein Atoll and Wake Island),
GEM-C missile variants intended to counter tactical ballistic missiles and cruise missiles, respectively). • The DoD intended MEADS to replace the Patriot system. The DoD decided not to field MEADS and concluded U.S. involvement in the design and development phase of the MEADS program in 2013. Mission Combatant Commanders use Patriot to defend deployed forces and critical assets from missile and aircraft attack and to defeat enemy surveillance air assets (such as unmanned aerial vehicles) in all weather conditions, and in natural and induced environments. Major Contractors • Prime: Raytheon Company, Integrated Defense Systems – Tewksbury, Massachusetts • PAC-3 Interceptors: Lockheed Martin Corporation, Missile and Fire Control – Dallas, Texas
Patriot performed a near-simultaneous engagement of a short-range ballistic missile target with two PAC-3 interceptors and a cruise missile target with another PAC-3 interceptor. FTI-01 was the first integrated flight test with multiple firing elements (Aegis Ballistic Missile Defense [BMD], Terminal High-Altitude Area Defense [THAAD], and Patriot) engaging
Patriot
121
Army PROGRAMS
• • •
•
•
multiple ballistic missile and air-breathing targets in a realistic BMDS-level architecture. During MEADS FT-1 in November 2012 at WSMR, Patriot intercepted a cruise missile target with an MSE interceptor. During FT-7-4 in December 2012 at WSMR, Patriot intercepted a short-range ballistic missile target with two MSE interceptors. During the demonstration flight of the Zombie tactical ballistic missile target in April 2013 at WSMR, Patriot intercepted the short-range ballistic missile target with two PAC-3 interceptors. During FT-7-5 in June 2013 at WSMR, Patriot performed a near-simultaneous engagement of a short-range ballistic missile target with two MSE interceptors and a cruise missile target with another MSE interceptor. The next Patriot operational test, the PDB-8 IOT&E, is scheduled to begin in 2015. This IOT&E will provide information to support the Full-Rate Production decision for the MSE interceptor.
Assessment • The PDB-7 LUT showed: - There were improvements in performance against some threats compared to PDB-6.5, but degradations in performance against other threats, the details of which can be found in the classified April 2013 Patriot PDB-7 LUT Assessment Report. - Patriot ground system reliability did not meet the threshold requirement, but would have met it had the Patriot radar achieved its allocated reliability goal. - Patriot training remains inadequate to prepare operators for complex Patriot engagements. This was true during the PDB 6.5 and PDB-6 LUTs as well. This problem was exacerbated in the PDB-7 LUT because many of the experienced Patriot operators in the test unit were transferred to deploying units prior to the LUT, resulting in many inexperienced users and a high variability in Soldier proficiency across the test unit. • During FTI-01, Patriot demonstrated the capability to detect, track, engage, intercept, and kill both a tactical ballistic missile target and a cruise missile target with PAC-3 missiles. - The first PAC-3 missile in the ripple method of fire intercepted the ballistic missile target at the planned altitude and range. - The second PAC-3 missile performed nominally throughout its flight and properly self-destructed after the first PAC-3 missile intercepted the target. - The third PAC-3 missile intercepted the cruise missile target at the planned altitude and range. • During FTI-01, Patriot also demonstrated tactical interoperability with BMDS participants including THAAD; Aegis BMD; and the Command and Control, Battle Management, and Communications System. - There was a Patriot radar fault, but the operators were able to put the system back online in time to conduct a nominal engagement.
122
Patriot
•
•
•
•
•
- All PAC-3 missile subsystems performed as expected. - The Patriot engagements were conducted in the debris field from the THAAD intercept and Patriot debris mitigation was nominal. - Aegis BMD failed to intercept its ballistic missile target during FTI-01; however, the Missile Defense Agency did not set up the flight test so that Patriot could engage targets that Aegis BMD or THAAD failed to intercept. DOT&E had recommended this be a feature of BMDS flight testing in its FY12 Patriot Annual Report. During MEADS FT-1, MEADS demonstrated the capability to detect, track, engage, intercept, and kill a cruise missile target with an MSE interceptor. The MEADS test configuration consisted of a Battle Management Command, Control, Communications, and Computers Intelligence tactical operations center; a Lightweight Launcher; and a Multifunction Fire Control Radar. This was the first MSE engagement of an air-breathing target. During FT-7-4, Patriot demonstrated the capability to detect, track, engage, intercept, and kill a tactical ballistic missile target with MSE interceptors in a ripple method of fire. The first MSE intercepted and killed the ballistic missile target at the planned altitude and range. The second MSE performed nominally throughout its flight and properly self-destructed after the first MSE intercepted the target. During the Zombie (tactical ballistic missile target) flight test, Patriot demonstrated the capability to detect, track, engage, intercept, and kill a tactical ballistic missile target with PAC-3 missiles. - The first PAC-3 missile intercepted the Zombie target at the planned altitude and range, although a missile autopilot error led to the guidance accuracy not being as good as the missile system specification requires. - The second PAC-3 missile failed to launch because a launcher problem led to external power not being provided to the missile. - A backup PAC-3 missile launched and intercepted debris from the first PAC-3 intercept. - Patriot also demonstrated the capability to detect, track, and perform a simulated PAC-3 MSE engagement on a low-altitude cruise missile surrogate target. During FT-7-5, Patriot demonstrated the capability to detect, track, engage, intercept, and kill both a tactical ballistic missile target and a cruise missile target with MSE interceptors. - The first MSE missile in the ripple method of fire intercepted and killed the ballistic missile target at the planned altitude and range within the MSE extended battlespace. - The second MSE performed nominally throughout its flight and properly self-destructed after the first MSE intercepted the target. - The third MSE intercepted and killed the cruise missile target at the planned altitude and range. Continuing obstacles to adequate testing and evaluation of the Patriot system include:
Army PROGRAMS - The lack of a robust interoperability event during PDB-7 testing. - The lack of data required to validate GEM interceptor blast lethality in the Lethality Endgame Simulation. - The lack of a robust Force Development Evaluation, preventing the Army from thoroughly examining tactical standard operating procedures prior to developing Patriot PDB-7 tactics, techniques, and procedures. As a result, the engagement procedures used during the PDB-7 LUT against some threats led to decreased system performance. Recommendations • Status of Previous Recommendations. The Army satisfactorily addressed 14 of the previous 21 open recommendations. The Army should continue to address the following recommendations: 1. Conduct Patriot air and missile defense testing during joint and coalition exercises that include large numbers of different aircraft types, sensors, battle management elements, and weapons systems. Conduct Red Team penetration testing during these exercises to test Patriot Information Assurance.
2. Conduct a Patriot flight test against an anti-radiation missile target to validate models and simulations. 3. Improve Patriot training. 4. Have Patriot participate with live interceptors in THAAD flight testing to determine Patriot-to-THAAD interoperability and the capability for Patriot to intercept tactical ballistic missile targets that are not intercepted by THAAD. 5. Collect reliability data on Patriot systems in the field so that the Mean Time Between Critical Mission Failure can be calculated. 6. Use test units for future Patriot operational tests that have operationally representative distributions in Soldier proficiency. 7. Conduct future operational flight tests with unannounced target launches within extended launch windows. • FY13 Recommendations. In addition to addressing the above recommendations, the Army should: 1. Improve Patriot radar reliability. 2. Obtain the data required to validate GEM interceptor blast lethality in the Lethality Endgame Simulation.
Patriot
123
Army PROGRAMS
124
ARMY PROGRAMS
Precision Guidance Kit (PGK) Executive Summary • Results from the October 2012 Early User Assessment (EUA) demonstrated a Field Artillery unit equipped with Precision Guidance Kit (PGK) can provide near-precision (less than 50 meters) accuracy when firing existing conventional, unguided 155 mm high-explosive projectiles. • In March 2013, DOT&E published a PGK Operational Assessment report. The assessment provided input for the Army’s PGK urgent fielding decision and PGK Program of Record Milestone C decision. • The government has accepted nearly 2,300 urgent fielding PGKs for the Army and Marines, fielding just under 1,300 PGKs to deployed units in combat. The Army indicates units are achieving accurate near-precision (less than 50 meters) target effects. System • The PGK is a combined fuze and GPS guidance kit that improves the ballistic accuracy of the current stockpile of high-explosive field artillery projectiles. • The Army plans to develop PGK for 155 mm high-explosive projectiles (M795 and M549A1) with threshold accuracy of 50 meters Circular Error Probable (CEP) and objective accuracy of 30 meters CEP. • The PGK will operate with existing and developmental artillery systems that have digital fire control systems and inductive fuze setters such as the M777A2 Lightweight Towed Howitzer, the M109A6 Paladin Self-Propelled Howitzer, and the M109 Family of Vehicles Paladin Integrated Management Self-Propelled Howitzer. Mission Field Artillery units employ PGK-fuzed projectiles in support of maneuver units to provide indirect fires with 30 – 50 meters
Activity • The Army is procuring and fielding the PGK in two program tracks. The first track focuses on meeting an Army directed requirement for urgent fielding of PGK. The Army authorized urgent fielding of PGK on March 4, 2013. The second track is the PGK Program of Record with full-rate production planned for 4QFY14. • In October 2012, the Army conducted an EUA, in accordance with a DOT&E-approved test plan, at Yuma Proving Ground, Arizona, as part of the urgent fielding track of the PGK program. This integrated operational and developmental test provided the first opportunity for Soldiers to fire a PGK-fuzed projectile. During the test, Soldier crews performed their tasks
accuracy. PGK-fuzed projectile accuracy allows Field Artillery units to fire fewer projectiles to achieve comparable effects of conventionally fuzed artillery ammunition. Major Contractor Alliant-Techsystems Advanced Weapons Division – Plymouth, Minnesota
successfully in 6 operationally realistic end-to-end missions, firing 20 PGKs from an M777A2-towed digital howitzer. The Army used the demonstrated performance, accuracy, and reliability results to support the PGK urgent fielding decision. • Following the EUA and into FY13, the Army continued with planned PGK developmental testing to address reliability failures observed in previous tests and the EUA. Firing multiple PGK-fuzed projectiles during each developmental test, the program determined the root causes of observed reliability and performance failures and is verifying proposed corrective actions.
PGK
125
Army PROGRAMS • In January 2013, in accordance with the DOT&E-approved Test and Evaluation Master Plan (TEMP), the Army began initial Lot Acceptance Testing of PGKs produced to support urgent fielding. The Army conducted 8 Lot Acceptance Tests throughout 2013 in support of the planned urgent fielding of up to 2,238 PGKs to Army units and 695 PGKs to Marine units. • In March 2013, DOT&E published an Operational Assessment report of the PGK program. The report analyzed data from two operational user assessments conducted by the Army Operational Test Command and developmental testing that occurred between August 2011 and January 2013. The assessment provided input for the Army’s PGK urgent fielding decision and PGK Program of Record Milestone C decision. • The Army Program Executive Officer for Ammunition conducted a Milestone C Decision Review in March 2013 and approved the PGK Program of Record for low-rate initial production (LRIP). • In late 2QFY13, the Army initiated action to move the PGK production line from Minnesota to the contractors’ permanent production facility in West Virginia. The Army validated the facility and its processes to produce LRIP test articles by firing PGKs manufactured on the new production line in developmental testing. The PGK test articles used in the production line validation testing incorporated hardware and software changes made to address remaining reliability and performance shortfalls. • DOT&E approved the PGK Milestone C TEMP on May 6, 2013. • In June 2013, the Army provided DOT&E an overview of its plan to conduct a combined PGK and Excalibur Increment Ib IOT&E. Excalibur is a precision-guided, extended-range, 155 mm artillery projectile. The combined IOT&E is scheduled for 2QFY14 at Yuma Proving Ground, Arizona. DOT&E concurred with the test concept and directed both programs to submit a TEMP update reflecting the combined IOT&E. Assessment • Results from the October 2012 EUA demonstrated a Field Artillery unit equipped with PGK can provide near-precision (less than 50 meters) accuracy when firing existing conventional, unguided 155 mm high-explosive projectiles.
126
PGK
• During the EUA, the median observed CEP accuracy for the PGK-fuzed projectiles fired by Soldier crews from an M777A2-towed digital howitzer was 32 meters (within the 50-meter threshold accuracy requirement and near the 30-meter objective requirement). • The demonstrated reliability of the PGK-fuzed projectiles fired during the EUA indicates the program is on the reliability growth path to meet its reliability requirements by Initial Operational Capability in 1QFY15. The program has not completed testing of the final corrective actions that address reliability failure modes observed in post-EUA developmental testing. • Using a test-fix-test approach, the program has developed corrective actions for the following failure modes: the GPS antenna/radome separating from the PGK in flight, causing a GPS drop lock; PGK-fuzed projectiles impacting several kilometers short of the intended target; and frequent fuze setting failures attributed to the flexible cables imbedded in the PGK canard covers. • Through September 2013, the program has completed 7 of 8 planned urgent fielding PGK Lot Acceptance Tests. The government has accepted nearly 2,300 PGKs for the Army and Marines, fielding just under 1,300 PGKs to deployed units in combat. The Army indicates units are achieving accurate near-precision target effects. • Performance and safety testing of 28 PGKs produced on the new LRIP line in West Virginia demonstrated a median miss distance of 12 meters with 94 percent reliability. • Test planning for the combined PGK and Excalibur Increment Ib IOT&E in 2QFY14 continues. Both program schedules remain on path for the combined IOT&E. Recommendations • Status of Previous Recommendations. The Army has satisfactorily addressed all previous recommendations. • FY13 Recommendations. The Army should: 1. Continue planned testing to validate corrective actions that address remaining reliability and performance shortfalls. 2. Provide an updated TEMP that documents the program’s reliability test strategy for incorporating validated corrective actions into LRIP articles and the now combined PGK‑Excalibur IOT&E.
ARMY PROGRAMS
Q-53 Counterfire Target Acquisition Radar System Executive Summary • In October and November 2012, the Army conducted the Q-53 radar Limited User Test (LUT) at Yuma Proving Ground, Arizona. Soldier crews operated two Q-53 radars during a 48-hour pilot test and three 72-hour record test scenarios observing mortar, artillery, and rocket fires. - During the LUT, the Q-53 acquired and provided targeting information consistent with user requirements in both the 90- and 360-degree modes against threat munitions fired simultaneously from multiple locations. - The Q-53 radar did not meet reliability growth estimates during the LUT. To meet reliability growth estimates, the Army expected the radars to operate 294 hours before a system abort during the LUT. The radars averaged a system abort every 51 hours. - Against threat munitions fired in volleys during the LUT, the Q-53 radar did not acquire or provide targeting information consistent with requirements in either the 90- or 360-degree modes. The Army has not established a radar performance requirement for threat munitions fired in volleys. • The Army Program Executive Officer for Missile and Space (PEO M&S) conducted a Q-53 radar program review on April 16, 2013, and approved the procurement of Lot 3 (21 systems). Lot 3 was the last of three planned low-rate production lots. • Testing previously planned to occur in October 2013 was delayed due to shutdown of the Federal Government and the lack of a Defense Appropriation. System • The Q-53 Counterfire Target Acquisition Radar System is a mobile radar system designed to detect, classify, and track projectiles fired from mortar, artillery, and rocket systems using a 90-degree or continuous 360-degree sector search. • The radar provides target location of threat indirect fire systems with sufficient accuracy for effective counterfire. • The Q-53 is designed to operate with the Counter – Rocket, Artillery, Mortar system and the future Indirect Fire Protection Capability system.
Activity • In October and November 2012, the Army conducted the Q-53 radar LUT at Yuma Proving Ground, Arizona, in accordance with a DOT&E-approved test plan. Soldier crews operated two Q-53 radars during a 48-hour pilot test and three 72-hour record test scenarios observing mortar, artillery, and rocket fires. DOT&E published
• The Army intends to field the Q-53 radar to the target acquisition platoons in Brigade Combat Teams (BCTs) and target acquisition batteries in Fire Brigades to replace the legacy AN/TPQ-36 and AN/TPQ-37 Firefinder radars. • The Q-53 is operated by a crew of four Soldiers and transportable by C-17 aircraft. Battlefield mobility is provided by two Family of Medium Tactical Vehicle trucks. • The Army contracted with Lockheed Martin Missile Systems and Sensors to develop and field 38 Quick Reaction Capability radars to support an Urgent Material Release. The Army intends to produce 136 program of record Q-53 radars. Mission Field Artillery units employ the Q-53 radar to protect friendly forces by determining timely and accurate location of threat rocket, artillery, and mortar systems for defeat with counterfire engagements. Air Defense Artillery integrate the Q-53 radar into the Counter – Rocket, Artillery, Mortar and Indirect Fire Protection Capability System to warn friendly forces and to engage incoming threat indirect fires. Major Contractor Lockheed Martin Missile Systems and Sensors – Syracuse, New York
an Operational Assessment report based on the LUT on April 5, 2013, providing input to the Army PEO M&S planned program review in mid-April 2013. • In January and February 2013, the Army conducted cold‑weather testing on an initial production Q-53 radar at the Cold Regions Test Center, Fort Greely, Alaska. A
Q-53
127
Army PROGRAMS Soldier crew operated the radar in 90-degree and 360-degree modes while observing artillery and mortar live firings. • The Army completed Q-53 radar Developmental Test Phase 2 (DT2) testing at Yuma Proving Ground, Arizona, and White Sands Missile Range, New Mexico, February through August 2013. The Army collected radar data for performance, reliability, operations in an electronic warfare environment, and environmental durability. - The government-operated radars completed 13 test cycles accumulating 2,118 test hours. - Radar crews conducted continuous operations during the 72-hour test cycles, employing the radar in 90-degree and 360-degree modes with tactical and survivability moves. • The Army PEO M&S conducted a Q-53 radar program review on April 16, 2013, and approved the procurement of Lot 3 (21 systems). Lot 3 was the last of three planned low-rate production lots. • The Army executed Phase 1 of the Q-53 radar Logistics Demonstration at the contractor’s Syracuse, New York, facility in June through August 2013. Q-53 radar Soldier maintainers and operators performed 288 radar logistical tasks during the demonstration. Phase 2 of the Logistics Demonstration scheduled for October 7-8, 2013, at Aberdeen Proving Ground, Maryland, was delayed due to shutdown of the Federal Government and the lack of a Defense Appropriation. The Army planned for Soldiers to perform 33 radar logistical tasks during the demonstration. Assessment • During the Q-53 radar LUT, the radars observed mortar, artillery, and rockets fired at various firing rates, trajectories, and radar-to-weapon ranges. - A workaround was required to overcome a Global Positioning System/Inertial Navigation Unit (GPS/ INU) problem that caused targeting errors as great as 1 kilometer. The program made changes to the radar software and the problem did not occur during post-LUT reliability developmental testing. - Against threat munitions fired one at a time during the LUT, the Q-53 acquired and provided targeting information (using the GPS/INU workaround) consistent with requirements in the 360-degree mode, but not in the 90-degree mode. - The Q-53 acquired and provided targeting information (using the GPS/INU workaround) consistent with requirements in both the 90- and 360-degree modes against threat munitions fired simultaneously from multiple locations. - Against threat munitions fired in volleys from the same general location during the LUT, the Q-53 did not acquire or provide targeting information (using the GPS/INU workaround) consistent with requirements in either the 90- or 360-degree modes. The Army has not established a radar performance requirement for threat munitions fired in volleys. Volley fire is a known threat artillery
128
Q-53
technique in which two or more howitzers located in the same unit engage the same target at the same time. - The Q-53 radar contractor has informed the Army that radars operating in the 360-degree mode within a BCT zone must be positioned 20 kilometers apart for optimal performance. Due to terrain restrictions in the LUT, radars operating in the 360-degree mode were positioned less than 20 kilometers apart. The Army cited radar-to- radar interference for the degraded radar performance during LUT. - The counterfire cell supporting the Q-53 LUT could not effectively employ the Q-53 radar. During combat operations, the counterfire cell is located in the tactical operations center of BCTs and Fires Brigades and controls the placement of the radars, establishes search sectors, coordinates frequency allocations to prevent interference, and directs the radars’ survivability and tactical moves. The expertise of counterfire cells to manage high volumes of incoming threat projectiles seen in major combat operations has atrophied in the last eight years due to a hybrid threat that engaged deployed BCTs and Fires Brigades with low volumes of incoming threat projectiles. • The Q-53 radar contractor has developed optimization modes to increase radar short- and long-range performance and performance in adverse weather conditions. The Army has conducted limited developmental testing and no operational testing of these new modes. • The Q-53 radar is not meeting planned reliability growth targets to achieve Army requirements. The user requires the Q-53 radar to operate 185 hours between system aborts. To achieve this requirement in the IOT&E, the Army established a reliability growth target of 361 hours between system aborts. - The LUT reliability growth target was 294 hours between system aborts. The radars demonstrated an average system abort every 51 hours at the conclusion of the LUT. - The IOT&E reliability entrance criterion was 352 hours between system aborts. The radars demonstrated an average system abort every 289 hours at the conclusion of DT2 and did not achieve the IOT&E reliability entrance criteria. - Demonstrating the reliability growth target of 361 hours between system aborts as a point estimate is consistent with having a high statistical probability of demonstrating 185 hours between system aborts in the IOT&E with 80 percent confidence. • Throughout Q-53 radar DT2 testing, the contractor installed three new versions of radar software. Radar performance and reliability decreased using the first two software upgrades. Operating performance improved and reliability increased using the final software version at the end of DT2 testing. The Army has not completed reliability testing of the software version planned for the IOT&E.
Army PROGRAMS Recommendations • Status of Previous Recommendations. The Army satisfactorily addressed all of the FY12 recommendations. • FY13 Recommendations. The Army should: 1. Confirm and characterize suspected radar-to-radar degradation caused by violating radar contractor positioning guidance. Develop and test techniques to overcome radar degradation if contractor positioning guidance is confirmed. 2. Characterize radar performance in all planned operational modes. 3. Determine if there is a valid requirement for Q-53 radar performance against threat munitions fired in volleys.
Q-53
129
Army PROGRAMS
130
ARMY PROGRAMS
Spider XM7 Network Command Munition Executive Summary • The Army uses Spider instead of persistent landmines to comply with the requirements of the 2004 National Landmine Policy. • The Army continued fielding Spider Increment 1 low-rate initial production (LRIP) systems to deployed and non‑deployed units during FY13. • During FOT&E conducted in 1QFY13 (FOT3), Spider Increment 1 demonstrated effectiveness and suitability, overcoming deficiencies identified during FOT2. • In February 2013, DOT&E published a post-FOT3 report to support an Army Full-Rate Production decision for the Spider Increment 1 system. • During FY13, the Army completed requirements for the Spider Increment 1A program to become a program of record and awarded an Engineering and Manufacturing Development (EMD) contract to Northrop Grumman.
System • The Army intends to use Spider as a landmine alternative to satisfy the requirements outlined in the 2004 National Landmine Policy that directs the DoD to: - End use of persistent landmines after 2010 - Incorporate self-destructing and self-deactivating technologies in alternatives to current persistent landmines • A Spider munition field includes: - Up to 63 Munition Control Units, each housing up to 6 miniature grenade launchers or munition adapter modules (the modules provide remote electrical and non-electrical firing capabilities) - A remote control station, used by the operator to maintain “man-in-the-loop” control of all munitions in a field - A communications relay device known as a “repeater” for use in difficult terrain or at extended ranges • Spider incorporates self-destructing and self-deactivating technologies to reduce residual risks to non-combatants. • Spider Increment 1A builds upon existing Spider Increment 1 capabilities with the addition of a new Remote Control Unit that will include an enhanced mapping capability and will provide the capability to communicate munition field status and location to the Mission Command System via radio frequency.
Activity Spider Increment 1 • The Army continued fielding Spider Increment 1 LRIP systems to deployed and non-deployed units during FY13. • During October through November 2012, the Army conducted Spider Increment 1 FOT3 in accordance with
Mission Engineer units of Brigade Combat Teams employ Spider to provide force protection and countermobility obstacles using lethal and non-lethal munitions. Spider functions as a stand-alone system or when combined with other obstacles to accomplish the following: • Provide Early Warning • Protect the Force • Delay and Attrite Enemy Forces • Shape the Battlefield Major Contractors Spider Increment 1 • Command and Control hardware and software: Textron Defense Systems – Wilmington, Massachusetts • Munition Control Unit and Miniature Grenade Launcher: Alliant-Techsystems, Advanced Weapons Division – Plymouth, Minnesota Spider Increment 1A • Command and Control hardware and software: Northrop Grumman Information Systems Sector, Defense Systems Division – Carson, California
a DOT&E-approved Test and Evaluation Master Plan (TEMP) and test plan. • In February 2013, DOT&E published a post-FOT3 report stating Spider Increment 1 is operationally suitable, and remains operationally effective, lethal, and survivable.
Spider
131
Army PROGRAMS • Following publication of the DOT&E Spider Increment 1 report, the Army scheduled a Spider Increment 1 Full‑Rate Production decision for 1QFY14. Spider Increment 1A • In June 2013, the Spider Increment 1A program achieved several key objectives to become a program of record: -- Headquarters, United States Army approved a Spider Increment 1A Capabilities Development Document. -- DOT&E approved the initial Spider Increment 1A TEMP in support of the scheduled Milestone B decision. This TEMP included a requirement for the Army to provide an updated TEMP in FY14 following selection of a system contractor and identification of a materiel solution. -- The Spider Increment 1A Milestone Decision Authority approved Milestone B and the system’s entry into EMD. • In August 2013, the Spider Increment 1A Program Management Office announced the selection of Northrop Grumman as the system contractor and awarded an EMD contract. • At the end of FY13, the Army was updating the June Spider Increment 1A TEMP to reflect the materiel solution proposed by the Spider Increment 1A contractor. Contractor developmental testing is expected to begin in 3QFY14 and a Limited User Test to support a pre-Milestone C assessment is projected for 1QFY16. Assessment • In FOT3, Spider Increment 1 demonstrated resolution of suitability deficiencies discovered in FOT2 conducted in May 2010. - Spider is operationally suitable, and remains operationally effective, lethal, and survivable as previously reported based on data from FOT2 and previous testing.
132
Spider
▪▪ O perational effectiveness – a trained unit can employ Spider Increment 1 as a component of a protective obstacle and provide obstacle effects as desired by the commander. ▪▪ Lethality – Spider Increment 1 grenades and remotely‑initiated munitions can produce personnel casualties. Army Modeling and Simulation determined the Spider Increment 1 can produce 30 percent casualties under the lethality Key Performance Parameter conditions. ▪▪ Survivability – Spider Increment 1 components are survivable in an operational environment. - The two major deficiencies observed in FOT2 were demonstrated in FOT3 to be corrected through hardware and software modifications. The deficiencies were: ▪▪ Failure to meet Munition Control Unit mission reliability and re-use requirements ▪▪ Inability of a unit to “train-as-you-fight” • FOT3 was the last test event for Spider Increment 1. Recommendations • Status of Previous Recommendations. The Army corrected Spider Increment 1 deficiencies addressed in previous recommendations. • FY13 Recommendation. 1. The Army should design the Spider Increment 1A Limited User Test to enable the characterization of the system’s end-to-end mission effectiveness over the operational envelope to inform the system operators of its capabilities and limitations in the various conditions that will be encountered during combat operations.
ARMY PROGRAMS
Stryker Mobile Gun System (MGS) Executive Summary • The Army has mitigated, by either material fixes or tactics, techniques, and procedures, 22 of 23 deficiencies identified in the 2008 Secretary of Defense report to Congress. DOT&E identified 20 of these deficiencies in its 2008 IOT&E report to Congress. • The Army, in coordination with DOT&E, submitted the final report to Congress in July 2013, updating the status of actions taken by the Army to correct or mitigate all Stryker Mobile Gun System (MGS) deficiencies, as directed in Section 115 of the FY09 Duncan Hunter National Defense Authorization Act. • In FY13, the Army corrected the coaxial machine gun low ammunition sensor deficiency and demonstrated improvements for four survivability deficiencies. • In live fire testing, Stryker Reactive Armor Tiles (SRAT II) demonstrated that it mitigates some MGS vulnerabilities and can serve as a vulnerability reduction measure for all flat-bottom Stryker vehicles. LFT&E did reveal SRAT II performance deficiencies. The details are classified. System • The Stryker Family of Vehicles consists of two variants on a common vehicle platform: Infantry Carrier Vehicle and the MGS. • The MGS provides the three-man crew with varying levels of protection against small arms, fragmenting artillery, mines, and rocket-propelled grenades (RPGs). Add-on armor options that provide RPG protection include slat armor (high hard steel arranged in a spaced array) and SRAT II (reactive armor tiles). • The MGS mission equipment includes the following: - M68A2 105 mm cannon system with an ammunition handling system - Coaxial 7.62 mm machine gun and a secondary M2HB, .50-caliber machine gun - Full solution fire control system with two-axis stabilization - Low-profile turret meant to provide survivability against specified threat munitions
Activity • During the December 2010 Stryker Double-V Hull (DVH) Configuration Steering Board, the Army decided not to pursue full-rate production for the Stryker flat-bottom MGS. The Army determined it could not integrate the DVH design onto the MGS platform without the Stryker modernization program to resolve weight and power shortfalls. • The Army has produced and fielded 142 MGSs. Three MGSs are total losses due to battle damage, so the current fleet has 139.
- Driver’s Vision Enhancer and Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance components as government-furnished equipment Mission • The Stryker Brigade Combat Team uses the MGS to create openings in walls, destroy bunkers and machine gun nests, and defeat sniper positions and light armor threats. The primary weapon systems are designed to be effective against a range of threats up to T-62 tanks. • The MGS operates as a three-vehicle platoon organic to the Stryker infantry company or as a single vehicle in support of a Stryker infantry platoon. Major Contractor General Dynamics Land Systems – Sterling Heights, Michigan
• In December 2012, the Army executed a gunnery demonstration to validate the correction to the coaxial machine gun low ammunition sensor deficiency identified during previous operational testing. • The Army completed the LFT&E program for the MGS with SRAT II. Test and evaluation activity in FY13 included armor performance testing of individual tiles and ballistic hull and turret testing to complete the characterization of the protection
Stryker MGS
133
Army PROGRAMS provided by the add-on armor. The Army also completed its modeling and simulation effort in support of the final ballistic vulnerability evaluation of MGS equipped with SRAT II. • The Army, in coordination with DOT&E, submitted the final report to Congress in July 2013, updating the status of actions taken by the Army to correct or mitigate all Stryker MGS deficiencies, as directed in Section 115 of the FY09 Duncan Hunter National Defense Authorization Act. • There have been five auxiliary battery box fires since 2008, the last one in August 2013. After this recent fire, ATEC identified the auxiliary battery box as a safety hazard. The Army released a Ground Precautionary Action detailing near-term corrective actions. • The Army conducted all operational testing in FY13 in accordance with DOT&E-approved test plans. Assessment • The Army has mitigated, by either material fixes or tactics, techniques, and procedures, 22 of 23 deficiencies identified in the 2008 Secretary of Defense report to Congress. DOT&E identified 20 of these deficiencies in its 2008 IOT&E report to Congress. • In 2013, the Army corrected the coaxial machine gun low ammunition sensor deficiency and demonstrated improvements for four survivability deficiencies. • In live fire testing, SRAT II demonstrated that it mitigates some MGS vulnerabilities and can serve as a vulnerability reduction measure for all flat-bottom Stryker vehicles. LFT&E did reveal SRAT II performance deficiencies. The details are classified.
134
Stryker MGS
• In the 2007 IOT&E report, DOT&E assessed the MGS as not operationally effective when operating in a degraded capacity. DOT&E assessed that the gun pod can be easily disabled, causing the MGS to operate in a degraded capacity, thereby making the MGS not operationally effective. Lack of adequate gun pod protection makes the MGS vulnerable to widely proliferated threats including RPGs, which increases the likelihood of the MGS operating in a degraded capacity. The Army has no plans to improve gun pod protection. • The C-130 Transportability Key Performance Parameter is a design constraint that limits MGS capabilities. Because of size and weight constraints for transporting equipment on the C-130, there is a limitation on the size and weight of the MGS. This limit results in several survivability deficiencies, including protection of the Commander’s Weapon Station, protection of 105 mm ammunition, gun pod protection, and hydraulic circuit separation. If the Army decides to move forward with full-rate production, a Stryker modernization program will have the opportunity to address these deficiencies. Recommendations • Status of Previous Recommendations. The Army has addressed one of the recommendations from FY12 but did not address the recommendation to increase gun pod protection. • FY13 Recommendation. 1. As part of DOT&E coordination with the Army, as directed in Section 115 of the FY09 National Defense Authorization Act, the Army should increase gun pod protection.
ARMY PROGRAMS
Warfighter Information Network – Tactical (WIN-T) Executive Summary • Based upon the 2012 Warfighter Information Network – Tactical (WIN-T) Increment 2 IOT&E results, the Defense Acquisition Executive (DAE) authorized a second WIN-T Increment 2 low-rate initial production (LRIP). The Acquisition Decision Memorandum (ADM) directed the Army to: - Conduct an FOT&E to demonstrate effectiveness and suitability of the Soldier Network Extension (SNE) and the Highband Networking Waveform (HNW) prior to accepting the LRIP - Present evidence that all WIN-T Increment 2 configuration items were on track to meet approved reliability thresholds • In 2013, the Army conducted a WIN-T Increment 2 FOT&E to confirm fixes of IOT&E deficiencies. • DOT&E published a WIN-T Increment 2 FOT&E report in September 2013, which assessed most configuration items as operationally effective. - The SNE, Tactical Relay – Tower (TR-T), and HNW were not operationally effective. - Most WIN-T Increment 2 configuration items were operationally suitable. The SNE and Point of Presence (PoP) were not operationally suitable. - WIN-T Increment 2 demonstrated improvement in survivability, but requires further improvement in Information Assurance. • In September 2013, the DAE conducted a WIN-T Increment 2 Full-Rate Production (FRP) decision. The resulting ADM: - Authorized the acceptance of the 2012 LRIP and the procurement of another LRIP without SNEs - Directed the Army to reduce SNE and PoP complexity, improve PoP reliability, fix survivability, and demonstrate these improvements in a second FOT&E System • The Army designed the WIN-T as a three-tiered communications architecture (space, terrestrial, and airborne) to serve as the Army’s high-speed and high-capacity tactical communications network. • The Army intends WIN-T to provide reliable, secure, and seamless communications for units operating at theater level and below. • The WIN-T program consists of four increments; however, Increment 4 is currently unfunded. - Increment 1: “Networking At-the-Halt” enables the exchange of voice, video, data, and imagery throughout the tactical battlefield using a Ku- and Ka-satellite-based network. The Army has fielded WIN-T Increment 1 to its operational forces. - Increment 2: “Initial Networking On-the-Move” provides command and control on-the-move down to the
company level for maneuver brigades and implements an improved network security architecture. WIN-T Increment 2 supports on-the-move communications for commanders with the addition of the PoP and the SNE and provides a mobile network infrastructure with the Tactical Communications Node. - Increment 3: “Full Networking On-the-Move” provides full mobility command and control for all Army field commanders, from theater to company level. Network reliability and robustness are enhanced with the addition of the air tier transport layer, which consists of networked airborne communications relays. - Increment 4: “Protected Satellite Communications On-the‑Move” includes access to the next generation of protected communications satellites while retaining all previous on-the-move capabilities. Mission Commanders at theater level and below will use WIN-T to: • Integrate satellite-based communications capabilities into an everything-over-Internet Protocol network to provide connectivity, while stationary, across an extended, non-linear battlefield and at remote locations (Increment 1) • Provide division and below maneuver commanders with mobile communications capabilities to support initial command and control on-the-move (Increment 2) • Provide all maneuver commanders with mobile communications capabilities to support full command and control on-the-move, including the airborne relay and protected satellite communications (Increments 3 and 4)
WIN-T
135
Army PROGRAMS Major Contractor General Dynamics, C4 Systems – Taunton, Massachusetts
Activity • In May 2012, the Army conducted a WIN-T Increment 2 IOT&E as part of Network Integration Evaluation (NIE) 12.2. DOT&E published a WIN-T Increment 2 IOT&E report to support a September 2012 FRP decision. The SNE, TR-T, and HNW waveform were not effective. All other configuration items and the Net Centric Waveform (NCW) were effective. WIN-T Increment 2 was not suitable due to poor reliability and maintainability, and not survivable due to Information Assurance deficiencies. • In September 2012, the DAE authorized a second LRIP. The ADM directed the Army to: - Conduct an FOT&E to demonstrate effectiveness and suitability of the SNE and the HNW waveform prior to accepting the LRIP. - Present evidence that all WIN-T Increment 2 configuration items were on track to meet approved reliability thresholds. • The Army conducted two Risk Reduction Events during 2QFY13 under benign conditions at the contractor’s facility at Taunton, Massachusetts. • In May 2013 as part of NIE 13.2, the Army conducted the WIN-T Increment 2 FOT&E in accordance with a DOT&E‑approved test plan. The test employed the 2nd Brigade, 1st Armored Division under operationally realistic conditions at Fort Bliss, Texas, and White Sands Missile Range, New Mexico. • In September 2013, DOT&E published a WIN-T Increment 2 FOT&E report in support of the September 2013 FRP decision. • In September 2013, the DAE conducted a WIN-T Increment 2 FRP decision. The ADM: - Authorized the acceptance of the 2012 LRIP and the procurement of another LRIP without SNEs - Directed the Army to reduce SNE and PoP complexity, improve PoP reliability, fix survivability, and demonstrate these improvements in a second FOT&E • The Army continues planning for a second FOT&E during the October through November 2014 NIE 15.1. Assessment • The Army’s Risk Reduction Events demonstrated improvements of the SNE, HNW switching, and Information Assurance. • Based on FOT&E performance, DOT&E assessed most of the WIN-T Increment 2 configuration items as operationally effective. The following components were not operationally effective in both FOT&E and IOT&E: - SNE. The SNE was useful for conducting Voice over Internet Phone (VoIP) calls but the utility of VoIP was limited by long call set-up times. The SNE’s mission command applications and Combat Net Radio Gateway did
136
WIN-T
not support the company’s mission. The SNE’s startup and shutdown procedures were complex, lengthy, and required the vehicle to be at-the-halt. Fifteen of fifteen company commanders found the SNE distracting and indicated they would not take it to war. - HNW. The HNW cycling problem noted in IOT&E was improved and did not affect the unit’s mission. The Army corrected this problem by adjusting HNW parameters that reduced the HNW’s ability to carry the brigade’s network traffic. During FOT&E, 30 percent of the brigade’s network traffic went over line-of-sight HNW compared to 60 percent during IOT&E. The transmission range of HNW in terrain with blockage (e.g., dense vegetation) remains unchanged and poor. - TR-T. The single TR-T employed at brigade was not sufficient to extend range and allow HNW to cover the area-of-operations of the brigade during combat. • DOT&E assessed the Tactical Communications Node, Vehicle Wireless Package, TR-T, and Network Operations and Security Center as operationally suitable during FOT&E. • DOT&E assessed the following WIN-T Increment 2 configuration items as not operationally suitable during FOT&E: - PoP. The PoP is not reliable, too complex to operate, and did not meet its maintainability requirement. - SNE. The SNE is not reliable, too complex to operate, and did not meet its maintainability requirement. • DOT&E assessed WIN-T Increment 2 survivability as improved, but the system continues to have Information Assurance vulnerabilities. The details are provided in the classified annex to the FOT&E report. Recommendations • Status of Previous Recommendations. The program addressed all FY12 recommendations. The program still needs to improve SNE and HNW deficiencies noted during the WIN-T Increment 2 IOT&E. • FY13 Recommendations. The Army should: 1. Continue the reliability growth plan to improve the WIN-T Increment 2 reliability shortfalls highlighted during FOT&E. Reliability improvements should be demonstrated during a future operational test event. 2. Reduce SNE and PoP complexity of operations and troubleshooting. Demonstrate their suitability in a future operational test event. 3. Improve HNW and TR-T to gain better transmission range from the radio and increase the number of TR-Ts available to support units in dispersed operations.
Navy Programs
Navy Programs
N av y P R O G R A M S
Acoustic Rapid Commercial Off‑the‑Shelf (COTS) Insertion (A-RCI) and AN/BYG‑1 Combat Control System
Executive Summary • DOT&E issued a classified FOT&E report on the Advanced Processor Build 2009 (APB-09) variants of the Acoustic Rapid Commercial Off-the-Self (COTS) Insertion (A-RCI) sonar system and AN/BYG-1 combat control system on November 15, 2012. In the report, DOT&E concluded that performance in the mission areas tested remain unchanged from previous versions of the A-RCI and AN/BYG-1 systems. • DOT&E approved a Test and Evaluation Master Plan (TEMP) covering APB-11 and APB-13 variants. • Operational testing of the APB-11 variants began in early FY13 and is expected to conclude in FY14. The Navy completed the first at-sea portion of operational testing in accordance with the DOT&E-approved test plan but testing was limited due to equipment failures: - During the Anti-Submarine Warfare (ASW) test in May 2013, the TB-29 towed array failed. As a result, data for the TB-29-specific modifications of the APB-11 system are to be captured in future testing. - The active operating mode of the Low Cost Conformal Array (LCCA) was unable to be evaluated due to a failure of the system’s software to detect contacts. The system performed adequately in earlier developmental testing but a previously unknown software deficiency caused the system to not function properly in the operational test environment. Although the Navy developed a software update to address this deficiency, future testing will need to be conducted to verify its performance. System A-RCI • The A-RCI sonar system is intended to maintain an advantage in acoustic detection of threat submarines. • A-RCI processes data from the submarine’s acoustic arrays (i.e., spherical array, hull array, wide aperture array, conformal array, and high-frequency array) along with the submarine’s two towed arrays (i.e., the fat line array consisting of the TB-16 or TB-34, and the thin line array consisting of the TB-23 or TB-29). AN/BYG-1 • The AN/BYG-1 combat control system provides operators with information to support appropriate tactical positioning and a means to employ weapons (i.e., torpedoes and missiles). • AN/BYG-1 is used for analyzing and tracking submarine and surface ship contacts, providing situational awareness with the capability to target and employ torpedoes and missiles.
• Both systems use commercial off-the-shelf computer technology and software to provide: -- Sonar and combat control for the Virginia class submarine -- Replacement sonar and combat control retrofitted into Los Angeles, Ohio, and Seawolf class submarines • The Navy updates the hardware and software every two years for these systems to take advantage of improved processing with new technology. Testing for the A-RCI sonar system and AN/BYG-1 combat control system occurs concurrently. Mission • Submarine crews use the A-RCI to: - Search, detect, and track submarine and surface vessels in open-ocean and littoral sea environments without being counter-detected - Search, detect, and avoid mines and other submerged objects - Covertly conduct Intelligence, Surveillance, and Reconnaissance - Covertly execute Naval Special Warfare missions - Perform under-ice operations • Operators use the AN/BYG-1 to: - Analyze submarine sensor contact information to track submarine and surface vessels in open ocean and littoral sea environments - Employ heavyweight torpedoes against submarine and surface ship targets - Receive strike warfare tasking, plan strike missions, and employ Tomahawk land-attack cruise missiles - Receive and synthesize all organic sensor data and external tactical intelligence to produce an integrated tactical picture
A-RCI and AN/BYG-1
137
N av y P R O G R A M S Major Contractors • A-RCI: Lockheed Martin Maritime Systems and Sensors – Washington, District of Columbia
Activity • DOT&E issued an FOT&E report on the APB-09 variants of the A-RCI sonar system and the AN/BYG-1 combat control system on November 15, 2012. • DOT&E approved a TEMP covering APB-11 and APB-13 variants. APB-11 operational testing will include at-sea evaluations focusing on ASW and situational awareness in High-Density Contact Management situations. In addition, testing will include Information Assurance evaluations and an at-sea event against a high-end diesel submarine, which has not been evaluated since 2007. • Operational testing of the APB-11 variants began in early FY13 and is expected to conclude in FY14. The Navy completed the first at-sea portion of testing in accordance with the DOT&E-approved TEMP and test plan but testing suffered from limitations due to equipment failures: - During the ASW test in May 2013, the TB-29 towed array failed. As a result, data could not be collected to characterize the performance of the TB-29 specific modifications of the APB-11 system that provided algorithms for determining range and a new range-azimuth display to aid in resolving bearing ambiguity. The Navy expects to capture these data in future testing. - The Navy was unable to evaluate the active operating mode of the LCCA due to a limitation of the system software. The system performed adequately in earlier developmental testing as the software problem was not readily apparent in a more benign developmental testing environment. The Navy developed a software update to correct this problem and verified proper functionality with in-lab testing by playing back and analyzing recorded data. Operational testing of the active operating mode of the LCCA, to include the software update, is not complete. The Navy incorporated the software update in a revision to the APB-11 variant. Assessment • The final assessment of APB-11 is not completed, as testing is expected to continue through FY14. As a result, DOT&E’s overall assessment remains unchanged from previous assessments. The DOT&E classified FOT&E report from November 2012 concluded the following regarding performance: - For ASW, A-RCI passive sonar capability is effective against older classes of submarines in some environments but is not effective in all environments or against modern threats. - The A-RCI and the AN/BYG-1 systems are not effective in supporting operator situational awareness and contact management in areas of high-contact density.
138
A-RCI and AN/BYG-1
• AN/BYG-1: General Dynamics Advanced Information Systems – Fairfax, Virginia, and Pittsfield, Massachusetts
- The A-RCI high-frequency mine performance is not effective for some types of minefields but meets threshold requirements against some mine types under certain environmental conditions. - The AN/BYG-1 system did not meet the Navy’s requirements for target localization; however, the targeting solutions were often sufficient for a trained crew to provide the torpedo an opportunity to detect the target. Nevertheless, AN/BYG-1 remains not effective in ASW scenarios. - Information Assurance is not effective and remains unchanged from the APB-07 variant, although APB-09 represents an improvement in Information Assurance over previous systems. - The AN-BYG-1 APB-09 system is operationally suitable and continues to exhibit excellent reliability and availability; however, the Navy needs to improve APB training. • Due to the biennial software and hardware development cycle, the Navy generates and approves the requirements documents and TEMPs in parallel with APB development and installation. As a result, the fleet assumes additional risk, since most operational testing is not completed before the system is initially deployed. • The Navy’s schedule-driven process prevents operational test results from directly supporting development of the follow-on APBs. For example, the Navy completed operational testing of the A-RCI APB-09 system in early FY12. Due to the combination of the late completion of testing and the Navy’s practice of issuing an updated version every two years, data from the test could not be included in the development of APB-11. Recommendations • Status of Previous Recommendations. - The Navy made progress in addressing 23 of the 39 previous recommendations outlined in the classified APB-09 DOT&E report. Of the 16 remaining outstanding recommendations, the significant unclassified recommendations are: 1. Conduct additional testing in shallow water to examine the ship’s ASW capabilities in those conditions. 2. Improve the detection and localization performance for submarines operating in high-density surface ship environments. Consider investing in automation that will assist the operator in processing the large amount of constantly changing contact data and determining which contacts pose an immediate collision or counter‑detection threat.
N av y P R O G R A M S 3. Improve operator training such that operators understand and effectively employ new APB functionality when fielded. 4. Evaluate the covertness of the high-frequency sonar during a future submarine-on-submarine test. 5. Determine the performance of the A-RCI system in detecting near surface mines. - The following recommendations from the FY12 Annual Report remain open. In the upcoming fiscal year, the Navy should: 1. Consolidate the A-RCI and AN/BYG-1 TEMPs and test plans into an Undersea Enterprise Capstone document to permit efficiencies in testing.
2. Improve its developmental testing processes and metrics used to determine if a system potentially improves effectiveness and suitability and to ensure that the system is ready for operational testing and subsequent fielding. 3. Evaluate its metrics to improve their robustness under varying environmental conditions and to focus on earlier and longer range operator detections. 4. Conduct a minefield video survey to evaluate the condition and location of the mines prior to the decision to use the minefield for testing, if future minefield testing requires use of existing fleet training minefields. • FY13 Recommendations. None.
A-RCI and AN/BYG-1 139
N av y P R O G R A M S
140
N av y P R O G R A M S
AGM-88E Advanced Anti-Radiation Guided Missile (AARGM) Program Executive Summary • The Advanced Anti-Radiation Guided Missile (AARGM) program remains operationally suitable but not operationally effective due to multiple deficiencies discovered during IOT&E in FY11-12. • The Navy expects that the software changes contained in the AARGM Block 1 Upgrade will address IOT&E deficiencies and Service-deferred Capability Production Document requirements in order to provide full operational capability. • The Navy Conducted a Resource and Requirements Review Board to clarify the Block 1 test requirements, identify the measures of effectiveness necessary for AARGM to achieve operational effectiveness, and determine the resources necessary for this effort. • The Commander, Operational Test and Evaluation Force (COTF), DOT&E, PMA-242, and the Navy have agreed to a framework that will adequately test the AARGM Block 1 Upgrade during an FOT&E in FY14-15. System • The AGM-88E AARGM is the follow-on to the AGM-88B/C High-Speed Anti-Radiation Missile (HARM) using a new guidance section and modified HARM control section and fins. The Navy intends to employ AARGM on F/A-18C/D/E/F and EA-18G platforms. • AARGM Block 0, intended for Initial Operational Capability, incorporates digital Anti-Radiation Homing, GPS, Millimeter Wave guidance, and a Weapon Impact Assessment transmitter. • Anti-Radiation Homing improvements over HARM include an increased field of view, and increased detection range. • The GPS allows position accuracy in location, time, and weapon impact assessment transmissions. • Millimeter Wave radar technology allows target discrimination and guidance during the terminal flight phase.
Activity • There were no operational test events scheduled or conducted during FY13. • In August 2012, DOT&E issued a classified report that stated AARGM Block 0 was operationally suitable but not operational effective. • The Navy’s Milestone Decision Authority conducted a Full-Rate Production (FRP) Decision Review during 4QFY12. At that review, the Navy authorized AARGM Block 0 for FRP. FRP1 is scheduled to begin delivery in January 2014 and FRP2 was awarded in September 2013. FRP3 negotiations are expected to begin in 1QFY14.
• The Navy expects the AARGM Block 1 Upgrade to deliver Full Operational Capability, including Block 0 capability improvements, as well as an Integrated Broadcast Service Receiver (enables reception of national broadcast data), and software changes to provide deferred capability requirements and address deficiencies identified during IOT&E. Mission Commanders employ aircraft equipped with AARGM to conduct pre-planned, on-call, and time-sensitive reactive anti-radiation targeting for the suppression, degradation, and destruction of radio frequency-enabled surface-to-air missile defense systems. Commanders receive real-time Weapons Impact Assessments from AARGM via a national broadcast data system. Major Contractor Alliant Techsystems, Defense Electronics Systems Division – Woodland Hills, California
• In December 2012, the Navy conducted a Requirements and Resources Review Board (R3B) to ensure the requirements to fully test the Block 1 Upgrade were clearly defined and the required funding was allocated. The Chief of Naval Operations N8 endorsed the R3B decision in January 2013. • COTF, with DOT&E oversight, developed an FOT&E framework that will adequately test the deficiencies and deferred capabilities discovered during developmental test and evaluation and IOT&E. • The AARGM Test and Evaluation Master Plan for Block 1 FOT&E is currently being reviewed and will reflect the
AARGM 141
N av y P R O G R A M S agreed-upon framework. The Block 1 Upgrade is intended to complete testing on deferred Capability Production Document capabilities, correct deficiencies identified in IOT&E, and provide derivative benefits. Assessment • The FY13 status remains unchanged from the FY12 report. • In 2012, the Navy, without DOT&E consent, modified the approved test scenario to alleviate a classified deficiency, and proceeded with two live missile firings. Due to the modification of the test scenario, DOT&E assessed those missile firings to be operational failures. With that exception, the AARGM Block 0 testing was adequate to support an evaluation of the weapon system’s operational effectiveness and operational suitability. • AARGM Block 0 is operationally suitable. Although the weapon demonstrated poor reliability during IOT&E, the program addressed the primary deficiency affecting reliability and satisfactorily demonstrated this during the verification of correction of deficiencies test period in FY12. • AARGM Block 0 is not operationally effective. The details of these deficiencies are detailed in the classified DOT&E IOT&E report published in August 2012. • Sequestration is currently affecting FOT&E planning. Reductions in funding in FY14 could delay FOT&E later into FY15 or possibly FY16.
142 AARGM
Recommendations • Status of Previous Recommendations. The FY12 recommendation to limit FRP quantities until operational effectiveness is properly demonstrated is no longer valid as the Milestone Decision Authority made the FRP decision in 4QFY12, and the Navy acquired additional lots of FRP missiles. The Navy intends to upgrade all Block 0 FRP missiles with Block 1 Upgrade at the completion of the FOT&E. The upgrade is a software-only upgrade and will be completed at the squadron level. The Navy addressed the second recommendation regarding telemetry kits to satisfy Block 1 FOT&E requirements. • FY13 Recommendation. 1. The Navy should adequately program and fund the AARGM Block 1 FOT&E. In the event full funding is not available, the Navy should prioritize targets and conduct FOT&E on the higher priority targets. The Navy should then develop an additional FOT&E period to fully test the remaining lower-priority targets, when funding becomes available.
N av y P R O G R A M S
AIM-9X Air-to-Air Missile Upgrade Executive Summary • On July 29, 2013, the Program Executive Officer formally decertified AIM-9X Block II due to deficiencies discovered during IOT&E that affected missile performance. As of November 2013, the root causes of these deficiencies were still under investigation. • The Navy and Air Force began IOT&E on April 27, 2012. Prior to decertification to continue operational testing, the Navy completed 18 of 22 planned captive-carry events, 5 of 9 live missile shots, and 1 repeat test shot. The Air Force completed 18 of 22 captive-carry events and 6 of 8 live missile shots. Of the 12 live missile shots, 7 were within lethal radius of the target. The Services plan to return to IOT&E in 3QFY14. • As of July 29, 2013, the Navy and Air Force accomplished 6,353 total operating hours with 22 failures resulting in a Mean Time Between Critical Failure (MTBCF) of 288.79 hours. The current system reliability is significantly below the value on the reliability growth curve consistent with reaching the requirement of 500 hours MTBCF at 80,000 hours.
capability for the OFS 9.3 upgrade. The AIM 9X-2 missile includes a new processor, a new ignition battery for the rocket motor, an electronic ignition safety/arm device, and the DSU-41/B Active Optical Target Detector fuze/datalink assembly. - OFS 9.3 is a software upgrade that is intended to add trajectory management to improve range, datalink with the launching aircraft, improved lock-on-after-launch, target re-acquisition, and improved fuzing.
System • AIM-9X is the latest generation short-range, heat-seeking, air-to-air missile. The currently fielded version of the missile is AIM-9X Block I, Operational Flight Software (OFS) 8.220, which includes limited lock-on-after-launch, full envelope off boresight capability without a helmet-mounted cueing system, and improved flare rejection performance. • AIM-9X is highly maneuverable, day/night capable, and includes the warhead, fuze, and rocket motor from the previous AIM-9M missile. • AIM-9X added a new imaging infrared seeker, vector controlled thrust, digital processor, and autopilot. • F-15C/D, F-16C/D, and F/A-18C/D/E/F aircraft are capable of employing the AIM-9X. • The AIM-9X Block II is the combination of AIM-9X-2 hardware and OFS 9.3. - AIM-9X-2 is the latest hardware version and is designed to prevent parts obsolescence and provide processing
Mission Air combat units use the AIM-9X to: • Conduct short-range offensive and defensive air-to-air combat • Engage multiple enemy aircraft types with passive infrared guidance in the missile seeker • Seek and attack enemy aircraft at large angles away from the heading of the launch aircraft
Activity • On July 29, 2013, the Program Executive Officer formally decertified AIM-9X Block II (AIM-9X-2 with OFS 9.311) for operational testing due to deficiencies discovered during IOT&E that affected missile performance. As of November 2013, the root causes of these deficiencies were still under investigation.
• Between the start of IOT&E on April 27, 2012, and decertification, the Navy completed 18 of 22 planned captive‑carry events, 5 of 9 planned live missile shots, and 1 repeat test shot. The Air Force completed 18 of 22 captive- carry events and 6 of 8 live missile shots. Of the
Major Contractor Raytheon Missile Systems – Tucson, Arizona
AIM-9X 143
N av y P R O G R A M S 12 live missile shots conducted during operational testing, 7 were within lethal radius of the target. • After the Program Office identifies the root causes for deficiencies and implements hardware and/or software solutions, they will request recertification for IOT&E. • Before decertification, the Navy and Air Force intended to complete IOT&E in July 2013, with a Full-Rate Production decision in April 2014 and Initial Operational Capability in September 2014. The Services plan to return to IOT&E in 3QFY14. • The Program Office conducted the IOT&E in accordance with the DOT&E-approved Test and Evaluation Master Plan. Assessment • During operational testing, 7 of 12 total AIM-9X Block II shots guided to within lethal radius of the drone. The developmental testing record was 9 of 12 shots within lethal radius; however, one missile did not receive a fuze pulse. As of July 29, 2013, the total hit rate was assessed as 15 of 24. • Aircrew observed missile flyout for 8 of the 12 operational test shots. For seven of the eight shots, they witnessed excessive oscillations, or “porpoising.” The Navy and Air Force assessed two of the observed missile shots as “misses” due to internal measurement unit errors. Data from four shots show possible deficiencies with the guidance, navigation, and control software or the internal measurement unit hardware.
144 AIM-9X
• All captive-carry missions were nominal, but the Air Force repeatedly highlighted one performance discrepancy with AIM-9X Block II Helmet-less High Off-Boresight (HHOBS) performance. Aircrew reported that Block II is slower to acquire targets in HHOBS than Block I. The Capability Production Document requires Block II performance be equal to or better than baseline AIM-9X performance. • At the Operational Test Readiness Review, reliability was 232 hours MTBCF and was projected to reach 316 hours at the end of IOT&E. The Navy and Air Force accomplished 6,353 total operating hours with 22 failures, resulting in an MTBCF of 288.79 hours. The current system reliability is significantly below the value on the reliability growth curve consistent with reaching the requirement of 500 hours MTBCF at 80,000 hours. DOT&E will track reliability in the IOT&E. Recommendations • Status of Previous Recommendations. The Navy addressed the previous recommendations. • FY13 Recommendation. 1. The Navy should submit for approval an updated operational test plan after implementing hardware and/or software solutions to fix identified deficiencies.
N av y P R O G R A M S
AN/BLQ-10 Submarine Electronic Warfare Support System Executive Summary • The Navy operationally tested the AN/BLQ-10 system with the Technical Insertion (TI) 2008 (TI-08) upgrade and the Multifunction Modular Mast (MMM) in October 2012. • DOT&E issued a classified report on that testing in September 2013 and concluded the TI-08 upgrade improves the system’s intercept capability against Low-Probability of Intercept (LPI) radars, and the MMM provides communications signal localization accuracy that would be sufficient for most missions. DOT&E assessed the AN/BLQ‑10 system as not operationally effective for use in collection of communications signals. • DOT&E is working with the Navy to develop a Test and Evaluation Master Plan (TEMP) to support assessment of the AN/BLQ-10 system with the TI-10 upgrade. Testing on the TI-10 version of the system is expected to occur during FY14.
System • The AN/BLQ-10 system is an electronic warfare support system for U.S. submarines. It provides automatic intercept capability (detection, classification, localization, and identification) for both radar and communications signals. Separate subsystems process radar and communications signals. • The AN/BLQ-10 processes signals collected with the submarine’s masts. Radar signals are collected by the imaging mast, which is either a photonics mast (on the Virginia class) or a periscope (on all other classes). Communications signals are collected from both the imaging mast and a dedicated communications intercept mast, which is either an AN/BRD-7 (on the Los Angeles and Seawolf classes), an AN/BSD-2 (on the Virginia class), or a MMM (recently fielded on some Los Angeles and Virginia class ships). These masts provide largely the same functionality but with different frequency coverage and localization accuracy. • The program is adopting an open-architecture, incremental development process. Hardware and software updates,
Activity • The Navy completed TEMP Revision C to cover testing of the TI-08 and TI-10 upgrades to the system. • After TEMP Revision C was signed, the Navy decided to accelerate the fielding of a new communications intercept algorithm into TI-10. This change will necessitate a new TEMP revision to cover the additional testing required for this capability. The Navy has begun the test design and TEMP revision processes.
referred to as TIs, will be fielded every two years. TI-08 was the first such upgrade, which added a subsystem to intercept some LPI radar signals. • The AN/BLQ-10 provides support for specialized, carry-on electronic warfare equipment and personnel. Mission Submarine crews use the AN/BLQ-10 electronic warfare support system whenever the submarine is at periscope depth. Crews use the information provided by AN/BLQ-10 for the following submarine force missions: • Threat warning to avoid counterdetection and collision • Determining the number and location of targets for subsequent prosecution • Conducting Intelligence, Surveillance, and Reconnaissance in support of fleet or battlegroup objectives Major Contractor Lockheed Martin Mission Systems and Training – Syracuse, New York
• During at-sea developmental and operational testing in October 2012, the Navy assessed the ability of TI-08 to intercept LPI radars and the ability of the MMM to localize communications signals. DOT&E issued a classified report on this testing in September 2013. • In September 2013, the Navy conducted cybersecurity testing of BLQ-10.
AN/BLQ-10 145
N av y P R O G R A M S Assessment • The AN/BLQ-10 system is limited in operational effectiveness. The system detects some radars at long ranges; however, operational testing was inadequate to determine the extent operators can use the AN/BLQ-10 to support submarine missions. The Navy has not yet conducted operational testing against some modern threat radars or appropriate surrogates. The AN/BLQ-10 system is not operationally effective for collecting communications signals due to its inability to automatically detect some signal types. • The TI-08 upgrade provides improved intercept capability against the intended LPI radars. However, the number of LPI radars is increasing and the Navy will need to develop future upgrades to stay current with newer technology. • The MMM provides communications localization accuracy that would be sufficient for most submarine missions. Operational testing showed the system did not meet the Navy’s established thresholds. • The most recent operational testing was partially adequate because it provided sufficient data, when supplemented with developmental testing results, to assess the technical performance of the AN/BLQ-10’s intercept capabilities. However, the Navy did not conduct testing in accordance with the October 2012 DOT&E-approved test plan. - Testing was not adequate to assess the operators’ ability to determine counterdetection risks, which is a primary use in submarine operations. In particular, the test plan required a Ticonderoga class cruiser to act as a surrogate threat; however, the ship scheduled to participate was unable due to a material casualty and no other ships were available.
146 AN/BLQ-10
- The submarine’s crew did not act realistically to the threat posed by the available P-3C aircraft, which was the only threat surrogate in the test. These problems limited the data available to evaluate the AN/BLQ-10’s support of threat avoidance. • The AN/BLQ-10 is not operationally suitable because the Navy’s training system is not sufficient to allow fleet operators to maintain proficiency on the system. Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. The Navy should: 1. Reconsider use of Probability of Communications Signal Intercept and Probability of Electronic Signal Intercept in establishing the AN/BLQ-10 system requirements and use measures that address the system’s capabilities against each of the signal types. 2. Develop a more robust training program to increase the proficiency of AN/BLQ-10 operators and maintainers on the communications subsystem. 3. Avoid conducting developmental testing immediately before operational testing unless measures are in place to prevent degraded operator performance due to desensitization. 4. Structure future tests to evaluate AN/BLQ-10’s support of threat avoidance, rather than limiting them to assessing the technical performance of the system.
N av y P R O G R A M S
AN/SQQ-89A(V)15 Integrated Undersea Warfare (USW) Combat System Suite Executive Summary • The Navy deployed the AN/SQQ-89A(V)15 with Advanced Capability Build (ACB)-11 onboard a DDG 51 class destroyer in July 2013. • The Navy Commander, Operational Test and Evaluation Force (COTF) conducted an operational assessment (OA) of the AN/SQQ-89A(V)15 in conjunction with two fleet training events in FY13. AN/SQQ-89A(V)15 demonstrated capability to detect submarines and incoming U.S. torpedoes during limited deep water testing. • IOT&E is expected to complete in 3QFY14.
System • AN/SQQ-89A(V)15 is the primary Undersea Warfare system used aboard U.S. Navy surface combatants to locate and engage threat submarines. AN/SQQ-89A(V)15 is an open‑architecture system that includes biannual software upgrades (ACBs) and four-year hardware upgrades called Technology Insertions. • AN/SQQ-89A(V)15 uses active and passive sonar to conduct Anti-Submarine Warfare (ASW) search. Received acoustic energy is processed and displayed to support operator detection, classification, localization, and tracking of threat submarines. • AN/SQQ-89A(V)15 uses passive sonar to provide early warning of threat torpedoes. • The Navy intends for the program to provide improvement in sensor display integration and automation, reduction in false alerts, and improvement in onboard training capability to better support operation within littoral regions against multiple sub-surface threats. • The system consists of: - Acoustic sensors – hull-mounted array, multi- function towed array (TB-37), towed acoustic intercept array, calibrated reference hydrophone, helicopter and/or ship‑deployed sonobuoys - Functional segments used for processing and display of active, passive, and environmental data
Activity • AN/SQQ-89A(V)15 with ACB-09 was delivered to the fleet in FY09 and installed on 12 DDG 51 class destroyers. In 2011, the Navy deferred IOT&E of AN/ SQQ-89A(V)15 with ACB‑09 due to imminent delivery of ACB-11. The only previous operational test on a version of AN/ SQQ- 89A(V)15 occurred in 2005 and did not include an
- Interface to Aegis Combat System for Mk 46 and Mk 54 torpedo prosecution using surface vessel torpedo tubes, Vertical Launch Anti-Submarine Rocket, or SH- 60B/ MH- 60R helicopters • The system is deployed on a DDG 51 class destroyer or CG 47 class cruiser host platform. Mission • Maritime Component Commanders employ surface combatants with AN/SQQ-89A(V)15 as escorts to high-value units to protect against threat submarines during transit. • Maritime Component Commanders use AN/SQQ-89A(V)15 to conduct area clearance and defense, barrier operations, and ASW support during amphibious assault. • Unit Commanders use AN/SQQ-89A(V)15 to support self- protection against incoming threat torpedoes. Major Contractor Lockheed Martin Mission Systems and Training – Syracuse, New York
evaluation of performance in shallow water. DOT&E placed AN/SQQ‑89A(V)15 under oversight in late FY10. • In January 2013, DOT&E sent a memorandum to the Assistant Secretary of the Navy (Research, Development, and Acquisition) outlining the need for a threat torpedo surrogate to support operational testing of the AN/SQQ-89A(V)15.
AN/SQQ-89A(V)15 147
N av y P R O G R A M S • COTF conducted an OA on AN/SQQ-89A(V)15 with ACB-11 in FY13. Test activities were conducted in accordance with a DOT&E-approved test plan and included ASW transit search and area search operations using AN/SQQ-89A(V)15 onboard a DDG 51 class destroyer. Testing was conducted in conjunction with the following two fleet events: - Submarine Command Course 12-4 Anti-Surface Warfare events in November 2012 - Tactical Development Exercise 6-13 in April 2013 • COTF conducted integrated testing on AN/SQQ-89A(V)15 with ACB-11 in accordance with a DOT&E-approved test plan and in conjunction with a fleet training event, SCC 13-2, in May 2013. • The Navy deployed a DDG 51 class destroyer with AN/SQQ‑89A(V)15 with ACB-11 in July 2013. • DOT&E will issue a classified Early Fielding Report for AN/ SQQ-89A(V) with ACB-11 in 2QFY14 based on observations and data obtained from the OA and integrated testing. • The Navy is scheduling dedicated IOT&E events for 2Q-3QFY14.
Assessment • Operationally relevant testing of AN/SQQ-89A(V)15 with ACB-11 to date has been limited to deep water environments. Due to the prevalence of submarines operating in littoral regions, the lack of testing in shallow water represents risk to fleet operation.
148 AN/SQQ-89A(V)15
• AN/SQQ-89A(V)15 with ACB-11 demonstrated capability to detect inbound U.S. torpedoes and will likely improve surface combatant survivability against sub-surface threats. The ability of surface combatants employing the AN/SQQ‑89A(V)15 to avoid torpedoes can only partially be assessed due to significant differences in U.S. torpedoes and untested wake homing torpedoes employed by other nations. • AN/SQQ-89A(V)15 with ACB-11 demonstrated some capability to detect and classify threat representative submarines during an OA. However, the limited testing was insufficient to assess the likelihood of a successful submarine prosecution.
Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. The Navy should: 1. Schedule and complete IOT&E to adequately assess the effectiveness and suitability of AN/SQQ-89A(V)15 with ACB-11 with a primary focus on performance in shallow water. 2. Identify and/or develop a threat torpedo surrogate to support operational test as identified in a DOT&E memorandum to the Assistant Secretary of the Navy (Research, Development, and Acquisition) dated January 09, 2013.
N av y P R O G R A M S
Cobra Judy Replacement (CJR) Executive Summary • During Operational Assessment-2 (OA-2), Commander, Operational Test and Evaluation Force (COTF) personnel were able to address a number of ship-based effectiveness and suitability measures. In parallel, the Air Force Operational Test and Evaluation Center (AFOTEC) conducted an assessment of the pre-Technical Evaluation (TECHEVAL) S- and X-band radar capabilities against live radar targets and conducted Information Assurance (IA) testing of the mission equipment. • With the exception of one run-for-record event, the Navy Program Office and the prime contractor (Raytheon Integrated Defense Systems) completed their TECHEVAL in July 2013. • AFOTEC conducted its Multi-Service Operational Test and Evaluation (MOT&E) from September through November 2013. In addition to conducting modeling and simulation scenarios and data collections against standard targets, the crew executed Cobra Judy Replacement’s (CJR) primary mission utilizing U.S. Air Force Glory Trip flights. System • CJR is a mobile radar suite permanently located on the USNS Howard O. Lorenzen. • The original Cobra Judy system has been deployed since 1981 and has reached the end of its service life. • The CJR radar suite consists of steerable, instrument-quality S- and X-band phased arrays, greatly expanding the data collection capability over the original system. The S-band radar primarily conducts large volume searches and is capable of performing radar tracks and collections on a large number of radar targets. The X-band radar provides high-resolution data on specific radar objects of interest and also has a search capability. • The ship’s crew will consist of civilian or contracted Military Sealift Command personnel responsible for the navigation,
Activity • During OA-2, conducted in accordance with the DOT&E‑approved Test and Evaluation Master Plan and begun in August 2012, the S- and X-band radars tracked balloon‑borne calibration spheres and satellites at full power while the ship conducted at-sea operations. Tracking standard targets, the crew exercised routine functions it will use to support its primary mission. AFOTEC, as the lead Operational Test Agency, conducted the radar and IA portion of OA-2. In parallel, COTF personnel assessed ship-based measures of effectiveness and suitability.
operations, and maintenance of the ship; a small, specialized group of contractors will be utilized for radar operations. An Air Force officer will serve as the mission commander. • Once the Air Force accepts the CJR as an operational capability, the ship platform will be designated as Cobra King. Mission The DoD uses CJR to conduct treaty monitoring and verification activities. Additionally, CJR can be used to provide data for comparison with other sources during domestic ballistic missile tests. Major Contractors • Raytheon Integrated Defense Systems – Sudbury, Massachusetts • Northrop Grumman Electronic Systems – Baltimore, Maryland • VT Halter Marine – Pascagoula, Mississippi
• In September 2012, the program manager decided to delay the start of both the TECHEVAL and MOT&E by three months to support completion of the integration effort. • On March 19, 2013, CJR successfully tracked and collected radar cross section data on the boosting phase of an Atlas V launch from Cape Canaveral Air Force Station, Florida. • With the exception of one run-for-record event, the Navy Program Office and the prime contractor (Raytheon Integrated Defense Systems) completed the TECHEVAL in July 2013. They verified that the ship, radar, and auxiliary mission
CJR 149
N av y P R O G R A M S equipment met the defined performance specifications and declared the system ready for MOT&E execution. • AFOTEC conducted its MOT&E from September through November 2013 in accordance with the DOT&E-approved test plan. In addition to conducting modeling and simulation scenarios and data collection against standard targets such as satellites and balloon-borne spheres, the crew executed CJR’s primary mission using Air Force Glory Trip flights. Assessment • The CJR program executed a compressed test schedule caused by programmatic complications and technical setbacks during the integration and developmental test phase DT-VI, which resulted in three-month delays of TECHEVAL and MOT&E. The Program Office balanced the need for adequate testing with the pressure to retire an aging and difficult-to-maintain legacy Cobra Judy system without any gaps in mission capability or technical performance. • During OA-2, COTF personnel were able to address many of the ship-based performance measures (i.e., ship speed, endurance, replenishment, habitability, etc.). However, during this event, none of the radar effectiveness measures were resolved, although the crew did collect metric and signature data against test balloons and satellites, mainly using S-band and a partial X-band array. At the time, the radar arrays were
150 CJR
• •
•
•
not fully calibrated and a failure of a replaceable unit in the X-band radar led to only a partial test of the radar’s capability. Following TECHEVAL, a number of Category II software deficiencies were noted and addressed. The fixes will be implemented following completion of the MOT&E. Modeling and simulation was an essential part of the test strategy as it is impossible to observe the wide range of ballistic missile phenomenology relying solely on targets of opportunity. Operational testing involved demonstrating not only the ship and radar performance during the conduct of a mission, but also the pre-mission planning and transmission of collection data and post-mission data analysis using software tools developed and largely tested under a separate acquisition program. The Atlas V launch provided valuable data, revealing system deficiencies that were corrected through system modifications to improve operator situational awareness and system operational procedures. An assessment of MOT&E test mission data with respect to the effectiveness and suitability of the CJR is ongoing.
Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. None.
N av y P R O G R A M S
Consolidated Afloat Networks and Enterprise Services (CANES) Executive Summary • The Commander, Operational Test and Evaluation Force (COTF) conducted an operational assessment of Consolidated Afloat Networks and Enterprise Services (CANES) from September 12 through October 10, 2012, in a laboratory environment. Testing was conducted to inform a Milestone C and limited fielding decision. DOT&E reported on the results in the December 2012 Operational Assessment report. • COTF will conduct the CANES IOT&E for unit-level ships onboard USS Milius in June 2014. Subsequent to the IOT&E, COTF will conduct follow-on test events on force-level ships and submarines. System • CANES is an evolving enterprise information environment consisting of computing hardware, software, and network services (e.g., phone, email, chat, video teleconferencing, web hosting, file transfer, computational resources, storage, network configuration) monitoring. CANES will replace legacy networks on ships, submarines, and shore sites. • The CANES program is intended to mitigate hardware and software obsolescence on naval vessels through the increased use of standard components and regular hardware and software updates. • The CANES network will provide a single consolidated physical network with logical sub-networks for Unclassified, Secret, Secret Releasable, and Top Secret security domains. It will include a cross-domain solution for information transfers across these security boundaries. This consolidation
Activity • COTF conducted an operational assessment of CANES in a laboratory environment from September 28 through October 10, 2012. • DOT&E reported on the results of the CANES operational assessment in the December 2012 Operational Assessment report. • USD(AT&L) approved the Milestone C decision in December 2012 and published an Acquisition Decision Memorandum authorizing limited fielding to 29 CANES units in addition to 8 procurements that were previously authorized at Milestone B. • COTF will conduct the CANES IOT&E for unit-level ships onboard USS Milius in April 2014. Subsequent to the IOT&E, COTF will conduct follow-on test events on force-level ships and submarines.
is expected to reduce the network infrastructure footprint on naval platforms and the associated logistics, sustainment, and training costs. Mission Shipboard and shore-based users will use the CANES network to: • Host their applications in support of naval and joint operations with computing resources and networks services • Support weapon systems, command and control, intelligence, and business information applications Major Contractor Northrop Grumman – San Diego, California
• The Milestone C Acquisition Decision Memorandum states USD(AT&L) will convene Interim Program Review Defense Acquisition Board meetings to review the CANES program upon completion of operational testing for force-level ships and submarines. Assessment • CANES provided network services at Unclassified, Secret, Secret, Releasable, and Top Secret classification levels, performing very limited sets of operations. Two future integrated test (IT) events (IT-C1 in December 2013 and IT-C2 in February 2014) and IOT&E (April 2014) are scheduled to address the full CANES functionality. • COTF has only tested 4 of 32 baseline applications for CANES. The Navy will conduct developmental test events
CANES 151
N av y P R O G R A M S before the start of IOT&E to test the remaining interfaces and representative applications. • As of November 14, 2012, CANES had a large number of cybersecurity vulnerabilities (29 Category 1 and 172 Category 2). The Navy must mitigate cybersecurity vulnerabilities prior to the IOT&E. • The Navy’s test planning documents do not provide an adequate description of the operational environment for the CANES IOT&E. DOT&E will approve the CANES Test and Evaluation Master Plan when the Navy provides an adequate description of the operational test environment for IOT&E.
152 CANES
Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. The Navy needs to: 1. Mitigate all CANES Category 1 and 2 cybersecurity vulnerabilities prior to IOT&E. 2. Provide a description of the operational environment for IOT&E that includes a discussion of the mission types that the ship’s crew will perform during the test.
N av y P R O G R A M S
Cooperative Engagement Capability (CEC) Executive Summary • In a September 16, 2013, report to Congress, DOT&E assessed the USG-3B Cooperative Engagement Capability (CEC) E-2D Advanced Hawkeye Carrier Airborne Early Warning aircraft variant to be operationally suitable, but not operationally effective based on the results of an FOT&E conducted from September 2012 to May 2013. • FOT&E testing identified performance deficiencies showing that the USG-3B CEC’s performance is inferior to the performance of the predecessor USG-3 CEC used in the E-2C Hawkeye 2000 aircraft. System • CEC is a real-time sensor netting system that enables high‑quality situational awareness and Integrated Fire Control capability. • There are four major U.S. Navy variants of CEC: - The USG-2A is used in selected Aegis cruisers and destroyers, LPD-17/LHD amphibious ships, and CVN-68 class aircraft carriers. - The USG-2B, an improved version of the USG-2, is used in selected Aegis cruisers and destroyers. - The USG-3 is used in the E-2C Hawkeye 2000 aircraft. - The USG-3B is used in the E-2D Advanced Hawkeye aircraft. • The two major hardware pieces are the Cooperative Engagement Processor, which collects and fuses radar data, and the Data Distribution System, which exchanges the Cooperative Engagement Processor data. • The CEC increases overall Naval Air Defense capabilities by integrating sensors and weapon assets into a single, integrated, real-time network that expands the battlespace; enhances
Activity • The Navy’s Commander, Operational Test and Evaluation Force (COTF) completed the first phase of CEC USG-3B FOT&E at the Naval Air Station (NAS) Patuxent River, Maryland; Eielson AFB, Alaska; NAS Fallon, Nevada; and NAS Point Mugu, California, from September 2012 through June 2013. Testing was conducted in accordance with a DOT&E-approved test plan. • DOT&E issued a classified report to Congress on the results of the CEC USG-3B FOT&E on September 16, 2013. Assessment • FOT&E testing identified performance deficiencies showing that the USG-3B CEC’s performance is inferior to the performance of the predecessor USG-3 CEC used in the E-2C Hawkeye 2000 aircraft.
situational awareness; increases depth-of-fire and enables longer intercept ranges; and improves decision and reaction times. Mission Naval forces use CEC to improve battle force air and missile defense capabilities by combining data from multiple battle force air search sensors on CEC-equipped units into a single, real-time, composite track picture. Naval surface forces also use CEC to provide accurate air and surface threat tracking data to ships equipped with the Ship Self-Defense System. Major Contractor Raytheon Systems Co., Command, Control and Communications, Data Systems – St. Petersburg, Florida
• The FOT&E demonstrated that, while the USG-3B CEC failed to meet its reliability requirement, the observed reliability would allow the E-2D to complete a typical 5-hour mission, without a mission-ending CEC hardware failure, 94 percent of the time. • Deficiencies found in FOT&E included the following: - Errors in the estimated alignment of one CEC unit’s sensors with another CEC unit’s sensors seriously degraded the USG-3B CEC’s ability to ensure that tracks on one CEC unit are identical to tracks on another CEC unit (i.e., Track File Concurrence). - Excessive numbers of dual tracks (i.e., multiple tracks for single objects) were well in excess of historical results. - Interoperability errors between the USG-3B CEC and the E-2D mission computer degraded the single integrated air
CEC 153
N av y P R O G R A M S picture presented to the various combat systems (e.g., other E-2Ds, Ship Self-Defense System Mk 2 Combat Systems, and Aegis Combat System) in the CEC network and datalink networks with the E-2Ds. - Electromagnetic interference between the USG-3B CEC and the E-2D radar altimeter caused the altimeter readings to be unreliable at certain altitudes. • The classified September 16, 2013, DOT&E report to Congress contains further USG-3B CEC related details and recommendations. • DOT&E approved the CEC Test and Evaluation Master Plan (TEMP) in May 2012. The TEMP requires an update to address all future phases of CEC operational testing. Recommendations • Status of Previous Recommendations. The Navy has satisfied all of the previous recommendations. • FY13 Recommendations. The Navy should: 1. Determine the root cause of the problem that degrades the USG-3B CEC’s Track File Concurrence and demonstrate corrections in a phase of FOT&E. 2. Implement changes to the USG-3B CEC interface with the E-2D mission computer that would allow data from the E-2D’s APY-9 radar to be used by the USG-3B CEC without first requiring the creation of an E-2D Mission Computer track.
154 CEC
3. Reassess the USG-3B CEC reliability requirement and whether the logistic supply system can support the demonstrated USG-3B CEC reliability. 4. Correct the cause of the electromagnetic interference between the USG-3B CEC and the E-2D radar altimeter and demonstrate the corrections in a phase of FOT&E. 5. Take action on the recommendations contained in the classified DOT&E report to Congress on the CEC USG-3B FOT&E. 6. Update the CEC TEMP to include details of: -- The second phase of the USG-3B FOT&E with the supersonic seaskimming target scenario -- FOT&E of corrections made to the CEC USG-3B -- FOT&E of the CEC USG-2B with the Aegis Baseline 9 Combat System -- FOT&E of the CEC USG-2B with the DDG 1000 Combat System -- FOT&E of the CEC USG-2B with the CVN-78 Combat System -- FOT&E of USG-3B CEC to demonstrate the system’s ability to support the E-2D’s Theater Air and Missile Defense and Battle Force Command and Control missions -- The test program supporting the Acceleration of Mid- term Interoperability Improvements Project
N av y P R O G R A M S
CVN-78 Gerald R. Ford Class Nuclear Aircraft Carrier Executive Summary • The Commander, Operational Test and Evaluation Force (COTF) completed a DOT&E-approved operational assessment of the CVN-78 in October 2013. • It is unlikely that CVN-78 will achieve its Sortie Generation Rate (SGR) (number of aircraft sorties per day) requirement. The target threshold is based on unrealistic assumptions including fair weather and unlimited visibility, and that aircraft emergencies, failures of shipboard equipment, ship maneuvers (e.g., to avoid land), and manning shortfalls will not affect flight operations. DOT&E plans to assess CVN-78 performance during IOT&E by comparing to the demonstrated performance of the Nimitz class carriers. A demonstrated SGR less than the requirement but equal to or greater than the performance of the Nimitz class could potentially be acceptable. • CVN-78 incorporates newly designed catapults, arresting gear, weapons elevators, and radar, which are all critical for flight operations. The current reliability estimates for the catapult and arresting gear systems are a small fraction of their projected target for the shipboard configuration, and an even smaller fraction of the required reliability. Reliability test data are not available for the radar and the weapons elevators. DOT&E assesses that the poor or unknown reliability of these critical systems will be the most significant risk to CVN-78’s successful completion of IOT&E. • The CVN-78 design is intended to reduce manning. As manning requirements have been further developed, analysis indicates the present design has insufficient berthing for some ranks. The ship will not be delivered with sufficient empty berthing for the CVN-78’s Service Life Allowance (SLA). The SLA provides empty bunks to allow for changes in the crew composition over CVN-78’s expected 50-year lifespan, as well as ship riders for repairs, assists, and inspections. • The CVN-78 combat system for self-defense is derived from the combat system on current carriers and is expected to have similar capabilities and limitations. • The Navy continues to work on integration challenges related to the F-35 Joint Strike Fighter (JSF) and its fleet of aircraft carriers, including CVN-78. • Although CVN-78 will include a new Heavy underway replenishment (UNREP) system that will transfer cargo loads of up to 12,000 pounds, the Navy’s plan to install Heavy UNREP systems on resupply ships beginning in FY16 is unfunded. Heavy UNREP is needed to transfer JSF engines to CVN-78 when it is at-sea. • The current Test and Evaluation Master Plan (TEMP) does not adequately address integrated platform-level developmental testing, significantly raising the likelihood that platform-level problems will be discovered during IOT&E. The Program
Office is said to be addressing the problem and is in the process of refining the post-delivery schedule. • The Navy began CVN-78 construction in 2008. The schedule to deliver the ship has slipped from September 2015 to March 2016. The Electromagnetic Aircraft Launching System (EMALS), Advanced Arresting Gear (AAG), Dual Band Radar (DBR), and Integrated Warfare System will continue to drive the timeline. • On June 12, 2012, DOT&E rescinded approval of the alternative LFT&E Management Plan pertaining to the Gerald R. Ford (CVN-78) class carrier program. The Navy has not yet addressed the Full Ship Shock Trial (FSST) issue satisfactorily. System • The CVN-78 Gerald R. Ford class nuclear aircraft carrier program is a new class of nuclear-powered aircraft carriers that replaces the previous CVN-21 program designation. It has the same hull form as the CVN-68 Nimitz class, but many ship systems, including the nuclear plant and the flight deck, are new. • The newly designed nuclear power plant is intended to operate at a reduced manning level that is 50 percent of a CVN-68 class ship and produce significantly more electricity. • The CVN-78 will incorporate EMALS (electromagnetic, instead of steam-powered), and AAG, and will have a smaller island with a DBR (a phased-array radar which replaces/combines several legacy radars used on current aircraft carriers). • The Navy intends for the Integrated Warfare System to be adaptable to technology upgrades and varied missions throughout the ship’s projected operating life including
CVN-78 155
N av y P R O G R A M S
• • •
•
increased self-defense capabilities compared to current aircraft carriers. The ship’s DBR replaces the myriad radars on Nimitz class carriers serving in air traffic control and in ship self- defense. The Navy redesigned weapons stowage, handling spaces, and elevators to reduce manning, increase safety, and increase throughput of weapons. CVN-78 has design features intended to enhance its ability to launch, recover, and service aircraft, such as a slightly larger flight deck, dedicated weapons handling areas, and increased aircraft refueling stations. The Navy set the SGR requirement for CVN-78 to increase the sortie generation capability of embarked aircraft to 160 sorties per day (12-hour fly day) and to surge to 270 sorties per day (24-hour fly day) as compared to the CVN-68 Nimitz class SGR demonstration of 120 sorties per day/240 sorties for 24-hour surge. The Consolidated Afloat Networks and Enterprise Services (CANES) program replaces five shipboard legacy network programs to provide a common computing environment for command, control, intelligence, and logistics.
Activity Test Planning • The Navy continues to develop the CVN-78 SGR test modeling. The Navy plans to reestablish the SGR working group in early FY14. The ship’s SGR requirement is based on a 30-plus-day wartime scenario. The Navy designed a test to demonstrate the SGR with 6 consecutive 12-hour fly days followed by 2 consecutive 24-hour fly days. This live testing will be supplemented with modeling and simulation from the Virtual Carrier (VCVN) model to extrapolate results to the 30-plus-day SGR requirement. DOT&E concurs with this approach. • The CVN-78 Gerald R. Ford class carrier Program Office continues revising the TEMP in an effort to align planned developmental tests with corresponding operational test phases and to identify platform-level developmental testing. The Program Office released an updated Post‑Delivery Test and Trials schedule. • The Navy conducted all operational testing in accordance with a DOT&E-approved test plan. Operational Assessment • COTF conducted an operational assessment (OT-B3) from September 2012 through September 2013 to assess the ability of CVN-78 to successfully undergo its IOT&E in 2017. The COTF assessment was a desktop mission‑based analysis with specific emphasis on the review of previously identified issues as well as risk assessments of new issues. DOT&E participated in the assessment. DOT&E published an Operational Assessment report in December 2013, which will inform the Defense Acquisition Board decision regarding future procurement of CVN-79.
156 CVN-78
• CVN-78 is intended to support the JSF. • The Navy plans to declare CVN-78 Initial Operational Capability in FY17 and achieve Full Operational Capability in FY19 (after the ship completes IOT&E and the Type Commander certifies that CVN-78 is Major Combat Operations Ready). Mission Carrier Strike Group Commanders will use the CVN-78 to: • Conduct power projection and strike warfare missions using embarked aircraft • Provide force protection of friendly units • Provide a sea base as both a command and control platform and an air-capable unit Major Contractor Huntington Ingalls Industries, Newport News Shipbuilding – Newport News, Virginia
EMALS • The EMALS system functional design test site at Joint Base McGuire-Dix-Lakehurst, New Jersey, continues to test the new electromagnetic catapult system. Aircraft compatibility testing continued in 2013. Approximately 400 aircraft launches are being conducted using EA-18G, F/A-18E, F/A-18C, E-2D, T-45, and C-2 aircraft. The Navy has also conducted an additional 1,200 dead-load launches (non-aircraft, weight equivalent, simulated launches). Approximately 55 percent of the EMALS government furnished equipment (GFE) has been delivered to the shipyard. AAG • The Navy continues testing the AAG on a jet car track at Joint Base McGuire-Dix-Lakehurst, New Jersey. Testing has prompted design changes for the system’s Water Twisters, Cable Shock Absorbers, Mechanical Brake, and Arresting Engine Controller. Performance testing began in April 2013, and approximately 71 dead-load performance tests have been conducted. About 43 percent of the AAG GFE has been delivered to the shipyard. CANES • The Navy has scheduled developmental and follow-on operational testing of the force-level CANES configuration used on the Nimitz and Gerald R. Ford classes for 1Q and 2QFY15. A full system test of the Aegis destroyer configuration occurred this year. Developmental testing and IOT&E of the Aegis destroyer configuration are scheduled for 2Q and 3QFY14. DBR • The Navy reactivated the Engineering Development Model of the Volume Search Radar portion of the DBR at the
N av y P R O G R A M S Surface Combat System Center at Wallops Island, Virginia. The Navy planned to begin testing in January 2013; however, the testing has slipped repeatedly. The first government-led integrated test events began in 1QFY14. JPALS • The Navy conducted the Joint Precision Approach and Landing System (JPALS) operational assessment on CVN‑77 from May through August 2013. During the assessment, the Navy conducted at-sea requirements verification and collected data to support Navy Data Link Model, Performance Model, and Availability Model Verification, Validation, and Accreditation. A variety of afloat operations with a King Air (simulating the C-2A), MH-60S, and two F/A-18C aircraft were conducted, including about 120 approaches and 20 captures. Associated land-based testing was conducted at the Patuxent River Landing System Test Facility and the St. Inigoes (Maryland) Air Traffic Control Integration Laboratory. Both the afloat and land-based testing was terminated before it was completed because of an anticipated Nunn-McCurdy breach. JSF • The Navy is working to address several JSF integration challenges on its aircraft carriers. In general, these issues affect all of the Navy’s carriers, not just CVN-78. • In FY12, a test of the JSF arresting hook identified problems with the design. After failing to engage the arresting cable and demonstrating insufficient load-carrying capacity, the Navy has redesigned the arresting hook system and will test it at Joint Base McGuire‑Dix‑Lakehurst, New Jersey, in 1QFY14. • The Navy is redesigning the cooling system in the CVN‑78’s Jet Blast Deflectors (JBDs). The JBDs deflect engine exhaust during catapult launches. The redesign is needed to handle JSF engine exhaust and will include improvements in side-cooling panels. The Navy will install the redesigned JBDs into CVN-78 after ship delivery. • CVN-78 will receive the new Heavy UNREP system. To use the Heavy UNREP capability, both the carrier and the resupply ship must be equipped with the system. This new Heavy UNREP system, along with heavy vertical lift aircraft not embarked on carriers, are the only systems currently capable of resupplying the JSF engine and container while the carrier is underway. Today, only one combat logistic ship has Heavy UNREP, USNS Arctic. The installation on other Combat Logistic Fleet ships is planned for FY16, but is currently unfunded. • The JSF engine container was unable to sustain the required sudden drop of 18 inches (4.5 g’s) without damage to the power module during shock testing. The Navy is redesigning the container to better protect the engine, which will likely result in an increase in container size and weight. The Navy estimates the new container will be available in late calendar year 2016. • The Navy is designing separate charging and storage lockers for the lithium-ion batteries required for the JSF.
The Navy is also designing a new storage locker for pilot flight equipment as the JSF helmet is larger and more fragile than legacy helmets. • The Navy has completed JSF cyclic thermal strain testing and concluded that repeated JSF sortie generation at combat rated thrust, i.e., afterburner, will not cause cyclic thermal strain on the CVN-78 flight deck structure. • The National Security Agency has determined that the JSF Prognostic Health Management (PHM) system downlink poses unacceptable security risks. The PHM reports on the health of the aircraft as it returns from a mission. The Navy has not established a path forward because the JSF Program Office does not have funding to address this issue. • Unlike current fleet aircraft, the JSF carries ordnance in internal bays. This will require changes to aircraft firefighting techniques for the JSF. The Navy has continued to conduct mock firefighting testing to develop new procedures in the event of a fire on the flight deck near aircraft carrying internal ordnance. • The JSF Program Office has initiated a tire redesign because of higher than predicted failure rates. The Navy has not yet settled on a strategy for dealing with a possible higher tire storage requirement. LFT&E • On June 12, 2012, DOT&E rescinded approval of the alternative LFT&E Management Plan pertaining to the Gerald R. Ford class carrier program because the Navy deferred the FSST to CVN-79. Assessment Test Planning • The current state of the VCVN model does not fully provide for an accurate accounting of SGR due to a lack of fidelity regarding manning and equipment/aircraft availability. Spiral development of the VCVN model continues in order to ensure that the required fidelity will be available to support the SGR assessment during IOT&E. • A new TEMP is under development to address problems with the currently-approved TEMP. The current TEMP does not adequately address platform-level developmental testing. The Program Office has begun to refine the Post Delivery Test and Trials schedule, but that schedule still lacks sufficient details to ensure reasonable developmental testing. Lack of platform-level developmental testing significantly raises the likelihood of the discovery of platform-level problems during IOT&E. • The Navy plans to deliver CVN-78 in February 2016. The ship’s post-shipyard shakedown availability will follow delivery in 2016. During the post-shipyard shakedown availability installations of some systems will be completed. The first at-sea operational test and evaluation of CVN-78 will begin in July 2017. Reliability • CVN-78 includes several systems that are new to aircraft carriers; four of these systems stand out as being critical to flight operations: EMALS, AAG, DBR, and the Advanced
CVN-78 157
N AV Y P R O G R A M S Weapons Elevators (AWEs). Overall, the uncertain reliability of these four systems is the most significant risk to the CVN-78 IOT&E. All four of these systems will be tested for the first time in their shipboard configurations aboard CVN-78. Reliability estimates derived from test data are available for EMALS and AAG and are discussed below. For DBR and AWE, estimates based on test data are not available and only engineering reliability estimates are available. SGR • It is unlikely that CVN-78 will achieve its SGR requirement. The target threshold is based on unrealistic assumptions including fair weather and unlimited visibility, and that aircraft emergencies, failures of shipboard equipment, ship maneuvers (e.g., to avoid land), and manning shortfalls will not affect flight operations. DOT&E plans to assess CVN-78 performance during IOT&E by comparing to the demonstrated performance of the Nimitz class carriers. A demonstrated SGR less than the requirement but equal to or greater than the performance of the Nimitz class could potentially be acceptable. • During the operational assessment, DOT&E conducted an analysis of past aircraft carrier operations in major conflicts. The analysis concludes that the CVN-78 SGR requirement is well above historical levels and that CVN-78 is unlikely to achieve that requirement. There are concerns with the reliability of key systems that support sortie generation on CVN-78. Poor reliability of these critical systems could cause a cascading series of delays during flight operations that would affect CVN-78’s ability to generate sorties, make the ship more vulnerable to attack, or create limitations during routine operations. DOT&E assesses the poor or unknown reliability of these critical subsystems will be the most significant risk to CVN-78’s successful completion of IOT&E. The analysis also considered the operational implications of a shortfall and concluded that as long as CVN-78 is able to generate sorties comparable to Nimitz class carriers, the operational implications of CVN-78 will be similar to that of a Nimitz class carrier. Manning • Current manning estimates have shortages of bunks for Chief Petty Officers (CPOs) and do not provide the required 10 percent SLA. Per Office of the Chief of Naval Operations Instruction 9640.1B, Shipboard Habitability Program, all new ships are required to have a growth allowance of 10 percent of the ship’s company when the ship delivers. The SLA provides empty bunks to allow for changes in the crew composition over CVN-78’s expected 50-year lifespan and provides berthing for visitors and Service members temporarily assigned to the ship. EMALS • EMALS is one of the four systems critical to flight operations. While testing to date has demonstrated that EMALS should be able to launch aircraft planned for CVN-78’s air wing, the system’s reliability is uncertain.
158
CVN-78
At the Lakehurst, New Jersey, test site, over 1,967 launches have been conducted and 201 chargeable failures have occurred. Based on available data, the program estimates that EMALS has approximately 240 Mean Cycles Between Critical Failure in the shipboard configuration, where a cycle represents the launch of one aircraft. Based on expected reliability growth, the failure rate is presently five times higher than should be expected. AAG • AAG is another system critical to flight operations. Testing to date has demonstrated that AAG should be able to recover aircraft planned for the CVN-78 air wing, but as with EMALS, AAG’s reliability is uncertain. At the Lakehurst, New Jersey test site, 71 arrestments were conducted earlier this year and 9 chargeable failures occurred. The Program Office estimates that AAG has approximately 20 Mean Cycles Between Operational Mission Failure in the shipboard configuration, where a cycle represents the recovery of one aircraft. Based on expected reliability growth, the failure rate is presently 248 times higher than should be expected. DBR • Previous testing of Navy combat systems similar to CVN-78’s revealed numerous integration problems that degrade the performance of the combat system. Many of these problems are expected to exist on CVN-78. The previous results emphasize the necessity of maintaining a DBR/CVN-78 combat system asset at Wallops Island. The Navy is considering long-term plans (i.e., beyond FY15) for testing DBR at Wallops Island, Virginia, but it is not clear if resources and funding will be available. Such plans are critical to delivering a fully-capable combat system and ensuring lifecycle support after CVN-78 delivery in 2016. JPALS • The Navy has proposed to the USD(AT&L) Milestone Decision Authority that the program be restructured from its current, land- and sea-based, multiple-increment structure to a single increment focusing on sea-based requirements primarily supporting JSF and future Unmanned Carrier Launched Airborne Surveillance and Strike aircraft. Under this proposed restructuring scheme, there will be no retrofitting of JPALS on legacy aircraft and the Navy will need to maintain both the legacy approach and landing system and JPALS onboard each aircraft-capable ship. JSF • The arresting hook system remains an integration risk as the JSF development schedule leaves no time for discovering new problems. The redesigned tail hook has an increased downward force as well as sharper design that may induce greater than anticipated wear on the flight deck. • JSF noise levels remain moderate to high risk in JSF integration and will require modified carrier flight deck procedures. - Flight operations normally locate some flight deck personnel in areas where double hearing protection
N av y P R O G R A M S would be insufficient during F-35 operations. To partially mitigate noise concerns, the Navy will procure new hearing protection with active noise reduction for flight deck personnel. -- Projected noise levels one level below the flight deck (03 level), which includes mission planning spaces, will require at least single hearing protection that will make mission planning difficult. The Navy is working to mitigate the effects of the increased noise levels adjacent to the flight deck. • Storage of the JSF engine is limited to the hangar bay, which will affect hangar bay operations. The impact on the JSF logistics footprint is not yet known. • Lightning protection of JSF aircraft while on the flight deck will require the Navy to modify nitrogen carts to increase their capacity. Nitrogen is used to fill fuel tank cavities while aircraft are on the flight deck. • JSF remains unable to share battle damage assessment and non-traditional Intelligence, Surveillance, and Reconnaissance information captured on the aircraft portable memory device or cockpit voice recorder in real-time. In addition, the CVN-78 remains unable to receive and display imagery transmitted through Link 16 because of bandwidth limitations. These capability gaps were identified in DOT&E’s FY12 Annual Report. The Combatant Commanders have requested these capabilities to enhance decision-making. LFT&E • While the Navy has made substantial effort in component and surrogate testing, this work does not obviate the need to conduct the FSST to gain the critical empirical data that past testing has repeatedly demonstrated are required to rigorously evaluate the ship’s ability to withstand shock and survive in combat. Shock Trials conducted on both the Nimitz class aircraft carrier and the San Antonio class Amphibious Transport Dock demonstrated the need for and substantial value of conducting the FSST. Postponing the FSST until CVN-79 would cause a five- to seven-year delay in obtaining the data critical to evaluating the survivability of the CVN-78 and would preclude timely modification of subsequent ships of this class to assure their survivability. • CVN-78 has many new critical systems that have not undergone shock trials on other platforms. Unlike past tests on other new classes of ships with legacy systems, the performance of CVN-78’s new critical systems under test is unknown.
• The Navy proposes delaying the shock trial by five to seven years because of the approximately four- to six-month delay required to perform the FSST. The benefit of having test data to affect the design of future carriers in the class outweighs the delay in delivery of CVN-78 to the fleet to conduct this test. The delay is not a sufficient reason to postpone the shock trial. Recommendations • Status of Previous Recommendations. The Navy should continue to address the seven remaining FY10 and FY11 recommendations. 1. Adequately test and address integration challenges with JSF; specifically: -- Logistics (unique concerns for storage and transportation) -- Changes required to JBDs -- Changes to flight deck procedures due to heat and noise -- Autonomic Logistics Information System integration 2. Finalize plans that address CVN-78 Integrated Warfare System engineering and ship’s self-defense system discrepancies prior to the start of IOT&E. 3. Continue aggressive EMALS and AAG risk-reduction efforts to maximize opportunity for successful system design and test completion in time to meet required in-yard dates for shipboard installation of components. 4. Continue development of a realistic model for determining CVN-78’s SGR, while utilizing realistic assumptions regarding equipment availability, manning, and weather conditions for use in the IOT&E. 5. Provide scheduling, funding, and execution plans to DOT&E for the live SGR test event during the IOT&E. 6. Continue to work with the Navy’s Bureau of Personnel to achieve adequate depth and breadth of required personnel to sufficiently meet Navy Enlisted Classification fit / fill manning requirements of CVN-78. 7. Conduct system-of-systems developmental testing to preclude discovery of deficiencies during IOT&E. • FY13 Recommendations. The Navy should: 1. Address the uncertain reliability of EMALS, AAG, DBR, and AWE. These systems are critical to CVN-78 flight operations, and are the largest risk to the program. 2. Conduct fully integrated, robust, end-to-end testing of the proposed JPALS, to include operations in neutral and potentially hostile electronic warfare environments.
CVN-78 159
N av y P R O G R A M S
160
N av y P R O G R A M S
DDG 51 Flight III Destroyer/Air and Missile Defense Radar (AMDR)/Aegis Modernization Executive Summary • On May 22, 2013, DOT&E disapproved the Air and Missile Defense Radar (AMDR) Test and Evaluation Master Plan (TEMP) because the proposed operational test approach did not adequately assess the capability of that radar to support the DDG 51 Flight III Destroyer’s self-defense mission. - Safety restrictions preclude realistic testing on manned ships in this region of the battlespace. Consequently, an unmanned test ship equipped with an AMDR and an Aegis DDG 51 Flight III Destroyer Combat System is required for adequate operational testing and assessment of the AMDR and DDG 51 Flight III Destroyer’s self-defense capabilities. - This approach is similar to the Self-Defense Test Ship (SDTS) currently used for testing the self-defense capabilities of ships equipped with Ship Self-Defense System (SSDS)-based combat systems • On August 9, 2013, DOT&E disapproved the Aegis Modernization TEMP because the proposed operational testing did not provide the credible modeling and simulation (M&S) effort needed to fully assess the DDG 51’s combat system self-defense capability, nor a means to validate the M&S (i.e., an unmanned SDTS equipped with an AMDR and the DDG 51 Flight III Combat System). System • The DDG 51 Flight III Destroyer is a combatant ship equipped with the: - AMDR three-dimensional (range, altitude, and azimuth) multi-function radar - AN/SQQ-89 Undersea Warfare suite that includes the AN/SQS-53 sonar - MH-60R helicopter - Close-In Weapon System - Five-inch diameter gun - Vertical Launch System that can launch Tomahawk, Standard (SM-2, -3, and -6), and Evolved SeaSparrow Missiles (ESSMs) • The Navy is developing the AMDR to provide simultaneous sensor support of integrated air and missile defense (IAMD) and air defense (including self-defense) missions. IAMD and air defense require extended detection ranges and increased radar sensitivity against advanced threats with high speeds and long interceptor fly out times. The three AMDR major components are: - The AMDR S-band radar (AMDR-S) will provide search, track, cueing, missile discrimination, air defense Non-Cooperative Target Recognition, S-band missile communications, surveillance capability for
ship self‑defense and area air defense, and S-band kill‑assessment support functions. - The AMDR X-band radar (AMDR-X) will provide horizon and surface search capabilities in addition to navigation and periscope detection/discrimination functions. The Navy is delaying development of the AMDR-X. The AN/SPQ-9B X-band radar will provide these functions in the interim. - The Radar Suite Controller will provide the open interface with the ship combat system. • The Aegis Combat System is an integrated naval weapons system that uses computers and radars to form an advanced command and decision, and a weapon control system to track and guide weapons to destroy enemy targets. - The Navy’s Aegis Modernization program is a planned, phased program that provides updated technology and combat systems for existing Aegis-guided missile cruisers (CG 47) and destroyers (DDG 51) as well as the DDG 51 Flight III Destroyers. - The Aegis Modernization program will provide an improved Advanced Capability Build combat system variant for the DDG 51 Flight III Destroyers equipped with the AMDR.
Mission • The Navy will use the DDG 51 Flight III Destroyer equipped with the Aegis Modernization program and AMDR to provide joint battlespace threat awareness and defense capability to counter current and future threats in support of joint forces ashore and afloat.
DDG 51/AMDR/Aegis Mod 161
N av y P R O G R A M S • The Navy will use the AMDR-S/Radar Suite Controller with the AN/SPQ-9B and the Aegis Modernization Program to support the following DDG 51 Flight III Destroyer missions: - Support area air defense (to include self-defense) to counter advanced air and cruise missile threats and increase ship survivability - Detect, track, discriminate, and provide missile engagement support (including kill assessment) to counter ballistic missile threats - Support surface surveillance, precision tracking, and missile and gun engagements to counter surface threats - Support Undersea Warfare with periscope detection and discrimination - Detect and track enemy artillery projectiles to support combat system localization of land-battery launch positions by the DDG 51 Flight III Combat System
Activity • DOT&E issued two classified memoranda to USD(AT&L) (February 25 and May 5, 2013) in preparation for the AMDR Milestone B decision. Both memoranda highlighted severe shortfalls in the operational test plans in the AMDR and DDG 51 Flight III ship self-defense test arena and stressed the requirement for an unmanned SDTS equipped with the AMDR and DDG 51 Flight III Combat System for adequate operational testing of the radar and ship’s combat system self-defense capability. • DOT&E disapproved the AMDR TEMP on May 22, 2013, because the proposed operational test approach did not adequately assess the capability of the AMDR to support the DDG 51 Flight III Destroyer’s self-defense mission. • DOT&E disapproved the Aegis Modernization TEMP on August 9, 2013, because the proposed operational testing did not provide a credible M&S effort needed to fully assess the ship’s combat system self-defense capability nor a means to validate the M&S (i.e., an unmanned SDTS equipped with an AMDR and the DDG 51 Flight III Combat System). Assessment • The operational test programs for the AMDR, Aegis Modernization, and DDG 51 Flight III Destroyer programs are not adequate to fully assess their self-defense capabilities in addition to being inadequate to test the following Navy‑approved AMDR and DDG 51 Flight III requirements. - The AMDR Capability Development Document describes AMDR's IAMD mission, which requires AMDR to support simultaneous defense against multiple ballistic missile threats and multiple advanced anti-ship cruise missile (ASCM) threats. The Capability Development Document also includes an AMDR minimum track range Key Performance Parameter. - The DDG 51 Flight III Destroyer has a survivability requirement directly tied to meeting a self-defense
162 DDG 51/AMDR/Aegis Mod
- Detect and track own-ship gun projectiles in support of surface warfare and naval surface fire support Major Contractors • DDG 51 Destroyer - General Dynamics Marine Systems Bath Iron Works – Bath, Maine - Huntington Ingalls Industries, Ingalls Shipbuilding Division – Pascagoula, Mississippi • AMDR - Raytheon – Sudbury, Massachusetts • Aegis Modernization Program - Lockheed Martin Marine Systems and Sensors – Moorestown, New Jersey
requirement threshold against ASCMs described in the Navy’s Surface Ship Theater Air and Missile Defense Assessment document of July 2008. It clearly states that area defense will not defeat all the threats, thereby demonstrating that area air defense will not completely attrite all ASCM raids and that individual ships must be capable of defeating ASCM leakers in the self-defense zone. • Conduct of operational testing with threat‑representative ASCM surrogates in the close-in, self-defense battlespace using manned ships is not possible since current Navy test range safety restrictions preclude testing on manned ships in this region because targets and debris from intercepts will pose an unacceptable risk to personnel at ranges where some of the engagements will take place. - In addition to stand-off ranges (on the order of 2 to 5 nautical miles for subsonic and supersonic surrogates, respectively), safety restrictions require that supersonic ASCM targets not be flown directly at a manned ship, but at some cross-range offset (approximately 1 nautical mile), which unacceptably degrades the operational realism of the test. - Similar range safety restrictions will preclude testing the AMDR minimum track range requirement against supersonic, sea-skimming ASCM threat-representative surrogates at the land-based AMDR test site at the Pacific Missile Range Facility. • Due to the inherent complexity and safety limitations, live testing (without an SDTS) cannot provide sufficient data to assess the self-defense capabilities of the AMDR and the DDG 51 Flight III Destroyer. - M&S will therefore play a major role in determining those capabilities. However, per public law, M&S cannot be the only contributor to the assessment; realistic operational test results are required.
N av y P R O G R A M S - M&S can support an operational evaluation, but must be accredited not only with manned test ship testing, but also through end-to-end testing against operationally realistic targets equipped with an ADMR and the DDG 51 Flight III Destroyer Combat System in the close-in, self-defense battlespace. - The extent to which the Navy can use M&S to assess AMDR and DDG 51 Flight III's self-defense capability depends critically on whether the M&S can be rigorously accredited for operational testing. - Side-by-side comparison between credible live fire test results and M&S test results form the basis for M&S accreditation. Without an Aegis SDTS, there will not be a way to gather the operationally realistic live fire test data needed for comparison to accredit the M&S. • The Air Warfare/Ship Self Defense Enterprise M&S accreditation paradigm being used in the test programs for LHA-6, Littoral Combat Ship (LCS), DDG 1000, LPD-17, LSD-41/49, and CVN-78 ship classes was approved by the Navy and DOT&E in 2005. It is based on live fire events conducted on manned ships and an SDTS, as well as M&S events conducted in the same configuration. - The live firings conducted in the close-in, self-defense battlespace can only be accomplished with an SDTS due to the range safety restrictions on testing with manned ships. - For the AMDR and DDG 51 Flight III, the paradigm will be the same; whatever end-to­ end M&S tool is developed must be accredited for use in operational testing by comparing live fire results in the close-in battlespace to simulated events in the close-in battlespace. - Those live fire events can only be conducted on an SDTS equipped with the AMDR and the DDG 51 Flight III Destroyer Combat System. DOT&E considers that paradigm to be the credible template for application by the AMDR and DDG 51 Flight III Destroyer operational test programs. • The Navy currently models the Aegis Weapon System (AWS) with Lockheed Martin’s Multi-Target Effectiveness Determined under Simulation by Aegis (MEDUSA) M&S tool. - MEDUSA encompasses several components of the AWS including the SPY-1 radar, Command and Decision, and Weapon Control System. MEDUSA models AWS performance down to the system specification and the Navy considers it a high-fidelity simulation of AWS. - However, it is not a tactical code model; so, its fidelity is ultimately limited to how closely the specification corresponds to the Aegis tactical code (i.e., the specification is how the system is supposed to work while the tactical code is how the system actually works). This adds to the need for realistic live fire shots to support validation efforts. - By comparison, the Air Warfare/Ship Self Defense Enterprise M&S test bed used for assessing USS San Antonio’s (LPD-17) self-defense capabilities used re-hosted SSDS Mk 2 tactical code.
• Recent test events highlight the limitations of specification models like MEDUSA. During Aegis Advanced Capability Build 08 testing in 2011, five AWS software errors were found during live fire events and tracking exercises. - Three software errors contributed to a failed SM-2 engagement, one to a failed ESSM engagement, and one to several failed simulated engagements during tracking exercises. - Since these problems involved software coding errors, it is unlikely that a specification model like MEDUSA (which assumes no software errors in tactical code) would account for such issues and hence it would overestimate the combat system’s capability. • Since Aegis employs ESSM in the close-in, self-defense battlespace, understanding ESSM's performance is critical to understanding the self-defense capabilities of the DDG 51 Flight III Destroyer. - Past DOT&E Annual Reports have stated that the ESSM’s operational effectiveness has not been determined. The Navy has not taken action to adequately test the ESSM’s operational effectiveness. - Specifically, because safety limitations preclude ESSM firing in the close-in self-defense battlespace, there are very little test data available concerning ESSM's performance, as installed on Aegis ships, against supersonic ASCM surrogates. - Any data available regarding ESSM's performance against supersonic ASCM surrogates are from an SSDS- based combat system configuration, using a completely different guidance mode or one that is supported by a different radar suite. • The cost of building and operating an Aegis SDTS is small when compared to the total cost of the AMDR development/ procurement and the eventual cost of the 22 (plus) DDG 51 Flight III ships that are planned for acquisition ($55+ Billion). Even smaller is the cost of the SDTS compared to the cost of the ships that the DDG 51 Flight III Destroyer is expected to protect (~$450 Billion in new ship construction over the next 30 years). - If DDG 51 Flight III Destroyers are unable to defend themselves, these other ships are placed at greater risk. - Moreover, the SDTS is not a one-time investment for only the AMDR/DDG 51 Flight III IOT&E, as it would be available for other testing that cannot be conducted with manned ships (e.g., the ESSM Block 2) and as the combat system capabilities are improved. Recommendations • Status of Previous Recommendations. There are no previous recommendations. • FY13 Recommendations. The Navy should: 1. Program and fund an SDTS equipped with the AMDR and DDG 51 Flight III Combat System in time for the AMDR/DDG 51 Flight III Destroyer IOT&E. 2. Modify the AMDR, Aegis Modernization, and DDG 51 Flight III TEMPs to include a phase of IOT&E using an
DDG 51/AMDR/Aegis Mod 163
N av y P R O G R A M S SDTS equipped with the AMDR and DDG 51 Flight III Combat System. 3. Modify the AMDR, Aegis Modernization, and DDG 51 Flight III TEMPs to include a credible M&S effort that will enable a full assessment of the AMDR and DDG 51 Flight III Combat System’s self-defense capabilities.
164 DDG 51/AMDR/Aegis Mod
N av y P R O G R A M S
DDG 1000 – Zumwalt Class Destroyer Executive Summary • The first ship in the DDG 1000 class was launched on October 28, 2013, with fabrication over 85 percent complete. The Navy initiated lead ship pre-delivery testing in FY13 and will continue in FY14. • The Navy conducted live fire tactical guided flight and lethality testing of the Long-Range Land Attack Projectile (LRLAP) in FY13. Analysis of test results is ongoing. • The Navy successfully completed initial integration and compatibility testing of the ship’s Engineering Control System at the DDG 1000 Integrated Power System (IPS) Land-Based Test Site (LBTS). System DDG 1000 is a new combatant ship with a wave piercing hull form designed both for endurance and to be difficult to detect on radar. It is equipped with the following: • Total Ship Computing Environment Infrastructure that hosts all ship functions on an integrated, distributed computing plant • Two 155 mm Advanced Gun Systems that fire LRLAPs • AN/SPY-3 Multi-Function (X-band) Radar modified to include a volume search capability (the Navy removed the Volume Search Radar (S-band) from the ship baseline design for cost reduction per an Acquisition Decision Memorandum of June 1, 2010) • Eighty vertical launch cells that can hold a mix of Tomahawk Land Attack Missiles, Standard Missiles, Vertical Launch Anti-Submarine Rockets, and Evolved Sea Sparrow Missiles • Integrated Undersea Warfare system with a dual frequency bow-mounted sonar and multi-function towed array sonar to detect submarines and assist in avoiding mines • An ability to embark and maintain MH-60R helicopters and vertical take-off unmanned aerial vehicles
Activity • The Navy continues to revise the Test and Evaluation Master Plan (TEMP). The most significant issues being addressed in the TEMP revision are: - Removal of Volume Search Radar - Replacement of an integrated Mk 110 57 mm close-in gun system with non-integrated Mk 46 30 mm guns - Changes in hardware and software delivery schedule, including the delivery of Anti-Submarine Warfare, mine avoidance, Tomahawk Land Attack Missile, and Advanced Gun System counterbattery capability during Post‑Shakedown Availability
Mission • The Joint Force Maritime Component Commander can employ DDG 1000 to provide: - Joint Strike - Joint Fire Support - Anti-Surface Warfare - Anti-Air Warfare - Anti-Submarine Warfare • DDG 1000 is intended to operate independently or in conjunction with an Expeditionary or Carrier Strike Group, as well as with other joint or coalition partners in a Combined Expeditionary Force environment. Major Contractors • General Dynamics Marine Systems Bath Iron Works – Bath, Maine • Huntington Ingalls Industries – Pascagoula, Mississippi • BAE Systems – Minneapolis, Minnesota • Raytheon – Waltham, Massachusetts
• The last annual report of DDG 1000 class ships was in FY09. Since that time, the Navy: - Successfully completed initial integration and compatibility testing of the ship’s Engineering Control System with the ship’s major IPS hardware at the DDG 1000 IPS LBTS in March 2012. - Continued guided flight testing and lethality testing of LRLAP at the Army Test and Evaluation Command’s White Sands Missile Range. Lethality testing was conducted in accordance with DOT&E-approved test plans and is now complete. The Navy’s Lethality Assessment
DDG 1000 165
N av y P R O G R A M S Report is expected in FY14. One remaining guided flight test is scheduled to occur in FY14. - Commenced development of the DDG 1000 Probability of Raid Annihilation (PRA) test bed, modeling and simulation that will be used to assess DDG 1000 capability against threat anti-ship cruise missiles and threat aircrafts. - Is developing and validating ship models to assess the vulnerability of DDG 1000 to above-water and below-water threats. These models will support Damage Scenario Based Engineering Analyses and the Total Ship Survivability Trials. The Navy’s Final Vulnerability Assessment Report is expected in FY16. • In January 2013, DOT&E sent a memorandum to the Assistant Secretary of the Navy (Research, Development, and Acquisition) outlining the need for a threat torpedo surrogate to support operational testing. Assessment • Analysis of the LRLAP lethality test data is ongoing. However, the preliminary assessment is that the LRLAP is lethal against expected realistic targets. • Integration and compatibility testing at the LBTS provided early identification and correction of deficiencies within the ship’s power distribution system and should reduce cost of post-installation deficiency correction. • A component shock qualification program is required for assessing ship vulnerability to below-water threats and is necessary for accurate damage simulations. However, the shock qualification program remains unfunded. • Two LRLAP failures in the first 20 guided flight tests were the result of accelerometer failure. The Navy incorporated
166 DDG 1000
additional quality screening requirements for its accelerometer and has seen no issues in the successive 14 flight tests. Recent performance indicates that the probability of accelerometer failure may have been reduced; however, the limited number of firings prior to operational test will not validate the effectiveness of these actions. Recommendations • Status of Previous Recommendations. The Navy should address the following open recommendations from FY09 or earlier: - Develop tactics and training that optimize employment of the Mk 46 gun systems against surface threats. - Fund and schedule component shock qualification to support the DDG 1000 class requirement to maintain all mission essential functions when exposed to underwater explosive shock loading. • FY13 Recommendations. The Navy should: 1. Determine a development and test strategy that mitigates the risk of delivering substantial mission capability during Post‑Shakedown Availability. 2. Complete the revision to the TEMP that accounts for DDG 1000 baseline changes and system delivery schedule. 3. Develop a strategy to validate reliability of the accelerometers used in LRLAP prior to shipboard operational test. 4. Develop and execute an accreditation plan that validates the acceptability of the PRA test bed to support operational test. 5. Identify and/or develop a threat torpedo surrogate to support operational test in FY16.
N av y P R O G R A M S
Distributed Common Ground System – Navy (DCGS-N) Executive Summary • The Commander, Operational Test and Evaluation Force (COTF) conducted an FOT&E of the Distributed Common Ground System – Navy (DCGS-N) Increment 1, Block 1 Early Adopters (EAs) Engineering Change Proposal (ECP) from November 2011 through August 2012. • DOT&E evaluated DCGS-N Increment 1, Block 1 EA ECP to be effective and suitable for the Navy to conduct intelligence missions but recommended more robust Information Assurance testing. System • DCGS-N is the Navy Service component of the DoD DCGS family-of-systems, providing multi-Service integration of Intelligence, Surveillance, Reconnaissance, and Targeting capabilities. • DCGS-N Increment 1 uses commercial off-the-shelf (COTS) and mature government off-the-shelf (GOTS) software, tools, and standards. It interoperates with the DCGS family- of- systems via implementation of the DCGS Integration Backbone and Net-Centric Enterprise Services standards. • Increment 1 is divided into two blocks: Block 1 delivered initial capability on the legacy ship networks and Block 2 was intended to host the DCGS-N application on the Consolidated Afloat Networks and Enterprise Services (CANES). • When the CANES program was delayed, the Navy proceeded to update ships equipped with the legacy networks with new COTS and GOTS hardware and software. These updated networks were called EAs. The DCGS-N program was forced to implement an ECP so that it could work with the modernized EA networks, vice the legacy networks. • Block 2 is intended to be hosted on CANES and is expected to deliver enhanced functionalities, including newer versions of both COTS and GOTS applications.
Mission • The operational commander will use DCGS-N to participate in the Joint Task Force-level targeting and planning processes and to share and provide Navy-organic Intelligence, Surveillance, and Reconnaissance, and Targeting data to Joint Forces. • Users equipped with DCGS-N will: - Identify, locate, and confirm targets through multi-source intelligence feeds - Update enemy track locations and provide situational awareness to the Joint Force Maritime Component Commander by processing data drawn from available sensors
Activity • COTF conducted an FOT&E of DCGS-N Increment 1, Block 1 EA ECP from November 2011 through August 2012 in accordance with the DOT&E-approved test plan. The FOT&E used data from two integrated tests conducted onboard the USS Bonhomme Richard from November 2011 through March 2012, and onboard the USS Nimitz from June through July 2012. COTF published the FOT&E test report on February 26, 2013. • From November 5 – 16, 2012, the Navy conducted Developmental Test (DT)-1, the first of three Block 2 developmental tests, in the Navy’s Enterprise Engineering and Certification laboratory in San Diego, California, in
accordance with the DOT&E-approved Test and Evaluation Master Plan. On March 27, 2013, COTF published the Operational Assessment report. • DOT&E submitted a memorandum report to the Milestone Decision Authority on the results of Block 1 EA ECP and Block 2 tests on March 25, 2013. • The Navy will conduct two more developmental test events: DT-2 in the Navy’s Enterprise Engineering and Certification laboratory and DT-3 onboard a CANES-equipped ship. The Navy will conduct an FOT&E on the same ship in 2Q/3QFY15.
Major Contractor BAE Systems, Electronics, Intelligence and Support (EI&S) – San Diego, California, and Charleston, South Carolina
DCGS-N 167
N av y P R O G R A M S Assessment • DOT&E evaluated the Block 1 EA ECP system to be operationally effective and suitable, but Information Assurance testing was not adequate to assess survivability against cyber threats to the system. • The first developmental test of the Block 2 software (DT-1) did not demonstrate software maturity. The Program Office is working towards the resolution of all Priority 1 and 2 software problems before the IOT&E.
168 DCGS-N
Recommendations • Status of Previous Recommendations. The Navy addressed all previous recommendations. • FY13 Recommendations. None.
N av y P R O G R A M S
E-2D Advanced Hawkeye Executive Summary • DOT&E provided details on the FY12 IOT&E in the classified E-2D Advanced Hawkeye (AHE) IOT&E report in February 2013. The E-2D was operationally effective for legacy missions and suitable for legacy missions conducted from land-based operations. Suitability for carrier-based operations was unresolved. • On March, 1, 2013, USD(AT&L) approved entry into full-rate production (FRP) and procurement of FRP Lot 1 aircraft. An In-Progress Review to receive approval for FRP Lot 2 and beyond is scheduled for FY14. • The Navy’s Cooperative Engagement Capability (CEC) FOT&E occurred between October 2012 and June 2013. It focused on testing the USG-3B, the CEC system specific for the E-2D and a critical enabler for the Theater Air and Missile Defense (TAMD) mission. The testing focused on interoperability with legacy CEC systems. During FOT&E, CEC performance was degraded relative to the CEC version in the E-2C. • In 4QFY13, Commander, Operational Test and Evaluation Force conducted a Verification of Correction of Deficiencies (VCD) in order to assess the program’s progress in addressing deficiencies found during IOT&E. Not all of the problems identified in the DOT&E IOT&E report will be resolved during the VCD, but will be corrected through a series of hardware and software changes that are incorporated and demonstrated through FY16. System • The E-2D AHE is a carrier-based Airborne Early Warning and Command and Control aircraft. • Significant changes to this variant of the E-2 include upgraded engines to provide increased electrical power and cooling relative to current E-2C aircraft; a strengthened fuselage to support increased aircraft weight; replacement of the radar system, the communications suite, and the mission computer; and the incorporation of an all-glass cockpit, which permits the co-pilot to act as a tactical fourth operator in support of the system operators in the rear of the aircraft.
Activity • In 4QFY13, Commander, Operational Test and Evaluation Force conducted a VCD to assess the program’s progress in addressing deficiencies found during IOT&E at Naval Air Station (NAS) Patuxent River, Maryland; NAS Fallon, Nevada; and Point Mugu, California. The Program Office has developed a multi-year test strategy to address all IOT&E deficiencies.
• The radar upgrade replaces the E-2C mechanically-scanned radar with a phased-array radar that has combined mechanical and electronic scan capabilities. • The upgraded radar provides significant improvement in littoral and overland detection performance and TAMD capabilities. • The E-2D AHE System includes all simulators, interactive computer media, and documentation to conduct maintenance, as well as aircrew, shore-based initial and follow-on training. Mission The Combatant Commander, whether operating from the aircraft carrier or from land, will use the E-2D AHE to accomplish the following missions: • Theater air and missile sensing and early warning • Battlefield management, command, and control • Acquisition, tracking, and targeting of surface warfare contacts • Surveillance of littoral area objectives and targets • Tracking of strike warfare assets Major Contractor Northrop Grumman Aerospace Systems – Bethpage, New York
• DOT&E provided details on the FY12 IOT&E in the classified E-2D AHE IOT&E report in February 2013. • The Navy conducted the CEC FOT&E in accordance with a DOT&E-approved test plan from October 2012 through June 2013 in overland and overwater environments with CEC networks of varying complexity, which included E-2D,
E-2D 169
N av y P R O G R A M S E-2C, USS Nimitz, USS Preble, USS Princeton, USS Shoup, USS Stockdale, USS William P. Lawrence, and various land-based towers. The most complex network consisted of nine participants or roughly two times the normal operational environment of four to five participants. Assessment • Based on the CEC FOT&E results, the current implementation of the CEC in the E-2D has degraded effectiveness relative to that of the E-2C. Software changes will be required to improve E-2D CEC performance. • The February 2013 IOT&E report included the following assessments: - Due to testing limitations, DOT&E could not fully assess E-2D’s TAMD mission, but currently assesses it as not operationally effective based on current identified deficiencies. - The E-2D was not operationally suitable for the TAMD mission based upon poor availability. - The E-2D is operationally effective for legacy missions and suitable for legacy missions conducted from land. Suitability of carrier operations was unresolved. E-2D has improved surveillance capabilities relative to the E-2C. Test aircrews identified performance shortfalls with operator workload in dynamic, high-target density environments where the E-2D mission system erroneously swapped identification labels for crossing/closely-spaced aircraft tracks. Subsequently, the tracks required manual aircrew re-labeling in the mission system, which can lead to operator overload and loss of situational awareness. - The E-2D AHE demonstrated significant improvements to the radar tracking capability over the E-2C. While the system provided improved overland performance, the Navy needs to continue radar and mission system development efforts to provide a robust capability in all overland environments. - The E-2D mission planning system, the Joint Mission Planning System, is not currently effective. Operators use
170 E-2D
the Joint Mission Planning System to provide a means of importing mission-planning data into the E-2D mission computer for use during flight. - Based on IOT&E reliability and availability data, DOT&E has identified shortfalls on some radar reliability and weapon system availability metrics. - The E-2D aircraft performed nominally during at-sea operations, but immature logistics and maintenance support precluded an adequate assessment to demonstrate carrier‑based flight operations. The current E-2C system operates in a four-aircraft-per-squadron configuration as opposed to the E-2D, which the Navy plans to operate in a five-aircraft-per-squadron configuration. ▪▪ The E-2D at-sea testing did not fully demonstrate the ability to support the logistics of the proposed five-aircraft E-2D squadron in the aircraft carrier environment. The limited number of at-sea sorties and the current limited spare parts support for E-2D precluded a full at-sea logistics supportability assessment of the five-aircraft E-2D squadron concept. ▪▪ Full demonstration of a five-aircraft E-2D squadron in the aircraft carrier environment is not expected until 3Q‑4QFY14. Recommendations • Status of Previous Recommendations. The Navy continues efforts to improve radar and mission system performance overland, improve radar and overall weapon system reliability and availability, and resolve the mission system track re‑labeling deficiency. • FY13 Recommendations. The Navy should: 1. Correct E-2D to CEC integration shortfalls identified during the E-2D CEC FOT&E. 2. Demonstrate full shipboard suitability and logistical supportability.
N av y P R O G R A M S
Enhanced Combat Helmet (ECH) Executive Summary • The Enhanced Combat Helmet (ECH) underwent a third First Article Test (FAT III) and a second Full-Up System-Level (FUSL) live fire test because the manufacturer changed the ballistic shell laminate material from that which was previously tested. • The ECH successfully met its ballistic and non-ballistic requirements during FAT III. However, while the ECH protects against perforation by the specified small arms threat, it does not provide a significant overall improvement in operational capability over currently-fielded helmets against the specified small arms threat. The deformation induced by the impact of a non-perforating small arms threat impact exceeds accepted deformation standards across most of the threat’s effective range. The ECH is therefore unlikely to provide meaningful protection over a significant portion of the threat’s effective range. The ECH provides improved fragmentation protection compared to the fielded Advanced Combat Helmet and the Light Weight Helmet (LWH). • The manufacturer has started ECH production, with first deliveries anticipated in early FY14. System • The Marine Corps developed the ECH in response to a 2009 Urgent Statement of Need to produce a helmet that provides ballistic protection from energetic fragments and selected small arms ammunition, yet maintains all other characteristics of the Marine Corps’ LWH and the Army’s Advanced Combat Helmet (ACH). • The ECH is compatible with and is typically worn in conjunction with other components of infantry combat equipment such as body armor systems, protective goggles, night vision equipment, and a camouflage fabric helmet cover. This new helmet is intended to provide Marines and Soldiers improved protection compared to the currently fielded LWH and ACH. • The ECH consists of a ballistic protective shell, a pad suspension system, and a 4-point chin strap/nape strap retention system. Unlike the ACH and LWH helmets, which are constructed with aramid fibers, the ECH is constructed
Activity • The Marine Corps approved full-rate production in 2012 following successful completion of FAT II. • During testing of Engineering Change Proposals intended to increase manufacturing capacity, the ECH failed small arms testing. Subsequent attempts to implement and verify corrective action failed to produce a helmet that could pass the small arms portion of the FAT.
using ultra-high-molecular-weight polyethylene fibers. Unlike aramid composites, the ultra-high-molecular-weight polyethylene ballistic material absorbs ballistic impact and dissipates energy via extensive plastic strains. This results in more resistance to penetration but it also results in large permanent helmet shell deformations and larger damaged areas following impact for a wide range of ballistic threats. Mission Forces equipped with the ECH will rely on the helmet to provide ballistic protection from selected threats when engaged with enemy combatants during tactical operations in accordance with applicable tactics, techniques, and procedures. Major Contractor Ceradyne, Inc. – Costa Mesa, California
• In February 2013, the manufacturer changed the ballistic shell laminate to improve small arms protection. This change required the helmet to undergo another FAT (FAT III) and a follow-on FUSL live fire test. • The Program Office conducted and successfully completed FAT III in March 2013 and the FUSL live fire test from April
ECH
171
N av y P R O G R A M S through May 2013. Testing was conducted in accordance with the DOT&E-approved test plan. • The manufacturer has started producing ECHs to support both Marine Corps and Army requirements, with the first deliveries anticipated in early FY14. Assessment • Although the ECH protects against perforation by the specified small arms threat, it does not provide a significant overall improvement in operational capability over currently fielded helmets against the specified small arms threat. It is unlikely to provide meaningful protection against this small arms threat over a significant portion of the threat’s effective range. However, the ECH does provide improved penetration protection against fragments relative to currently fielded helmets. The ECH met all ballistic performance requirements. • In stopping high-energy threats, the helmet absorbs the projectile energy by deforming inward toward the skull. It is unknown, definitively, whether the ECH provides protection against injury when the deforming helmet impacts the head. There is, however, reason to be concerned because the deformation induced by the impact of a non-perforating small arms threat exceeds accepted deformation standards (established for a 9 mm round) across most of the threat’s effective range. • There are no definitive medical criteria or analytic methods to correlate the extent of helmet deformation to injury. However, the potential for helmet deformation to cause significant blunt force and/or penetrating trauma to the head is a concern.
172 ECH
• Structural degradation as a result of prolonged temperature and humidity exposure may be a concern for the ECH. Published data document the degradation of ballistic performance in ultra-high-molecular-weight polyethylene materials, but the long-term performance of the ECH’s specific ballistic material is unknown. The ECH Program Office plans to study the durability of the helmet’s ballistic material. Recommendations • Status of Previous Recommendations. As the Program Office is not procuring the helmet described in the FY12 report, those recommendations are no longer valid. • FY13 Recommendations. The ECH Program Office should: 1. Conduct durability testing to determine whether moderate blunt impacts degrade ECH ballistic performance. 2. Conduct testing to determine whether long-term exposure to elevated temperatures and humidity degrades ECH ballistic performance. 3. Carefully monitor the results of lot acceptance testing when ECH production begins for indications of variations in the manufacturing process that could affect the ECH’s ballistic protection. 4. Improve ECH protection by reducing the amount of helmet deformation caused by non-perforating small arms impacts, as improvements in materials and manufacturing processes permit. 5. Continue to support development of test methodologies and techniques that would reduce limitations associated with the current, single-sized clay-filled headform used for testing.
N av y P R O G R A M S
F/A-18E/F Super Hornet and EA-18G Growler Executive Summary • While System Configuration Sets (SCSs) H8E and 23X demonstrate acceptable suitability, the Active Electronically Scanned Array (AESA) radar’s reliability continues to suffer from software instability. The radar’s failure to meet reliability requirements remains a shortfall from previous test and evaluation periods. • Although the F/A-18E/F Super Hornet weapon system continues to be operationally effective and suitable for many threat environments, it has critical shortfalls. The details are addressed in DOT&E’s classified report issued following the SCS H6E, SCS 23X, and AESA FOT&E. • The EA-18G Growler weapon system is operationally effective and operationally suitable with the same radar limitations as the E/F. • The Navy is conducting the F/A-18E/F and EA-18G SCS H8E System Qualification Test (SQT) in two phases. Phase I was completed in 4QFY13. The Navy expects to conduct Phase II testing from 4QFY13 through 2QFY14. DOT&E will issue a single report covering both H8E phases after the completion of Phase II. System F/A-18E/F Super Hornet • The Super Hornet is the Navy’s premier strike-fighter aircraft that replaces earlier F/A-18 variants in carrier air wings. The F/A-18E is a single-seat aircraft while the F model has two seats. • F/A-18E/F Lot 26+ aircraft provide functionality essential for integrating all Super Hornet Block 2 hardware upgrades, which include: -- Single pass multiple targeting for GPS-guided weapons -- Use of off-board target designation -- Improved datalink target coordination precision -- Implementation of air-to-ground target points • Additional systems include: -- APG-73 or APG-79 radar -- Advanced Targeting and Designation Forward-Looking Infrared System -- AIM-9 infrared-guided missiles and AIM-120 and AIM-7 radar-guided missiles -- Shared Reconnaissance Pod -- Multifunctional Information Distribution System for Link 16 tactical datalink connectivity -- Joint Helmet-Mounted Cueing System -- Integrated Defensive Electronic Countermeasures EA-18G Growler • The Growler is the Navy’s land- and carrier-based, radar and communication jamming aircraft. • The two-seat EA-18G replaces the four-seat EA-6B. The new ALQ-218 receiver, improved connectivity, and linked
displays are the primary design features implemented to reduce the operator workload in support of the EA-18G’s two-person crew. • The Airborne Electronic Attack system includes: -- Modified EA-6B Improved Capability III ALQ-218 receiver system -- Advanced crew station -- Legacy ALQ-99 jamming pods -- Communication Countermeasures Set System -- Expanded digital Link 16 communications network -- Electronic Attack Unit -- Interference Cancellation System that supports communications while jamming -- Satellite receive capability via the Multi-mission Advanced Tactical Terminal • Additional systems include: -- APG-79 AESA radar -- Joint Helmet-Mounted Cueing System -- High-speed Anti-Radiation Missile -- AIM-120 radar-guided missiles System Configuration Set (SCS) Software • Growler and Super Hornet aircraft employ SCS operational software to enable major combat capabilities. All EA-18Gs and Block 2 F/A-18s (production Lot 26 and beyond) use high-order language or “H-series” software, while F/A-18E/F prior to Lot 26 and all legacy F/A-18 A/B/C/D aircraft use “X-series” software. -- The current fleet-release software versions are H8E Phase I (H-series) and 23X (X-series). -- Software versions currently under test are H8E Phase II (H-series) and 25X (X-series).
F/A-18E/F and EA-18G 173
N AV Y P R O G R A M S Mission • Combatant Commanders use the F/A-18E/F to: - Conduct offensive and defensive air combat missions - Attack ground targets with most of the U.S. inventory of precision and non-precision weapon stores - Provide in-flight refueling for other tactical aircraft - Provide the fleet with an organic tactical reconnaissance capability • Combatant Commanders use the EA-18G to: - Support friendly air, ground, and sea operations by countering enemy radar and communications - Jam integrated air defense systems - Support non-integrated air defense missions and emerging non-lethal target sets
Activity • The Navy is conducting F/A-18E/F and EA-18G SCS H8E SQT in two phases. Phase I was completed in 4QFY13. The Navy expects to conduct Phase II testing from 4QFY13 through 2QFY14. DOT&E will issue a single report covering both H8E phases after the completion of Phase II. - The Navy conducted Phase I of F/A-18E/F and EA-18G SCS H8E SQT from July 2012 tthrough May 2013 in accordance with a DOT&E-approved Test and Evaluation Master Plan and operational test plan. Developmental delays caused an approximately six-month slip in the schedule. H8E Phase I was released to the fleet in September 2013. - The Navy began H8E Phase II SCS operational testing in September 2013 and plans to complete testing in March 2014. • F/A-18A/C/D/E/F SCS 25X SQT was scheduled to begin in 1QFY13. Developmental delays have pushed back the start of SCS 25X SQT to 1QFY14. • The Navy has continued to defer development of the AESA’s electronic warfare capability to later software builds. • The Navy deferred several enhancements that it intended to deliver with SCS H8E to later software builds. These enhancements included integrated electronic support, integrated high-gain electronic support measures, specific emitter identification, single-ship geolocation, integration of the ALQ-214(V)4 jammer, and RNAV (Area Navigation). Assessment • The Navy has not yet addressed long-standing deficiencies with the APG-79 AESA radar. As stated in the FY12 Annual Report, the AESA demonstrated marginal improvements during FOT&E from prior testing and provides improved performance relative to the legacy APG-73 radar. However, operational testing has yet to demonstrate a statistically significant difference in mission accomplishment between F/A-18E/F aircraft equipped with AESA and those equipped with the legacy radar.
174
F/A-18E/F and EA-18G
- Enhance crew situational awareness and mission management - Enhance connectivity to national, theater, and tactical strike assets - Provide enhanced lethal suppression through accurate High-speed Anti-Radiation Missile targeting - Provide the EA-18G crew air-to-air self-protection with the AIM-120 Major Contractor The Boeing Company, Integrated Defense Systems – St. Louis, Missouri
• Though aircraft software has demonstrated acceptable suitability, the continued poor reliability of the AESA radar appears to be a result of software instability. The radar’s reliability and poor built-in test (BIT) performance remain deficient. • The Navy did not attempt to address long-standing deficiencies in air warfare or AESA radar reliability with SCS H8E. Overall, the F/A-18E/F/G is not operationally effective for use in certain threat environments, the details of which are addressed in DOT&E’s classified report issued following SCS H6E, SCS 23X, and AESA FOT&E. • SCS H8E testing does not include an end-to-end multi-AIM-120 missile shot. This capability has not been successfully operationally tested. The Navy has tentatively planned to conduct a multi-missile test with SCS H12 testing in FY16 or FY17. • DOT&E will report on the Super Hornet and Growler SCS H8E capability improvements after both Phase I and Phase II operational testing are complete in FY14. • Preliminary results from the H8E testing indicate the EA-18G is likely to continue to be operationally effective; however, analysis is ongoing. Preliminary H8E results also indicate that the EA-18G has met all suitability thresholds except Mean Flight Hours Between BIT False Alarm. Recommendations • Status of Previous Recommendations. The Navy has made minimal progress in addressing FY07 recommendations to continue to improve APG-79 AESA reliability and BIT functionality, to conduct an operationally representative end-to-end missile shot to demonstrate APG-79 radar and current SCS ability to support multi-AIM-120 engagement, and to develop and characterize the APG-79 AESA’s full electronic warfare capability. DOT&E made no new recommendations in FY12.
N av y P R O G R A M S • FY13 Recommendations. The Navy should: 1. Address the F/A-18E/F/G radar deficiencies. 2. Continue to improve maintainability and BIT software maturity by reporting key suitability parameters during future FOT&E, such as Mean Flight Hours Between Operational Mission Failure and Mean Corrective Maintenance Time for Operational Mission Failure.
F/A-18E/F and EA-18G 175
N av y P R O G R A M S
176
N av y P R O G R A M S
Ground/Air Task Oriented Radar (G/ATOR) Executive Summary • The Ground/Air Task Oriented Radar (G/ATOR) is a short- to medium-range, air-cooled phased array radar that will provide an Air Defense/Air Surveillance (AD/SR) radar capability to the Marine Air Ground Task Force (MAGTF) commander. A total of 57 G/ATOR systems are planned for procurement. • The Program Executive Office, Land Systems Marine Corps is executing the G/ATOR program as an evolutionary acquisition consisting of four capabilities, now referred to as G/ATOR blocks. - Block 1 will complete the primary material system acquisition and can support the short-range air defense mission, as well as provide an AD/SR radar capability to the MAGTF Commander. - Block 2 will include software to perform the missions of ground counter-battery/fire control (Ground Locating Weapons Radar). - Block 3 capabilities have been deferred indefinitely. - Block 4 will provide air traffic control capabilities (Expeditionary Airport Surveillance Radar). • In March 2013, the Program Executive Office, Land Systems Marine Corps delayed an operational assessment scheduled for the spring Weapons and Tactics Instructor (WTI) course (WTI 2-13) to the fall WTI course (WTI 1-14) and rescheduled the Milestone C decision from 4QFY13 through 2QFY14. The change in WTI schedule was due to concerns about software stability affecting overall system reliability. In place of the operational assessment, the Marine Corps Operational Test and Evaluation Activity (MCOTEA) and the Program Office jointly conducted a Field Users Evaluation (FUE) at Marine Corps Air Station Yuma, Arizona, during WTI 2-13 to assess system performance and collect additional reliability data in an operational environment. • The FUE provided an opportunity to assess G/ATOR’s progress toward Critical Operational Issue (COI) resolution. DOT&E assessed that the test methodology and data collected for the FUE were mostly sufficient to support an assessment of the objective COIs and Measures of Performance (MOPs); however, data were lacking for resolution of radar false track MOPs. • In September 2013, DOT&E concurred with a MCOTEA assessment that data collected from the FUE could be used to support an operational assessment of G/ATOR. However, DOT&E and MCOTEA concluded additional data would also be required from the developmental test (DT)-1B4 test period and WTI 1-14 course in order to adequately develop an operational assessment in support of Milestone C. • Reliability performance remains low with a reported 42.8 hours Mean Time Between Operational Mission Failure (MTBOMF) during the FUE. Although reliability performance is improving, a robust reliability growth plan
needs to be put in place to address both hardware and software failures. The system’s current reliability key system attribute threshold requirement of 500 hours MTBOMF cannot be realistically achieved within the context of the current G/ATOR test schedule through IOT&E. • Approval of the Milestone C Test and Evaluation Master Plan is not likely to occur in time to support the program’s current January 2014 Milestone C decision. The program has not yet finalized an acceptable reliability growth strategy, has not completed an adequate test design for the IOT&E, and the production configuration (Gallium arsenide (GaAs) versus Gallium nitride (GaN) radar modules) for IOT&E has not yet been agreed to. The Navy desires to conduct IOT&E on a GaAs radar module configuration and currently plans to switch to a GaN configuration during low-rate initial production. Over 80 percent of the Block 1 and Block 2 procurement is planned with GaN radar modules, yet it remains unclear if adequate production representative versions of the system will be available in time for IOT&E. IOT&E must be conducted on the production configuration representing the majority of the planned procurement.
System • G/ATOR is a short- to medium-range, air-cooled phased array radar under development for the Marine Corps. It is intended to replace five current radar systems and augment the AN/ TPS-59 long-range radar. A total of 57 G/ATOR systems are planned for procurement. • The Program Executive Office, Land Systems Marine Corps is developing G/ATOR in three blocks: - Block 1 develops the basic hardware and provides AD/SR radar capability. It replaces the AN/UPS-3, AN/MPQ-62, and AN/TPS-63 radar systems.
G/ATOR 177
N av y P R O G R A M S - Block 2 adds a ground counter battery/fire control mission capability and replaces the AN/TPQ-46 radar system. - Block 3 has been deferred and Mode 5/S will be incorporated into Block 4. - Block 4 provides an air traffic control capability and replaces the AN/TPS-73 radar system. • The G/ATOR baseline system configuration is comprised of three subsystems: - Radar Equipment Group (REG). The REG consists of the phased-array radar mounted on an integrated trailer. The trailer is towed by the Medium Tactical Vehicle Replacement. - Power Equipment Group (PEG). The PEG includes a 60-kilowatt generator and associated power cables mounted on a pallet. The generator pallet is carried by the Medium Tactical Vehicle Replacement. - Communications Equipment Group (CEG). The CEG provides the ability to communicate with and control the radar and is mounted inside the cargo compartment of the High Mobility Multi-purpose Wheeled Vehicle.
• The G/ATOR program completed Milestone B and entered the Engineering and Manufacturing Development phase in August 2005 as an Acquisition Category II program. However, in October 2011, G/ATOR was re-designated an Acquisition Category IC program due to increases in Research, Development, Test, and Evaluation funding required to meet mandatory Force Protection requirements.
Activity • The Marine Corps conducted three developmental test periods of G/ATOR Block 1 from July 2012 until February 2013 with a corrective action period following each developmental test. • In March 2013, Program Executive Office, Land Systems Marine Corps delayed an operational assessment scheduled for the spring WTI course (WTI 2-13) to the fall WTI course (WTI 1-14) and rescheduled the Milestone C decision from 4QFY13 through 2QFY14. The change in schedule was due to concerns about G/ATOR reliability metrics and system performance. In place of the operational assessment, MCOTEA and the Program Office conducted an FUE at Marine Corps Air Station Yuma, Arizona, during WTI 2-13 to assess system performance and collect additional reliability data in an operational environment. • The Program Office initiated a re-evaluation of its reliability growth program because of G/ATOR reliability concerns. • The Program Office added a fourth developmental test period (DT-1B4) from July through September 2013 to evaluate software updates primarily for reliability improvements. • In September 2013, DOT&E concurred with a MCOTEA assessment that data collected during the FUE would be sufficient to support a G/ATOR operational assessment if supplemented by additional data collected during DT-1B4 and WTI 1-14. • The Program Office and MCOTEA will use data collected during DT-1B4 and WTI 1-14 to support an operational milestone assessment report of G/ATOR. Milestone C and low-rate initial production contract award for G/ATOR Block 1 is currently scheduled for 2QFY14.
Assessment • The FUE provided an opportunity to assess G/ATOR’s progress toward COI resolution. DOT&E assessed the FUE test methodology and data collected as nearly sufficient to support an assessment of the objective COIs and MOPs; however, data were lacking for resolution of radar false track MOPs and data supporting track ambiguity metrics were not sufficiently analyzed. • MCOTEA and DOT&E concluded data collected during the FUE were not sufficient to support an operational assessment in support of a Milestone C decision and additional data collection during the DT-1B4 event and WTI 1-14 was required. DOT&E and MCOTEA agreed that additional testing and data collection were necessary to assess G/ATOR reliability performance metrics, false track rates, radar track ambiguity metrics, training, and user workload to better identify G/ATOR’s technical maturity in preparation for the Milestone C decision. • Reliability performance remains low relative to the requirement with 42.8 hours MTBOMF reported during the FUE. Although reliability performance is improving, a defensible reliability growth plan needs to be established to address both hardware and software failures. The system’s current reliability threshold requirement of 500 hours MTBOMF cannot be realistically achieved within the context of the current G/ATOR test schedule through IOT&E. Moreover, the operational rationale for the 500-hour requirement is unclear. • An update to the Milestone C Test and Evaluation Master Plan will be required. The program has not yet finalized an
178 G/ATOR
Mission The MAGTF commander will employ G/ATOR within the Marine Air Command and Control System to provide enhanced situational awareness and additional capabilities to conduct short- to medium-range radar surveillance and air defense, ground counter-battery/fire control, and air traffic control missions using a single system. Major Contractor Northrop Grumman Electronic Systems – Linthicum, Maryland
N av y P R O G R A M S acceptable reliability growth strategy, and has not completed an adequate test design for the IOT&E. Progress is being made in resolving these issues; however, it is unlikely they will be fully resolved prior to the currently planned January 2014 Milestone C review. The Navy now plans to produce about 20 percent of these radars using GaAs transmit/receive modules, with 80 percent using GaN modules. IOT&E will be conducted on the GaN production configuration representing the majority of the planned procurement.
• FY13 Recommendation. 1. The Program Office should re-evaluate the G/ATOR reliability program and ensure that reliability growth plans and curves are realistic and that reliability requirements are based on a clear operational rationale. In addition, appropriate adjustments should be made to meet and demonstrate operational reliability, availability, and maintainability requirements.
Recommendations • Status of Previous Recommendations. This is the first annual report for this program.
G/ATOR 179
N av y P R O G R A M S
180
N av y P R O G R A M S
H-1 Upgrades – U.S. Marine Corps Upgrade to AH-1Z Attack Helicopter and UH-1Y Utility Helicopter Executive Summary • The Navy conducted an FOT&E from October 2012 to January 2013 to evaluate the aircraft System Configuration Set 6.0 (SCS 6.0) software, which was designed to enhance capabilities and correct previously identified problems. • The H-1 Upgrades aircraft with SCS 6.0 remain operationally effective and survivable. The test unit successfully completed 19 of 23 missions. Operational test aircraft met reliability and maintainability requirements but did not meet the availability requirement of an 85-percent mission-capable rate. • Effectiveness, suitability, and survivability of H-1 Upgrades aircraft with SCS 6.0 are degraded by occasional software blanking of the electronic warfare display. • An FOT&E of SCS 7.0, scheduled for 4QFY14, will evaluate corrections to the blanking of the electronic warfare display, other corrections to SCS 6.0, and any new features. The VX-9 test squadron tested this correction in September 2013 at China Lake and the Verification of Correction of Deficiencies report is in staffing. System • This program upgrades two Marine Corps H-1 aircraft: - The AH-1W attack helicopter becomes the AH-1Z - The UH-1N utility helicopter becomes the UH-1Y • The aircraft have identical twin engines, drive trains, four‑bladed rotors, tail sections, digital cockpits, and helmet‑mounted sight displays. By parts count, the aircraft are 84-percent common. • The UH-1Y has twice the payload and range of legacy UH‑1N aircraft and can deliver eight combat-ready Marines 118 nautical miles and return without refueling. The AH-1Z has a high-fidelity targeting sensor for delivery of air‑to‑ground and air-to-air missiles, rockets, and guns.
Activity • The Navy conducted operational testing (OT-IIIB) of the AH‑1Z and UH-1Y aircraft from October 2012 from January 2013 at Yuma Proving Ground, Arizona; Marine Corps Base Twenty‑nine Palms, Camp Pendleton; and Naval Air Weapons Station China Lake, California. - Commander, Operational Test and Evaluation Force executed OT-IIIB in accordance with a test plan that DOT&E approved on October 12, 2011. - An exception was made to cancel the planned shipboard phase of testing as no ship was available.
• As of July 2013, Bell Helicopter has delivered 79 of the planned 160 UH-1Y aircraft and 32 of the planned 189 AH-1Z aircraft. Mission • Marine light/attack helicopter squadron detachments are typically deployed with a mix of UH-1Y and AH-1Z helicopters. • Detachments equipped with the AH-1Z attack helicopter conduct rotary-wing close air support, anti-armor, armed escort, armed and visual reconnaissance, and fire support coordination missions. • Detachments equipped with the UH-1Y utility helicopter conduct command, control, assault support, escort, air reconnaissance, and aeromedical evacuation missions. Major Contractor Bell Helicopter – Amarillo, Texas
• Four aircraft, two AH-1Zs and two UH-1Ys, completed a total of 163.8 flight hours in pre-test training and 62.3 flight hours during FOT&E. Crews completed 19 operational missions during FOT&E in an operationally realistic desert environment including real-world scenarios against simulated threats. • The primary focus of OT-IIIB was to evaluate the newly‑installed SCS 6.0 software, which added or enhanced several capabilities and corrected some previously identified deficiencies.
H-1 Upgrades 181
N av y P R O G R A M S Assessment - This means that if any failure (actual or false) is detected • The H-1 Upgrades aircraft with SCS 6.0 remain operationally in the suite of aircraft survivability equipment (APR-39 effective. and AAR-47, both missile approach and ballistic warning - The OT-IIIB unit successfully completed 19 of 23 missions. functions), SCS 6.0 causes the electronic warfare display to This 83-percent mission success rate is consistent with go blank. demonstrated mission success rates in previous H-1 - SCS 6.0 detection of a single, failed electronic warfare Upgrades operational test events. component results in total loss of visual threat displays for - Two of the four mission failures were caused by blanking all threat detection systems. When this loss of situational of the electronic warfare display. SCS 6.0 blanks the awareness occurs mid-mission, pilots have the option to electronic warfare display if any failure is detected in the abort the mission or continue the mission by relying on aircraft survivability equipment before or during mission countermeasures afforded by still-functioning aircraft execution. survivability components and cues from the wingman to - Aside from the electronic warfare software blanking, detect and counter threat activity. The VX-9 test squadron SCS 6.0 software enhances pilot situational awareness tested this correction in September 2013 at China Lake and with pilot-to-pilot cueing, improved cockpit lighting, more the Verification of Correction of Deficiencies report is in editable waypoints, more efficient zoom control of the staffing. AH-1Z sensor, and increased awareness of hostile fire. - This software blanking of the electronic warfare display • OT-IIIB aircraft met reliability and maintainability caused two of the four mission failures. requirements, but did not meet availability requirements. As observed in the non-deployed AH-1Z/UH-1Y fleet, the Recommendations OT-IIIB unit did not meet mission-capable rates because of • Status of Previous Recommendations. The Program Office is long downtimes while awaiting repair parts, particularly those satisfactorily addressing previous recommendations. associated with the tail and main rotor systems. Deployed • FY13 Recommendations. The Navy should consider the H-1 aircraft in combat in Afghanistan and with Marine following recommendations and verify the corrections to Expeditionary Units afloat have higher priority for repair parts, deficiencies during the next FOT&E period: shorter parts delays, and higher mission-capable rates. 1. Eliminate software blanking of the electronic warfare • H-1 Upgrades units remain survivable against small arms display. and automatic weapons fire (up to 12.7 mm) and legacy 2. Continue efforts to increase the availability of spare parts, Man‑Portable Air Defense Systems. With SCS 6.0, pilots especially of critical rotor system components. have increased awareness of hostile fire from small arms and 3. Continue to resolve H-1 survivability concerns identified rocket-propelled grenades, as long as all electronic warfare during live fire testing. Redesign the main rotor components are operating properly. transmission and combine gearbox housings to increase • Effectiveness, suitability, and survivability of H-1 Upgrades run-dry capabilities following loss of lubricant, and improve aircraft with SCS 6.0 are degraded by occasional software the self-sealing capability of fuel bladders. blanking of the electronic warfare display.
182 H-1 Upgrades
N av y P R O G R A M S
Integrated Defensive Electronic Countermeasures (IDECM) Executive Summary • The Navy completed the Integrated Defensive Electronic Countermeasures (IDECM) Block IV operational assessment (OA) in March 2013. - The OA included laboratory testing at the Navy’s Electronic Combat System Evaluation Laboratory (ECSEL), Point Mugu, California, against two classified threats and flight testing at the Electronic Combat Range (ECR), China Lake Naval Air Station, California. - At the conclusion of the OA, IDECM Block IV demonstrated progress toward being operationally effective but not operationally suitable due to poor reliability. - System instability, a high built-in test false alarm rate, and lack of software maturity were the primary causes of poor reliability. - DOT&E documented the OA in a classified report in April 2013. • IDECM Block IV developmental testing confirmed two interoperability shortfalls identified on previous IDECM system blocks, both of which reduce aircrew situational awareness: - The interaction between the ALR-67(V)2 and (V)3 radar warning receivers and IDECM Block IV system causes false threat symbols to be displayed. - The APG-79 radar is falsely identified by the ALR‑67(V)2 and (V)3 radar warning receivers. • The Navy has focused on resolving or mitigating IDECM Block IV shortfalls with the goal of accomplishing successful operational testing beginning 2QFY14. System • The IDECM system is a radio frequency, self-protection electronic countermeasure suite on F/A-18 aircraft. The system is comprised of onboard and off-board components. The onboard components receive and process radar signals and can employ onboard and/or off-board jamming components in response to identified threats. • There are four IDECM variants: Block I (IB-1), Block II (IB‑2), Block III (IB-3), and Block IV (IB-4). All four variants include an onboard radio frequency receiver and jammer. - IB-1 (fielded FY02) combined the legacy onboard receiver/jammer (ALQ-165) with the legacy (ALE-50) off-board towed decoy.
- IB-2 (fielded FY04) combined the improved onboard receiver/jammer (ALQ-214) with the legacy (ALE-50) off-board towed decoy. - IB-3 (fielded FY11) combines the improved onboard receiver/jammer (ALQ-214) with the new (ALE-55) off-board fiber optic towed decoy that is more integrated with the ALQ-214. - IB-4 (currently in development) is intended to replace the onboard receiver/jammer (ALQ-214(V)3) with a lightweight, repackaged onboard jammer (ALQ-214(V)4 and ALQ-214(V)5). • An additional program to provide IDECM Block IV the capability to deny or delay targeting of the F/A-18 by enemy radars, known as the Software Improvement Program, is in early development. • The F/A-18E/F installation includes off-board towed decoys. The F-18C/D installation includes only the onboard receiver/jammer components and not the towed decoy.
Mission • Combatant Commanders will use IDECM to improve the survivability of Navy F/A-18 strike aircraft against radio frequency-guided threats while flying air-to-air and air‑to‑ground missions. • The Navy intends to use IB-3’s and IB-4’s complex jamming capabilities to increase survivability against modern radar‑guided threats. Major Contractors • ALE-55: BAE Systems – Nashua, New Hampshire • ALQ-214: ITT Electronic Systems – Clifton, New Jersey • ALE-50: Raytheon Electronic Warfare Systems – Goleta, California
IDECM 183
N av y P R O G R A M S Activity IDECM Block III • DOT&E completed its IDECM Block III IOT&E report in June 2011, assessing the system as operationally effective and suitable for combat. The Navy authorized IDECM Block III full-rate production (FRP) in July 2011. IDECM Block IV • The Navy completed the IDECM Block IV OA in March 2013. The OA included laboratory testing at the Navy’s ECSEL, Point Mugu, California, against two classified threats and flight testing at the ECR, China Lake Naval Air Station, California. DOT&E published a classified report on the OA in April 2013. • The Navy conducted all testing in accordance with a DOT&E-approved Test and Evaluation Master Plan and test plan. • The Navy held Intermediate Progress Review (IPR) #4 in April 2013 to determine if the system should be approved for FRP decisions 10 and 11. The Navy decided the following at IPR #4: -- Approve FRP decision 10. -- Delay the decision on whether or not to exercise FRP 11 until IPR #5. -- Add IPR #6 following completion of the FOT&E and in support of FRP 12. -- Postpone the FOT&E six months to continue to mature and test IDECM Block 4 software prior to beginning testing. -- Accomplish a developmental assisted test phase using developmental test resources and personnel that will result in a Letter of Observation from the Commander, Operational Test and Evaluation Force in 1QFY14. • The Navy completed a hardware-in-the-loop (HWIL) test at an Air Force facility in February 2013. Data analysis is ongoing and should be complete by January 2014. • The Navy conducted an additional HWIL test and a dense electromagnetic threat environment test at the ECSEL in October and November 2013, respectively. DOT&E will report on the results of both tests in the IDECM Block IV FOT&E report. • Integrated developmental and operational test flights at the ECR and the Air Force’s Nevada Test and Training Range took place from July through December 2013. The results will be included in DOT&E’s IDECM Block IV FOT&E report. Assessment • At the conclusion of the OA, IDECM Block IV demonstrated progress toward being operationally effective but not operationally suitable due to poor reliability. System instability, a high built-in test false alarm rate, and lack of software maturity were the primary causes of poor reliability. DOT&E documented the results of the OA in a classified report in April 2013.
184 IDECM
• Testing at the ECSEL, which included simulated aircraft and threats and actual IDECM Block IV jammer systems, was inadequate. DOT&E recommended the Navy re-accomplish those tests; the Navy agreed and began re-testing in October 2013. • IDECM Block IV developmental testing confirmed two interoperability shortfalls identified on previous IDECM system blocks, both of which reduce aircrew situational awareness: - The interaction between the ALR-67(V)2 and (V)3 radar warning receivers and IDECM Block IV system causes false threat symbols to be displayed. - The APG-79 radar is falsely identified to the ALQ-214(V)4 by the ALR-67(V)2 and (V)3 radar warning receivers. • The Navy has made progress on resolving or mitigating IDECM Block IV shortfalls with the goal of accomplishing successful operational testing beginning 2QFY14. Recommendations • Status of Previous Recommendations. The Navy has adequately addressed several previous recommendations. However, four recommendations from FY12 remain outstanding. IDECM System 1. The Navy should restructure and reorganize the complex and poorly organized IDECM system software code. This will minimize potential software problems yet to be discovered and simplify future modifications. 2. The Navy should develop hardware and/or software changes to provide pilots with correct indications of whether a decoy was completely severed. This recommendation does not apply to the F/A-18 C/D installation since that installation does not include a towed decoy. 3. The Navy should investigate the effects of IDECM on threat missile fuses. Electronic Warfare Warfighting Improvements 4. In coordination with the Defense Intelligence Agency, the Navy should update the threat lethal radii and/or the evaluation processes that are used to determine whether simulated shots are hits or misses. • FY13 Recommendations. The Navy should: 1. Use an iterative process of fine-tuning the radar warning receivers and the IDECM Block IV system to alleviate the two interoperability shortfalls. 2. Resolve built-in test and system maturity problems before FOT&E. 3. The Navy should continue to improve data collection processes and reporting methods to support an adequate suitability assessment.
N av y P R O G R A M S
Joint High Speed Vessel (JHSV) Executive Summary • The Navy conducted the Joint High Speed Vessel (JHSV) IOT&E July 15 – November 8, 2013, on the USNS Spearhead (JHSV-1) and the USNS Choctaw County (JHSV-2). DOT&E will provide a combined IOT&E and LFT&E report once data analysis is complete. • Initial results indicate the following: -­ USNS Spearhead cannot make the required 23 knot, 4,700 nautical mile light ship self-deployment transit; current analysis shows a 682 nautical mile deficit assuming a 90 percent starting fuel load with an ending fuel load of 10,000 gallons. -­ It appears that USNS Spearhead cannot make the required 35 knot, 1,200 nautical mile, fully loaded (600 short tons) transit. DOT&E is currently conducting analysis. -­ USNS Spearhead is roughly 12.5 short tons over the predicted outfitted weight. This contributes to the range deficiency for the fully loaded transit, since 12.5 short tons translate to approximately a 4 percent fuel load or 3,565 gallons. -­ USNS Spearhead can support 354 passengers for 96 hours, exceeding the requirement of 312 passengers. System • The JHSV is a high-speed, shallow-draft surface vessel designed for intra-theater transport of personnel and medium payloads of cargo for the joint force. It bridges the gap between large capacity low-speed sealift, and small capacity high-speed airlift. • JHSV is a redesign of a commercial catamaran capable of accessing relatively austere ports. Classified as a non‑combatant, JHSV has limited self-protection capability. Design characteristics include the following: - Propelled by four water jets powered by diesel engines - Transport capacity of 600 short ton of troops, supplies, and equipment 1,200 nautical miles at an average speed of 35 knots through wave height of up to 4 feet - Support 312 embarked troops for up to 96 hours or 104 troops for 14 days - Integrated ramp capable of load/off-load of military vehicles to include combat-loaded main battle tanks (M1A2)
Activity • On November 13, 2012, DOT&E approved the JHSV IOT&E test plan. The IOT&E test plan adopted an integrated test approach where developmental and operational testing were conducted concurrently, with each having its own set of metrics and data collection.
- Flight deck with helicopter refueling capability - Unrefueled self-deploying range of 4,700 nautical miles Mission Combatant Commanders will use the JHSV to support the flexible, agile maneuver and sustainment of combat forces between advanced bases, ports, austere littoral access points, and the sea base. Specifically, Combatant Commanders may employ the JHSV in a transport/resupply role in benign, permissive environments to: • Rapidly transport medium payloads of Marine Corps or Army cargo and combat-ready troops over intra-theatre distances between shore nodes • Deliver troops, combat-loaded vehicles, and equipment ready to be employed, requiring only ports with pier or quay wall access and no other infrastructure. • Support sustainment of forces between advanced bases, ports, and austere littoral access points that would be prohibitive for larger ships to access Major Contractor Austal USA – Mobile, Alabama
• Commander, Operational Test and Evaluation Force (COTF) and Marine Corps Operational Test and Evaluation Activity conducted the IOT&E on USNS Spearhead (JHSV-1) and USNS Choctaw County (JHSV-2). Testing began on July 15, 2013, and completed on November 8, 2013.
JHSV 185
N av y P R O G R A M S • The Navy accepted delivery of the USNS Spearhead (JHSV- 1) on December 5, 2012. After the Post-Delivery Availability, the ship transited from the manufacturing facility in Mobile, Alabama, to the port of Joint Expeditionary Base Little Creek – Fort Story, Virginia. On January 18, 2013, the ship started the Post-Delivery Test and Trials (PDT&T) period. • The Navy conducted the following integrated tests during PDT&T as part of the DOT&E-approved IOT&E plan. - The ship’s crew completed the self-deployment unrefueled range testing. - COTF and Navy Information Operations Command personnel performed an Operational Information Vulnerability Evaluation to uncover Information Assurance vulnerabilities in the ship’s information systems in February and March 2013. - The ship’s crew demonstrated the capability to feed 354 people (the crew, test personnel, and embarked troops) during the 96-hour end-to-end test. - Personnel from the Naval Surface Warfare Center (NSWC), Port Hueneme, California, conducted an underway replenishment (fuel only) ship qualification on USNS Spearhead both in port at Norfolk Naval Base and underway in April 2013 without transferring fuel. - Personnel from NSWC, Port Hueneme, California, conducted an underway replenishment (fuel only) ship qualification on USNS Choctaw County while underway in October 2013, transferring 22,000 gallons of fuel. - Personnel from the U.S. Army Aberdeen Test Center assisted with a ramp test at a commercial port in Morehead City, North Carolina, during March 2013. - A Navy security team embarked with their weapons along with a Naval Surface Warfare Center Dahlgren inspector in April 2013 to perform a Structural Test Firing to certify the ship’s 0.50 caliber mounts. The security team then conducted an Anti-Terrorism/Force Protection exercise firing on a towed surface target. - Naval Aviation personnel from Patuxent River Naval Air Station oversaw Aircraft Dynamic Interface testing underway in May 2013. Fleet assets included MH-60R, MH-60S, and MH-53E helicopters. - COTF and Navy Information Operations Command personnel conducted Information Assurance Penetration testing to demonstrate the crew’s ability to protect, detect, respond, and restore from a cyber-attack on the ship’s information systems in July 2013. • The end-to-end IOT&E test period included the following tests: - Naval aviators from Norfolk Virginia Naval Air Station flew an MH-60S helicopter to perform night vertical replenishment and Night Vision Device landings on USNS Spearhead during the transit from Joint Expeditionary Base Little Creek-Fort Story to Morehead City, North Carolina. - Marines from II Marine Expeditionary Force performed a day into night loading of a reinforced rifle company
186 JHSV
rolling stock (29 vehicles ranging from High Mobility Multi‑purpose Wheeled Vehicles to Amphibious Assault Vehicles) and transportation storage units. - Marine aviators from Marine Test and Evaluation Squadron 22 (VMX 22), Jacksonville, North Carolina, flew a Marine Corps Osprey (MV-22) to perform day and night vertical replenishment tests with USNS Spearhead. - Personnel from the Combatant Craft Division of Naval Surface Warfare Center Carderock Division assisted in the testing of the Rigid Hull Inflatable Boat (RHIB) launches and recoveries at sea. - Testers shifted to USNS Choctaw County (JHSV-2) to execute in port and underway cargo handling testing. • The Program Office conducted the Total Ship Survivability Trial (TSST) in conjunction with the IOT&E on USNS Spearhead. The TSST consisted of four simulated damage scenarios. For each scenario, the crew attempted to control the damage and recover lost mission capability caused by simulated weapons effects on shipboard systems and equipment. Assessment This report provides only preliminary assessments of the JHSV based on test observations on USNS Spearhead and USNS Choctaw County. The final assessments will be provided in the IOT&E Report in 2QFY14. • JHSV is capable of fueling at sea. • The JHSV ramp can accommodate an M1A2 tank (heaviest of required load items) and a Heavy Expanded Mobility Tactical Truck (least maneuverable of required load items) both to a pier/quay wall and to a floating causeway. • With an embarked security team, which includes both personnel and weapons, JHSV can engage a moving surface threat. • JHSV manning and facilities can accommodate handling of all required helicopters, with the exception of fuel and power. • The JHSV crew demonstrated day and night vertical replenishment with MH-60, MH-53, and MV-22. • The JHSV crew demonstrated efficient loading, securing, and unloading of Marines from II Marine Expeditionary Force personnel and equipment, to include rolling stock. • The JHSV crew demonstrated they could exceed the 96-hour requirement for transporting and feeding 312 combat troops by supporting 354 people over that period. • The crew demonstrated the requirement to launch 2 11-meter RHIBs within 40 minutes in Sea State 2 (wave heights up to 2.0 feet). The requirement is up to Sea State 3 (wave heights up to 4 feet). • The ship’s crew demonstrated the underway requirement to move a 27,000-pound container from the mission bay to the flight deck and back in Sea State 3. • The JHSV’s organic container load trailer is not effective for loading 20-foot long metal storage containers. During the IOT&E, the test team took five hours to connect the container load trailer with a storage container and failed to load it aboard
N av y P R O G R A M S the ship. A 20-foot container was loaded at the ship home port where ramp access from the pier was less restricted. • JHSV cannot make the required 23 knot, 4,700 nautical mile light ship self-deployment transit. Initial calculations show a 682 nautical mile deficit assuming a 90 percent starting fuel load to a 10,000-gallon ending fuel load. USNS Spearhead is currently performing an Energy Audit to determine the ship’s best fuel economy. Results of this testing will clarify best average speed for self-deployment transit. • It appears that JHSV cannot make the required 35 knot, 1,200 nautical mile fully loaded (600 short tons) transit. DOT&E is currently conducting an analysis. • Initial analyses indicate USNS Spearhead was roughly 12.5 short tons over the predicted outfitted weight. This contributes to the fully loaded range deficiency since 12.5 short tons translate to approximately a 4 percent fuel load or 3,565 gallons.
Recommendations • Status of Previous Recommendations. The Navy has addressed all previous recommendations. • FY13 Recommendations. The Navy should: 1. Determine the best self-deployed transit speed to explore the feasibility of a 4,700 nautical mile unrefueled range requirement. 2. Determine a transit speed that allows for a 600 short ton load delivery to 1,200 nautical miles. 3. Determine outfitted JHSV weight, hull by hull, to enable mission planners to characterize fully loaded transit capability. 4. Resolve helicopter fueling and powering deficiencies. 5. Demonstrate 11-meter RHIB launch capability in Sea State 3 (wave heights up to 4 feet). 6. Evaluate design improvements identified during the TSST and implement those that will enhance the ship’s survivability.
JHSV 187
N av y P R O G R A M S
188
N av y P R O G R A M S
LHA-6 Executive Summary • The LHA-6 will likely satisfy its Key Performance Parameters for vehicular stowage space, F-35 Joint Strike Fighter capacity, vertical take-off and landing spots, cargo space, and troop accommodations. However, all personnel, vehicles, and cargo must be off-loaded via aircraft because the ship does not have a well deck. The Amphibious Ready Group (ARG) Commander will not be able to rapidly offload the ship in support of an amphibious assault due to the lack of a surface means to move heavy equipment ashore. Additionally, the Navy and Marine Corps are finalizing a new concept of operations for deploying LHA-6 as the centerpiece of an ARG. • The LHA-6 Ship Self-Defense System (SSDS) has demonstrated capability against some classes of anti-ship cruise missile (ASCM) threats. However, based on combat systems testing on other platforms, it is unlikely that LHA‑6’s SSDS Mk 2-based combat system will meet the ship’s Probability of Raid Annihilation (PRA) requirement against all classes of ASCMs. • LFT&E analysis completed so far identified potential problems in susceptibility and vulnerability that would likely result in the LHA-6 being unable to maintain or recover mission capability following a hit by certain threat weapons, the details of which are classified. The Navy’s required updated analysis is behind schedule jeopardizing planning for follow-on ship survivability improvements and final LHA-6 LFT&E. System LHA-6 is a large-deck amphibious ship designed to support a notional mix of fixed- and rotary-wing aircraft consisting of 12 MV-22s, 6 F-35B Joint Strike Fighters (Short Take‑Off/ Vertical Landing variant), 4 CH-53Es, 7 AH-1s/UH-1s, and 2 embarked H-60 Search and Rescue (SAR) aircraft, or a load‑out of 20 F-35Bs and 2 embarked H-60 SAR aircraft. Key ship features and systems include: • Greater aviation storage capacity and an increase in the size of the hangar bay, which is necessary to accommodate the increased maintenance requirements of the F-35B and the MV- 22. Additionally, two maintenance areas with high- overhead clearance are incorporated into the design of the ship to accommodate wings-open MV-22 maintenance. • Shipboard medical spaces reduced by approximately two‑thirds compared to contemporary LHDs to expand the hangar bay. • An SSDS Mk 2-based combat system with the following seven major components. - The SSDS Mk 2 Mod 4 control and decision system supports the integration and control of most other combat system elements
- The ship’s AN/SPS-48E and AN/SPS-49A air search radars and the AN/SPQ-9B horizon search radar - USG-2 Cooperative Engagement Capability radar tracking systems - The Rolling Airframe Missile and the Evolved SeaSparrow Missile (ESSM), with the NATO SeaSparrow Mk 9 Track Illuminators - The AN/SLQ-32B(V)2 electronic warfare system with the Nulka electronic decoy equipped Mk 53 Decoy Launching System - The Phalanx Close-in Weapon System for air and small boat defense - The Mk 38 Mod 2 Gun Weapon System for small boat defense • Propulsion is provided by two marine gas turbine engines, two electric auxiliary propulsion motors, and two controllable pitch propellers. Six diesel generators provide electric power. • Command, Control, Communications, Computers, and Intelligence (C4I) facilities and equipment to support Marine Corps Landing Force operations are part of the program of record. • It does not have a well deck, which is traditionally used to move large volumes of heavy equipment needed for amphibious operations. Mission The Joint Maritime Component Commander will employ LHA-6 to: • Be the primary aviation platform within an ARG with space and accommodations for Marine Corps vehicles, cargo, ammunition, and more than 1,600 troops • Serve as an afloat headquarters for a Marine Expeditionary Unit (MEU), Amphibious Squadron, or other Joint Force commands using its C4I facilities and equipment
LHA-6 189
N av y P R O G R A M S • Accommodate elements of a Marine Expeditionary Brigade when part of a larger amphibious task force • Carry and discharge combat service support elements and cargo to sustain the landing force
Activity • The Navy conducted an operational assessment in December 2012 in accordance with the DOT&E-approved test plan to assess the ship’s design. • The Navy and Marine Corps conducted a wargame in May 2013 to support the development of the concept of operations for an LHA-6 configured ARG. • The Navy conducted Phase 1 of the Enterprise Test-05 on the SSDS using the LHA-6 combat system configuration in May and June 2013. In two firing exercises, a single subsonic aerial target and a supersonic high-diving aerial target were engaged with Rolling Airframe Missiles (Block 2). The SSDS program is discussed in a separate section of this report. Additional test events against the Multi-Stage Supersonic Target have been delayed until FY17 because of problems with the target’s development. • The Navy has conducted a variety of LFT&E testing and analyses using surrogate ship platforms (including the ex- Saipan (LHA-2) and scale models to develop an understanding of vulnerabilities of LHA-6 design against typical weapons effects. The Navy survivability assessment report that was due in FY12 will not be completed until FY14. This delay has an adverse effect on the planning for future LFT&E test events and limits the opportunity to improve the survivability on follow-on ships. • DOT&E approved the Test and Evaluation Master Plan Revision A in July 2012. Assessment • LHA-6 will likely meet its Key Performance Parameters for vehicular stowage space, Joint Strike Fighter capacity, vertical take-off and landing spots, cargo space, and troop accommodations. However, as the ship does not have a well deck, its capability to offload vehicles and cargo will be limited to those that can be air lifted from the ship, which will limit the capability of the ARG to support the MEU. • LHA-6 will have an enhanced aviation capability compared to the LHD-1 class ships. In particular, unlike the LHD-1 class ships, LHA-6 should be able to support operations of the entire Marine aviation combat element. • The reduction in the number of operating rooms and the size of the intensive care unit relative to the LHD-1 class limits the ability of an LHA-6 to support a medical augmentation team. The Navy may increase the number of medical personnel assigned to LPD-17 class ships when operating in LHA-6-led ARGs to compensate for the reduced medical capability on LHA-6.
190 LHA-6
Major Contractor Huntington Ingalls Industries, Ingalls Shipbuilding Division – Pascagoula, Mississippi
• The ARG and MEU Commander are able to execute all their missions utilizing an LHA-6-led three ship ARG and MEU. However, relative to an LHD-1-led ARG, an ARG with LHA-6 will require more time to complete some missions due to the reduced number of surface connectors. • The LHA-6 SSDS has demonstrated capability against some classes of ASCM threats. Based on combat systems testing on other platforms, it is unlikely that LHA-6’s SSDS Mk 2-based combat system will meet the ship’s PRA requirement against all classes of ASCMs. • LFT&E analysis completed to date identified potential problems in susceptibility and vulnerability that would likely result in the LHA-6 being unable to maintain or recover mission capability following a hit by some threat weapons. In particular, some fluid systems need additional isolation valves, sensors, and remote operators to allow rapid identification and isolation of damage and reconfiguration for restoration of the mission capability they support. Additionally, the egress from some of the troop and crew berthing spaces may result in crew causalities and delay damage control actions. The Navy has plans to incorporate some corrective actions for follow-on ships. Recommendations • Status of Previous Recommendations. The Navy satisfactorily addressed some of the FY08 and FY11 recommendations. However, the Navy needs to finalize the concept of employment for LHA-6, which is still in progress. The Navy is developing a means to provide real-time feedback on weapon system effectiveness against small boat attacks during testing. Additionally, the Navy has partially addressed a recommendation to install a capability to isolate damage and restore vital fluid systems to improve survivability, but testing to verify that improvement still needs to be planned. The Navy conducted a study to determine the benefit of hangar bay divisional doors for LHA‑7 to improve the ability to contain a fire and limit the spread of smoke and damage; however, it still needs to evaluate the mission impact for loss of the hanger bay. The Navy has not taken sufficient action on the five recommendations listed below. 1. Continue to study what effects F-35Bs and MV‑22s – particularly aircraft exhaust/noise and required logistic support – will have on the ship and make appropriate adjustments to the design.
N av y P R O G R A M S 2. Correct systems engineering deficiencies related to SSDS Mk 2-based combat systems and other combat system deficiencies so that LHA-6 can satisfy its PRA requirement. 3. Consider the use of solid state automatic bus transfer switches to improve the survivability of electrical power to vital C4I and self-defense systems to improve survivability. 4. Study flight deck manning needs to support surge operations. Mitigation plans should be demonstrated during IOT&E.
5. The survivability improvement recommendations resulting from the analysis of the LHA-6 design should be evaluated for incorporation into the LHA-7 design. • FY13 Recommendations. The Navy should: 1. Implement improvements to the SSDS Mk 2-based combat system and test those changes during FOT&E. 2. Make the Multi-Stage Supersonic Target available to support an assessment of LHA-6 PRA requirement.
LHA-6 191
N av y P R O G R A M S
192
N av y P R O G R A M S
Light Armored Vehicle (LAV) Upgrade Executive Summary • The Marine Corps completed ballistic and fuel cell upgrade tests in October 2013 at Aberdeen Test Center, Maryland. • The emerging data indicate that the upgraded fuel cells for the mission role variants (MRVs) meet technical specifications. System • The Family of Light Armored Vehicles (LAVs) shares a common base platform configuration (eight wheels, armored hull, suspension, power plant, drive train, and auxiliary automotive subsystem) among eight MRVs. The LAV-25 is the predominant MRV. • The Marine Corps completed a Service Life Extension Program in FY05 primarily to address obsolescence deficiencies. • The Marine Corps undertook the Survivability Upgrade I program based on an FY04 Urgent Need Statement from the operating forces. - This upgrade became the LAV A2 configuration standard, and involved developing and installing a Ballistic Protection Upgrade Package (BPUP), power pack enhancements, upgraded suspension, and other modifications. - The BPUP system consists of three kits, two of which provide additional protection against threats, while the third provides an internal and external stowage system. • In 2007, the LAV Program Office designed an underbody kit (known as a D-Kit) that can be incorporated to counteract under-vehicle blasts. The D-kit has been fielded since 2009. • The LAV A2 D-Kit is designed to work with the previously installed BPUP system and is a special purpose mission kit used in theater at the discretion of the operational commander. The BPUP provides armor protection to the sides and front
Activity • The program manager initiated a Survivability Upgrade program that includes improvements to the fuel cell (MRV and LAV-25) and the occupant seating. The program manager plans to conduct a subsequent Mobility and Obsolescence Upgrade program to improve the suspension (adjustable ride height) and address obsolescence issues (driveline, powerpack, steering, electrical) of the LAV platforms. • Phase I ballistic specification tests began in August 2013 at Aberdeen Proving Ground, Maryland. The LAV Program Office provided three self-sealing fuel cells for testing. • Phase II system-level tests utilized a previously used LAV-25 personnel carrier asset as a test stand to relocate the new fuel cell design for testing. The fuel cell location was consistent with the other MRVs.
of the vehicle, whereas the D-Kit provides additional armor protection with a V-shaped hull attachment under the vehicle. Mission Marine Corps commanders will use LAVs to provide combined arms reconnaissance, security missions, and mobile electronic support. Major Contractors • General Dynamics Land Systems – Canada • Conversion of a LAV A1 to a LAV A2 is conducted at Marine Corps Logistics Base – Albany, Georgia, and Marine Corps Logistics Base – Barstow, California
• DOT&E approved the Ballistic and Live Fire Event Design Plan for the Family of LAVs Fuel Cell Upgrade in August 2013. Assessment • Testing and analysis confirm that the LAV-25 A2 D-Kit increases crew protection against some under-vehicle mine and IED strikes. The details are available in the January 2013 classified DOT&E LFT&E report. • Emerging results from the Phase I fuel cell upgrade tests indicate the technical specifications have been met. • Emerging results from the Phase II system-level tests indicate that the relocated LAV MRV fuel cell is survivable up to a threshold-level underwheel blast.
LAV 193
N av y P R O G R A M S Recommendations • Status of Previous Recommendations. The Marine Corps acted upon the recommendation to consider relocating the fuel cell of the LAV-25 A2, by utilizing the LAV MRV fuel cell relocation program as a pre-cursor to a LAV-25 A2 fuel cell relocation program. The results of the MRV fuel cell relocation program will aid the program manager with engineering analysis for the subsequent LAV-25 A2 fuel cell relocation.
194 LAV
• FY13 Recommendation. 1. Despite the reduction to the Survivability Program in the Alternative Program Objective Memorandum 2015, the program manager should continue to analyze the LAV-25 fuel cell relocation effort along with ballistic seat upgrades.
N av y P R O G R A M S
Littoral Combat Ship (LCS) Executive Summary • DOT&E approved a revision to the Littoral Combat Ship (LCS) Test and Evaluation Master Plan (TEMP) in August 2013, and issued an Early Fielding Report in December 2013 providing an assessment of the LCS seaframes and mission packages. • The Navy has not yet conducted comprehensive operational testing of the LCS but has scheduled some initial operational test events in FY14. • The Navy completed the second phase of a Quick Reaction Assessment (QRA) of the capabilities and limitations of the Freedom variant seaframe and Increment II Surface Warfare (SUW) mission package on LCS 1 in December 2012. - Results from the QRA revealed performance, reliability, and operator training deficiencies for both the 30 mm and 57 mm guns. Developmental tests of the SUW mission package in October 2013 show improvement over past performance. - The Freedom variant demonstrated a capability to conduct maritime interdiction operations when the mission module is embarked. • The core combat capabilities of the Independence variant seaframe remain largely untested. Developmental testing focused on evaluating the performance of the seaframe and the Mine Countermeasures (MCM) mission package. • Analysis of data from an operational assessment of the Airborne Laser Mine Detection System (ALMDS) conducted in FY12 showed that the system does not meet the Navy’s desired probability of detection over the required depth zone and produces many false contacts. These deficiencies will increase the time required for the LCS to complete MCM operations. LCS has yet to demonstrate whether the first increment of MCM capability will meet the Navy’s reduced expectations for mine clearance. Even if this MCM package meets all of its final increment requirements, legacy systems will be needed to perform the full range of mine clearance operations. • LCS is not expected to be survivable in high-intensity combat because its design requirements do not require the inclusion of survivability features necessary to conduct sustained combat operations in a major conflict as expected for the Navy’s other surface combatants. • Equipment reliability problems have degraded the operational availability of LCS 1 and LCS 2. The Navy reports that recent reliability improvements made to the affected seaframe components have led to improved operational availability; however, no formal developmental or operational testing has occurred to verify and quantify any improvement.
System Seaframes • The LCS is designed to operate in the shallow waters of the littorals where larger ships cannot maneuver as well. • The Navy plans to acquire a total of 52 LCSs. • The Navy is procuring two (seaframe) variants of the LCS: -- USS Freedom (LCS 1, 3, 5, and follow-on ships) is a semi-planing monohull design constructed of steel (hull) and aluminum (deckhouse) with two steerable and two fixed boost water jets driven by a combined diesel and gas turbine main propulsion system. -- USS Independence (LCS 2, 4, 6, and follow-on ships) is an aluminum trimaran design with two steerable water jets driven by diesel engines and two steerable water jets driven by gas turbine engines. • Common design specifications include: -- Sprint speed in excess of 40 knots, draft of less than 20 feet, and an unrefueled range in excess of 3,500 nautical miles at 14 knots -- Accommodations for up to 76 personnel -- A Common Mission Package Computing Environment for mission package control
LCS 195
N av y P R O G R A M S -- H angars sized to embark MH-60R/S and Vertical Take-Off Unmanned Aerial Vehicles (VTUAVs) -- 5 7 mm Bofors Mk 3 gun • The designs have different core combat systems to provide command and control, situational awareness, and self‑defense against anti-ship cruise missiles and surface craft. -- Freedom Variant: COMBATSS-21, an Aegis-based integrated combat weapons system with a TRS-3D air/surface search radar, Ship Self-Defense System Rolling Airframe Missile (RAM) system (one 21-cell launcher), TERMA Soft Kill Weapon System, and a DORNA electro-optical/infrared system for Mk 110 57 mm gun fire control. -- Independence Variant: Integrated Combat Management System (derived from Dutch TACTICOS system) with a Sea Giraffe air/surface search radar, one Mk 15 Mod 31 SeaRAM launcher mount (which integrates the search, track, and engagement scheduler of the Close-in Weapon System with an 11-round RAM launcher assembly), ALEX (Automatic Launch of Expendables) System (off-board decoy countermeasures), and Sea Star SAFIRE electro-optical/infrared systems for 57 mm gun fire control. Mission Packages • LCS is intended to accommodate a variety of individual warfare systems (mission modules) assembled and integrated into interchangeable mission packages. The Navy currently plans to field MCM, SUW, and Anti‑Submarine Warfare (ASW) mission packages. Mission modules provide the seaframes with mission capability. • Multiple individual programs of record involving sensor and weapon systems and off-board vehicles make up the individual mission modules. SUW Mission Package -- I ncrement 1 ▪▪ G un Mission Module (two Mk 46 30 mm guns) ▪▪ A viation Module (embarked MH-60R) -- I ncrement 2 ▪▪ M aritime Security Module (small boats) -- I ncrement 3 ▪▪ S urface-to-Surface Missile system intended to provide limited “interim” SUW capability in response to an urgent operational need ▪▪ A viation Module (two VTUAVs) -- I ncrement 4 ▪▪ L onger range Surface-to-Surface Missile MCM Mission Package -- I ncrement 1 ▪▪ R emote Minehunting System (RMS), consisting of the Remote Multi-Mission Vehicle (RMMV) and the AN/AQS-20A sonar system
196 LCS
▪▪ MH-60S Block 2A/B Airborne Mine Countermeasures (AMCM) System, consisting of an AMCM system operator workstation, a tether system, and the two MCM systems currently under development – ALMDS for detection and classification of near surface mines, and the Airborne Mine Neutralization System (AMNS) for identification and neutralization of in volume and bottom mines (the AN/AQS-20A sonar system and Organic Airborne Sweep and Influence System are no longer being developed for use in the AMCM System) -- Increment 2 ▪▪ Coastal Battlefield Reconnaissance and Analysis (COBRA) Block I system (and VTUAVs) for unmanned aerial tactical reconnaissance to detect and localize minelines and obstacles in the daylight in the beach zone and partially in the surf zone -- Increment 3 ▪▪ Unmanned Influence Sweep System (UISS) to activate acoustic-, magnetic-, and combined acoustic/magnetic-initiated volume and bottom mines in shallow water so they self-destruct -- Increment 4 ▪▪ COBRA Block II system (and VTUAVs), which has Block I capability with the addition of night-time minefield and obstacle detection capability and full detection capability in surf zone; and Knifefish Unmanned Undersea Vehicle, a self propelled, untethered, autonomous underwater vehicle, employing a low-frequency broadband sonar sensor to detect, classify, and identify volume and bottom mines in shallow water ASW Mission Package (only Increment 2) ▪▪ Torpedo Defense and Countermeasures Module (Lightweight Tow torpedo countermeasure) ▪▪ ASW Escort Module (Multi-Function Towed Array and Variable Depth Sonar) ▪▪ Aviation Module (embarked MH-60R and two VTUAVs) Mission • The Maritime Component Commander will employ LCS to conduct MCM, ASW, or SUW tasks depending on the mission package fitted into the seaframe. Commanders can employ LCS in a maritime presence role in any configuration because of capabilities inherent to the seaframe. With the Maritime Security Module, installed as part of the SUW mission package, the ship can conduct Visit, Board, Search, and Seizure maritime interception operations. • The Navy can employ LCS alone or in company with other ships. The Navy is still developing the concept of employment for these ships in each of the mission areas.
N av y P R O G R A M S Major Contractors • Freedom Variant (LCS 1, 3, 5, 7, and follow-on odd numbered ships) - Prime: Lockheed Martin Maritime Systems and Sensors – Washington, District of Columbia - Shipbuilder: Marinette Marine – Marinette, Wisconsin • Independence Variant (LCS 2, 4, 6, 8, and follow-on even numbered ships) - Prime for LCS 2 and LCS 4: General Dynamics Corporation Marine Systems, Bath Iron Works – Bath, Maine
Activity LCS Program • DOT&E approved a revision to the LCS TEMP in August 2013. The TEMP governs test and evaluation of both LCS seaframe and mission package programs through FY15 and has the strategy and resources identified to support completing IOT&E in FY19. • DOT&E published an Early Fielding Report providing an assessment of the LCS seaframes and mission packages in December 2013. Seaframe • Freedom Variant (LCS 1): -- Commander, Operational Test and Evaluation Force (COTF) completed the second phase of a QRA of Freedom’s capabilities and limitations in November and December 2012 along with an assessment of the ship’s cyber defense and maritime interdiction capabilities in preparation for the ship’s overseas deployment to Singapore. -- The Navy continued developmental testing of the seaframe’s 57 mm gun system in November and December 2012 and January 2013. -- USS Freedom (LCS 1) departed San Diego, California, for operations in the Western Pacific in March 2013. • Independence Variant (LCS 2): -- The Navy completed calm water performance trials in May and June 2013 to evaluate the seaframe’s speed, power, fuel consumption, and maneuvering characteristics. -- The Navy completed acoustic trials in August 2013 to evaluate the radiated and structure-borne noise signatures. -- The Navy completed a scheduled phase of developmental testing of structural improvements to the RMMV launch, handling, and recovery system, and multi-vehicle communications system (MVCS) upgrades in dockside and at-sea testing in 4QFY13. • Freedom Variant (LCS 3): -- USS Fort Worth (LCS 3) completed initial Combat System Ship Qualification Trial events in November and December 2012.
- Prime for LCS 6, LCS 8, and follow-on even numbered ships: Austal USA – Mobile, Alabama - Shipbuilder: Austal USA – Mobile, Alabama • Mission Packages - Future Mission Package Integration contract awarded to Northrop Grumman – Los Angeles, California
-- T he Navy completed fuel economy trials in September 2013 to evaluate the seaframe’s speed, power, fuel consumption, and maneuvering characteristics. -- The Navy commenced developmental testing of LCS 3 and the Increment II SUW mission package in September 2013. SUW Mission Package • The Navy continued developmental testing of the 30 mm gun mission modules on LCS 1 in December 2012 and January 2013. • The Navy established incremental performance requirements for the Increment II SUW mission package. • The Navy completed the second phase of a QRA of the capabilities and limitations Increment II SUW mission package on LCS 1 in December 2012. • The Navy conducted additional developmental testing of the SUW mission package in October 2013. MCM Mission Package • DOT&E issued a formal report on the Phase A (shore‑based and training phase) operational assessment of the MH-60S Block 2 AMCM System with ALMDS. The Navy intends to conduct Phase B (LCS-based phase) of the ALMDS operational assessment in conjunction with the MCM mission package Developmental Test Phase 4 Period 2 on the Independence variant seaframe that is scheduled to occur in 4QFY14-1QFY15. • The Navy established performance requirements for the Increment I MCM mission package. • The RMS program completed a second and final phase of reliability growth improvements of the RMMV, and completed 438 hours of in-water contractor testing in 2QFY13. • The AMNS program completed developmental testing using explosive destructors against moored explosive-filled targets (live-on-live testing) at Aberdeen Test Center, Maryland; explosive destructors against inert targets in the Gulf of Mexico; and training neutralizers against inert targets in the Gulf of Mexico and at the South Florida Test Facility in the Atlantic. COTF plans to conduct an
LCS 197
N av y P R O G R A M S operational assessment in two phases: shore-based in mid-FY14 and LCS-based in late FY14. • The Navy continued developmental testing of the RMMV launch, handling, and recovery system, and MVCS interoperability in LCS 2. LFT&E • In July 2013, the Navy began 30 mm ammunition lethality testing in accordance with the DOT&E-approved plan at Naval Surface Warfare Center – Dahlgren, Virginia. Testing will continue into FY14. • Component-level aluminum survivability testing began in June 2013 that will generate data to address the aluminum structural collapse due to fire exposure. Also, the Navy conducted a series of large-panel tearing tests of aluminum structural elements unique to the Independence variant of the LCS. Additional surrogate tests to address knowledge gaps related to the vulnerability of the aluminum ship structure to weapon induced blast and fire damage will be conducted during FY14. • The Navy updated the LCS TEMP with a plan to assess LCS vulnerability against the latest Capability Development Document requirements. The results of this assessment will be included in a Detail Design Survivability Assessment Report that is scheduled for completion in FY16. Assessment This assessment is based on information from DOT&E’s observations of post-delivery testing and trial events, fleet operations, and developmental test data and results provided by the Navy Program Offices. No formal at-sea operational tests were conducted. Program • The Navy intends to field LCS capabilities incrementally as mission package systems mature and become ready for fleet use. Additionally, the Navy directed changes to the seaframe designs based on the results of early developmental testing and operations. -- The Navy has indicated that the seaframe designs will be stabilized in the third ship of each variant (LCS 5 and LCS 6). -- Since the Navy expects each increment to deliver significant increases in mission capability, DOT&E is requiring the Navy to conduct an appropriately-designed phase of OT&E on all delivered increments on each seaframe variant. -- The initial phases of OT&E are scheduled in FY14, but the final phases will not be completed until the FY19 timeframe. Seaframes • While both seaframe variants are fast and highly maneuverable, they are lightly armed for ships of this size and possess no significant offensive capability without the planned SUW Increment IV mission package. -- They have very modest self-defense capabilities; their air defense capabilities cannot be characterized fully until
198 LCS
tests on LCS 5 and LCS 6 (the production-representative seaframes) and the Navy’s unmanned Self-Defense Test Ship feed the Navy Probability of Raid Annihilation high‑fidelity modeling and simulation analyses in FY18. -- The surface self-defense capability is scheduled to undergo limited testing in the first OT&E events on LCS 2 and LCS 3 in FY14, but the Navy has deferred testing of the ships’ capability to defeat unmanned aerial vehicles and slow-flying aircraft until production‑representative seaframes are available. -- The seaframes have no systems designed to detect torpedo attacks or mines without the appropriately configured mission packages installed. • Results from the QRA revealed performance, reliability, and operator training deficiencies for the 57 mm gun on LCS 1 that prevented the ship from demonstrating it can meet the Navy’s SUW performance requirements. -- The Navy reported that the observed deficiencies have been corrected on LCS 1; and that those corrections were satisfactorily demonstrated during developmental testing in October 2012; however, no data were collected during that testing to facilitate an independent assessment. -- The preliminary analysis of data collected during recent testing of the 57 mm gun conducted on LCS 3 in October 2013, which was observed by DOT&E, indicates that the gun reliability has improved. DOT&E expects to issue a formal test report in 4QFY14 after the Navy has completed IOT&E of the Freedom variant seaframe and Increment II SUW mission package. • Crew size can limit the mission capabilities and combat endurance of the ship. The Navy continues to review manning to determine appropriate levels. The Navy installed 20 additional berths in LCS 1 for flexibility during its deployment and has stated that additional berths will be installed in all seaframes. • Freedom Variant (LCS 1 and 3): -- Developmental testing demonstrated that this variant can position, launch, and recover the 11-meter boats included in the SUW mission package as long as the launch, recovery, and handling system is operational. Replacement of the aluminum launch ramp with one constructed of steel allows a boat to be stored on the ramp to reduce the launch time and improve responsiveness. The Navy has not tested the ship’s capability to handle, launch, and recover other watercraft. -- COMBATSS-21 and TRS-3D performance deficiencies have affected target detection and tracking capabilities in developmental testing. -- The QRA performed by COTF uncovered vulnerabilities in the ship’s capability to protect the security of information. -- Failures of diesel-powered generators, air compressors, and propulsion drive train components have degraded
N av y P R O G R A M S the seaframe’s operational availability. The Navy reports that recent reliability improvements made to the affected seaframe components have led to improved operational availability of the seaframe; however, no formal developmental or operational testing has occurred to quantify that improvement. • Independence Variant (LCS 2): -- DOT&E has no data to assess the core mission capabilities of the Independence variant seaframe. -- T he Independence crew encountered multiple problems with the twin-boom extensible crane (TBEC) and other mission package support systems during initial developmental testing of the MCM mission package. Since then, the vendor improved the TBEC and the Navy made RMMV hardware changes. Developmental testing in August 2013 demonstrated the ship’s capability to launch and recover the RMMV has improved. -- A vailability of the Independence to support testing has been degraded by equipment failures, including problems with operator consoles, power generation equipment, components of the ship’s computing and networking equipment, propulsion drive train components, and communications systems. The Navy reports that recent reliability improvements made to the affected seaframe components have led to improved operational availability of the Independence; however, no formal developmental or operational testing has occurred to quantify that improvement. SUW Mission Package • Results from the QRA revealed performance, reliability, and operator training deficiencies for both the 30 mm guns that prevented the ship from demonstrating that it can meet the Navy’s SUW performance requirements. However, as installed in the Freedom variant, the Increment II SUW mission package does enhance existing surface self-defense capability and provides additional capability to conduct maritime interdiction operations; it has not been tested in the Independence variant seaframe. MCM Mission Package • The Navy has not yet demonstrated the interim sustained area coverage rate requirement through end-to-end testing. Developmental testing has focused primarily on integrating the Increment I MCM mission package on the Independence. The MCM mission package has not been tested with the Freedom variant seaframe. • During operational assessments completed in 2011 and 2012, the AN/AQS-20A and ALMDS systems that compose the Increment I minehunting sensors demonstrated some capability in favorable benign operating environments, but failed to meet all performance requirements established by the Navy. -- AN/AQS-20A contact depth localization errors in all operating modes and false contacts in two of the three search modes exceeded Navy limits. ALMDS failed to achieve the desired detection performance over the depth
range prescribed by the Navy and the system’s false contacts exceeded Navy limits by a wide margin. -- While the Navy has identified mitigations for some of these deficiencies, they require additional search missions to weed out most of the false contacts. The additional search missions will reduce LCS’s search rate. -- Data from these operational assessments also bring into question the ability of the two minehunting systems to search the full water column; the Navy is conducting additional tests to determine if there are gaps in coverage. The Navy is also developing an improved version of the AN/AQS-20A and expects to begin developmental testing in FY14. • AMNS, intended to provide identification and neutralization of in-volume and bottom mines, will provide the only mine neutralization capability in the Increment I MCM mission package. -- Since the Navy has stopped the development of the Rapid Airborne Mine Clearance System (RAMICS), Increment I will not provide near-surface mine neutralization capability. -- The operational assessment that the Navy planned to conduct in FY13 has slipped to FY14. -- The Navy plans to develop an improved version of AMNS that will include the capability to neutralize near-surface mines; however, that development is not currently funded. The Navy expects AMNS to achieve initial operating capability (IOC) in FY16. • The RMS, which is critical to achieving the Navy’s sustained area coverage rate requirement, has also experienced developmental delays. -- The Navy expects RMS to achieve IOC in 4QFY15. Contractor tests completed in FY13 suggest that RMMV reliability has grown since the RMS program emerged from the Nunn-McCurdy review in FY10; however, these tests were not conducted in an operationally realistic manner and the measure used was not operationally relevant resulting in artificially high estimates of reliability. Data from the recent development testing suggest that reliability may not have improved sufficiently to enable an LCS with two RMMVs onboard to complete the desired area search without having to return to port more often than currently planned and desired to obtain replacements. An accurate quantitative assessment of achieved RMMV reliability cannot be evaluated until the RMS is tested in operationally realistic minehunting missions (test conditions not achieved during the contractor testing). -- The analysis of test data collected during developmental testing of structural improvements for the RMMV and the RMMV recovery system as well as MVCS upgrades is still in progress. The Navy expects to issue a formal test report in 2QFY14.
LCS 199
N av y P R O G R A M S • Even if this MCM package meets all of its final increment requirements, legacy systems will be needed to perform the full range of mine clearance operations. LFT&E • The initial aluminum fire testing focused on the strength degradation of aluminum panels and welds at elevated temperatures. Follow-on testing in FY14 will investigate structural collapse of a multi-compartment aluminum structure due to fire exposure. The tearing tests collected data needed to develop methodologies suitable for the simulation of ductile fracture on the structural scale within the framework of whole-ship finite element analyses. Data analysis continues; the Detail Design Survivability Assessment Report is scheduled to complete in FY16. • LCS is not expected to be survivable in high-intensity combat because the design requirements do not require the inclusion of survivability features necessary to conduct sustained combat operations in a major conflict as expected for the Navy’s other surface combatants. Recommendations • Status of Previous Recommendations. - The Navy partially addressed one FY09 recommendation to develop an LFT&E program with the approval of the LFT&E Management Plan; however, the details of the surrogate testing and the lethality testing still need to be developed. - The Navy partially addressed the FY10 recommendations to implement recommendations from DOT&E’s Combined Operational and Live Fire Early Fielding Report. Significant remaining recommendations include enhancing seaframe sensors and improving capability of seaframe and SUW mission package gun systems. - With respect to FY11 recommendations regarding AN/AQS-20A and ALMDS, the Navy is adjusting tactics and, for the AN/AQS-20A, funding improvements to address deficiencies. The FY11 recommendation for the Navy to continue to report vulnerabilities during live fire tests remains valid.
200 LCS
- The Navy partially addressed the FY12 recommendations to complete the revised capabilities document defining the incremental approach to fielding mission packages. ▪▪ The Navy has released requirements letters for Increments I and II SUW and Increment I MCM mission packages only; however, the requirements have not been codified in an approved Capabilities Production Document. The Navy published the LCS Platform Wholeness Concept of Operations Revision D in January 2013. ▪▪ The Navy has not published the concept of employment for all the mission packages, but advises that initial manning level studies have been completed. The Navy has adjusted ship and mission package manning levels and is continuing studies to determine the final manning levels. ▪▪ The Navy has stated that gun reliability problems identified during the QRA have been resolved based on limited testing conducted in October 2012. Preliminary analysis of additional testing conducted aboard LCS 3 in October 2013, which was observed by DOT&E, indicates that the gun reliability has improved. ▪▪ The Navy intends to conduct LCS ship-based phases of the planned operational assessments of the MH-60S Block 2/3 and ALMDS and the MH-60S Block 2/3 and AMNS MCM systems starting in late FY14. ▪▪ Throughout FY13, the Navy focused on correction of material deficiencies with mission package launch and recovery systems, and manpower and training deficiencies that prevent safe and effective shipboard launch and recovery of the RMS, and can now launch and recover the RMMV without damaging equipment in Sea States 1 and 2. Developmental testing is scheduled to continue in FY14. • FY13 Recommendation. 1. The Navy should provide a Surface-to-Surface Missile LFT&E Management Plan for DOT&E approval.
N av y P R O G R A M S
LPD-17 San Antonio Class Amphibious Transport Dock Executive Summary • The Navy is working to correct deficiencies identified during IOT&E that led DOT&E to assess the ship not operationally effective, not operationally suitable, and not survivable in a hostile environment. However, correction of a number of these deficiencies has not yet been verified by follow-on operational testing and some deficiencies have not been corrected. • The Navy conducted operational testing of the Ship Self- Defense System (SSDS) Mk 2-based combat system on CVN-68 and LHA-6, but has not yet conducted any formal operational testing to demonstrate that improvements to LPD-17’s combat system are sufficient to satisfy the ship’s self-defense requirements. • The Navy’s Board of Inspection and Survey (INSURV) assessed the material condition of LPD-23 as satisfactory. System • LPD-17 is a diesel engine-powered ship designed to embark, transport, and deploy ground troops and equipment. Ship‑to‑shore movement is provided by Landing Craft Air Cushion (LCAC), Landing Craft Utility (LCU), Amphibious Assault Vehicles (AAVs), MV-22 tilt rotor aircraft, and/or helicopters. Key ship features and systems include: - A floodable well deck for LCAC, LCU, and AAV operations - A flight deck and hangar to support Navy and Marine Corps aircraft and helicopters - Command, Control, Communications, Computers, and Intelligence facilities and equipment to support Marine Corps Landing Force operations - Self-defense against anti-ship cruise missile capability provided by the SSDS Mk 2-based combat system, which includes the Cooperative Engagement Capability radar tracking system and data distribution system, the Rolling Airframe Missile point defense system, the SLQ-32B(V)2 (with Mk 53 Decoy Launching System with Nulka electronic decoys) passive electronic warfare system, and radars (SPQ-9B horizon search radar and SPS-48E long‑range air search radar) - Two Mk 46 30 mm gun systems and smaller caliber weapons (e.g., Mk 2 50-caliber machine guns) to provide the ship’s self-defense against small surface threats
Activity • The Navy’s INSURV assessed the material condition of LPD-23, the seventh ship of the class, as satisfactory during Final Contract Trials in July 2013. • The Navy conducted preliminary modeling and simulation using an unaccredited model to examine whether upgrades and
- A Shipboard Wide Area Network that serves as the data backbone for most of the ship’s computer systems (LPD‑17 is one of the first ships built with a fully integrated data network system.) - Design features that reduce the ship’s radar cross section and are intended to make the ship less susceptible to attack Mission • A Fleet Commander will employ LPD-17 class ships as part of a notional three-ship Amphibious Ready Group or independently to conduct Amphibious Warfare. • The Commanding Officer will use these ships to: - Transport combat and support elements of a Marine Expeditionary Unit or Brigade - Embark, launch, and recover LCACs, LCUs, and AAVs for amphibious assault missions - Support aerial assaults by embarking, launching, and recovering Marine Corps aircraft - Carry and discharge cargo to sustain the landing force Major Contractor Huntington Ingalls Industries – Pascagoula, Mississippi
corrections to the ship’s combat system will improve the ship’s capability to defeat raids of anti-ship cruise missiles. • In February 2013, the Navy developed a plan of action to address deficiencies that affected survivability of the LPD-17 ship class.
LPD-17 201
N av y P R O G R A M S Assessment • In IOT&E, the Probability of Raid Annihilation and Self‑Defense Test Ship events revealed deficiencies with LPD‑17’s self-defense capability. While some potential improvements have been made, the Navy has not conducted any operational testing to permit a reassessment of that capability. • Operational testing on other SSDS Mk 2 platforms revealed similar combat system deficiencies to those found during LPD-17’s IOT&E, confirming these problems are not LPD‑17 specific. In some cases, however, the effects of these deficiencies are significant on LPD-17 because of the combat system’s design. DOT&E’s classified November 2012 Ship Self-Defense Operational Mission Capability Assessment Report provides further details. • Although improvements have been made, the Navy has not yet demonstrated the Command, Control, Communications, Computers, and Intelligence capabilities needed to support LPD-17 when performing amphibious assault operations. The Navy still needs to successfully test the Advanced Field Artillery Tactical Data System onboard LPD-17. • The Navy has improved the reliability of critical systems based on results from INSURV and a review of Casualty Reports from the Operational Commander. Further reliability improvements (described below in the recommendations section) are necessary and the Navy must validate these reliability improvements in FOT&E to confirm the ship class is operationally effective and survivable. • As the Navy has not conducted testing to demonstrate the effectiveness of deficiency corrections, DOT&E’s assessment that the LPD-17 class is not survivable in combat remains unchanged. Recommendations • Status of Previous Recommendations. The Navy has partially addressed previous recommendations by improving the reliability of Shipboard Wide Area Network, amphibious support equipment, propulsion, and Magnetic Signature
202 LPD-17
Control System. However, these material fixes have not been tested in FOT&E. The Navy should act on the remaining 12 recommendations: 1. Test fixes to critical systems including the Shipboard Wide Area Network and review the effect of ship’s manning, training, and logistics support on the reliability and maintainability of ship systems. 2. Address and test fixes to reliability problems with amphibious support equipment and propulsion equipment during FOT&E. 3. Continue to pursue mitigations to address integration problems with self-defense in multiple warfare areas. 4. Conduct FOT&E in order to demonstrate improvements to performance problems related to the Advanced Enclosed Mast Structure (verify installation of the shroud on the SPS-48E radar corrects performance problems). 5. Improve reliability of critical systems including gun systems, Magnetic Signature Control System, and effectiveness of the SSDS Mk 2-based combat system. 6. Measure Total Ship Operational Availability over an extended period after completing reliability improvements. 7. Correct remaining deficiencies from Shock Trial Reports. 8. Complete FOT&E to test Information Assurance. 9. Conduct FOT&E using the Advanced Mine Simulation System to determine susceptibility of LPD-17 to enemy mines. 10. Incorporate outstanding test events as FOT&E into the LPD-17 Test and Evaluation Master Plan. 11. Develop an FOT&E test plan for adequate, rigorous testing of the critical ship systems that must perform reliably to assure LPD-17 is operationally effective and survivable. 12. Conduct the Probability of Raid Annihilation study using an accredited model. • FY13 Recommendation. 1. The Navy should conduct testing to determine the effectiveness of the planned corrective actions in improving survivability.
N av y P R O G R A M S
Mark XIIA Identification Friend or Foe (IFF) Mode 5 Executive Summary • Independent Mark XIIA Mode 5 Identification Friend or Foe (IFF) (referred to as “Mode 5”) development efforts exist in each U.S. Military Service as well as some NATO countries. Since Mode 5 is not a joint program, the Services are separately developing IFF equipment for use on various land, sea, and air platforms. - Of these separate Service efforts, only the Navy has an established Acquisition Category II program. - The Army and Marine Corps are procuring Mode 5 transponders developed in the Navy program. - The Air Force is developing its own Mode 5 transponders and interrogator capabilities. • Although the Services are designing and building Mode 5 systems to comply with NATO and DoD IFF standards, DOT&E initiated oversight in 2006 because of the concern that the multiple programs and vendors add risk to achieving joint IFF systems interoperability. • The Navy conducted an IOT&E of Mode 5 capability that included significant joint Service participation in FY12. During the June 2013 Joint Staff J-6-led Bold Quest Coalition Capability Demonstration and Assessment event, the Navy conducted a major joint operational test event off the U.S. East Coast that focused on Mode 5 interoperability and identification in a system-of-systems context. This two‑week event included extensive participation by joint Service and allied systems equipped with a wide variety of Mode 5 equipment produced by different U.S. and allied manufacturers. Test results are currently being analyzed with the final assessment due for completion in time to support Initial Operational Capability in 2014. - This realistic operational test event has helped resolve earlier DOT&E concerns about lack of testing of Mode 5 interoperability and identification in a system-of-systems context. - Similar future events will evaluate Mode 5 interoperability and identification as other IFF systems in development are integrated into Service platforms. System • The Mark XIIA Mode 5 IFF is a cooperative identification system that uses interrogators and transponders located on host platforms to send, receive, and process friendly identification data. • Mode 5 is a military-only identification mode, which modifies the existing Mark XII Mode 4 IFF (referred to as “Mode 4”) system and addresses known shortcomings of the legacy Mode 4 identification mode. Mode 5 will eventually replace Mode 4 and allows National Security Agency-certified secure encryption of interrogations and replies. Primary system features include:
- A lethal interrogation format, which is used by a weapons‑capable platform prior to weapons release as a final attempt to get a valid Mode 5 reply from the target, even with the target’s interrogated Mode 5 transponder system in standby; this is intended to reduce the possibility of fratricide. - A random-reply-delay, which prevents overlapping replies and provides better display discrimination for closely‑spaced platforms. • Mode 5 offers more modern signal processing, compatibility with legacy Mode 4 systems and civilian air traffic control, and secure and encrypted data exchange through use of the new waveform. • Mode 5 serves as a component of the combat identification process used on ground-based systems such as the Army’s Patriot missile system, sea-based systems such as Aegis‑equipped ships, and military aircraft to include the E-3 Sentry Airborne Warning and Control System (AWACS) and E-2 Hawkeye command and control platforms. • Independent Mode 5 development efforts exist in each U.S. Military Service as well as some NATO countries. Although not a joint program, the Services are developing equipment capable of employment on multiple Service platforms. - Of the four separate Service efforts, only the Navy has the established Acquisition Category II Program of Record, with incorporation of Service-specific Mode 5 capability through platform-specific Engineering Change Proposals. - The Army and Marine Corps are leveraging the Navy program, and the Air Force will execute individual Engineering Change Proposals on its affected hardware.
Mark XIIA IFF 203
N av y P R O G R A M S Mission The Combatant Commander employs the Mode 5 system to provide positive, secure, line-of-sight identification of friendly platforms equipped with an IFF transponder. In the future, this system’s information will be combined with other cooperative and non-cooperative combat identification techniques in order to provide identification of all platforms – enemy, neutral, and friendly.
Major Contractors • Navy Transponder and Interrogator: BAE Systems – Arlington, Virginia • Air Force Transponder and Interrogator, Army Air Defense Interrogator: Raytheon Systems – Waltham, Massachusetts • Air Force E-3 Interrogator: Telephonics Corporation – Farmingdale, New York
Activity • In July 2012, the Navy Acquisition Executive approved full-rate production of the Navy Mode 5 system that includes both transponders and shipboard interrogators following the Navy Mode 5 IOT&E. • The Army and Air Force are separately developing and testing Service-specific Mode 5 capabilities: - The Army developed, tested, and is fielding a Mode 5 Air Defense Interrogator for the Patriot and Sentinel air defense systems. - The Air Force is developing a Mode 5 interrogator for AWACS. - The Air Force-developed, integrated, and tested Mode 5 interrogators and transponders into F-15C/E and F-16C aircraft. • USD(AT&L) and DOT&E worked with the Services to develop and approve a revised Joint Operational Test Approach (JOTA) document to guide Mode 5 interoperability testing across the DoD. -­ Utilizing the approved JOTA guidance, the Navy led the development of a DOT&E-approved joint test concept and test plan for the conduct of an operationally realistic JOTA evaluation of Mode 5 capability. - During the June 2013 Joint Staff J-6-led Bold Quest Coalition Capability Demonstration and Assessment event, the Navy conducted a JOTA event in 3QFY13 off the U.S. East Coast, which involved a variety of joint Service and allied aircraft equipped with interrogators and transponders produced by different U.S. and allied manufacturers. The Navy executed air warfare events under Navy Aegis destroyer, AWACS, or ground controlled intercept control. During the event, U.S. and allied aircraft flew 272 of 294 planned aircraft sorties. Representative operational flight profiles and tactics were used during the event. - This JOTA event will inform the DoD-wide FY14 Mode 5 Initial Operational Capability declaration. Future JOTA events will support the planned FY20 Full Operational Capability declaration.
Assessment • The 3QFY13 JOTA test event addressed DOT&E concerns about joint interoperability and identification in a system‑of‑systems context for the systems under test. The JOTA schedule included a mixture of blue and red forces consisting of a variety of platforms equipped with transponders and interrogators from different vendors. Preliminary JOTA results revealed no new Mode 5-associated deficiencies. • Following the Navy IOT&E, the Navy Program Office developed new software builds for both its transponder and interrogator systems to address discrepancies encountered during IOT&E. The installed performance of these software fixes, as well as Mode 5 interoperability with both existing and planned IFF systems, is being validated in combined development/integration testing. The fixes will be incorporated into Navy Mode 5 systems over the next several years. • The Navy and DOT&E are currently assessing the results from the 2013 JOTA event and will report them in a subsequent annual report.
204 Mark XIIA IFF
Recommendations • Status of Previous Recommendations. The Navy has adequately addressed all previous recommendations. • FY13 Recommendations. 1. In order to ensure interoperability between interrogators, transponders, and combined interrogator-transponders, Service program managers must ensure that developmental and operational testing of Mode 5 capabilities and systems address compatibility with both joint Service and allied IFF systems. 2. The Services must fully participate in future JOTA interoperability and identification exercises to ensure that Mode 5 capabilities continue to be tested in a realistic joint Service environment.
N av y P R O G R A M S
MH-60R Multi-Mission Helicopter Executive Summary • The overall assessment of the MH-60R airframe remains operationally effective, operationally suitable, and survivable for all mission areas. • Commander, Operational Test and Evaluation Force (COTF) completed testing focused on corrections made to resolve previously identified Multi-spectral Targeting System (MTS) deficiencies. The analysis of that test data is still in progress. DOT&E expects to issue a formal test report in 2QFY14. • COTF completed IOT&E of MH-60R with the Automatic Radar Periscope Detection and Discrimination (ARPDD) upgrade in 4QFY13. Data analysis is ongoing; DOT&E will submit a formal test report in 2QFY14. System • The MH-60R is a ship-based helicopter designed to operate from cruisers, destroyers, frigates, littoral combat ships, and aircraft carriers. • It incorporates dipping sonar and sonobuoy acoustic sensors, multi-mode radar, electronic warfare sensors, a forward-looking infrared sensor with laser designator, and an advanced mission data processing system. • It employs Mk 46 and Mk 54 torpedoes, Hellfire air‑to‑surface missiles, and crew-served mounted machine guns. • It has a three-man crew: two pilots and one sensor operator.
Activity • COTF conducted testing focused on corrections made to resolve previously identified MTS deficiencies from 4QFY12 to 2QFY13. Testing was conducted in accordance with a DOT&E-approved plan. • COTF completed IOT&E of MH-60R with the ARPDD upgrade in 4QFY2013. Testing was conducted in accordance with a DOT&E-approved test plan. • All LFT&E activities have been completed and reported in the Combined OT&E/LFT&E report to Congress in 2006. Assessment • Preliminary analysis of test data collected during testing of the upgraded software for MTS on the MH-60R indicates that most of the deficiencies identified during previous testing events have been resolved. However, testing was limited in scope and did not support an assessment of the Surface Warfare mission capability of MH-60R when equipped with
Mission The Maritime Component Commander employs the MH-60R from ships or shore stations to accomplish the following: • Surface Warfare, Under Sea Warfare, Area Surveillance, Combat Identification, and Naval Surface Fire Support missions previously provided by two different helicopters (SH-60B and SH-60F) • Support missions such as Search and Rescue at-sea and, when outfitted with necessary armament, maritime force protection duties Major Contractors • Sikorsky Aircraft Corporation – Stratford, Connecticut • Lockheed Martin Mission System and Sensors – Owego, New York
MTS and the Hellfire missile. DOT&E expects to issue a formal test report in 2QFY14. • The analysis of test data collected during IOT&E of MH-60R with the ARPDD upgrade is in progress. DOT&E expects to issue a formal test report in 2QFY14. Recommendations • Status of Previous Recommendations. The Navy has partially addressed the FY12 recommendation by conducting FOT&E to assess corrections made to resolve previously identified MTS deficiencies. • FY13 Recommendations. The Navy should: 1. Conduct comprehensive live fire lethality testing of the Hellfire missile against a complete set of threat representative small boat targets. 2. Test the Surface Warfare mission capability of MH-60R equipped with Hellfire missiles.
MH-60R 205
N av y P R O G R A M S
206
N av y P R O G R A M S
MH-60S Multi-Mission Combat Support Helicopter Executive Summary • The overall assessment of the MH-60S airframe remains operationally effective, operationally suitable, and survivable for all mission areas. • DOT&E provided a report to Congress on the Quick Reaction Assessment (QRA) of the MH-60S with the 20 mm Gun System (Forward Fixed Firing Weapon) in January 2013, which stated that the 20 mm Gun System as employed by the MH-60S provides the Operational Commander with additional Surface Warfare capability. • Commander, Operational Test and Evaluation Force (COTF) completed testing focused on corrections made to resolve previously identified Multi-spectral Targeting System (MTS) deficiencies. The analysis of that test data is still in progress. DOT&E expects to issue a formal test report in 2QFY14. • DOT&E provided an analysis of the results of the QRA of the Unguided Rocket Launcher to the Navy in September 2013. The analysis showed that the Unguided Rocket system does provide a limited enhancement to the Surface Warfare capability of the MH-60S. • COTF completed Phase A (shore-based and training phase) of the planned operational assessment of the MH-60S Block 2 Airborne Laser Mine Detection System (ALMDS) in 4QFY12. DOT&E issued an Operational Assessment report in May 2013 that identified the system did not meet Navy requirements for False Classification Density (FCD) and has low reliability. System • The MH-60S is a helicopter modified into three variants (blocks) from the Army UH-60L Blackhawk for operation in the shipboard/maritime environment. • The blocks share common cockpit avionics and flight instrumentation with the MH-60R. • Installed systems differ by block based on mission: - Block 1, Fleet Logistics – precision navigation and communications, maximum cargo or passenger capacity - Block 2A/B, Airborne Mine Countermeasures (AMCM) System – AMCM system operator workstation, a tether/towing system, and the two MCM systems currently under development; ALMDS for detection and classification of near-surface mines, and the Airborne Mine Neutralization System for neutralization of in‑volume and bottom mines. The AQS-20A sonar system and Organic Airborne and Surface Influence Sweep are no longer being developed for use in the AMCM system
- Block 3A, Armed Helicopter – 20 mm Gun system, forward-looking infrared with laser designator, crew‑served side machine guns, dual-sided Hellfire air-to-surface missiles, and defensive electronic countermeasures - Block 3B, Armed Helicopter – Block 3A with addition of tactical datalink (Link 16) Mission The Maritime Component Commander can employ variants of MH-60S from ships or shore stations to accomplish the following missions: • Block 1 – Vertical replenishment, internal cargo and personnel transport, medical evacuation, Search and Rescue, and Aircraft Carrier Plane Guard • Block 2 – Detection, classification, identification and/or neutralization of sea mines depending on which AMCM systems are employed on the aircraft • Block 3 – Combat Search and Rescue, Surface Warfare, Aircraft Carrier Plane Guard, Maritime Interdiction Operations, and Special Warfare Support Major Contractors • Sikorsky Aircraft Corporation – Stratford, Connecticut • Lockheed Martin Mission System and Sensors – Owego, New York • Raytheon Company, Integrated Defense Systems – Tewksbury, Massachusetts • Northrop Grumman – Melbourne, Florida
MH-60S 207
N av y P R O G R A M S Activity • COTF conducted the following test events in accordance with a DOT&E-approved test plan: - A QRA of the MH-60S with the 20 mm Gun System (Forward Fixed Firing Weapon) in 3QFY12 - FOT&E focused on corrections made to resolve previously identified MTS deficiencies from 4QFY12 to 2QFY13 - A QRA of MH-60S with the LAU 61C/A Unguided Rocket Launcher during 2Q-3QFY13 - Phase A (shore-based and training phase) of the planned operational assessment of the MH-60S Block 2 ALMDS in 4QFY12 • DOT&E issued an Operational Assessment report in May 2013 that identified the system did not meet Navy requirements for FCD and has low reliability. The Navy intends to conduct Phase B (Littoral Combat Ship (LCS) Ship-based phase) of the operational assessment in conjunction with the MCM mission package Developmental Test Phase 4 Period 2 on the Independence variant seaframe that is scheduled to occur in 4QFY14-1QFY15. • All LFT&E activities were completed and reported in the Combined OT/LFT&E report to Congress in 2008. Assessment • DOT&E submitted a test report to Congress for the QRA of MH-60S with the 20 mm Gun System (Forward Fixed Firing Weapon) in January 2013. The 20 mm Gun System provides enhanced Surface Warfare performance to the MH‑60S helicopter. The complete details of the evaluation are classified. • Preliminary analysis of test data collected during testing of the upgraded software for MTS on the MH-60S indicates that most of the deficiencies identified during previous test events have been resolved. Testing was limited in scope and did not support an assessment of the Surface Warfare mission capability of MH-60R when equipped with MTS and the Hellfire missile. DOT&E expects to issue a formal test report in 2QFY14. • DOT&E provided an analysis of the results of the QRA of the Unguided Rocket Launcher to the Navy in September 2013. Testing demonstrated that proper employment of the Unguided
208 MH-60S
Rocket Launcher on the MH-60S may assist the Operational Commander in executing the Surface Warfare mission. • DOT&E assessed that the MH-60S helicopter equipped with the ALMDS cannot be operationally effective until ALMDS detection and classification performance is improved. ALMDS did not demonstrate the required rapid mine reconnaissance rate, did not detect mines at depths and percentage required, and did not meet the Navy’s requirements for FCD. To mitigate the FCD problem, the Navy devised alternate tactics requiring multi-passes and reacquisition attempts that adversely affect the area coverage rate sustained (ACRS). ACRS is a key measure of LCS effectiveness in MCM operations. • DOT&E assessed that ALMDS cannot be operationally suitable until its reliability is improved. System performance is degraded by numerous nuisance faults and periodic mission critical failures. Persistent faults delayed missions and required flight crews to search areas and revisit contacts multiple times. The test team replaced the system 4 times to complete a total of 16 minehunting missions. The high-failure rate will impose additional logistics burdens to support LCS MCM operations. Recommendations • Status of Previous Recommendations. The Navy has partially addressed the FY12 recommendation by conducting FOT&E to assess corrections made to resolve previously identified MTS deficiencies. The Navy still needs to address the FY11 recommendation to investigate solutions and correct the ALMDS FCD and reliability deficiencies prior to IOT&E. • FY13 Recommendations. The Navy should: 1. Separately complete comprehensive survivability studies for the MH-60S employing the 20 mm Gun System and the Unguided Rocket Launcher. 2. Conduct comprehensive live fire lethality testing of the Hellfire missile against a complete set of threat representative small boat targets. 3. Test the Surface Warfare mission capability of MH-60S equipped with Hellfire missiles.
N av y P R O G R A M S
Mk 48 Advanced Capability (ADCAP) Torpedo Modifications Executive Summary • In FY13, the Navy completed operational testing of the Spiral 4 operational software for the Mk 48 Advanced Capability (ADCAP) Modification (Mod) 7 Common Broadband Advanced Sonar System (CBASS) torpedo and Mk 48 ADCAP Mod 6 Advanced Common Torpedo (ACOT). - DOT&E issued an FOT&E report on that testing in May 2013. - Operational testing and regression results indicate overall Mk 48 Spiral 4 performance in deep-water and shallow‑water areas has not substantially changed over legacy Mk 48 torpedo performance. - The Spiral 4 software does show some limited improvements in certain specific warfare scenarios but does not meet the Navy’s original key performance goals. Performance in Anti-Surface Warfare (ASuW) needs improvement. - The Navy authorized fielding of Spiral 4 in May 2013. • DOT&E is working with the Navy to develop a Test and Evaluation Master Plan to support assessment of Advanced Processor Build (APB) 5 software. (Note that the Navy changed the naming convention for updates to the software from “spiral” to “APB.”) • Initial developmental testing is scheduled to begin in FY15, with operational testing commencing in FY18 to support Initial Operational Capability (IOC) in FY20. System • The Mk 48 Advanced Capability torpedo is the only Anti‑Submarine Warfare and Anti-Surface Ship Warfare weapon used by U.S. submarines. • Mk 48 Mod 6, Mod 6 ACOT, and Mod 7 CBASS are currently fielded in the fleet. • The Mk 48 Mod 7 CBASS upgraded the Mk 48 ACOT with a new sonar designed to improve torpedo effectiveness through future software upgrades. Phase 1 torpedoes (IOC 2006)
Activity • In FY13, the Navy employed Spiral 4 weapons during four Submarine Command Course exercises at the Atlantic Undersea Test and Evaluation Center and the Pacific Missile Range Facility, thus completing operational testing of the Spiral 4 operational software for the Mk 48 ADCAP Mod 6 ACOT and the Mk 48 ADCAP Mod 7 CBASS torpedoes. - The majority of Mk 48 test data come from fleet training exercises, in particular the Submarine Command Courses,
delivered the initial hardware and software; Phase 2 torpedoes (IOC 2013) were required to deliver full capability. • The Navy determined the Spiral 4 software developed for CBASS Phase 2 can run on ACOT weapons as well. The Navy has authorized the fielding of Mk 48 Mod 6 ACOT and Mod 7 CBASS torpedoes with Spiral 4 software. • CBASS is a co-development program with the Royal Australian Navy. Mission The Submarine Force employs the Mk 48 ADCAP torpedo as a long-range, heavy-weight weapon against surface ships or submarines in both deep-water open ocean and shallow-water littoral environments. Major Contractor Lockheed Martin Sippican Inc. – Marion, Massachusetts
which serve as exams for prospective U.S. submarine commanders. - To conserve test resources, DOT&E agreed to utilize these torpedo events as regression testing to evaluate the performance of the Mk 48 Spiral 4 in some deep-water scenarios. - The Navy conducted testing in accordance with DOT&E‑approved test plans.
Mk 48 209
N av y P R O G R A M S • In August 2013, the Navy conducted two successful Service Weapons Tests using war-shot torpedoes. These test events confirmed the warhead performance of in-service and stored Mk 48 torpedoes. • DOT&E issued a Spiral 4 FOT&E report in May 2013. The Navy authorized fielding of Spiral 4 in May 2013. • DOT&E is working with the Navy to develop a Design of Experiments and Test and Evaluation Master Plan to support assessment of APB 5 software. Initial developmental testing is scheduled to begin in FY15, with operational testing commencing in FY18 and an IOC goal of FY20. Assessment • Operational testing and regression results indicate overall Mk 48 Spiral 4 performance in deep-water and shallow-water areas has not substantially changed over legacy performance. Spiral 4 does show some limited improvements in certain specific warfare scenarios but it still does not meet the Navy’s original key performance goals. Performance in ASuW needs improvement. • The Mk 48 Mod 6 and Mod 7 weapons continue to be operationally suitable. • Additional information on Mk 48 Spiral 4 performance can be found in DOT&E’s classified Mk 48 ACOT and CBASS Spiral 4 FOT&E report dated May 2013. Recommendations • Status of Previous Recommendations. Of the previous years’ recommendations, the following three remain unresolved: 1. While the Navy is in the process of improving the Weapons Analysis Facility simulations with the development of
210 Mk 48
the Torpedo Operational Testing Using Modeling and Simulation (TOTUMS) project, further work is required to complete the TOTUMS project and determine its usefulness in support of testing. TOTUMS is intended to implement improved false target emulation, multiple wake models, and range-dependent propagation environments where ocean composition and depth vary to allow more realistic emulation of representative threat environments. 2. As the Navy continues to conduct limited torpedo training and testing in shallow water, they should develop shallow‑water test and training areas and modernize the exercise torpedo locating and recovery systems. 3. The Navy should complete development of threat representative target and countermeasure surrogates for torpedo testing. In addition to representing the physical and signature characteristics of the threat, the surrogate should be capable of emulating appropriate operational profiles of the threat. • FY13 Recommendations. The Navy should: 1. Evaluate torpedo performance against small diesel-electric submarine threats using in-water testing against a validated surrogate. 2. Evaluate alternate acoustic technologies that can be incorporated to enhance ASuW performance.
N av y P R O G R A M S
Mk 54 Lightweight Torpedo Executive Summary • Initial analysis of completed operational testing indicates that the Mk 54 Block Upgrade (BU) software provides some limited operational capability in certain scenarios and against the Urgent Operational Needs Statement (UONS) threat, but does not meet all the original program requirements. • Reconstruction and analysis of the September 2013 testing is in progress to determine if sufficient information to assess performance has been obtained. • In preparation for the May 2013 test, Navy operational testers uncovered inconsistencies in tactical guidance, documentation, and training for the employment of the Mk 54 BU torpedo, some of which date from the introduction of the Mk 54 Mod 0 to the fleet in 2004. These problems could prevent fleet operators from effectively presetting and employing the Mk 54 BU. System • The Mk 54 Lightweight Torpedo is the primary Anti‑Submarine Warfare (ASW) weapon used by U.S. surface ships, fixed-wing aircraft, and helicopters. • The Mk 54 combines the advanced sonar transceiver of the Mk 50 torpedo with the legacy warhead and propulsion system of the older Mk 46. Mk 46 and Mk 50 torpedoes can be converted to an Mk 54 via an upgrade kit. • The Mk 54 sonar processing is an expandable, open architecture system. It combines algorithms from the Mk 50 and Mk 48 torpedo programs with the latest commercial off-the-shelf technology. • The Navy designed the Mk 54 sonar processing to operate in shallow-water environments and in the presence of sonar countermeasures. • The Navy has designated the Mk 54 torpedo to replace the Mk 46 torpedo as the payload section for the Vertical Launched Anti-Submarine Rocket for rapid employment by surface ships. • The High-Altitude Anti-Submarine Warfare Weapons Capability program will provide an adapter kit to permit long
Activity • The Navy started operational testing of the Mk 54 BU torpedo in FY12, but fielded the software in January 2012 to address a Fifth Fleet UONS threat. In FY12, the Navy conducted testing involving 34 weapons deployed from surface ships, fixed‑wing aircraft, and helicopters targeting U.S. attack submarine targets. • In FY13, the Navy conducted seven Mk 54 BU test events off Kauai, Hawaii, in conjunction with fleet training to satisfy
range, high-altitude, GPS-guided deployment of the Mk 54 by a P-8A Multi-mission Maritime Aircraft. • The Mk 54 BU is a software upgrade to the Mk 54 baseline torpedo designed to correct deficiencies identified during the 2004 Mk 54 IOT&E. • The Mk 54 must be interoperable and compatible with the analog or digital combat control systems and software variants installed on all ASW fixed-wing and helicopter aircraft, and on the surface ship combat control system variants used for torpedo tube or ASW rocket-launched torpedoes. Mission Navy surface ships and aircraft employ the Mk 54 torpedo as their primary anti-submarine weapon: • For offensive purposes, when deployed by ASW aircraft and helicopters • For defensive purposes, when deployed by surface ships • In both deep-water open ocean and shallow-water littoral environments • Against fast, deep-diving nuclear submarines; and slow moving, quiet, diesel-electric submarines Major Contractors • Raytheon Integrated Defense Systems – Tewksbury, Massachusetts • Progeny Systems Corporation – Manassas, Virginia • Boeing Company – St. Charles, Missouri • Northrop Grumman – Annapolis, Maryland
Mk 54 deep-water regression requirements. P-3C aircraft delivered four weapons and MH-60R helicopters delivered three weapons. • Following an in-progress review of completed Mk 54 operational tests, DOT&E identified a problem with how a surrogate target portrayed the threat that could have biased the results, requiring some test events to be repeated. However, due to budget constraints, resources were limited. Since
Mk 54 211
N av y P R O G R A M S
•
•
•
•
•
performance in legacy scenarios was similar to IOT&E results, DOT&E agreed to utilize the limited remaining Mk 54 BU resources to further examine the UONS scenario to assess homing-to-hit performance and to repeat the compromised test scenario. The remainder of the planned tests were deferred to the next torpedo version (Mk 54 Mod 1). The Navy conducted seven, set-to-hit Mk 54 BU firings by MH-60R helicopters against the Steel SSK surrogate target off the coast of Southern California in May 2013 and seven delivered by P-8A aircraft in the Cape Cod operating areas in September 2013. The Navy also reran the compromised test scenario but was only able to launch four set-not-to-hit Mk 54 BU torpedoes against manned submarine targets due to poor weather in the Cape Cod operating area in September 2013. The Navy plans to continue Mk 54 development with the Mk 54 Mod 1 torpedo. The Navy started development of the Mk 54 Mod 1 torpedo and plans to approve a new set of requirements documents and the Test and Evaluation Master Plan (TEMP) in FY14. DOT&E is participating in the Navy’s Torpedo Target Strategy Working Group to identify and develop test target surrogates for the Mk 54. The Navy proposed a short term strategy that utilizes three separate targets, each appropriate for specific limited scenarios and a long-term strategy that develops a mobile set-to-hit submarine surrogate that will permit realistic testing. Currently, the strategy is not funded. The Navy conducted all testing in accordance with a DOT&E- approved test plan.
Assessment • Initial analysis of completed operational testing indicates that the Mk 54 BU software provides some limited operational capability in certain scenarios and against the UONS threat, but does not meet all the original program requirements. Reconstruction and analysis of the September 2013 Cape Cod testing is in progress to determine if sufficient information to assess performance has been obtained with the reduced number of shots conducted or if another event must be scheduled. • In preparation for the May 2013 test: - Navy operational testers uncovered inconsistencies in the operator’s tactical guidance, documentation, and training for the employment of the Mk 54 BU torpedo, some of which date from the introduction of the Mk 54 Mod 0 to the fleet in 2004. The inconsistent documentation, tactical guidance, and training could prevent fleet operators from effectively presetting and employing the Mk 54 BU. - Testing also discovered some required weapon presets were not selectable by crews using the MH-60R combat control system introduced to the fleet in 2010. The Navy’s early fielding and Quick Reaction Assessment processes did not
212 Mk 54
identify these critical shortfalls. The Navy investigated and found it had a problem in communication between the torpedo developers, platform fire control system developers, tactics developers, the training community, and the fleet users. The Navy is instituting new processes intended to rectify this situation. • In August 2013, the Navy updated and issued interim Mk 54 BU employment guidance to MH-60R fleet operators and trainers. • Almost two years after the early fielding, the Navy has not yet provided fleet operators and trainers adequate employment guidance or completed required operational testing. Recommendations • Status of Previous Recommendations. All of the previous recommendations remain outstanding. The Navy still needs to: 1. Conduct mobile target set-to-hit testing. The Navy completed an initial terminal homing assessment against the set-to-hit Steel SSK static target surrogate; however, the Navy deferred the mobile testing due to the lack of a suitable target surrogate. 2. Continue to develop a lethality strategy that includes the firing of the Mk 54 against appropriate targets. The Navy has identified the plan to conduct this testing with the Mk 54 Mod 1 torpedo upgrade. 3. Fund an operationally realistic mobile set-to-hit target to complete the terminal homing testing of the Mk 54 torpedo. The Navy continues to investigate possible surrogates; however, the proposals are unfunded. 4. Propose alternatives, such as the use of a portable range, to minimize or eliminate the test and safety limitations that minimize operational realism in Mk 54 testing. • FY13 Recommendations. The Navy should: 1. Complete development of the Mk 54 Mod 1 requirements and TEMP. This TEMP should include all the necessary resources or plans to develop the necessary resources, including target and range needs, to complete the remaining testing. 2. Pursue development of an evasive mobile set-to-hit target and threat representative countermeasures to support operationally realistic development and test of the Mk 54 Mod 1 torpedo. The targets identified by the Navy’s Torpedo Target Strategy Working Group will support some Mk 54 development and testing. 3. Institute processes to verify the incremental upgrades to the Mk 54 are interoperable with the variety of combat systems on surface ship, aircraft, and helicopter platforms. 4. Institute processes to update the operator’s tactical guidance, documentation, and training when implementing upgrades to the Mk 54.
N av y P R O G R A M S
MQ-4C Triton Unmanned Aircraft System Executive Summary • Due to a series of system integration and software maturity problems, the Navy delayed MQ-4C first flight and the planned developmental flight test program from May 2012 to May 2013. As a result, the Navy was unable to execute previously approved program development, test, and production schedules leading to an operational assessment in FY13 and a Milestone C decision in FY14. The Navy is currently developing revised program plans and schedules necessary to update the acquisition program baseline. • Since the MQ-4C first flight test in May 2013, initial safety of flight and air vehicle envelope expansion testing has proceeded as planned with only minor problems or delays. At the current pace of test execution, initial air vehicle testing will continue into FY14, while software development timelines will drive mission system integration and sensor performance testing to late FY14. • The Northrop Grumman Multi-Function Active Sensor (MFAS) risk reduction flight test program identified several system performance problems for resolution prior to MFAS integration on to the MQ-4C platform. The contractor implemented a series of radar software changes to improve sensor stability, maritime target surveillance and tracking performance, and synthetic aperture radar image quality. System • The MQ-4C Triton Unmanned Aircraft System is an Intelligence, Surveillance, and Reconnaissance system- of- systems consisting of the high-altitude, long- endurance MQ-4C air vehicle, sensor payloads, and supporting ground control stations. The MQ-4C system is a part of the Navy Maritime Patrol and Reconnaissance family- of-systems, with capabilities designed to complement the P-8A Poseidon. • The MQ-4C air vehicle design is based on the Air Force RQ‑4B Global Hawk air vehicle with significant modifications that include strengthened wing structures, anti-ice and de-icing systems, and an air traffic de-confliction and collision avoidance radar system. • The MQ-4C is equipped with the MFAS maritime surveillance radar to detect, identify, and track surface targets and produce high-resolution imagery. The MQ-4C
Activity • Due to a series of system integration and software maturity problems, the Navy delayed MQ-4C first flight and the planned developmental flight test program from May 2012 to May 2013. As a result, the Navy was unable to execute previously approved program development, test, and
electro‑optical/infrared sensor provides full motion video and still imagery of surface targets. An Electronic Support Measures system detects, identifies, and geo-locates radar threat signals. An Automatic Identification System (AIS) receiver permits the detection, identification, geo-location, and tracking of cooperative vessels equipped with AIS transponders. • Onboard line-of-sight and beyond line-of-sight datalink systems transmit sensor data from the air vehicle to ground control stations for dissemination to fleet tactical operation centers and intelligence exploitation sites. Mission • Commanders use units equipped with MQ-4C to conduct maritime surveillance operations and provide high-altitude, long-endurance intelligence collection. • MQ-4C operators detect, identify, track, and assess maritime and littoral targets of interest and collect imagery and signals intelligence information. Operators disseminate sensor data to fleet units to support a wide range of maritime missions to include surface warfare, intelligence operations, strike warfare, maritime interdiction, amphibious warfare, homeland defense, and search and rescue. Major Contractor Northrop Grumman Aerospace Systems, Battle Management and Engagement Systems Division – Rancho Bernardo, California
production schedules leading to an operational assessment in FY13 and a Milestone C decision in FY14. The Navy is currently developing revised program plans and schedules necessary to update the acquisition program baseline.
MQ-4C 213
N av y P R O G R A M S • Since beginning the MQ-4C flight test in May 2013, the Navy has accomplished a series of flight tests focusing on air vehicle guidance and control, flight envelope expansion, flying qualities, communication systems, and other basic air vehicle functions. • The Navy continued to expand ground test activity using the Navy Systems Integration Laboratory and other software development and verification facilities. Ground testing focused on supporting flight test activities, development of sensor software, and interoperability risk reduction testing. • The Navy continued MFAS radar risk reduction flight testing on a Northrop Grumman surrogate test bed aircraft to identify and resolve potential radar performance problems prior to integration on the MQ-4C air vehicle. The contractor completed more than 25 test flights by the end of FY13 with plans to continue this risk reduction activity through December 2013. Assessment • The Navy is currently revising program test and production schedules due to technical problems encountered during early developmental testing. Since first flight and air vehicle envelope expansion test activities were delayed for one year due to system technical difficulties, the operational assessment and associated Milestone C decision will likely be delayed until FY15 with IOT&E rescheduled for FY17. A final decision on program schedule revisions is on-hold pending resolution of FY14 budget uncertainties. • Since the MQ-4C first flight test in May 2013, initial safety of flight and air vehicle envelope expansion testing has proceeded as planned with only minor problems or delays. At the current pace of test execution, initial air vehicle testing will continue into FY14, while software development timelines will drive mission system integration and sensor performance testing to late FY14.
214 MQ-4C
• The Northrop Grumman MFAS risk reduction flight test program identified several system performance problems for resolution prior to MFAS integration on to the MQ-4C platform. Radar software changes have been implemented to improve sensor stability, maritime target surveillance and tracking performance, and synthetic aperture radar image quality. • The Navy encountered significant technical difficulties during early development of the planned MQ-4C air traffic de- confliction and collision avoidance radar system. The program is currently analyzing other technical options to provide air traffic collision avoidance capabilities. This is a critical mission capability for operation of the MQ-4C in civil and international airspace in support of global naval operations. Recommendations • Status of Previous Recommendations. The Navy made progress implementing the FY12 recommendation to retain previously approved system demonstrations and operational assessments in revised program schedules leading to a Milestone C decision. The Navy is integrating this recommendation into a revised acquisition program baseline and schedule expected to be submitted for approval in FY14. • FY13 Recommendations. The Navy should: 1. Develop a revised program test schedule that reflects the extensive FY13 program test delays. 2. Develop and submit for approval a revised Test and Evaluation Master Plan that reflects a revised program test and evaluation strategy through the Milestone C decision, IOT&E, and initial operational fielding.
N av y P R O G R A M S
Multi-Static Active Coherent (MAC) System Executive Summary • The Navy completed the initial operational testing of the Multi-Static Active Coherent (MAC) system on P-3C aircraft in October 2013. • Initial operational test results indicate that the MAC system provides P-3C aircraft with some limited wide-area Anti-Submarine Warfare (ASW) search capability in select scenarios but it does not meet the program’s requirements in some operational environments. • The IOT&E did not fully examine the capability of MAC across all operational conditions, representative operational environments, and target types. DOT&E agreed to limit testing during the initial phase because sufficient active source buoys were not available and because the MAC system would be installed and further tested on P-8A aircraft. System • The MAC system is an active sonar system composed of two types of buoys (source and receiver) and an acoustic processing software suite. It is employed by the Navy’s maritime patrol aircraft (P-3Cs and eventually P-8As) to search for and locate threat submarines in a variety of ocean conditions. To plan MAC missions, the Navy is updating the Active System Performance Estimate Computer Tool (ASPECT)/Multi-static Planning Acoustics Toolkit (MPACT) currently used to plan Improved Extended Echo Ranging (IEER) system missions. • MAC replaces the Navy’s current IEER system, which employs non-coherent sources to produce loud sounds that reflect off submarine targets. MAC employs new coherent source buoys that enable multiple pings, optimized waveforms, and various ping durations, none of which the legacy IEER system provided. • The Navy initially intends to employ MAC on P-3C aircraft in a limited set of acoustic environments. Future increments
Activity • The Navy certified the MAC system ready for initial operational testing on P-3C aircraft in October 2012, but waived testing of ASPECT/MPACT because its bottom environment database was poorly populated causing it to inaccurately predict the probability of detection for the planned MAC search. Because of technical problems integrating the existing multi-static wide-area ASW search system (IEER) on P-8A aircraft, the Navy delayed testing the P-8A wide-area requirements until MAC was initially tested on P-3C and installed on P-8A. • Due to a shortage of MAC system source buoys, the Navy identified four system developmental test events that used
of MAC will be employed on P-8A aircraft and in a wider variety of acoustic ocean environments in order to span the operational envelope of threat submarine operations. MAC will be the primary wide-area acoustic search system for the P-8A. • MAC is expected to have fewer effects on marine mammals and the environment than the legacy IEER system. Mission The Navy intends for P-3C and P-8A crews equipped with MAC to support the search, detect, and localization phases of the ASW mission. MAC is particularly focused on large-area active acoustic searches for threat submarines. Major Contractors • Lockheed Martin – Manassas, Virginia • Sparton Electronics Florida, Inc. – De Leon Springs, Florida • Ultra Electronics, Undersea Sensor Systems Incorporated (USSI) – Columbia City, Indiana
realistic buoy placement that could supplement operational test data and reduce the initial phase of operational testing. DOT&E reviewed the available data and test execution and determined that three of the four events were conducted with sufficient operational realism to be valid for the operational evaluation. The Navy conducted the three events on P-3C aircraft in the deep-water operating area off the coast of Jacksonville, Florida, in the spring of 2012. The test design supplemented these three events with five additional deep‑water events near San Diego, California. • The Navy conducted seven deep-water operational test flights with P-3C aircraft in the Southern California operating areas
MAC 215
N av y P R O G R A M S
•
• • •
•
in January 2013, to obtain five events that met the operational conditions specified in the Navy’s requirements documents. Two events were invalid because of P-3 system-of-system and target problems or because the test platform was diverted to higher-priority tasking (counter-drug operations) during the test. The Navy conducted five of the eight planned shallow water MAC events with P-3C aircraft in the Narragansett Bay operating area in May 2013. The Navy paused operational testing to investigate observed performance problems. The Navy identified operator training and material problems on the P-3C aircraft as probable causes of the degraded performance. The Navy required that the testing be repeated. The Navy completed eight additional MAC test events in the Narragansett Bay operating area in October 2013, all eight of which were valid for assessment. MAC test data analysis is in progress to support an initial assessment of the MAC operational effectiveness and suitability. The Navy and DOT&E are developing a Test and Evaluation Master Plan for the future installations and incremental upgrades of MAC capability on both P-3C and P-8A aircraft that reflects the test program in the recently approved P-8A Increment 2 Test and Evaluation Master Plan. Funding for the MAC operational testing on P-8A must still be obtained. The Navy conducted all operational testing in accordance with a DOT&E-approved test plan.
Assessment • Preliminary operational test results indicate that the MAC system provides P-3C aircraft with some limited wide-area ASW search capability in select scenarios but it falls short of what the fleet identified as the capability they need to protect high value units. Initial testing revealed unexpected performance shortfalls that are still being investigated. The latest results from the test events conducted in October 2013 in the benign environment of the Narragansett Bay operating area appear to meet the desired low threshold, but cannot be used to characterize the system’s capability in other, more
216 MAC
difficult environments where it will be used in war. Testing to understand the effects different threat types and environments have on performance will continue through FY19 in conjunction with the P-8 program. • The IOT&E did not fully examine the capability of MAC across all operational conditions, representative operational environments, and target types. DOT&E agreed to limit testing of the initial phase of MAC because sufficient active source buoys were not available and because the MAC system would be installed and tested on P-8A aircraft in FY14. Additional testing is also required to examine planned MAC system-of-system upgrades. • Although the MAC system detection algorithms display possible submarine contacts, the operator must quickly distinguish the actual submarine target from a variety of clutter and false contact presentations. Complicating this task, completed test analysis identified that the MAC system presentation of the target, clutter, and false targets varies with environmental conditions and likely target types. The data also suggest operators are only able to recognize a small fraction of the valid system detections as targets. • The Navy uses ASPECT/MPACT to predict the expected system performance while planning MAC missions. In addition to the known limitations in ASPECT/MPACT that were deferred, the planning tool also appears to overestimate performance because it does not have a good estimate for operator recognition of a submarine target. Recommendations • Status of Previous Recommendations. The Navy has addressed all previous recommendations. • FY13 Recommendations. The Navy should: 1. Conduct testing to identify target and false target characteristics in a variety of threat environments and with a variety of submarine target types. 2. Incorporate information about the characteristics of both valid and false target presentations into the training program as future MAC training and testing occurs.
N av y P R O G R A M S
Navy Enterprise Resource Planning (ERP) Executive Summary • The Commander, Operational Test and Evaluation Force (COTF) conducted an FOT&E of the Navy Enterprise Resource Planning (ERP) Single Supply Solution Release 1.1 from April 1 through May 31, 2013. DOT&E gathered data from COTF and observed Navy users performing logistics operations at Naval Supply Systems Command (NAVSUP) Weapon Systems Support, Mechanicsburg, Pennsylvania; Fleet Logistics Center (FLC) Pearl Harbor, Hawaii; and FLC Norfolk, Virginia. • During FOT&E, COTF evaluated whether corrective actions had resolved IOT&E deficiencies in the following areas: - Initial Source Processing Time (ISPT) (a Key Performance Parameter (KPP)) - Intermediate Document (IDOC) processing - Organic repair contract modifications - System defect management • The FOT&E also evaluated the Warehousing and the Environmental Health and Safety (EH&S) capabilities, which were not available during IOT&E. • Navy ERP is operationally effective. The Navy ERP contribution to ISPT is minor and is acceptable to both users and evaluators. IDOC processing has made substantial progress since the IOT&E, meets threshold requirements, and continues to improve. The automated organic repair contract award and modification capability exceeds threshold requirements. Navy ERP effectively manages warehousing operations with some limitations that have acceptable workarounds. The Navy ERP EH&S capability adequately facilitates procurement, tracking, transportation, and handling of hazardous material (HAZMAT). • Navy ERP is operationally suitable. The system achieved all reliability, availability, and maintainability thresholds. The program’s configuration and defect management processes have improved since the IOT&E. The total number of outstanding defects has remained constant at around 500, but none are Severity 1 or Severity 2 deficiencies and the workarounds are acceptable. Most of the outstanding defects are longstanding, low-severity, low-priority deficiencies with viable workarounds. New deficiencies, particularly high-severity ones, are being corrected expeditiously. The regression testing process was efficient, with 87 percent of critical business test scripts automated. System • Navy ERP is an integrated financial, acquisition, and logistics information technology system that provides financial and budgetary management for all Navy system commands. It is fully deployed to approximately 72,000 users worldwide in support of NAVSUP and its FLCs, Naval Air Systems
Command, Naval Sea Systems Command, Space and Naval Warfare Systems Command, naval air stations, HAZMAT centers, Strategic Systems Program locations, and the Office of Naval Research. • The Navy ERP application architecture is based on the commercial off-the-shelf System Applications and Products (SAP) Business Suite and NetWeaver products. Navy ERP uses SAP ERP Central Component, SAP Supply Chain Management from the Business Suite and Enterprise Portal, Business Intelligence, Process Integration, and Knowledge Management modules. • The Navy ERP program is a major component of the Navy’s Global Combat Service Support family-of-systems and is compliant with the Global Information Grid. The system interfaces with 50 external automated systems to exchange acquisition, financial, manpower and personnel, and logistics data. Mission The Navy uses the system to: • Implement an ERP business management system for the Navy to modernize and standardize financial, workforce, and supply chain management across the naval enterprise • Manage more than one-half of its Total Obligation Authority • Produce auditable financial statements in the future, enabling compliance with federal financial and security standards, the Chief Financial Officers Act of 1990, and the DoD Information Assurance Certification and Accreditation Process
Navy ERP 217
N av y P R O G R A M S Major Contractors • International Business Machines (IBM) – Bethesda, Maryland • Deloitte – New York, New York
Activity • The Navy completed fielding of Navy ERP to the FLCs, partner sites, the Strategic Systems Program, and the Office of Naval Research in 1QFY13. • COTF conducted an FOT&E of Navy ERP Single Supply Solution Release 1.1 from April 1 through May 31, 2013. DOT&E observed Navy users performing logistics operations at Weapon Systems Support, Mechanicsburg, Pennsylvania; FLC Pearl Harbor, Hawaii; and FLC Norfolk, Virginia. • COTF conducted all testing in accordance with the DOT&E‑approved Test and Evaluation Master Plan and FOT&E plan. Assessment • The FOT&E evaluated whether corrective actions had resolved IOT&E deficiencies in the following areas: - ISPT (KPP) - IDOC processing - Organic repair contract modifications - System defect management • The FOT&E also evaluated the Warehousing and the EH&S capabilities, which were not available during IOT&E. • ISPT is the average time (in days) required to process material from a customer’s request to shipment from the warehouse. The measure is applicable to four classes of material: Aviation Repairables, Aviation Consumables, Maritime Repairables, and Maritime Consumables. During IOT&E, measurements of ISPT exceeded thresholds in all material classes except Aviation Consumables. Subsequent analysis showed that Navy ERP was not the primary factor affecting these times; the foremost cause of lengthy ISPT was backordered material. • ISPT is a poorly chosen KPP; it may measure the supply chain, but it is an invalid measure of Navy ERP effectiveness unless qualified further. Consequently, DOT&E, in coordination with NAVSUP and COTF, developed a new methodology to determine the Navy ERP contribution to ISPT. The Navy ERP contribution to ISPT is defined to include Navy ERP system and business process time, while excluding backorder time, an ISPT logistics factor that is independent of the ERP and its associated business processes. • During FOT&E, the Navy supply chain did not meet ISPT threshold values with 24.4 days for Aviation Repairables (22‑day threshold), 45.9 days for Maritime Repairables (23‑day threshold), and 18.2 days for Maritime Consumables (10-day threshold). The Navy ERP contribution to ISPT for each of these categories was 2.8 days for Aviation Repairables, 5.8 days for Maritime Repairables, and 4.2 days
218 Navy ERP
• Electronic Consulting Services (ECS) iLuMinA Solutions, Inc. – Fairfax, Virginia
•
•
•
•
for Maritime Consumables. This is well below supply chain ISPT threshold values, is minor compared to non-ERP factors, and is acceptable to both users and evaluators. Navy ERP communicates certain transactions with external systems via IDOCs. If an IDOC is defective when it is received, Navy ERP is programmed not to process it. The failed document must then be processed manually. This safety mechanism prevents populating the system with bad information, but too many failures can adversely affect operations, require more time and manpower to process orders, and pay vendors. Following IOT&E, NAVSUP established a goal of less than 10 percent failures overall and accomplished this threshold with an IDOC failure rate of less than 7 percent over the past year. During IOT&E, the organic repair capability did not provide for automated processing of contract awards and modifications, resulting in users performing most of the process off-line. The Program Office developed an automated organic repair contract award and modification capability. Tests at all FOT&E sites resulted in a success rate of over 96 percent (between 93.4 and 99.0 percent at an 80 percent confidence level.) The program’s configuration and defect management processes have improved since the IOT&E. A Configuration Control Board effectively manages software changes, prioritizing them by criticality, user need, and cost. The total number of outstanding defects has remained constant at around 500, but none are Severity 1 or Severity 2 deficiencies and the workarounds are acceptable. Most of the outstanding defects are longstanding, low-severity, low-priority deficiencies with viable workarounds. New deficiencies, particularly high-severity ones, are being corrected expeditiously. The regression testing process was efficient, with 87 percent of critical business test scripts automated. Navy ERP effectively manages warehousing operations with some limitations that have acceptable workarounds. Logistics personnel use a time consuming workaround to address discrepancies when reconciling depot inventories with the Naval Aviation Logistics Command Management Information System. Non-deployable air wing unit and stock replenishment requisitions were sometimes referred against deployable unit allowances, causing a manual review of each such action by warehouse managers. NAVSUP implemented a new, single national inventory management strategy to prioritize and streamline inventory management, making warehouse managers’ manual review process ineffectual.
N av y P R O G R A M S • The Navy ERP EH&S capability adequately facilitates procurement, tracking, transportation, and handling of HAZMAT. • Financial fraud testing could not be included in the FOT&E because a Federal Information System Controls Audit Manual (FISCAM) Phase II assessment was ongoing and the Program Office had not yet corrected financial vulnerabilities identified in the FISCAM Phase I report and during an Independent Verification and Validation. Recommendations • Status of Previous Recommendations. The Navy addressed all previous recommendations.
• FY13 Recommendations. 1. The Program Office and NAVSUP should continue to execute their current processes for reducing defect and IDOC failure backlogs. 2. The Program Office should develop a Naval Aviation Logistics Command Management Information System interface solution to increase the accuracy of warehouse inventories and reduce time-consuming workarounds. 3. NAVSUP should make fleet personnel aware of the new single national inventory management strategy. 4. The Program Office and COTF should address financial vulnerabilities and plan for financial fraud penetration testing in 2014.
NAVY ERP 219
N av y P R O G R A M S
220
N av y P R O G R A M S
P-8A Poseidon Multi-Mission Maritime Aircraft Executive Summary • The Navy conducted the P-8A Increment 1 IOT&E from September 2012 through March 2013. Based on IOT&E results, the P-8A Increment 1 system provides maritime patrol mission capabilities similar to the legacy P-3C system in selected mission areas, but it is not effective for executing the full range of mission tasks required by the P-8A Increment 1 concept of operations. - The P-8A Increment 1 system provides effective small- area, cued Anti-Submarine Warfare (ASW) search, localization, and attack mission capabilities similar to the legacy P-3C system. Fundamental limitations in current sensor technology restrict search capabilities against more stressing adversary targets, making the P-8A not effective in some mission scenarios. The P-8A does not have an equivalent broad-area ASW acoustic search capability similar to that provided by the P-3C Improved Extended Echo Ranging system. The Navy intends to install the next generation multi-static active system to provide broad-area ASW search capabilities as part of the P-8A Increment 2 program. As a result of these two sensor shortfalls, the P-8A cannot execute the full range of mission tasks required by the ASW concept of operations. In fact, current P-8A ASW search capabilities provide only a small fraction of what is needed for most Navy operational plans. P-8A non-acoustic search capabilities are also very limited for evasive targets attempting to limit exposure to detection by radar and other sensors. Existing Mk 54 torpedo limitations also reduce P-8A attack effectiveness against evasive targets. - The P-8A is effective in conducting unarmed Anti-Surface Warfare (ASuW) missions against maritime surface targets. The radar and supporting sensor systems provide an effective, all-weather surface target search, detection, and classification capability at short to medium ranges for all maritime surface targets and at longer ranges for larger target vessels. - The P-8A is not effective for the Intelligence, Surveillance, and Reconnaissance (ISR) mission. Radar performance deficiencies, sensor integration problems, and data transfer system interoperability shortfalls degrade imagery intelligence collection and dissemination capabilities. The Electronic Support Measures (ESM) sensor provides a limited electronic intelligence (ELINT) capability, when supported by well-defined signal signature libraries. The P-8A demonstrated the capability to collect exploitable acoustic signature intelligence data. - P-8A aircraft flight performance meets or exceeds operational requirements and fully supports execution of the ASW, ASuW, and ISR concept of operations. The
system provides increased range, payload, and speed compared to the legacy P-3C aircraft. • The P-8A Increment 1 system is operationally suitable. The P-8A offers significant improvements in system hardware reliability, maintainability, and availability compared to the legacy P-3C aircraft. However, frequent mission software faults indicate that mission system stability and software maturity require further improvement. Over 75 percent of observed critical mission system failures resulted from software-related events. • The P-8A is survivable in permissive threat environments. Survivability in other threat environments presented by peer, second-tier adversary, or non-state actors depends primarily on the threat detection capabilities of organic sensor systems and threat intelligence updates from off-board sources via datalinks and communication systems. - Current P-8A systems provide sufficient information for crews to effectively remain outside most threat engagement zones. However, some combinations of environmental factors, target density, and increased crew workload due to system integration problems can degrade threat situational awareness, which increases the likelihood of inadvertent entry into these threat engagement zones. - If engaged, the P-8A Early Warning Self-Protection (EWSP) system capabilities to prevent Man-Portable Air Defense System missile hits are similar to those for comparable aircraft with similar protection systems. - The P-8A has no radar warning receiver capability or countermeasures to provide warning or protection against radio frequency (RF)-guided threats. - The P-8A vulnerability reduction features including On-Board Inert Gas Generator (OBIGGS) and Dry Bay
P-8A Poseidon 221
N av y P R O G R A M S Fire Protection System (DBFPS) improve the P-8A survivability when hit by likely gun threats. • The Navy completed developmental and integration testing of the AGM-84 Harpoon Block 1C anti-ship missile on the P-8A aircraft in September 2013. FOT&E to verify system integration and effective employment of armed ASuW capabilities is scheduled for early FY14. • In October 2013, DOT&E approved Test and Evaluation Master Plans (TEMPs) for the P-8A Increment 2 and Increment 3 programs that identified test strategies and required test resources necessary to execute operational testing for these programs through FY19. These programs are intended to significantly improve P-8A ASW and ASuW mission capabilities by integrating improved sensors, weapons, and mission system technologies. These TEMPs also incorporate test strategies for the next generation multi-static active system. This key P-8A sensor system upgrade is intended to provide P-8A with wide area ASW search capabilities necessary to execute both the current ASW concept of operations and future high-altitude ASW employment concepts. System • The P-8A Poseidon Multi-mission Maritime Aircraft design is based on the Boeing 737-800 aircraft with significant modifications to support Navy maritime patrol mission requirements. It will replace the P-3C Orion. • The P-8A incorporates an integrated sensor suite that includes radar, electro-optical (EO), and electronic signal detection sensors to detect, identify, locate, and track surface targets. An integrated acoustic sonobuoy launch and monitoring system detects, identifies, locates, and tracks submarine targets. The P-8A carries Mk 54 torpedoes and is currently integrating
Activity • The Navy conducted the P-8A Increment 1 IOT&E from September 2012 through March 2013. IOT&E events included testing conducted in conjunction with fleet exercises in Guam, the United Kingdom, and Japan, and during dedicated operational test events in the United States. IOT&E included 93 flight missions totaling 561 flight hours to evaluate operational effectiveness and survivability. DOT&E evaluated 1,620 maintenance actions performed in the course of integrated and operational test missions, totaling 727 flight hours. Testing was completed in accordance with the DOT&E-approved TEMP and IOT&E plan. • The Navy completed live fire test events on an actual P-8A airframe – the S-1 structural test article – to assess P-8A vulnerability to ballistically-induced structural failure and sustained dry bay fire. The Navy also completed the performance verification testing of the P-8A vulnerability reduction features including OBIGGS and DBFPS. Incorporating the results from these tests, the Navy used
222 P-8A Poseidon
the AGM-84 Harpoon missile system to engage identified submarine and surface targets. Sensor systems also provide tactical situational awareness information for dissemination to the fleet and ISR information for exploitation by the joint intelligence community. • The P-8A aircraft incorporates aircraft survivability enhancement and vulnerability reduction systems. An integrated infrared (IR) missile detection system, flare dispenser, and directed IR countermeasure system is designed to improve survivability against IR missile threats. On and off-board sensors and datalink systems are used to improve tactical situational awareness of expected threat systems. Fuel tank inerting and fire protection systems reduce aircraft vulnerability. Mission • Theater Commanders primarily use units equipped with the P-8A Multi-mission Maritime Aircraft to conduct ASW. P-8A units detect, identify, track, and destroy submarine targets. • Additional P-8A maritime patrol missions include: - ASuW operations to detect, identify, track, and destroy enemy surface combatants or other shipping targets - Maritime and littoral ISR operations to collect and disseminate imagery and signals information for exploitation by the joint intelligence community - Collection and dissemination of tactical situation information to improve the fleet common operational picture - Identification and precise geo-location of targets ashore to support fleet strike warfare missions Major Contractor Boeing Defense, Space, and Security – St. Louis, Missouri
standard DoD-sponsored vulnerability analysis tools to determine the overall P-8A vulnerable area and probability of kill given a hit as well as the likelihood of crew casualties. In assessing P-8A susceptibility, the Navy completed hardware-in-the-loop simulation and flight testing of the EWSP system. Testing was completed in accordance with the DOT&E‑approved TEMP and the Live Fire Alternative test plan. • In September 2013, the Navy completed development of P-8A Increment 1 Operational Flight Program software upgrades and integration testing to support carriage and employment of the AGM-84 Harpoon Block 1C anti-ship missile. This upgrade will provide P-8A with an armed ASuW mission capability. The Navy also implemented Operational Flight Program software changes to correct a limited number of system performance deficiencies identified during IOT&E. The Navy is planning to conduct FOT&E
N av y P R O G R A M S to verify AGM-84 Harpoon integration and deficiency corrections in early FY14 prior to initial operational fielding. • The Navy completed P-8A Increment 2 TEMP development and initiated early software development testing for this program in FY13. During the Increment 2 program, the Navy intends to install and upgrade the Multi-static Active Coherent (MAC) system (currently in IOT&E on P-3C aircraft) to provide a limited broad-area search capability on P-8A, complete delayed IOT&E testing, add high-altitude ASW capability, and correct some IOT&E deficiencies. • The Navy completed P-8A Increment 3 TEMP development for the P-8A Increment 3 program. This program is intended to provide additional ASW sensor capabilities and upgrades to mission system architectures in the FY19 timeframe. Assessment • Based on IOT&E results, the P-8A Increment 1 system provides effective small-area, cued ASW search, localization, and attack mission capabilities, similar to the legacy P-3C system. - Fundamental limitations in current sensor technology restrict search capabilities against more stressing adversary targets, making the P-8A not effective in some mission scenarios. - The P-8A does not have an equivalent broad-area ASW acoustic search capability similar to that provided by the P-3C Improved Extended Echo Ranging system. The Navy intends to install the next generation multi-static active system to provide broad-area ASW search capabilities as part of the P-8A Increment 2 program. - As a result of these two sensor shortfalls, the P-8A cannot execute the full range of mission tasks required by the ASW concept of operations. In fact, current P-8A ASW search capabilities provide only a small fraction of what is needed for most Navy operational plans. P-8A non-acoustic search capabilities are also very limited for evasive targets attempting to limit exposure to detection by radar and other sensors. Existing Mk 54 torpedo limitations also reduce attack effectiveness against evasive targets. • The P-8A Increment 1 system is effective in conducting unarmed ASuW missions against maritime surface targets. - The P-8A radar provides an effective, all-weather surface target search and detection capability at short to medium ranges for all maritime surface targets and at longer ranges for larger target vessels. - P-8A sensors effectively support surface surveillance operations and cue other Navy surveillance and strike platforms. However, the P-8A radar track-while-scan mode does not provide reliable track information for targets outside the radar field-of-view. Operational workarounds require P-8A crews to track each target of interest manually, which significantly increases sensor operator workload in target-dense operational environments.
- P-8A unarmed ASuW maritime surface target search, classification, track, and cue-to-attack capabilities are equivalent to P-3C capabilities. The Navy deferred armed ASuW mission capability until successful integration of the AGM-84 Harpoon anti-ship missile in FY14. • The P-8A Increment 1 system is not effective for the ISR mission. Imagery intelligence collection and dissemination capabilities are limited by radar performance deficiencies, sensor integration problems, and data transfer system interoperability shortfalls. - The P-8A sensor suite can effectively collect EO and IR imagery in clear weather, day/night conditions. However, the P-8A does not have an effective high-resolution synthetic aperture radar imagery collection capability. - The P-8A ESM sensor provides a limited ELINT capability with high-signal detection and identification rates when supported by well-defined, signal signature libraries specifically tailored to the expected electronic order of battle in a specific theater of operations. However, ELINT signal identification capabilities are limited by ESM signature library-size constraints. - The P-8A demonstrated the capability to collect exploitable acoustic signature intelligence data during test events utilizing surface vessel targets. • The P-8A Increment 1 system provides a limited command, control, and communications mission capability to monitor and disseminate maritime target information to enhance the tactical awareness of maritime forces and on-scene commanders. - During fleet training exercises, P-8A crews developed, maintained, and disseminated key elements of the fleet common operating picture to participating units while simultaneously conducting ASW, ASuW, and ISR operations. However, radar track-while-scan performance deficiencies and data display limitations often require manual target position tracking by the operator, which reduced tactical awareness and the capability to disseminate timely information to fleet forces. - Communication system interoperability shortfalls related to the International Maritime Satellite, Common Data Link, and voice satellite communication systems limit crew access to off-board intelligence updates, preclude participation in some real-time tactical communication forums, and reduce capabilities to transmit tactical and intelligence data updates to on-scene commanders. - Recent developmental test results indicate that the Navy has improved performance in the majority of these areas. Mission capability improvements will be evaluated during FOT&E planned for early FY14. • P-8A aircraft flight performance meets or exceeds operational requirements and fully supports execution of the ASW, ASuW, and ISR concept of operations. - The aircraft can effectively self-deploy from main operating base locations to primary theater deployment sites and sustain long-term operations at more remote forward operating locations. Unrefueled range exceeds
P-8A Poseidon 223
N av y P R O G R A M S 4,000 nautical miles and increased transit speeds reduce transit times as compared to the legacy P-3C system. The P-8A is compatible with planned operating locations and meets worldwide navigation and airspace operating requirements. - Weapons and expendable store carriage and employment capabilities support planned ASW and ASuW mission operations. - The P-8A provides an adequate all-weather operating capability in most operational environments, although main tank fuel overheating problems currently preclude ground and flight operations during peak temperature periods in extreme hot weather environments. - Cyber-security measures implemented for the P-8A are effective. • The P-8A Increment 1 system is operationally suitable for ASW, ASuW, and ISR mission operations. The P-8A offers significant improvements in system reliability, maintainability, and availability compared to the legacy P-3C aircraft. - During fleet exercise missions conducted from main operating bases and worldwide forward operating locations, the P-8A demonstrated high-mission reliability with an on-time take-off rate of 93.6 percent and airborne mission abort rate of only 1.6 percent. - Operational availability exceeded the established Navy requirement of 60 percent for initial fielding. P-8A hardware reliability, system maintenance frequency, and maintenance corrective action times surpass operational requirement thresholds, directly contributing to high- operational availability rates. However, frequent mission software faults indicate that mission system stability and software maturity require further improvement. Over 75 percent of observed critical mission system failures resulted from software-related events. • The P-8A is survivable in permissive threat environments. Survivability in conflicts against peer adversaries with advanced military technologies, second-tier adversary nations with less sophisticated threat systems, or non-state actors, depends on the P-8A capability to use off-board intelligence sources and onboard sensor performance to maintain safe standoff distances from all expected threats. The P-8A systems provide sufficient information for the crew to remain outside most threat engagement zones. However, some combination of environmental conditions, target density, and increased crew workload due to system integration problems can degrade threat situational awareness, which increases the likelihood of inadvertent entry into these threat engagement zones. - If engaged, the EWSP testing demonstrated the effectiveness of the system against a range of simulated Man-Portable Air Defense System missiles. The EWSP system has no radar warning receiver capability or countermeasures to provide warning or protection against RF-guided threats.
224 P-8A Poseidon
•
•
•
•
- The P-8A vulnerability reduction features (e.g., OBIGGS, DBFPS, etc.) improve its survivability when hit by likely gun threats: ▪▪ OBIGGS is capable of reducing fuel tank oxygen levels to a non-combustible 9 percent throughout most flight conditions except for an emergency dive when concentrations went as high as 9.5 percent. The aircraft fuel tanks can withstand the pressure rise expected from combustion at this oxygen concentration. ▪▪ The effectiveness of the P-8A DBFPS was lower than demonstrated in developmental tests using surrogate test articles. The DBFPS system reduces the P-8A vulnerability against ballistic threats from that of the unprotected aircraft. The P-8A vulnerability to dry bay fire could be further reduced by changing the DBFPS suppressor design and footprint. Following developmental and integration testing of the AGM-84 Harpoon Block 1C anti-ship missile on the P-8A, the Navy certified the system for missile carriage, safe separation, and employment. FOT&E to verify system integration and effective employment is on schedule for early FY14. The Navy also conducted additional developmental testing to correct a limited number of system deficiencies identified during IOT&E. System improvements in the following areas are expected to be delivered for re-evaluation during FOT&E in FY14: - Radar track-while-scan mode target tracking - Radar pointing and high-resolution imagery collection - EO/IR sensor cueing and target tracking - International Maritime Satellite and Common Data Link interoperability - Radar periscope detection and search capability - Initial MAC broad-area ASW search capability The Navy completed P-8A Increment 2 TEMP development and initiated early software development testing for this program in FY13. During Increment 2, the Navy intends to install and upgrade the MAC system (currently in IOT&E on P-3C aircraft) to provide a limited broad-area search capability on P-8A, complete delayed IOT&E, add high-altitude ASW capability, and correct some IOT&E deficiencies. The Navy completed P-8A Increment 3 TEMP development for the P-8A Increment 3 program. This program is intended to provide additional ASW sensor capabilities and upgrades to mission system architectures in the FY19 timeframe.
Recommendations • Status of Previous Recommendations. The Navy made progress on three of the four FY12 recommendations. The Navy completed recommended LFT&E events prior to completion of IOT&E. The Navy accelerated efforts to correct a number of key system deficiencies identified in FY12 testing and is planning to conduct FOT&E to verify fix effectiveness prior to operational deployment in FY14. Remaining deficiency corrections were deferred to future test periods.
N av y P R O G R A M S • FY13 Recommendations. The Navy should: 1. Implement corrective actions for deficiencies identified in the DOT&E IOT&E report and conduct FOT&E to verify improved mission capabilities. 2. Complete adequate operational testing of delayed capabilities and of new system improvements intended to provide P-8A a broad-area and high-altitude ASW search and attack capability.
3. Consider integrating RF threat warning and countermeasure self-protection systems on the P-8A aircraft to improve threat situational awareness and to provide protection against RF-guided threat systems. 4. Modify the DBFPS design, i.e., increase the number and volume of suppressors or change their type and location to improve DBFPS effectiveness.
P-8A Poseidon 225
N av y P R O G R A M S
226
N av y P R O G R A M S
Remote Minehunting System (RMS) Executive Summary • Contractor testing completed in FY13 suggests that vehicle reliability has grown since the Remote Minehunting System (RMS) program emerged from the Nunn-McCurdy review in FY10; however, these tests were not conducted in an operationally realistic manner. Data from the recent developmental testing suggest that reliability may not have improved sufficiently to enable a Littoral Combat Ship (LCS) with two Remote Multi-Mission Vehicles (RMMVs) onboard to complete the desired area search without having to return to port more often than currently planned and desired to obtain replacements. An accurate assessment of achieved RMMV reliability cannot be made until the RMS is tested under operationally realistic end-to-end minehunting missions. • As observed during operational assessment and developmental testing of the MH-60S Organic Airborne Mine Countermeasures, the AN/AQS-20A does not meet all Navy requirements in all operating modes. • The analysis of test data collected during developmental testing of RMS communications and launch, handling, and recovery improvements, and the AN/AQS-20A sonar is still in progress. The Navy expects to issue formal developmental test reports in 2QFY14. System • The RMS is designed to provide an organic, off-board mine reconnaissance capability to detect, classify, and localize non-buried bottom and moored mines, as well as to identify shallow-water bottom mines only. • The RMS will be launched, operated, and recovered from the LCS as part of the Mine Countermeasures (MCM) mission package (when embarked). • The RMS is comprised of four components: - RMMV ▪▪ The RMMV is an unmanned, semi-submersible, un‑tethered vehicle designed to conduct autonomous or semi-autonomous mine reconnaissance missions. ▪▪ The RMMV physically transports AN/AQS-20A sensors, processors, and datalink equipment to the operations area where mine reconnaissance data are collected, recorded, and transmitted to the host LCS platform. - AN/AQS-20A sensor ▪▪ The AN/AQS-20A is a variable depth forward-looking and side-scanning sonar that is deployed and retrieved by the RMMV. ▪▪ The sensor tow body automatically controls depth based on specific mission planning parameters, providing a stable platform for integral mine reconnaissance sensors. ▪▪ The AN/AQS-20A provides detection, classification, and localization of non-buried bottom (on the ocean floor)
and volume (in the water column) mine-like-contacts. The sensor utilizes a port and starboard Side-Looking Sonar and a Gap Filler Sonar for detection of bottom and tethered volume mines. A Volume Search Sonar (VSS) and a Forward-Looking Sonar are utilized for all mine type detection. An Electro-Optic Identification Device can replace the VSS for missions requiring (mine versus non-mine) identification of shallow-water bottom mine‑like‑contacts via high-resolution imaging. - Remote Minehunting Functional Segment (RMFS) ▪▪ R MFS is the software that will be hosted in the mission package computing environment on the LCS. ▪▪ R MFS is a two-operator system that enables the Remote Vehicle Operator and Remote Sensor Operator to command and monitor RMS operations. ▪▪ S pecific RMFS functionality enables the operator to (1) command and monitor the RMMV; (2) receive, process, and display real-time mission data; (3) conduct performance monitoring/fault detection/fault localization; and (4) perform network communication to the Data Link System (DLS). ▪▪ RMFS also exchanges data with the Global Command and Control System – Maritime/Mine Warfare Environmental Decision Aids Library for mission planning and interface to the Global Information Grid. - DLS ▪▪ The DLS enables the RMMV to communicate with the LCS MCM mission package via one of two radio frequency datalink subsystems. ▪▪ The Multi-Vehicle Communications System (MVCS) consists of two radios – an Ultra High Frequency line-of-sight datalink that is used for vehicle launch and recovery and near-ship operations and a low-band Very
RMS 227
N av y P R O G R A M S High Frequency datalink that is used for over-the-horizon mission operations. Both datalinks provide encrypted continuous communications between the LCS MCM mission package and the RMMV for real-time command and control and mission data capture. Mission MCM Commanders will employ the RMS from an MCM mission package-equipped LCS, to detect, classify, and localize non‑buried bottom and moored mines, as well as to
Activity • The RMMV contractor completed the second and final phase of system reliability growth improvements and completed 438 hours of in-water validation testing in 2QFY13. - An FY10 Acquisition Decision Memorandum, at the conclusion of the program’s Nunn-McCurdy review, directed the implementation of a reliability growth program for the vehicle and this testing to assess vehicle reliability improvements against a reduced reliability requirement. - An earlier phase of contractor testing was completed in 1QFY11. - Vehicles with the full complement of reliability improvements have not been tested on an LCS. Testing was conducted in benign conditions from shore, which did not subject the RMMV to the handling stresses imposed by the LCS handling system. • The Navy funded development of pre-planned product improvements for the AN/AQS-20A and is investigating improved tactics, techniques, and procedures for employment of the sensor. Both efforts are intended to correct effectiveness deficiencies observed during operational assessment and developmental testing of the AN/AQS-20A conducted in FY11. • The Navy requested approval to deviate from the operational assessment strategy prescribed in the approved RMS Test and Evaluation Master Plan (TEMP). - The approved RMS TEMP directs the conduct of ship‑based developmental testing and operational assessment of RMS in FY14 from an LCS at sea. Due to the unavailability of an RMS-compatible LCS seaframe to facilitate conduct of ship-based RMS testing, the planned FY14 testing will be conducted from a shore base. - Dedicated end-to-end mission testing of the RMS from an LCS ship-base may not occur until the programs’ Technical Evaluation starting in 4QFY14. DOT&E expects to approve the requested deviation in 2QFY14. • The Navy completed a scheduled phase of developmental testing of some structural improvements for the RMMV and the RMMV launch, handling, and recovery system and MVCS upgrades in dockside and at-sea testing in 4QFY13. • The Navy completed a supplemental phase of developmental testing of the AN/AQS-20A in 4QFY13. The testing of the
228 RMS
identify shallow-water bottom mines only in support of theater minehunting operations in shallow-water and deep-water minefields. Major Contractors • RMMV: Lockheed Martin – West Palm Beach, Florida • AN/AQS-20A: Raytheon Corporation – Portsmouth, Rhode Island
sensor, towed behind the Athena Research Vessel System, was intended to characterize detection/classification performance against moored mines located near the surface. • In December 2013, the Navy proposed a new RMS acquisition strategy to support a Milestone C decision. DOT&E did not concur with the proposal because the selected measure for the RMMV reliability was not appropriate to ensure the new units would be operationally suitable and the quantity of units being procured prior to the completion of IOT&E was excessive. • DOT&E recommends strongly that the planned operational assessment previously expected to be conducted 2QFY14 be postponed until 3/4QFY14. DOT&E will not approve the Navy’s plan to conduct an operational assessment until the intended test article is representative of the system that will be tested during the LCS MCM mission package IOT&E and ultimately provided to the fleet at Initial Operational Capability. Upgrades to both the RMMV as well as the AN/AQS-20A are planned and the upgraded RMS is expected to start developmental testing in June 2014.
Assessment • Contractor testing completed in FY13 suggests that vehicle reliability has grown since the RMS program emerged from the Nunn-McCurdy review in FY10. However, these tests were not conducted in an operationally realistic manner. The defined reliability measurement for the Nunn-McCurdy reliability growth program is not operationally relevant in that it includes post-mission analysis time when the RMMV is not operating, doesn’t require the RMMV to be operating under a realistic load, permits additional maintenance if completed within two hours, and does not count several critical failures that would be termed operational mission failures in operational testing because they affect the performance of the mission. Hence, the reliability derived from the contractor testing is artificially inflated by at least a factor of two. - Data from the recent developmental testing, also conducted from shore but in a more operationally realistic manner, suggest that reliability may not have improved sufficiently to enable an LCS with two RMMVs onboard
N av y P R O G R A M S to complete the desired area search without having to return to port more often than currently planned and desired to obtain replacements. - An accurate quantitative assessment of operational availability of the RMS (a Key Performance Parameter) will not be obtainable until the reliability, maintainability, and logistics supportability of the RMS can be assessed during ship‑based testing from an LCS as part of the MCM mission package. • As observed during operational assessment and developmental testing of the MH-60S Organic Airborne Mine Countermeasures, the AN/AQS-20A still does not meet all Navy requirements in all operating modes. - Contact depth (vertical localization) errors exceeded Navy limits in all AQS-20A operating modes. False classification density (number of non-mine like objects erroneously classified as mine-like per unit area searched) also exceeded Navy limits in two of three search modes. If left uncorrected, a large number of false targets and vertical localization errors generated by the AN/AQS-20A will reduce the minehunting capability of the LCS with an embarked MCM mission package. - In 2008, developmental testing of the RMS revealed that the system has problems meeting the probability of reacquisition requirement when attempting to identify bottom objects in deeper waters. The Navy expects to implement fixes in the next version of the vehicle to correct this deficiency. • The analysis of test data collected during developmental testing of structural improvements for the RMMV and the RMMV recovery system, and for MVCS upgrades is still in progress. However, sailors reported that communications between an RMMV equipped with MVCS upgrades and LCS 2 were unreliable throughout the test. The Navy expects to issue a formal test report in 2QFY14 and to complete additional MVCS and launch and recovery testing in 2QFY14 and 4QFY14. • The Navy has not yet demonstrated the system can meet its single pass detection and classification requirements against moored and bottom mines spanning the portion of the shallow water regime not covered by the Airborne Laser Mine Detection System (ALMDS).
- The Navy is weighing the need for multiple search passes with the sensor towed at different depths under some conditions. Use of multi-pass search tactics would require more time to cover the same area and would negatively affect the LCS area coverage rate. - Recent testing suggests that the AN/AQS-20A search envelope might be able to be extended upward to restore the desired overlap with the demonstrated ALMDS envelope. The analysis of test data collected during recent developmental testing of the AN/AQS-20A sensor is still in progress. The Navy expects to issue a formal developmental test report in 2QFY14. The Navy still must complete tactics development and operational testing to verify whether the use of the AN/AQS-20A will mitigate ALMDS shortfalls in expected threat environments. Recommendations • Status of Previous Recommendations. This is the first annual report for this program since 2008. The program was restructured in 2010 as a result of a Nunn-McCurdy breach. • FY13 Recommendations. The Navy should: 1. Conduct reliability testing of the RMS under operationally realistic end-to-end minehunting missions as soon as possible to accurately assess achieved RMS, RMMV, and AN/AQS‑20A reliability. 2. Conduct ship-based testing of the RMS that includes end‑to‑end minehunting missions from an LCS as part of the MCM mission package as soon as possible to: -- Assess operational availability of the RMS. -- Assess the RMMV launch, handling, and recovery system performance under operational conditions. -- Assess fixes to resolve communications problems observed in FY13 testing. -- Verify the RMS and LCS with MCM mission package are ready for IOT&E. 3. Investigate solutions and correct AN/AQS-20A False Classification Density and Vertical Localization deficiencies prior to IOT&E. 4. Update the RMS and AN/AQS-20A TEMPs and test plans to develop adequate testing to verify corrected deficiencies and assess operational capability of the systems the Navy expects to employ to meet LCS’s mission requirements.
RMS 229
N av y P R O G R A M S
230
N av y P R O G R A M S
Ship Self-Defense Executive Summary • The ship self-defense mission for aircraft carriers, destroyers, and amphibious warfare ships coordinates several legacy shipboard systems, as well as six major acquisition programs: Ship Self-Defense System (SSDS), Rolling Airframe Missile (RAM), Evolved SeaSparrow Missile (ESSM), Cooperative Engagement Capability (CEC), Surface Electronic Warfare Improvement Program (SEWIP), and the Air and Missile Defense Radar (AMDR). These comprise a self-defense capability for in-service ships, as well as the LPD-17, LHA-6, DDG 51 Flight III, and CVN-78 ship classes still in acquisition. • The Navy successfully completed the first phase of the RAM Block 2 IOT&E with four missile firings in May 2013 from the Self-Defense Test Ship (SDTS). • While the integration of sensor and weapon systems with the command and decision system enhances the ships’ self-defense capability over non-integrated combat systems, the Navy has not successfully demonstrated the ability to effectively complete the self-defense mission against the types of threats and threat scenarios for which the overall system was designed. • The Navy must complete the currently planned operational test programs and conduct additional testing to demonstrate the correction of significant deficiencies with SSDS Mk 2, RAM, ESSM, CEC, and legacy ship self-defense combat system elements. System Surface ship self-defense is addressed by several legacy combat system elements (ship class-dependent) and five acquisition programs: SSDS, RAM, ESSM, CEC, SEWIP, and AMDR. SSDS • SSDS is a local area network that uses open computer architecture and standard Navy displays to integrate a surface ship’s sensors and weapons systems to provide an automated detect-track-engage sequence for ship self- defense. SSDS Mk 1 is the command and control system for LSD-41/49 class ships. • SSDS Mk 2 has six variants: -- M od 1, used in CVN-68 class aircraft carriers -- M od 2, used in LPD-17 class amphibious ships -- M od 3, used in LHD-7/8 class amphibious ships -- M od 4, in development for LHA-6 class amphibious ships -- Mod 5, in development for LSD-41/49 class amphibious ships -- Mod 6, in development for CVN-78 class aircraft carriers
RAM • The RAM, jointly developed by the United States and the Federal Republic of Germany, provides a short-range, lightweight, self-defense system to defeat Anti-Ship Cruise Missiles (ASCMs). RAM is currently installed in all aircraft carriers and amphibious ships (except LPD-4 class). • There are three RAM variants: -- RAM Block 0 uses dual mode, passive radio frequency/ infrared guidance. -- RAM Block 1A adds infrared guidance improvements to extend defense against non-radio-frequency-radiating ASCMs. -- RAM Block 2 is in development and will extend the capability of RAM Block 1A against newer classes of ASCM threats. ESSM • The ESSM, cooperatively developed among 13 nations, is a medium-range, ship-launched self-defense guided missile designed to defeat ASCM, surface, and low- velocity air threats. The ESSM is currently installed on DDG 51 Flight IIA destroyers, as well as CVN-68 class aircraft carriers equipped with the SSDS Mk 2 Mod 1 Combat System. The Navy is planning for future ESSM installations in CG 47 class cruisers, LHA-6 class amphibious assault ships, CVN-78 class aircraft carriers, DDG 1000 class destroyers, and DDG 51 Flight III class destroyers. • There are two variants of ESSM. -- ESSM Block 1 is a semi-active radar-guided missile that is currently in-service. -- ESSM Block 2 is in development and will have semi‑active radar-guidance as well as active radar guidance.
Ship Self-Defense 231
N av y P R O G R A M S CEC • CEC is a sensor network with integrated fire control capability that is intended to significantly improve battle force air and missile defense capabilities by combining data from multiple battle force air search sensors on CEC-equipped units into a single, real-time, composite track picture. The two major hardware pieces are the Cooperative Engagement Processor, which collects and fuses radar data, and the Data Distribution System, which exchanges the Cooperative Engagement Processor data. CEC is an integrated component of, and serves as the primary air tracker for, SSDS Mk 2-equipped ships. • There are four major variants of CEC: -- The CEC USG-2 is used in selected Aegis cruisers and destroyers, LPD-17/LHD amphibious ships, and CVN‑68 class aircraft carriers. -- The CEC USG-2B, an improved version of the USG-2, is used in selected Aegis cruisers and destroyers. -- The CEC USG-3A is used in the E-2C Hawkeye 2000 aircraft. -- The CEC USG-3B is in development for use in the E-2D Advanced Hawkeye aircraft. AMDR • The AMDR is the Navy’s next generation radar system that is being developed to provide DDG 51 Flight III Destroyer combat systems with simultaneous sensor support of ballistic missile defense and air defense (to include self‑defense) missions. SEWIP • The SEWIP is an evolutionary development program providing block upgrades to the AN/SLQ-32 Electronic Warfare (EW) System to address critical capability, integration, logistics, and performance deficiencies. • There are three major SEWIP block upgrades: -- SEWIP Block 1 replaced obsolete parts in the AN/ SLQ- 32 in addition to incorporation of a new, user- friendly operator console, an improved electronic emitter identification capability, and an embedded trainer.
Activity • The Navy’s Commander, Operational Test and Evaluation Force (COTF) completed the first phase of RAM Block 2 IOT&E testing and the first phase of SSDS Mk 2 Mod 4 FOT&E testing on the SDTS in May 2013 with four RAM Block 2 missile firings. Testing was conducted in accordance with a DOT&E-approved test plan. • COTF continued planning for operational testing of the ship self-defense mission area during IOT&E of the RAM Block 2 and FOT&E of the SSDS Mk 2 Mod 4 and ESSM on the SDTS. The Navy plans to continue testing in March 2014. • The Navy instituted the Fire Control Loop Improvement Program (FCLIP) to address a number of the ship self defense
232 Ship Self-Defense
-- SEWIP Block 2 is in development and will incorporate a new receiver antenna system intended to improve the AN/SLQ-32’s passive EW capability. -- SEWIP Block 3 is in development and will incorporate a new transmitter antenna system intended to improve the AN/SLQ-32’s active EW capability. Mission Naval Component Commanders use SSDS, RAM, ESSM, and CEC, as well as many legacy systems, to provide faster, more effective accomplishment of ship self-defense missions. • Naval surface forces use SSDS to provide automated and integrated detect-to-engage ship self-defense capability against ASCM, air, and surface threats. • Naval surface forces use RAM to provide a short-range hard kill engagement capability against ASCM threats. • Naval surface forces use ESSM to provide a medium-range hard kill engagement capability against ASCM, surface, and low velocity air threats. • Naval surface forces use CEC to provide accurate air and surface threat tracking data to SSDS. • Naval surface forces will use AMDR as a primary sensor for simultaneous ballistic missile defense and air defense (to include self-defense) missions. • Naval surface forces will use the SEWIP-improved AN/SLQ‑32 as the primary EW sensor and weapons system for air defense (to include self-defense) missions.
Major Contractors • SSDS (all variants): Raytheon – San Diego, California • RAM and ESSM (all variants): Raytheon – Tucson, Arizona • CEC (all variants): Raytheon – St. Petersburg, Florida • AMDR: Raytheon – Dallas, Texas • SEWIP - Block 1: General Dynamics Advanced Information Systems – Fair Lakes, Virginia - Block 2: Lockheed Martin – Syracuse, New York - Block 3: To be determined
deficiencies identified in the classified November 2012 DOT&E report to Congress on the ship self-defense mission area. Assessment • The RAM Block 2 firings, while successful, were not conducted with any FCLIP improvements. The initial FCLIP improvements are not planned for testing until March 2014. Many of the recommended improvements identified in the classified November 2012 DOT&E report to Congress will not be addressed until FY16.
N av y P R O G R A M S • The test infrastructure remains inadequate to support self-defense testing on the DDG 51 Flight III Destroyers. The Navy has not planned or programmed funding for an unmanned, at-sea test capability to safely demonstrate the self‑defense capabilities of the DDG 51 Flight III against anti‑ship missile threats. The test capability must be in place by 2021 to support DDG 51 Flight III Destroyer Combat System and AMDR self-defense operational testing. The DDG 51 and AMDR programs are discussed in a separate section of this report. Recommendations • Status of Previous Recommendations. The Navy has satisfactorily completed some of the previous recommendations. The Navy has not resolved the following previous recommendations: 1. Optimize SSDS Mk 2 weapon employment timelines to maximize weapon probability of kill. 2. Develop a credible open-loop seeker subsonic ASCM surrogate target for ship self-defense combat system operational tests. 3. Correct the identified SSDS Mk 2 software reliability deficiencies. 4. Correct the identified SSDS Mk 2 training deficiencies. 5. Develop and field deferred SSDS Mk 2 interfaces to the Global Command and Control System – Maritime and the TPX-42A(V) command and control systems. 6. Continue to implement the Program Executive Office for Integrated Warfare Systems’ plan for more robust, end‑to‑end systems engineering and associated developmental/operational testing of ship self-defense combat systems. 7. Provide a capability to launch a raid of four supersonic sea‑skimming targets at the Naval Air Warfare Center/ Weapons Division, Point Mugu, California, test range to support Test and Evaluation Master Plan- approved Air Warfare/Ship Self-Defense Enterprise testing planned for FY16. 8. Improve the ability of legacy ship self-defense combat system sensor elements to detect threat surrogates used in specific ASCM raid types.
9. Develop adequate and credible target resources for ship self-defense and EW operational testing. 10. Continue to take action on the classified recommendations contained in the March 2011 DOT&E report to Congress on the ship self-defense mission area. 11. Improve the SSDS Mk 2 integration with the Mk 9 Track Illuminators to better support ESSM engagements, as well as preventing the Mk 9 Track Illuminators from contributing to the composite track during certain threat raid types. 12. Develop combat system improvements to increase the likelihood that ESSM and RAM will home on their intended targets. 13. Conduct additional operational testing on the CVN-68 class once the ship is equipped with additional self-defense weapons. This additional testing will determine whether the additional weapons are sufficient to meet the ship’s self-defense requirements. 14. Develop an unmanned, at-sea self-defense test capability that will allow safe demonstration of the self-defense mission of the DDG 51 Flight III Destroyers, ESSM, and AMDR against anti-ship missile threats. 15. Continue to take action on the classified recommendations contained in the November 2012 DOT&E report to Congress on the ship self-defense mission area. • FY13 Recommendations. The Navy should: 1. Continue planning for operational testing of the ship self‑defense mission area during IOT&E of the RAM Block 2 and FOT&E of the SSDS Mk 2 Mod 4 and ESSM on the SDTS. 2. Continue to implement and demonstrate with adequate operational testing the ship self-defense FCLIP improvements. 3. Develop SDTS to permit testing the close-in self-defense capability of ships equipped with AMDR and the DDG 51 Flight III Combat System in FY21. (The DDG 51 and AMDR programs are discussed in a separate section of this report.)
Ship Self-Defense 233
N av y P R O G R A M S
234
N av y P R O G R A M S
Small Tactical Unmanned Aerial System (STUAS) Tier II Executive Summary • The Navy approved Milestone C for the Small Tactical Unmanned Aerial System (STUAS) on May 16, 2013. Operational testing demonstrated that the RQ-21A air vehicle possesses the ability to provide tactical commanders with accurate and timely Intelligence, Surveillance, and Reconnaissance (ISR) coverage. • The Navy delayed IOT&E, scheduled for October 2013, until January 2014 due to delays in the delivery of the low-rate initial production system and the need to conduct additional integrated testing to address identified deficiencies. • The program’s adoption of a test-fix-test philosophy and early involvement of Marines serves as a good model for other programs. • The Navy projects that STUAS will not meet its Mean Flight Hours Between Abort (MFHBA) threshold requirement until the system has achieved 3,300 flight hours. According to current planning documents, this will occur three years after IOT&E. At that point, the program will have purchased 16 of the planned 33 systems. Discussions with the Navy are ongoing to mitigate this. System • Each STUAS consists of five RQ-21A unmanned air vehicles, surface components, and assorted government‑provided equipment. The surface components consist of ground control stations, launch and recovery equipment, datalinks, multi‑mission payloads, and support systems. Government- provided equipment includes vehicles and generators to transport and power ground components and intelligence workstations. • The Marine Corps intends the STUAS with the RQ-21A to have the following capabilities: - Reliability to support an operating tempo of 12 hours on station per day at a sustained rate for 30 days, and the capability for one surge of 24 hours on station per day for a 10-day period during any 30-day cycle - Air vehicle with 10 hours endurance, airspeed up to 80 nautical miles per hour, and a service ceiling of 15,000 feet density altitude - Operating radius of 50 nautical miles
Activity • The Navy conducted Operational Test Period B2 (OT-B2) in accordance with a DOT&E-approved Test and Evaluation Master Plan and test plan in November 2012 to support the Milestone C decision in May 2013. It is notable that OT-B2 occurred two months after the start of the program’s first
- Electro-optical sensor capable of identifying a 1-meter sized object from 3,000 feet altitude; infrared sensor capable of identifying a 3-meter sized object from 3,000 feet altitude - Entire system transportable by CH-53E helicopter • STUAS will replace the Shadow RQ-7 unmanned aerial vehicles (UAVs) currently operated by Marine UAV Squadrons. Mission • Marine Corps commanders will use the STUAS to provide units ashore with a dedicated persistent battlefield ISR capability that will reduce their dependence on higher headquarters for ISR support. • The persistence of the system allows commanders greater coverage of their areas of interest, while providing the capability to concentrate for longer periods of time on a specified target of interest. • In addition to operating from land bases, detachments from Marine Corps UAV Squadrons will embark the requisite personnel and equipment aboard L-class ships and conduct operations in the maritime domain. Major Contractor Insitu, Inc. – Bingen, Washington
integrated test period. The system flew 42.3 hours over the course of 11 flights during OT-B2. Because of ship propulsion problems, sea-based testing was limited to a single flight of 1.8 hours of the 24 hours planned. • The Navy approved Milestone C on May 16, 2013.
STUAS 235
N av y P R O G R A M S • The Navy delayed IOT&E, scheduled for October 2013, until January 2014 due to delays in the delivery of the low-rate initial production system and the need to conduct additional integrated testing to address identified deficiencies. - IOT&E is scheduled to occur as part of a Marine Corps Integrated Training Exercise at the Marine Corps Air Ground Combat Center at Twenty-nine Palms, California. - During IOT&E, STUAS will support Marine Corps ground units preparing to deploy. • The system experienced air vehicle mishaps on September 19, 2012, and January 14, 2013. In the first mishap, the air vehicle experienced a structural failure during launch. The second mishap also occurred during launch. In this case, a loose circuit board pin on the Electronic Control Unit processor contacted the unit’s housing, producing a short circuit that resulted in engine shutdown and a hard landing. Assessment • OT-B2 demonstrated that the RQ-21A possesses the ability to provide tactical commanders with accurate and timely ISR coverage. The program’s adoption of a test-fix-test philosophy and early involvement of Marine operators and maintainers serves as a good model for other programs. • The Navy projects that STUAS will not meet its MFHBA threshold requirement until the system has achieved 3,300 flight hours. According to current planning documents, this will occur three years after IOT&E. At that point, the
236 STUAS
program will have purchased 16 of the planned 33 systems. Discussions with the Navy are ongoing to mitigate this. • The Marine Corps based the MFHBA threshold criteria of 50 hours on the performance of other unmanned systems. It is not readily apparent that the 50-hour threshold does or does not fully support the desired operating tempo and operating and support costs budgeted for system operations. • While the occurrence of mishaps is not uncommon in unmanned systems early in their development, it is noteworthy that both mishaps might be attributable to the manufacturing process. The Navy has taken steps to address quality control during production. Recommendations • Status of Previous Recommendations. This is the first annual report for the program. • FY13 Recommendations. The Navy and Marine Corps should: 1. Conduct a comprehensive review of STUAS reliability versus requirements. The MFHBA threshold criterion of 50 hours should be reviewed to assess how this value supports operational effectiveness and suitability. 2. Increase annual operating hours in order to reach the projected 3,300 flight hours sooner than 2017. This increase in operating tempo would allow the Navy to identify and correct failure modes before committing to buy a significant number of systems.
N av y P R O G R A M S
SSBN Ohio Class Replacement Program Executive Summary • The Ohio Replacement will replace the current Ohio class fleet ballistic missile submarine (SSBN). The Navy is continuing to refine the design and requirements for the Ohio Replacement submarine. • The Navy conducted an Early Operational Assessment (EOA) from September 2012 to July 2013. • Initial results indicate the following: - The modeling and simulation study conducted as part of the EOA had limitations, making the results informative but inconclusive. The Ohio Replacement and Virginia class programs are collaborating to update the model for future analysis. - The EOA identified a few risks to the program achieving operational effectiveness and suitability. These risks are classified. System • The Ohio Replacement Program recapitalizes the aging Ohio class fleet SSBN. • The design of the Ohio Replacement submarines will include: - A new propulsor, a new electric drive system, and a degaussing system, which will provide improved covertness over the Ohio class to ensure the survivability of the platform against potential future threats. - A new nuclear reactor that will not require mid-life refueling. This shortens the required mid-life overhaul period, allowing a fleet of 12 Ohio Replacement submarines to maintain the same at-sea presence as a fleet of 14 legacy Ohio class submarines, which do require refueling. - A new design Common Missile Compartment to host the existing Trident II Life Extension Strategic Weapon System. The Strategic Weapon System includes the Trident II D5 Life Extension missile, launcher, fire control, navigation systems, and associated support systems. - The existing Ohio class basing, maintenance and training infrastructure. Many ship components, such
Activity • From September 2012 to July 2013, the Navy conducted an EOA of the Ohio Replacement Program in accordance with the DOT&E-approved test plan. The assessment consisted of an extensive review of Ohio and Ohio Replacement documentation to identify risks to the Ohio Replacement Program. The assessment also included a modeling and simulation study to compare the survivability of the two submarine classes. • DOT&E will publish a classified EOA Report in 2QFY14.
as communications, sonar, tactical control system, and internal computer networks, will be carried over from other submarine classes to reduce both cost and risk as well as expand commonality across the submarine force. • The Navy plans to procure 12 Ohio Replacement submarines to support U.S. Strategic Command presence requirements. Initial Operating Capability and the first Strategic Patrol will be in FY31. The fielding rate will be one per year. • Ohio Replacement submarines are being designed to have a 42-year service life, a mixed gender crew, and to be in service until the mid-2080s. Mission The Commander, United States Strategic Command will employ Ohio Replacement submarines as the survivable leg of the United States nuclear triad providing an effective sea-based strategic nuclear deterrent. Major Contractor General Dynamics Electric Boat – Groton, Connecticut
Assessment • DOT&E is currently analyzing the data obtained during the EOA. • The modeling and simulation study conducted as part of the EOA had limitations, making the results informative but inconclusive. The Ohio Replacement and Virginia class programs are collaborating to update the model for future analysis. • The EOA identified a few risks to the program achieving operational effectiveness and suitability. The risks are classified.
SSBN Ohio 237
N av y P R O G R A M S Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendation. 1. The Navy should update and accredit the acoustic and threat models for the next operational assessment to reduce modeling and simulation limitations.
238 SSBN Ohio
N av y P R O G R A M S
SSN 774 Virginia Class Submarine Executive Summary • The Navy conducted a Virginia class FOT&E event in FY13 that examined the submarine’s ability to support Naval Special Warfare (NSW) missions using an installed Dry Deck Shelter (DDS). • DOT&E issued a classified report in October 2013 on the results of the FOT&E. DOT&E concluded that: -­ Virginia class submarines are capable of hosting the DDS system. - Virginia class submarines can remain covert during NSW missions in some environments against some threat forces. The Navy’s metrics for assessing this covert capability was a binary probability that cannot reasonably be assessed by testing so it was not used in DOT&E’s assessment. • In May 2013, DOT&E issued a classified report on a combined FOT&E event that occurred in FY11. - The first portion of the report assessed the Virginia class submarine’s ability to operate under-ice and to conduct Anti-Submarine Warfare (ASW) in the Arctic. - The second portion of the report assessed the Virginia class submarine’s susceptibility to detection by passive acoustic arrays. ▪▪ DOT&E concluded that the Virginia class submarine is effective at supporting general operations in the Arctic but remains ineffective at ASW against some targets, which is unchanged from the results of previous testing reported on by DOT&E. ▪▪ DOT&E also concluded that the Virginia class submarines are among the quietest submarines in the world and are difficult to detect with passive acoustic sensors. Like all other classes of U.S. submarines, when operating at high speeds Virginia class submarines become more susceptible to detection by passive acoustic sensors. • DOT&E issued a separate November 2012 classified report on a combined FOT&E event that began in FY11 and extended into FY12. This report assessed the Virginia class submarine’s performance with the Navy’s latest combat system and sonar suite. DOT&E concluded that the modernization of the combat system and sonar suite did not change the performance of the Virginia class submarines for the missions tested. System • The Virginia class submarine is the Navy’s latest fast attack submarine that is capable of targeting, controlling, and launching Mk 48 Advanced Capability torpedoes and Tomahawk cruise missiles.
• The Navy is procuring Virginia class submarines incrementally in a series of blocks. The block strategy is for contracting purposes, not necessarily to support upgrading capabilities. - Block I (hulls 1-4) and Block II (hulls 5-10) ships were built to the initial design of the Virginia class. - Block III (hulls 11-18) ships will include the following enhancements: ▪▪ A Large Aperture Bow array will replace the spherical array in the front of the ship. ▪▪ Two Virginia payload tubes will replace the 12 vertical launch tubes. Each payload tube is capable of storing and launching six Tomahawk land attack missiles used in strike warfare. - The Navy has not designed Block IV and beyond ships. Mission The Operational Commander will employ the Virginia class submarine to conduct open ocean and littoral covert operations in support of the following submarine mission areas: • Strike Warfare • Anti-Submarine Warfare • Intelligence, Surveillance, and Reconnaissance; Indications and Warnings; and Electronic Warfare • Anti-Surface Ship Warfare • Naval Special Warfare • Battle Group Operations Major Contractors • General Dynamics Electric Boat – Groton, Connecticut • Huntington Ingalls Industries, Newport News Shipbuilding – Newport News, Virginia
SSN 774 Virginia 239
N av y P R O G R A M S Activity • In November 2012, DOT&E issued a classified FOT&E report on the modernized Virginia with the Advanced Processor Build (APB) 09 sonar and combat control systems. • In May 2013, DOT&E issued a classified report on Virginia’s ability to conduct operations in the Arctic environment and the submarine’s susceptibility to low-frequency passive acoustic sensors. • During November through December 2012, the Navy conducted developmental and operational tests to assess the ability of the Virginia class submarine to perform NSW missions with a DDS installed. DOT&E issued a classified report in October 2013 on the results of the FOT&E. • The Block III design requires shock testing of the Common Weapons Launcher and the Virginia Payload Tube (VPT) hatch. The VPT hatch shock qualification test series to support the first Block III delivery in August 2014 was scheduled for April 2013. However, the test series is on-hold due to a work stoppage at the Aberdeen Test Center. The Program Office is planning to restart the test series in early 2014. • The Navy is performing a verification and validation of the Transient Shock Analysis (TSA) modeling method used for the design of Virginia class Block III items. The TSA modeling method is scheduled to be accredited in April 2014. • The Navy has planned an update to the Vulnerability Assessment Report to include the Block III modifications for January 2015. Assessment • The October 2013 DOT&E classified report details Virginia’s ability to host NSW missions from a DDS and concluded the following: - Virginia class submarines are capable of hosting the DDS system. - Virginia class submarines can remain covert during NSW missions in some environments against some threat forces. Testing was not sufficient to fully evaluate the covertness of the class during DDS operations against expected threats. DOT&E’s report provided estimates for probability to remain covert based on the data available. Furthermore, the Navy’s primary metric for assessing success in these missions is a binary probability, which is infeasible to measure. - Operational testing was adequate for an assessment of the Virginia class submarine’s effectiveness and suitability for NSW missions using a DDS only against a low-end threat. The Navy’s Commander, Operational Test and Evaluation Force (COTF) did not conduct test execution in accordance with the DOT&E-approved test plan. Specifically, COTF failed to collect positional data from the assigned simulated opposing forces, which limited the ability to assess covertness during these operations. Additionally, the testing did not provide data to address acoustic vulnerabilities during NSW operations using a DDS.
240 SSN 774 Virginia
- The Virginia class submarine is suitable for NSW operations using a DDS; however, the Navy identified shortcomings in the Virginia class in testing. ▪▪ Space limitations onboard the submarines restrict movement to and from the control room, which potentially impedes the ship’s ability to execute damage control procedures in the event a casualty occurs during NSW operations using a DDS. ▪▪ During conditions of low visibility, including nighttime operations, Special Operations Force (SOF) members on the surface may have difficulty seeing the photonics mast of a submerged submarine, which is used to guide the movement of the SOF as they return to the submarine. ▪▪ The Navy made modifications to the SEAL Delivery Vehicle (SDV) Auxiliary Life Support System (ALSS) used in some DDS operations. These modifications allow for increased air pressure and as a result, more available man-hours to support missions. The Virginia class air supply system to pressurize the ALSS does not support operating at the higher pressures. • The May 2013 DOT&E report on Virginia’s operational capabilities in the Arctic and the Virginia’s susceptibility to low-frequency passive acoustic detection concluded that: - Testing was adequate for an assessment of effectiveness and suitability to support general Arctic operations and of the susceptibility of the submarine to detection by passive acoustic sensors. The Navy conducted the testing in accordance with the DOT&E-approved Test and Evaluation Master Plan and test plan but data were not available to conduct the desired quantitative assessment because the Navy did not retain the data following the testing. - Virginia class submarines are effective at supporting general operations in the Arctic but remain ineffective at ASW against some targets, which is unchanged from previous testing reported on by DOT&E. During testing, the Virginia class submarine was hampered with a failure of its sonar system’s TB-29 towed array. The failure of the towed-array affected the submarine’s performance because it provided the longest-range detections of acoustic contacts. However, these arrays are known to be fragile and do frequently fail during operations. - As part of the operational testing, an evaluation of the Depth-Encoded Ice-Keel Avoidance (IKA) mode of the Acoustic Rapid Commercial Off-the-Shelf Insertion (A-RCI) sonar system was included. Ice-keels extend down from the ice canopy above the submarine when operating in regions of the Arctic covered by ice. This Depth-Encoded IKA mode uses active sonar with the intention of providing operators with location, size, and depth of ice-keels so that the submarine can avoid colliding with them. The testing showed that the Depth‑Encoded IKA is fundamentally limited by the precision to which a submarine can know the
N av y P R O G R A M S propagation path of the active sonar and as a result, the Depth-Encoded IKA is unable to achieve the threshold for accuracy established by the Navy. - Virginia class submarines are difficult to detect with low‑frequency passive acoustic sensors. Like all other classes of U.S. submarines, when operating at high speeds Virginia class submarines become more susceptible to detection by passive acoustic sensors. - Virginia class submarines provide less Arctic capability than the Seawolf and improved Los Angeles class submarines. Some regions of the Arctic are characterized by tight vertical clearances between the shallow ocean floor below and the thick ice canopy above. Virginia lacks a hardened sail, and is therefore limited in the thickness of ice through which the submarine can safely surface. - The Virginia class submarine is operationally suitable for supporting general Arctic operations but suffers from some reliability shortcomings: ▪▪ The IKA modes of the A-RCI sonar system reliability require improvement to support extended periods of challenging under-ice operations. After a decade of development and fielding, no hardware or software variant of A-RCI has come close to the Navy’s reliability requirement, which is based on an operational need. More reliable sonar processing hardware is typically brought onboard because of the poor A-RCI reliability. ▪▪ The common methods of removing carbon dioxide and hydrogen waste gas consistently failed during operations in the cold Arctic environment. ▪▪ The handling system for the Virginia class submarine’s Buoyant Cable Antenna, used for communications during operations under the ice canopy, is susceptible to freezing preventing subsequent deployment or retrieval. ▪▪ The Virginia class submarine suffers from excessive condensation in the cold Arctic environment. In general, this is an insulation problem since water vapor will condense on any surface with a temperature below the local dew point. Excessive condensation has the potential to cause problems with electronic systems. • DOT&E’s classified report on Virginia’s modernization FOT&E, issued in November 2012, concluded the following: - Virginia’s operational effectiveness is dependent on the mission conducted. The modernization of the sonar and fire control systems (A-RCI and AN/BYG-1) with the APB 09 software did not change (improve or degrade) the performance of the Virginia class for the missions tested. DOT&E’s assessment of mission effectiveness remains the same for ASW; Intelligence, Surveillance, and Reconnaissance; High-Density Contact Management; situational awareness; and Mine Avoidance. DOT&E’s overall assessment of Information Assurance remains unchanged from IOT&E, although the new software represents an improvement in Information Assurance over previous systems. - Although Virginia was not effective for some of the missions tested, it remains an effective replacement for the
Los Angeles class submarine, providing similar mission performance and improved covertness. - Testing to examine ASW-attack and situational awareness in high-density environments was adequate for the system software that was tested but not adequate for the software version that the Navy fielded. After completion of operational testing, the Navy issued software changes intended to address the severe performance problems observed with the Wide Aperture Array. The Navy has not completed operational testing on the new software, which is fielded on deployed submarines. DOT&E assesses that the late fix of the array’s deficiencies is a result of the Navy’s schedule-driven development processes, which fields new increments without completing adequate developmental testing. - The Navy collected adequate data to assess the suitability of the sonar and fire control systems. Insufficient data were collected to reassess the suitability of Virginia’s hull, mechanical, electrical, or electronic systems; however, these data were not expected to demonstrate significantly different reliability compared to what was observed in IOT&E. Of note, the installation of the new APB 09 on Virginia’s A-RCI sonar system will degrade the reliability of the sonar system on these submarines relative to what was demonstrated in the IOT&E. Recommendations • Status of Previous Recommendations. - The Navy has made progress in addressing 23 of the 30 recommendations contained in the November 2009 classified FOT&E report. Of the seven outstanding recommendations, the significant unclassified recommendations are: 1. Test against a diesel submarine threat surrogate in order to evaluate Virginia’s capability, detectability, and survivability against modern diesel-electric submarines. 2. Conduct an FOT&E to examine Virginia’s susceptibility to airborne ASW threats such as Maritime Patrol Aircraft and helicopters. - The following recommendations from the FY12 Annual Report remain open and the Navy should work to address them in the upcoming fiscal year: 3. Coordinate the Virginia, A-RCI, and AN/BYG-1 Test and Evaluation Master Plans and utilize Undersea Enterprise Capstone documents to facilitate testing efficiencies. 4. Complete the verification, validation, and accreditation of the TSA method used for Virginia class Block III items. 5. Repeat the FOT&E event to determine Virginia’s susceptibility to low-frequency active sonar and the submarine’s ability to conduct Anti-Surface Warfare in a low-frequency active environment. This testing should include a Los Angeles class submarine operating in the same environment to enable comparison with the Virginia class.
SSN 774 Virginia 241
N av y P R O G R A M S • FY13 Recommendations. The Virginia DDS and Arctic reports generated 16 recommendations. The following are unclassified recommendations listed in the October 2013 FOT&E report. The Navy should: 1. Reconsider the metrics used to assess Virginia class submarine’s ability to covertly conduct mass swimmer lockout operations using the DDS. 2. Evaluate the possible acoustic vulnerabilities associated with SDV employment. 3. Seek additional evaluations of Virginia class operations with a DDS to improve understanding of deployment time for operations and operationally evaluate covertness. 4. Confirm that the access to and from the Control Room during DDS operations meet the requirements of the Submarine Safety Program for accessibility and are sufficient to provide for adequate damage control in the event of casualties. 5. The Navy should investigate and implement methods to aid the SOF in identifying the submarine during operations in conditions of low visibility.
242 SSN 774 Virginia
6. Investigate modifying the reducer in the air charging system to allow higher air pressure for the SDV Auxiliary Life Support System in order to provide increased flexibility for SDV missions that can be hosted from Virginia class submarines. 7. Re-evaluate the accuracy requirements for the IKA sonar modes and investigate the calibration of those modes. 8. Continue the reliability improvement program for the TB-29 towed-array or pursue the development of a new array. 9. Improve the reliability of the A-RCI IKA sonar modes. 10. Modify atmosphere control subsystems to operate properly in the freezing waters of the Arctic Ocean. 11. Modify the handling system of the Buoyant Antenna Cable to prevent its freezing in the cold Arctic environment. 12. Continue to collect data on the susceptibility of the Virginia class to low-frequency passive systems and conduct a more quantitative assessment (e.g., determine detection ranges for different ship postures).
N av y P R O G R A M S
Standard Missile-6 (SM-6) Executive Summary • The Navy will not demonstrate the Standard Missile-6 (SM-6) Capability Production Document performance requirement for interoperability until the fielding of the Navy Integrated Fire Control-Counter Air (NIFC-CA) From the Sea (FTS) capability in FY14/15. The Navy expects to demonstrate the maximum range and launch availability Key Performance Parameters during SM-6 FOT&E and Aegis Baseline 9 operational testing in FY14. • The Navy will fire 16 SM-6 missiles during SM-6 FOT&E/Aegis Baseline 9 operational testing and NIFC-CA FTS demonstrations scheduled for FY14/15. These firings will demonstrate SM-6 integration with Aegis Baseline 9 software and SM-6 performance as part of NIFC-CA FTS. • As reported in DOT&E’s May 2013 IOT&E and Live Fire Report, the Navy conducted high-temperature wind tunnel tests of the improved missile uplink/downlink antenna shrouds. - During these tests, the Navy discovered inter-layer delamination in the antenna shroud insulation on three of the five wind tunnel test articles, which questioned the efficacy of the Navy’s previous corrective actions. - Failure review and analysis determined the observed anomaly was not a high risk for aggravating the original removal of insulation material failure mode, as there was no observed delamination or removal of material. - DOT&E will monitor and assess the uplink/downlink antenna shroud reliability issue throughout FOT&E. • The performance deficiency discovered during IOT&E and outlined in the classified SM-6 IOT&E and Live Fire Report remains unresolved and continues to affect DOT&E’s final assessment of effectiveness. - The Navy is assessing several options for a solution, each with varying degrees of complexity. A primary concern is to ensure the solution causes no degradation to the existing SM-6 performance envelope. - The Navy anticipates making a final decision on corrective action by 3QFY14; however, funding for final implementation and testing of the solution remains unresolved.
System • SM-6 is the latest evolution of the Standard Missile family of fleet air defense missiles that incorporates components from two existing Raytheon product lines: - SM-2 Block IV - Advanced Medium-Range Air-to-Air Missile (AMRAAM)
• SM-6 is employed from cruisers and destroyers equipped with Aegis combat systems. • The SM-6 seeker and terminal guidance electronics derive from technology developed in the AMRAAM program. SM-6 retains the legacy Standard Missile semi-active radar homing capability. • SM-6 receives midcourse flight control from the Aegis combat system via ship’s radar; terminal flight control is autonomous via the missile’s active seeker or supported by the Aegis combat system via the ship’s illuminator. Mission • The Joint Force Commander/Strike Group Commander will use SM-6 for air defense against fixed-/rotary-winged targets and anti-ship missiles operating at altitudes ranging from very high to sea-skimming. • The Joint Force Commander will use SM-6 as part of the NIFC-CA FTS operational concept to provide extended‑range, over-the-horizon capability against at-sea and overland threats. Major Contractor Raytheon Missile Systems – Tucson, Arizona
SM-6 243
N av y P R O G R A M S Activity • DOT&E submitted its SM-6 IOT&E and Live Fire Report to Congress in May 2013. • The SM-6 entered full-rate production in FY13 and will achieve Initial Operational Capability in 1QFY14. • The Navy conducted the Flight Test Round (FTR)-25A test mission at White Sands Missile Range, New Mexico, in accordance with a Navy-approved SM-6 developmental test plan. FTR-25 demonstrated the flight reliability of a missile equipped with the Processor Replacement Program computer hardware update that mitigated parts obsolescence. • The Navy conducted SM-6 Live Fire-02 (LF-02) at the Pacific Missile Test Center, Point Mugu, California, in accordance with the Aegis Baseline 9 developmental test plan. LF-02 demonstrated the ability of an Aegis Baseline 9 cruiser, utilizing the SM-6 missile, to engage and intercept a target using targeting data provided by an off-board sensor on the Cooperative Engagement Capability network. DOT&E collected SM-6 flight reliability data during this event. • The Navy conducted SM-6 Live Fire-04 (LF-04) at the Pacific Missile Test Center, Point Mugu, California, in accordance with the Navy-approved plan for NIFC- CA testing. LF-04 demonstrated the ability of an Aegis Baseline 9 cruiser, utilizing the SM-6 missile, to engage and intercept a target using targeting data provided by an off-board sensor on the Cooperative Engagement Capability network. This was the first at-sea demonstration of the NIFC-CA FTS capability. DOT&E collected SM-6 flight reliability data during this event. • In FY14/15, the Navy plans to fire up to 16 SM-6 missiles during SM-6 FOT&E/Aegis Baseline 9 operational testing and NIFC-CA FTS demonstrations. These firings will demonstrate SM-6 integration with Aegis Baseline 9 software and SM-6 performance as part of NIFC-CA FTS. DOT&E will collect SM-6 performance and flight reliability data during these events. • The Navy concluded its Failure Review Board for the Mk 54 Safe-Arm Device anomaly. Assessment • As reported in DOT&E’s May 2013 IOT&E and Live Fire Report, the Navy conducted high-temperature wind tunnel tests of the improved missile uplink/downlink antenna shrouds. These tests discovered inter-layer delamination in the antenna shroud insulation on three of the five wind tunnel test articles, which raised questions regarding the efficacy of the Navy’s previous corrective actions. As there was no observed
244 SM-6
•
•
•
•
•
delamination or removal of material, the Navy’s failure review and analysis determined the insulation inter-layer delamination observed was not a high risk for aggravating the original removal of insulation material failure mode. DOT&E will monitor and assess this reliability issue throughout FOT&E. The Navy Failure Review Board’s analysis of the Mk 54 Safe-Arm Device anomaly, as reported in the IOT&E and Live Fire Report, concluded that the anomalous data observed during live testing was not indicative of a device malfunction and is not expected to affect lethality of the SM-6 missile. The FY13 SM-6 flight tests were all successful. There were no occurrences of the uplink/downlink antenna shroud flight reliability deficiency or other anomalies during these tests. DOT&E and the Navy will continue to collect data on this deficiency throughout FOT&E flight-testing. In the FY13 IOT&E and Live Fire Report, DOT&E assessed SM-6 as suitable. This assessment considered combined data from the IOT&E and developmental/operational flight tests. DOT&E will collect reliability data and assess suitability throughout SM-6 FOT&E testing in FY14/15. The performance deficiency discovered during IOT&E and outlined in the classified SM-6 IOT&E and Live Fire Report remains unresolved and continues to affect DOT&E’s final assessment of effectiveness. The Navy is assessing several options for a solution, each with varying degrees of complexity. A primary concern is to ensure the solution causes no degradation to the existing SM-6 performance envelope. The Navy anticipates making a final decision by 3QFY14; however, funding for final implementation and testing of the solution remains unresolved. The Navy will not demonstrate the SM-6 Capability Production Document performance requirement for interoperability until the fielding of the NIFC-CA FTS capability in FY14/15. The Navy expects to demonstrate the maximum range and launch availability Key Performance Parameters during Aegis Baseline 9 operational testing in FY14.
Recommendations • Status of Previous Recommendations. The Navy is addressing all previous recommendations. • FY13 Recommendation. 1. The Navy should correct the classified performance deficiency discovered during IOT&E and test those corrective actions in flight.
N av y P R O G R A M S
Surface Ship Torpedo Defense (SSTD) System: Torpedo Warning System and Countermeasure Anti- torpedo Torpedo
Executive Summary • The Navy installed a prototype Torpedo Warning System (TWS) and early engineering development model of the Countermeasure Anti-torpedo Torpedo (CAT) aboard USS George H. W. Bush (CVN-77) in March 2013. It demonstrated some capability to detect certain types of threat torpedoes. However, the system has not been fully tested and most TWS and CAT testing to date have been conducted in areas with benign acoustic conditions when compared to the expected threat operating areas. • The Navy’s decision to add an acoustic operator to monitor TWS displays and supplement the automated detection and alerting functions of TWS improved threat detection performance during the November 2013 Quick Reaction Assessment (QRA). However, the test area did not offer the same number of opportunities for false alerts as expected in the threat area, it is not known if the presence of the operator could also reduce the false alert rate. • When properly targeted, the CAT demonstrated a capability to detect and home on some threat surrogates. However, because of safety requirements, the surrogate threat torpedoes and CATs used were operated at depths that were deeper than most threat torpedoes are expected to operate. The Navy’s CAT developmental testing before the QRA focused on predicting the performance in scenarios planned for the QRA. Shallower torpedo scenarios that would force the CAT to track and attack the surrogate threat torpedoes in challenging areas of the water column were not investigated. Therefore, CAT’s ability to neutralize these threats cannot be fully assessed. • The Navy intends to field the prototype TWS and early engineering development model of the CAT in FY14. Additional information on the TWS and CAT performance will be provided in DOT&E’s classified Early Fielding Report in 2QFY14. System • The Surface Ship Torpedo Defense (SSTD) is a system‑of‑systems that includes two new sub-programs: the TWS program (an Acquisition Category III program) and CAT (not an acquisition program until FY16). • TWS is being built as an early warning system to alert on and localize incoming threat torpedoes and consists of three major subsystems: - The Target Acquisition Group consists of a towed acoustic array, tow cable, winch, power supply, and signal
processing equipment. Data from the array and the ship’s radar system are processed into contact tracks and alerts to be forwarded to the Tactical Control Group. The array will eventually be capable of both passive and active sonar operations. - The Tactical Control Group consists of duplicate consoles on the bridge and Combat Direction Center (on CVNs) that displays contacts, issues torpedo alerts to the crew, and automatically develops CAT placement presets using information sent from the Target Acquisition Group. The operator will use this console to manage the threat engagement sequence and command CAT launches. - The Ready Stow Group will consist of the steel cradles housing the CATs. • CAT is a hard-kill countermeasure intended to neutralize threat torpedoes and consists of the following: - The Anti-torpedo Torpedo (ATT) is a 6.75-inch diameter interceptor designed for high-speed and maneuverability to support rapid engagement of the threat torpedo. - The All-Up Round Equipment consists of a nose sabot, ram plate, launch tube, muzzle cover, and breech mechanism to encapsulate and launch the ATT. Mission Commanders of nuclear-powered aircraft carriers and Combat Logistic Force ships will use SSTD to defend against incoming threat torpedoes.
SSTD 245
N av y P R O G R A M S Major Contractors TWS • 3Phoenix – Wake Forest, North Carolina • In-Depth Engineering – Fairfax, Virginia • Pacific Engineering Inc. (PEI) – Lincoln, Nebraska
Activity • The Navy has been working on a hard-kill torpedo defensive system for surface ships for over 10 years, but accelerated the development of TWS and CAT as a result of the March 2010 sinking of the South Korean ship, ROKS Cheonan, and a Navy Fifth Fleet Urgent Operational Needs Statement. The Navy also decided to have the systems protect high-value ships (aircraft carriers and combat logistic ships) rather than destroyers as originally planned. • The Navy conducted early ATT (a previous version of the CAT) warhead testing against select representative torpedo threats in 2002 and 2008. These tests were conducted to gain early insights into the lethality of the ATT and to begin development of a lethality prediction model. • In March 2013, the Navy installed a prototype TWS aboard USS George H. W. Bush (CVN-77). The Navy conducted the following five sea tests of this TWS configuration: - Approximately 24 hours of TWS operations in the Virginia Capes Fleet Operating Areas (VCOAs) during March 2013. During this test, the Navy completed the TWS installation checkouts including functional system operation of the Target Acquisition Group and TWS array deployment. - Approximately 25 hours of TWS operations in the VCOAs during April 2013. During this test, the Navy further exercised the system deployment and collected additional data with the TWS towed array deployed. - Approximately 20 hours of TWS operations with the array deployed and 10 surrogate threat torpedo alertment opportunities in the VCOAs in May 2013. During the test, a barge fired exercise torpedoes at the ship for the TWS to detect and alert the crew. The crew then responded to these alerts by firing CATs to intercept the surrogate threat torpedo. This was the initial integrated test of the TWS and CAT system. - Approximately 58 hours of TWS operations in the VCOAs during August 2013. During this test, the Navy further exercised system employment and collected additional data with the TWS towed array deployed. - Approximately 15 hours of TWS operation with the TWS array deployed and 6 surrogate threat torpedo alertment opportunities in the VCOAs in November 2013. The Navy Program Office enhanced the system for this test (as it will be for the USS George H. W. Bush’s deployment) by adding civilian acoustics specialists to operate TWS and alert the crew of potential threat torpedoes. The Navy conducted this test event as a QRA to support a rapid
246 SSTD
CAT • Pennsylvania State University Applied Research Laboratory – State College, Pennsylvania • Pacific Engineering Inc. (PEI) – Lincoln, Nebraska
fielding assessment of the TWS and CAT system’s ability to defend against threat torpedoes. • The Navy, with the Pennsylvania State University Applied Research Laboratory – State College, Pennsylvania, developed and built CAT engineering development models (designated EDM-2). CAT EDM-2s are planned to be fielded on USS George H. W. Bush. During late FY12 and FY13, the Navy and Pennsylvania State University Applied Research Laboratory conducted contractor and developmental testing of CAT in three configurations at Dabob Bay, Washington, and Nanoose Bay, British Columbia, Canada, acoustic tracking ranges. CAT EDM-2 contractor and developmental testing included: - Twenty-six structured events to develop, analyze, and verify CAT EDM-2 electronics, sonar, and processor (front end) functionality. Ten of the events used the front end of the CAT attached to and propelled by a modified heavyweight torpedo propulsion section. Sixteen events used the front end of the CAT propelled by a rechargeable electric propulsion system. The electric propulsion CAT variant was built as a reusable test asset because of the cost and difficulty in reusing the Stored Chemical Energy Propulsion System (SCEPS) used on the tactical CAT. Aside from the propulsion system, which determines the vehicles’ speed and endurance, the CAT variants are identical. - Six structured CAT EDM-2 events using production representative SCEPS propulsion sections to evaluate performance, maneuverability, and noise characteristics of the tactical CAT. - Twenty-seven structured events to develop the CAT EDM-2’s ability to detect, track, and intercept surrogate threat torpedoes. Six of these events used CAT EDM-2 front ends propelled by heavyweight torpedo back ends; 16 events used electrically-propelled CAT front ends; and 5 events used CAT EDM-2s with the SCEPS propulsion. • In May 2013, the Navy conducted the first integrated TWS and CAT test in the VCOAs aboard the USS George H. W. Bush. The Navy completed seven structured events. During each event, a barge fired a surrogate threat torpedo at the USS George H. W. Bush to allow the TWS system to detect and target the CAT. The USS George H. W. Bush’s crew, with contractor support, engaged the surrogate threat torpedo with an electrically-propelled CAT. • In November 2013, the Navy conducted a QRA aboard the USS George H. W. Bush in the VCOAs. During
N av y P R O G R A M S each event, a surrogate threat torpedo was fired at the USS George H. W. Bush for the TWS system to detect and target. The USS George H. W. Bush’s crew, with contractor support that will accompany the ship on their deployment, engaged the threat torpedo surrogate with a CAT. During the QRA, two representative tactical CATs with SCEPS propulsion were fired; the remaining three CATs used electric propulsion. Analysis of TWS and CAT data is in progress. DOT&E will issue a classified Early Fielding Report on the TWS and CAT in 2QFY14. • The Navy plans to field the TWS system and the CAT EDM-2 with the SCEPS propulsion system when the USS George H. W. Bush deploys in 2014. • The Navy and DOT&E are developing a Test and Evaluation Master Plan (TEMP) for the TWS system. The Navy has not started the CAT system TEMP. Assessment • The prototype TWS and early engineering developmental model CAT installed on USS George H. W. Bush demonstrated some capability to detect certain types of threats. However, the system has not been fully tested and most TWS and CAT testing to date has been conducted in areas with benign acoustic conditions when compared to the expected threat operating areas. • The Navy’s decision to add a highly-trained acoustic operator, to supplement the automated detection and alerting functions of TWS, improved threat detection performance during the QRA. However, the test area did not offer the same number of opportunities for false alerts as expected in the threat area; thus, it is not known if the presence of the operator could also reduce the false alert rate. For safety reasons, the QRA testing was highly structured and allowed the operators to focus on torpedo detections and firing the CAT. Therefore, QRA testing was inadequate to resolve the rate of false alerts or their impact on mission accomplishment. • During developmental testing and the QRA, a properly targeted CAT EDM-2 demonstrated a capability to detect and home on some surrogates torpedoes. However, all of the
surrogate threat torpedoes and CATs were operating deeper than most expected threat torpedoes. During the testing from the USS George H. W. Bush, both the threat surrogate and the CAT were required to operate deeper than either system normally would for safety reasons. Shallower scenarios that would force the CAT to track and attack the surrogate threat torpedo in the challenging areas of the water column were not investigated during the CAT’s contractor or developmental testing. Therefore, these tests cannot be used to assess CAT’s overall ability to neutralize these threats. • The Navy intends to field the prototype TWS and early engineering development model of the CAT in FY14. Additional information on the testing of TWS and CAT performance will be included in DOT&E’s classified Early Fielding Report in 2QFY14. • The ATT warhead tests indicate that the ATT should be lethal against select representative torpedo threats provided that both the CAT’s closest point of approach to the threat torpedo and the CAT’s fuzing occurs within the explosive kill zone. Further test and analysis is required to determine the comprehensive lethal capability of the ATT. Recommendations • Status of Previous Recommendations. This is the first annual report for the TWS and CAT system. • FY13 Recommendations. The Navy should: 1. Develop TEMPs for both the TWS and CAT system and an LFT&E strategy for the ATT lethality as soon as possible. 2. Conduct additional testing in challenging, threat representative environments. 3. Conduct additional CAT testing using operationally realistic threat target profiles closer to the surface to assess the CAT’s terminal homing, attack, and fuzing within the lethality range of the warhead. 4. Retest TWS performance once the sensor is upgraded with an active component, the threat torpedo alertment algorithms are updated, and when a member of the ship’s crew replaces the contractor acoustic specialist.
SSTD 247
N av y P R O G R A M S
248
N av y P R O G R A M S
Surveillance Towed Array Sensor System (SURTASS) and Compact Low Frequency Active (CLFA) Executive Summary • IOT&E for Surveillance Towed Array Sensor System (SURTASS)/Compact Low Frequency Active (CLFA) remains ongoing from September 2012. Due to fiscal constraints and platform availability, the Navy conducted no operational testing in FY13. Testing completed so far is insufficient to assess operational effectiveness and suitability. • DOT&E intends to publish a classified SURTASS/CLFA Early Fielding Report (EFR) in FY14. • Completion of IOT&E is intended in FY14; however, the Navy currently projects that sequestration cuts to SURTASS research, development, test, and evaluation will not support funding Commander, Operational Test and Evaluation Force IOT&E efforts in FY14. System • SURTASS/CLFA is a low-frequency, passive and active, acoustic surveillance system installed on tactical auxiliary general ocean surveillance (T-AGOS) ships as a component of the Integrated Undersea Surveillance System (IUSS). • SURTASS provides passive detection of quiet nuclear and diesel submarines and enables real-time reporting of surveillance information to Anti-Submarine Warfare (ASW) commanders. • CLFA is a low frequency, active sonar system developed to provide an active detection capability of quiet submarines operating in environments that support active sonar propagation. • The system consists of: - A T-AGOS host ship with array-handling equipment - A towed vertical string of active acoustic projectors - A towed horizontal twin line (TL-29A) acoustic array - An integrated common processor (ICP) for processing active and passive acoustic data - A High-Frequency Marine-Mammal Monitoring (HFM3) active sonar used to ensure local water space is free of marine mammals prior to low frequency active transmission - A communications segment to provide connectivity to shore-based Integrated Undersea Surveillance System processing facilities and to fleet ASW commanders
Activity • One engineering development model and two production CLFA systems were available for operation on three of the five Western Pacific-based T-AGOS ships during 2013.
Mission • Maritime Component Commanders employ T-AGOS ships equipped with SURTASS/CLFA systems to provide long‑range active and passive ASW detection, classification, and tracking of submarines in support of Carrier Strike Group and theater ASW operations. • Maritime Component Commanders use SURTASS/CLFA to provide blue force ASW screening and threat submarine localization information to theater ASW commanders to support coordinated prosecution of detected threat submarines. Major Contractors • Overall Integrator: Maritime Surveillance Systems Program Office (PMS 485) • ICP: Lockheed Martin – Manassas, Virginia • CLFA Projectors: BAE – Nashua, New Hampshire • CLFA Handling System: Naval Facilities Engineering Service Center (NAVFAC ESC) (Government Lab) – Port Hueneme, California • HFM3 Active Sonar: Scientific Solutions Incorporated (SSI) – Nashua, New Hampshire • TL-29A Towed Arrays: Lockheed Martin – Syracuse, New York
• In September 2012, the Navy commenced IOT&E, which included SURTASS/CLFA participation in the fleet exercise, Valiant Shield 12, and a dedicated four-day test phase.
SURTASS/CLFA 249
N av y P R O G R A M S - Although testing was conducted in accordance with a DOT&E-approved test plan, target submarine availability limited execution to 4 of 20 planned interaction events and resulted in insufficient data to characterize system performance. - Due to fiscal constraints and platform availability, the Navy did not conduct any operational test events in FY13. - The Navy currently projects that sequestration cuts to SURTASS research, development, test, and evaluation will not support funding Commander, Operational Test and Evaluation Force IOT&E efforts in FY14. • Remaining IOT&E is required to: - Adequately characterize CLFA long-range active detection and localization capability against threat representative submarines. - Assess the ability of the fleet to acquire and prosecute CLFA localizations with ASW-capable assets. - Evaluate SURTASS/CLFA vulnerabilities and protection against cyberspace threats. • DOT&E intends to publish a classified EFR in FY14 due to extended delay in the completion of operational test and system availability to forward-deployed T-AGOS ships. Assessment • Limited IOT&E data demonstrated that the SURTASS/CLFA is capable of detecting submarines at long ranges using both active and passive sonar. Data collected are insufficient to fully characterize the detection capability. • Reliability of HFM3 active sonar during the IOT&E significantly affected the availability of CLFA and contributed
250 SURTASS/CLFA
to insufficient data collection during this event. HFM3 active sonar is required by federal law to mitigate the taking of marine mammals by low-frequency active sonar, but its operation does not affect the capability of CLFA. Having an inoperable HFM3 active sonar would not limit availability or capability of CLFA in wartime. • The fleet did not demonstrate the ability to correlate non‑submarine CLFA detections to real-time surface ship positions during Valiant Shield 12. Failure to exclude surface ship detections coupled with limited ASW-capable assets will not support fleet prosecution of CLFA submarine localizations. • Further assessment of the SURTASS/CLFA will be in DOT&E’s classified EFR. Recommendations • Status of Previous Recommendations. IOT&E of SURTASS/ CLFA was not completed in FY13. The Navy is strongly recommended to complete it as soon as feasible in FY14. • FY13 Recommendations. The Navy should: 1. Improve procedures and training for correlating CLFA non-submarine active detections with real-time surface vessel positions. 2. Evaluate procedures used by SURTASS operators to classify active returns that have submarine characteristics and determine if higher confidence can be assigned to suspected submarine detections. 3. Include an event in the remaining IOT&E that assesses the ability of the fleet to reacquire long-range CLFA localizations with ASW-capable assets.
N av y P R O G R A M S
Tomahawk Missile and Weapon System Executive Summary • As demonstrated during FY13 test flights, the Tomahawk Weapon System (TWS) continues to meet Navy standards for reliability and performance. • The FOT&E Operational Test Launch (OTL) program concluded in 2013. This phase of operational testing ran from 2004 to 2013. Flight testing of Tomahawk All-Up Round changes, stockpile monitoring, emerging deficiencies requiring immediate correction, and hardware obsolescence will be conducted in future developmental tests. • In 2013, DOT&E removed the TWS from operational testing oversight. This decision was based upon TWS history of consistent satisfactory performance over the past nine years in test planning, test execution, and the TWS in meeting reliability and performance requirements. System • The Tomahawk Land Attack Missile is a long-range, land attack cruise missile designed for launch from submarines and surface ships. • There are three fielded variants: a Block III with a conventional unitary warhead, a Block III with a conventional submunitions warhead, and a Block IV with a conventional unitary warhead. Production of Tomahawk Block II and III missiles is complete. • Block IV Tomahawk is in production as the follow-on to the Block III conventional unitary warhead variant. These missiles are produced at lower cost and provide added capability, including the ability to communicate and be redirected to an alternate target during flight. • The TWS also includes the Tomahawk Command and Control System (TC2S) and the shipboard Tactical Tomahawk Weapon Control Systems (TTWCS). The TC2S and TTWCS provide
Activity • In accordance with the DOT&E-approved Test and Evaluation Master Plan and operational test plan, the Navy continued to conduct FOT&E OTLs to verify reliability and performance of Block IV Tomahawk missiles, their associated weapon control systems, and the TC2S. The Navy conducted a total of four Tomahawk missile test launches in FY13. These constitute the final launches in the nine year test series, which completed in FY13. • In 2013, DOT&E removed the TWS from operational testing oversight. This decision was based upon TWS history of consistent satisfactory performance over the past nine years in test planning, test execution, and the TWS in meeting reliability and performance requirements. Flight testing
for command and control, targeting, mission planning, distribution of Tomahawk tactical and strike data, and post‑launch control of Block IV missiles. Mission The Joint Force Commander employs the TWS for long-range, precision strikes against land targets. Major Contractors • Missile element: Raytheon Missile Systems – Tucson, Arizona • Weapon Control System element: Lockheed Martin – Valley Forge, Pennsylvania • Command and Control element: - QinetiQ North America LLC – San Jose, California - Boeing Inc. – St. Louis, Missouri
to evaluate All-Up Round changes, stockpile monitoring, emerging deficiencies requiring immediate correction, and hardware obsolescence will be conducted in future developmental tests. Assessment • The final OTL (OTL-423) of FY12 resulted in a failure when the missile self-terminated. As this test was late in the year, no final failure analysis was available for the FY12 DOT&E Annual Report. - The cause of the flight termination was a missile electrical bus under-voltage initiated by an engine flame out.
Tomahawk 251
N av y P R O G R A M S - Range safety procedures required a chase aircraft to maintain visual contact with the missile at all times. To avoid clouds, the chase pilot directed the missile to climb to a higher altitude resulting in an unplanned departure from the flight plan. As a result, the test missile was 4,000 feet higher than expected when it began its descent towards the target. - The missile engine’s fuel control algorithm was unable to compensate for the pace of descent, resulting in unstable combustion and engine flame out. As the engine flamed out and engine speed decreased, missile electrical bus voltage fell below the minimum, causing the missile to self-terminate. • Pending a flight software update, OES 0056, the Navy has developed and implemented procedures to ensure future tests and operational flight plans avoid the conditions that led to the OTL-423 failure. The OES 0056 software update is in development and planned for end-to-end testing in May 2014.
252 Tomahawk
• As demonstrated during FY13 test flights, the TWS continues to meet Navy standards for reliability and performance. • The OTL program (OT-IIIB/IT-CB) concluded in 2013. - This phase of operational testing ran from 2004 through 2013, consisting of 67 flight tests of Tomahawk Block II/ III/IV missiles. - In later years, only Block III and IV missiles were tested, with the last Block III test being in 2012. - Using cruise reliability data from 45 Block IV tests, the 80 percent confidence interval for cruise reliability spans 11 percent and indicates a true cruise reliability value consistent with operational requirements. Recommendations • Status of Previous Recommendations. All previous recommendations have been addressed. • FY13 Recommendations. None.
N av y P R O G R A M S
Vertical Take-Off and Landing Unmanned Aerial Vehicle (VTUAV) (Fire Scout) Executive Summary • The Navy stopped production of the MQ-8B air vehicle after procuring 30 MQ-8Bs. The program focus now shifts to the MQ-8C air vehicle (also known as the “Endurance Upgrade”) as a Rapid Deployment Capability. The Program Office is considering plans to transition the MQ-8C into the Vertical Take-Off and Landing Unmanned Aerial Vehicle (VTUAV) Program of Record. This will replace the Schweizer 333 (MQ-8B) airframe with the Bell 407 (MQ‑8C). • The Test and Evaluation Master Plan (TEMP) approved in 2007 is outdated and does not contain a clear path to successful development, integration, and testing of the MQ-8B or the MQ-8C-based Fire Scout system. • In August, one Fire Scout system completed a 28-month deployment to the Regional Command North area of operations in Afghanistan. The system flew 5,100 hours supporting U.S. and allied forces. • Operational testing demonstrated that the program successfully integrated the Advanced Precision Kill Weapon System (APKWS) aboard the MQ-8B. Additional sea-based testing is required before the Navy can field a sea-based, weaponized unmanned aerial system in response to the U.S. Naval Forces Central Command request for a Rapid Deployment Capability. • At the end of FY13, the Navy conducted a Military Utility Assessment in support of transitioning control of the majority of the MQ-8Bs to the U.S. Navy’s Fleet Forces Command. Once Fleet Forces Command receives control of the MQ‑8Bs, the Program Office will reduce its involvement in the day‑to‑day activities required to fund, train, equip, and support deployed Fire Scout detachments. System • The Fire Scout is a helicopter-based tactical unmanned aerial system comprised of up to three MQ-8 air vehicles with payloads, a shipboard integrated Ground Control Station with associated Tactical Common Data Link, and the UAV Common Automatic Recovery System.
Activity • Between 2006 and 2013, the VTUAV program flew over 10,000 MQ-8B flight hours. Of the 30 MQ-8B aircraft procured, 5 have been lost and are no longer flyable (one was a maintenance trainer, one was lost during operations in Libya, two were lost to design failures, and one was lost flying into icing conditions).
• The Navy intends the Fire Scout with the MQ-8B airframe to have the following capabilities: - Combat radius – 110 nautical miles - Endurance at combat radius – 3 hours on station - Target Identification – small fast-attack boats at 6 kilometer range - Initial payload consists of the AN/AAQ-22D Bright Star II electro-optical and infrared imaging system with laser designator • The Navy plans to replace the Schweizer 333 (MQ-8B) airframe with the Bell 407 (MQ-8C) airframe. MQ-8B vehicles are planned to deploy on Littoral Combat Ships (LCS) and will be phased out via attrition. Mission Aviation detachments equipped with VTUAVs perform reconnaissance, surveillance, target acquisition, and communications relay missions in support of littoral Anti‑Submarine Warfare, Anti-Surface Warfare, and Mine Warfare operations. System deployments during 2013 provided reconnaissance and surveillance to units conducting combat operations ashore and maritime commanders conducting anti‑piracy operations. Major Contractor Northrop Grumman – San Diego, California
• The Navy has stopped production of the MQ-8B air vehicle after procuring 30 MQ-8Bs. The program focus now shifts to the MQ-8C air vehicle (also known as the “Endurance Upgrade”) as a Rapid Deployment Capability. The Program Office is considering plans to transition the MQ-8C into
VTUAV 253
N av y P R O G R A M S
• •
• •
•
•
the VTUAV Program of Record. This will replace the Schweizer 333 (MQ-8B) airframe with the Bell 407 (MQ-8C). The Navy continues to use the MQ-8B to support development of additional payloads for the Navy and other DoD customers. The Navy is continuing development of the MQ-8C air vehicle in response to a Special Operations Command Joint Universal Operational Needs Statement. - In 2012, the Navy issued a sole source contract to Northrop Grumman for $262.3 Million for 2 developmental aircraft and 6 low-rate initial production aircraft. The Navy plans to conduct a Quick Reaction Assessment in 4QFY14. - The Navy plans to transition the MQ-8C from a Rapid Deployment Capability to the VTUAV Program of Record. The current plan is to procure 96 MQ-8C aircraft; 14 are under contract. In June 2013, the Navy conducted operational testing of the APKWS aboard the MQ-8B at China Lake, California, in accordance with a DOT&E-approved TEMP and test plan. In August 2013, one VTUAV system completed a 28-month deployment to the Regional Command North area of operations within Afghanistan. The system flew just under 5,100 hours supporting U.S. and allied forces. The system has returned to the United States for refurbishment and to support testing, training, and deployments with the Navy. At the end of FY13, the Navy conducted a Military Utility Assessment in support of transitioning control of the majority of the MQ-8Bs to the U.S. Navy’s Fleet Forces Command. Once Fleet Forces Command receives control of the MQ‑8Bs, the Program Office will reduce its involvement in the day‑to‑day activities required to fund, train, equip, and support deployed VTUAV detachments. The Navy has begun efforts to integrate the Telephonics 1700B-Plus Search, Surveillance, Tracking, Imaging and Weather Avoidance Radar System into the MQ-8B air vehicle. The AN/ZPY-4(V)1 is intended to detect and track maritime surface targets and cue the electro‑optical and infrared sensor. Fire Scout continues to deploy aboard the Navy’s Oliver Hazard Perry class of frigates. The MQ-8B system is providing Special Operations Forces some Intelligence, Surveillance, and Reconnaissance capability. Each detachment consists of four MQ-8B air vehicles that support the Navy’s forward presence mission and Special Operations Forces. Frigate deployments will continue into 2015. MQ-8B
254 VTUAV
deployment on LCS will commence in 2014; MQ-8C will deploy on LCS in 2015. • Other VTUAV developmental testing during 2013 focused on software upgrades to correct deficiencies identified during deployment and previous testing and address parts obsolescence. Assessment • The TEMP approved in 2007 is outdated and does not contain a clear path to successful completion of IOT&E. The TEMP does not address the transition from the MQ-8B to the MQ-8C as the VTUAV Program of Record. DOT&E has concerns regarding the scope of operational testing the Navy intends to conduct to support the MQ-8C Milestone C decision. • Operational testing demonstrated that the Navy successfully integrated the APKWS aboard the MQ-8B. MQ-8B operators successfully launched 12 APKWS rockets with 11 rockets hitting the designated targets. The sole miss is attributable to an APKWS guidance system malfunction. • Additional sea-based testing is required before the Navy can field a sea-based, weaponized unmanned aerial system in response to the U.S. Naval Forces Central Command request for a Rapid Deployment Capability. • Analysis of the Military Utility Assessment is not yet complete. DOT&E will make an assessment of the MQ-8B transition when all data are available. Recommendations • Status of Previous Recommendations. The Navy has made satisfactory progress on the FY12 recommendations. Continued frigate deployments have allowed detachments to optimize Tactical Common Data Link performance. While the Navy will not conduct an IOT&E on the MQ-8B air vehicle, the recent MQ-8B Military Utility Assessment will highlight risk areas as the Navy transfers these systems to Fleet Forces Command. • FY13 Recommendations. The Navy should: 1. Place a high priority on ship availability to complete testing of a sea-based, weaponized unmanned aerial system. 2. Update the TEMP to describe operational testing that addresses the transition from the MQ-8B to the MQ-8C as the VTUAV Program of Record.
Air Force Programs
Air Force Programs
A i r F o r c e P RO G R A M S
AC-130J Ghostrider Executive Summary • U.S. Special Operations Command (USSOCOM) is developing AC-130J through the integration of a modular Precision Strike Package (PSP) onto existing MC-130J aircraft. The PSP was previously developed and tested on several AC-130W aircraft since 2009. • Modification of the first aircraft is underway and expected to be complete by the end of FY13 to support first flight in January 2014. • The Test and Evaluation Master Plan (TEMP) submitted for Milestone B requires updates to reflect a new test and evaluation concept for IOT&E and a plan for testing intended future capabilities. • The Live Fire Alternative Test Plan (ATP) will provide the data to assess differences in AC-130J survivability from that of the existing MC-130J aircraft given the changes in AC-130J systems (e.g., addition and location of the PSP), missions, and respective threat environments. The assessment will leverage comparable live fire, developmental, and operational test data from previously assessed C-130 legacy platforms, including the U.S. Marine Corps KC-130J Harvest Hawk. System • The AC-130J is a medium-sized, multi-engine, tactical aircraft with a variety of sensors and weapons for air-to-ground attack. • USSOCOM is developing AC-130J through the integration of a modular PSP onto existing MC-130J aircraft. The AC-130J will retain the ability to be refueled in flight, but it will not retain the external hose-and-drogue pods used to refuel other aircraft. • The PSP provides a weapons suite composed of a 30 mm side- firing chain gun; wing-mounted, GPS-guided Small Diameter Bombs; and Griffin laser-guided missiles mounted internally and launched through the rear cargo door. Future increments of AC-130J may incorporate a side-firing 105 mm howitzer and wing-mounted, laser-guided Hellfire missiles. • The PSP also provides an Intelligence, Surveillance, and Reconnaissance suite composed of two electro-optical/infrared sensor/laser designator pods; a synthetic aperture radar pod; a pilot helmet-mounted cueing system; and multiple video, data, and communication links. All PSP subsystems are controlled from a dual-console Mission Operator Pallet in the cargo bay. • The AC-130J retains all survivability enhancement features found on the HC/MC-130J aircraft. Susceptibility reduction
Activity • DOT&E approved the Milestone B TEMP and the live fire ATP in May 2013. DOT&E developed a test and evaluation concept to guide the Milestone C TEMP update and IOT&E plan.
features include the AN/ALR-56M radar warning receiver, AN/AAR-47(V)2 missile warning system, and AN/ALE- 47 countermeasures dispensing system. Vulnerability reduction features include fuel system protection (fuel tank foam to protect from ullage explosion), redundant flight critical components, and armor for crew and oxygen supply protection. • The AC-130J will replace legacy AC-130H/U aircraft.
Mission The Joint Task Force or Combatant Commander will use: • The AC-130J to provide persistent strike operations, including close air support (CAS), air interdiction, and armed reconnaissance. These operations may also include time- sensitive CAS for troops in contact, helicopter/convoy escort, air base defense, and strike coordination and reconnaissance. • The AC-130J sensor, data, and communications suite to provide battlespace-wide area surveillance and situational awareness; execute non-traditional Intelligence, Surveillance, and Reconnaissance operations; and support combat search and rescue operations.
Major Contractor Lockheed Martin – Bethesda, Maryland
• The Air Force delivered the first MC-130J for conversion to an AC-130J to Eglin AFB in January 2013. Modification of the aircraft is ongoing with first flight planned for January 2014.
AC-130J
255
A i r F o r c e P RO G R A M S • The program conducted a Preliminary Design Review in March 2013 and a Critical Design Review in August 2013. • The Integrated Test Team Working Group conducted its first Certification of Operational Test Readiness review in April 2013. • The LFT&E Integrated Product Team (IPT) held its first meeting in September 2013. The IPT started an initial qualitative assessment of AC-130J survivability based on existing developmental, live fire, and operational test data. The IPT agreed to report on the applicability of prior data while considering any changes in AC-130J concept of operations/employment relative to legacy platforms. Assessment • USSOCOM has been developing and testing three increments of the PSP with increasing capabilities on the AC-130W aircraft since 2009. Experience on the AC-130W will provide risk reduction for development of the AC-130J. However, it is not clear whether the Air Force has collected sufficient reliability data on the AC-130W to augment the limited data to be collected during AC-130J testing. • The Milestone B TEMP and the LFT&E ATP do not include any follow-on testing for intended future capability increments, such as a 105 mm side-firing gun or Hellfire missiles. Future capabilities will be included in the Milestone C TEMP update. • To support the survivability assessment, USSOCOM will develop a list of AC-130J operational/tactical scenarios in projected theaters to include any low-level missions and relevant expected threats. The LFT&E IPT requires operator- defined scenarios for the AC-130J survivability assessment to show traceability from operational scenarios to realistic threats and associated testing and analyses. • Armor requirements and the amount of armor differ significantly between the AC-130U and AC-130J aircraft. The AC-130U armor was designed to provide protection to the aircrew stations, personnel, ammunition, and critical systems against a single 37 mm high-explosive incendiary round at a range of 10,000 feet while the AC-130J’s primary
256 AC-130J
crewmember positions and oxygen supplies should be protected against single 7.62 mm ball projectile at 100 meters (threshold). The Program Office will provide a rationale behind this difference to DOT&E and other members of the LFT&E IPT. The LFT&E IPT will quantify the effects of these changes on the survivability of the AC-130J for realistic threats. • The planned armor layout on the AC-130J does not include the Mission Operator Pallet, which should be considered a “primary crewmember” position and protected in accordance with the associated Force Protection Key Performance Parameter (KPP). • The Survivability KPP states that the AC-130J defensive systems will use spiral development to meet the threshold MC-130J Commando II capabilities established in the draft Commando II Capability Development Document. It will not be possible for the program or DOT&E to evaluate this KPP unless the Commando II capabilities are more explicitly stated. Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. 1. The Program Office should update the TEMP and the ATP to reflect intended future capabilities and related follow-on testing as well as modified IOT&E conditions based on the DOT&E test and evaluation concept. 2. The Program Office should collect and provide DOT&E with all available reliability data on the AC-130W that can augment the suitability evaluation for AC-130J. 3. The survivability evaluation scenarios that USSOCOM will develop should differentiate between the current increment of capabilities and intended employment as well as planned future capability increments and intended employments (e.g., new weapons/defensive systems). 4. The test team should identify the data needed to successfully run and verify the models used in support of the overall survivability assessment.
A i r F o r c e P RO G R A M S
Advanced Extremely High Frequency (AEHF) Satellite Communications System Executive Summary • Advanced Extremely High Frequency (AEHF) satellites 1 and 2 are on orbit. The Air Force launched AEHF satellite 3 on September 18, 2013. The Air Force anticipates the satellite will reach its initial orbital position by January 2014. • Air Force Space Command accepted the Mission Control Segment (MCS) Increment 5 for operational use as the Milstar and AEHF ground segment on August 1, 2013. • MCS Increment 5 provides a capability that is an improvement over the previous MCS Increment 4. MCS Increment 5 can support Low Data Rate and Medium Data Rate communications over a combined constellation of Milstar and AEHF satellites. MCS Increment 5 can support Extended Data Rate (XDR) for command and control and limited XDR tactical communications support. • DOT&E requires a robust Multi-Service Operational Test and Evaluation (MOT&E) planned for 2014 to verify that the AEHF system provides the Initial Operational Capability required by the strategic and tactical users. System • The AEHF system represents the second generation of Extremely High Frequency Satellite Communications capability protected from nuclear effects and jamming activities. • The AEHF system will follow the Milstar program as the protected backbone of the DoD’s integrated military satellite communications architecture. AEHF is expected to increase system throughput capacity by a factor of 10. • The overall AEHF system has three segments: - Space segment – comprised of an integrated constellation of Milstar and AEHF satellites - Mission Control segment – includes fixed and mobile telemetry, tracking, and commanding sites; fixed and transportable communication planning elements; the Test
Activity • The Air Force Operational Test and Evaluation Center (AFOTEC) conducted an Operational Utility Evaluation of the MCS Increment 5 with an integrated test period from September 24, 2012, through November 16, 2012, and a dedicated test period from February 19, 2013, through April 5, 2013. • The Air Force launched AEHF satellite 3 on September 18, 2013. Orbit-raising is progressing as anticipated and the satellite is expected to reach its initial orbit in January 2014.
and Training Simulation Element; and the Operational Support and Sustainment Element - Terminal (or User) segment – includes ground-fixed, ground-mobile, man-portable, transportable, airborne, submarine, and shipboard configurations • The AEHF Operational Requirements Document, dated October 2, 2000, defines the operational AEHF constellation as four interconnected satellites. Mission Combatant Commanders and operational forces worldwide will use the AEHF system to provide secure, responsive, and survivable space-based, strategic, and tactical military communications. Major Contractors • Lockheed Martin Space Systems – Sunnyvale, California • Northrop Grumman – Redondo Beach, California
• Air Force Space Command accepted the MCS Increment 5 for operational use as the Milstar and AEHF ground segment on August 1, 2013, replacing the MCS Increment 4 as the system of record. • AFOTEC has successfully demonstrated pre-test execution events to support the test of AEHF anti-jamming capability during preparations for the 2014 MOT&E. • AFOTEC and the Arnold Engineering Center have successfully conducted pre-test activities that demonstrated
AEHF 257
A i r F o r c e P RO G R A M S their ability to conduct scintillation testing in preparation for the 2014 MOT&E. Scintillation is a rapid fluctuation in radio wave signals that can result from atmospheric effects or from a nuclear detonation. • AFOTEC conducted all testing in accordance with a DOT&E- approved Test and Evaluation Master Plan and test plan. Assessment • AFOTEC’s Operational Utility Evaluation of MCS Increment 5 was limited in scope because the Air Force has not deployed the full AEHF MCS capability; therefore, DOT&E cannot provide conclusions about operational effectiveness and suitability. DOT&E requires a robust MOT&E, planned for 2014, to verify that the AEHF system is operationally effective and suitable and provides the Initial Operational Capability required by the strategic and tactical users. • The AEHF MCS Increment 5 provides a capability that is an improvement over the previous MCS Increment 4. MCS Increment 5 can support Low Data Rate and Medium Data
258 AEHF
Rate communications over a combined constellation of Milstar and AEHF satellites. MCS Increment 5 can support XDR for command and control and limited XDR tactical communications support. • The AEHF MCS Increment 5 can command and control the constellation in a timely and accurate manner. Increment 5 met the system requirements for communication management: five of six requirements for timeliness and three of six requirements for accuracy in communications planning. • The MCS Increment 5 demonstrated a significantly improved Information Assurance posture relative to the previous MCS Increment 4. • The MCS Increment 5 demonstrated improved reliability, dependability, and maintainability relative to the previous MCS Increment 4. Recommendations • Status of Previous Recommendations. The Air Force has made satisfactory progress on all previous recommendations. • FY13 Recommendations. None
A i r F o r c e P RO G R A M S
AIM-120 Advanced Medium-Range Air-to-Air Missile (AMRAAM) Executive Summary • In 2009, key stakeholders, including the Program Office and DOT&E, suspended the progression of AIM-120D operational testing due to four new performance and reliability deficiencies. Raytheon addressed the four deficiencies, and DOT&E signed the revised AIM-120D Test and Evaluation Master Plan (TEMP) and separate, detailed operational test plan on May 25, 2012. • The Air Force completed an Operational Test Readiness Review on May 31, 2012, and certified the AIM-120D Advanced Medium-Range Air-to-Air Missile (AMRAAM) to begin operational testing in June 2012. AIM-120D operational testing consists of multiple live missile shots and captive-carry events. The Air Force and Navy are projected to complete operational testing in FY14. • AIM-120D operational testing was suspended in 2012 after discovery of hardware and software problems. With solutions to the problems, the program was approved to restart OT&E in May 2013. • During operational testing to date, the Air Force has accomplished multiple AIM-120D shots and captive carry missions. • The AMRAAM Electronic Protection Improvement Program (EPIP), a software upgrade to AIM-120C3-C7 variants, is currently in integrated testing under a separate EPIP TEMP that DOT&E approved in April 2012. • Testing previously planned to occur in October and November 2013 has been delayed due to shutdown of the Federal Government and the lack of a Defense Appropriation. System • The AIM-120 AMRAAM is a radar-guided air to-air missile with capability in both the beyond-visual-range and within‑visual-range arenas. A single launch aircraft can engage multiple targets with multiple missiles simultaneously when using AMRAAM. • The latest fielded version, the AIM-120C7, incorporated an upgraded antenna, receiver, signal processor, and new
Activity AIM-120D • The Air Force adequately addressed fixes to four performance and reliability deficiencies that precluded AIM-120D from proceeding to operational testing. They conducted an Operational Test Readiness Review in May 2012 and certified the program to begin operational testing in June 2012.
software algorithms to counter new threats. The use of smaller system components created room for future growth. • The AMRAAM program periodically develops and incorporates phased upgrades. The AMRAAM EPIP is a software upgrade to AIM-120C3-C7. • The AIM-120D is currently in development and the Air Force intends for it to deliver performance improvements beyond the AIM-120C7 through the use of an internal GPS, an enhanced datalink, and new software. Following FOT&E, the contractor will develop a System Improvement Program that will consist of software upgrades to AIM-120D. Mission • The Air Force and Navy, as well as several foreign military forces, use various versions of the AIM-120 AMRAAM to shoot down enemy aircraft. • All U.S. fighter aircraft use the AMRAAM as the primary beyond-visual-range air-to-air weapon. Major Contractors • Raytheon Missile Systems – Tucson, Arizona • Rocket Motor Subcontractors: - Alliant Techsystems (ATK) – Arlington, Virginia - Nammo (Nordic Ammunition Group) – Raufoss, Norway
• AIM-120D operational testing was suspended in 2012 after discovery of new hardware and software problems. After the Air Force fixed the problems, the program resumed OT&E in May 2013. FOT&E is progressing in accordance with the DOT&E-approved TEMP and test plan. • A QF-4 drone crash on July 17, 2013, further delayed operational testing until early September 2013.
AIM-120 AMRAAM 259
A i r F o r c e P RO G R A M S • AIM-120D operational testing consists of multiple live missile shots and captive-carry events. The Services are projected to complete operational testing in FY14. • Testing previously planned to occur in October/ November 2013 has been delayed due to shutdown of the Federal Government and the lack of a Defense Appropriation. AMRAAM EPIP • DOT&E approved the AMRAAM EPIP TEMP on April 13, 2012, after which integrated testing began. Lot Acceptance Test (LAT)/Rocket Motors • Beginning in December 2011, rocket motor Lot Acceptance Test (LAT) performance became unpredictable at low temperatures (-65 degrees Fahrenheit) due to propellant hot spots and case burn-through failures. ATK, the subcontractor who produces the rocket motors, is developing a new propellant with projected availability in FY16. • The Program Office suspended performance-based payments to Raytheon and negotiated restructuring of the AMRAAM delivery schedule after delivery delays reached an unacceptable level. • The Program Office, Raytheon, and AMRAAM safety communities coordinated to certify Nammo to be an approved alternative rocket motor supplier. As of October 2013, Nammo had manufactured 1,000 motors in their role as the sole source provider for new production motors. • The Program Office has suspended performance-based payments to ATK until resolution of the shortage of rocket motors due to unacceptable LAT performance.
260 AIM-120 AMRAAM
Assessment • The Air Force originally planned for AIM-120D to begin operational testing in 2008; it is now approximately four years behind schedule. • Since the start of operational testing, the Air Force has executed multiple live missile shots. Captive-carry performance has exceeded the interim Mean Time Between Failure requirement and is approaching the mature requirement of 450 hours. There are insufficient data to quantify free flight reliability with confidence. • The shortage in rocket motors due to unacceptable LAT performance should not significantly affect AIM-120D testing, but it has created a backlog in production. After Insensitive Munitions certification in October 2012, Nammo became the only AIM-120D and AIM-120C7 rocket motor producer for the foreseeable future. The government and Raytheon are still reviewing a path forward for ATK production. Recommendations • Status of Previous Recommendations. The Air Force satisfactorily addressed the previous recommendations. • FY13 Recommendation. 1. The Program Office should finalize and submit a test strategy and AIM-120D TEMP annex for DOT&E review and approval to ensure System Improvement Program 1 completion within one year after operational testing is completed.
A i r F o r c e P RO G R A M S
Air Operations Center – Weapon System (AOC-WS) Executive Summary • The Air Operations Center – Weapon System (AOC-WS) 10.1 is a system-of-systems that contains numerous third-party software applications, including the Global Command and Control System – Joint (GCCS-J), Theater Battle Management Core Systems – Force Level (TBMCS-FL), Master Air Attack Plan Toolkit (MAAPTK), and Joint Automated Deep Operations Coordination System (JADOCS). • The Air Force tests AOC-WS 10.1 during a three-phase Recurring Event (RE) test cycle, which includes event-based test periods primarily focused on software upgrades. The software upgrades and associated test event are designated using similar terms; for example, AOC-WS 10.1 RE11 is the software upgrade tested during RE11. - Phase 1 developmental testing is conducted at the Combined Air Operations Center – Experimental (CAOC-X) at Langley AFB, Virginia. - Phase 2 operational testing is conducted to assess effectiveness at CAOC-X. - Phase 3 operational testing is conducted at a fielded site to assess suitability. • In March 2013, the Air Force delivered its final report on RE11 that included the results of Phase 3 operational testing at 613 AOC, Hickam AFB, Hawaii. • RE11 results demonstrated operators can successfully execute all critical missions and produce threshold or larger-sized target lists and Air Tasking Orders on schedule, although significant workarounds are required to avoid errors in products due to the existing Category 1 (CAT I) deficiencies. AOC-WS 10.1 RE11 provided a significant improvement to AOCs both in internal functionality and in interoperability with Combatant Commands. • AOC-WS 10.1 RE11, based on RE11 Phase 3 testing, was not operationally suitable. It could not be built, configured, or maintained adequately at an operational site. Additionally, the RE11 training schedule did not prepare operators across multiple AOC divisions to complete tasks using AOC‑WS 10.1 RE11, and it did not provide system administrators sufficient instruction in maintenance and troubleshooting of an operational system. • Air Combat Command (ACC) conducted a thorough analysis of the AOC-WS 10.1 RE11 outstanding CAT I deficiencies and decided to accept the risk of fielding it to meet critical operational needs, while maintaining the expectation that the AOC-WS Program Office will fix those deficiencies in an expeditious manner. • In August 2013, the Air Force conducted Phase 2 operational testing of AOC-WS 10.1 RE12 at CAOC-X, Langley AFB, Virginia. The test demonstrated that AOC-WS 10.1 RE12 enables operators to successfully execute all critical missions and meet threshold requirements. AOC-WS 10.1 RE12
is proceeding to Phase 3 of operational testing during November to December 2013 at 612 AOC, Davis-Monthan AFB, Arizona, for an assessment of operational suitability. System • The AOC-WS is the senior command and control element of the U.S. Air Force’s Theater Air Control System and provides operational-level command and control of air, space, and cyberspace operations, as well as joint and combined air, space, and cyberspace operations. Capabilities include command and control of joint theater air and missile defense; time-sensitive targeting; and Intelligence, Surveillance, and Reconnaissance management. • The AOC-WS 10.1 (AN/USQ-163 Falconer) is a system- of-systems that contains numerous third- party- developed software applications and commercial off-the-shelf products. Each third-party system integrated into the AOC‑WS provides its own programmatic documentation. • The AOC-WS consists of: - Commercial off-the-shelf hardware - Separate third-party software applications GCCS-J, TBMCS-FL, MAAPTK, and JADOCS, from which the AOC-WS draws its capabilities - Additional third-party systems that accept, process, correlate, and fuse command and control data from multiple sources and share them through multiple communications systems • AOC-WS 10.1 operates on several different local area networks (LANs), including Secret Internet Protocol Router Network, Joint Worldwide Intelligence Communications System, and a coalition LAN, when required. The LANs connect the core operating system and primary applications to joint and coalition partners supporting the applicable area of operation. Users
AOC-WS 261
A i r F o r c e P RO G R A M S can access web-based applications through the Defense Information Systems Network. • The Air Force tests AOC-WS 10.1 software upgrades during REs. The Air Force refers to each software upgrade by the event during which it was tested. For example, AOC-WS 10.1 RE11 is the software upgrade tested during RE11. • The future AOC-WS 10.2 delivers a modernized, integrated, and automated approach to AOC-WS operations. Mission The Commander, Air Force Forces, or the Joint/Combined Forces Air Component Commander use the AOC-WS to exercise control
Activity • The Air Force uses a three-phase RE test cycle for major AOC‑WS 10.1 upgrades, along with lower-level testing events to sustain interoperability and Information Assurance (IA), and to provide low-risk upgrades to third-party systems as required. - Phase 1 developmental testing is conducted at CAOC-X, Langley AFB, Virginia. - Phase 2 operational testing is conducted to assess effectiveness at CAOC-X. - Phase 3 operational testing is conducted at a fielded site to assess suitability. • The Air Force completed its report on RE11 in March 2013, which included results from Phase 3 testing at 613 AOC, Hickam AFB, Hawaii, from July through September 2012. This testing focused on the ability of the install team to correctly upgrade and configure the AOC from legacy AOC‑WS 10.1 RE10 to AOC-WS 10.1 RE11 and perform backup and recovery actions on AOC-WS 10.1 RE11. • DOT&E observed and reported on RE11 testing, both at CAOC-X and at 613 AOC. The data from this phase of testing form the basis for DOT&E’s assessment of AOC-WS 10.1 RE11’s operational suitability. • The Air Force conducted operational testing of AOC-WS 10.1 RE12 in August 2013. AOC-WS 10.1 RE12 incorporated Defense Information Systems Agency upgrades to GCCS-J, updates to other third-party applications, and improvements to the system cybersecurity posture. • The Air Force conducted all RE11 and RE12 testing in accordance with the DOT&E-approved test plans. Assessment • The Air Force adequately tested AOC-WS 10.1 RE11 through a combination of developmental and operational testing; however, there were significant known limitations to IA and Reliability, Availability, and Maintainability (RAM) data collection. The DOT&E-approved test plan anticipated the lack of RAM data, so the Air Force adopted a mitigation strategy in which they will collect and provide the required
262 AOC-WS
of joint (or combined) air forces including planning, directing, and assessing air, space, and cyberspace operations to meet operational objectives and guidance. An operational AOC is fundamental in enabling centralized command and decentralized execution of a theater air campaign. Major Contractors • AOC-WS 10.1 Production Center: Jacobs Technology Inc., Engineering and Technology Acquisition Support Services – Hampton, Virginia • AOC-WS 10.2 Modernization: Northrop Grumman – Hampton, Virginia
•
•
•
•
data from fielded sites, allowing DOT&E to refine the assessment results based on the ongoing analysis. Following the completion of Phase 3 testing at 613 AOC, ACC conducted a thorough analysis of the 12 outstanding CAT I Urgent deficiencies and decided to accept the risk of fielding AOC-WS 10.1 RE11 to meet critical operational needs, while maintaining the expectation that the AOC-WS Program Office will fix unresolved CAT I deficiencies in an expeditious manner. Of the 12 CAT I deficiencies, 4 affected operational effectiveness, 7 affected operational suitability (2 were subsequently closed), and 1 affected IA. AOC-WS 10.1 RE11 has the capability to produce the primary products necessary to meet the established AOC battle rhythm at threshold levels. AOC-WS 10.1 RE11 demonstrated interoperability with other mission-critical systems and provides a significant improvement to AOCs in both internal functionality as well as the ability to interoperate with respective Combatant Commands. However, significant workarounds are required to avoid errors in products due to the existing CAT I deficiencies. AOC-WS 10.1 RE11 is not operationally suitable. Phase 3 testing showed that the system could not be built, configured, or maintained adequately at an operational site. All seven suitability-related CAT I deficiencies describe inadequacies in documentation of required plans and procedures for fielding and maintaining the system. Additionally, the RE11 training schedule did not prepare operators across multiple AOC divisions to complete tasks using AOC-WS 10.1 RE11, and it did not provide system administrators sufficient instruction in maintenance and troubleshooting of the operational system. The AOC-WS 10.1 RE11 test article and associated documentation that entered OT&E was the direct output of the developmental test-fix-test cycle. Time constraints precluded entering OT&E with a 'clean rebuild' of the test article and a cohesive consolidation of the documentation that incorporated all the supplements (software and configuration modifications) used to 'fix' the previously discovered problems.
A i r F o r c e P RO G R A M S • AOC-WS 10.1 RE11 has a valid “Authority to Operate” through November 2015. AOC-WS 10.1 RE11 has yet to be assessed completely for IA. Since AOC-WS 10.1 RE12 and recurring periodic software patches have significantly improved the cybersecurity posture of the system, the Air Force should complete IA testing on AOC-WS 10.1 RE12 by a representative, emulated adversary (a DoD cyber Red Team) in an operationally realistic environment, preferably during a major command post exercise. • The duration and nature of RE11 test events provided insufficient time to allow DOT&E to accurately assess RAM under operationally realistic system usage. Additional data must be collected at operational sites to assess the effects of RAM on AOC mission operations. • Phase 2 of RE12 testing successfully validated the closure of 11 of the 12 open CAT I deficiencies planned for remediation and demonstrated the ability of AOC-WS 10.1 RE12 to execute all critical missions and produce threshold or larger‑sized target lists and Air Tasking Orders on schedule. • The 605th Test and Evaluation Squadron recommended the Air Force proceed to field AOC-WS 10.1 RE12. This fielding will occur during Phase 3 testing (November to December 2013) at the first operational site, 612 AOC, Davis-Monthan AFB, Arizona. Phase 3 will assess operational suitability, contingent upon resolving the twelfth CAT I deficiency related to executing the build process without needing to rely on extensive high-level help desk support. This CAT I deficiency was successfully resolved with a successful regression build (rebuilding the entire AOC-WS software increment from a clean system, rather than on top of an existing software increment) at the 46th Test Squadron prior to proceeding to Phase 3 testing. • The AOC-WS 10.1 RE12 test article and associated documentation that entered OT&E was also the direct output of the developmental test-fix-test cycle. Extended developmental test and evaluation efforts ensured that this test article was based on a “clean rebuild” of the AOC-WS 10.1 RE11 baseline first followed by the AOC-WS 10.1 RE12 build, consistent with the plan for fielding to operational sites. • The key to successful testing and fielding of AOC-WS 10.1 RE11 and AOC-WS 10.1 RE12 continues to be closer collaboration between the AOC-WS Program Office and the Defense Information Systems Agency to increase the
likelihood that GCCS-J meets the operational needs of the AOCs. AOC-WS tester involvement in GCCS-J testing continues to identify critical problems early for corrective action. Recommendations • Status of Previous Recommendations. The Air Force has made progress to adequately address previous recommendations. Over the past year, the Air Force has increased its efforts with the two long-term FY11 recommendations (below), and this engagement needs to continue. Additionally, the Joint Staff has not yet responded to the FY12 recommendation for a systems integration group for command and control systems. 1. Coordinate with third-party programs to ensure that critical AOC-WS third-party systems (such as GCCS-J) have testable requirements that meet AOC-WS requirements. The requirements should be vetted within the appropriate user and program communities for schedule and funding priority. 2. Ensure the AOC-WS users and test community continue to actively participate in GCCS-J developmental and operational testing and collaborate to develop a capability to adequately test GCCS-J to AOC-WS threshold stress levels. • FY13 Recommendations. The Air Force should: 1. Continue to improve the process of build documentation, production, and validation to include assessing the utility of conducting regression builds following Phase 2 testing at CAOC-X prior to conducting subsequent builds. Conduct an assessment of operational risk to the AOC warfighting mission using DoD cyber Red Teams in an operationally realistic environment (at an operational AOC where AOC-WS 10.1 RE12 is fielded), consistent with DOT&E IA procedures. 2. Require operational AOC sites to collect and report all significant RAM data to the Program Office, assess the data for needed system improvements, and report on RAM improvement efforts monthly to the Configuration Review Board. DOT&E will continue to review RAM data periodically and adjust our findings in accordance with this analysis.
AOC-WS 263
A i r F o r c e P RO G R A M S
264
A i r F o r c e P RO G R A M S
Battle Control System – Fixed (BCS-F) Executive Summary • The Air Force completed IOT&E and a Force Development Evaluation (FDE) on the Battle Control System – Fixed (BCS-F) Increment 3, Release 3.2 (R3.2) at all U.S. air defense sites in September 2012. R3.2 was not fielded due to a critical software deficiency of not consistently sending air track information via Link 16. R3.2 presented some Information Assurance (IA) improvements and achieved a three-year Authority to Operate. • Air Combat Command (ACC) completed an FDE on BCS-F Increment 3, Release 3.2.0.1 (R3.2.0.1) and fielded it at all U.S. air defense sites in November and December 2012. R3.2.0.1 corrected the critical deficiency that prevented the fielding of R3.2. • The Air Force completed developmental testing of R.3.2.2 in August 2013 and Air Force Operational Test and Evaluation Center (AFOTEC) is scheduled to complete the FOT&E in December 2013; however, testing previously planned to occur in October/November 2013 has been delayed due to shutdown of the Federal Government and the lack of a Defense Appropriation. System • BCS-F is the tactical air battle management command and control system for the two continental U.S. North American Aerospace Defense Command (NORAD) air defense sectors, as well as the Hawaii and Alaska Regional Air Operations Centers. The system utilizes commercial off-the-shelf hardware within an open-architecture software configuration, and operates within the NORAD and Pacific Command air defense architecture. The system is employed by the U.S. and Canada. • The R3.2 upgrade includes the following system enhancements: - Improved tactical datalinks with additional Link 16 and Link 11 message types that enable the operators to better digitally control fighters, send amplifying intelligence information, and create a more comprehensive air picture - Air Tasking Order and Airspace Control Order integration with Theater Battle Management Core Systems data sources that enables the operators to view the most current Air Tasking Order/Airspace Control Order and correlate the information with military aircraft - Adaptation data modification tools that enable system administrators to field changes to system adaptation files and to perform error checks with greater fidelity than R3.1 - System control manager interface improvements that enable the system administrator improved system performance monitoring and diagnostics
- Global Area Reference System coordinate conversion tool that facilitates a NORAD interface with global search and rescue efforts by using a common set of coordinates - Remote Gateway Manager control through the virtual network computing interface that provides the operators a complete picture of the available datalinks and the flexibility to access link information from an operator workstation - Auxiliary server for offline training and support capabilities at the U.S. air defense sectors - Improved system capacities from 10,300 to 15,000 system tracks to support single sector continental U.S. operations • The R3.2.2 upgrade includes the following enhancements: - Ability to operate with mandatory International Civil Aviation Organization flight plan changes - Addition of external firewall/intrusion detection system sensor - Implementation of remote administrative management and log server capabilities - Audible and visual alert capabilities on the Computer Network Defense components - New network switch to support the IA-Demilitarized Zone architecture Mission • NORAD and U.S. Pacific Command Commanders use BCS-F to execute command and control and air battle management in support of air sovereignty and air defense missions for North American Homeland Defense and Pacific Command air defense.
BCS-F 265
A i r F o r c e P RO G R A M S • Air defense operators employ BCS-F to conduct surveillance, identification, and control of U.S. sovereign airspace and control air defense assets, including fighters, to intercept and identify potential air threats to U.S. airspace.
Activity • The Air Force completed IOT&E and an FDE on the BCS-F Increment 3, R3.2 at all U.S. air defense sites in September 2012. R3.2 was not fielded due to a critical software deficiency of not consistently sending air track information via Link 16. R3.2 presented some IA improvements and achieved a three-year Authority to Operate. • ACC completed an FDE of R3.2.0.1 at the System Support Facility at Tyndall AFB, Western Air Defense Sector, Eastern Air Defense Sector, Alaska Regional Air Operations Center, and the Hawaii Regional Air Operations Center in December 2012. • The Air Force fielded R3.2.0.1 to all U.S. air defense sectors in December 2012. Canada fielded the release in March 2013. • The Air Force conducted developmental testing on R3.2.2 at the System Support Facility at Tyndall AFB from January through August 2013. Additionally, the Air Force accomplished an IA certification test at this time. • In September 2013, AFOTEC began FOT&E on R3.2.2 at the System Support Facility. Testing previously planned to occur in October/November 2013 has been delayed due to shutdown of the Federal Government and the lack of a Defense Appropriation. • AFOTEC and ACC conducted operational testing in accordance with a DOT&E-approved Test and Evaluation Master Plan and test plan. Assessment DOT&E analyses of R3.2.0.1 concluded: • R3.2.0.1 is operationally effective and suitable but with significant limitations. • R3.2.0.1 resolved the R3.2 critical software deficiency of not consistently sending air tracks via Link 16. During 190 hours of testing, BCS-F 3.2.0.1 transmitted 100 percent of track information via Link 16 to other sites including the Joint Air Defense Operations Center at Joint Base Anacostia Bolling, Washington, D.C. • There were several deficiencies associated with battle management that limited R3.2.0.1 operational effectiveness. Operator workarounds mitigated these deficiencies to an acceptable level.
266 BCS-F
Major Contractor Thales-Raytheon Systems – Fullerton, California
• R3.2.0.1 was assessed as operationally suitable, although ACC did not collect sufficient operational test data to demonstrate the availability and reliability requirements with statistical confidence. During 190 hours of testing, R3.2.0.1 did not experience any critical failures or downtime. Additionally, as of September 30, 2013, the system has operated at all four U.S. air defense sector sites without a critical failure since fielding in December 2012. This equates to over 7,200 hours at each site without a critical failure. The system requirement for Mean Time Between Critical Failure is greater than or equal to 10,000 hours. • While R3.2.0.1 is operationally suitable, technical documentation and training for the system administrators was deficient. • R3.2.0.1 remains deficient in all IA assessment areas. The system is poorly equipped to detect, protect, react, and restore/ recover from attacks by current cyber threats. R3.2.2 is designed to resolve many critical IA deficiencies. Operational testing on R3.2.2 is scheduled to conclude in December 2013. Recommendations • Status of Previous Recommendations. The Air Force satisfactorily addressed all but three of the previous recommendations. The Air Force still needs to: 1. Correct and formalize all BCS-F Increment 3 system documentation and training deficiencies. 2. Develop a plan for remote workstation management to include sustainment, training, documentation, and IA compliance. 3. Upgrade the System Support Facility to support a more robust BCS-F developmental and operational testing capability in order to minimize the impact of overall testing at the operational sites. • FY13 Recommendation. 1. The Air Force should continue to track and correct IA deficiencies.
A i r F o r c e P RO G R A M S
CV-22 Osprey Executive Summary • Air Force Special Operations Command (AFSOC) tested the upgraded Suite of Integrated Radio Frequency Countermeasures (SIRFC) hardware and software version 7.0 from January through July 2012 and conducted an operational test of the GAU-21 .50 caliber Ramp‑Mounted Weapon System (RMWS) in February through May 2013. They also evaluated the reliability of the upgraded Icing Protection System (IPS) during flights into actual icing conditions during January 2013. AFSOC evaluated relocated communications antennas from October through December 2012. • In spite of shortcomings, operational testing indicates that current SIRFC performance, combined with appropriate tactics, techniques, and procedures, results in CV-22 survivability against most expected radar threat systems. • The GAU-21 .50 caliber RMWS is more reliable than the previously fielded GAU-18 RMWS. • The CV-22 IPS has improved since the 2008 IOT&E. • During the 2012 radio antenna relocation testing, the CV-22 communicated with ground troops at distances from 0.5 to 25 nautical miles (nm) and with another aircraft at distances from 0.5 to 120 nm. System • There are two variants of the V-22: the Marine Corps MV-22 and the Air Force/U.S. Special Operations Command CV-22. The air vehicles for Air Force and Marine Corps missions are nearly identical, with common subsystems and military components sustainable by each Service’s logistics system. • The CV-22 is the replacement for aging Special Forces MH-53 helicopters. It is a tilt-rotor aircraft capable of conventional fixed-wing flight and vertical take-off and landing over the range of Special Operations missions. • The speed and range of the CV-22 enable support for Special Operations missions that were not possible with legacy rotary-or fixed-wing aircraft. • The CV-22 can carry 18 combat-ready Special Operators 538 nm and return. It can self-deploy up to 2,100 nm with one aerial refueling.
Activity • To address 2008 CV-22 IOT&E deficiencies with SIRFC, AFSOC tested upgraded SIRFC hardware and software version 7.0 from January from July 2012. • AFSOC evaluated the relocation of CV-22 communication antennas from October through December 2012. The antennas were relocated to address limited operating range and
• The CV-22 will augment Air Force Special Operations MC-130 aircraft. It has terrain-following/terrain-avoidance radar, an advanced multi-frequency communication suite, and a more robust electronic defense suite. • Future capabilities will include engine sub-assembly upgrades, strategic refueling capability, and various fixes to deficiencies identified during IOT&E. • As of August 13, 2013, 34 of 50 CV-22 aircraft have been fielded. Mission Air Force squadrons equipped with the CV-22 will provide high- speed, long-range insertion and extraction of Special Operations Forces to and from high-threat objectives. Major Contractors Bell-Boeing Joint Venture: • Bell Helicopter – Amarillo, Texas • The Boeing Company – Ridley Township, Pennsylvania
inadequate reliability with the CV-22 radios used during the 2008 IOT&E. • AFSOC evaluated the reliability of the upgraded IPS. CV-22 pilots flew 21 sorties totaling 73 flight hours searching for icing conditions, with approximately 26 hours in light-icing conditions from January 10 – 29, 2013.
CV-22 267
A i r F o r c e P RO G R A M S • AFSOC conducted an operational test (in accordance with an informal oversight arrangement with DOT&E) of the GAU‑21 .50 caliber RMWS from February through May 2013. The GAU-21 was fired from the ramp following dust-out landings at test ranges and from aircraft deployed to Afghanistan. Assessment • Block 5 SIRFC shortfalls during the 2008 CV-22 IOT&E included inaccurate and late threat situational awareness, limited countermeasure effectiveness against some threat systems, and a high rate of reliability failures. Since IOT&E, the Air Force upgraded CV-22 SIRFC with new, higher-power transmitters, cabling, radio-frequency switches, antennas, and Block 7 operational flight software. The DOT&E FY12 CV- 22 Annual Report included partial analysis of FY12 testing. Completed analysis shows: - Block 7 SIRFC exhibited significant improvement in threat situational awareness displayed and some improvement in reliability. The countermeasure dispenser does not function properly in automatic mode, requiring manual dispense of countermeasures. - The Block 7 electronic countermeasures performed no better than the Block 5 electronic countermeasures. The system is still subject to software failures requiring reboots, during which the aircraft may be more vulnerable to threat systems. In spite of these shortcomings, operational testing indicates that current SIRFC performance combined with appropriate tactics, techniques, and procedures results in CV-22 survivability against most expected radar threat systems. • CV-22 radio communications with ground forces during the 2008 IOT&E were not effective beyond approximately 5 nm and frequently failed to establish radio contact with the ground troops within 0.5 nm. During the 1QFY13 radio antenna relocation testing, the CV-22 communicated with ground troops at distances from 0.5 to 25 nm and with another aircraft
268 CV-22
at distances from 0.5 to 120 nm. The test was limited to a small subset of operational and atmospheric conditions. • The CV-22 IPS has improved since the 2008 IOT&E. During the IOT&E, the IPS frequently failed the ground built-in test checks and two failure modes led to damage to the aircraft. This damage resulted in restrictions on V-22 flight in icing conditions. Testing in 2013 demonstrated that the probability that the upgraded IPS can operate for 45 minutes in light icing conditions without a failure is 87 percent (80 percent confidence interval: 77 to 92 percent). This improvement expands the operational envelope for the CV-22 to include deliberate operations in light icing, if required. • The GAU-21 .50 caliber RMWS is more reliable than the previously fielded GAU-18 RMWS. During testing in a dust-out landing, the GAU-18 was able to fire only 16 rounds, jamming 3 times before the gun stopped firing. The overall stoppage rate of the GAU-21 is approximately 2,000 rounds between stoppages, whether in dust-out or dust-free landings. The aircrew noted that during testing, none of the GAU-21 stoppages appeared to have been caused by the accumulation of dust and debris. Recommendations • Status of Previous Recommendations. The program has addressed previous recommendations with the exception of battle damage repair and strategic refueling capability. • FY13 Recommendations. 1. The Services should address SIRFC deficiencies and demonstrate improved performance and reliability in future operational testing. Meanwhile, crews should combine the enhanced situational awareness provided by Block 7 SIRFC with appropriate tactics, techniques, and procedures to defeat threat systems. 2. AFSOC should conduct future radio communications testing in the context of end-to-end operational missions in a variety of operational and atmospheric conditions.
A i r F o r c e P RO G R A M S
Defense Enterprise Accounting and Management System (DEAMS) Executive Summary • The Air Force Operational Test and Evaluation Center (AFOTEC) began a second Operational Assessment (OA-2) of the Defense Enterprise Accounting and Management System (DEAMS) Release 2.2 in August 2013. • The Air Force is conducting OA-2 in two phases. During Phase I, DOT&E sent teams to all of the sites where DEAMS is currently fielded (Headquarters U.S. Transportation Command (HQ USTRANSCOM) and Headquarters Air Mobility Command (HQ AMC), both located at Scott AFB, Illinois; the Defense Finance and Accounting Service (DFAS) in Limestone, Maine; and base operations at McConnell AFB, Kansas). The purpose of Phase 1 was to make an initial assessment of DEAMS Release 2.2 as deployed to the current user base. Prior to OA-1, the DEAMS program claimed that all software defects were resolved, but OA-1 then found 200+ software defects. The Milestone Decision Authority (MDA) asked DOT&E to determine if DEAMS had made progress since OA-1 on its ability to predict the quality of the software it fielded. Phase 1 completed in September 2013. • DOT&E’s Phase 1 observations were consistent with DEAMS program assertions of improved software management. Based on its initial assessment, DOT&E concurred with the MDA’s decision to release DEAMS to four new AMC bases, provided that updated training material based on current user feedback is developed and administered to new users. Extended on-site technical support must also be provided to the new bases. • OA-2 Phase 2 began on September 26, 2013, and will continue through January 2014. OA-2 Phase 2 is a second look at the testing performed in OA-1. It is occurring at the previously fielded bases and four new bases (Dover, Grand Forks, and Little Rock AFBs and Pope Field). • An Air Force Blue Team from the 92d Information Operations Squadron (92 IOS) performed a Cyber Vulnerability Assessment (CVA) of DEAMS at Maxwell AFB-Gunter Annex, Montgomery, Alabama, in June 2013. Although the team found a significant number of compliance issues, DEAMS Information Assurance (IA) posture has improved overall. System • DEAMS is a Major Automated Information System that uses commercial off-the-shelf Enterprise Resource Planning software to provide accounting and management services.
• The Program Office is following an evolutionary acquisition strategy that adds additional capabilities and users incrementally. There are seven scheduled releases. Release 2 has several sub-releases, including Release 2.2, which will support all AMC bases. Releases 3 through 7 will deploy to the Air National Guard and all Air Force major commands, including those overseas. There are expected to be nearly 30,000 users worldwide by 2017. • DEAMS operates on the Global Combat Support System – Air Force Integration Framework. It interfaces with approximately 40 other systems that provide travel, payroll, disbursing, transportation, logistics, acquisition, and accounting support. Mission • USTRANSCOM and Air Force financial managers use DEAMS to compile and share accurate, up-to-the-minute financial management data and information across USTRANSCOM and the Air Force. • USTRANSCOM, Air Force, and DoD leadership use DEAMS to access vital, standardized, real-time financial data and information to make strategic business decisions. • USTRANSCOM and the Air Force use DEAMS to satisfy congressional and DoD requirements for auditing funds, standardizing financial ledgers, timely reporting, and reduction of costly rework. Major Contractor Accenture Federal Services – Dayton, Ohio
DEAMS 269
A i r F o r c e P RO G R A M S Activity • To support a limited fielding decision, AFOTEC conducted an OA from May 30 through June 15, 2012. The OA results showed that DEAMS Increment 1 Release 1 had numerous substantial deficiencies and was not making adequate progress toward achieving operational effectiveness and operational suitability. OA-2 was added to gauge program progress and to inform the DEAMS MDA’s decision for further base deployments and Milestone C. • AFOTEC began OA-2 of DEAMS Release 2.2 in August 2013 in accordance with a DOT&E-approved test plan. The Phase 1 test locations include HQ USTRANSCOM and HQ AMC, both located at Scott AFB, Illinois; the DFAS facility in Limestone, Maine; and base operations at McConnell AFB, Kansas. • The Air Force is conducting OA-2 in two phases. During Phase I, DOT&E sent a team to all of the Phase 1 test sites to make a limited initial assessment of DEAMS progress since OA-1. Phase 1 was completed in September 2013. • Phase 2, which includes four new bases, began on September 26, 2013, and will continue through January 2014. • An Air Force Blue Team from the 92 IOS performed a CVA of DEAMS at Maxwell AFB-Gunter Annex, Montgomery, Alabama, in June 2013. Although the team found a significant number of compliance problems, DEAMS IA posture has improved overall. Assessment • The DOT&E assessment of DEAMS prior to the October limited fielding included interviews with current users and system managers. The assessment determined how well DEAMS has progressed since OA-1 in its ability to manage software defects, patches, and pre-deployment regression testing. DOT&E observers sought to understand the operational impacts of the DEAMS Release 2.2 fielding on the current DEAMS users. The assessment plan expected that major deficiencies, such as were found in OA-1, would be immediately apparent and that interviews with the individuals responsible for configuration management would confirm or deny program claims of greatly improved rigor.
270 DEAMS
• DEAMS appears to have improved configuration management by incorporating a new incident reporting system known as Serena. The systems integrator is now conducting regression testing prior to operational fielding (which should have been the case previously), and the need for software rollbacks have consequently decreased. Effective workarounds for existing software defects have been well documented at DFAS‑Limestone. However, few of the workarounds below that level have been documented, particularly at the base level. • Although configuration management has improved, there are still a large number of unresolved defects and several currently required capabilities and enhancements are scheduled for implementation in future software releases. • Feedback from new users at McConnell AFB indicated that the training they had received was inadequate. They noted that it focused on navigating DEAMS but did not provide them with a real understanding of the system and its application to their day-to-day work process. Users also stated that they need more on-site technical support during DEAMS implementation. • Based on its assessment, DOT&E concurred with the MDA’s decision to release DEAMS to four new AMC bases, provided that training and on-site support are improved. Recommendations • Status of Previous Recommendations. The program is addressing five of the six previous recommendations. The Program Office and the Functional Management Office still need to document workarounds out to the base level. • FY13 Recommendations. The program manager should: 1. Correct IA deficiencies noted in the June 2013 CVA and perform both IA and financial fraud penetration testing. 2. Update training material based on current user feedback before training new users. 3. Provide more on-site technical support to the new users at the base level.
A i r F o r c e P RO G R A M S
F-15E Radar Modernization Program (RMP) Executive Summary • The Air Force Operational Test and Evaluation Center (AFOTEC) conducted IOT&E from March through September 2013 to assess the system’s operational effectiveness, operational suitability, and mission capability. The IOT&E included 85 F-15E Radar Modernization Program (RMP) sorties and 175 hours of dedicated flight testing in which AFOTEC evaluated the F-15E RMP in an operationally representative cross-section of counterair and counterland operations employing both live and simulated air-to-air and air-to-ground weapons in realistic tactical scenarios. • Preliminary IOT&E results indicate the F-15E RMP: - Is operationally effective and provides significantly improved capability in the air-to-air operational environment compared to that of the legacy radar system. - Demonstrated comparable air-to-ground radar performance compared with that of the legacy system and realized some improvements in target location accuracy. - Achieved improved system reliability and maintainability with a reduced deployment footprint of personnel, spare parts, and maintenance equipment compared to that of the legacy system. - Does not meet the user’s software stability requirement of 30 hours Mean Time Between Software Anomaly (MTBSA). The inability to meet this requirement diminishes the effect of the overall mission capability improvements that the RMP system is intended to provide to operational F-15E units. • The Air Force plans to conduct the Full-Rate Production decision in March 2014. System • The F-15E is a twin engine, tandem seat, fixed wing, all weather, multi-role fighter aircraft. The F-15E has a fully missionized cockpit and a multimode air intercept and air- to- ground radar, giving the aircrew the capability to employ air-to-air and air-to-ground munitions, a 20 mm cannon, and countermeasures for evading enemy fire. • The RMP replaces the F-15E legacy APG-70 mechanically scanned radar with an active electronically scanned array (AESA) system designated the APG 82(V)1. The RMP is designed to retain functionality of the legacy radar system while providing expanded mission employment capabilities to include: - Near-simultaneous interleaving of selected air-to-air and air-to-ground functions - Enhanced air-to-air and air-to-ground combat identification capabilities - Longer range air-to-air target detection and enhanced track capabilities
- Longer range and higher resolution air-to-ground radar mapping - Improved ground moving target track capability • The RMP upgrade is also intended to address legacy F-15E radar system suitability shortfalls including: poor reliability, parts obsolescence, and high sustainment costs. The Air Force intends to retrofit the RMP across the existing F-15E fleet. • The RMP APG-82(V)1 design leverages capabilities from currently fielded AESA radar systems. The APG 82(V)1 antenna and power supply are currently in use on the F-15C APG-63(V)3 program, and the radar receiver/exciter and Common Integrated Sensor Processor are based on the F/A-18E/F APG-79 AESA system. • Other hardware and software modifications comprising the RMP effort include a more powerful Environmental Control System, updates to the aircraft Operational Flight Program and Electronic Warfare software, a new radio frequency tunable filter, and aircraft modifications to include a new wideband radome and wiring changes. Mission A unit equipped with the F-15E conducts all weather, day and night missions to include: • Offensive and Defensive Counterair • Conventional Air Interdiction and Nuclear Strike • Close Air Support and Strike Coordination and Reconnaissance • Suppression of Enemy Air Defenses • Combat Search and Rescue Major Contractors • The Boeing Company – St. Louis, Missouri • Raytheon – El Segundo, California
F-15E RMP 271
A i r F o r c e P RO G R A M S Activity • The Air Force conducted F-15E RMP testing in accordance with the DOT&E-approved Test and Evaluation Master Plan and test plan. • AFOTEC conducted IOT&E from March through September 2013 to assess the system’s operational effectiveness, operational suitability, and mission capability. The IOT&E included 85 F-15E RMP sorties and 175 hours of dedicated flight testing in which AFOTEC evaluated the F-15 E RMP in an operationally representative cross-section of counterair and counterland operations employing both live and simulated air- to-air and air-to-ground weapons in realistic tactical scenarios. • The Air Force plans to conduct the Full-Rate Production decision in March 2014. Assessment • Preliminary IOT&E results indicate the F-15E RMP: - Is operationally effective and provides significantly improved capability in the air-to-air operational environment compared to that of the legacy radar system. - Demonstrated comparable air-to-ground radar performance compared with that of the legacy system and realized some improvements in target location accuracy. - Achieved improved system reliability and maintainability with a reduced deployment footprint of personnel, spare parts, and maintenance equipment compared to that of the legacy system. - Does not meet the user’s software stability requirement of 30 hours MTBSA. The inability to meet this requirement diminishes the effect of the overall mission capability improvements that the RMP system is intended to provide to operational F-15E units. • The primary emphasis behind the RMP upgrade is to improve the reliability, maintainability, and sustainability of the F-15E
272 F-15E RMP
radar system while significantly improving the aircraft’s air-to-air and air-to-ground capabilities. As has been the case in similar fighter AESA upgrades (e.g., F/A-18 APG-79 AESA), preliminary RMP IOT&E results indicate improved operational capabilities, hardware reliability, and system maintainability. However, as has also been the case with similar AESA upgrades, the inability to achieve the level of software stability necessary to meet the users’ operational mission requirements detracts from the overall effectiveness and mission capability of the F-15E RMP system. • The F-15E RMP system software architecture shares significant commonality with that of the F/A 18 APG-79, and the APG-79 has yet to resolve the software stability deficiencies identified in its 2007 IOT&E. Therefore, it is unlikely that the Air Force will achieve the stability necessary to achieve the full potential operational capability of the F-15E RMP system unless significant effort and resources are directed towards improved software stability. Recommendations • Status of Previous Recommendations. In FY12, DOT&E recommended that the Air Force should consider either amending the RMP 30-hour MTBSA requirement or structuring the program (in particular, adding time and resources for additional development) to achieve the desired performance measure. The Air Force did not amend the requirement, and preliminary IOT&E results indicate the Air Force did not meet the MTBSA requirement. • FY13 Recommendation. 1. The Air Force should place increased emphasis and provide necessary resources to improve RMP software stability in order to achieve the user’s desired MTBSA requirement and realize the full operational potential of the F-15E RMP system.
A i r F o r c e P RO G R A M S
F-22A Advanced Tactical Fighter Executive Summary • The Air Force Operational Test and Evaluation Center (AFOTEC) completed FOT&E of F-22A Increment 3.1 Enhanced Global Strike capabilities in November 2011, and fleet-wide Increment 3.1 retrofits of Block 30 F-22As continued throughout FY13. • F-22A Increment 3.2A developmental testing proceeded throughout FY13 and will continue in FY14. Increment 3.2A is a software-only modernization effort integrating Link 16 Receive, enhanced Combat Identification, and enhanced Electronic Protection (EP) capabilities. • The F-22A Modernization integrated test construct enabled operational test pilots to fly familiarization, training, regression, and developmental test support missions with F-22As configured with early developmental Increment 32.A Operational Flight Program (OFP) software releases throughout FY13. This enabled the F-22A Increment 3.2A program to identify problems early in system development and preserve the overall Increment 3.2A developmental test schedule throughout FY13. • F-22A Modernization Increment 3.2B, a separate Major Defense Acquisition Program, achieved Milestone B in June 2013. System • The F-22A is an air superiority fighter that combines low observability to threat radars, sustained high speed, and integrated avionics sensors. • Low observability reduces threat capability to engage F-22As with current adversary weapons. • The aircraft maintains supersonic speeds without the use of an afterburner. • Avionics that fuse information from the Active Electronically Scanned Array radar, other sensors, and datalinked information for the pilot enable employment of medium- and short-range air-to-air missiles, guns, and air-to-ground munitions. • The Air Force designed the F-22A to be more reliable and easier to maintain than legacy fighter aircraft. • F-22A air-to-air weapons are the AIM-120C radar-guided missile, the AIM-9M infrared-guided missile, and the M61A1 20 mm gun. • F-22A air-to-ground precision strike capability consists of the 1,000-pound Joint Direct Attack Munition and the 250-pound Small Diameter Bomb (SDB) Increment One. • The F-22A program delivers capability in increments. Incremental Enhanced Global Strike modernization efforts include the following current and projected increments: - Increment 3.1 provides enhanced air-to-ground mission capability, to include geo-location of selected emitters, electronic attack, air-to-ground synthetic aperture
radar mapping and designation of surface targets, and SDB integration. Increment 3.1 is currently fielding in operational F-22A units. - Increment 3.2A is a software-only upgrade intended to provide improved EP, Link 16 Receive, and Combat Identification capabilities in early FY15. Increment 3.2A is a modernization effort within the scope of the F-22A Advanced Tactical Fighter baseline acquisition program of record. - Increment 3.2B is a separate Major Defense Acquisition Program modernization effort intended to integrate AIM‑120D and AIM-9X missile systems and provide additional EP enhancements and improved emitter geolocation capability. Increment 3.2B IOT&E is currently planned for FY17. Mission A unit equipped with the F-22A: • Provides air superiority over friendly and non-permissive, contested enemy territory • Defends friendly forces against fighter, bomber, or cruise missile attack • Escorts friendly air forces into enemy territory • Provides air-to-ground capability for counter-air, strategic attack, counter-land, and enemy air defense suppression missions Major Contractor Lockheed Martin Aeronautics Company – Fort Worth, Texas
F-22A 273
A i r F o r c e P RO G R A M S Activity • The Air Force conducted F-22A testing in accordance with the DOT&E-approved Test and Evaluation Master Plan and test plan. • The Air Force completed F-22A Increment 3.1 FOT&E in November 2011. Fleet-wide Increment 3.1 retrofits of Block 30 F-22As continued throughout FY13. • F-22A Increment 3.2A developmental testing proceeded throughout FY13 and will continue in FY14. Increment 3.2A FOT&E is scheduled to begin in June 2014. • F-22 Increment 3.2B achieved Milestone B in June 2013. Assessment The F-22A Increment 3.2A integrated testing construct enabled the program to progress in accordance with the planned FY13 development schedule. Air Combat Command’s 53d Wing operational test pilots flew familiarization, training, regression, and developmental test support missions with F-22As configured
274 F-22A
with early developmental OFP releases throughout FY13. This effort provided operational testers early insight into capabilities and helped shape development efforts and the scope of testing that will be required to vet system capabilities in the FY14 AFOTEC FOT&E. Recommendations • Status of Previous Recommendations. The Air Force continues to address all previous recommendations. • FY13 Recommendation. 1. The Air Force should continue to utilize the integrated testing construct for F-22A Increment development, and should provide increased opportunities, where feasible, for operational test unit pilots to conduct familiarization, training, regression, and developmental flight test support with early OFP releases.
A i r F o r c e P RO G R A M S
Global Broadcast System (GBS) Executive Summary • The Air Force is transitioning the Global Broadcast Service (GBS) system Satellite Broadcast Manager (SBM) subsystem from Navy sites to Defense Information System Agency (DISA) Defense Enterprise Computing Center (DECC) sites and integrating them into an enterprise architecture for long-term sustainment. • The Air Force established a Global Broadcast Service Operations Center (GBSOC) at Schriever AFB, Colorado, to manage GBS operations and provide help desk support to users with technical issues. • The Air Force’s 17th Test Squadron conducted a Force Development Evaluation (FDE) from July 15, 2013, through August 29, 2013, that was not concluded. The Test Squadron Commander paused the test because real-world operations in the Central Command Area of Responsibility preempted planned testing. • The test squadron conducted the remainder of the FDE from September 20 – 24, 2013. • The FDE results will inform an Air Force Space Command decision to accept the GBS DECC SBM for operation in early FY14. • Although analysis of the operational testing data is ongoing, developmental testing (DT) and preliminary analysis of operational testing suggest that: - GBS does not properly transmit unmanned aerial vehicle video to users. - The procedures for properly restoring GBS operations after transitioning from the primary DECC SBM to the alternate DECC SBM are incomplete. - GBSOC procedures are immature. - GBS broadcast support organizational roles and responsibilities need to be better defined. - Tier II help desk support needs to be sustained at the current levels. The around-the-clock Tier II help desk that provided technical support during the operational test is not on contract past the 2013 calendar year to support operations. • The GBS Program Manager is updating the GBS system to satisfy National Security Agency (NSA) security requirements. The Air Force Operational Test and Evaluation Center is planning to conduct an FOT&E in 1QFY15 to evaluate the operational effectiveness and suitability of the updated architecture.
satellites, and an additional government-leased satellite capability to meet operational demand. - The broadcast segment includes: ▪▪ S BMs that manage the flow of selected information to the orbiting satellites for broadcast to the appropriate theaters of operation. The SBMs are being relocated from Navy sites to DISA DECCs. They interface through DoD Teleport sites for the Wideband Global and commercial satellites and fixed Primary Injection Points for Ultra High Frequency Follow-On satellites. ▪▪ T heater Injection Point antennas that are mobile and provide support for all satellites. They also provide the Combatant Commanders an in-theater uplink capability that broadcasts information products to regional forces. System - The receive segment consists of ground- and sea-based • The GBS system is a satellite-based broadcast system providing mobile terminals that process and provide the appropriate near worldwide, high-capacity, one-way transmission of information to the end users within selected areas of operational military data. operation. • The GBS system consists of three segments: - The space segment includes GBS transponders on Wideband • The program manager is updating the broadcast and receive segments to satisfy NSA security requirements. Global Satellites, Ultra High Frequency Follow‑On
GBS 275
A i r F o r c e P RO G R A M S Mission • Operational forces worldwide use GBS to provide a continuous high-speed and high-volume flow of data, audio, imagery, and video at multiple classification levels for sustained operations. • Combatant Commanders use the GBS capability to provide theater-specific intelligence and battlespace weather
Activity • The Air Force is transitioning the GBS SBMs from Navy sites to DISA DECCs. The Air Force is integrating the SBMs into an enterprise architecture to meet the long-term sustainment strategy directed in the Deputy Secretary of Defense’s Program Directive Memorandum IV. • The Air Force established a GBSOC at Schriever AFB, Colorado, to manage GBS operations and provide help desk support to users with technical issues. • The Air Force Program Office conducted government DT from September 24, 2012, through October 12, 2012. Due to poor DT results, the Air Force made fixes to the system and conducted a limited-scope developmental regression test from March 18 – 29, 2013, to demonstrate the program manager had fixed the performance problems and the system was ready for operational test. • The Air Force’s 17th Test Squadron conducted an FDE from July 16, 2013, through August 29, 2013, that was not concluded. The Test Squadron Commander paused the test because real-world operations in the Central Command Area of Responsibility preempted planned testing. • The test squadron conducted the remainder of the FDE from September 20 – 24, 2013, once Central Command assets were once again available. The 17th Test Squadron conducted the FDE in accordance with the DOT&E-approved Test and Evaluation Master Plan and test plan. • The FDE results will inform an Air Force Space Command decision to accept the GBS DECC SBM for operation in early FY14. • The GBS Program Manager is updating the GBS system to satisfy NSA security requirements. The Air Force Operational Test and Evaluation Center is planning to conduct an FOT&E in 1QFY15 to evaluate the operational effectiveness and suitability of the updated architecture. Assessment • The program manager conducted DT from September through October 2012. The DT demonstrated the GBS did not meet the operational requirement for product reception rate and product integrity rate. The DT also showed that GBSOC
276 GBS
information, thereby increasing the joint operations mission data available to regional military forces. Major Contractor Lockheed Martin Corporation Information Systems and Global Solutions – Gaithersburg, Maryland
•
• •
•
procedures were immature and not able to adequately support operational users. The Air Force’s developmental regression test demonstrated improved product reception rate and product integrity rate; however, the test was limited in scope, lacked operational realism, and was inadequately resourced. The Central Command Area of Responsibility broadcast was not tested. The lack of operational realism in the DT regression test prevented the GBSOC from adequately refining and exercising their help desk procedures prior to operational test. During the DT, the ability to transition GBS operations from the primary DECC SBM to the alternate DECC SBM and back to the primary SBM did not meet requirements. The program manager took corrective actions, but chose not to retest this capability during the DT regression test to meet schedule and cost constraints. Although analysis of the operational testing data is ongoing, DT and preliminary analysis of operational testing suggest that: - GBS does not properly transmit unmanned aerial vehicle video to users. - The procedures for properly restoring GBS operations after transitioning from the primary DECC SBM to the alternate DECC SBM and back to the primary SBM are incomplete. - GBSOC procedures are incomplete and immature. - GBS broadcast support organizational roles and responsibilities need to be better defined. - Tier II help desk support needs to be sustained at the current levels. The around-the-clock Tier II help desk that provided technical support during the operational test is not on contract past the 2013 calendar year for operations.
Recommendations • Status of Previous Recommendations. The Air Force addressed all previous recommendations. • FY13 Recommendation. 1. The Air Force should correct the deficiencies identified in the FDE and verify they are corrected in the FY15 FOT&E or other operational test events.
A i r F o r c e P RO G R A M S
HC/MC-130J Executive Summary • DOT&E issued an IOT&E report for the HC/MC-130J in April 2013. The HC/MC-130J is operationally effective, suitable, and survivable in the low- to medium-threat environment where the Air Force expects to operate this aircraft. • Problems with the enhanced electrical system have limited Increment 2 aircraft to the existing 200 Amp power supply instead of the planned 400 Amp system, which is required to support future capability upgrades and the AC-130J gunship modification. • Air Combat Command (ACC) is developing additional communications capabilities for the HC-130J, but additional work is required before the aircraft can receive an updated interoperability certification. • Air Force Special Operations Command (AFSOC) is preparing for developmental test and evaluation of Increment 3 in April 2014. The Air Force should update the Test and Evaluation Master Plan (TEMP) and include sufficient detail to assess future operational testing of Increment 3. System • The HC/MC-130J is a medium-sized, four-engine turboprop tactical transport aircraft with hose and drogue aerial refueling, airdrop, and command and control capabilities. The core configuration is based on the Marine Corps KC-130J refueling tanker design with modifications including the ability to receive fuel in flight, a nose-mounted electro-optical/infrared sensor, and a combat systems operator flight deck station. • The HC/MC-130J program delivers capability in increments. Increment 1 modifications include additional countermeasure dispensers, high-altitude ramp and door hydraulics, an additional (fourth) flight deck crew member station, an additional cargo compartment intercom panel, and cargo compartment 60-Hertz electrical outlets. Increment 2 includes increased 28-volt direct current internal power capacity, crash- worthy loadmaster scanner-position seats, and provisions for Large Aircraft Infrared Countermeasures. Increment 3 includes dual special mission processors and a secure file server. • The HC-130J will replace legacy HC-130P/N and MC-130P (rescue) aircraft; the MC-130J will replace legacy MC-130E/P
Activity • The Air Force Operational Test and Evaluation Center conducted the IOT&E from March 1 through May 30, 2012, in accordance with a DOT&E-approved test plan. DOT&E issued an IOT&E report for the HC/MC-130J in April 2013 and approved an updated TEMP in August 2013.
(special operations) aircraft. The Air Force intends to procure 37 HC-130Js and 94 MC-130Js; only the MC-130Js will be developed to Increment 3 capability. Of the 94 MC-130Js, 37 will be converted to AC-130J gunships under a separate U.S. Special Operations Command program. Mission • ACC uses the HC-130J to support the personnel recovery mission through: - Aerial and ground refueling of vertical lift assets used during personnel recovery missions - Para-rescue jumper deployment with rescue-related equipment - Infiltration/exfiltration and resupply by airdrop or air-land operations • AFSOC uses the MC-130J to support special operations missions requirements, including: - Aerial refueling and forward arming and refueling point operations of Special Operations Forces rotary and tilt-rotor aircraft - Infiltration/exfiltration, resupply, or delivery of Special Operations Forces personnel and equipment via airdrop or landing on austere, short runways in hostile or denied territory Major Contractor Lockheed Martin Aeronautics Company – Bethesda, Maryland
• The Milestone Decision Authority approved the Air Force request to enter full-rate production on October 4, 2013. • The contractor is now delivering all aircraft in the Increment 2 configuration with the exception of the planned 400 Amp regulated transformer rectifier unit (RTRU), which has
HC/MC-130J 277
A i r F o r c e P RO G R A M S experienced mechanical, electrical, and software problems. The additional power (over the current 200 Amp RTRU) will be required for future system upgrades on the MC-130J and to support modification of 37 MC-130J aircraft into AC-130J gunships. Contractor efforts to troubleshoot the 400 Amp RTRU are ongoing and regression testing is anticipated in 1QFY14. • ACC intends to field a “T-1” communications modification for the HC-130J to address shortfalls in some legacy HC-130P/N aircraft enhancements. The T-1 modification includes the Specialized Automated Mission Suite/Enhanced Situational Awareness (SAMS/ESA) system, Blue Force Tracker, and the Joint Precision Airdrop System. The SAMS/ESA system provides Situational Awareness Data Link, High Power Waveform, and Air Force Tactical Radio System-Ruggedized. • ACC conducted limited operational evaluations in FY13, including participation in the Angel Thunder search and rescue exercise, which the Joint Interoperability Test Command (JITC) observed in order to update the HC-130J interoperability certification. Development of the T-1 modification is ongoing and DOT&E expects to review a follow-on operational test plan from ACC prior to any final testing that supports a fielding decision. • AFSOC continues to work with the contractor on development of Increment 3 with government developmental test and evaluation scheduled to begin in April 2014. Assessment • The HC/MC-130J is operationally effective for most combat search and rescue and Special Operations Forces missions as described in the classified annex to the IOT&E report. The combat search and rescue and Special Operations Forces- specific modifications to the HC/MC-130J supported the ability of aircrews to conduct these missions. The aircrew successfully conducted air-land, airdrop, aerial and ground refueling, and formation flight operations across a range of environmental conditions and operations. • The HC/MC-130J is operationally suitable. The aircraft met all quantitative suitability requirements except for Mean Time to Diagnose a Fault (required to be less than 30 minutes; achieved 119 minutes) and probability of completing a 4-hour mission without a failure (required to be at least 91 percent;
278 HC/MC-130J
•
•
•
•
achieved 83 percent). These shortfalls are not operationally significant – the system nonetheless exceeded the materiel availability Key Performance Parameter objective requirement of 89 percent by demonstrating a mission-capable rate of 95 percent. The HC/MC-130J is survivable in the low- to medium-threat environment where the Air Force expects to operate this aircraft. Specific threats and mission scenarios are detailed in the classified annex to the IOT&E report. Planned testing of the 400 Amp RTRU slipped significantly in FY13. Without the enhanced RTRU, the MC-130J will not be able to support intended future upgrades such as a terrain‑following radar and the aircraft modified to the AC-130J configuration will not be able to operate all necessary systems in certain scenarios. Based on current data and documentation, JITC cannot issue an interoperability certification for the T-1 modified HC-130J. ACC has not updated the Information Support Plan for the HC-130J to reflect the T-1 modification. Furthermore, JITC was unable to collect operationally relevant data on the modification during observation of the Angel Thunder exercise. There were also anomalies in part of the SAMS/ESA software that the Air Force must correct. As of the recent TEMP update, AFSOC had not defined specific software capability requirements for the Increment 3 special mission processors, so there was insufficient detail in the TEMP to assess any planned FOT&E.
Recommendations • Status of Previous Recommendations. The Air Force has made progress toward all previous recommendations. • FY13 Recommendations. 1. ACC should address interoperability shortfalls in the HC-130J T-1 modification and present a detailed test plan to DOT&E no later than 60 days prior to the start of any follow-on operational test activities. 2. The program should submit a TEMP addendum that describes a robust statistical test design for Increment 3 once specific capability requirements are defined and no later than 180 days prior to the start of any follow-on operational test activities.
A i r F o r c e P RO G R A M S
Joint Air-to-Surface Standoff Missile – Extended Range (JASSM-ER) Executive Summary • The Air Force and Lockheed Martin are in the process of engineering development of the FMU-162/B Electronic Safe and Arm Fuze (ESAF) as a replacement for the mechanical fuzes currently used in Joint Air-to-Surface Standoff Missile – Extended Range (JASSM-ER). • The Air Force should continue to conduct Reliability Assessment Program testing on Lot 8 and later JASSM-ER missiles in the Weapons System Evaluation Program (WSEP). System • The AGM-158B JASSM-ER is a stealthy cruise missile that flies a preplanned route from launch to a target, using GPS guidance and an internal navigation system. • JASSM-ER: - Has a 1,000-pound penetrating warhead - Has an imaging infrared seeker that can be used for greater accuracy and precision; the seeker uses image templates prepared by a rear echelon intelligence unit - Can be launched by B-1B aircraft, with plans to integrate it on the F-15E, F-16, and B-52 - Includes a container that protects the weapon in storage and aids ground crews in moving, loading, and checking the missile - Uses the same Air Force mission planning systems utilized for aircraft and other weapons • FMU-162/B ESAF takes advantage of advances in fuze technology and is intended to be a more reliable fuze with the same capabilities as the baseline mechanical fuze. The FMU-162/B ESAF would be used in JASSM baseline and ER variants.
Activity AGM-158B JASSM-ER • The Air Force completed the JASSM-ER IOT&E in November 2012. • DOT&E submitted an IOT&E report in May 2013 with the determination that the JASSM-ER is operationally effective and suitable. • The Air Force conducted all testing in accordance with the DOT&E-approved Test and Evaluation Master Plan and test plan.
Mission Operational units equipped with JASSM-ER will employ the weapon from the B-1B Lancer against high-value or highly- defended targets from outside the lethal range of many threats. Units equipped with JASSM-ER will: • Destroy targets with minimal risk to flight crews and support air dominance in the theater • Strike a variety of targets greater than 500 miles away • Execute missions using automated preplanning or manual pre-launch retargeting planning • Attack a wide range of targets including soft, medium, and very hard (not deeply buried) targets Major Contractor Lockheed Martin, Missile and Fire Control – Orlando, Florida
• The Air Force has not taken delivery of any production JASSM-ER weapons yet, so no Reliability Assessment Program missions have been flown for Lot 8 or future JASSM- ER missiles. FMU-162/B JASSM ESAF • The FMU-162/B ESAF is in technical development testing. • The Air Force and contractor are currently re-designing the fuze based on the results of the sled testing conducted.
JASSM-ER 279
A i r F o r c e P RO G R A M S Assessment AGM-158B JASSM-ER • There was no operational test activity to report following the conclusion of the JASSM-ER IOT&E in November 2012. • Despite improvements in workmanship and production processes, there is still a need to evaluate the inherent reliability of production lot missiles (through Lot 8, at a minimum) to ensure that the reliability growth plan is successful. Since there has been no Lot 8 JASSM-ER production missiles flown in WSEP, it is not possible to evaluate the reliability of these weapons. FMU-162/B JASSM ESAF • The FMU-162/B ESAF program has the potential to increase the overall reliability of all JASSM variants. The FMU-162/B ESAF program would replace the current
280 JASSM-ER
electro-mechanical fuze, which relies on moving parts prone to reliability failures. In addition, the FMU-162/B ESAF has more built-in test capability than the current electro-mechanical FMU-156/B ESAF. Recommendations • Status of Previous Recommendations. The Air Force has not addressed either of the FY12 recommendations. The Air Force should continue Reliability Assessment Program testing for Lot 8 and beyond in the WSEP. In addition, the Air Force should, in conjunction with the contractor, continue the technical development and evaluation of the FMU-162/B ESAF. • FY13 Recommendations. None.
A i r F o r c e P RO G R A M S
Joint Space Operations Center (JSpOC) Mission System (JMS) Executive Summary • The Air Force Operational Test and Evaluation Center (AFOTEC) conducted an Operational Utility Evaluation (OUE) of the Joint Space Operations Center (JSpOC) Mission System (JMS) Increment 1, which included the initial delivery of system infrastructure and data display capabilities. DOT&E published the final test report on February 19, 2013. • The OUE informed Air Force operational acceptance of Increment 1 and milestone decisions for the acquisition of Increment 2. System • The JMS program provides applications, net-centric services and databases, and dedicated hardware to improve space situational awareness and command and control of space forces. • JMS will replace legacy Space Defense Operations Center and space‑specific portions of the Correlation, Analysis, and Verification of Ephemerides Network systems, both of which are aging and unsustainable, and cannot meet changing threat, operating environment, and mission requirements. The Air Force is developing JMS in two increments. - Increment 1 delivered an initial Service-Oriented Architecture infrastructure and user tools, including a User Defined Operational Picture that allows access to and analysis of data from legacy systems, integrated collaboration/messaging/data sharing tools, and Space Order of Battle processing. - Increment 2 is being developed to deliver most of the required mission functionality, including replacement of legacy data processing and analysis capabilities to directly task sensors, ingest sensor data, produce and sustain a high-accuracy space catalog, increase orbit determination and prediction accuracy, and improve capacity to support conjunction assessment (predicting orbit intersection and potential collision), orbital safety, threat modeling, and operational decisions. Mission The Joint Functional Component Command for Space (JFCC SPACE), a component of U.S. Strategic Command, will use JMS to enable the coordination, planning, synchronization, and execution of continuous, integrated space operations in support of national and Combatant Commander objectives. JFCC SPACE will use JMS to execute these five lines of operations: • Space Object Identification – Identify, analyze, and maintain a thorough database of objects in space at a given time as
•
•
• •
necessary to provide a robust and accurate space-operating picture. Spectrum Characterization – Identify, analyze, and resolve interference problems in the electromagnetic spectrum and operationally characterize non-kinetic activities and effects within the realm of space operations. Launch/Reentry Support – Provide awareness and warning of potential threats to space systems, including thorough knowledge and rapid identification of all objects being launched into, traveling through, or deorbiting from the space domain. Joint Forces Support – Provide space products, services, and effects to military, commercial, and civil entities worldwide in support of the joint/coalition force and national users. Support to Contingency Operations – Protect space capabilities and provide on-order, tailored delivery of space effects.
Major Contractors • System Integrator, Increments 1 and 2: - Space and Naval Warfare Systems Command – San Diego, California • Increment 1 sub-contractors: - Integrated Support Systems, Inc. – Colorado Springs, Colorado - The Design Knowledge Company – Fairborn, Ohio • Increment 2 sub-contractors: - Integrated Support Systems, Inc. – Colorado Springs, Colorado - Artificial Intelligence Solutions – Lanham, Maryland - Analytical Graphics Incorporated – Exton, Pennsylvania
JMS 281
A i r F o r c e P RO G R A M S Activity • AFOTEC, the JMS Program Office, and the 46th Test Squadron led an Integrated Test and Evaluation of JMS Increment 1 from August through October 2012. • AFOTEC conducted a dedicated OUE of JMS Increment 1 from November 2012 through January 2013. • DOT&E published a final OUE test report on February 19, 2013, which informed Air Force operational acceptance of Increment 1 and milestone decisions for the acquisition of Increment 2. • AFOTEC conducted all testing in accordance with the DOT&E-approved Test and Evaluation Master Plan and test plan. Assessment • AFOTEC’s OUE of JMS Increment 1 was adequate to assess JMS Increment 1 as effective and suitable for the limited capabilities delivered, including initial delivery of system infrastructure and data display functions. While useful relative to the existing legacy systems, Increment 1 delivered only a small subset of the full operational capabilities needed and expected with the full JMS, which are programmed and planned for delivery over the next several years. Increment 1 is not yet suitable for the JMS end-to-end mission, due to non-availability of some external data sources and infrastructure. • AFOTEC employed an integrated test methodology to optimize the use of data collection for future OT&E. Nothing was observed during the OUE that would preclude using Increment 1 as the basis for further JMS development.
282 JMS
Recommendations • Status of Previous Recommendations. This is the first annual report for this program. • FY13 Recommendations. Consistent with the February 2013 DOT&E report on the JMS OUE, the Air Force should: 1. Develop an acquisition strategy for delivery of capabilities post-Increment 2, including facilities and capabilities to support continuity of operations. 2. Investigate and resolve problems with external data source consistency, external interfaces, and support networks that will otherwise impede JMS end-to-end mission performance. 3. Complete the planned technology refresh for Increment 1 equipment that currently limits system capacity and continue acquisition, development, testing, and fielding of JMS Increment 2. 4. Assess new Increment 2 capabilities and reassess JMS User Defined Operational Picture and net-centric capabilities to verify full JMS functionality. 5. Develop and validate model and simulation tools to support evaluation of system capacity under high-user loading and evaluation of JMS high-accuracy catalog size and accuracy. 6. Develop operationally-relevant measures to assess when JMS system performance degradation due to excessive loading or cyber-attack is no longer acceptable. Provide capabilities to allow system administrators to monitor performance and take appropriate actions to mitigate operational impacts based on these measures. 7. Conduct independent, non-cooperative, threat representative penetration testing to assess protect, detect, react, and restore components of cyber security for Increment 2.
A i r F o r c e P RO G R A M S
KC-46A Executive Summary • DOT&E approved the post-Milestone B Test and Evaluation Master Plan (TEMP) in January 2013. The TEMP approval memorandum identified planned test program shortfalls that require resolution prior to the Milestone C TEMP submittal. • The KC-46A successfully completed the Critical Design Review (CDR) in July 2013. DOT&E published an Operational Assessment report on the program’s progress toward IOT&E in June 2013 to support the CDR. • Readiness for the scheduled start of IOT&E continues to be high risk with a 6- to 12-month delay expected. • The ALR-69A Radar Warning Receiver (RWR) has effectiveness shortfalls that require resolution prior to integration on the KC-46A. The contractor has made some hardware and software changes to improve performance. • The program has made advances in collecting and analyzing live fire test data needed to address the KC-46A vulnerability to dry bay fires. System • The KC-46A aerial refueling aircraft is the first increment (179) of replacement tankers for the Air Force’s fleet of KC-135 tankers (more than 400). The KC-46A design uses a modified Boeing 767-200ER commercial airframe with numerous military and technological upgrades, such as the fly-by-wire refueling boom, the remote air refueling operator’s station, additional fuel tanks in the body, and defensive systems. The KC-46A is intended to provide boom (pictured) and probe-drogue refueling capabilities. The Air Force intends to equip the KC-46A with an air-refueling receptacle so that it can also receive fuel from other tankers, including legacy aircraft. • The KC-46A is designed to have significant palletized cargo and aeromedical capacities, chemical/biological/radiological/ nuclear survivability, and the ability to host communications gateway payloads. • Survivability enhancement features are incorporated into the KC-46A design. Susceptibility is reduced with an Aircraft
Activity • The KC-46A Integrated Test Team has met quarterly since April 2011. • DOT&E approved, with caveats, the post-Milestone B TEMP in January 2013. • The Air Force Operational Test and Evaluation Center conducted an operational assessment in accordance with a DOT&E-approved test plan to support the CDR in July 2013.
Survivability Equipment suite consisting of Large Aircraft Infrared Countermeasures, the ALR-69A RWR, and a Tactical Situational Awareness System (TSAS). The suite is intended to compile threat information from the RWR and other on- and off-board sources and prompt the crew with an automatic re-routing suggestion in the event of a threat. Vulnerability is reduced through the addition of fuel tank inerting and integral armor to provide some protection to the crew and critical systems. Mission Commanders will use units equipped with the KC-46A to: • Perform air refueling to accomplish six primary missions-- nuclear operations support, global strike support, air bridge support, aircraft deployment, theater support, and special operations support. Secondary missions will include airlift, aeromedical evacuation, emergency aerial refueling, air sampling, and support of combat search and rescue. • Operate in day/night and adverse weather conditions over vast distances to support U.S., joint, allied, and coalition forces. • Operate in a non-permissive environment. Major Contractor The Boeing Company, Commercial Aircraft in conjunction with Defense, Space & Security – Seattle, Washington
• DOT&E published an Operational Assessment report of the program’s progress towards IOT&E in June 2013 in support of the CDR. • The Air Force conducted the CDR in July 2013. • Developmental, operational, and Federal Aviation Administration test planning is ongoing. The contractor’s Stage 4 (final build) test plans are in development.
KC-46A 283
A i r F o r c e P RO G R A M S • The Air Force is developing a KC-46A Design Reference Mission document to provide operator-defined mission scenarios in projected theaters, which are required to adequately complete an operationally realistic survivability assessment. • The LFT&E Integrated Product Team provided a detailed description of planned survivability analyses but is still identifying the developmental and operational test data requirements for these analyses and model verification. • The Air Force completed most of the testing needed to address the aircraft’s vulnerability to dry bay fires. The Air Force finalized the center wing dry bay fire vulnerability test plan and scheduled testing for 1QFY14. The remaining relevant tests, i.e., wing dry bay fire sustainment and fuselage dry bay fire vulnerability, are scheduled for FY14 and FY15, respectively. Assessment • The TEMP approval memorandum identified planned test program shortfalls that require resolution prior to the Milestone C TEMP submittal to gain DOT&E approval. The TEMP requires increased detail in a number of areas. The planned test program includes the following shortfalls that the Air Force has partially addressed but still require complete resolution to gain DOT&E approval at Milestone C: - The Air Force should mitigate the impact of concurrent activities and planned flying hours for the Engineering and Manufacturing Development (EMD) program that place a high demand on limited aircraft and simulator resources. - The task loading across EMD test aircraft is unbalanced. - The operational test aircrew and maintenance personnel must have time to attain their training requirements and establish proficiency in their tasks before the start of IOT&E. - The technical order verification process must be completed before the start of IOT&E. - Sufficient calendar time must be allotted for correction of discrepancies and/or deficiencies discovered during developmental testing prior to the planned start of operational testing. • The Air Force is continuing to analyze existing schedule risks and potential mitigations. However, until the final detailed plans (referred to as Stage 4 test plans) are delivered, DOT&E will not have sufficient insight to determine if there are adequate mitigations to reduce the risk in the EMD test schedule. Delivery of approximately 375 Stage 4 test plans started in September 2013 and delivery is planned to be complete in March 2014. • DOT&E analysis of initial Boeing schedules with regard to aerial refueling certifications, aircraft and support equipment technical orders, and operator/maintainer training indicates that operational testing will likely slip at least 6 to 12 months. • Recent ALR-69A RWR operational testing on the C-130H revealed that it was not effective due to integration and
284 KC-46A
performance problems. ALR-69A RWR is being provided as Contractor Furnished Equipment, and in addition to previously identified shortfalls, requires additional complex integration with TSAS. The contractor has made some hardware and software changes to ALR-69A, including antennas, wingtip inertia measuring units, and some software modifications, which have yet to be proven in testing. • Preliminary ALR-69A RWR effectiveness test plans, TSAS, and crew situational awareness test plans needed for the survivability analyses and assessment are not well defined. Current test planning events and proposed test facilities require changes and upgrades to test the KC-46A against operationally realistic threat systems, consistent with the KC-46A concept of operations. Boeing intends to finalize these test plans by March 2014. • The KC-46A survivability requirements focused on less likely threats and did not thoroughly consider all survivability enhancement alternatives. Preliminary analysis of the wing leading edge, wing trailing edge, and center wing dry bay fire live fire test data confirmed the vulnerability of the KC-46A to dry bay fires. A dry bay fire suppression system was not considered in the design, even though it could have reduced KC-46A vulnerability more effectively than cockpit armor (less weight) against more operationally realistic threats. • The TEMP and other test documents do not address detailed Information Assurance (IA) protect, detect, react, and restore requirements. The program has begun to address these problems by planning additional testing and crew IA training through the IA Working Group. Recommendations • Status of Previous Recommendations. The Air Force is addressing some of the FY12 recommendations to incorporate realistic assumptions in test plans; however, additional work is still needed. The Air Force should still: 1. Submit a TEMP with a realistic schedule mitigating the above mentioned shortfalls. 2. Provide an approach to correct the ALR-69A RWR shortfalls prior to integration on the KC-46A. 3. Plan to begin IOT&E at least 6 to 12 months later than the current TEMP indicates to allow for completion of developmental test and initial training. • FY13 Recommendations. The Air Force should: 1. Provide a comprehensive aerial refueling certification plan for the KC-46A including all EMD Phase 1 and 2 receivers. 2. Plan testing against realistic cybersecurity threats to identify vulnerabilities for correction. In addition, plan follow-on penetration testing to assess IA performance in terms of protect, detect, react, and restore functions. 3. Consider the integration of a dry bay fire suppression system with the potential to reduce aircraft and crew vulnerability against operationally realistic threats.
A i r F o r c e P RO G R A M S
Massive Ordnance Penetrator (MOP) Executive Summary • DOT&E published a classified Massive Ordnance Penetrator (MOP) Early Fielding Report in April 2012 and an update in January 2013, which summarized testing during FY08 through FY12, including the successful re-design of certain aspects of the weapon system. • The Air Force, between April and July 2013, successfully completed two additional weapon drops from the B-2 aircraft on representative targets. The tests, conducted at the White Sands Missile Range, New Mexico, demonstrated weapon behavior in a GPS jamming environment after planned enhancements were incorporated. • DOT&E published a classified Early Fielding Report in September 2013 to summarize the FY13 testing of the Enhanced Threat Reduction Phase 1 effort. System • The GBU-57 MOP is a large, GPS-guided, penetrating weapon with the ability to attack deeply buried and hardened bunkers and tunnels. The warhead case is made from a special high-performance steel alloy and its design allows for a large explosive payload while maintaining the integrity of the penetrator case during impact. • The B-2 Spirit is the only aircraft in the Air Force programmed to employ the MOP. • The GBU-57 warhead is more powerful than its predecessors, the BLU-109 and GBU-28. • The MOP is an Air Force-led, Quick Reaction Capability that is on DOT&E oversight as well as a Secretary of Defense special interest effort.
Activity • Prior to the live-flight missions, the Air Force flew two captive-carry missions to validate the hardware and software changes implemented in the MOP. Both of the missions were successful and allowed the program to proceed to live weapons employment. • Between May and July 2013, the Air Force executed two weapon drops at White Sands Missile Range, New Mexico, on representative targets. This testing was to evaluate the effect of the Enhanced Threat Reduction modification to the weapon system performance. An Air Force B-2 aircraft flew four missions to complete two planned drops with live warheads. • During the mission on May 13, the Air Force aborted a weapon drop due to a fault in the Monitor and Control Equipment. The analysis uncovered a poorly soldered wire in the Monitor and Control Equipment that prevented the solenoids from receiving the command to release. The fault
Mission Combatant Commanders use MOP to conduct pre-planned, day or night attacks against defended point targets vulnerable to blast and fragmentation effects and requiring significant penetration, such as hardened and deeply-buried facilities. Major Contractor The Boeing Company, Defense, Space & Security – St. Louis, Missouri
has been isolated to MOP-unique aircraft equipment. Further inspections did not identify the failure to be a systemic fault. • During the mission on June 13, the B-2 had a recessed connector pin that failed to complete the release circuit. The Air Force aborted the mission when the aircrew could not receive a valid release signal to employ. The Air Force determined that the fault was inside the connector and was not associated with any MOP-unique equipment. Assessment In the September 2013 Early Fielding report, DOT&E concluded that the MOP is capable of effectively prosecuting selected hardened, deeply buried targets. The captive-carry missions and two weapon drops indicate that the weapon modification is adequate for the successful prosecution of all of the elements of the currently defined target set.
MOP 285
A i r F o r c e P RO G R A M S Recommendations • Status of Previous Recommendations. While there were no previous recommendations for this program, the Air Force addressed all recommendations in the September 2013 Early Fielding Report. • FY13 Recommendations. None.
286 MOP
A i r F o r c e P RO G R A M S
Miniature Air Launched Decoy (MALD) and MALD-Jammer (MALD-J) Executive Summary • In FY12, the Program Office converted the Miniature Air Launched Decoy (MALD) procurement line to MALD‑Jammer (MALD-J). The Air Force will no longer procure any MALDs without the jammer. • The Air Force Test and Evaluation Center (AFOTEC) completed IOT&E except for full mission-level testing, which is scheduled for October 2013. • The Air Force launched 11 MALD-J vehicles in operational environments. • Preliminary analysis indicates that MALD-J (and MALD) did not satisfy navigational accuracy requirements in operationally relevant environments. • Preliminary results indicate that the Air Force’s corrective actions for MALD have improved the materiel reliability of both MALD and MALD-J. System • MALD is a small, low-cost, expendable, air-launched vehicle that replicates how fighter, attack, and bomber aircraft appear to enemy radar operators. • The Air Force designed the MALD-J as an expendable, close‑in jammer to degrade and deny an early warning or acquisition radar’s ability to establish a track on strike aircraft while maintaining the ability to fulfill the MALD decoy mission. • In FY12, the Program Office converted the MALD procurement line to MALD-J. The Air Force will no longer procure any MALDs without the jammer. • The F-16C/D and B-52 are the lead aircraft to employ MALD and MALD-J.
Activity • AFOTEC completed testing of the MALD in FY12; the Air Force will no longer procure MALDs. • The Air Force tested design changes in MALD intended to improve reliability in flight tests during FY12 under the Reliability Assessment Program (RAP). • The MALD-J Program Office fired eight MALD-J missiles in FY13 as part of the Jammer Reliability Assessment Program (JRAP), which builds upon the MALD RAP and is intended to verify correction of reliability shortfalls in previous MALD testing. No failures were noted during these events. • AFOTEC executed 4 operational test events as part of the IOT&E in accordance with a DOT&E-approved test plan, launching a total of 11 live missiles in FY13.
Mission Combatant Commanders will use units equipped with: • MALD and MALD-J to improve battlespace access for airborne strike forces by deceiving, distracting, or saturating enemy radar operators and Integrated Air Defense Systems. • MALD to allow an airborne strike force to accomplish its mission by deceiving enemy radars and forcing air defense systems to treat MALD as a viable target. • MALD-J to allow an airborne strike force to accomplish its mission by jamming enemy radars and air defense systems to degrade or deny detection of friendly aircraft or munitions. Major Contractor Raytheon Missile Systems – Tucson, Arizona
- AFOTEC launched 11 free-flight missiles, captive-carried 9 missiles on Sabreliner aircraft to assess MALD-J performance, and flew 13 missiles on either B-52H or F-16C/D aircraft wings to accumulate carriage time on the weapon. - Four of the 11 free-flight vehicles experienced excessive navigational drift in their operational environment. Two of the remaining missiles were prematurely terminated due to range safety system failures. - Full mission-level simulation, the final stage of the IOT&E, is scheduled for October 2013. • The MALD-J program participated in a full mission employment test event, which included fifth-generation aircraft. No interoperability issues were observed.
MALD 287
A i r F o r c e P RO G R A M S Assessment • Preliminary analysis of IOT&E data indicates that MALD/ MALD-J may not satisfy navigational requirements in operationally relevant environments. • DOT&E is currently awaiting range data information and expects to publish a classified IOT&E report examining MALD-J system effectiveness and suitability, including deficient navigational accuracy problems upon completion of IOT&E. • Mission planning testing events (during full mission employment testing) for the MALD-J program indicate the time needed to plan a full load of MALD-J vehicles is excessive (averaging seven hours per missile to plan). • DOT&E will use a combination of MALD and MALD-J data to evaluate whether vehicle reliability problems observed during previous testing have been resolved. Since no failures in the MALD-J payload to date have occurred, and the other systems are otherwise essentially identical, combining these data is appropriate.
288 MALD
• Preliminary results of the JRAP show reliability corrective actions have improved the materiel reliability of MALD/ MALD-J. • The Air Force has not yet validated and accredited full mission-level simulation; validation and accreditation are necessary to ensure authentic, usable data. Recommendations • Status of Previous Recommendations. The Air Force satisfactorily addressed the FY12 recommendation. • FY13 Recommendations. The Air Force should: 1. Improve navigational accuracy in operational environments. 2. Improve mission planning capabilities for the MALD-J program to reduce the time needed to plan a full load of MALD-J vehicles. 3. Validate and accredit the Digital Integrated Air Defense System simulation facility for the MALD-J program.
A i r F o r c e P RO G R A M S
Mission Planning System (MPS)/Joint Mission Planning System – Air Force (JMPS-AF) Executive Summary • Following FY12 Air Force regression testing of a revised E-8 Mission Planning Environment (MPE) Version 1.3, the Air Force Operational Test and Evaluation Center (AFOTEC) executed the FY13 IOT&E in its entirety in accordance with the DOT&E-approved IOT&E plan. • The Air Force executed an FY13 IOT&E of a revised E-8 MPE from October through November 2012. A prior FY11 IOT&E was terminated before completion of testing due to critical deficiencies in flight planning calculations and data transfer functionality. The Air Force subsequently accomplished FY12 regression testing and re-accomplished the IOT&E in its entirety in FY13. • In April 2013, the DOT&E Major Automated Information System (MAIS) report determined E-8 MPE Version 1.3 to be operationally effective and operationally suitable. Previously identified critical deficiencies were corrected and the user defined Key Performance Parameters were met. However, manual workarounds are required to overcome some uncorrected shortfalls that detract from overall capability and Information Assurance shortfalls identified in 2011 testing remain to be corrected. System • A Mission Planning System (MPS) is a Standard Desktop Configuration (SDC)-based common solution for Air Force aircraft mission planning (the current SDC is based on a Windows XP® environment). It is a package of common and platform-unique mission planning applications. • An MPE is a set of developed applications built from a Framework, common components, and Unique Planning Components for a particular aircraft. The Framework is the basis of the MPE. Software developers add common components (e.g., Weather, Electronic Warfare Planner, etc.) and federated applications that support multiple users to the framework. Developers then add a Unique Planning Component for the specific aircraft type (e.g., E-8) to complete the MPE.
Activity • The Air Force conducted all MPE operational testing in accordance with a DOT&E-approved Test and Evaluation Master Plan and IOT&E Plan. • In 2011, the Air Force Operational Test and Evaluation Center (AFOTEC) conducted IOT&E on the E-8 MPE Version 1.0 but terminated testing prior to completion due to critical deficiencies in flight planning calculations and data
• The MPE can operate as an unclassified system or a classified system. • Although the Framework software is being co-developed among DoD components, MPS is not a joint program. Each Service tests and fields its own aircraft-specific MPEs. • The representative test platform for Joint Mission Planning System – Air Force (JMPS-AF) Increment IV mission planning functionality is the E-8 MPE. Mission Aircrews use MPS to conduct detailed mission planning to support the full spectrum of missions, ranging from simple training to complex combat scenarios. Aircrews save the required aircraft, navigation, threat, and weapons data on a data transfer device that they load into their aircraft before flight. Major Contractors • Northrop Grumman – Carson City, California • Boeing – St. Louis, Missouri • TYBRIN Corporation – Fort Walton Beach, Florida
transfer failures. Following Air Force regression testing of a revised E-8 MPE Version 1.3 in August 2012, AFOTEC re-accomplished IOT&E in its entirety in accordance with the DOT&E-approved IOT&E plan. • AFOTEC executed IOT&E of the E-8 MPE Version 1.3 from October through November 2012 to re-evaluate the operational
MPS 289
A i r F o r c e P RO G R A M S effectiveness, suitability, and mission capability of the E-8 MPE. • AFOTEC conducted the 2012 IOT&E due to the discovery of deficiencies during the 2011 IOT&E of the E-8 MPE Version 1.0 that led to termination prior to completion. Problems identified in the earlier IOT&E of Version 1.0 included: navigational functionality; magnetic variation computation; unreliable system set up/installation; and excessive time needed for routine maintenance. Assessment • In April 2013, DOT&E published a MAIS report concluding that E-8 MPE Version 1.3 was operationally effective and operationally suitable. Significant findings include: - Aircrews are able to plan all representative missions well within the required 4-hour time period, and the critical deficiencies identified during the 2011 IOT&E have been corrected. - The system meets the user defined Key Performance Parameters to include flight route creation and manipulation; mission planning time; data transfer device operations; and data exchanges from JMPS workstations to the aircraft. - Manual workarounds are required to overcome some uncorrected shortfalls that detract from overall capability. These shortfalls include the inability to-▪▪ Calculate take-off and landing data ▪▪ Automatically plan in-flight orbits ▪▪ Generate correct printed flight plan materials for routes north of 60 degrees latitude ▪▪ Implement the MPE’s vertical vector obstruction data terrain avoidance planning tools
290 MPS
• E-8 MPE Information Assurance shortfalls identified in earlier 2011 DOT&E MAIS reporting remain uncorrected. These deficiencies include the following: - Missing critical software patches - Outdated anti-virus signature definitions - Poor printer authentication - Default guest accounts allowed network access - Internal system communication processes accessible with anonymous credentials - Remotely accessible unauthorized registry paths - Planning computer Basic Input and Output System allowing booting from a Universal Serial Bus device Recommendations • Status of Previous Recommendations. The Air Force is addressing the previous recommendations. However, the Air Force did not automate take-off and landing data capabilities into the MPE tested during the 2012 IOT&E. • FY13 Recommendations. The Air Force should: 1. Address the shortfalls identified in the 2012 IOT&E to enable the E-8 MPE to compute automated take-off and landing data calculation; automatically plan in-flight orbits; correct deficiencies in the E-8 MPE vertical vector obstruction data terrain avoidance planning tools; allow importing threat information with ease; reduce lengthy software installation time; and improve training for JMPS personnel and users 2. Address the uncorrected Information Assurance shortfalls identified in the 2011 DOT&E JMPS E-8 MPE MAIS report.
A i r F o r c e P RO G R A M S
MQ-9 Reaper Armed Unmanned Aircraft System (UAS) Executive Summary • The MQ-9 Reaper Unmanned Aircraft System (UAS) continues to support ongoing global combat operations with primary programmatic focus on production and delivery of Remotely Piloted Aircraft (RPA) and incorporation of evolving and emergent sensor and system technologies outside of the MQ-9 baseline program of record. • The MQ-9 program notified the Secretary of the Air Force of an Acquisition Program Baseline (APB) breach in May 2013 due to the inability to meet the program of record development schedule. Ongoing schedule challenges, combined with RPA production emphasis, increase the likelihood that the MQ-9 UAS will complete the delivery of planned RPAs under low-rate initial production. • The program will not be able to execute the planned 2014 FOT&E of the final configuration of the Increment One system consisting of the Block 5 RPA, Block 30 Ground Control Station (GCS), and Operational Flight Program (OFP) 904.6 due to delays in software and technical order development. • Ongoing developmental challenges delayed operational testing and subsequent fielding of enhanced baseline program capabilities to operational MQ-9 units in FY13 including OFP 904.2, and GBU-38 Joint Direct Attack Munition (JDAM). Challenges are likely to persist in the long-term and significantly delay the operational testing and fielding of the final configuration of the Increment One system. • Air Force Air Combat Command (ACC) began a Force Development Evaluation (FDE) of OFP 904.2 in July 2013. Operational testing of this software OFP will continue through early FY14. System • The MQ-9 Reaper UAS is a remotely-piloted, armed, air vehicle that uses optical, infrared, and radar sensors to locate, identify, target, and attack ground targets. - The MQ-9 RPA is a medium-sized aircraft that has an operating ceiling up to 50,000 feet, an internal sensor payload of 800 pounds, an external payload of 3,000 pounds, and an endurance of approximately 14 hours. - The GCS commands the MQ-9 RPA for launch, recovery, and mission control of sensors and weapons. C-band line‑of-sight datalinks are used for RPA launch and recovery operations, and Ku-band satellite links are used for RPA mission control.
• The MQ-9 RPA carries AGM-114, Hellfire II anti-armor precision laser-guided missiles and GBU-12, 500-pound laser guided bombs. • The Air Force is using an evolutionary acquisition approach for meeting Increment One Capability Production Document (CPD) requirements, with Block 1 and Block 5 RPAs and Block 15 and Block 30 GCSs. • The Air Force is currently fielding the Block 1 RPA and the Block 15 GCS. • The Air Force designed the Block 5 RPA to incorporate improved main landing gear, an upgraded electrical system with more power, an additional ARC-210 radio, encrypted datalinks, a redesigned avionics bay and digital electronic engine control system, the BRU-71 bomb rack, high-definition video, and upgraded software to allow the two-person aircrew to operate all onboard systems. Mission • Combatant Commanders use the MQ-9 onboard sensors and weapons to conduct armed reconnaissance and pre-planned strikes. Units equipped with MQ-9s can find, fix, track, target, engage, and assess critical emerging targets (both moving and stationary). • MQ-9 units can also conduct aerial intelligence gathering, reconnaissance, surveillance, and target acquisition for other airborne platforms. Major Contractor General Atomics Aeronautical Systems Inc. – San Diego, California
MQ-9 Reaper 291
A i r F o r c e P RO G R A M S Activity • The Air Force conducted MQ-9 testing in accordance with the DOT&E-approved Test and Evaluation Master Plan and test plan. • The MQ-9 program notified the Secretary of the Air Force of an APB breach in May 2013 due to the inability to meet the program’s development schedule. The program will not be able to execute the planned 2014 FOT&E due to delays in software and technical data development. • ACC began the FDE of OFP 904.2 in July 2013 on the Block 1 RPA to test improvements to optical and infrared sensor target location accuracy, establish a baseline measurement of radar target location accuracy, and evaluate system user interface improvements. The FDE will continue into early FY14. • Air Force Special Operations Command (AFSOC) executed a limited evaluation of OFP 904.4 in September 2013 in order to deliver a limited capability of encrypted high-definition full motion video transmission to remote video terminal-equipped ground units with Video Oriented Transceiver for Exchange of information. The Air Force completed risk reduction demonstration flights of the Block 5 RPA and Block 30 GCS in FY13; however, planned formal developmental testing did not begin as planned in FY13. Formal Block 5 RPA and Block 30 GCS developmental testing is projected to begin in early FY14. • DOT&E rescinded the 2009 GBU-38 500-pound JDAM FDE plan in February 2013 due to lack of progress in maturing software capabilities to support an operational evaluation with the current MQ-9 OFPs. AFOTEC will test JDAM during FOT&E of the MQ-9 Increment One system. • Significant programmatic and developmental delays caused by software maturity challenges, technical data development, and competing schedule priorities for non-program of record capabilities continued to delay the program test schedule. Assessment • The MQ-9 program continues to face systemic challenges in prioritizing and maturing software OFPs and developing technical order data to meet development and fielding timelines for the MQ-9 Increment One program of record. The lack of a program Integrated Master Schedule to support the development of capabilities continues to exacerbate these difficulties. As in previous years, such challenges significantly extended the time to complete development of planned software upgrades (OFPs 904.2 and 904.4). The planned FY12 OFP 904.2 FDE test did not begin until late FY13. OFP 904.4 development delays led the Air Force to cancel planned operational testing and fielding within the ACC fleet, and defer incorporation of intended capabilities to future OFP 904.6.
292 MQ-9 Reaper
• Development, operational testing, and fielding of Increment One program of record capabilities will likely experience continued delays until the program is able to better prioritize and control maturation of these capabilities in accordance with a predictable schedule. Ongoing schedule challenges, combined with RPA production emphasis, increase the likelihood that the MQ-9 UAS will complete the delivery of all planned MQ-9 RPAs under low-rate initial production. FOT&E of the Increment One UAS configuration, originally planned for 2013, will likely be delayed several years beyond FY14. • The Air Force intends to fulfill the MQ-9 Increment One CPD requirements with a final UAS configuration consisting of the Block 5 RPA, Block 30 GCS, and OFP 904.6. AFOTEC will conduct formal operational testing of the final MQ-9 Increment One UAS. This operational testing will assess Increment One UAS effectiveness, suitability, mission capabilities, and satisfaction of CPD key performance parameters. • AFSOC demonstrated the successful transmission of encrypted, high-definition full motion video from the RPA to remote video terminal-equipped ground units in support of urgent AFSOC capabilities needs. AFOTEC will conduct formal evaluation of full motion video transmission during FOT&E of the MQ-9 Increment One system. • As has been the case since FY11, Information Assurance (IA) vulnerabilities and deficiencies are not well characterized because the Air Force has only completed limited IA testing on the MQ-9 system. Currently, the system is operating under an Interim Authority to Test, pending full system IA testing. Recommendations • Status of Previous Recommendations. In FY13, the Air Force made progress toward, but did not satisfy, the FY12 recommendation to complete the development of the Increment One UAS hardware and software to support FOT&E of the Increment One system. The Air Force also did not satisfy the outstanding FY11 recommendations to complete the JDAM FDE and conduct IA testing. • FY13 Recommendations. The Air Force should: 1. Complete the MQ-9 Increment One UAS Integrated Master Schedule. 2. Complete the development of the Increment One UAS hardware and software to support FOT&E of the Increment One system. 3. Complete the development of the GBU-38 JDAM capability for MQ-9 and test it during the FDE or FOT&E. 4. Complete IA vulnerability testing and correct or mitigate any deficiencies prior to FOT&E.
A i r F o r c e P RO G R A M S
RQ-4B Global Hawk High-Altitude Long-Endurance Unmanned Aerial System (UAS) Executive Summary Block 30 • Since the RQ-4B Block 30 IOT&E in 2011, which also served as the IOT&E event for the Airborne Signals Intelligence Payload (ASIP) program, the Air Force has corrected most RQ-4B air vehicle reliability and availability problems and implemented a limited number of previously planned system improvements. However, due to the decision to retire this system, the Air Force reduced developmental test activities and has not conducted a comprehensive FOT&E to verify correction of all major IOT&E deficiencies. As a result, currently fielded RQ-4B Block 30 systems continue to operate with some operational performance and ASIP signals intelligence (SIGINT) mission deficiencies identified during IOT&E. Current program plans call for limiting future test activity to the minimum necessary to sustain and support fielded systems. • The Air Force has not completed all ASIP corrective action plans or conducted follow-on operational testing to verify improved ASIP SIGINT mission capabilities. No formal plan to re-evaluate documented ASIP system performance deficiencies has been established, even though the Air Force continues to acquire and field additional ASIP production units. The Air Force is currently re-evaluating options to conduct ASIP FOT&E on either the RQ-4B Block 30 or the U-2 Dragon Lady. Block 40 • In August 2013, DOT&E published the RQ-4B Block 40/ Multi-Platform Radar Technology Insertion Program (MP-RTIP) Early Fielding Report that provided an assessment of system capability to support U.S. Central Command (USCENTCOM) early fielding requirements for surveillance of vehicle ground moving targets. This report concluded that RQ-4B Block 40/MP-RTIP early operational capabilities are limited, but adequate to provide additional near real-time vehicle ground moving target capabilities necessary to support the USCENTCOM early fielding concept of employment. The Air Force deployed two RQ-4B Block 40 systems to the USCENTCOM operating area in September 2013. • Based on operational testing conducted to support early fielding, the RQ-4B Block 40 system provides an effective vehicle moving target and detection capability at short to medium ranges. The system is interoperable with interim USCENTCOM command and control networks and target data dissemination architecture. For early fielding operations, operational units are capable of generating long-endurance sorties at the planned operational tempo of
two to three sorties per week using two aircraft. However, contractor maintenance and supply support is required to compensate for immature system-level reliability, maintenance training, documentation, and logistics support systems. • Although not required for USCENTCOM early fielding, early operational test results indicate that RQ-4B Block 40/MP-RTIP synthetic aperture radar (SAR) stationary target imagery capabilities are immature and do not currently meet established operational requirement thresholds for image resolution. MP-RTIP operator displays and control interfaces are also immature, which significantly increases operator workload during target-intense operations. During operational testing, frequent MP-RTIP sensor faults required sensor operators to halt intelligence collection operations to reset or restart the system. Resulting sensor downtime reduced on-station intelligence collection time by 23 percent. The demonstrated sensor availability rate of 77 percent falls short of the 90 percent availability expected at system maturity. The Air Force is conducting additional development and test activities to improve performance in these areas prior to the planned FY14 IOT&E.
System • The RQ-4 Global Hawk is a remotely-piloted, high-altitude, long-endurance airborne Intelligence, Surveillance, and Reconnaissance system that includes the Global Hawk unmanned air vehicle, various intelligence and
RQ-4B Global Hawk 293
A i r F o r c e P RO G R A M S communications relay mission payloads, and supporting command and control ground stations. • The RQ-4B Global Hawk Block 30 system is equipped with a multi-intelligence payload that includes both the Enhanced Integrated Sensor Suite (EISS) imagery intelligence payload and the ASIP signals intelligence sensor. • The RQ-4B Global Hawk Block 40 system is equipped with the MP-RTIP SAR payload designed to simultaneously collect imagery intelligence on stationary ground targets and track ground moving targets. Mission Commanders use RQ-4 Global Hawk reconnaissance units to provide high-altitude, long-endurance intelligence collection capabilities or theater communications relay capabilities to supported commanders. • Operators collect imagery and signals data in order to support ground units and to identify intelligence essential elements
Activity Block 30 • Since RQ-4B Block 30 IOT&E completion and initial fielding in 2011, the Air Force has implemented some minor system upgrades and deficiency corrections. -- In March 2013, the Air Force completed a limited force development evaluation to support fielding of previously planned air vehicle and sensor upgrades and a new satellite communications link. -- Due to the decision to retire the RQ-4B Block 30 system, the Air Force halted planning for a comprehensive RQ‑4B Block 30 FOT&E intended to evaluate correction of all major RQ-4B and ASIP SIGINT mission capability shortfalls identified during IOT&E. • The Air Force is continuing to acquire and pursue upgrade programs for the ASIP sensor. In addition, the Air Force is planning to modify some RQ-4B Block 30 ASIP sensors for transfer and deployment on the U-2 Dragon Lady in FY14. Block 40 • The Air Force continued to execute the RQ-4B Block 40/MP-RTIP test program leading to a projected Initial Operational Capability in FY15. In FY13, the program successfully completed initial MP-RTIP sensor integration and performance developmental testing. Additional performance and interoperability testing will continue into FY14. Planning is in progress for the RQ-4B Block 40/MP-RTIP IOT&E event in 4QFY14. Based on current production and delivery schedules, the Air Force will deliver all 11 RQ-4B Global Hawk Block 40 systems to Grand Forks AFB, North Dakota, prior to IOT&E. • In response to a USCENTCOM urgent operational need request, the Air Force accelerated the fielding of two RQ-4B Block 40/MP-RTIP aircraft to provide additional surveillance and tracking capabilities for vehicle ground moving targets. In March 2013, the Air Force Operational
294 RQ-4B Global Hawk
of information for theater commanders. Units equipped with RQ-4B Global Hawk use line-of-sight and beyond lineof-sight satellite datalinks to control the Global Hawk system and transmit collected intelligence data. • Distributed intelligence processing, exploitation, and dissemination systems receive intelligence data directly from the air vehicle or from the Global Hawk ground station via intelligence data transmission systems. • Ground-based intelligence analysts exploit collected imagery and signals information to provide intelligence products in support of theater operations. • Global Hawk can also provide imagery intelligence directly to forward-based personnel through direct line-of-sight datalink systems.
Major Contractor Northrop Grumman Aerospace Systems, Strike and Surveillance Systems Division – San Diego, California
Test and Evaluation Center (AFOTEC) conducted a DOT&E-approved RQ-4B Global Hawk Block 40 Operational Utility Evaluation (OUE) to evaluate early system capabilities for this purpose. DOT&E and AFOTEC published RQ-4B Global Hawk Block 40 Early Fielding Reports based on this test in August 2013. The Air Force deployed two aircraft to support USCENTCOM operations in September 2013. • Due to continuing uncertainty regarding the future of the RQ-4B program, USD(AT&L) again deferred the originally planned June 2011 RQ-4B Global Hawk Block 40 Milestone C decision to FY14 and the Joint Staff did not proceed with approval of the RQ-4B Block 40 Capabilities Production Document (CPD). As a result, the Air Force deferred completion of a program Test and Evaluation Master Plan intended to define the RQ-4B Block 40/MP‑RTIP developmental and operational test and evaluation strategy. Assessment Block 30 • Since the combined RQ-4B Block 30 IOT&E and ASIP IOT&E event in 2011, the Air Force has corrected most RQ-4B air vehicle reliability and availability problems and implemented a limited number of previously planned system improvements. However, due to the decision to retire this system, the Air Force reduced developmental test activities and has not conducted a comprehensive developmental test and FOT&E to verify correction of all major deficiencies identified during the 2011 IOT&E. As a result, fielded RQ-4B Block 30 systems continue to operate with some operational performance and ASIP SIGINT mission deficiencies identified during IOT&E. Current program plans call for limiting future RQ-4B Block 30 test
A i r F o r c e P RO G R A M S activity to the minimum necessary to sustain and support maintenance and supply support is required to fielded systems. No additional RQ-4B Block 30 follow-on compensate for immature system-level reliability, operational testing is currently planned. maintenance training, documentation, and logistics • The Air Force has not completed all post-IOT&E ASIP support systems. corrective action plans or conducted follow-on operational • Although not required for USCENTCOM early fielding, testing to verify improved ASIP SIGINT mission the March 2013 OUE results indicate that RQ-4B capabilities. No formal plan to re-evaluate previously Block 40/ MP‑RTIP SAR stationary target imagery identified ASIP system performance deficiencies has been capabilities are immature and do not currently meet established, even though the Air Force continues to acquire established operational requirement thresholds for image and field additional ASIP production units and pursue resolution. MP-RTIP operator control interfaces, sensor incremental system upgrades. The Air Force is currently stability, and sensor reliability are also immature. During re-evaluating options to conduct the required ASIP SIGINT OUE missions, frequent MP-RTIP sensor faults required mission FOT&E on either the RQ-4B Block 30 or the sensor operators to halt intelligence collection operations U-2 Dragon Lady. Conducting ASIP FOT&E on the U-2 to reset or restart the system. Resulting sensor downtime may be a viable option since the Air Force is planning to reduced on-station intelligence collection time by transfer additional RQ-4B Block 30 ASIP sensors to the U-2 23 percent. The demonstrated sensor availability rate of platform. 77 percent falls short of the 90 percent availability expected Block 40 at system maturity. Sensor stability problems, combined • In August 2013, DOT&E published the RQ-4B with identified sensor control and interface deficiencies, Block 40/ MP- RTIP Early Fielding Report based on test significantly increase operator workload in target-dense results from the RQ-4B Block 40 OUE. This report operating environments. The Air Force is conducting provided an assessment of system capability to support additional development and test activities to improve USCENTCOM early fielding requirements for surveillance performance in these areas prior to the planned FY14 of vehicle ground moving targets and an in-progress IOT&E. assessment of system progress toward full maturity and IOT&E readiness. The Air Force deployed two RQ-4B Recommendations • Status of Previous Recommendations. The Air Force Block 40 systems to the USCENTCOM operating area in September 2013. conducted an operational test to support early fielding of • Based on the March 2013 OUE results, RQ-4B the RQ-4B Block 40 systems to support USCENTCOM Block 40/ MP- RTIP early operational capabilities are operational requirement. Due to the decision to retire the limited, but adequate to provide additional near real-time RQ-4B Block 30 system, the Air Force did not address the vehicle ground moving target capabilities necessary to following previous recommendations. 1. Develop or implement a comprehensive development and support the USCENTCOM early fielding concept of employment. FOT&E strategy to complete correction of RQ-4B Block 30 -- The system provides an effective vehicle ground moving system deficiencies. target and detection capability at short to medium ranges. 2. Establish a plan to conduct an ASIP sensor FOT&E on -- RQ-4B Block 40 air vehicle and ground station either the RQ-4B Block 30 or U-2 to verify correction of performance is similar to previously fielded RQ-4B ASIP SIGINT operational capability deficiencies identified Block 30 systems and compatible with the planned during IOT&E. USCENTCOM operating environment. 3. Complete an RQ-4B Block 40 Test and Evaluation Master -- Air vehicle long endurance flight capabilities exceed Plan to guide developmental and operational testing of this 30 hours. system. -- Adverse weather operations remain limited due to a lack • FY13 Recommendations. The Air Force should: of anti-ice/de-icing systems and real-time severe weather 1. Plan and conduct an RQ-4B Block 40/MP-RTIP IOT&E detection and avoidance capabilities. event to evaluate delivered mission capabilities. -- The system is interoperable with interim USCENTCOM 2. Identify and correct persistent RQ-4B Block 40/MP-RTIP command and control networks and target data sensor stability problems. Operational and developmental dissemination architecture. testing has consistently identified sensor instability as a -- For early fielding operations, operational units are significant operational performance shortfall since the initial capable of generating long-endurance sorties at the AFOTEC MP-RTIP Operational Assessment in 2008. 3. Identify and correct RQ-4B Block 40/MP-RTIP SAR image planned operational tempo of two to three sorties per week using two aircraft. However, contractor resolution performance prior to IOT&E.
RQ-4B Global Hawk 295
A i r F o r c e P RO G R A M S
296
A i r F o r c e P RO G R A M S
Space-Based Infrared System (SBIRS) Executive Summary • The Air Force Operational Test and Evaluation Center (AFOTEC) conducted an Operational Utility Evaluation (OUE) of the Space-Based Infrared System (SBIRS) Effectivity 5 from September 27, 2012, through October 11, 2012. Testing included the SBIRS ground architecture, the first SBIRS satellite in geosynchronous orbit (GEO-1), two hosted infrared payloads in Highly Elliptical Orbit (HEO), and legacy Defense Support Program (DSP) assets. DOT&E published a classified test report in December 2012. • Along with two Air Force Space Command (AFSPC) operational trial periods, the OUE informed Air Force operational acceptance of Effectivity 5 in May 2013, and the National Geospatial Intelligence Agency’s acceptance of GEO-1 data for technical intelligence. • SBIRS Effectivity 5 is operationally effective and suitable since the Air Force resolved an open deficiency identified in the December 2012 classified DOT&E report. System • The SBIRS program provides infrared sensing from space to support DoD and other user organizations. SBIRS will replace the legacy DSP ground station and satellites and improve upon DSP timeliness, accuracy, and detectable threats. The SBIRS program is being developed in two system increments. - Increment 1 uses the SBIRS Control Segment and User Segment, operating with DSP satellites, to provide current military capability. Initial Operational Capability for Increment 1 was attained December 18, 2001, consolidating the operations of the DSP and Attack and Launch Early Reporting to Theater missions. - Increment 2 includes a space segment consisting of two hosted payloads in HEO and four satellites in geosynchronous orbit. Increment 2 also provides new ground system software and hardware for consolidated data processing across all sensor families. - The contractor is delivering Increment 2 capabilities in phases, with both ground system software and on-orbit
Activity • AFOTEC conducted a dedicated OUE from September 27, 2012, through October 11, 2012. AFOTEC conducted the testing in accordance with the DOT&E‑approved Test and Evaluation Master Plan and test plan. The OUE was executed in conjunction with AFSPC’s initial operational trial period. AFSPC conducted an additional trial period from April 15, 2013, through May 17, 2013.
assets, which require several dedicated test and evaluation activities. Two HEO payloads and two SBIRS GEO satellites are now on-orbit. Additional GEO satellites will continue to launch to complete the constellation over the next few years. Concurrently, the ground system replacement will proceed in blocks, completing in 2018. Mission The Joint Functional Component Command for Space, a component of U.S. Strategic Command (USSTRATCOM), employs SBIRS to provide reliable, unambiguous, timely, and accurate missile warning and missile defense information to the President of the United States, the Secretary of Defense, Unified Commanders, and other users, as well as to provide technical intelligence and battlespace awareness to those same users. Major Contractors • Lockheed Martin Space Systems – Sunnyvale, California • Northrop Grumman Electronic Systems – Azusa, California • Lockheed Martin Information Systems and Global Solutions – Denver, Colorado
- AFOTEC tested Effectivity 5, which includes the SBIRS ground architecture, GEO-1, two hosted infrared payloads in HEO, and legacy DSP assets. - DOT&E published a classified OUE test report in December 2012. The report informed the Air Force decision to hold an additional trial period and the subsequent Air Force operational acceptance
SBIRS 297
A i r F o r c e P RO G R A M S of Effectivity 5 on May 17, 2013. The National Geospatial- Intelligence Agency also utilized the operational test results for its acceptance of GEO-1 data for technical intelligence on April 12, 2013. On August 23, 2013, USSTRATCOM certified the GEO-1 space and ground systems for Integrated Theater Warning/Attack Assessment. • The Air Force successfully launched SBIRS GEO-2 on March 19, 2013, completed on-orbit check-out, tuning, and trial periods, and operationally accepted the satellite on November 25, 2013. • An update to the Enterprise Test and Evaluation Master Plan is in coordination to address future testing of Increment 2. Finalizing this document has been contingent upon a ground architecture definition, a concept of operations, and operational requirements for each key SBIRS Increment 2 delivery.
Assessment • SBIRS Effectivity 5 is operationally effective. Integration of GEO-1 into the operational constellation improved accuracy of both strategic and theater missile warning mission data and did not degrade overall mission performance. SBIRS also demonstrated improved performance against the missile defense mission. SBIRS support to the technical intelligence and battlespace awareness missions was functional and effective. • There were no major problems observed during the integrated and operational test periods. The SBIRS enterprise system accomplished its strategic and theater missile warning missions, successfully detecting and reporting all missile events during both real-world and simulation scenarios during these test periods.
298 SBIRS
• The SBIRS GEO-1 scanning sensor payload is meeting accuracy and sensitivity requirements, based on developmental and integrated test activities. It is at least as capable as legacy DSP sensors, while providing detection over a given location twice as frequently. This increased revisit rate is operationally significant as it enables the ability to determine target missile type with higher confidence by providing more data points for analysis during the target missile’s powered flight. • SBIRS Effectivity 5 is operationally suitable since the Air Force resolved the open deficiency identified in the classified DOT&E OUE report. The Air Force continues to address problems identified during the OUE with the overall system, technical intelligence missions, and specific Information Assurance postures. • The classified OUE test report includes more information on additional observations, detailed findings, and recommendations. Recommendations • Status of Previous Recommendations. Of nine previous recommendations contained in the FY12 Annual Report and the December 2012 classified DOT&E OUE report, the Air Force satisfactorily addressed one, is in the process of addressing five, and made insufficient progress with three. The Air Force should still: 1. Clarify and revalidate the intended use case for SBIRS support to missile defense operations. 2. Confirm user format requirements for intelligence products and develop SBIRS to deliver to that need. 3. Verify that operational unit procedural changes have remedied configuration-related reliability concerns. • FY13 Recommendations. None.
Ballistic Missile Defense Systems
Ballistic Missile Defense Systems
BALLISTIC MISSILE DEFENSE SYSTEMS
Ballistic Missile Defense System (BMDS)
Executive Summary • The Ballistic Missile Defense System (BMDS) capability against theater threats increased during the fiscal year. The deployment of Command and Control, Battle Management, and Communications (C2BMC) S6.4 MR2 software to multiple Combatant Commands, a Terminal High-Altitude Area Defense (THAAD) battery to Guam, and an AN/TPY-2 (Forward-Based Mode [FBM]) radar to U.S. Central Command (USCENTCOM) provided capabilities in several theaters against short-, medium-, and intermediate-range ballistic missile threats. The BMDS capability against strategic threats did not increase. • During FY13, the Missile Defense Agency (MDA) conducted system-level flight tests. - Flight Test Integrated-01 (FTI-01) was an important milestone in BMDS testing because, for the first time, three missile defense weapon elements and an external sensor operated in the same theater engaging a small raid of ballistic missiles and air-breathing threats. - The Flight Test Operational (FTO-01) test mission followed the FTI-01 test mission with a full demonstration of a layered upper-tier regional/theater BMDS defense.
Full assessment of the FTO-01 test mission data with respect to the effectiveness, suitability, and interoperability of the BMDS is ongoing. • The MDA has restructured the Integrated Master Test Plan (IMTP) so that testing in support of each phase of the European Phased-Adaptive Approach has a dedicated chapter in the document. The test schedule is based on input from the MDA, the operational testers, and the Combatant Commands. System • BMDS is a distributed system currently comprised of five elements (four shooter elements and one command and control element) and five sensor systems (four radar systems and one space-based system). Elements • Aegis Ballistic Missile Defense (BMD) (shooter) • C2BMC (command and control) • Ground-Based Midcourse Defense (GMD) (shooter) • Patriot (shooter) • THAAD (shooter)
BMDS
299
BALLISTIC MISSILE DEFENSE SYSTEMS Sensors • Aegis BMD AN/SPY-1 Radar • COBRA DANE Radar • Upgraded Early Warning Radars • AN/TPY-2 (FBM) Radar • Space-Based Infrared System/Defense Support Program (SBIRS/DSP) • Sea-Based X-band (SBX) Radar (primarily a test asset that can be operationally deployed as needed)
USCENTCOM, and USPACOM also use the C2BMC to provide sensor management of theater AN/TPY-2 (FBM) radars across the full mission engagement space. • The Army employs Patriot to provide theater defense for deployed forces against short- and medium-range threats.
Mission • The U.S. Strategic Command (USSTRATCOM) synchronizes operational-level global missile defense planning and operations support for the DoD. When directed, it provides alternate missile defense execution. • U.S. Northern Command (USNORTHCOM), U.S. Pacific Command (USPACOM), U.S. European Command (USEUCOM), and USCENTCOM employ the assets of the BMDS to defend U.S. territory, deployed forces, friends, and allies against ballistic missile threats of all ranges. Current capability permits limited defense of U.S. territory against simple ballistic missile threats and defending deployed forces, friends, and allies from theater-level ballistic missile threats. • USSTRATCOM, USNORTHCOM, USEUCOM, USCENTCOM, and USPACOM use the C2BMC element of the BMDS to maintain situational awareness. USEUCOM,
Major Contractors • BMDS Integration: The Boeing Company, Network and Space Systems – Huntsville, Alabama • Aegis BMD and Aegis Ashore: Lockheed Martin Corporation, Mission Systems and Training – Moorestown, New Jersey • C2BMC: Lockheed Martin Corporation, Information Systems and Global Solutions – Gaithersburg, Maryland • GMD: The Boeing Company, Network and Space Systems – Huntsville, Alabama • Patriot: Raytheon Company, Integrated Defense Systems – Tewksbury, Massachusetts • THAAD: Lockheed Martin Corporation, Missile and Fire Control – Dallas, Texas • Sensors: Raytheon Company, Integrated Defense Systems – Tewksbury, Massachusetts; Lockheed Martin Corporation, Mission Systems and Training – Moorestown, New Jersey; Lockheed Martin Corporation, Space Systems – Sunnyvale, California; Northrop Grumman Corporation, Aerospace Systems – Redondo Beach, California
Activity • The MDA conducted system-level flight testing during FY13 in accordance with the DOT&E-approved IMTP. System‑level ground testing is reported in the individual article on C2BMC later in this section. • The MDA conducted FTI-01 in October 2012, which included Aegis BMD and Patriot engagements of short-range ballistic missiles while defending against cruise missile attacks, and a THAAD first time engagement of a medium-range ballistic missile. The Aegis BMD and THAAD engagements were designed for near-simultaneous intercept. SBIRS/DSP provided early warning and an AN/TPY-2 (FBM) radar provided acquisition cues via C2BMC. Soldiers performed command and control functions from the Air and Space Operations Center at Hickam AFB, Hawaii. • The BMDS Operational Test Agency and the MDA conducted FTO-01 in September 2013. The FTO-01 test mission was designed to demonstrate a layered upper-tier regional/theater BMDS defense against a raid of two simultaneously-launched and threat-representative medium-range ballistic missiles threatening a shared defended area. The Aegis BMD element engaged the first target while the THAAD element engaged the second target and provided a secondary engagement capability against the first target. • The MDA, in collaboration with DOT&E, updated the FY12 version of the IMTP to incorporate BMDS element
maturity, program modifications, and fiscal constraints. In parallel, the MDA continued efforts to align the IMTP with BMDS modeling and simulation verification, validation, and accreditation data requirements. • During FY13, the Lethality Focus Group continued collaboration to identify lethality data gaps for all BMDS weapon elements. Further, the Group began reviewing the performance of “first principles” physics-based software tools for potential use in predicting the lethality of BMDS intercepts. • During FY13, the MDA conducted numerous war games and exercises that enhanced Combatant Command BMD readiness and increased confidence in the deployed elements of the BMDS. • To support FTO-01 directly, plus future operational testing scheduled in the IMTP, the MDA completed approximately 80 improvement and modernization efforts at the Reagan Test Site (Kwajalein Atoll) and Wake Island. These efforts included power, water, and fuel infrastructure; lodging and office accommodations for system operators and flight test personnel; lightning protection for deployed test assets; instrumentation improvements and security facilities construction; test site upgrades to accommodate unique test support equipment; and installation of communications infrastructure to support increased mission data and voice networks.
300 BMDS
BALLISTIC MISSILE DEFENSE SYSTEMS Assessment • Significant to a system-level characterization of BMDS, the MDA conducted the first integrated flight test that included Aegis BMD, Patriot, and THAAD, as well as C2BMC and an AN/TPY-2 (FBM). FTI-01 included basic system-level integration, but not layered defense. The weapon elements operated independently of one another, although they did exchange track data with each other and received cues from the AN/TPY-2 (FBM) radar via C2BMC. In spite of the test limitations, FTI-01 was an important milestone in BMDS testing because, for the first time, three missile defense weapon elements and an external sensor operated in the same theater engaging a small raid of ballistic missiles and air‑breathing threats. • The FTO-01 test mission demonstrated an integrated and layered upper-tier regional/theater BMDS defense. The initial assessment of data indicated that the simultaneous launch of the two medium-range ballistic missile targets occurred. The Aegis BMD intercepted one target and THAAD intercepted the other target while simultaneously engaging the debris from the Aegis BMD engagement. Full assessment of FTO-01 test mission data with respect to the effectiveness, suitability, and interoperability of the BMDS is ongoing. • The MDA implemented significant improvement for tracking modeling and simulation verification, validation, and accreditation completion over FY13. This tracking capability is the first step in adjusting the IMTP to better align with the overall modeling and simulation effort. The MDA developed a software tool, which correlates these key performance parameters, the BMDS mission threads, and the IMTP-approved test schedule. However, many of the models and simulations used in the ground tests remain with limited accreditation, which constrains performance assessment, thereby limiting quantitative assessments based on their results.
• Although the Lethality Focus Group has developed a plan of action to address BMDS lethality data voids, the MDA has made little progress in retiring them. • The BMDS capability against theater threats increased during the fiscal year. The deployment of C2BMC S6.4 MR2 software to multiple Combatant Commands, a THAAD battery to Guam, and an AN/TPY-2 (FBM) radar to USCENTCOM provided capabilities in several theaters against short-, medium-, and intermediate-range ballistic missile threats. These capabilities were demonstrated through ground testing. During FTI-01, sensors and weapon systems worked together to engage five theater-level targets. Initial results show that the FTO-01 test mission also will contribute significantly to the system-level body of knowledge. • The BMDS capability against strategic threats has not increased. The GMD program experienced a flight test failure in Flight Test, Ground-Based Interceptor-07 (FTG-07) where the Capability Enhancement I Exoatmospheric Kill Vehicle failed to separate from the third stage booster. A Failure Review Board has been convened to address this failure. Recommendations • Status of Previous Recommendations. The MDA satisfied the outstanding FY08 and FY09 recommendations. Any remaining recommendations specific to the BMDS elements can be found in the reports for those programs (i.e., the recommendation to repeat flight tests to verify root causes and Failure Review Board results for Aegis BMD and GMD flight test failures). Additionally, the MDA still needs to continue addressing the interoperability and command and control deficiencies uncovered during the GT-04 test campaign and FTI-01. Resolution of these deficiencies should be demonstrated through ground and/or flight testing • FY13 Recommendations. None.
BMDS 301
BALLISTIC MISSILE DEFENSE SYSTEMS
302
BALLISTIC MISSILE DEFENSE SYSTEMS
Aegis Ballistic Missile Defense (Aegis BMD) Executive Summary • In FY13, the Aegis Ballistic Missile Defense (BMD) program completed most of the combined developmental test (DT)/ operational test (OT) and IOT&E flight test program for the Aegis BMD 4.0 system and Standard Missile-3 (SM-3) Block IB guided missile. • The Aegis BMD program conducted five intercept missions in FY13 and one in early FY14. All but one resulted in successful intercepts. • During an integrated flight test of the Ballistic Missile Defense System (BMDS), an Aegis BMD 3.6.1 destroyer intercepted an anti-air warfare target using an SM-2 missile and failed to intercept a short-range ballistic missile target using an SM-3 Block IA guided missile. • Aegis BMD participated in the first system-level operational flight test conducted by the Missile Defense Agency (MDA). During the mission, an Aegis BMD 3.6.2e destroyer intercepted a medium-range ballistic missile target using an SM-3 Block IA guided missile. • Aegis BMD continued to improve interoperability with other BMDS elements and sensors during flight and ground testing in FY13. • Hardware-in-the-loop (HWIL) ground testing demonstrated Aegis BMD capability to contribute to theater, regional, and strategic-level defense missions spanning a range of ballistic missile defense scenarios. System • Aegis BMD is a sea-based missile defense system that employs the multi-mission shipboard Aegis Weapon System, with improved radar and new missile capabilities to engage ballistic missile threats. Capabilities of Aegis BMD include: - Computer program modifications to the AN/SPY-1 radar for long-range surveillance and track (LRS&T) of ballistic missiles of all ranges. - A modified Aegis Vertical Launch System, which stores and fires SM-3 Block IA and Block IB guided missiles (on select ships), and modified SM-2 Block IV guided missiles (on select ships). - SM-3 Block IA and Block IB guided missiles, which use a maneuverable kinetic warhead to accomplish midcourse engagements of short-, medium-, and intermediate-range ballistic missiles. - Modified SM-2 Block IV guided missiles, which provide terminal engagement capability against short‑range ballistic missiles and air-breathing threats.
• Aegis BMD is capable of performing autonomous missile defense operations and operations that exploit networked sensor information; it can send or receive cues to or from other BMDS sensors through tactical datalinks. Mission The Navy can accomplish three missile defense-related missions using Aegis BMD: • Defend deployed forces and allies from short- to intermediate‑range theater ballistic missile threats • Provide forward-deployed radar capabilities to enhance defense against ballistic missile threats of all ranges by sending cues or target track data to other elements of the BMDS • Provide all short- to long-range ballistic missile threat data to the Command and Control, Battle Management, and Communications (C2BMC) system for dissemination to Combatant Commanders’ headquarters to ensure situational awareness Major Contractors • Prime: Lockheed Martin Corporation, Mission Systems and Training – Moorestown, New Jersey • AN/SPY-1 Radar: Lockheed Martin Corporation, Mission Systems and Training – Moorestown, New Jersey • SM-3 Missile: Raytheon Company, Missile Systems – Tucson, Arizona
Aegis BMD 303
BALLISTIC MISSILE DEFENSE SYSTEMS Activity • In FY13, the Aegis BMD program completed most of the combined DT/OT and IOT&E flight test program for the Aegis BMD 4.0 system and SM-3 Block IB guided missiles, in accordance with the DOT&E-approved Integrated Master Test Plan. • Although the program completed FOT&E for the Aegis BMD 3.6.1 system in FY11, the program continued to use variants of the Aegis BMD 3.6 system (i.e., 3.6.1 and 3.6.2e) in BMDS-level tests in FY13 to assess system-level engagement capability and interoperability. • The Aegis BMD program conducted five intercept missions in FY13 and one in early FY14; five ballistic missile targets were intercepted, one anti-air warfare target was intercepted, and one ballistic missile target was not intercepted. - During Flight Test Integrated-01 (FTI-01) in October 2012, an Aegis BMD 3.6.1 destroyer simultaneously engaged a short-range simple separating ballistic missile target with an SM-3 Block IA guided missile and an anti-air warfare target with an SM-2 missile. The SM-2 missile successfully engaged its intended target, but the SM-3 Block IA guided missile failed to intercept. FTI-01 was the first integrated flight test with multiple firing elements (Aegis BMD, Terminal High-Altitude Area Defense [THAAD], and Patriot) engaging multiple ballistic missile and air-breathing targets in a realistic BMDS-level architecture. - In the Flight Test Standard Missile-20 (FTM-20) mission in February 2013, an Aegis BMD 4.0.2 cruiser intercepted a medium-range non-separating ballistic missile target with an SM-3 Block IA guided missile using remote data provided by Space Tracking and Surveillance System – Demonstrators. The FTM-20 intercept was the first performed with an Aegis BMD 4.0 ship’s fire control system set up with remote engagements authorized, and the first firing of an SM-3 Block IA guided missile from an Aegis BMD 4.0 ship. - During FTM-19 in May 2013, an Aegis BMD 4.0.2 cruiser intercepted a short-range complex separating ballistic missile target with an SM-3 Block IB guided missile. The FTM-19 engagement was the third successful intercept mission conducted with the Aegis BMD 4.0 system with an SM-3 Block IB guided missile, and the second combined DT/OT flight test for that system. In addition to the flight mission, the ship participated in a four-event multi-warfare exercise, including live fire events with an SM-2 against an air-breathing threat and guns against a high-speed maneuvering surface threat, to assess simultaneous BMD radar loading while exercising surface warfare, electronic warfare, undersea warfare, and anti-air warfare capabilities. - An Aegis BMD 3.6.2e destroyer participated as a shooter in Flight Test Operational-01 (FTO-01) in September 2013. During the mission, the ship intercepted a medium-range ballistic missile target with an SM-3 Block IA guided missile. FTO-01 was the first system-level operational flight test conducted by the MDA.
304 Aegis BMD
- During FTM-21 in September 2013, an Aegis BMD 4.0.2 cruiser intercepted a short-range complex separating ballistic missile target with the first of two salvo-fired SM-3 Block IB guided missiles. This was the first salvo firing of two SM-3 guided missiles against a live ballistic missile target in an Aegis BMD flight test. The FTM-21 engagement was the fourth successful intercept mission conducted with the Aegis BMD 4.0 system and SM-3 Block IB guided missiles, and the first Aegis BMD flight test designated as an IOT&E mission supporting a Full‑Rate Production decision for the SM-3 Block IB guided missile. - In the FTM-22 flight mission in October 2013, an Aegis BMD 4.0.2 cruiser intercepted a medium-range separating target with an SM-3 Block IB guided missile. The FTM-22 engagement was the fifth successful intercept mission conducted with the Aegis BMD 4.0 system with an SM-3 Block IB guided missile, and the second of two IOT&E flight missions. The FTM-22 engagement was the first intercept of a medium-range target with the Aegis BMD 4.0 system and an SM-3 Block IB guided missile. • In FY13, Aegis BMD ships or HWIL facilities participated in several flight and ground tests to assess Aegis BMD 3.6.1/3.6.2e and 4.0.1/4.0.2 system functionality and interoperability with the BMDS. - Ground Test Integrated-04e (GTI-04e) Part 1 in November 2012 tested the engagement capabilities of fielded and to-be-fielded missile defense elements and sensors against ballistic missiles of all ranges in an HWIL environment. Participants included Aegis BMD 3.6.2e and 4.0.2 (laboratory sites); C2BMC, Patriot, THAAD; Space‑Based Infrared System (SBIRS), AN/TPY-2 Forward-Based Mode (FBM), Joint Tactical Ground Station, and Arrow. - Fast Eagle 2 Increment 2 in February 2013 explored ballistic missile defense capabilities using laboratory and distributed assets for Aegis BMD 3.6.1, AN/TPY-2 (FBM), C2BMC, SBIRS, and Patriot. The event was designed to evaluate the capability of AN/TPY-2 (FBM) and the associated C2BMC to augment existing U.S. Central Command BMDS capability. Also, a primary focus of the test was the development of tactics, techniques, and procedures (TTPs) for Aegis BMD use of AN/TPY-2 (FBM) data and for the provision of AN/TPY-2 (FBM) track data to Aegis, THAAD, and Patriot. - Fast Assurance in February 2013 used laboratory assets of Aegis BMD 4.0.2, C2BMC, Ground-based Midcourse Defense (GMD), COBRA DANE, AN/TPY-2 (FBM), and SBIRS to demonstrate the interoperability between Aegis BMD and GMD fire control in relation to the Aegis BMD LRS&T mission. - GTI-04e Part 1a runs-for-the-record in October 2013 further explored theater/regional defense capabilities (beyond those tested in GTI-04e Part 1) using updated
BALLISTIC MISSILE DEFENSE SYSTEMS
-
-
-
- -
software builds for AN/TPY-2 (FBM), C2BMC, and THAAD, in addition to Aegis BMD 3.6.2e and 4.0.2. An Aegis BMD 4.0.2 cruiser participated in Flight Test Ground-Based Interceptor-07 (FTG-07) in July 2013. The cruiser successfully performed LRS&T duties in support of the GMD engagement against an intermediate-range ballistic missile target. FTG-07 was the first live-target LRS&T mission performed by an Aegis BMD 4.0 ship where the ship’s data were used to create GMD’s weapon task plan. The FTO-01 System Pre-Mission Test in July 2013 explored integrated engagement capability for Aegis BMD 3.6.2e and THAAD in an operationally relevant architecture using HWIL assets to reduce risks for the FTO-01 flight mission. Aegis BMD participated in the Fast Aim HWIL event in August 2013, which demonstrated the use of the Sea-Based X-band (SBX) radar in strategic defense scenarios. The test included laboratory assets of Aegis BMD, C2BMC, GMD Fire Control, AN/TPY-2 (FBM), SBX, and SBIRS. The MDA conducted a Maintenance Demonstration (M-Demo) in August 2013 using an Aegis BMD 4.0.2 ship to collect reliability, maintainability, and availability data. The MDA performed a set of warfighter simulations in an HWIL environment in August and September 2013 as part of the FTO-01 campaign. The warfighter used the simulations to explore and refine TTPs, and for training of operators for regional/theater engagement scenarios. Participants included Aegis BMD, THAAD, SBIRS, AN/TPY-2 (FBM), and C2BMC. Aegis BMD 3.6.1 participated in the FTI-01 System Post-Flight Reconstruction in September 2013, which was a BMDS HWIL-based event designed to provide data in support of modeling and simulation verification, validation, and accreditation efforts.
-
•
•
•
Assessment • In FY13, Aegis BMD demonstrated the capability to perform end-to-end engagements against complex separating short‑range and separating medium-range ballistic missiles with the Aegis BMD 4.0 system and SM-3 Block IB guided missiles. • Flight testing in FY13 exercised Aegis BMD 4.0 launch‑on‑remote and demonstrated the capability of the 4.0 system to fire deployed SM-3 Block IA guided missiles. • Test data from FY13, in combination with data collected during previous flight testing, suggest that overall Aegis BMD 4.0 Weapon System reliability is adequate for the midcourse defense mission against short- and medium-range ballistic missiles. However, the SM-3 Block IB third stage rocket motor (TSRM) has experienced flight test failures that require further investigation and the identification of the underlying root cause(s). • Aegis BMD 4.0’s participation in FTG-07 verified the system’s capability to perform LRS&T against long-range targets. However, the test highlighted the need to further explore and
•
•
•
refine TTPs for the transmission and receipt of Aegis BMD track data for use by GMD. With the completion of FTM-21 and FTM-22, the IOT&E flight testing phase for Aegis BMD 4.0 and SM-3 Block IB guided missiles is nearly complete. However, the program needs to complete Flight Test Other-18 (FTX-18) and planned HWIL testing of raid engagement capability and Information Assurance testing using accredited models and simulations in the test runs-for-the-record before an assessment of effectiveness and suitability can be made. Additionally, the program needs to test Aegis-Aegis, Aegis-THAAD, and Aegis‑Patriot engagement coordination; only the first of these three types of engagement coordination is planned for live‑target testing before the SM-3 Block IB Full-Rate Production decision in 4QFY14. The program has addressed and tested corrections for the SM-3 TSRM problems found in FTM-15 and FTM-16 Event 2. - The program re-designed the TSRM cold gas regulator in response to the FTM-15 anomalous TSRM behavior; the new cold gas regulator has now been flight tested five times without incident. - To correct the failure exhibited in the FTM-16 Event 2 TSRM energetic event, the program modified the TSRM’s inter-pulse delay time; the now greater minimum inter‑pulse delay has been exercised without incident in three flight tests and a number of ground-based static firings. During FTM-21, the second of two salvo-launched SM-3 Block IB guided missiles suffered a reliability failure of the TSRM during second pulse operations (the first missile had already achieved a successful intercept). The MDA has established a Failure Review Board to determine the root cause of this failure. A Failure Review Board concluded that the failure to intercept in FTI-01 was caused by a faulty memory chip in the SM-3 Block IA guided missile’s Inertial Measurement Unit (IMU). The specific brand of IMU with this problem is confined to a small fraction of fielded SM-3 Block IA guided missiles, and the program and U.S. Navy are working to mitigate any potential impact from those rounds. The faulty chip is not present in the IMU’s design for the SM-3 Block IB guided missile. An Aegis BMD 3.6.2e destroyer, using an SM-3 Block IA guided missile, successfully intercepted its medium-range ballistic missile target during FTO-01. A full assessment of FTO-01 test mission data with respect to the effectiveness, suitability, and interoperability of the participating BMDS elements is ongoing. Continued post-deployment system-level ground testing with the Aegis BMD 3.6 system has helped to refine TTPs and overall interoperability of that system with the BMDS. However, the test events routinely demonstrated that inter‑element coordination and interoperability are still in need of improvement.
Aegis BMD 305
BALLISTIC MISSILE DEFENSE SYSTEMS Recommendations • Status of Previous Recommendations. - The program addressed the remaining part of the recommendation from FY11 to demonstrate that the SM-3 TSRM problem that caused the failure in FTM-16 Event 2 has been corrected when it completed the FTM‑19, FTM- 20, and FTM-21 flight missions with TSRM inter‑pulse delays at the revised minimum value. - The program addressed the first recommendation from FY12 to conduct further live-target testing of the Aegis BMD 4.0.2 LRS&T capability when it successfully sent track data for use by GMD fire control in FTG-07. - The program addressed the second FY12 recommendation to engage a medium-range target before the Full-Rate Production decision for the SM-3 Block IB guided missile to support assessment of midcourse capability when it completed the FTM-22 flight mission. • FY13 Recommendations. The program should: 1. Conduct flight testing of Aegis BMD 4.0’s remote authorized engagement capability against a medium-range
306 Aegis BMD
ballistic missile or intermediate-range ballistic missile target using an SM-3 Block IB guided missile. 2. Conduct operationally realistic testing that exercises Aegis BMD 4.0’s improved engagement coordination with THAAD and Patriot. 3. Continue to assess an Aegis BMD 4.0 intercept mission where the ship simultaneously engages an anti-air warfare target to verify BMD/anti-air warfare capability. 4. Use the Failure Review Board process to identify the failure mechanism responsible for the FTM-21 second missile failure and determine if there is an underlying root cause common to both the FTM-16 Event 2 and FTM-21 second missile failures. 5. Deliver sufficient Aegis BMD 4.0 validation data and evidence to support BMDS modeling and simulation verification, validation, and accreditation of the Aegis HWIL and digital models.
BALLISTIC MISSILE DEFENSE SYSTEMS
Command and Control, Battle Management, and Communications (C2BMC) System Executive Summary • The Missile Defense Agency (MDA) continued to demonstrate the increased capability of Command and Control, Battle Management, and Communications (C2BMC) Spiral 6.4 software during FY13. Ground and flight testing demonstrated automated management of multiple AN/TPY‑2 Forward-Based Mode (FBM) sensors, as well as limited battle management capabilities allowing Combatant Command sensor managers to direct AN/TPY-2 (FBM) radars to execute focused search plans or respond to precision cues. • C2BMC also demonstrated timely and accurate radar track forwarding during numerous ground and flight tests. • C2BMC remains the key situational awareness tool used by the Combatant Commanders and National Command Authority to stay abreast of both homeland and regional ballistic missile defense operations. System • C2BMC is a Combatant Command’s interface to the fully integrated Ballistic Missile Defense System (BMDS). • More than 70 C2BMC workstations are fielded at U.S. Strategic, Northern, European, Pacific, and Central Commands (USSTRATCOM, USNORTHCOM, USEUCOM, USPACOM, and USCENTCOM); numerous Army Air and Missile Defense Commands; Air and Space Operations Centers; and other supporting warfighter organizations. - The current C2BMC provides Combatant Commands and the National Command Authority with situational awareness on BMDS status, system coverage, and ballistic missile tracks by displaying selective data from the Global Communications Network for strategic/national missile defense and the Joint Tactical Information Distribution System for tactical/regional missile defense. - C2BMC also provides upper echelon deliberate planning at the Combatant Command and component level, permitting a federation of planners across the BMDS. BMDS elements (Aegis BMD, Ground-based Midcourse Defense [GMD], Patriot, and Terminal High-Altitude Area Defense [THAAD]) use their own command and control battle management systems, and mission planning tools for stand-alone engagements. - C2BMC S6.4 Combatant Command suite provides command and control for one AN/TPY-2 (FBM) radar. The S6.4 Global Engagement Manager Suite provides command and control for multiple radars, as well as updated sensor management, track processing, and reporting. • Through the Global Communications Network, C2BMC provides track forwarding of AN/TPY-2 (FBM) and
AN/ SPY- 1 tracks to GMD. Additionally, through the Joint Tactical Information Distribution System, it provides track forwarding of AN/TPY-2 (FBM) tracks for THAAD and Patriot cueing and Aegis BMD engagement support. • C2BMC S8.2 is intended to improve and expand the initial S6.4 capabilities as the next step toward integrated sensor management. Mission U.S. Strategic, Northern, European, Central, and Pacific Commands currently use C2BMC to support ballistic missile defense engagements. Commanders use C2BMC specifically for: • Deliberate and dynamic planning • Situational awareness • Track management • AN/TPY-2 (FBM) sensor management and control • Engagement monitoring • Data exchange between C2BMC and BMDS elements • Network management Major Contractor Lockheed Martin Corporation, Information Systems and Global Solutions – Gaithersburg, Maryland
C2BMC 307
BALLISTIC MISSILE DEFENSE SYSTEMS Activity • The MDA conducted testing during FY13 in accordance with the DOT&E-approved Integrated Master Test Plan. • In October 2012, C2BMC S6.4 managed an AN/TPY-2 (FBM) radar from which it forwarded acquisition cues to Aegis BMD and THAAD during the MDA’s combined developmental/ operational Flight Test Integrated-01 (FTI- 01). • In November 2012, C2BMC participated in Ground Test Integrated-04e Part 1 (GTI-04e Part 1), an MDA combined developmental/operational ground test that focused on the defense of USEUCOM and USCENTCOM. The purpose of GTI-04e Part 1 was to assess the new mission functionality of the BMDS operational architecture consisting of Aegis BMD, THAAD, Patriot, AN/ TPY-2 (FBM), and C2BMC. In particular, the warfighters used C2BMC S6.4 to receive AN/TPY-2 (FBM) radar tracks and Link 16 data and forward system tracks. C2BMC S6.4 had no new functionality (software or hardware) during this test event. • C2BMC S6.4 managed a single AN/TPY-2 (FBM) radar during the Fast Eagle II Increment 2 ground test in February 2013 that used both deployed and hardware‑in‑the‑loop (HWIL) representations of BMDS weapon assets focusing on the defense of USCENTCOM. • In February 2013, C2BMC participated in Flight Test Standard Missile-20 (FTM-20). It provided tracks generated by the Space Tracking and Surveillance System and processed by the External Sensors Laboratory and the C2BMC Experimental Laboratory to an Aegis BMD 4.0.2 ship. The ship used these data to successfully intercept a target with an SM-3 Block IA interceptor. • In April 2013, the MDA started GTI-04e Part 1a integration testing. For this test event, C2BMC S6.4 was upgraded with Maintenance Release 1 and 2 (MR1 and MR2) with a focus on debris mitigation. • In July 2013, the MDA conducted Flight Test GBI-07 (FTG- 07) in which C2BMC S6.4 forwarded tracks from Aegis BMD to the GMD Fire Control software. • The MDA conducted Flight Test Operational (FTO-01) in September 2013 to demonstrate an integrated and layered upper-tier regional/theater BMDS defense. During the test, C2BMC S6.4 MR2 managed one deployed AN/TPY-2 (FBM) radar, including demonstrating MR2’s debris mitigation functionality, and passed tracks of two medium-range targets between that radar and an Aegis BMD ship. C2BMC also received and responded properly to J-series messages from Aegis BMD and THAAD. • The MDA conducted an HWIL test called Fast Aim in August 2013. The MDA used an HWIL representation of C2BMC S6.4 MR2 to forward track data from Aegis BMD, report data from an AN/TPY-2 (FBM) radar, and receive data from the Sea-Based X-band radar for simulated intercontinental ballistic missile threats to a portion of the U.S. Homeland.
308 C2BMC
Assessment • C2BMC S6.4/Global Engagement Manager allows for automated management of multiple AN/TPY-2 (FBM) sensors located in one area of responsibility. It also provides greater automation of sensor management functions and improved track processing and reporting while requiring less operator involvement as compared to S6.2 software. • C2BMC has limited battle management capabilities allowing Combatant Command sensor managers to direct AN/TPY-2 (FBM) radars to execute focused search plans or respond to a precision cue. S6.4 demonstrated command and control of a single AN/TPY-2 (FBM) radar in ground and flight tests. S6.4 demonstrated command and control of two AN/TPY-2 (FBM) radars in both an HWIL and distributed test environment, but not in a flight test using deployed assets. • The C2BMC engagement planner provides performance analysis of the composition and location of U.S. and allied BMD assets but does not currently provide a system-level capability to coordinate engagement decisions. Such a capability is planned for S8.4. • The MDA tested C2BMC S6.4 interactions with theater elements throughout the GTI-04e Parts 1, 1a, and Fast Eagle II Increment 2 ground test campaigns in FY13. In addition to providing situational awareness, C2BMC S6.4 (and in the case of GTI-04e Part 1a, S6.4 MR2) demonstrated interoperability with theater BMDS elements and command and control of up to two AN/TPY-2 (FBM) radars contributing to the defensive capability for the USEUCOM and USCENTCOM theaters. • During the GTI-04e Parts 1, 1a, and Fast Eagle II Increment 2 campaigns, C2BMC generally performed nominally receiving AN/TPY-2 (FBM) and Link 16 data and forwarding system tracks to Link 16. • C2BMC did experience some minor latency issues during stressing test cases with large numbers of threats. These latencies ultimately did not adversely affect the outcome for the test cases run during GTI-04e Part 1. • The MDA and BMDS Operational Test Agency team identified S6.4 interoperability and command and control deficiencies during GTI-04e Part 1 that affected track processing, situational awareness, and battle management. Some of these problems are exacerbated by increasing the density of blue forces in any given theater. The MDA is currently testing solutions to these deficiencies. • C2BMC selected and reported AN/TPY-2-based system tracks to Link 16 for all major objects from all threats in support of radar cueing in FTI-01. • C2BMC demonstrated the ability to cue multiple weapon elements in addition to the management and forwarding of cues from the AN/TPY-2 (FBM) supporting FTI-01. It also forwarded track data to Aegis BMD, THAAD, and Patriot. • During FTO-01, C2BMC demonstrated the ability to manage a deployed AN/TPY-2 (FBM) radar while forwarding track data from multiple targets between the radar and an Aegis BMD
BALLISTIC MISSILE DEFENSE SYSTEMS ship. C2BMC also received and responded properly to J-series messages from Aegis BMD and THAAD. • During the Fast Aim HWIL test, C2BMC demonstrated its role in the strategic defense of the U.S. Homeland by forwarding simulated intercontinental ballistic missile tracks between the multiple sensors and the GMD fire control. Recommendations • Status of Previous Recommendations. The MDA addressed eight of the previous nine recommendations. The FY12 recommendation has been combined with the FY13 recommendation. The MDA continues to make progress on the one outstanding FY06 recommendation to include
assessments of Information Assurance during BMDS-centric C2BMC testing. • FY13 Recommendations. The MDA should: 1. Perform additional flight testing with multiple AN/TPY-2 (FBM) radars in a single Area of Regard or theater to assess C2BMC’s ability to correctly task and coordinate track data from multiple radars. 2. Continue to address the C2BMC interoperability and command and control deficiencies uncovered during the GTI-04e Part 1, Fast Eagle II Increment 2, and FTI-01 campaigns. Resolution of these deficiencies should be demonstrated through ground and/or flight testing.
C2BMC 309
BALLISTIC MISSILE DEFENSE SYSTEMS
310
BALLISTIC MISSILE DEFENSE SYSTEMS
Ground-Based Midcourse Defense (GMD) Executive Summary • Ground-based Midcourse Defense (GMD) has demonstrated a partial capability to defend the U.S. Homeland from small numbers of simple intermediate or intercontinental ballistic missile threats launched from North Korea or Iran. • The performance of GMD during flight tests in FY13 prevented any improvement in the assessment of GMD capability. The Missile Defense Agency (MDA) successfully flew a redesigned Capability Enhancement-II (CE-II) Exoatmospheric Kill Vehicle (EKV) in a planned non‑intercept flight test; however, the MDA experienced a failure with a CE-I EKV in an unrelated intercept flight test. The flight test failures that have occurred during the past three years raise questions regarding the robustness of the EKV’s design. • The MDA continues to make progress on the return- to- intercept for the CE-II EKV, but will need to successfully conclude its investigation of the CE-I EKV failure before returning the CE-I EKV to intercept flight testing. System GMD is a Ballistic Missile Defense System element that counters intermediate-range and intercontinental ballistic missile threats to the U.S. Homeland. The GMD “system” includes: • COBRA DANE Upgrade Radar at Eareckson Air Station (Shemya Island), Alaska • Upgraded Early Warning Radars at Beale AFB, California; Royal Air Force Fylingdales, United Kingdom; and Thule Air Base, Greenland • Ground-based Interceptor (GBI) missiles at Fort Greely, Alaska, and Vandenberg AFB, California • GMD ground system including GMD Fire Control (GFC) nodes at Schriever AFB, Colorado, and Fort Greely, Alaska; Command Launch Equipment at Vandenberg AFB, California, and Fort Greely, Alaska; and In-Flight Interceptor Communication System Data Terminals at Vandenberg AFB, California, Fort Greely, Alaska, and Shemya Island, Alaska • GMD secure data and voice communication system including long-haul communications using the Defense Satellite Communication System, commercial satellite communications, and fiber-optic cable (both terrestrial and submarine) • External interfaces that connect to Aegis BMD; North American Aerospace Defense – U.S. Northern Command Command Center and Command and Control, Battle Management, and Communications at Peterson AFB, Colorado; Space Based Infrared System/Defense Support
Program at Buckley AFB, Colorado; and AN/TPY-2 (Forward‑Based Mode [FBM]) radar at Shariki Air Base, Japan • Sea-Based X-band radar, which is a sea-based mobile sensor platform used primarily as a test asset, but which can be operationally deployed as needed Mission Military operators for the U.S. Army Space and Missile Defense Command/Army Forces Strategic Command (the Army service component to U.S. Strategic Command) will use the GMD system to defend the U.S. Homeland against intermediate-range and intercontinental ballistic missile attacks using its weapon, the GBI, to defeat threat missiles during the midcourse segment of flight. Major Contractors • The Boeing Company, Integrated Defense Systems, Missile Defense Systems – Huntsville, Alabama • Orbital Sciences Corporation – Chandler, Arizona • Raytheon Missile Systems – Tucson, Arizona • Northrop Grumman Information Systems – Huntsville, Alabama
GMD 311
BALLISTIC MISSILE DEFENSE SYSTEMS Activity • The MDA conducted an interceptor-only flight test of a GBI equipped with a CE-II EKV in January 2013. The MDA planned this test, GMD Control Test Vehicle-01 (GM CTV- 01), as part of its return-to-intercept effort in response to the failed intercept attempt, Flight Test GBI-06a (FTG- 06a), in December 2010. - A Failure Review Board, which the MDA had convened subsequent to FTG-06a, attributed the failure to a faulty design of a CE-II EKV component. - The MDA redesigned the EKV component, and in GM CTV-01, tested a GBI equipped with a CE-II EKV that incorporated the component redesign. The MDA collected data in this interceptor flight test on the EKV flight environment and EKV performance in the flight environment. - The MDA assessed the data collected in GM CTV-01 and is preparing to conduct an intercept flight test, FTG-06b, in March 2014 as a redo of FTG-06a using a GBI equipped with the redesigned CE-II EKV component. • The MDA conducted an intercept flight test of a GBI equipped with a CE-I EKV against an intermediate-range ballistic missile (IRBM) target in July 2013. The MDA planned this test, FTG-07, to demonstrate CE-I EKV performance under more challenging threat engagement conditions than had been demonstrated in previous intercept flight tests with CE-I EKVs. - The MDA launched an IRBM target from the U.S. Army’s Reagan Test Site on Kwajalein Atoll, Republic of the Marshall Islands. The BMDS sensors detected and tracked the target. - The GFC planned an engagement, and a warfighter manning the GFC launched a GBI from Vandenberg AFB, California, to intercept the target. The GBI, however, failed to intercept. The MDA convened a Failure Review Board that investigated the failure and reported its initial results in August 2013. • The MDA conducted a hardware-in-the-loop test called Fast Aim in August 2013. The MDA used hardware and software representations of GFC; Space-Based Infrared System; Command and Control, Battle Management, and Communications; the AN/TPY-2 (FBM) radar; the Aegis BMD radar in its Long Range Surveillance and Track mode; and the Sea-Based X-band radar to investigate additional BMD capability against intercontinental ballistic missile threats. • The MDA conducted testing in accordance with the DOT&E‑approved Integrated Master Test Plan. Assessment • In GM CTV-01, the GBI boost vehicle and the CE-II EKV with the redesigned component performed adequately and mostly as expected. - The MDA noted several unexpected results that did not negatively affect test execution or data collection. The
312 GMD
MDA is analyzing these unexpected results to determine if any of them pose a risk to GBI operational or test performance. - The CE-II EKV fly-out in GM CTV-01 was, as planned, developmental in nature in order to stress specific aspects of EKV performance and to acquire data in specific environments. CE-II EKV performance in the more operationally-representative intercept flight environment of the failed test, FTG-06a, remains to be demonstrated. - The MDA plans FTG-06b to be a redo of FTG-06a, which should enable assessment of CE-II EKV performance, including target intercept, in that same flight environment. • In FTG-07, the CE-I EKV failed to separate from the GBI boost vehicle and, consequently, was unable to complete all further inflight actions including intercept of the IRBM target. This was the first failure to intercept for a GBI equipped with a CE-I EKV. - The three prior tests, FTG-02, FTG-03a, and FTG‑05, all resulted in target intercepts albeit in less challenging engagement conditions than presented in FTG-07, which had a longer time of flight and a faster closing velocity than the previous CE-I-equipped GBI flight tests. - The MDA convened a Failure Review Board and reported its initial results in August 2013. The board is expected to publish its final report by the end of calendar year 2013. • The MDA is currently analyzing the data that it acquired in the August 2013 Fast Aim test. • The flight test failures that have occurred during the past three years raise questions regarding the robustness of the EKV’s design. Recommendations • Status of Previous Recommendations. The MDA has started, but not completed, the FY11 recommendation to repeat the FTG-06a mission to verify (1) failure root causes, (2) Failure Review Board results, and (3) permanent fixes for the deficiencies found during the flight test. They have identified root cause issues, implemented solutions, and successfully completed the first (CTV-01) of a planned two-flight test series designed to demonstrate the fixes. The MDA has scheduled the second flight test in the series, FTG-06b. • FY13 Recommendations. The MDA should: 1. Conduct a redo of the FTG-07 test with a GBI equipped with a CE-I EKV in order to accomplish the test objectives of FTG-07. 2. Consider whether to re-design the EKV using a rigorous systems engineering process to assure its design is robust against failure.
BALLISTIC MISSILE DEFENSE SYSTEMS
Sensors Executive Summary • The MDA has gained significant operational experience with each of the Ballistic Missile Defense System (BMDS) sensors since the completion of sensor upgrade and development programs. • BMDS sensors participated in two major ground tests, five flight tests, Fast Eagle II Increment 2, and Fast Aim testing during the reporting period. • The Missile Defense Agency (MDA) placed the Sea-Based X-band (SBX) radar into a limited test support and standby operational status, and includes it in ground and flight tests when appropriate. • Accreditation of each of the sensor models for use in performance assessments continues to progress but is still incomplete. The BMDS Operational Test Agency Team has completed some limited accreditation; however, the Team will be unable to accredit the COBRA DANE radar model until after the MDA completes a 2QFY15 flight test involving the radar. System The BMDS sensors are systems that provide real-time boosting and ballistic missile threat data to the BMDS. The data are used to counter ballistic missile attacks. These sensor systems are operated by the Army, Navy, Air Force, and the MDA, and include a satellite‑based, infrared sensor system and five phased array radar system types. The sensor systems are: • Space-Based Infrared System/Defense Support Program (SBIRS/DSP), a satellite constellation of infrared sensors operated by the Air Force with an external interface to the BMDS located at Buckley AFB, Colorado • Fixed site, fixed orientation, phased array radars - COBRA DANE Upgrade (CDU) Radar, an L-band radar (one radar face that provides 120-degree azimuth field of view) operated by the Air Force and located at Eareckson Air Station (Shemya Island), Alaska - Upgraded Early Warning Radars (UEWRs), ultra high frequency radars operated by the Air Force and located at Beale AFB, California (two radar faces that provide
240-degree azimuth field of view); Fylingdales, United Kingdom (three radar faces that provide 360-degree azimuth field of view); and Thule, Greenland (two radar faces that provide 240-degree azimuth field of view). (The MDA and Air Force Space Command awarded a contract in July 2012 for the upgrade of the Early Warning Radar [EWR] at Clear Air Force Station, Alaska. In December 2012, a contract option was exercised for the upgrade of the EWR at Cape Cod Air Force Station, Massachusetts.) • Mobile/transportable variable orientation, phased array radars - AN/TPY-2 (Forward‑Based Mode [FBM]) radars, X-band radars (one radar face that provides a classified azimuth field of view) operated by the Army and located at Shariki Air Base, Japan, and sites in Israel, Turkey, and the U.S. Central Command area of responsibility - Aegis Ballistic Missile Defense (Aegis BMD) AN/SPY-1 radars, S-band radars (four radar faces that provide 360-degree azimuth field of view) operated by the Navy and located aboard Aegis BMD-capable cruisers and destroyers - SBX radar, an X-band radar operated by BMDS and located aboard a twin-hulled, semi-submersible, self‑propelled, ocean‑going platform (primarily a test asset that can be operationally deployed as needed)
Sensors 313
BALLISTIC MISSILE DEFENSE SYSTEMS Mission Military operators for the U.S. Strategic Command, U.S. Northern Command, U.S. European Command, U.S. Pacific Command, and U.S. Central Command will use the BMDS sensors to: • Detect, track, and classify ballistic missile threats that target the United States, U.S. allies, and U.S. friends • Provide data for situational awareness and battle management to the BMDS Command and Control, Battle Management, and Communications (C2BMC) • Provide data that support engagement of ballistic missile threats by ballistic missile defense systems Major Contractors Aegis BMD Radar • Lockheed Martin Corporation, Mission Systems and Training – Moorestown, New Jersey
Activity The MDA conducted testing during FY13 in accordance with the DOT&E-approved Integrated Master Test Plan. Aegis BMD Radar • The Aegis BMD radar participated in Flight Test Ground‑Based Interceptor-7 (FTG-07). In FTG-07, the Aegis BMD radar detected and tracked the intermediate‑range ballistic missile (IRBM) target and forwarded the track data to the C2BMC system. • The MDA conducted a hardware-in-the-loop (HWIL) test called Fast Aim in August 2013. The MDA used hardware and software representations of the Aegis BMD radar. In this test, the Aegis BMD radar representation detected and tracked simulated intercontinental ballistic missile (ICBM) threats to a portion of the U.S. Homeland and forwarded the track data to an HWIL representation of the C2BMC. AN/TPY-2 (FBM) Radar • The AN/TPY-2 (FBM) radar participated in Flight Test Integrated-01 (FTI-01) in October 2012. In FTI-01, the AN/TPY-2 (FBM) radar provided up-range track data to C2BMC for processing, down-select, and forwarding of tracks to Aegis BMD, Patriot, and Terminal High-Altitude Area Defense (THAAD). • The AN/TPY-2 (FBM) radar participated in Flight Test Operational-01 (FTO-01) in August 2013. In FTO-01, the AN/TPY-2 (FBM) radar detected and tracked multiple Regional/Theater ballistic missile threats and provided track reports to the C2BMC. • HWIL representations and distributed AN/TPY‑2 (FBM) radar assets participated in Ground Test Integrated‑04e Part 1 (GTI-04e Part 1), an MDA combined developmental/operational ground test, in November 2012, Fast Eagle II Increment 2 and distributed testing in February 2013, and GTI-04e Part 1a integration testing in April 2013. AN/TPY‑2 (FBM) interactions with
314 Sensors
AN/TPY-2 (FBM) Radar • Raytheon Company, Integrated Defense Systems – Tewksbury, Massachusetts COBRA DANE Radar • Raytheon Company, Integrated Defense Systems – Tewksbury, Massachusetts SBIRS/DSP • Lockheed Martin Space Systems – Sunnyvale, California SBX Radar • Raytheon Company, Integrated Defense Systems – Tewksbury, Massachusetts UEWRs/EWRs • Thule, Beale, and Fylingdales: Northrop Grumman Corporation, Aerospace Systems – Redondo Beach, California • Clear and Cape Cod: Raytheon Company, Integrated Defense Systems – Tewksbury, Massachusetts
C2BMC, interoperability, and some engagement support capabilities in various U.S. European Command and U.S. Central Command theater scenarios against short- and medium-range ballistic missiles were tested using BMDS configurations that are deployed or deployable. • The MDA used an HWIL representation of the AN/TPY‑2 (FBM) radar in Fast Aim in August 2013. In Fast Aim, the AN/TPY-2 (FBM) radar representation detected and tracked simulated ICBM threats to a portion of the U.S. Homeland and forwarded the track data to an HWIL representation of the C2BMC. COBRA DANE Radar • In FY13, the U.S. Air Force used the COBRA DANE radar to observe targets of opportunity. The Air Force Space Command (AFSPC) also used the COBRA DANE radar as a contributory sensor to the Space Surveillance Network to track orbital debris and active satellites. SBIRS/DSP • In FY13, the U.S. Air Force used the SBIRS/DSP system to observe domestic and foreign launch events and provide launch event data to the operational BMDS. The SBIRS/DSP system also participated in multiple BMDS intercept flight tests including FTI-01, FTO-01, FTM-20, and FTG‑07. • A digital representation of the SBIRS/DSP system participated in Fast Aim in August 2013. SBX Radar • The MDA has placed the SBX radar in a limited test support status. The SBX radar can be reactivated based on warning of an ICBM threat to the U.S. Homeland and for BMDS flight testing. The SBX was deployed from limited test support status in FY13 for both flight test and operational contingency.
BALLISTIC MISSILE DEFENSE SYSTEMS • The SBX radar participated in FTG-07. In FTG-07, the SBX radar accepted sensor task plans from the Ground‑Based Midcourse Defense (GMD) Fire Control (GFC), detected and tracked the IRBM target, and forwarded track data to the GFC. • An HWIL representation of the SBX radar participated in Fast Aim in August 2013. In Fast Aim, the SBX radar representation detected and tracked simulated ICBM threats to a portion of the U.S. Homeland and forwarded the track data to an HWIL representation of the GFC. • SBX performed track and discrimination on Minuteman III launches as targets of opportunity in Glory Trips 207, 208-1, and 209. UEWRs/EWRs • In FY13, the U.S. Air Force used the Beale, Fylingdales, and Thule UEWRs, and the Clear and Cape Cod EWRs, to observe targets of opportunity. The AFSPC also used these radars as collateral sensors to the Space Surveillance Network to track orbital debris and active satellites. • In FY14, the MDA will complete the transfer of the Beale, Fylingdales, and Thule UEWRs to AFSPC. Assessment • The MDA has gained significant operational experience with each of the BMDS sensors since the completion of the sensor upgrade and development programs. • The MDA and the BMDS Operational Test Agency Team, however, have not fully accredited models and simulations of the BMDS sensors for performance assessment. Representations of the AN/TPY-2 (FBM) radar, the SBX radar, the UEWR, the Aegis BMD radar, and the SBIRS/DSP system have been accredited for limited uses. Representations of the COBRA DANE radar have not been accredited. The MDA is analyzing the radar performance data that were collected in the FY12 satellite tracking campaign for its use toward accreditation of multiple radar models and simulations. • In FY14, AFSPC will take over responsibility for the sustainment of the COBRA DANE radar and the UEWRs. Aegis BMD Radar • The MDA used the Aegis BMD radar as the primary data source for the GMD engagement planning in FTG-07. Although the interceptor failed to intercept the target, post-test analysis demonstrated that the Aegis BMD radar supported GMD engagement planning and generation by the GFC of a successful sensor task plan. In this test, however, the Aegis BMD employed an alternate concept of operations (CONOPs) that was different from the operational CONOPs that Aegis BMD currently employs. Therefore, the Aegis BMD performance that was demonstrated pertains to the Aegis BMD with the alternate CONOPs and does not pertain to Aegis BMD performance within the current, operational CONOPs. • In Fast Aim, the MDA demonstrated a capability of the Aegis BMD radar to support a potential new BMD
capability against ICBM threats, which will be reported in the classified appendix to DOT&E’s annual BMDS report in February 2014. • In previously conducted BMDS integrated ground tests, the MDA demonstrated a capability of the Aegis BMD radar to support GMD engagement of IRBM and ICBM threats. The Aegis BMD radar provided data that enabled the GMD system to generate sensor cueing and missile engagement plans. AN/TPY-2 (FBM) Radar • In FTI-01, the MDA demonstrated AN/TPY-2 (FBM) radar capability to provide up-range track data to C2BMC to support cueing of Aegis BMD, AN/TPY-2 (Terminal Mode), and Patriot. Interoperability was sufficient for the flight test, but there were track correlation issues for one of the targets that the AN/TPY-2 (FBM) radar tracked that could be problematic in other scenarios. • In FTO-01, the MDA demonstrated AN/TPY-2 (FBM) radar capability to support a BMDS Regional/Theater layered defense against a small raid of threat-representative, medium-range ballistic missile threats flying challenging and realistic attack profiles. • The GTI-04e and Fast Eagle ground tests demonstrated interoperability and engagement support using AN/TPY‑2 (FBM) for theater scenarios, revealing problems that the MDA is now addressing for multi-sensor and multi-element coordination. • In Fast Aim, the MDA demonstrated a capability of the AN/TPY-2 (FBM) radar to support a potential new BMD capability against ICBM threats. • In previously conducted BMDS integrated ground tests, the MDA demonstrated AN/TPY-2 (FBM) radar capability to provide real-time track data that supported BMDS situational awareness, BMDS sensor tasking, and GMD engagement planning. COBRA DANE Radar • Due to its location and field of view, the COBRA DANE radar has not participated in BMDS intercept flight tests. The MDA currently plans to conduct a target flight test through the COBRA DANE radar field of view in 2QFY15 to support model and simulation accreditation. • In previously conducted BMDS integrated ground tests, the MDA demonstrated a capability of the COBRA DANE radar to provide real-time data that enabled the GMD system to generate missile engagement plans and supported GMD system engagement of IRBM and ICBM threats. SBIRS/DSP • SBIRS/DSP performance and its capability to support BMDS engagement of IRBM and ICBM threats will be provided in the classified appendix of DOT&E’s annual BMDS report to Congress. SBX Radar • In FTG-07, the MDA demonstrated a capability of the SBX radar to detect and track an IRBM target and to provide data to GMD that supported GFC engagement planning and
Sensors 315
BALLISTIC MISSILE DEFENSE SYSTEMS generation of in-flight target updates. The employment of the SBX radar in that test, however, was not operationally realistic. • In Fast Aim, the MDA demonstrated a capability of the SBX radar to support a potential new BMD capability against ICBM threats, which will be reported in the classified appendix of DOT&E’s annual BMDS report in February 2014. • The MDA demonstrated a capability of the SBX radar in intercept flight testing to support GMD engagement planning against an IRBM target. However, the MDA has not gathered adequate SBX radar performance data against IRBM and ICBM threats and targets to enable accreditation of the SBX radar models and simulations that are required for performance assessment. • SBX successfully performed track and discrimination on Minuteman III launches as targets of opportunity in Glory Trips 207, 208-1, and 209. UEWRs/EWRs • Due to their locations and fields of view, the UEWRs at Thule and Fylingdales have not participated in BMDS intercept flight tests in an operationally realistic manner. Beale has participated in all flight tests within its field of view and has supplied critical data in analysis of these flight tests. Data from targets of opportunity and ground tests support performance estimates for the current configuration of the UEWRs. UEWRs have participated in GTIs in the Huntsville labs and all field Distributed Ground Tests.
316 Sensors
• The MDA and the U.S. Air Force have not yet upgraded the EWRs at Clear and Cape Cod Air Force Stations, and these radars are not yet part of the MDA’s sensor network. Recommendations • Status of Previous Recommendations. - The MDA has satisfactorily addressed all but three of the previous sensor recommendations. The MDA and Combatant Commanders have made progress on developing concepts of operations for the sensors, but this FY09 recommendation remains open pending completion of those concepts and implementation in operational testing. - The DOT&E February 2012 THAAD and AN/TPY-2 Radar Operational and LFT&E Report made three recommendations for the MDA and Army to consider for AN/TPY-2(FBM). One of the recommendations aligns directly with the Army materiel release conditions, which are being addressed through a plan agreed upon by the MDA Sensors Product Office and the Army. The recommendation to conduct additional flight testing with multiple AN/TPY-2 (FBM) radars in a single Area of Regard or theater has been moved to C2BMC. The recommendation to conduct independent, dedicated AN/TPY-2 (FBM) Information Assurance testing remains open. • FY13 Recommendations. None.
BALLISTIC MISSILE DEFENSE SYSTEMS
Terminal High-Altitude Area Defense (THAAD) Executive Summary • The Missile Defense Agency (MDA) has completed development of Terminal High-Altitude Area Defense (THAAD) 1.0, which includes fundamental capability against short- and medium-range ballistic missiles, and the Conditional Materiel Release of the first two THAAD fire units. Development of the more advanced THAAD capability (Release 2.0) is scheduled through FY18. • The THAAD system successfully intercepted a medium‑range ballistic missile target for the first time in October 2012. In addition, THAAD intercepted one medium-range ballistic missile target while simultaneously engaging debris from a second medium-range ballistic missile target that had been intercepted by the Aegis Ballistic Missile Defense (BMD) element during an operational test in September 2013. • Eight of the 39 Conditional Materiel Release conditions imposed on the first two THAAD batteries have been closed. Fixes and testing of the remaining conditions are scheduled through 2017. • THAAD reliability and maintainability measures are still fluctuating greatly between test events, indicating system immaturity with respect to consistent reliability and maintainability growth. System • The THAAD ballistic missile defense system consists of five major components: - Missiles - Launchers - Radar (designated AN/TPY-2 Terminal Mode [TM]) - THAAD Fire Control and Communications - Unique THAAD support equipment • THAAD can accept target cues for acquisition from the Aegis BMD, satellites, and other external theater sensors and command and control systems.
Activity • The MDA conducted testing during FY13 in accordance with the DOT&E-approved Integrated Master Test Plan. • The combined developmental/operational Flight Test, Integrated-01 (FTI-01) in October 2012 included a THAAD engagement against a medium-range target for the first time. The test evaluated interoperability between THAAD; Aegis BMD; Patriot; Command and Control, Battle Management, and Communications; and AN/TPY-2 Forward-Based Mode elements with multiple live targets.
• THAAD is intended to complement the lower-tier Patriot system and the upper-tier Aegis BMD system. Mission U.S. Strategic Command intends to deploy and employ THAAD, a rapid response weapon system, to protect critical assets worldwide. Commanders will use the THAAD to intercept an incoming threat ballistic missile in the endo-atmosphere or exo- atmosphere, limiting the effects of weapons of mass destruction on battlefield troops and civilian populations. Major Contractors • Prime: Lockheed Martin Corporation, Missile and Fire Control – Dallas, Texas • AN/TPY-2 Radar (TM): Raytheon Company, Integrated Defense Systems – Tewksbury, Massachusetts
• The Flight Test Operational-01 (FTO-01) in September 2013 included two THAAD engagements. The FTO-01 was designed to demonstrate an integrated and layered upper‑tier regional/theater BMDS defense against a raid of two threat‑representative medium-range ballistic missiles threatening a shared defended area. • The Army reviewed and assessed reliability and maintainability data from FTI-01 in December 2012. Additional data were collected throughout the FTO-01 test mission.
THAAD 317
BALLISTIC MISSILE DEFENSE SYSTEMS Assessment • In FY13, MDA realigned THAAD capability deliveries such that THAAD 1.0 is a baseline capability delivery and THAAD 2.0 contains advanced capability development. As such, the MDA has completed THAAD 1.0 development, which includes fundamental capability against short- and medium-range ballistic missiles with initial discrimination capability, interoperability with other BMDS elements, and Conditional Materiel Release of the first THAAD fire units. • THAAD 2.0 work continues, which includes improving THAAD performance in a high debris environment, advanced discrimination algorithms, improved engagement coordination with Patriot and Aegis BMD, the ability to initiate THAAD engagements using sensor data from other BMDS sources, and other upgrades. THAAD 2.0 development is scheduled through FY18. • The THAAD system successfully intercepted one medium- range target during the FTI-01 mission. - This test demonstrated positive performance in a significantly different portion of the battlespace than previous missions with increased ground range, interceptor flight time, and closing velocity, as well as new target re-entry vehicle characteristics. - Mission software reporting of the operational capability of the system components was insufficient to assess the status of the equipment. Specific instances of incorrect and inconsistent reporting were observed during testing and some critical faults were not relayed through the system at all. - THAAD experienced interoperability problems such as data latency with Aegis BMD messages (because the primary network connection was unavailable) and track correlation concerns with the AN/TPY-2 (TM) radar. • The initial assessment from the FTO-01 test mission indicated that THAAD intercepted one medium-range ballistic missile target while simultaneously engaging the debris from the second medium-range ballistic missile target that had been intercepted by the Aegis BMD element. This engagement sequence was by test mission design. Full assessment of FTO-01 test mission data with respect to the effectiveness, suitability, and interoperability of THAAD is ongoing. • The Conditional Materiel Release of the first two THAAD batteries in February 2012 included 39 conditions that need to be resolved before a Full Materiel Release could be granted. The THAAD Project Office and the Army continue to address these conditions. In addition to the four conditions that were closed in FY12, an additional four have been closed in FY13 (verification of technical manuals, procedures for a post-launch launcher inspection, verifying capability against medium-range targets, and procedures and equipment to measure soil density for emplacement). Fixes and testing of remaining conditions are scheduled through 2017. • Comparing the reliability and maintainability data from FTI-01 to the results from the Reliability Confidence Test in July 2011
318 THAAD
and the Flight Test THAAD-12 (FTT-12) in October 2011 shows that reliability and maintainability measures are still fluctuating greatly between test events. This indicates that the THAAD system may not be mature enough to exhibit consistent reliability growth. The additional reliability and maintainability data from FTO-01 will help determine any emerging trends. Recommendations • Status of Previous Recommendations. The classified DOT&E February 2012 THAAD and AN/TPY-2 Radar Operational and Live Fire Test and Evaluation Report contained seven recommendations in addition to and not associated with the 39 Conditional Materiel Release conditions established. The MDA should continue to address the three classified recommendations (Effectiveness #2, Effectiveness #5, and Survivability #4) and the following remaining four: 1. The MDA and the Army should reassess the required spares and tools (including their quantities) that should be onsite with the battery based on all available reliability and maintainability data (Suitability #5). An assessment of the proper number of spares is ongoing and is scheduled to complete in FY14. 2. The MDA and the Army should define duties related to THAAD at the brigade level. Until a battalion is established for THAAD, it should also define duties and training for THAAD battery personnel on any required battalion-level duties (Suitability #10). This recommendation has been addressed, although DOT&E does not concur with the response. The Army has assigned the two fielded THAAD batteries to an existing Army battalion. This battalion currently lacks a comprehensive understanding of THAAD requirements, which significantly reduces the THAAD batteries’ effectiveness by forcing them to assume typical higher headquarters responsibilities for personnel, logistics, plans, and operations. 3. The MDA and the Army should implement equipment redesigns and modifications identified during natural environment testing to prevent problems seen in testing (Suitability #11). During FTO-01, a total radar power failure was observed to be caused by a connector that was missing a gasket, which allowed water to enter a sealed area. Periodic inspection of all gaskets was a recommendation from the natural environment testing. 4. The MDA and the Army should conduct electronic warfare testing and analysis (Survivability #3). This recommendation remains open. Some preliminary testing was conducted during FY13, but additional testing is required. • FY13 Recommendation. 1. The THAAD Program Office should reassess their reliability and maintainability growth planning curve.
Live Fire Test and Evaluation
Live Fire Test and Evaluation
LFT&E PROGRAM
Live Fire Test and Evaluation (LFT&E) Program DOT&E executed oversight of survivability and lethality test and evaluation for 121 acquisition programs in FY13. Of those 121 programs, 21 operated under the waiver provision of U.S. Code, Title 10, Section 2366, by executing an approved alternative Live Fire Test and Evaluation (LFT&E) strategy in lieu of full-up system-level testing. In addition, Section 2366 also requires DOT&E to report on a program’s LFT&E results prior to that program entering into full-rate production. DOT&E published reports on the following programs during the past year (reports marked with an asterisk were sent to Congress): LFT&E Reports • Hellfire Romeo Interim Lethality Assessment • CH-47F Cargo On-Off Loading System (COOLS) Ballistic Protection System (BPS) Armor Live Fire Test and Evaluation Assessment • Enhanced Combat Helmet (ECH) Follow-On Live Fire Test and Evaluation Report* • Live Fire Test and Evaluation Report for the Mk 248 Mod 0 .300 Caliber Cartridge* • Mine-Resistant Ambush-Protected (MRAP) All-Terrain Vehicle (M-ATV) Underbody Improvement Kit (UIK) Final Live Fire Test and Evaluation Assessment* • United States Marine Corps Light Armored Vehicle (LAV) with A2 Upgrades Live Fire Test and Evaluation Report* DOT&E Reports (with combined OT&E/LFT&E elements) • 20 mm Fixed Forward Firing Weapons (FFFW) for the MH-60 Armed Helicopter Weapon System (AHWS) Early Fielding Report*
• Massive Ordnance Penetrator (MOP) Early Fielding Report Phase 2* • Standard Missile 6 (SM-6) Initial Operational Test and Evaluation Report* • HC/MC-130J Initial Operational Test and Evaluation Report* • E-2D Advanced Hawkeye (AHE) Initial Operational Test and Evaluation Report* • M109 Family of Vehicles Paladin Integrated Management (PIM) Limited User Test Operational Assessment • CH-47F Cargo On-Off Loading System (COOLS) Operational Assessment (Integrated Test and Live Fire Test) • KC-46A Operational Assessment #1 • H-1 Upgrades Follow-On Operational Test and Evaluation (FOT&E) Report* • USNS Lewis & Clark (T-AKE) Class of Dry Cargo and Ammunition Ships Follow-On Test and Evaluation Report* In addition to satisfying acquisition oversight requirements, the LFT&E program: • Funds and executes technical oversight on investment programs that provide joint munitions effectiveness data (Joint Technical Coordinating Group for Munitions Effectiveness) • Funds projects to develop advanced technologies and analytical methods to increase aircraft survivability (Joint Aircraft Survivability Program) • Conducts vulnerability and lethality testing of fielded platforms and weapons systems and improves survivability analysis tools (Joint Live Fire Program) • Supports quick reaction efforts addressing urgent operational commander’s needs.
JOINT TECHNICAL COORDINATING GROUP FOR MUNITIONS EFFECTIVENESS The Joint Technical Coordinating Group for Munitions Effectiveness (JTCG/ME) was chartered more than 40 years ago in 1968 to serve as the DoD’s focal point for munitions effectiveness information. They produce Joint Munitions Effectiveness Manuals (JMEMs), which provide tri-Service approved effectiveness data for all major non-nuclear U.S. weapons. JTCG/ME authenticates weapons effectiveness data for use in operational weaponeering, strike mission planning, training, systems acquisition, weapon procurement, and combat modeling and simulation. JTCG/ME also develops and standardizes methodologies for evaluating munitions effectiveness and maintains databases for target vulnerability, munitions lethality, and weapon system accuracy. The Armed Forces of the U.S., NATO, and other allies use JMEMs to plan operational missions, support training and tactics development, and support force-level analyses. The JMEM requirements and development process continues to be driven by operational lessons learned (e.g., Enduring Freedom, Iraqi
Freedom, Odyssey Dawn, and ongoing Task Force operations) and the needs of Combatant Commands, Services, the Military Targeting Committee, and Operational Users Working Groups for specific weapon-target pairings and methodologies. The primary JMEM application is weaponeering, which is the detailed technical planning of a weapon strike that occurs at multiple levels in the operational chain of command before actual combat. JTCG/ME produces, distributes, and regularly updates JMEMs, which provide computerized effectiveness tools and data for rapid weaponeering, i.e., evaluation of alternative weapons and their delivery against specific targets. In many cases, effectiveness and collateral damage estimates generated by these tools are part of the decision criteria for strikes approved at the highest levels of the U.S. Government. A formal data call is issued annually via J-2 Joint Staff Action Process to the Services and Combatant Commands. DOT&E sponsors the JTCG/ME and provides an annual budget. The
LFT&E
319
LFT&E PROGRAM Director, Army Materiel Systems Analysis Activity chairs the JTCG/ME Executive Steering Committee and oversees the Program Office at Aberdeen Proving Ground, Maryland. JMEM TARGETING AND WEAPONEERING SOFTWARE In FY13, the fielded JMEM Weaponeering System (JWS) version 2.1 (v2.1) software and the JTCG/ME‑generated Chairman of the Joint Chiefs of Staff Instruction (CJCSI) 3160.01 Collateral Effects Radii (CER) tables were used for operational weaponeering and collateral damage estimation calls in direct support of combat operations in the U.S. Africa Command and U.S. Central Command Areas of Responsibility. JWS is a source for air-to-surface and surface‑to‑surface weaponeering, munitions, and target information; and evaluates the effectiveness of a multitude of munition‑target combinations for numerous air-to-surface and surface-tosurface munitions against a variety of target types in real-time.
JTCG/ME is developing JWS v2.2, which will contain Fast Integrated Structural Tool updates to include an interface to the Digital Precision Strike Suite Collateral Damage Estimation (DCiDE) tool. JWS v2.2 will also contain new/updated targets, new/updated munitions, trajectory simulation updates, browser updates, and an enhanced viewer. In addition, development is ongoing to support the release of the JWS product to coalition partners. This capability represents a significant improvement to coalition warfare.
The Joint Anti-air Combat Effectiveness System (J-ACE) simulates air-to-air and surface-to-air engagements. JTCG/ME released v5.2 in September 2013. Blue, Red, and Gray air-to-air missile models as well as Red and Gray surface-to-air missile fly-out models are included. J-ACE v5.2 provides an updated missile fly-out model, including hundreds of weapon target pairings and an interface to Enhanced Surface-to-Air Missile Simulation for countermeasures.
J-ACE v5.2 also provides the new Endgame Manager (EM) software and data sets. The EM is a new application that adds missile lethality and target vulnerability. EM allows explicit evaluation of weapon miss distance, fuse performance, weapon lethality, and target vulnerability. EM provides the Probability of Kill given an intercept for the entered mission. To more effectively support operational mission planning, particularly at U.S. Strategic Command, the J-ACE v5.2 also provides a direct interface to force-level simulations. The fidelity is adequate for studying tactics, training evaluation, relative missile performance, and scenario planning. OPERATIONAL SUPPORT TO MISSION PLANNING In support of mission planning for the Combatant Commands and the CJCSI 3160.01, the JTCG/ME supported the release of DCiDE v1.1 for operational use in FY13. This tool displays JTCG/ME accredited Collateral Damage Estimate effective radii reference tables. Additionally, JTCG/ME provided incremental updates in FY13 for CER values for newly fielded/updated systems (e.g., M1130 Projectile, AGM-65-E2/L, and AGM‑176‑3/2M).
The JTCG/ME continues to have a Senior Weaponeering Instructor stationed at MacDill AFB with U.S. Central Command, CCJ2-JOT, to support the Combatant Commands. The instructor has deployed on numerous occasions in support of current operations, most recently to provide training to weaponeers and targeteers from U.S. Naval Forces Central Command Bahrain, International Security Assistance Force Joint Command in Afghanistan, a Task Force in Afghanistan, and the 609th Air and Space Operations Center at Al Udeid Airbase Qatar. JTCG/ME trained nearly 250 users at 10 different commands to support realtime, operational Collateral Damage Estimation decisions.
INFORMATION OPERATIONS TOOLS AND CAPABILITIES In conjunction with the Air Force Targeting Center, the JTCG/ME has established a working group to develop JMEMs for cyberspace operations. This effort led to the development of a first generation of tools, including the Computer Network Attack‑Risk and Effectiveness Analyzer, the Weapons Characteristics Manual, Target Vulnerability Assessments, and the Network Risk Assessment Tool. These capabilities are being incorporated as modules within the Joint Capabilities Analysis and Assessment System (JCAAS), formerly known as Information Operations JMEM. JCAAS is intended to be a single point of access for analysts, targeteers, planners, and others to identify and analyze non-kinetic options for military operations.
JOINT AIRCRAFT SURVIVABILITY PROGRAM The Joint Aircraft Survivability Program (JASP) develops techniques and technologies to improve the survivability of U.S. military aircraft. Working with joint and Service staffs, other government agencies, and industry, the JASP funds development of new capabilities and works to assure they are pursued jointly by the Services.
320
LFT&E
DOT&E sponsors and funds JASP. The Naval Air Systems Command, Army Aviation and Missile Command, and Air Force Life Cycle Management Center charter the program. DOT&E establishes objectives and priorities for the JASP and exercises oversight of the program.
LFT&E PROGRAM JASP is supporting the Joint Multi-Role Technology Capabilities Demonstration program as a member of the Platform Integrated Product Team. The program intends to demonstrate transformational vertical lift capabilities to prepare the DoD for developing the next generation vertical lift fleet. JASP was a driving force in establishing the assumptions and requirements for the Vulnerability Analysis used in evaluating the initial three government model prototypes. JASP funded 52 multi-year survivability projects for $9.4 Million and delivered 45 final reports in FY13. The following summaries illustrate current JASP efforts in four focus areas: susceptibility reduction, vulnerability reduction, survivability assessment, and combat damage assessment. SUSCEPTIBILITY REDUCTION These efforts address urgent aircraft survivability needs from current combat operations, as well as provide improved aircraft survivability against future threats. Special Threats Investigation and Modeling. The Naval Research Laboratory is performing an in-depth analysis of newly-obtained threat infrared seekers, which operate in a new portion of the electromagnetic spectrum, to develop flare and jammer countermeasures. The first objective is to develop countermeasure solutions and parametric requirements for flares and jammers to defeat the new missiles. The second objective is to enhance the DoD modeling and simulation tools to support countermeasure analysis in the new band of the electromagnetic spectrum. Advanced Pre/Post-Launch Man-portable Air Defense System (MANPADS) Identification. The U.S. Army Communications‑Electronics Research, Development, and Engineering Center/Intelligence and Information Warfare Directorate is developing a methodology for using coded optical waveforms that would allow direct measurement of relative position and reflectivity of optical elements within an optical sight. The system is intended provide the ability to include the unique optical “fingerprint” of the missile seeker. If successful, this technique offers the potential of rapid missile identification, before or after launch, for more effective countermeasures. Multi-Digital Radio Frequency Memory (DRFM) Coherence for Self-Protection and Escort Jamming. The Naval Research Laboratory is developing algorithms to demonstrate a DRFM jammer pair that operates in concert, but without any link between two independent jammers. Once the algorithms are
developed, they will be implemented and demonstrated in laboratory and field radar experiments. These technologies and techniques will enhance the self-protection and support the jamming capability of U.S. military aircraft. VULNERABILITY REDUCTION In FY13, JASP vulnerability reduction projects focused on developing lighter-weight opaque ballistic protection systems, fuel containment and related fire protection technologies, and structures and materials, including composites that are self-healing. Three of the most highly successful projects are highlighted below. MH-47 Sub Deck Armor. The Army Aviation Applied Technology Directorate (AATD), contracting with The Protective Group, completed work on this project during FY13. They developed a non‑permanent armor to fit under the floor of the MH-47 helicopter cabin. The goal was to maintain the same minimum level of ballistic protection as the fielded armor, with better durability and less installed weight. Locating the armor under the cabin floor panels greatly reduces the wear and increases its lifespan. The designers also developed an installation and removal system that is lightweight, requires minimal aircraft modification and manpower, and does not interfere with maintenance requirements, mission equipment, or cargo loading systems. The project demonstrated armor panel installation and removal in minutes and achieved a 34 percent reduction in weight over the currently fielded ballistic protection system. Self-Contained Fire Protection System (FPS). The U.S. Air Force 96th Test Group and Engineering and Scientific Innovations, Inc. teamed up to develop self-contained fire protection technology. The system will be lightweight, quick- reacting, and easy to install on aircraft without structural modification to the airframe and without requiring an external power source. The FPS will incorporate
LFT&E
321
LFT&E PROGRAM both detection of the fire within 4 microseconds, and suppression within 500 microseconds. In addition, the system will be rechargeable and contained within a single unit. Lightweight Conformal Armor. The United Technologies Research Center (part of the United Technologies Corporation), in coordination with AATD, worked to transition their high impedance ceramic composite backing layer technology from flat armor to conformal (simple curvature) armor. They demonstrated a curved armor technology capable of defeating armor piercing projectiles at a weight savings of 21 percent (over current solutions) and providing suitable protection for objects up to a 7-inch diameter. This solution is most applicable to components such as servos, actuators, and hanger bearings. In addition, the monolithic ceramic-fiber reinforced ceramic matrix composite (CMC) hybrid layering system utilized further enhanced performance and provided additional weight reduction over current solutions. The armor consists of a continuous fiber-reinforced glass-CMC strike face, bonded directly (no adhesive) to a monolithic ceramic, which is in turn directly bonded to a second layer of fiber-reinforced glass‑CMC, backed by a final layer of highly cross-linked polyethylene fiber. SURVIVABILITY ASSESSMENT The JASP continues to develop and maintain survivability assessment methodologies from the engineering through the few‑on-few engagement levels of analysis. These methodologies are widely used to support system acquisition through design studies; specification development and compliance; and test and evaluation through pre-test predictions, post-test analysis, operational test kill removal, and countermeasure effectiveness assessment.
322
LFT&E
Suite of Anti-air Kill chain – Models and Data (SAK-MD). JASP continues to work with the JTCG/ME and the U.S. Strategic Command (USSTRATCOM) to improve the data and methodologies employed by USSTRATCOM to assess options for global strike missions. These assessments are combined with other information at USSTRATCOM into a decision support package that goes to the President to enable strategic power deployment decisions. These tools and data are also used extensively in the Air Force and Navy fighter aircraft community for training and tactics development. The primary FY13 efforts centered on the usability of the SAK-MD software, expanding the EM user base, and subject matter expert reviews of the methodology and data. Enhanced Surface-to-Air Missile Simulation (ESAMS) Upgrades. There were three projects in FY13 to improve the credibility of ESAMS by updating threat system information and conducting verification and validation: • A multi-year effort by JASP to incorporate the latest surface‑to-air missile (SAM) threat system descriptions from the Missile and Space Intelligence Center into ESAMS • Upgraded modeled threat system radars from analog to digital processing • Verification and validation of the anti-helicopter mode that was developed for a specific SAM system as a previous JASP effort, which will provide a credible modeling tool for assessing helicopter survivability against radio frequency-guided SAM threat systems. Infrared Countermeasure (IRCM) Modeling. The Naval Surface Warfare Center (NSWC) is developing physics-based models for pyrotechnic and pyrophoric IRCM. These models address combustion, heat, and mass transfer, as well as infrared radiation, trajectory, and spatial extent/image presentation; ultimately providing time‑dependent plume or cloud characteristics for use in missile-flare engagement models. Current engagement models, including hardware-in-the-loop simulations, rely on oversimplified inputs that do not have the resolution needed to address the capability of imaging seekers to discern variations within radiation sources.
LFT&E PROGRAM The NSWC is also modernizing the Flare Aerodynamic Modeling Environment (FLAME) and the Tri-Service Flare Database (TFD) software architecture to improve usability, add 3D aircraft flow-fields to FLAME, and develop Linux versions of both tools. The final product will be enhanced versions of both FLAME and TFD that can be run on either Linux or Windows® operating systems. This project will configure FLAME and TFD for distribution through the Survivability/Vulnerability Information Analysis Center (SURVIAC). These projects will enhance the DoD capability to develop countermeasure techniques for advanced infrared-guided missiles. COMBAT DAMAGE ASSESSMENT JASP continued to support the Joint Combat Assessment Team (JCAT) in FY13. JCAT is a team of Air Force, Army, Marine Corps, and Navy personnel deployed in support of combat operations. JCAT continued its operation in Afghanistan with full-time deployments in Regional Commands – South, Southwest, and East. Iraq and other areas of the world were supported remotely or by rapid deployment from Afghanistan or the United States. JCAT inspects damaged and destroyed aircraft, acquires maintenance records, and conducts interviews with aircrew and intelligence personnel to develop an accurate and comprehensive assessment of each aircraft combat damage event. They provide consultation to weapons, tactics, and logistics personnel and comprehensive briefings to commanders in charge of daily air operations. These efforts inform battlefield commanders, allowing them to adjust operational tactics, techniques, and
procedures (TTPs) based on accurate threat assessments. In FY13, the JCAT had initiated 225 and completed 219 aircraft combat damage assessments. The JCAT strengthened aircraft combat damage incident reporting in the Services and the DoD. The Combat Damage Incident Reporting System hosted by the SURVIAC is the repository for all U.S. aircraft combat damage reports. JCAT and SURVIAC worked with the Office of the Deputy Assistant Secretary of Defense for Systems Engineering (ODASD(SE)) and U.S. Central Command on a successful demonstration linking the Combat Damage Incident Reporting System and U.S. Central Command databases to more quickly identify, assess, document, and distribute aircraft combat damage incident data to the Services and DoD. JASP and ODASD(SE) submitted major weapon system combat damage reporting requirement language for the revision of DoD Instruction 5000.02, T&E Enclosure, c.7; and has drafted language for the aircraft combat damage reporting process for inclusion in the Defense Acquisition Guide. The JCAT trains the U.S. aviation community on potential aircraft threats and combat damage. This training includes but is not limited to: capabilities briefs, intelligence updates, recent “shoot-down” briefs to discuss enemy TTPs, and the combat damage collection and reporting mentioned above. The attendees include aircrews, maintenance personnel, intelligence sections, Service leadership, symposia attendees, and coalition partners. Pre-deployment training was provided to 1,100 aircrew bound for combat duty and another 1,200 survivability community members in professional military education courses and DoD symposia.
JOINT LIVE FIRE (JLF) The goal of the Joint Live Fire (JLF) program is to test fielded systems, identify vulnerable areas, understand damage mechanisms, and provide the information needed to make design changes, modify TTPs, or improve analytical tools. The need for these tests results from systems being exposed to new threats; being used in new, unanticipated ways; or being operated in new combat environments, thereby requiring an updated assessment of their performance. JLF supplements LFT&E of systems by testing systems against new threats that the requirements community did not anticipate during original development or against old threats employed in new ways. The rocket-propelled grenade (RPG) is an example of a threat employed differently than initially intended. Originally developed as an anti-tank or anti-personnel weapon, hostile forces in Afghanistan often use the RPG as an anti-aircraft weapon. AIRCRAFT SYSTEMS PROGRAM JLF-Air conducted five test series in FY13, with a focus towards collecting blast and penetration data to improve LFT&E assessments through improvements to our vulnerability modeling and simulation capabilities. Data were collected for selected small arms, anti-aircraft artillery, and missile threats.
V50 Yaw of Projectiles. This project is exploring how a projectile, such as those impacting fast moving targets, penetrates typical aircraft materials when impacting the materials at other than normal (i.e., 90 degrees) incidence. Test results will provide immediate feedback on the accuracy of the analytical vulnerability tools now being commonly used and will be used to update the Computation of Vulnerable Area Tool (COVART) and the Advanced Joint Effectiveness Model (AJEM). The JLF program executed 76 test events in early FY13. Data were collected for projectile yaw angles of 0, 10, 20, and 30 degrees. Test fixture limitations prevented testing of yaw angles greater than 30 degrees. The Army Research Lab (ARL) is currently analyzing data, and will compare results to pre-test predictions.
LFT&E
323
LFT&E PROGRAM MANPADS Missile Debris Penetration. This project involves launching MANPADS debris (collected during previous static MANPADS detonations) through a series of aluminum panels, which represent a generic aircraft structure. Collected data will provide model developers insight into the physics of debris threat penetration, and allow an assessment of current penetration methodology.
Vulnerability models are currently incorporating the capability to model non-spherical blast, but data are needed to support those models.
The 30-shot test matrix was completed in September 2013. Testing was executed by Army Research Laboratory (ARL)/ Survivability and Lethality Analysis Division personnel at Aberdeen Proving Ground, Maryland, with Air Force 96th Test Group oversight, and support from Air Force, Army, and Navy model developers.
With matching JTCG/ME funding, the first phase of testing completed in late FY13 against a relevant and representative threat air-to-air missile. Testing was at the Air Force Research Laboratory (AFRL) blast pad facility located on Eglin AFB, Florida, where an array of 48 air-blast measurement pressure gages collected the data. Data will be used by COVART, EM, and various JTCG/ME predictive models, which require the free-field characterization of the explosive force. A second phase is planned for FY14, where an additional class of threat weapon utilized by the J-ACE community will be characterized.
Advanced Hit Efficiency and Destruction (AHEAD) Sub‑projectile Characterization Testing. This project is determining the penetration characteristics for a modern anti‑aircraft artillery projectile. For the last quarter of a century, rotorcraft vulnerability testing and analyses primarily focused on armor-piercing incendiary (API) and high-explosive incendiary projectiles. However, more advanced threats are being fielded, such as air burst munitions. This project looks at the air bursting AHEAD round and, in particular, the penetration characteristics of its sub‑projectiles. Initial testing started in late FY13 and will continue into FY14. When complete, data from up to 150 shots spread across three obliquity angles and two thicknesses of aluminum will be available for the development of a penetration model to effectively model air burst munitions sub-projectiles, providing an analysis capability presently unavailable for the most commonly used vulnerability codes (AJEM/COVART). Non-Spherical Blast Measurement for Missiles. This project addresses a lack of data needed to understand and characterize non-spherical air-blast pressure distributions produced by missile warheads. Current vulnerability methodologies assume a spherical blast profile, whereas warhead configuration and previous testing indicate blast is non-spherical in nature. Modelers account for this non-spherical nature by increasing the spherical blast pressures by a “fudge” factor, but this may be resulting in over-predicting Probability of Kill due to blast.
324
LFT&E
Vulnerability Characteristics of the PT-6A Family of Engines. This project evaluates the vulnerability of the PT6 turboprop family of engines to ballistic threats and identifies and recommends vulnerability reduction measures. Phase I, conducted in FY13, examined the penetration and damage effects of ball, API, and fragment simulating projectile threats against static engine components to characterize the lower bounds of engine component vulnerability. Test data will be used to validate and expand the previous vulnerability estimates for Probability of Component Damage given a Hit and plan for follow-on testing in FY14 using an operating engine. Phase I completed 29 tests (23 planned plus 6 retests) in early FY13 against components from a PT6A-34 engine. Components included gears, bearings, axial rotor, impellor, main shaft/tie bolt, and the fuel control unit. Results varied from no significant effect likely to likely loss of engine power within one minute. GROUND SYSTEMS PROGRAMS Validation of JTCG/ME Joint Blast Analysis Methodology (JBAM) Tool. In FY13, ARL conducted testing to support validation efforts for the JTCG/ME JBAM tool. This was a continuation of previous testing efforts conducted over the past few years on both simple plate and full vehicle targets. The FY13 program focused on generating data needed to support the validation efforts for the different plate response algorithms in the JBAM tool. Testers detonated bare explosive spheres at various distances from plates and measured loading on the plates, as well as dynamic and permanent deflections of the
LFT&E PROGRAM panels. Digital image correlation techniques were used to capture dynamic deflection data. The plate response models in the JBAM tool drive the development of blast lethal volumes that are essential to assess weapons effects on material targets for the JTCG/ME. Improved Modeling for Small Arms Protective Inserts (SAPI). ARL conducted tests of small-caliber rounds against personnel protective equipment Enhanced Small Arms Protective Insert and X-threat Small Arms Protective Insert plates. Data collected from this test program, including V50 (the velocity at which 50 percent of the rounds penetrate the armor plates), residual velocity, and residual mass, will be used to develop better penetration algorithms for vulnerability/ lethality models, and thus to assess personnel survivability. United Kingdom & Canadian Torso Device Evaluation for Behind Armor Blunt Trauma. This project quantified and examined the repeatability and reliability of thoracic test devices developed by the United Kingdom and Canada for evaluating behind armor blunt trauma. In order to use the technology offered by these devices for system acquisition, ARL equipped each system with body armor materials and shot at them with ballistic threats. Analysis examined the sensitivity of these
devices to evaluate body armor materials from a regime of high and low-velocity impacts. This effort will provide data and in- depth analysis for the body armor community. Detonation of Solid Propellant. This project addresses the inaccuracies in engineering models to predict sympathetic detonation of solid rocket propellant when subjected to non‑reactive fragments and shaped charge threats. The Air Force 780th Test Squadron tested the ability of the small diameter bomb warhead to detonate 122 mm rocket motors. The test results were compared with predictions from Sandia National Laboratories’ Combined Hydro and Radiation Transport Diffusion Hydrocode by Applied Research Associates. Analysis is ongoing, and is expected to enable further development of concepts and methodologies for enhanced vulnerability, lethality, and survivability in the area of insensitive munitions and non-reactive materials. Development of an Engineered Soil Standard for Theater Representation in LFT&E. The objective of this project is to determine standard parameters for the characterization of engineered soil repeatability, testability, and measurability in LFT&E through a series of sub-scale and full-scale explosive experiments performed by Aberdeen Test Center (ATC) and ARL. Emplacement parameters have been developed for engineered soil to meet roadbed and loose (cross country) emplacement conditions. Fifteen explosive experiments will be conducted at Aberdeen Proving Ground: nine sub-scale at the ARL Vertical Impulse Measurement Facility and six full-scale at ATC’s C-field range. The program results will be compared with those of the current DOT&E directive Homemade Explosive Characterization (HME-C) effort, funded by the Army Test and Evaluation Command. ARL has completed nine
LFT&E
325
LFT&E PROGRAM sub-scale events. ATC’s full-scale testing is underway, utilizing the Army intelligence community’s approved HME surrogate developed by the HME-C program. Full-scale explosive experiments will conclude in December 2013.
reduced ship availability time required to install and remove the instrumentation suite, in addition to substantially reducing the amount of cable required and the number of bulkhead and deck penetrations.
Assessment of Ocular Pressure as a Result of Blast for Protected and Unprotected Eyes. This testing, conducted by ATC, consisted of shock tube testing and free field blast testing using a Facial and Ocular Countermeasure for Safety head form. The primary test objectives were to assess the level of blast protection that goggles and spectacles provide Soldiers during blast events, to quantify pressure levels seen in the ocular region during blast events, and to indicate any design features that could be changed to mitigate effects of blast overpressure and enhance stability of future ocular protection systems. ATC is preparing a final report, which is expected to be published in early 2014.
Bomb Scalability Tests. This project conducted an experimental evaluation of a foreign asymmetric threat weapon at half-scale. Data collected from the test series are being combined with quarter-scale data collected in an FY12 evaluation for the Office of Naval Intelligence to better understand the characteristics of this threat. Parametric studies are also being conducted to investigate the effects of changing several of the warhead-design variables in order to assess the range of lethal effects.
Fragment Testing Against Adobe Walls. NSWC, Dahlgren, Virginia, conducted a series of tests with fragment replicas fired against adobe walls to obtain depth of penetration and/or residual speed and weight. This collected information will be utilized to improve the fragment penetration methodology, Fast Air Target Encounter Penetration. The improved Fast Air Target Encounter Penetration methodology will in turn be utilized to calculate protection provided by adobe walls to support collateral damage and lethality estimates in current theaters. SEA SYSTEMS PROGRAM The Joint Live Fire Sea Systems Program (JLF-Sea) funded projects to improve the capability to assess the survivability of submarines and surface ships. These projects benefit ship and submarine acquisition programs, as well as the fleet of fielded U.S. Navy vessels. Finnish-U.S. Cooperative KATANPÄÄ Shock Trials. In June 2013, the Finnish Navy and U.S. Navy jointly conducted a series of shock tests against the newly acquired Finnish mine countermeasure vessel KATANPÄÄ. In addition to JLF-Sea, U.S. sponsors of the trial were the Naval Sea Systems Command Chief Technology Office, the Program Executive Office for Littoral Combat Ships, and the Navy International Program Office. Of particular interest to JLF-Sea were Navy efforts to reduce the cost of shock trials. A new instrumentation scheme that used existing network connections to connect each recording station was proposed and demonstrated. The approach significantly
326
LFT&E
U.S. Coast Guard 41-foot Utility Boat Vulnerability Model. This project developed a target vulnerability model for the U.S. Coast Guard 41-foot utility boat, which is being taken out of service. The Coast Guard has made a number of these boats available to the Navy as targets representing fast-attack craft. JLF-Sea funded NSWC, Dahlgren, to develop a target geometry model for the boat and to develop the failure-analysis logic tree and failure mode effects analysis. Once approved by the JTCG/ME, the model will be made available to acquisition programs with testing requirements against fast-attack craft. Sea Bottom Underwater Explosion Effects Testing. The latest project agreement between the U.S. Navy and German Navy continues development and validation of simulation tools for assessing ship survivability against various explosive threats. In FY12, JLF-Sea provided funding to conduct underwater explosion testing for charges located on the bottom and near the bottom to quantify the loading environment near the bottom, in the middle of the water column, and at keel depth for floating structures. In FY13, the test results were analyzed to better understand the test pond bottom characteristics, explosion bubble migration, and shock loading. This project effectively leverages a joint U.S./ German investment of nearly $20 Million and provides data to increase the fidelity of models and the accuracy of survivability assessments, as well as addresses fleet urgent operational needs.
LFT&E PROGRAM LFT&E SPECIAL INTEREST PROGRAMS PERSONNEL PROTECTION EQUIPMENT DOT&E continued oversight of personnel protection equipment testing. The Services and U.S. Special Operations Command continue to implement the DoD testing protocols for hard body armor inserts and military combat helmets. The Defense Logistics Agency has utilized the hard armor testing protocol in new contracts for sustainment stocks of hard armor inserts, and included the military combat helmet protocol in its solicitation for the Light Weight Advanced Combat Helmet (ACH). The DoD Inspector General conducted a Technical Assessment of the ACH, with a focus on first article test standards. The DoD Inspector General also initiated a follow-up audit to its 2009 audit of hard body armor testing requirements. The ACH Technical Assessment found that the protocol adopted a statistically principled approach and an improvement with regard to the number of helmets tested, and made recommendations to improve the protocol. DOT&E agreed to implement the recommendations. The National Academy of Sciences’ Committee on Review of Test Protocols Used by the DoD to Test Combat Helmets began its work in January 2013. DOT&E has asked the National Academy of Sciences to review the military helmet test protocols and to evaluate the appropriate use of statistical techniques, the performance metrics, and the adequacy of current test procedures to determine the protection current helmet performance specifications provide. The committee will also comment on considerations for efficient scoping of future helmet characterization efforts. This study is expected to be complete by March 2014. The Army and U.S. Special Operations Command have developed multi-sized headforms as potential replacements for the single-sized headform currently used for military combat helmet testing. Initial characterization testing should begin in FY14. WARRIOR INJURY ASSESSMENT MANIKIN (WIAMan) DOT&E continued its oversight of the WIAMan project, an Army-led research and development effort to design a biofidelic prototype anthropomorphic test device (ATD) specifically for underbody blast testing. In FY13, the project underwent significant restructuring to address delays in execution and to streamline management and funding lines. The Army Research Laboratory, under the Research, Development, and Engineering Command, is now the home of the newly-formed WIAMan Project Management Office (PMO), which is responsible for execution of all parts of the project, including medical research and ATD development. Currently, the PMO projects that delays from prior fiscal years will push the delivery of the prototype ATD by approximately 12 – 16 months, out to 2018. While under PMO oversight, execution of the medical research associated with WIAMan has been transitioned from the Army to the Johns Hopkins University Applied Physics Laboratory. Most of the extramural medical researchers have had test
plans approved by the PMO and several have initiated body region-specific research activities.
A key programmatic accomplishment in FY13 was the completion of an initial series of experiments, which conducted paired-comparison tests to determine the differences in response between a human and an ATD in an explosively-driven, LFT&E‑representative environment. This test series utilized a unique fixture, purpose-built for the WIAMan program, which allows the use of small amounts of explosive to fine-tune loads imparted to occupants seated on a platform on the fixture. This test series demonstrates a stark difference in the kinematic response of a human when compared to that of an ATD in an underbody blast environment, to the point where the loading recorded by the ATDs is likely not representative of actual loads to a person during the course of such an event. These differences highlight the critical need to continue this type of work in order to enhance the DoD’s understanding of the human response to the underbody blast environment. Such knowledge will form the basis for significantly improving underbody blast LFT&E capabilities and building better, more protective vehicle platforms for our Soldiers, Sailors, Airmen, and Marines. SMALL BOAT SHOOTER’S WORKING GROUP Small boats represent a growing threat class to ships operating in littoral waters. They have been identified as a required class of targets within a wide variety of tactical missile, rocket, and gun weapon programs under DOT&E oversight, including 25 mm, 30 mm, and 57 mm ammunition; Hellfire, Joint Air-to-Ground, Evolved Sea Sparrow, and Rolling Airframe missiles; and Small Diameter Bomb II; as well as for ships such as the Littoral Combat Ship and the DDG 1000. The target sets, evaluation approaches, and test methodologies for these targets vary widely from program to program. In an attempt to coordinate across these interests, DOT&E sponsored the second Small Boat Shooter’s Working Group on August 29, 2013, hosted at the NSWC, Dahlgren. Fifty- nine weapon system operators, weapons designers, and evaluators met to discuss the nature of the small boat threat, the availability of targets and lethality models representing those threats, and the data collection, test techniques, and instrumentation that have been applied to tests against small boats. Threat experts suggested the inclusion of larger patrol boats such as the CG-41 (being phased out by the Coast Guard), and evaluators encouraged the development of small, easy-to-place “in-situ” camera and overpressure measurement packages on the boats in order to better observe and record the types of damage to target boats during fleet weapons tests. COMBAT DATA ANALYSIS The conflicts in Afghanistan and Iraq have resulted in a large number of rotary-wing aircraft hit by enemy fire resulting in aircraft losses and personnel casualties (fatalities and injuries).
LFT&E
327
LFT&E PROGRAM In 2009, Congress directed the DoD to conduct a study on rotorcraft survivability with the specific intent of identifying key technologies that could help reduce rotary-wing losses and fatalities. However, since non-hostile and non-combat mishaps accounted for more than 80 percent of the losses and 70 percent of the fatalities, conclusions from the 2009 study are more heavily weighted towards preventing mishaps than surviving direct combat engagements. This year, DOT&E analyzed combat damage to the four primary U.S. Army helicopters (AH-64 Apache, H-47 Chinook, H-60 Blackhawk, and OH-58 Kiowa Warrior) to provide insight on the threats (including small arms, MANPADS, and RPGs), aircraft components and systems, and operational conditions that led to the loss or damage of aircraft and personnel casualties. Additionally, combat damage to these four helicopters was compared to live fire testing to determine if any changes need to be made in how live fire test programs are conducted. Conclusions from this study showed: • Analyses of combat damage have led to multiple hardware and TTP changes, some of which have already been instituted by the Army, such as the installation of the AN/AAR-57 Common Missile Warning System.
328
LFT&E
• Results of LFT&E provide good predictors of the types of damage seen in combat. • The primary causes of threat-induced fatalities and injuries were-- The threat directly hitting personnel - Catastrophic crashes (i.e., crashes where there are no survivors) caused by the threat hitting a component, which subsequently caused loss of control of the helicopter. DOT&E made recommendations to the Army to improve design requirements for all helicopters to make them similarly robust to those of the Blackhawk and Apache and to implement existing vulnerability reduction technologies to improve the survivability of all the Army helicopters. DOT&E also made a recommendation to the DoD to institutionalize combat data collection and reporting to avoid losing the capability to collect and analyze this valuable information. DOT&E staff have briefed the results of this study to senior Army leaders, and the Army is using the study to make informative decisions on its future aviation enterprise. A parallel effort for the Navy and Marine Corps helicopters (i.e., H-1, V-22, and H-53) will be conducted next year.
Information Assurance and Interoperability
Information Assurance and Interoperability
i N F O R M A T I O N a S S U R A N C E an d i ntero p era b i l i t y
Information Assurance (IA) and Interoperability (IOP) Summary Although 16 assessments were planned for FY13, 8 of those were associated with Combatant Command (CCMD) or Service exercises that either were cancelled, reduced in scope, or split into smaller events because of funding cuts and limitations related to sequestration as shown in Figure 1. Nonetheless, the DOT&E Information Assurance (IA) and Interoperability (IOP)
Assessment Program completed 12 assessments: 9 of which were conducted at 8 CCMDs and 3 at Service exercises. These were conducted during either exercises or real-world activities and DOT&E was able to analyze these events for trends in context with the prior six years of assessments, as shown in Figure 2.
Figure 1: FY13 Exercise Assessments
Figure 2: FY07-FY13 Exercise Assessments USAFRICOM – U.S. Africa Command USCENTCOM – U.S. Central Command USCYBERCOM – U.S. Cyber Command USEUCOM – U.S. European Command USNORTHCOM – U.S. Northern Command USPACOM – U.S. Pacific Command USSOCOM – U.S. Special Operations Command ECRE – Enterprise Cyber Range Environment
Most FY13 assessments were at smaller venues than previous years and often included only the lowest tier of computer network defense (local network defenders).1 At the same time, many assessed commands continued an ongoing transition from direct CCMD management of network resources to an enterprise 1
Computer Network Defense (CND) is divided by responsibility into three tiers: Tier 3 (local), Tier 2 (CND Service Providers, e.g., Service and Agency cyber commands), and Tier 1 (Dod-wide, e.g., U S Cyber Command)
USSOUTHCOM – U.S. Southern Command USSTRATCOM – U.S. Strategic Command USTRANSCOM – U.S. Transportation Command USA – U.S. Army USAF – U.S. Air Force USMC – U.S. Marine Corps USN – U.S. Navy C2IS – Command, Control, and Intelligence Systems
model of consolidated network defenses – a trend that will continue with the Joint Information Environment (JIE). As a result, the actual Computer Network Defense Service Providers (CNDSP) were not usually assessed during FY13 exercises. To offset this, three events explored new approaches for assessments without a training exercise: (1) an extended Theater Cyber Readiness Campaign assessment, (2) a Cyber Key Terrain
IA/IOP
329
i N F O R M A T I O N a S S U R A N C E an d i ntero p era b i l i t y methodology assessment, and (3) the IA and IOP Assessment Program also explored making better use of cyber range facilities by sponsoring the Enterprise Cyber Range Environment (ECRE).2 Based on FY13 assessments, the demonstrated capabilities of the local network defenses are insufficient to protect against a determined or well-resourced cyber adversary and warfighter missions should be considered “at moderate to high risk” until they can be demonstrated to be resilient in a contested cyber environment. Overall IA (soon to be referred to as “cybersecurity”) compliance observed during the FY13 exercise assessments reflected continued and even improved conformance with standards and policies as shown in Figure 3.3 However, network scans continued to find missing patches and IA vulnerability alerts at rates consistent with previous years.
usually resulted in increased operator workloads, increased errors, and slowed mission performance, but did not affect the accomplishment of the assigned missions and tasks. Less than one third of all fielded systems observed in assessments over the past five years have had current Interoperability certifications. Given the generally effective interoperation of the systems assessed, both certified and uncertified, it is clear the Interoperability certification process provided little to no confidence in system readiness and has not eliminated the need for such workarounds. Attainment of the milestones from the Chairman, Joint Chiefs of Staff (CJCS) Execute Order (EXORD) to Incorporate Realistic Cyberspace Conditions into Major DoD Exercises of February 2011 remained low. Portrayal of denied, manipulated, or contested cyber conditions was seldom permitted in FY13 assessments, providing little opportunity for the continued development of more sophisticated tactics and procedures. Currently, the Joint Staff intends to allow the EXORD to expire in February 2014 but will replace it with a CJCS Instruction. Increased emphasis on cybersecurity test planning improved the level of rigor and cyber-threat realism in acquisition tests, but the majority of cybersecurity problems identified during operational testing in FY13 could have been uncovered and resolved in early phases of development and testing. DOT&E and USD(AT&L) are coordinating to update procedures for developmental and operational cybersecurity testing to increase the scope and rigor for an integrated test strategy to improve discovery and correction of vulnerabilities earlier in the acquisition development cycle.
Figure 3: Network Standards Compliance Red Teams were consistently able to penetrate and exploit networks, but seldom were permitted to conduct disruptive activities – and the lack of exercise participation by upper-tier CNDSPs limited the ability to fully assess the impact of Red Team activities. This lack of participation in IA evaluations must be addressed as it raises questions regarding CNDSP effectiveness in guarding against, recognizing, and responding to attacks. By extension, it also raises questions regarding the approach JIE will implement for computer network defense. IOP assessments were limited in FY13 for the same reasons as cited earlier, but anecdotal findings confirmed that operators frequently implement workarounds to complete assigned missions and tasks when information systems encounter difficulties exchanging data automatically. These workarounds
2
An assessment of Cyber Key Terrain identifies critical components and nodes related to missions of interest, and focuses on the protection and defense of those key components and nodes. 3 Revised DoD Instruction 8500.01, anticipated release in late 2013.
330
IA/IOP
Essential observations for FY13 include: • DoD is moving towards more centralized and enterprisebased management of cyber capabilities, including the implementation of JIE. • Local network (proactive) defenses were insufficient to counter the portrayed cyber adversaries. • Inclusion of upper tier CNDSP participation is essential for both effective training and effective network defense. • While standards compliance has improved, such compliance is necessary but not sufficient to ensure effective network defense. • DoD cybersecurity training policies should require participation by all relevant cybersecurity activities/tiers operating in contested cyber conditions with realistic threats. • The currently evolving tools needed to automate the management and defense of enterprise networks will require ongoing testing and evaluation. • Cybersecurity testing of acquisition programs must emphasize earlier discovery and remediation of vulnerabilities.
i N F O R M A T I O N a S S U R A N C E an d i ntero p era b i l i t y FY13 Activities In FY13, the five assessing organizations were the Army Test and Evaluation Command; Commander, Operational Test and Evaluation Force; the Marine Corps Operational Test and Evaluation Activity; the Joint Interoperability Test Command; and the Air Force Operational Test and Evaluation Center. These
five Operational Test Agencies completed 12 assessments under the DOT&E IA and IOP Assessment Program that included 9 CCMD and 3 Service exercise assessments (see Table 1). Two of the assessments involved units preparing to deploy (or already deployed) to Iraq and Afghanistan.
Table 1. Information Assurance and Interoperability Exercise Events in FY13 Assessment/Exercise Authority U.S. Africa Command
U.S. Central Command
Assessment/Exercise Venue
designated assessment lead
Judicious Response 2013 (Exercise cancelled)
ATEC
Headquarters Vulnerability Assessment (Multiple events)
ATEC
Marine Forces CENTCOM Site Assessment
ATEC
Internal Look 2013 (Exercise cancelled)
ATEC
Headquarters Vulnerability Assessment
ATEC
U.S. Cyber Command
Cyber Flag 2013
ATEC
U.S. European Command
Theater Cyber Readiness Campaign (Multiple events)
ATEC
North American Aerospace Defense Command/U.S. Northern Command
Vigilant Shield 2013
AFOTEC
U.S. Pacific Command
Terminal Fury 2013 (Exercise cancelled)
COTF
U.S. Special Operations Command
Emerald Warrior 2013
ATEC
U.S. Southern Command
Integrated Advance 2013
ATEC
U.S. Strategic Command
Global Lightning 2013 (Exercise cancelled)
JITC
Turbo Challenge 2013 (Exercise cancelled)
JITC
Real World Assessment
JITC
Warfighter Exercise 13-4
ATEC
USS Harry S. Truman Sustainment Exercise
COTF
Bold Quest 2013 (Exercise cancelled)
COTF
Blue Flag 2013 (Exercise cancelled)
AFOTEC
U.S. Transportation Command U.S. Army U.S. Navy U.S. Air Force U.S. Marine Corps
Ulchi Freedom Guardian 2013 (Deferred to 2014)
AFOTEC
Dawn Blitz 2013-2
MCOTEA
I MEF Site Assessment
MCOTEA
AFOTEC – Air Force Operational Test and Evaluation Center ATEC – Army Test and Evaluation Command CENTCOM – U.S. Central Command COTF – Commander, Operational Test and Evaluation Force IOW – Information Operations Wing JITC – Joint Interoperability Test Command MCOTEA – Marine Corps Operational Test and Evaluation Activity MEF – Marine Expeditionary Force
DOT&E and the Operational Test Agencies began an ongoing in-depth analysis on a number of topics germane to the conduct and improvement of IA and IOP assessments including: • Consolidated assessment program guidance and practices into a handbook-style document • Revised cybersecurity compliance metrics to attain consistency with the National Institute for Standards Risk Management Framework • Revised IOP metrics to capture expanded areas of interest and better integrate with IA as part of a holistic cybersecurity assessment • Revised data collection forms to incorporate lessons learned and capture new areas of interest • Reviewed IA/cybersecurity compliance inspection and review programs to identify data sharing opportunities
• Designed a scorecard for measuring compliance with guidance to improve training in contested cyber environments • Developed a Cyber Key Terrain assessment methodology when exercise events are not available • Developed a scoring mechanism to rate potential exercise venues as well as evaluate the quality of an assessment Many of the lessons learned during exercise assessments have provided insight on better test methods for systems under acquisition and test. To enhance the cybersecurity for acquisition programs, DOT&E continued to revise and refine the guidance, templates, and process for planning IA testing for acquisition programs. The templates facilitate development and review of Test and Evaluation Master Plans and test plans to ensure that IA is adequately addressed. The templates and new process were
IA/IOP
331
i N F O R M A T I O N a S S U R A N C E an d i ntero p era b i l i t y applied to reviews of 67 separate Service and DoD systems, including 67 Test and Evaluation Master Plans, 14 operational test plans, and 12 related test documents. DOT&E IA subject matter experts specifically observed IA tests and reviewed report data for 21 systems that showed the majority of cybersecurity problems identified during operational testing in FY13 could have been uncovered and resolved in early phases of developmental testing. DOT&E and USD(AT&L) are working together to revise and update procedures for developmental and operational cybersecurity testing. The purpose of these revisions is to expand the opportunities to discover and correct vulnerabilities earlier in the acquisition development cycle. This will be accomplished by systematically examining the stated
system cybersecurity requirements, analyzing the inherent cybersecurity requirements that arise from the system operating environment, and constructing tests that realistically depict the ways an adversary would attempt to compromise the system under test. DOT&E conducted site visits in support of cyber assessments for the U.S. Air Force’s (USAF) Joint Space Operation Center Mission System and the U.S. Navy’s (USN) Joint High Speed Vessel and Los Angeles/Virginia submarines. DOT&E has provided active support to assist in the development of cyber testing for systems such as the USN CVN-78 aircraft carrier, USAF Joint Strike Fighter and KC-46 aircraft, and the U.S. Army (USA) M1 ABRAMS tank.
findings, trends, and analysis Assessment Structure Ownership, architecture, and command and control relationships governing DoD networks are all in considerable flux. The European-based networks are in transition to a JIE structure, Navy networks are in transition from an outsourced service to a partially outsourced service, and the division of duties between network defense tiers continues to evolve. In addition, the resource constraints from sequestration of DoD funds resulted in fewer and smaller exercises in FY13, constraining the ability of DOT&E assessment teams to observe and assess network defenses. Most FY13 assessments were at smaller venues and only included the lower tiers of computer network defense. As the Department continues to migrate to more centralized and enterprise network and cybersecurity management models, the majority of key network defense activities are now performed by the upper tier commands, such as the CNDSPs, the Service Cyber Component Commands, or U.S. Cyber Command (USCYBERCOM). Therefore, the focus in FY13 was principally toward local/proactive defenses (standards compliance, patch management, vulnerability management) and not the reactive (detection, remediation) activities conducted at higher layers of network defense. The FY13 assessments were focused on the lower tier defenses, and it was clear that local network (proactive) defenses were insufficient to counter the portrayed cyber adversaries. To be more realistic and effective for both training and assessment, future events should include the upper tier cybersecurity services.
Three of the FY13 assessments explored new approaches for cybersecurity assessments without a training exercise venue: an extended Theater Cyber Readiness Campaign assessment at U.S. European Command and a Cyber Key Terrain methodology assessment at U.S. Africa Command and U.S. Central Command. These assessments were intended to develop consistent assessment approaches for normal operating conditions that would not depend on a scheduled exercise to perform or necessitate harmful effects to operations and networks.
332
IA/IOP
Capability Assessment While compliance with key cybersecurity standards continued to improve in FY13, assessment teams observed that good fundamental network maintenance, while necessary, was not sufficient to fully protect DoD networks and systems. Local network defenses are insufficient to protect against a determined or well-resourced cyber adversary and warfighter missions should be considered “at moderate to high risk” until they can be demonstrated to be resilient in a contested cyber environment. Assessments continued to identify the risks posed to operational missions from cyber events, primarily affecting information intensive missions of commanding and controlling forces. The primary mission effects encountered in assessments involved degradations to operational security from compromise of information. IOP problems affecting missions were largely due to the inherent costs associated with the workarounds devised to exchange needed information when automation failed--these costs include the additional personnel and workload required, errors introduced during manual transcriptions, and delays in mission tasks. The risks to operational missions were generally moderate to high when considering the expected severity of the operational effects and the likelihood from portrayed cyber threats, and were generally low when IOP problems were encountered. Overall, compliance with network standards continues to improve in almost every key area reflecting the continuing efforts across the DoD to implement cybersecurity policies and procedures. Compliance determines whether network defensive measures are in place; however, the observed defensive performance against portrayed threats confirms that these measures can be defeated. Red Teams increasingly circumvented network defenses using default or stolen credentials despite improved compliance with identity management policies. The asymmetric nature of cyber operations permits even a single default or discovered password to lead to rapid exploitation of the network. Further, Red Teams continued to encounter systems with known vulnerabilities that remained unpatched and improper configurations that permitted relatively easy paths for exploitation.
i N F O R M A T I O N a S S U R A N C E an d i ntero p era b i l i t y Some fundamental problems appear to be improving. Exercise adversary teams found fewer default or poorly selected passwords, but stolen and default credentials were a principal pathway to intrusion and exploitation activities. Additionally, key network infrastructure components, such as domain controllers, web servers, and printers remained focus areas for surveillance and possible exploitation, often because these components have inconsistent configuration management. Analysis of cybersecurity acquisition testing in FY13 (conducted separate from these exercise assessments) also shows a large body of cybersecurity vulnerabilities, the majority of which derive from either password and software configuration management, missing patches, or network vulnerabilities of systems under test. Many of these fundamental problems go undiscovered until operational testing is conducted late in the acquisition cycle, or discovered during normal fielded operations (such as these exercise assessments). The Red Teams and CCMD exercise planners emphasized realistically portrayed cyber-adversary activities, but continued to restrict activities needed to create contested conditions that include adversely affecting network resources or mission processes. FY13 assessments increasingly noted that improvements in portrayed threat realism have not been matched by improvements in network defense realism (specifically, the inclusion of upper-tier defensive capabilities). Assessments of CCMD exercises continue to find a more balanced mix of experience levels for network defenders, but Service exercises remain heavily biased towards lower-skilled personnel. Figure 4 shows the distribution of personnel with
beginner, intermediate, and expert skillsets. The difference between the distribution of skill levels at the CCMDs and within the Services likely reflects both the skill and experience requirements levied for assignment of Service personnel to joint tours, and the higher levels of contract support at the CCMD headquarters. Host Base Security System (HBSS) is intended to provide key monitoring and automated reporting support to the future JIE and continuous monitoring solutions for DoD, but in-depth reviews of HBSS in FY12 and FY13 found that a number of problems remain to be resolved with HBSS, including: • Inconsistencies in the asset management inventories, apparently caused by common configuration errors and hardware. These errors could be exploited to bypass HBSS protections. • Incomplete or inconsistent information provided by analysis tools to support the investigation of some errors and failed actions. Query tools are also difficult to use. • Misunderstood system setup rules and interfaces caused by configuration errors. • Intrusion protection rules that are difficult to access or understand. Little Interoperability data were gathered in FY13 due to the reduced opportunities for exercise assessments. In those assessments conducted, however, Interoperability issues were noted ranging from minor (e.g., systems freezing but easily rebooted with little-to-no loss of data exchange but minor processing delays) to moderate (e.g., two fires coordination systems locked up due to data transfer backlogs requiring operators to shift to voice communications which took three to five times longer to accomplish). In each case, local operators had developed workarounds, which, while effective in completing the mission tasks, required extra time, extra workload, and personnel, and introduced errors that would not have occurred had the automated data transfers worked properly. Less than one third of all fielded systems observed in assessments over the past five years have had current Interoperability certifications. Given the generally effective interoperation of the systems assessed, both certified and uncertified, it is clear that the Interoperability certification process provided little to no confidence in system readiness and has not eliminated the need for such workarounds.
Figure 4: Personnel Skill Levels
IA/IOP
333
i N F O R M A T I O N a S S U R A N C E an d i ntero p era b i l i t y infrastructure DOT&E conducted a variety of events to demonstrate and stress the capabilities of the National Cyber Range with support from other ranges and assets to include the Joint Cyberspace Operations Range, the DoD IA Range, Sandia National Laboratories, U.S. Pacific Command/J81, and the Threat Systems Management Office. These events also provided insights on how a DoD Enterprise Cyber Range Environment (ECRE) might work and enabled development of specific environments as part of the ECRE. The ECRE development effort is a DOT&E-led partnership to build representative mission environments where Red Teams can conduct attacks and demonstrate effects not permitted on operational networks and systems. These environments will be available via the DoD ECRE for use during exercises and in pre- and post-exercise events to demonstrate cyber effects, develop cyber playbooks, and enhance cyber tactics, techniques, and procedures. Each ECRE environment under development was motivated by an earlier exercise assessment where Red Team activities were restricted by operational or training limitations. The first such environment, ECRE-Command, Control, and Intelligence Systems (C2IS), in development by the Joint Staff J6’s Command, Control, Communications, and Computers (C4) Assessment Division, involves the common operating picture and supporting situational awareness systems. The ECRE-C2IS Team completed several phases of risk-reduction activities during late FY13, including integration of a Joint Information Operations Range (JIOR) node to support distributed Red Team and assessment activities. Preliminary events with Red Teams were also executed, providing the first look at the potential effects that a cyber adversary could deliver to the networks and systems
of this critical mission area. ECRE-C2IS will support the assessment of the USNORTHCOM exercise Vigilant Shield 2014 in October 2013. The second environment, ECRE-Command and Control, Battle Management, and Communications (C2BMC), is composed of the command and control elements of the Ballistic Missile Defense System. ECRE-C2BMC capabilities will be provided by the Missile Defense Agency, with augmentation of JIOR nodes. Planning is underway for risk-reduction activities and active Red Teaming. Activities in the missile defense mission area were part of the FY13 U.S. European Command Theater Cyber Readiness Campaign assessment, and ECRE-C2BMC will support follow-on events in FY14.
The third environment, ECRE-AEGIS, focuses on the Aegis Combat Systems and will be developed in four “spirals” or phases during FY14. Collaboration with the Navy Red Team, Wallops Island and Dahlgren test facilities, and Combat Direction Systems Activity Dam Neck began in 4QFY13. Phase 1 activities were conducted in August 2013 and included successful proof-of-concept testing by the Navy Red Team. Phase 2 activities are planned in 1QFY14 to generate initial results regarding the scope and duration of cyber effects. ECRE-AEGIS is expected to support several CCMD assessments in FY14. Additional ECRE environments are under consideration that will provide realistic data regarding the scope and duration of impacts on critical missions due to cyber attacks. Nonetheless, the management and resourcing of DoD ECRE remains fragmented and inefficient. DOT&E strongly recommends management and resourcing be brought under an Executive Agent.
partnerships and coordination DOT&E continued the long-standing partnerships with the Joint Staff and DoD Chief Information Officer (CIO) on the oversight and coordination of the IA and IOP Assessment Program. Metrics and observations generated from these assessments are provided to the DoD CIO for use in enterprise-wide IA assessments and programs. DOT&E coordinates efforts with USD(AT&L), Developmental Test and Evaluation (DT&E) in matters of test and evaluation for acquisition and development of information handling systems. Together with AT&L, DOT&E is reviewing and revising the existing guidelines for cybersecurity testing of acquisition programs. The revised process, once approved, will allow for earlier development of cybersecurity test strategies that are better focused on the operational role of the system under test. This will be accomplished by examining system requirements and intended mission environments early in development and designing developmental and operational tests that cumulatively examine the system. DOT&E is establishing a standing working group with USCYBERCOM and the National Security Agency to develop and synchronize priorities for Cyber Opposing Force missions
334
IA/IOP
consistent with the USCYBERCOM Exercise Support Plan, the Chairman of the Joint Chiefs of Staff training guidance, and DOT&E’s CCMD and Service assessment schedule. This group will work to ensure a Cyber Opposing Force has timely ground rules in place for their operations, detailed cyber threat information, and the training and resources to portray representative cyber adversaries. In addition, the working group will track significant vulnerabilities, recommend priorities for development of cyber range environments, and oversee persistent access to the DoD information networks for cyber test teams. DOT&E worked closely with many members of the intelligence community to improve both the scheduling and portrayal of the representative cyber threats during FY13 exercises. The Defense Intelligence Agency continued to enhance realism during these exercises by helping to write representative cyber threat scenarios and coordinating with Red Teams to ensure they knew adversarial practices and could apply them against DoD networks for training. The Defense Intelligence Agency team, in coordination with other intelligence community members, is building detailed
i N F O R M A T I O N a S S U R A N C E an d i ntero p era b i l i t y cyber-adversary threat folders to improve overall understanding and portrayal of adversary capabilities. Recognizing that not all adversary actions and effects are suitable for conduct on live networks, DOT&E continues to support the development of methods and environments to exercise and assess advanced actions on appropriate closed-loop cyber ranges. Cyber ranges such as the DoD JIOR were used in four assessed exercise venues and emphasis will continue for increasing the integration and operational realism of JIOR events associated with assessments in FY14. DOT&E also conducted a variety of events in FY13 to demonstrate and stress the capabilities of the National Cyber Range that included participation of several other cyber range capabilities. The National Cyber Range is now accredited to all classification levels required to support OT&E. The use of other ranges, including the Defense IA Range, and expanded range tools such as persistent range environments is also supported by DOT&E. DOT&E and the Test Resources Management Center used funding targeted for cyber enhancements to develop advanced cyber-threat assessments, improve the capabilities of cyber
Red Teams so they can emulate the advanced threats, develop range environments to demonstrate advanced cyber effects, and create a team of cyber/range/test and evaluation experts to plan and execute rigorous cyber-range events. The Test Resource Management Center’s resources are being applied to field the next-generation Regional Service Delivery Points for the JIOR; improvements to traffic generation, instrumentation, and visualization capabilities; creation of persistent cyber environments, and incorporation of Live-Virtual-Constructive capabilities into the cyber ranges. DOT&E has already seen early effects of these improvements, which will be reported on fully in the FY14 DOT&E Annual Report. DOT&E, in partnership with the Naval Postgraduate School, supports research for improved tools for testing and assessing cybersecurity. Thus far, this has led to the design and development of network test tools, which simulate intrusion and malware symptoms; validation of this tool as a training asset for network operators; and the ongoing development of cause/effect models for use in network event simulations.
reports Each assessment provided a specific report for the exercise authority (CCMD or Service) detailing results and observations including discovered vulnerabilities. DOT&E provided additional direct feedback to the exercise authorities for problems of high priority. In addition to these exercise assessment reports, DOT&E published six memoranda of findings and initiated research of three additional areas of concern in FY13. Finding memoranda detail specific shortfalls and vulnerabilities that have the potential to significantly degrade operations and warrant senior leadership attention. Shortfalls and vulnerabilities were identified to the responsible leadership and replies were provided to DOT&E detailing mitigation efforts, which then are subject to subsequent re-evaluation and validation in future assessments. During the fiscal year, solutions to prior findings were reviewed or validated in the field where observable. New findings released in FY13: • HBSS (released October 2012) – documented discrepancies in the operation of the asset management functions. Response received from the Defense Information Systems Agency. • Unsecured Chat Capabilities (released October 2012) – documented the use of unsecure collaboration tools in DoD. Awaiting JCS response. • Network Access Controls (released November 2012) – investigated the use of commonly available devices to compromise DoD networks. Response received from DoD CIO.
• Identity and Access Management (released January 2013) – documented frequently encountered problems with the use of credentials on DoD networks. Response received from USSTRATCOM. • Adaptive Network Defense (released January 2013) – documented the completion of a joint test at USPACOM to implement a rapidly-deployed virtual secure enclave capability to protect key data and components. Response received from JCS. • Assessment of DoD IA during Major CCMD and Service Exercises (published April 2013) – documented a detailed follow-up to the April 2012 report of the same title, and addressed classified issues identified in FY12. Response received from DoD CIO. New research initiated in FY13: • Defense Connect Online (initiated April 2013) – investigating new vulnerabilities in DoD collaboration tools. • HBSS (initiated June 2013) – investigating new issues discovered with the use of HBSS on DoD networks. • Shipboard Systems (initiated July 2012, re-initiated July 2013) – validating original findings and remediations were put into place as a result of research into potential vulnerabilities to afloat systems.
IA/IOP
335
i N F O R M A T I O N a S S U R A N C E an d i ntero p era b i l i t y FY14 goals and plans For FY14, the goal of the DOT&E IA and IOP Assessment Program is to complete at least one full assessment of each CCMD and Service. A full assessment is a holistic cybersecurity assessment (IA and IOP components) with the associated mission assurance analysis that focuses on the ability of the training audience to execute critical missions in denied, manipulated, or contested cyber conditions. The FY14 Program has 12 CCMD assessments, 5 Service assessments, and 3 observation-only assessments (See Table 2). The observation-only assessments evaluate specific exercises as potential venues for FY15 and beyond assessments. For FY14, the goals of DOT&E cybersecurity operational test and evaluation are: • Update procedures for operational testing to improve the DoD’s ability to identify and resolve issues earlier in system development and testing • Portray representative cyber threats to determine resilience of tested systems The FY14 detailed plans for DOT&E efforts in cybersecurity operational test and evaluation, and field assessments include: • Full implementation of the CJCS EXORD (and/or applicable follow-on instructions) to provide training opportunities for
• •
• • •
•
CCMDs and Services to execute critical missions in denied, manipulated, or contested cyber conditions. Improved realism of the cyber threat levels and effects portrayed during all tests and assessments. Increased coordination with USCYBERCOM in scheduling and synchronizing requirements for certified and accredited Red Team assets in support of approved CCMD and Service assessments. Improved data collection methodologies to enhance the end-to-end analysis of Cyber Opposing Force activities. Expanded capability of DoD JIOR and other cyber range facilities to support field assessments, training events, and tests. Implementation of a process to track remediation and verification of corrections for discovered vulnerabilities and shortfalls identified during CCMD and Service assessments. Increased completeness of the portrayed DoD cybersecurity defensive capabilities in field assessments and tests by improving participation of upper Tier computer network defense service providers.
Table 2. Information Assurance and Interoperability Exercise Events Proposed for FY14 Exercise Authority
Exercise
Assignment Agency
U.S. Africa Command
Epic Guardian 2014
ATEC
AOR Site Assessment – Special Operations
ATEC
U.S. Central Command
Internal Look 2014
ATEC
U.S. Cyber Command
Cyber Flag 2014
ATEC
U.S. European Command
EUCOM Theater Cyber Readiness Campaign 2014
ATEC
North American Aerospace Defense Command/U.S. Northern Command
Vigilant Shield 2013
AFOTEC
U.S. Pacific Command
Tempest Wind 2014
COTF
U.S. Special Operations Command
Tempest Wind 2014
ATEC
U.S. Southern Command
JIATF-South Assessment
ATEC
Global Lightning 2014
JITC
Global Thunder 2014
JITC
U.S. Strategic Command
Gypsy Juliet 2014 (Observation only)
JITC
U.S. Transportation Command
Turbo Challenge 2014
JITC
U.S. Army
Warfighter Exercise 2014-4
ATEC
U.S. Navy U.S. Air Force
U.S. Marine Corps
Valiant Shield 2014
COTF
Ulchi Freedom Guardian 2014 (7th Air Force)
AFOTEC
Green Flag 2014 (Observation only)
AFOTEC
Red Flag 2014 (Observation only)
AFOTEC
Ulchi Freedom Guardian 2014 (III MEF)
MCOTEA
Large Scale Exercise 2014 (I MEF)
MCOTEA
AOR – Area of Responsibility AFOTEC – Air Force Operational Test and Evaluation Center ATEC – Army Test and Evaluation Command COTF – Commander, Operational Test and Evaluation Force JITC – Joint Interoperability Test Command MEF – Marine Expeditionary Force MCOTEA – Marine Corps Operational Test and Evaluation Activity
336
IA/IOP
Test and Evaluation Resources
Test and Evaluation Resources
t e s t a n d e va l u at i o n r e s o u r c e s
Test and Evaluation Resources Public law requires DOT&E to assess the adequacy of operational and live fire testing conducted for programs under oversight, and to include comments and recommendations on resources and facilities available for operational testing and evaluation (OT&E) and on levels of funding made available for OT&E activities. DOT&E monitors and reviews DoD and Service-level strategic plans, investment programs, and resource management decisions to ensure capabilities necessary for realistic operational tests are supported. This report highlights general areas of concern in testing current systems and discusses significant issues, DOT&E recommendations, and testing and evaluation (T&E) resource and infrastructure needs to support operational and live fire testing. FY13 focus areas included: • General Test Infrastructure • Operational Test Agency (OTA) Capabilities and Resources • Continuing Radio Frequency Spectrum Concerns • Advanced Electronic Warfare (EW) Test Resources
General Test Infrastructure The Budget Control Act of 2011 and the continuing impasse on federal funding present significant challenges for DoD planning and budget formulation. Limitations on test infrastructure funding and related impacts are uncertain, but potentially significant. For example, the Major Range and Test Facility Base (MRTFB) faces negative impacts due to potential loss of funding needed to maintain and enhance capabilities; this funding comes from both institutional sources and customers (whose own funding losses cause schedule delays or cancellations). In turn, unavailable MRTFB assets and capabilities may delay testing for acquisition programs. Specific FY13 impacts due to sequestration alone include: • One-month delay for the F-35 Joint Strike Fighter (JSF) test program due to limited availability of the 412th Test Wing and the 96th Test Wing in Air Force Materiel Command • Reduction of flying operations at both the Eglin and Edwards open-air ranges to a four-day-per-week schedule resulting from civilian furloughs and contractor workforce layoffs • Air Force Space Command ceasing operations of several MRTFB test infrastructure assets (e.g., tracking and imaging radars, telemetry, and imaging optics) for the remainder of FY13. The test infrastructure continues to face technological and policy challenges and risks in maintaining capabilities to test and evaluate the effectiveness, suitability, survivability, and lethality of current and future defense systems. For example, ensuring the availability of sufficient Radio Frequency spectrum for operational testing of many current weapons systems (such as the JSF) requires policy solutions, technology innovations, and significant funding to maintain existing capabilities. Radio Frequency spectrum concerns are described in greater detail below.
• • • • • • • • • • •
Aegis-Capable Self-Defense Test Ship (SDTS) Aegis Probability of Raid Annihilation (PRA) Test Bed Warrior Injury Assessment Manikin (WIAMan) Cyber Warfare Fifth-Generation Aerial Target Real-Time Casualty Assessment (RTCA) Additional EW Simulator Units for Surface EW Improvement Program (SEWIP) Operational Testing Anti-Ship Cruise Missile (ASCM) Seekers for GQM-163A Supersonic Target Modification of GQM-163A Coyote Target to Represent another ASCM Threat Long-term Improvement in Fidelity of ASCM Seeker/Autopilot Simulators for EW Testing Torpedo Surrogates for Operational Testing of Anti-Submarine Warfare (ASW) Platforms and Systems
The rapid growth in renewable energy projects across the United States has the potential to adversely affect test capabilities on ranges for current and future systems. Renewable energy proponents have proposed construction of projects in close proximity to and within critical T&E ranges. The abundance of these projects causes a significant concern for DoD test ranges, emphasizing the need for resolution of competing policy objectives for alternative energy and national defense needs. The 2012 Strategic Plan for DoD T&E Resources identifies some near-term test infrastructure needs: • Addressing near-term maintenance, sustainment, and modernization needs of T&E facilities across the Services due to obsolescence and equipment deterioration • Managing the current workforce while shaping future workforce requirements to meet the sophisticated T&E and acquisition challenges brought about by emerging technology • Developing an investment and operational strategy to produce test capability for unmanned and autonomous systems in the air, land, and maritime domains • Continuing initial efforts to develop a cyberspace test infrastructure capability that provides friendly force, opposing force, and background environments with representative threats to offer both defensive and offensive cyber operations The remainder of this section focuses on test infrastructure specific to OT&E. The test infrastructure provides critical support for operational and live fire testing, and DOT&E engages in the DoD budget and review process to address continuing problems related to T&E resources and infrastructure.
T&E Resources
337
t e s t a n d e va l u at i o n r e s o u r c e s Operational Test Agency (OTA) Capabilities and Resources OT&E is performed by independent OTAs, which each Service is required to maintain. OTA capabilities and resources reside principally in a technically competent and available workforce to plan, execute, and evaluate operational test results. Table 1 provides a census of OTA personnel every two years from FY04-FY12. The data indicate military staffing for both the Navy and United States Marine Corps (USMC) OTAs was fairly constant, while the number of military billets in Army
and Air Force OTAs decreased. The Air Force decrease is quite significant, at approximately 36 percent. For civilian personnel, the most significant change is the Marine Corps Operational Test and Evaluation Activity (MCOTEA) increase from FY10-FY12 that reflects policy decisions to insource support for inherited government duties with government civilians and enhance scientific and technical competencies at MCOTEA.
Table 1. OTA Workforce FY02-FY12 Trend (Number of Personnel) Military
FY04
FY06
FY08
FY10
FY12
Army (ATEC-OT)
350
322
306
277
307
Air Force (AFOTEC)
577
548
456
378
369
Navy (COTF)1
223
240
221
217
224
USMC (MCOTEA)
26
28
25
28
28
Total Military
1,176
1,138
1,008
900
928
Civilian
FY04
FY06
FY08
FY10
FY12
Army (ATEC-OT)
650
729
756
715
627
Air Force (AFOTEC)
195
199
166
191
221
Navy (COTF)1
73
76
71
74
73
USMC (MCOTEA)
18
20
24
26
56
Total Civilian
936
1,024
1,017
1,006
977
TOTAL
2,112
2,162
2,025
1,906
1,905
ATEC-OT – Army Test & Evaluation Command – Operational Test COTF – Commander, Operational Test & Evaluation Force 1
AFOTEC – Air Force Operational Test & Evaluation Command MCOTEA – Marine Corps Operational Test & Evaluation Activity
COTF totals exclude VX squadrons and Marine Corps Air Detachments.
Degrees in science, technology, engineering, and mathematics provide a strong understanding of the scientific method and the analytical skills important to rigorous T&E. Additionally, degrees in statistics, operations research, and systems engineering are especially useful when constructing designed experiments
and analyzing data from tests. Table 2 displays the numbers of these targeted degree fields that focus on test design and analysis. Of note is the lack of civilian personnel with degrees in mathematics/statistics in the Navy.
Table 2. Number of civilian personnel with degrees in fields suited to test design and analysis degree area
atec
afotec
cotf
mcotea
Mathematics/Statistics
53
5
0
4
Computer & Information Science
71
13
8
1
Engineering
242
37
9
0
Engineering Technology
23
7
1
0
Biological Sciences
13
1
1
0
Physical Sciences
20
6
3
1
422/531 (79%)
69/89 (76%)
22/27 (81%)
6/41 (15%)
TOTAL
(Percent of non-admin personnel)
ATEC– Army Test & Evaluation Command – Operational Test COTF – Commander, Operational Test & Evaluation Force
338
T&E Resources
AFOTEC – Air Force Operational Test & Evaluation Command MCOTEA – Marine Corps Operational Test & Evaluation Activity
t e s t a n d e va l u at i o n r e s o u r c e s Commander, Operational Test and Evaluation Force (COTF) not only has the lowest number of civilian personnel among all the service OTAs but also only 27 T&E civilians with technical backgrounds. In an October 30, 2013 briefing to DOT&E, COTF recognized the need for increasing civilian staffing to improve scientific, technical, engineering, and mathematical/statistical competencies to effectively meet OT&E needs and proffered his plan to improve his workforce’s technical competency within the Navy’s constraints on the number of civilian billets authorized. DOT&E supports enhancing COTF’s civilian workforce and recommends improving both the size and composition of the workforce to improve operational and live fire test planning, execution, and analysis of Navy systems. Furthermore, COTF (as well as the other OTAs) would benefit from having a senior technical advisor to the Commander who is well versed in the science of experimental design and data analysis and is responsible for ensuring technical rigor across the entire Command.
Continuing Radio Frequency Spectrum Concerns T&E spectrum needs, like those of the rest of DoD, are growing. Bandwidths required by systems under test are expanding as the new system capability expands. Additionally, the number of test conditions and monitored conditions requiring telemetry data has been rising. Test activities are constrained by overlapping signal footprints, so that test schedules of nearby ranges must be interleaved. In June 2010, the White House directed the Secretary of Commerce, working through the National Telecommunications and Information Administration (NTIA) to collaborate with the Federal Communications Commission (FCC) to make available a total of 500 Megahertz (MHz) of federal and non-federal spectrum over 10 years, suitable for both mobile and fixed wireless broadband use. The spectrum must be available to be licensed by the FCC for exclusive use or made available for shared access by commercial and government users to enable deployment of wireless broadband technologies. In January 2011, the NTIA focused on the 1755–1850 MHz spectrum. On March 20, 2013, the FCC issued formal notice to the NTIA that the lower portion of the band (1755–1780 MHz) would be auctioned for wireless broadband as early as September 2014. In July 2013, the White House directed DoD to vacate that portion of the band, which is extensively used by major DoD systems, including: • Small Unmanned Aerial Systems • Tactical Targeting Network Technology • Tactical Radio Relay • High Resolution Video • Precision Guided Munitions • Point-to-Point Microwave Links • DoD Video Surveillance/Robotics • Satellite Operations • Electronic Warfare • Air Combat Training System • Joint Tactical Radio System (JTRS) • Aeronautical Mobile Telemetry
DoD capabilities in the 1755–1850 MHz band, such as the Air Combat Training System, JTRS, and T&E operations using Aeronautical Mobile Telemetry, will be compressed into the upper portion of the band (1780–1850 MHz). The DoD Chief Information Officer (CIO) has proposed the 2025–2110 MHz band (currently used by television broadcasters and video producers) as a possible alternative band for some of the dislocated systems. However, a July 17, 2013 DoD CIO proposal to NTIA is not consistent with prior DoD findings identified in “Spectrum Reallocation Feasibility Study 1755–1850 MHz Band” issued on September 8, 2011, and from revision 1 of this study issued on March 20, 2012. In a July 17, 2013 letter to NTIA, the DoD CIO proposed shared usage of 1755–1780 MHz for a limited but unspecified time. The proposal would establish protection zones applicable to JTRS radios only around Forts Irwin, Polk, Bliss, Bragg, and Hood; White Sands Missile Range; and Yuma Proving Ground based on the propagation within the 1755–1780 MHz spectrum using an antenna height of 5 to 10 feet. Once JTRS radios are decommissioned, DoD’s shared usage of 1755–1780 MHz would end. If this spectrum-sharing plan was implemented, it would significantly impede JTRS operation and use for test and training in the United States. For example, brigade-level training activities at Forts Drum, Campbell, Stewart, Lewis, Riley, Benning, and Sill, and numerous other sites in Alaska and Hawaii plan to use JTRS. Army Reserve and Army National Guard units plan to operate JTRS at sites throughout the United States. The proposed protection zones do not account for radio propagation placement of JTRS radios on air platforms (i.e., helicopters or unmanned aircraft systems such as those used by Combat Aviation Brigades). Additionally, Fort Huachuca Electronic Proving Ground provides JTRS test operations, including the Unmanned Aerial System training center. DoD needs a thoughtful transition plan that adequately supports programs such as JTRS and other critical test and training capabilities. DOT&E anticipates funding for the engineering and equipment acquisitions necessary to vacate the 1755–1780 MHz band will come from the Spectrum Relocation Fund provided for under law to support this change. The DoD CIO estimates the cost to move all operations out of the currently available spectrum at about $3.5 Billion. This estimate assumes only $100 Million will adequately cover the transition costs for only 4 of the 10 systems: Aeronautical Mobile Telemetry, Air Combat Training Systems, JTRS, and Satellite Operations/Electronic Warfare. The DoD “Spectrum Reallocation Feasibility Study 1755–1850 MHz Band,” issued September 8, 2011, determined that reallocation cost to Aeronautical Mobile Telemetry alone would be at least $3.10 Billion, and it would take at least 15 years to make the transition. In 2012, the Test Resource Management Center (TRMC) estimated the cost to retain the current capacity of the ranges (i.e., the number of test operations) to be on the order of $400 Million over 5 years due to continued growth of data transmission rates, the associated costs of developing the
T&E Resources
339
t e s t a n d e va l u at i o n r e s o u r c e s technologies needed to support these data transmission rates, and continuing constraints on the spectrum needed for testing. Advanced EW Test Resources In February 2012, DOT&E identified shortfalls in EW test resources that prevent development, testing, and timely fielding of U.S. systems capable of operating successfully against threats that currently exist, are proliferating, and are undergoing an accelerating pace of significant upgrades. FY13-18 funding was identified to address these shortfalls and assure the needed test resources would be available in time to support developmental and operational testing of systems, including the JSF. DOT&E recommendations include: • Developing a combination of open- and closed-loop threat simulators in the numbers required for operationally realistic open-air range testing of JSF and other systems beginning in 2018 • Upgrading the government anechoic chambers with adequate numbers of signal generators for realistic threat density • Upgrading the JSF mission data file reprogramming lab to include realistic threats in realistic numbers • Providing Integrated Evaluation and Analysis of Multiple Sources intelligence products needed to guide threat simulations • Accelerating the Next Generation Electronic Warfare Environment Generator (NEWEG) program’s production of high-fidelity signal generators Capabilities under development in JSF, F-22 Increment 3.2 A/B, B-2 Defensive Management System, Long-Range Strike Bomber, Next Generation Jammer for the EA-18G, Integrated Defensive Electronic Countermeasures upgrades, as well as several other programs, require the combination of improved government‑owned anechoic chambers and new open-air range test assets. These test resources are necessary for development and adequate, realistic testing of the systems noted above. Unfortunately, progress in initiating this critical program during the past year has lagged expectations considerably. Aegis-Capable Self-Defense Test Ship (SDTS) The close-in ship self-defense battle space is complex and presents a number of challenges for OT&E. For example, this environment requires: • Weapon scheduling with very little time for engagement • Air and Missile Defense Radar (AMDR) and Close-In Weapons System (CIWS) (to deal with debris fields due to previous successful engagements of individual ASCMs within a multi-ASCM raid) • Rapid multi-salvo kill assessments for multiple targets • Transitions from Evolved Sea-Sparrow Missile (ESSM) Command Midcourse Guidance mode to Home-All-the­Way guidance mode • Conducting ballistic missile defense and area air defense missions (i.e., integrated air and missile defense) while simultaneously conducting ship self-defense
340
T&E Resources
• Contending with stream raids of multiple ASCMs attacking along the same bearing, in which directors illuminate multiple targets (especially true for maneuvering threats) • Designating targets for destruction very close-in by CIWS Multiple hard-kill weapons systems operate close-in, including the Standard Missile 2 (SM-2), the ESSM, and the CIWS. Soft‑kill systems such as the Nulka Mk-53 decoy launching system also operate close-in. The short timelines required to conduct successful ship self-defense place great stress on combat system (CS) logic, CS Element (CSE) synchronization, CSE integration, and end-to-end performance. Navy range safety restrictions prohibit close-in testing on a manned ship because the targets and debris from successful intercepts will pose an unacceptable risk to the ship and personnel at the ranges where these self-defense engagements take place. These restrictions were imposed following a February 1983 incident on the USS Antrim (FFG 20), which was struck with a BQM-74 aerial target during a test of its self‑defense weapon systems, killing a civilian instructor. The first unmanned, remotely-controlled SDTS (the Ex-Stoddard) was put into service that same year. A similar incident occurred in November 2013, where two sailors were injured when the same type of aerial target struck the USS Chancellorsville (CG-62) during what was considered to be a low-risk test of its combat system. This latest incident underscores the inherent dangers of testing in the close-in battlespace. While it is expected the investigation into the Chancellorsville incident may cause the Navy to rethink how they will employ these subsonic targets neared manned ships, the Navy has always considered supersonic ASCM targets a high risk to safety, and will not permit flying them directly at a manned ship. The Navy has invested in a current at-sea, unmanned, remotely‑controlled test asset (the SDTS) and is using it to overcome these safety restrictions. The Navy is accrediting a high-fidelity modeling and simulation (M&S) capability utilizing data from the SDTS, as well as data from manned ship testing, so that a full assessment of ship self-defense capabilities of non‑Aegis ships can be completely and affordably conducted. While the Navy recognizes the capability as integral to the test programs for certain weapons systems (the Ship Self-Defense System, Rolling Airframe Missile Block 2, and ESSM Block 1) and ship classes (LPD-17, LHA-6, Littoral Combat Ship, DDG 100, and CVN-78), the Navy has not made a similar investment in an Aegis-capable SDTS for adequate operational testing of the DDG 51 Flight III Destroyer (with Aegis Advanced Capability Build “Next” Combat System and AMDR) capabilities. The current SDTS lacks the appropriate sensors and other combat system elements to test these capabilities. Although the Navy is investigating an improved flight termination system that would permit closer approach of the current GQM-163A supersonic target to manned Aegis ships, it will only permit cross-range offset reduction from the ship to 1 nautical mile (from the current 2.5 nautical miles for the
t e s t a n d e va l u at i o n r e s o u r c e s GQM-163A). That is the same cross-range offset that existed for the GQM-163A predecessor, the MQM-8G (EER) from 1998 to 2005, and Aegis was not able to conduct self-defense scenarios at that time because of the hazard posed by the proximity of the predecessor supersonic target to a manned ship. The November 2013 incident on USS Chancellorsville (CG‑62) underscores the inherent dangers of testing in the close-in battlespace. This leaves no safe alternative but to use an SDTS for complete, end-to‑end ship self-defense testing. Moreover, the cross-range offsets imposed under the closer approach concept would still result in unacceptable lack of realism in threat presentations for purposes of operational testing. DOT&E strongly recommends development of an Aegis-capable SDTS to test ship self-defense systems’ performance in the final seconds of the close-in battle and to acquire sufficient data to accredit ship self-defense performance M&S. Absent this critical resource, the lives of our Sailors and their success in battle will be placed at unacceptable risk. This is because use of the SDTS during the past decade has demonstrated clearly and repeatedly that shortfalls in combat system self-defense performance cannot be found and fixed without the realistic testing possible only using the SDTS. The estimated cost for development and acquisition of this capability over the Future Years Defense Program is approximately $284 Million. Of that, $228 Million would be recouped after the test program completes by installing the hardware in a future DDG 51 Flight III hull. DOT&E has disapproved the AMDR Test and Evaluation Master Plan (TEMP) because, contrary to its predecessor AMDR Test and Evaluation Strategy, the TEMP did not provide for the resources needed to equip an SDTS. Similarly, DOT&E will disapprove the DDG 51 Flight III TEMP if it omits the resources needed to equip an SDTS. Aegis Probability of Raid Annihilation (PRA) Test Bed The Navy has a robust strategy for evaluating the Probability of Raid Annihilation (PRA) for the LHA-6, Littoral Combat Ship 1 and 2, DDG 1000, and CVN-78 ship classes. This strategy, documented in the Air Warfare Ship Self-Defense Enterprise TEMP, is based on a paradigm in which data from lead ship testing are combined with data from testing on the Navy’s unmanned SDTS to accredit an end-to-end M&S tool, known as the PRA Test Bed. In addition to providing the accreditation data, SDTS and lead ship testing satisfy the statutory requirements for testing under operationally realistic conditions, and provide for a qualitative and quantitative assessment of the ship class’ capability. Once accredited, the PRA Test Bed assesses the numeric PRA requirement. Each phase of testing – lead ship, SDTS, and PRA Test Bed – is needed to assess the ship class’ capability. Starting with Aegis Weapon System’s Advanced Capability Build 12, all Aegis platforms must demonstrate that they meet their respective PRA requirements during operational testing. However, the Navy does not have an acceptable strategy for assessing PRA on Aegis Platforms until an Aegis-equipped SDTS
is available. The Navy has stated that they will not acquire an Aegis-equipped SDTS. Consequently, the Navy cannot assess PRA for Aegis platforms. In addition to not having an Aegis-equipped SDTS, the Navy’s M&S suite for the Aegis combat system is not nearly as capable as the Navy’s PRA Test Bed. The Aegis M&S suite falls short of the PRA Test Bed in three important areas: • First, the representation of Aegis in the M&S suite uses a specification-based model as opposed to a tactical code model. While specification-based models can be useful, depending on their intended uses, they are generally of lesser fidelity than tactical code models. This is because they are ultimately limited to how accurately the specifications were implemented in the tactical code. Thus, a perfect specification model of the Aegis Weapon System would accurately represent how it is intended to work, while a tactical code model would represent how it actually works. Almost all models in the Navy’s PRA Test Bed use tactical code representations of the combat system elements. • Second, the Navy’s Aegis M&S suite does not account for all the elements of the Aegis Combat System’s kill-chain in an end-to-end fashion. Although each part of the kill-chain is considered, interactions between the different kill-chain elements are not considered. Live fire test events conducted on the Navy’s current SDTS for other combat systems (e.g., the Ship Self-Defense System) have shown that such interactions can have profound effects on the ship’s capability. The Navy’s PRA Test Bed, via a virtual test range architecture, considers interactions between elements of the kill-chain. • Third, the Navy’s Aegis M&S suite does not adequately account for how ESSM and SM-2’s performance might be affected by different ASCM raid types. The Aegis Weapon System’s strategy attempts to account for these effects, but the current architecture of the M&S suite does not adequately support their inclusion. Live fire test events conducted on the Navy’s current SDTS for other combat systems (e.g., the Ship Self-Defense System) show that these effects can be very important. By comparison, the PRA Test Bed includes these effects via its virtual range. To account for these shortcomings, the Navy should implement an M&S strategy for Aegis Cruisers and Destroyers that is similar to the PRA Test Bed. In order to accredit such a model for operational testing, the Navy should acquire an Aegis-equipped SDTS. Because of the time and cost associated with acquiring an Aegis-equipped SDTS, it is difficult to see how the Navy can provide such an asset prior to DDG 51 Flight III testing in 2022. The strategy and the timelines for developing such a model and acquiring an SDTS should be documented in the Advanced AMDR, the Aegis Modernization, and the DDG 51 Flight III TEMPs. The Navy should also consider adding the DDG 51 Flight III PRA assessment to the existing Air Warfare Ship Self‑Defense Enterprise TEMP to better coordinate the planning and execution of events intended to support the PRA assessment.
T&E Resources
341
t e s t a n d e va l u at i o n r e s o u r c e s Warrior Injury Assessment Manikin (WIAMan) In 2011, DOT&E initiated a research program to improve the Department’s understanding of the cause and nature of injuries incurred in combat by underbody blast (UBB) events and to develop appropriate instrumentation to assess such injuries in testing. Critical research needs include adequate medical data to improve injury assessments during live fire testing and the development of instrumentation designed specifically for the UBB environment. The proposal resulted in an Army-led, five‑year research and development program, known as the Warrior Injury Assessment Manikin (WIAMan), to improve knowledge of occupant injuries due to UBB events. WIAMan utilizes expertise from both inside and outside the Department to develop and execute a widely-scoped, critical medical research plan, which will provide critical data to the materiel and T&E communities. For example, university research partners specializing in injury biomechanics underpin the WIAMan program. The medical data generated under the WIAMan program will support development of a biofidelic prototype anthropomorphic test device (ATD) designed to capture vertical occupant loading, the primary load axis to which occupants are exposed in a UBB event. The WIAMan ATD is a novel approach for understanding the vulnerability of a vehicle’s occupants to the effects of UBB, which supports LFT&E requirements. These advances will better inform users, vehicle designers, testers, and evaluators about the nature and severity of injuries incurred from UBB events. The WIAMan project also supported fabrication of the Accelerative Loading Fixture (ALF), which is a unique research and test facility for replicating the full-scale UBB environment for mounted Soldiers, at Aberdeen Proving Grounds, Maryland. Experiments conducted in the ALF are already contributing new information and insights on human response to UBB. The WIAMan system will use ALF throughout the life of the program for research and for verification testing. The WIAMan Program Office at the Army Research Laboratory manages all aspects of WIAMan development. The medical research is ongoing, and research results are transitioning to the ATD developer. A study of options for a suitable data acquisition system is also underway. In its June 20, 2013 report, 113-44, the Senate Committee on Armed Services recommended a $10 Million increase for WIAMan noting that “…the development of such a test manikin would significantly improve the Department’s ability to measure the projected injuries that could be caused by various blast events caused by improvised explosive devices. Such information would lead to improved survivability of ground combat vehicles.” If received, this funding will help ensure the program meets its schedule for delivering critical information for ground combat vehicles. Cyber Warfare Experimentation, development, testing, training, and mission rehearsal of offensive and defensive cyber-warfighting capabilities require representative cyber environments. Such
342
T&E Resources
environments are made up of distributed cyber ranges capable of interacting and interoperating with other DoD ranges, since cyber‑warfighting capability is a critical enabler of operations in the air, land, sea, and space domains. DOT&E proposed enhancements to existing facilities to create the DoD Enterprise Cyber Range Environment (DECRE) comprised of the National Cyber Range (NCR), the DoD Information Assurance Range, the Joint Information Operations Range (JIOR), and the Joint Staff J6’s C4 Assessments Division (C4AD). DECRE will provide for: • Consistent portrayal of operationally realistic, threat‑representative cyber environments • Expansion of JIOR operations capacity to plan and rigorously execute approximately 100 distinct events per year • Upgrades to introduce cloud-based Regional Service Delivery Points (RSDPs) • Incorporation of technologies emerging from the NCR for rapid design, reconfiguration, and sanitization of networks • Incorporation of various Live, Virtual, and Constructive capabilities • Range environments where advanced cyber-attacks can be conducted to understand the scope and duration of cyber effects, and where training and tactics, techniques, and procedures (TTPs) development can be performed • Implementation of archival capabilities to record and play back live events, and blend mixes of live and previously recorded events • Creation of a stand-alone cyber lab for testing and rehearsal of advanced offensive capabilities Preliminary work in each of the above areas is underway, but development and delivery of these capabilities will depend on the actual funding levels across the Future Years Defense Program. Of note, the first operational RSDP is expected to be fielded in 3QFY14, and will provide the foundation for greater traffic and realism, hosting of NCR technologies and persistent environments, and an expanded number of simultaneous DECRE events. With assistance from DOT&E, the C4AD team developed a high-fidelity environment to examine the effects of cyber-attacks on systems that support Combatant Commands’ Common Operating Picture. This environment will feature prominently in the assessment of U.S. Northern Command’s Vigilant Shield 14 exercise, allowing realistic demonstration of the effects of an advanced cyber-attack. U.S. Pacific and European Commands have also expressed interest in employing this environment in FY14 to confirm and/or improve their abilities to perform their command and control missions in a contested cyber environment. Two other environments are currently under development (Command and Control Battle Management Communications and Aegis weapon systems), and these environments are expected to come online later in FY14. Each of the above environments was motivated by vulnerabilities identified during
t e s t a n d e va l u at i o n r e s o u r c e s DOT&E Information Assurance and Interoperability (IA/IOP) assessments. As funding permits, DOT&E expects to initiate development of several additional environments each year. DOT&E expects that these high-fidelity cyber environments will become essential to IA/IOP assessments, OT&E, and also to the training of the DoD Cyber Mission Force being implemented by U.S. Cyber Command. Although many improvements are in progress, DOT&E expects the demand for high-fidelity cyber environments and range events will exceed the nascent capabilities. For example, U.S. Cyber Command alone estimates that the Cyber Mission Force will require more than 100 training activities each month, a great deal more than the current capability for 100 events per year across all DoD customers. The integration of key U.S. and coalition range nodes and labs for distributed, secure, operationally realistic, and threat-representative cyber environments will further expand the demand. DOT&E will closely monitor and report on the evolution of DECRE during FY14. DOT&E strongly recommends that the currently fragmented management and resourcing of DECRE be consolidated under an Executive Agent. Fifth-Generation Aerial Target Current aerial targets, including the QF-16 (in development) and sub-scale drones, do not adequately represent enhanced fifth-generation fighter capabilities, including low observability, low probability of intercept sensors, and embedded electronic attack. Aerial targets with the capacity to represent these characteristics are necessary for the operational test adequacy of U.S. air-to-air and surface-to-air weapons systems. Over the next five years, the feasibility of completing operationally realistic testing will decline significantly without an aerial target solution. The risk to the DoD in assessing the mission effectiveness of surface-to-air and air-to-air missile systems will be unacceptable without a representative fifth-generation aerial target. Over the next decade, the production and proliferation of fifth-generation fighter aircraft will enhance Anti-Access/Area Denial strategies and, without question, will challenge U.S. air superiority in future conflicts. Current weapon system testing is limited to segmented approaches using a combination of captive‑carry against the F-22 and live-fire against sub-scale and fourth-generation full-scale targets. The capacity to conduct end-to-end testing, from postlaunch acquisition to end-game fusing, against a fifth-generation fighter threat does not exist and constitutes a critical shortfall.
DOT&E initiated studies on the design and fabrication of a dedicated fifth-generation aerial target to evaluate U.S. weapon systems effectiveness. DOT&E requested $40 Million (out of $80 Million total) in the FY14 program review to complete final design, tooling, and prototyping efforts. The Canadian Government informally expressed interest in funding the remaining $40 Million as part of a joint U.S./Canada Defense Development Sharing Agreement. This agreement allows joint research and development efforts funded by DoD and the Canadian Department of Defence Production.
Real Time Casualty Assessment (RTCA) Simulated force-on-force battles must contain enough realism to cause Soldiers and their units to make tactical decisions and react to the real-time conditions on the battlefield. RTCA systems integrate Live, Virtual, and Constructive (LVC) systems to enable these simulated force-on-force battles. RTCA capability provides a means for simulated engagements to have realistic outcomes based on the lethality and survivability characteristics of both the systems under test and the opposing threat systems; therefore, RTCA systems must exhibit critical attributes of real-world combat engagements such as direct and indirect fires, IEDs and mines, realistic battle damage and casualties, a mix of ground and air vehicles, and a competent and capable threat force. RTCA systems must record the time-space position information and firing, damage, and casualty data for all players in the test event. Playback of these data provides a critical evaluation tool when determining the combat system’s capability to support Soldiers as they complete their unit mission. In recent years, Army Test and Evaluation Command (ATEC) has used a portion of its RTCA capability (a combination of the ATEC Player and Event Tracking System, Multiple Integrated Laser Engagement System, and LVC components) to support tests. For Network Integration Evaluation (NIE) 14.1 (scheduled for 1QFY14), DOT&E requested that ATEC use their full RTCA capability to collect data in support of the AN/PRC-117G radio and Nett Warrior evaluations. Shortfalls found during NIE 14.1 will be captured and used to augment the findings of the ongoing Army RTCA study. The Army initiated this study in FY13 to identify capability gaps based on upcoming operational tests and to provide a recommended course of action for necessary improvements. The results of the study were not available as of this writing. A finalized report is due in 1QFY14. DOT&E expects the report to include near-term plans for improving the existing RTCA system in support of upcoming tests, as well as plans for a long-term sustained capability. In addition to improving the existing system, and due to their common requirements and limited budgets, the Army test and training communities are working together on a future system called the Army – Tactical Engagement Simulation System. DOT&E supports this test and training synergy since the training community can use RTCA instrumentation developed for OT&E once the system is fielded. RTCA is essential to realistic force-on-force testing of current and future land and expeditionary warfare systems, and DOT&E requires RTCA for systems such as Ground Combat Vehicle, Amphibious Combat Vehicle, Bradley and Abrams Modernization, Armored Multi-purpose Vehicle, Apache Block III, Joint Lightweight Tactical Vehicle, and Stryker upgrades. The estimated cost for improvements to the current ATEC RTCA system is $35 Million over the next five years. The cost to develop the Army – Tactical Engagement Simulation System is not known at this time.
T&E Resources
343
t e s t a n d e va l u at i o n r e s o u r c e s Additional EW Simulator Units for Surface Electronic Warfare Improvement Program (SEWIP) Operational Testing At present, there exists only one each of the Kappa, Uniform, and Gamma EW simulators to support SEWIP operational testing. These simulators use Lear Jets as platforms to fly against shipboard EW systems. SEWIP Block 2 is the latest EW system under development. More than one of each type of simulator are needed (e.g., one for each Lear Jet) for adequate SEWIP Block 2 testing in FY14 using threat-realistic stream raid profiles. An estimated development/procurement cost is $5 Million. The SEWIP Block 3 program needs a Lear Jet-mountable Gamma asset for the FY17 IOT&E to present multiple simulated threats to SEWIP simultaneously. The estimated cost for acquisition of a second asset is $15 Million. Anti-Ship Cruise Missile (ASCM) Seekers for GQM-163A Supersonic Target Operationally realistic emissions from the GQM-163A supersonic target require threat-representative ASCM seekers that will stay locked on the target ship. This capability will provide threat‑representative stimulation for shipboard EW systems, in addition to ensuring that the ship’s combat system has a constant track of the incoming target emissions for launching (and guiding, on those same emissions) Rolling Airframe Missile Block 1 and/or Block 2 missiles as interceptors. This unit would be similar to the seeker used in the BQM-34 Open Loop Seeker subsonic target and the STEERAN unit currently used in the BQM-74E subsonic target. Since the diameters of the GQM and BQM targets differ greatly, the new ASCM seeker requires extensive re-engineering and testing to adapt the BQM unit to fit the GQM without disturbing the GQM kinematics/maneuverability. CVN-78/Rolling Airframe Missile Block 2 requires this capability for adequate operational testing in FY17. Estimated development cost is $10 Million to $20 Million. Estimated unit cost is $500 Thousand.
Modification of GQM-163A Coyote Target to Represent another ASCM Threat The Navy’s GQM-163A Coyote Validation Report of May 2006 identified two threats that the Coyote could fundamentally represent. Thus far, attention has focused mostly on a Coyote representation of one of the two threats. DOT&E recommends an engineering analysis to determine what alterations to the Coyote vehicle should be made to use it as a surrogate for the second threat discussed in the GQM-163A Coyote Validation Report. The results of the engineering analysis will inform the Coyote alteration to provide targets for IOT&E of the Aegis Modernization program in FY17 as well as the Aegis DDG Flight III program in FY23. The estimated cost of the analysis is $3 Million. Estimated cost for alteration of existing Coyotes is $150 Thousand per target for 12 targets, or $1.8 Million total. Four targets (two primary plus two backups) would be for the Aegis Modernization IOT&E, and eight targets (four primary plus four backups) would be for the Aegis DDG Flight III IOT&E.
344
T&E Resources
Long-term Improvement in Fidelity of ASCM Seeker/Autopilot Simulators for EW Testing Fidelity of ASCM threat representation during electronic warfare testing in operational environments remains an area for improvement due to the continued reliance on manned aircraft for captive carry of the simulators. The aircraft cannot fly at the high speeds and low altitudes needed for a full representation of ASCM threats. Some plausible improvements needing examination and proposed solutions include: • Recoverable, unmanned aerial vehicles using embedded, miniaturized simulators that are maneuverable at ASCM speeds and altitudes • Encrypted telemetry to track system responses to electronic attack against these simulators • Human-controlled override capability These aerial vehicles would support IOT&E of SEWIP upgrades and FOT&E with new ship classes in the post-FY23 timeframe. Estimated development cost is $120 Million. Estimated unit cost is $15 Million. Torpedo Surrogates for Operational Testing of Anti‑Submarine Warfare (ASW) Platforms and Systems Operational testing of ASW platforms and related systems includes the ability to detect, evade, counter, and/or destroy an incoming threat torpedo. The determination of system or platform performance is critically dependent on a combination of the logic used for acquisition, the dynamic and noise characteristics, and fusing methods of the incoming torpedo. Due to differences in technological approach and development, U.S. torpedoes are not representative in many of these torpedo characteristics for many highly proliferated torpedoes, particularly those employed in Anti-Surface Warfare (ASuW) by other nations. Operational testing that is limited to U.S. exercise torpedoes will not allow the identification of existing limitations of ASW platforms and related systems against threat torpedoes and will result in uninformed decisions in the employment of these same platforms in wartime. A January 9, 2013 DOT&E memorandum to the Assistant Secretary of the Navy (Research, Development & Acquisition) identifies specific threat torpedo attributes that the threat torpedo surrogate(s) must be evaluated against. The non-availability of threat-representative torpedo surrogates will prevent adequate operational testing for ASW platforms and related systems, as well as adversely affect tactics development and validation of these tactics within the fleet. DOT&E estimates that DoD will need approximately $500 Thousand to conduct a study of torpedo surrogate development options, including life-cycle and operation cost, quantity and types of torpedo surrogates required, and employment methodology. DOT&E believes that surrogate development and production for threat torpedoes will benefit from an enterprise approach to prevent burdening a single acquisition program.
Joint Test and Evaluation
Joint Test and Evaluation
JT&E Program
Joint Test and Evaluation (JT&E) The primary objective of the Joint Test and Evaluation (JT&E) Program is to provide rapid solutions to operational deficiencies identified by the joint military community. The program achieves this objective by developing new tactics, techniques, and procedures (TTPs) and rigorously measuring the extent to which their use improves operational outcomes. JT&E projects may develop products that have implications beyond TTPs. Sponsoring organizations submit these products to the appropriate Service or Combatant Command as doctrine change requests. Products from JT&E projects have been incorporated into joint and multi-Service documents through the Joint Requirements Oversight Council process and through coordination with the Air, Land, Sea Application Center. The JT&E Program also develops operational testing methods that have joint application. The program is complementary to, but not part of, the acquisition process. The JT&E Program has two test methods available for customers: the traditional Joint Test and the Quick Reaction Test (QRT). The traditional Joint Test method is, on average, a two-year joint test project, preceded by a six-month Joint Feasibility Study. A Joint Test involves an in-depth, methodical test and evaluation of issues and seeks to identify solutions. DOT&E funds the sponsor-led test team, which provides the customer periodic feedback and useful, interim test products. The JT&E Program annually charters two new Joint Tests. The program managed six Joint Tests in FY13 that focused on the needs of operational forces. Projects annotated with an asterisk (*) completed in FY13. • Joint Advanced Capability Employment (J-ACE) • Joint Counter-Low, Slow, Small Unmanned Aircraft Systems (JCLU) • Joint Cyber Operations (JCO)* • Joint Deployable Integrated Air and Missile Defense (JDIAMD)
• Joint Unmanned Aircraft Systems (UAS) Digital Information Exchange (JUDIE)* • Unmanned Aircraft Systems – Airspace Integration (UAS-AI) QRTs are intended to be less than a year in duration and solve urgent issues. The program managed 16 QRTs in FY13: • Battlefield Airborne Communications Node (BACN) Intra Flight Data Link Subsystem and Multi-Domain Integration (BIS-MDI)* • Civil Intelligence Fusion Concept of Operations (CIFC)* • Computer Network Defense Service Provider (CNDSP)* • Electromagnetic Battle Management Concept of Operations Development and Evaluation (E-CODE) • En-Route Mission Command Capability (EMCC) • Heterogeneous Sensor Integration (HSI) • Joint All-Domain Situational Awareness (J-ADSA)* • Joint Battlespace Awareness via Data Link (J-BADL)* • Joint Beyond Line-of-Sight Command and Control (JBC2)* • Joint Graphical Rapid Assessment of Mission Impact (J-GRAMI) • Joint Integration of Cyber Effects (J-ICE) • Joint Logistics Enterprise Data Sharing (JLEDS) • Joint Positive Hostile Identification (J-PHID) • Joint Sensor Awareness to Target Tracking (J-SATT) • Joint Threat Assessment and Negation for Installation Infrastructure Control Systems (JTANIICS)* • Unmanned Aircraft Systems – Airspace Integration (UAS- AI)* As directed by DOT&E, the program executes special projects that address DoD-wide problems. The program managed two special projects in FY13: • Rapid Acquisition by Sniper1K Track and Attack (RASTA)* • Joint Personnel Recovery Collaboration and Planning (JPRCaP)
JOINT TESTS JOINT ADVANCED CAPABILITY EMPLOYMENT (J-ACE) Sponsor/Start Date: U.S. Strategic Command (USSTRATCOM)/August 2011 Purpose: To develop, test, and evaluate a standardized process to support the Joint Force Commander’s ability to employ enhanced advanced capabilities to overcome complex targeting challenges. Products/Benefits: • A repeatable operational employment process that will enhance planning by developing, evaluating, and coordinating concepts of employment (CONEMPs) that can be used by the Joint Staff, Combatant Commands, Services, and National Security Agency to solve complex targeting challenges
• Multiple enhanced advanced capability CONEMPs to overcome complex targeting challenges that are approved and signed at the General Officer/Flag Officer level and maintained by the appropriate Combatant Command or Service component • Relevant training scenarios and vignettes • Documented effects associated with techniques against representative targets • Developed CONEMPs allow for expeditious development of operational concept of operations (CONOPS) and improved special program capability approval packages
JT&E
345
JT&E Program JOINT COUNTER LOW, SLOW, SMALL UNMANNED AIRCRAFT SYSTEMS (JCLU)
Purpose: To develop and test joint planning and execution processes and procedures for deployable IAMD for the homeland.
Sponsor/Start Date: Air Force/August 2012
Products/Benefits: • IAMD process modeling that provides a comprehensive view of the integrated planning and execution process • NORAD and USNORTHCOM current operations planning processes, checklists, and procedures for IAMD • Continental NORAD Region, Alaska NORAD Region, and Air Forces North planning and execution TTPs for IAMD • Naval Forces North and Third Fleet planning and execution TTPs for naval support of IAMD • Army North planning and execution TTPs for operational control of ground-based IAMD forces • 263rd Army Air and Missile Defense Command planning and execution TTPs for IAMD
Purpose: To develop, test, and evaluate integrated air and missile defense (IAMD) operator TTPs to increase operators’ ability to detect, track, and identify adversary low, slow, and small Unmanned Aircraft Systems (UAS) and provide timely notification to the Area Air Defense Commander. Products/Benefits: • TTPs to increase the operators’ ability to detect, track, and identify this UAS threat category • Integration of information from National Technical Means into a tactical datalink to support situational awareness and target identification • Development of the operational architecture and organizational relationships that will increase the cross-sharing of tactical information to increase the operators’ ability to execute the joint engagement sequence
JOINT CYBER OPERATIONS (JCO)* (Closed January 2013) Sponsor/Start Date: U.S. Pacific Command (USPACOM)/August 2010
Purpose: To assess, develop, and evaluate joint TTPs to employ an adaptive cyber defense Virtual Secure Enclave strategy to enhance and ensure the protection and availability of critical command and control services. Products/Benefits: • Developed CONOPS, TTPs, and related training packages to provide the following capabilities-- Addressed network vulnerabilities of critical command and control services by enabling Joint Task Force Commanders to employ an adaptive cyber defense Virtual Secure Encalve to protect against, detect, and respond to cyber threats against specific command and control applications at the operational level - Provided the Commander with situational awareness and cyber defense options to maintain a proactive defensive posture - Facilitated a systematic approach to implement the principles of war in the cyber domain • Tested and validated operational effectiveness of joint task force implementation • Received CONOPS endorsement by the Joint Requirements Oversight Council for DoD-wide use JOINT DEPLOYABLE INTEGRATED AIR AND MISSILE DEFENSE (JDIAMD) Sponsor/Start Date: North American Aerospace Defense (NORAD), U.S. Northern Command (USNORTHCOM), Army Space and Missile Defense Command/August 2011
346 JT&E
JOINT UNMANNED AIRCRAFT SYSTEMS (UAS) DIGITAL INFORMATION EXCHANGE (JUDIE) (Closed in September 2013) Sponsor/Start Date: Air Force/August 2010 Purpose: To develop, test, and evaluate cross-component UAS information exchange TTPs used to improve joint battlespace situational awareness and target prosecution capabilities for tactical commanders at the brigade level and below. Products/Benefits: • Standardized terminology for UAS information exchange • Recommended information portal and situational awareness display technology currently in use by the components to improve the efficiency of UAS information exchange • Introduced information exchange TTPs to combat training centers and formal training units • Provided comprehensive UAS Information Exchange TTPs and associated Quick Reference Guide • Integrated best practices and lessons learned into both joint and Service-specific TTPs UNMANNED AIRCRAFT SYSTEMS AIRSPACE INTEGRATION (UAS-AI) Sponsor/Start Date: NORAD, USNORTHCOM, and the Army Test and Evaluation Command/August 2012 Purpose: Standardize and evaluate procedures to effectively operate UAS in the National Airspace System (NAS). The UAS-AI Joint Test utilizes the product and builds upon the working relationships developed in the UAS-AI QRT.
Products/Benefits: • Standardized procedures for predictably operating UAS in the NAS under routine, lost command link, lost two-way radio communications, and lost sense and avoid conditions • A common lexicon for UAS operations in the NAS • Partnership and collaboration with the Federal Aviation Administration to integrate UAS in the NAS by 2015
JT&E Program Quick Reaction Tests BATTLEFIELD AIRBORNE COMMUNICATIONS NODE (BACN) INTRA-FLIGHT DATALINK SUBSYSTEM AND MULTI-DOMAIN INTEGRATION (BIS-MDI)* (Closed February 2013) Sponsor/Start Date: USPACOM/November 2011 Purpose: To develop and evaluate TTPs for the BIS-MDI capability, an upgrade to the basic BACN system, to provide interoperability across multi-band voice and datalink communications in order to bridge widely separated Link 16 networks. This will greatly enhance situational awareness, information sharing, and operational effectiveness, especially between fourth- and fifth-generation fighter aircraft and surface shooters. Products/Benefits: • Fusion of sensor information from multiple sources, including fourth- and fifth-generation platforms, to enhance the operator’s common operational picture • Joint and coalition operator CONOPS and TTPs to employ the BIS-MDI capability in support of potential combat support operations conducted in an anti-access and area-denial environment in the USPACOM theater
CIVIL INTELLIGENCE FUSION CONCEPT OF OPERATIONS (CIFC)* (Closed January 2013) Sponsor/Start Date: Joint Staff/January 2012 Purpose: To test and validate the Joint Staff CIFC that addresses how intelligence organizations provide sufficient support to collecting and integrating civil information, in order to allow the Joint Force Commander to obtain a holistic view of the operational environment. Products/Benefits: • Validated and improved CONOPS for fusion of civil intelligence • Joint doctrine change requests submitted to the Joint Staff for consideration • Connects sources of civil information with planners, operators, and intelligence professionals, creating a community of interest • Provided processes and architecture for improved information sharing resulting in better knowledge of the operational environment COMPUTER NETWORK DEFENSE SERVICE PROVIDER (CNDSP)* (Closed September 2013) Sponsor/Start Date: DoD Chief Information Office/July 2012 Purpose: To develop, evaluate, and formalize DoD-level TTPs to ensure the capability exists within DoD’s CNDSPs to guide day‑to‑day operations and ensure an acceptable level
of performance by the CNDSP when facing a capable cyber adversary. Products/Benefits: • Developed and validated CNDSP Performance Evaluation TTPs that provide a methodical, repeatable, and verifiable framework and instructions to measure DoD’s CNDSPs from a performance perspective • Developed measures of performance for detect and respond services that will be incorporated into the next release of the Evaluator’s Scoring Metrics for use by DoD’s CNDSPs to conduct self-assessments and the DoD certification authorities to conduct formal certification and accreditation evaluations • Mitigated vulnerabilities to product sponsors and hosting sites discovered as a result of the project’s work and updated organizational cyber defense TTPs, thus enhancing DoD’s cyber defense posture ELECTROMAGNETIC SPECTRUM BATTLE MANAGEMENT CONCEPT OF OPERATIONS DEVELOPMENT AND EVALUATION (E-CODE) Sponsor/Start Date: USSTRATCOM/March 2013 Purpose: To validate a CONOPS establishing a Combatant Command or Joint Task Force-level Joint Electromagnetic Spectrum Operations cell. Products/Benefits: The E-CODE-developed product is a validated CONOPS to provide-• Integrated Joint Electromagnetic Spectrum Operations cell planning, tasking, coordination, and conflict resolution processes • Synchronized operations to shape the electromagnetic battlespace to meet the Commander’s objectives • Codified processes to gain and maintain freedom of movement in the electromagnetic operating environment while denying access to adversaries • Improved information exchange, situational awareness, and command and control decision processes to reduce the timeline for dynamic reallocation of the congested and contested electromagnetic spectrum • Improved processes for prioritizing, nominating, and neutralizing electromagnetic spectrum targets EN-ROUTE MISSION COMMAND CAPABILITY (EMCC) Sponsor/Start Date: XVIII Airborne Corps/May 2013 Purpose: To develop, test, and refine TTPs for installation and operational use of a robust EMCC that provides global response forces with the ability to establish and maintain optimal situational awareness while airborne, en-route, and on the ground to conduct forcible entry operations.
JT&E
347
JT&E Program Products/Benefits: • Formalize TTPs for EMCC installation and operation • Provide guidance for leveraging EMCC to support forcible entry operations • Measure the increase in the Commander’s situational awareness during forcible entry operations compared to current communications systems • Develop supporting architectures for EMCC connectivity HETEROGENEOUS SENSOR INTEGRATION (HSI) Sponsor/ Start Date: USPACOM/March 2013 Purpose: To develop and test TTPs for training, alignment, and integration of experimental sensors with existing, signature-based sensors to enhance situational awareness of key terrain in cyberspace. The objective is to demonstrate, through the test and evaluation process, a significant improvement in the rate (low false-positive rate) and precision of detection of intrusions when employing heterogeneous sensor pairs in accordance with the developed TTP.
Products/Benefits: • Employment of the HSI-developed TTPs will greatly improve network defenders’ detection rates while reducing false-positive alert rates associated with network intrusions. Thus, the TTPs will provide a means to enhance the Joint Force Commander’s situational awareness of key terrain in cyberspace. • The operational CONEMP being developed will describe when and where it will be appropriate to employ the developed TTPs by showing how the capability fits within the broader context of joint operations. JOINT ALL DOMAIN SITUATIONAL AWARENESS (J-ADSA)* (Closed June 2013) Sponsor/Start Date: NORAD and USNORTHCOM/June 2012 Purpose: To develop and test necessary TTPs to overcome challenges associated with integrating disparate cross-domain activities and events that must be processed, synthesized, and disseminated in a timely, comprehensive manner in order to enable NORAD-USNORTHCOM leadership to gain and maintain comprehensive, integrated situational awareness and decision superiority. Products/Benefits: The J-ADSA-developed TTPs improved internal command and multi-component coordination and increased the ability to synthesize cross-domain information. Specific TTPs delivered to the NORAD-USNORTHCOM staff included • Crew Information Form and Analysis Checklist • Homeland Defense Decision Support Matrix • Significant Activities Tracker • A geospatial presentation capability for daily operations
348
JT&E
JOINT BATTLESPACE AWARENESS VIA DATA LINK (J-BADL)* (Closed August 2013) Sponsor/Start Date: NORAD and USNORTHCOM/August 2012
Purpose: To research and develop TTPs that will focus NORAD, USNORTHCOM, and supporting commands’, use of joint global sensor information to provide cuing to address priorities, adjust surveillance assets, or position existing forces in executing the joint engagement sequence against advanced air threats in defense of the Homeland. Products/Benefits: The expected test product includes TTPs that describe the execution of joint engagement sequence capabilities to be used operationally by NORAD and USNORTHCOM, as well as by other Combatant Commands and government agencies, against advanced air threats. JOINT BEYOND LINE-OF-SIGHT COMMAND AND CONTROL (JBC2)* (Closed July 2013) Sponsor/Start Date: Air Force/July 2012 Purpose: To develop and evaluate TTPs for operations centers to plan and employ the Beyond-Line-of-Sight (BLOS) Command and Control system- of-systems to support real-time, collaborative command and control capabilities. Products/Benefits: • TTPs for collaboration between U. S. Central Command operations centers in support of responsive fleet defense and strike operations • Integrated planning and employment of the BLOS Command and Control network within a joint theater of operations • Enhanced responsiveness of the theater component operations centers through improved exchange of critical information and data • Enhanced real-time situational awareness to avoid fratricide, mitigate civilian casualties, and accurately locate and identify enemy combatants JOINT GRAPHICAL RAPID ASSESSMENT OF MISSION IMPACT (J-GRAMI) Sponsor/Start Date: USSTRATCOM/December 2012 Purpose: To develop and evaluate TTPs for mission impact documentation, collaboration, and visualization of problem sets for USSTRATCOM’s nuclear command and control and space missions. The TTPs will leverage the Graphical Mission Impact Tool that USSTRATCOM’s Mission Assurance Division created to graphically display mission impacts resulting from loss or disruption of critical systems, assets, and infrastructure. J-GRAMI also provides capability to USPACOM, which operationally endorses the QRT and will receive its final product.
JT&E Program Products/Benefits: • TTPs that provide USSTRATCOM and USPACOM an operational mission impact evaluation methodology for loss or disruption of critical systems, assets, or infrastructure • Detailed directions for using Graphical Mission Impact Tool to do the following-- Dynamically identify vulnerabilities in critical systems, assets, and defense infrastructure needed to support assigned missions and mission-essential tasks - Assess and graphically represent potential impacts resulting from loss or disruption of critical systems, assets, or infrastructure - Provide Combatant Command leadership with an enhanced capability for informed decision making JOINT INTEGRATION OF CYBER EFFECTS (J-ICE) Sponsor/Start Date: USPACOM/October 2012 Purpose: To develop and evaluate TTPs that enable a joint cyber center to integrate cyber effects into joint operation planning, joint targeting, and operations. Products/Benefits: • Establish and refine processes for planning, targeting, and execution of offensive cyber operations • Enable the Combatant Commander’s application of operational art to project cyber power’s capability to achieve an objective • Provide a framework for command and control of newly- formed cyber forces within the command • Develop a doctrine, organization, training, materiel, leadership and education, personnel, facilities change request on factors that impede planning for offensive cyber operations • Validate TTPs through an assessment of developed processes across Combatant Commands JOINT LOGISTICS ENTERPRISE DATA SHARING (JLEDS) Sponsor/Start Date: Joint Staff, U.S. Transportation Command/ January 2013 Purpose: To implement enterprise data exposure methods necessary to overcome information sharing impediments and inefficiencies imposed by point-to-point systems interfaces. The project will develop and test credentialed-access, web-based enterprise interfaces to multiple sources of data regarding redeployment and retrograde of equipment and materiel from the U.S. Central Command theater. The interface will present these data with aggregated or detailed visualizations. Products/Benefits: • Improves awareness of logistics movement status, allowing for better management decisions and significant transportation cost savings • Exposes logistics data to the enterprise, eliminating the need for point-to-point interfaces and eliminating the overhead associated with managing individual user accounts
JOINT POSITIVE HOSTILE IDENTIFICATION (J-PHID) Sponsor/Start Date: NORAD and NORTHCOM/March 2013 Purpose: To develop and evaluate TTPs to improve IAMD decision-making processes that will enable faster and more accurate responses in an increasingly dynamic air and missile defense environment. The goal of this QRT is to minimize the time required to positively identify a contact of interest and increase the time available to take action to counter air and missile threats. Products/Benefits: • IAMD TTPs to more efficiently and effectively execute the joint engagement sequence in defense of the Homeland • J-PHID-developed algorithm will assign a confidence level to a contact of interest, resulting in improved IAMD decision- making processes, reduced response time, and increased accuracy while executing the joint engagement sequence JOINT SENSOR AWARENESS TO TARGET TRACKING (J-SATT) Sponsor/Start Date: USPACOM/March 2013 Purpose: To develop and evaluate TTPs for the rapid injection of fused track data derived from the Dynamic Time Critical Warfighting Capability into available tactical datalinks Products/Benefits: • Provide methods to disseminate unverified intelligence to provide timely situational awareness of mobile threats to warfighters over tactical networks • Enable joint operators at the tactical edge to find, fix, track, target, and engage time-sensitive targets, with intelligence- derived situational awareness JOINT THREAT ASSESSMENT AND NEGATION FOR INSTALLATION INFRASTRUCTURE CONTROL SYSTEMS (JTANIICS)* (Closed January 2013) Sponsor/Start Date: Air Force/January 2012 Purpose: To develop and validate a risk assessment handbook for use by installation commanders to strengthen their industrial control system security posture. Products/Benefits: This QRT developed a JTANIICS handbook that-• Enabled an installation Commander to conduct self- assessments of industrial control system vulnerabilities • Provided guidelines for assigning priority to vulnerabilities based on mission requirements • Validated a methodology that aids in identifying commonly overlooked systems that can potentially allow unauthorized access to mission-critical and safety-critical systems
JT&E
349
JT&E Program UNMANNED AIRCRAFT SYSTEMS – AIRSPACE INTEGRATION (UAS-AI)* (Closed October 2012) Sponsor/Start Date: NORAD and USNORTHCOM/ January 2012 Purpose: To test and evaluate the flight profiles in the Joint CONOPS for UAS Airspace Integration in a simulation environment prior to increased DoD access to the NAS. The
UAS-AI QRT, initiated while the UAS-AI Joint Test was in the feasibility study phase, produced results that were utilized by the UAS-AI Joint Test. Products/Benefits: Recommended improvements to the CONOPS and provided all test results to the USD(AT&L) UAS Task Force and UAS-AI Joint Test project, identifying CONOPS gaps revealed by the QRT.
Special Projects JOINT PERSONNEL RECOVERY COLLABORATION AND PLANNING (JPRCaP) Sponsor/Start Date: Joint Personnel Recovery Agency/ January 2013 Purpose: To employ multi-Service and other DoD agency support, personnel, and equipment to develop, test, and evaluate procedures that will formalize planning, crisis response, and information sharing between the Combatant Commands, senior defense officials, and State Department defense attachés prior to and during personnel recovery responses where a State Department Chief of Mission (generally the ambassador), and not a DoD official, is the lead U.S. Government authority for activity in a country. Currently, no formal personnel recovery planning or training takes place. Ad hoc responses during a crisis can waste time and resources, which puts the isolated person, and any rescue force, at additional risk. Products/Benefits: Processes and documents that provide formal personnel recovery planning and training protocols that inform senior defense officials and State Department defense attachés in United States embassies, in coordination with the Combatant Commands, on how to build and implement country- specific personnel recovery plans tailored for the Chief of mission setting.
350 JT&E
RAPID ACQUISITION BY SNIPER 1K TRACK AND ATTACK (RASTA)* (Closed February 2013) Sponsor/Start Date: USPACOM/November 2011 Purpose: To develop and test TTPs that improve the timely generation of specific target tracking capabilities for tactical fighter aircraft during combat employment in an environment that includes Advanced Electronic Attack waveforms. Products/Benefits: The RASTA-developed TTPs provided Service members the ability to generate target-quality information to enhance kill chain effectiveness while operating in an Advanced Electronic Attack waveform environment. The TTPs will support USPACOM, its functional components, other Combatant Commands, and Service missions.
Center for Countermeasures
Center for Countermeasures
Center for countermeasures
Center for Countermeasures The Center for Countermeasures (the Center) is a joint activity that directs, coordinates, supports, and conducts independent countermeasure/counter-countermeasure (CM/CCM) test and evaluation (T&E) activities of U.S. and foreign weapon systems, subsystems, sensors, and related components. The Center accomplishes this work in support of DOT&E, Deputy Assistant Secretary of Defense (DASD) for Developmental Test and Evaluation (DT&E), weapon system developers, and the Services. The Center’s testing and analyses directly support evaluation of the operational effectiveness and suitability of CM/ CCM systems. Specifically, the Center: • Performs early assessments of CM effectiveness against threat and DoD systems and subsystems. • Determines performance and limitations of missile warning and aircraft survivability equipment (ASE) used on rotary- wing and fixed-wing aircraft. • Determines effectiveness of precision guided weapon (PGW) systems and subsystems when operating in an environment degraded by CMs. • Develops and evaluates CM/CCM techniques and devices. • Develops and tests new CMs in operationally realistic environments. • Provides analysis and recommendations on CM/CCM effectiveness to Service Program Offices, DOT&E, DASD(DT&E), and the Services. • Supports Service member exercises, training, and pre- deployment activities.
and reporting on more than 40 DoD electro-optical systems or subsystems, with special emphasis on rotary-wing survivability. The Center participated in operational/developmental tests for rotary- and fixed-wing ASE, PGWs, hostile fire indicator (HFI) data collection, experimentation tests, and pre- deployment/ exercise support involving the use of CM/CCM. Approximately 49 percent of the Center’s efforts were spent on ASE testing, with the majority of these efforts in support of rotary-wing aircraft. About 11 percent of the Center’s efforts were spent on PGW, foreign system, and other types of field testing not related to ASE. Approximately 6 percent of the Center’s efforts were dedicated to overseas contingency operations support, with emphasis on CM-based, pre-deployment training for rotary-wing units. Thirty-two percent of the Center’s efforts were spent on internal programs to improve test capabilities and to develop test methodologies for new types of T&E activities. The Center continued to develop multiple test tools for evaluating ASE infrared countermeasure (IRCM) systems and hostile fire signature (HSIG) database models used to support development of HFI systems. In addition, the Center is expanding in the electronic warfare realm with a new internally funded Portable Range Threat Simulator capability. The Center dedicates about 2 percent of its efforts to providing subject matter expertise to numerous working groups and task forces. The following activities are representative of those conducted by the Center during the past year.
During FY13, the Center completed over 50 T&E activities. The Center’s support of these activities resulted in analysis aSE and HSI Activities RESEARCH AND DEVELOPMENT ACTIVITY
ROTARY-WING TEST EVENTS
Army: Distributed Aperture Directed Infrared Countermeasures System (DADS) • Sponsor: Information Intelligence Warfare Directorate (I2WD), Communications-Electronics Research, Development, and Engineering Center, U.S. Army Research, Development and Engineering Command • Activity: The Center provided Joint Mobile IRCM Test System (JMITS) infrared (IR) simulations, high-temperature thermal sources, and a select assortment of post-launch configured Man Portable Air Defense System (MANPADS) IR seekers. The DADS was stationary with respect to the JMITS and seekers during the data collection events. • Benefit: The results and measurements obtained from these tests will directly benefit and enhance the DADS tracker development and I2WD’s related modeling and simulation efforts.
Navy: Future Naval Capabilities of Advanced IR Countermeasure Techniques Technology Demonstration Phase II • Sponsors: Navy Program Executive Officer, Advanced Tactical Aircraft Protection Systems Program Office, and Naval Research Laboratory • Activity: The Center provided JMITS two-color IR simulations and reactive captive IR seekers to verify the performance of advanced IRCM techniques. The Center provided all data collected to the sponsors for their assessments. • Benefit: The data collected from this effort allowed the sponsors to assess the performance of the advanced IRCM techniques against reactive IR static threat seekers and to modify these advanced IRCM techniques for improved performance.
Center for Countermeasures
351
Center for countermeasures Navy: Department of the Navy (DoN) Large Aircraft Infrared Countermeasures (LAIRCM) Super Back End Processor (SBEP) Regression Flight Test • Sponsor: Navy Program Executive Officer, Advanced Tactical Aircraft Protection Systems Program Office • Activity: The Center provided JMITS two-color IR simulations to support a proof of Engineering Change Proposal upgrade to the DoN LAIRCM. The Center provided all data collected to the sponsors for their assessments. • Benefit: The testing provided a cost-effective test venue for collecting critical data needed to assess the performance of the DoN LAIRCM SBEP prior to installation on fleet aircraft. Navy: CH-53E DoN LAIRCM Advanced Threat Warner (ATW) Risk Reduction Flight Test • Sponsor: Navy Program Executive Officer, Advanced Tactical Aircraft Protection Systems Program Office • Activity: The Center provided JMITS two-color IR missile simulations and threat-representative laser beamrider, designator, and rangefinder to collect system response data for assessing the ATW sensors and processor. • Benefit: The testing provided a cost-effective test venue for collecting critical data needed to assess performance of the DoN LAIRCM ATW sensors and processor. Navy: CH-53E DoN LAIRCM ATW Sensor Upgrade, Missile Warning and Laser Warning Flight Test • Sponsor: Navy Program Executive Officer, Advanced Tactical Aircraft Protection Systems Program Office • Activity: The Center provided JMITS two-color IR missile simulations and threat-representative laser beamrider, designator, and rangefinder to collect system response data for assessing the ATW missile and laser warning systems. • Benefit: The testing provided the Navy with a cost-effective test venue for collecting critical data needed to assess performance of the DoN LAIRCM ATW sensors and software. Navy: Naval Research Laboratory Laser Beam Rider Detection Experiment • Sponsor: Naval Research Laboratory • Activity: The Center provided test assets and crew to support a joint U.S./Canada laser warning experiment. • Benefit: The sponsor used the data from this test effort to improve laser warning algorithms. OSD: Rotorcraft Aircraft Survivability Equipment (RASE) Experiment 2013 • Sponsor: Assistant Secretary of Defense for Research and Engineering • Activity: The Center served as experiment director and radiometric data collector during the RASE 2013 Tower event at the Weapons Survivability Laboratory Remote Test Site, China Lake, California. Twenty-three different systems mounted on an SH-60 helicopter installed on a hover stand participated in the experiment. • Benefit: The RASE Experiment is a venue focused on ASE that enhances decision makers’ understanding of ASE performance and advances the ASE state-of-the-art testing.
352
Center for Countermeasures
The RASE Experiment is expected to improve realism and standardization in the testing of ASE, improve the extent of testing prior to fielding, and provide an opportunity for multiple developers to save costs overall. Fixed-Wing Test Events Air Force: LAIRCM EC-130J Operational Flight Test • Sponsor: 46th Test Wing Test Squadron Defensive Systems and Mobility Directorate, Air Force Life Cycle Management Center • Activity: The Center provided JMITS missile simulators and crews to perform two-color IR simulations to collect system response data for assessing the LAIRCM system as installed on the EC-130J. The test was conducted at Eglin AFB, Florida. • Benefit: The testing provided the Air Force with a cost- effective test venue for collecting critical data needed to assess performance of the LAIRCM system as installed on the new platform, the EC-130J. Air Force: LAIRCM KC-135 Operational Flight Test • Sponsors: 46th Test Wing Test Squadron Defensive Systems and Arizona National Guard, Air National Guard Air Force Reserve Test Center • Activity: The Center provided JMITS missile simulators and crews to perform two-color IR simulations to collect system response data for assessing the LAIRCM system as installed on the KC-135 pod. The tests were conducted at Eglin AFB, Florida. • Benefit: The testing provided the Air Force with a cost- effective test venue for collecting critical data needed to assess performance of the pod-based LAIRCM system as installed on the KC-135. Air Force: LAIRCM AC-130U Operational Flight Test • Sponsors: 46th Test Wing Test Squadron Defensive Systems and Mobility Directorate, Air Force Life Cycle Management Center • Activity: The Center provided JMITS missile simulators and crews to perform two-color IR simulations to collect system response data for assessing the LAIRCM system as installed on the AC-130U. The tests were conducted at Eglin AFB, Florida. • Benefit: The testing provided the Air Force with a cost-effective test venue for collecting critical data needed to assess performance of the LAIRCM system as installed on the new platform, the AC-130U.
Air Force: Advanced Strategic and Tactical IR Expendables Fall 2012 Test • Sponsors: Air National Guard Air Force Reserve Command Test Center and Air Mobility Command • Activity: The Center provided test assets and crew to collect test data on five different aircraft against post-launch configured IR missile seekers and three different aircraft against pre-launch configured IR missile seekers. These tests
Center for countermeasures evaluated new flare CM sequences, variations of current flare CM sequences using improved flares, or different flares within the sequences. • Benefit: Sponsors are using these effectiveness results from flare sequence testing to enhance the protection of various aircraft such as the C-17, C-130H, F-15C, F-16, and A-10 against IR MANPADS. Air Force: Advanced Strategic and Tactical IR Expendables Spring 2013 Test • Sponsors: Air Force Special Operations Command and Air Mobility Command • Activity: The Center provided test assets and crew to collect test data on four different aircraft against reactive captive IR missiles. These tests evaluated new flare CM sequences, variations of current flare CM sequences using improved flares, or different flares within the sequences. • Benefit: Sponsors are using these effectiveness results from flare sequence testing to enhance the protection of various aircraft against IR MANPADS. Air Force: F-35 Electro-Optical Distributed Aperture System (EO DAS) • Sponsor: F-35 Lightning II Joint Program Office • Activity: The Center provided the Towed Airborne Plume Simulator (TAPS) and JMITS missile simulators and crews to perform IR simulations, allowing the F-35 Team to collect data on the EO DAS. The Air Force conducted the tests at Naval Air Station Pensacola and Eglin AFB, Florida, using the Cooperative Avionics Test Bed aircraft fitted with the F-35 EO DAS.
• Benefit: The testing provided the Air Force with an opportunity to evaluate the potential of TAPS and JMITS to support future open-air testing of F-35 capabilities. Starbuck III Tests • Sponsor: Other Government Agency • Activity: The Center provided test assets and crew to provide immediate feedback on the effectiveness of flares and flare sequences against reactive captive IR missiles. These tests evaluated new CM sequences, variations of current CM sequences using improved flares, or different flares within the sequences. • Benefit: These test results were used to verify the effectiveness of flare sequences used on aircraft deployed in theater and under development. Rotary- and Fixed-Wing Test Events Army: Seeker Bowl VIII • Sponsors: U.S. Army Research Development and Engineering Command, Engineer Research and Development Center, and Aviation Applied Technology Directorate • Activity: The Center provided test assets and crew to collect test data on flare protection effectiveness for one fixed-wing and two rotary-wing aircraft against reactive captive IR missiles. The test evaluated the effectiveness of new flare CM sequences or variations of current flare CM sequences. • Benefit: Sponsors are using these flare sequence effectiveness test results to enhance the protection of various aircraft against IR MANPADS.
PGW CM ACTIVITIES National Ground Intelligence Center: Smoke Week 2012 • Sponsor: National Ground Intelligence Center • Activity: The Center coordinated, directed, and conducted this event. The Center also provided vehicle-launched smoke grenades and several contaminated battlefield obscurant environments. • Benefit: This event provided a venue for PGW system developers, including Hellfire and a variety of Navy combat optics, to evaluate their EO and IR systems in the presence of various obscurant environments. It also provided an opportunity to improve obscurant characterization methodology and collect characterization data on several new obscurant environments.
Air Force: RQ-4B Block 40 Global Hawk Operational Utility Evaluation • Sponsor: Air Force Operational Test and Evaluation Center • Activity: The Center provided camouflage, concealment, and deception elements consisting of inflatable surface-to-air missile decoys, inflatable armored vehicle decoys, and one radar scattering camouflage net deployed in scenarios in which the RQ-4B Block 40 Global Hawk attempted to detect, locate, and identify those elements. • Benefit: This test was a pre-deployment event held prior to the fielding of the RQ-4 Block 40 Multi-Platform Radar Technology Insertion Program in theater in summer 2013.
Center for Countermeasures
353
Center for countermeasures CM-BASED PRE-DEPLOYMENT TRAINING FOR SERVICE MEMBER EXERCISES Surface Attack Training – Nellis AFB, Nevada 160th Special Operations Aviation Regiment Radio Frequency Training – White Sands Missile Range, New Mexico Texas Air National Guard Pre-Deployment Training – San Antonio, Texas Joint Forcible Entry – Nellis AFB, Nevada Mission Employment Exercise – Nellis AFB, Nevada Destruction of Enemy Air Defense United States Air Force Warfare Center Training – Nellis AFB, Nevada 58th Special Operations Wing Training Support – Albuquerque, New Mexico Joint Readiness Training Center Training Support – Fort Polk, Louisiana Emerald Warrior – Hurlburt Field, Florida 10th Aviation Brigade, 6th Squadron, 6 Cavalry Training – Fort Drum, New York 509th Weapons Squadron KC-135 Support – Roswell, New Mexico • Sponsors: Various • Purpose: The Center’s equipment and personnel provided a simulated threat/CM environment and subject matter expertise to observe aircraft sensor/ASE systems and crew reactions to this environment. Specifically, the Center emphasized simulated MANPADS and Radio Frequency threat engagements for participating aircraft. Additionally, the Center provided MANPADS capabilities and limitations briefings to pilots and crews and conducted “hands-on” training at the end of the briefings. • Benefit: Provided realism to the training threat environment for the pilots and crews to facilitate understanding and use of CM equipment, especially ASE. The Center provided collected data to the trainers for assisting units in the development/refinement of techniques, tactics, and procedures to enhance survivability. Survivability Initiatives HSIG Model The Center led development of the HSIG model to support HFI T&E and modeling efforts. The HSIG Model project, sponsored by the Test and Evaluation Threat Resource Activity, has developed a physics-based EO model that produces signatures for the 12.7 mm Armor Piercing Incendiary Tracer round and a rocket-propelled grenade (RPG 7) tracer and hardbody. Model validation and integration to Navy and Army facilities were completed in FY13. Joint Countermeasures T&E Working Group (JCMT&E WG) The JCMT&E WG is co-chartered by DOT&E and DASD(DT&E) to improve the integration of: • Aircraft self-protection developments • Live weapon-fire T&E • Developmental and Operational T&E • Development of standardized test methodologies • Common instrumentation and standards This group includes DOT&E, DASD(DT&E), all four of the U.S. Services, Australia, Canada, Great Britain, New Zealand, and NATO Air Force Armaments Group Sub-Group 2 as members of a coalition warfare sub-WG. The group is tasked with actively- seeking, mutually-beneficial T&E opportunities to measure performance and suitability data necessary to provide relevant operational information to deploying joint/coalition Service members and for U.S. acquisition decision makers. Specific efforts included the following: • The JCMT&E WG, in the capacity of the Chairman of the eight-year bilateral ASE Cooperative Test and Evaluation Project Arrangement Steering Committee, worked with Great Britain to ensure smooth and highly effective testing. The
354
Center for Countermeasures
two nations have developed and successfully implemented three Working Groups in order to more effectively manage the growing level of efforts. The two nations’ defense organizations, ASE Program Offices, development testing, operational testing, and LFT&E agencies have been able to collaborate on common test equipment and procedures and measure operationally relevant ASE and environmental data that will continue to improve Service member survivability. • The JCMT&E WG worked with the Deputy Under Secretary of the Air Force, International Affairs, Armaments Cooperation Division to synchronize the U.S. Air Force Information Exchange Annexes with the United Kingdom to effectively strengthen the cooperation between the two nations. Due to the Center’s efforts, DOT&E Air Warfare was identified as one of two essential U.S. National Technical Establishments in the Information Exchange Annexes, ensuring that the Center remains in a leadership role. • The JCMT&E WG, in the capacity of the Chairman of the 10-year bilateral ASE Cooperative Test and Evaluation Project Arrangement Steering Committee, worked with Australia to ensure smooth and highly-effective testing on both sides of the Pacific. The two nations developed and successfully implemented three Working Groups to more effectively manage the growing level of efforts. As a result, the Center participated in the planning of the Australian hostile fire data collection Trial OXIDIZER II and other data collection opportunities that expanded the U.S. threat database and improved U.S. threat detection algorithms while reducing both nations’ test costs.
Center for countermeasures • The JCMT&E WG was the U.S. Technical Advisor to the official negotiations of the Multinational Test and Evaluation Program memorandum of understanding with Australia, Canada, Great Britain, New Zealand, and the United States. In support of high-level NATO multinational approaches initiatives and DOT&E initiatives to NATO, the Center developed, organized, and conducted a highly-successful, seven-nation NATO Quick Reaction Assessment (QRA) in Slovenia. The calibrated data and expert analysis in the Center’s Trial Report was hailed as the model for NATO to use for future QRAs. Due to the Center’s efforts, the NATO
National Armaments Directors Representative designated the Defensive Aids Suite effort a Smart Defence Tier 2 project. Helicopter Survivability Task Force The Center is participating with the Assistant Secretary of Defense for Research and Engineering in an effort to increase aircraft survivability by coordinating Research and Development activities and JCMT&E WG initiatives using tailored projects for DoD programs of record and out-of-cycle emergent Service member projects.
THREAT SIMULATOR TEST AND EVALUATION TOOLS The Center, in conjunction with the Test Resource Management Center (TRMC), completed the IRCM Test Resource Requirements Study (ITRRS) “refresh.” The end product from this effort is an updated roadmap of prioritized projects necessary to perform T&E of advanced IRCM and HFI systems. The TRMC completed the original ITRRS roadmap in 2007, which led to the Central Test and Evaluation Investment Program’s (CTEIP) funding of several projects to fill the identified IRCM T&E gaps. Each product has a functional description of the project; the priority is based on Program of Record test schedules, requirements, and Service input. The Center has continued to develop tools for T&E of IRCM systems funded by the USD(AT&L), TRMC, and CTEIP. Currently, the Center is leading the development of the Multi‑Spectral Sea and Land Test Simulator (MSALTS) and the Joint Standard Instrumentation Suite (JSIS). • The MSALTS is a small, mobile missile simulator that can fire while moving and simulate all current tier-one missile threats. The Center has designed the MSALTS to provide simulated signatures for the new and more capable missile warning systems, such as LAIRCM Next Generation, DoN LAIRCM, and Joint and Allied Threat Awareness System. The Center initiated development of the first two systems in FY11 and the third system in FY12. The developer completed fabrication, assembly, and integration of the first system in FY13, along with two demonstration events to show system maturity and
alleviate risk to the program. The developer plans to execute government acceptance testing of the first MSALTS system in October 2013. • The JSIS is a transportable, fully integrated instrumentation suite that will be utilized for collecting signature, Time- Space- Position Information, acoustic, and related metadata of threat missile and hostile fire munitions. JSIS data collected during these live fire events will be used to support ASE systems development, modeling and simulation activities, T&E ground truth data, and anomaly investigation. All data collected from JSIS will be calibrated, measured, and stored according to the standards defined by the Joint Tactical Missile Signatures Handbook and will be available to the ASE community. The JSIS has been endorsed by the U.S. Navy (Program Manager Air – 272), Army (Program Management Office – ASE), and the Air Force (LAIRCM System Program Office) and will be an integral part in each Program Office’s ASE development. In July 2013, the JSIS was selected as a “Resource Enhancement Project New Start” project and will receive FY14 funding from the TRMC and CTEIP. In FY13, the Center, partnered with the Arnold Engineering Development Center, actively created program plans, refined requirements from the ASE T&E community, created and refined a concept of operation, and began identifying specific instrumentation that meets JSIS requirements.
LIVE FIRE TEST AND EVALUATION TOOLS The Center has continued to develop tools for the T&E community for live fire IRCM testing. Included in these developments are two new dual MANPADS missile launchers developed by Missile and Space Intelligence Center for the Center. These systems have been delivered and their operation verified during live fire acceptance tests. These launcher systems feature:
• Compatibility with a large variety of MANPADS missile types • Single, dual, and salvo launch capability (up to four missiles of the same or different types) • Precision launch synchronization and timing capable of simultaneous or programmable launch delays • High-mobility, self-contained operation
Center for Countermeasures
355
Center for countermeasures
356
Index
Index
Index of Programs A
AC-130J Ghostrider......................................................................... 255 Acoustic Rapid Commercial Off‑the‑Shelf (COTS) Insertion (A-RCI).............................. 15, 137 Advanced Extremely High Frequency (AEHF) Satellite Communications System........................ 257 Aegis Ballistic Missile Defense (Aegis BMD).................................................... 303 Aegis Modernization........................................................................ 163 AGM-88E Advanced Anti-Radiation Guided Missile (AARGM) Program........................... 30, 141 AIM-9X Air-to-Air Missile Upgrade......................................................... 15, 143 AIM-120 Advanced Medium-Range Air-to-Air Missile (AMRAAM)............................ 15, 19, 259 Air and Missile Defense Radar (AMDR)........................................................ 163 Air Operations Center – Weapon System (AOC-WS)............................................ 22, 261 AN/BLQ-10 Submarine Electronic Warfare Support System ........................................ 145 AN/BYG‑1 Combat Control System........................................................... 137 AN/SQQ-89A(V)15 Integrated Undersea Warfare (USW) Combat System Suite......................... 147 Armored Tactical Vehicles..................................................................... 83 Automated Biometric Information System (ABIS)............................................ 16, 25, 95 B
Ballistic Missile Defense System (BMDS)....................................................... 299 Battle Control System – Fixed (BCS-F)................................................... 22, 30, 265 Bradley Engineering Change Proposal (ECP)................................................... 30, 87 C
CH-47F (Chinook) – Improved Cargo Helicopter................................................... 89 Cobra Judy Replacement (CJR)................................................................ 149 Command and Control, Battle Management, and Communications (C2BMC) System..................... 307 Compact Low Frequency Active (CLFA)..................................................... 17, 249 Consolidated Afloat Networks and Enterprise Services (CANES)..................................... 151 Cooperative Engagement Capability (CEC)................................................... 19, 153 Countermeasure Anti- torpedo Torpedo (CAT)................................................. 29, 245 CV-22 Osprey............................................................................. 267 CVN-78 Gerald R. Ford Class Nuclear Aircraft Carrier.......................................... 24, 155 D
DDG 51 Flight III Destroyer.................................................................. 161 DDG 1000 – Zumwalt Class Destroyer.......................................................... 165 Defense Enterprise Accounting and Management System (DEAMS)............................ 16, 25, 269
357
Index of Programs Defense Readiness Reporting System (DRRS)..................................................... 31 Distributed Common Ground System – Army (DCGS-A)......................................... 30, 93 Distributed Common Ground System – Navy (DCGS-N)........................................... 167 DoD Automated Biometric Information System (ABIS)........................................ 16, 25, 95 E
E-2D Advanced Hawkeye.............................................................. 19, 30, 169 EA-18G Growler........................................................................ 22, 173 Enhanced Combat Helmet (ECH).............................................................. 171 F
F-15E Radar Modernization Program (RMP)............................................... 19, 30, 271 F-22A Advanced Tactical Fighter.............................................................. 273 F-35 Joint Strike Fighter (JSF)................................................................. 33 F/A-18E/F Super Hornet.................................................................. 22, 173 Fire Scout.............................................................................. 23, 253 G
Gerald R. Ford Class Nuclear Aircraft Carrier................................................. 24, 155 Global Broadcast System (GBS)............................................................ 20, 275 Global Combat Support System – Army (GCSS-Army).............................................. 99 Global Command and Control System – Joint (GCCS-J).......................................... 20, 53 Global Hawk High-Altitude Long-Endurance Unmanned Aerial System (UAS)....................... 28, 293 Ground/Air Task Oriented Radar (G/ATOR)................................................... 23, 177 Ground-Based Midcourse Defense (GMD)....................................................... 311 H
H-1 Upgrades – U.S. Marine Corps Upgrade to AH-1Z Attack Helicopter and UH-1Y Utility Helicopter..... 20, 181 Handheld, Manpack, and Small Form Fit (HMS) Manpack Radio............................ 21, 25, 30, 101 Handheld, Manpack, and Small Form Fit (HMS) Rifleman Radio............................... 26, 30, 103 HC/MC-130J.............................................................................. 277 I
Identification Friend or Foe (IFF) Mode 5........................................................ 203 Integrated Defensive Electronic Countermeasures (IDECM)...................................... 26, 183 Integrated Electronic Health Record (iEHR).................................................... 26, 57 J
Joint Air-to-Surface Standoff Missile – Extended Range (JASSM-ER) ................................ 279 Joint Battle Command – Platform (JBC-P).................................................... 16, 105
358
Index of Programs Joint Biological Tactical Detection System (JBTDS)................................................ 61 Joint High Speed Vessel (JHSV)............................................................... 185 Joint Information Environment (JIE)............................................................. 63 Joint Land Attack Cruise Missile Defense Elevated Netted Sensor System (JLENS)...................... 107 Joint Light Tactical Vehicle (JLTV) Family of Vehicles (FoV)........................................ 111 Joint Mission Planning System – Air Force (JMPS-AF)....................................... 21, 30, 289 Joint Space Operations Center (JSpOC) Mission System (JMS) ...................................... 281 Joint Strike Fighter (JSF)...................................................................... 33 Joint Warning and Reporting Network (JWARN)................................................ 26, 65 K
KC-46A.................................................................................. 283 Key Management Infrastructure (KMI)........................................................ 30, 67 L
LHA-6................................................................................ 27, 189 Light Armored Vehicle (LAV) Upgrade.......................................................... 193 Littoral Combat Ship (LCS)............................................................. 27, 30, 195 LPD-17 San Antonio Class Amphibious Transport Dock ............................................ 201 M
M109 Family of Vehicles (FoV) Paladin Integrated Management (PIM)............................. 28, 113 Manpack Radio................................................................... 21, 25, 30, 101 Mark XIIA Identification Friend or Foe (IFF) Mode 5.............................................. 203 Massive Ordnance Penetrator (MOP)........................................................... 285 MH-60R Multi-Mission Helicopter............................................................. 205 MH-60S Multi-Mission Combat Support Helicopter............................................... 207 Mine Resistant Ambush Protected (MRAP) Family of Vehicles (FoV).................................. 69 Miniature Air Launched Decoy (MALD) and MALD-Jammer (MALD-J)........................ 16, 30, 287 Mission Planning System (MPS)......................................................... 21, 30, 289 Mk 48 Advanced Capability (ADCAP) Torpedo Modifications....................................... 209 Mk 54 Lightweight Torpedo............................................................... 16, 211 Mk248 Mod 0 Sniper Round.................................................................. 117 MQ-4C Triton Unmanned Aircraft System....................................................... 213 MQ-9 Reaper Armed Unmanned Aircraft System (UAS)............................................ 291 Multi-Static Active Coherent (MAC) System............................................... 17, 30, 215
359
Index of Programs N
Navy Enterprise Resource Planning (ERP)....................................................... 217 Nett Warrior............................................................................ 26, 119 Network Integration Evaluation (NIE)........................................................... 81 Next Generation Diagnostics System (NGDS).................................................. 28, 73 O
Ohio Class Replacement Program.............................................................. 237 Osprey................................................................................... 267 P
P-8A Poseidon Multi-Mission Maritime Aircraft............................................... 21, 221 Paladin Integrated Management (PIM)....................................................... 28, 113 Patriot Advanced Capability-3 (PAC-3)...................................................... 30, 121 Precision Guidance Kit (PGK)................................................................. 125 Public Key Infrastructure (PKI)........................................................... 17, 28, 75 Q
Q-53 Counterfire Target Acquisition Radar System............................................. 28, 127 R
Reaper Armed Unmanned Aircraft System (UAS)................................................. 291 Remote Minehunting System (RMS)........................................................... 227 Rifleman Radio...................................................................... 26, 30, 103 RQ-4B Global Hawk High-Altitude Long-Endurance Unmanned Aerial System (UAS)................ 28, 293 S
San Antonio Class Amphibious Transport Dock ................................................... 201 Sensors................................................................................... 313 Ship Self-Defense.......................................................................... 231 Small Tactical Unmanned Aerial System (STUAS) Tier II........................................ 23, 235 Space-Based Infrared System (SBIRS).......................................................... 297 Spider XM7 Network Command Munition....................................................... 131 SSBN Ohio Class Replacement Program........................................................ 237 SSN 774 Virginia Class Submarine.......................................................... 30, 239 Standard Missile-6 (SM-6)................................................................ 30, 243 Stryker Mobile Gun System (MGS)............................................................ 133 Surface Ship Torpedo Defense (SSTD) System................................................ 29, 245 Surveillance Towed Array Sensor System (SURTASS).......................................... 17, 249
360
Index of Programs T
Terminal High-Altitude Area Defense (THAAD).................................................. 317 Theater Medical Information Program – Joint (TMIP-J)............................................. 79 Tomahawk Missile and Weapon System......................................................... 251 Torpedo Warning System (TWS)............................................................ 29, 245 Triton Unmanned Aircraft System.............................................................. 213 V
Vertical Take-Off and Landing Unmanned Aerial Vehicle (VTUAV) (Fire Scout)...................... 23, 253 Virginia Class Submarine.................................................................. 30, 239 W
Warfighter Information Network – Tactical (WIN-T)......................................... 17, 30, 135 Z
Zumwalt Class Destroyer..................................................................... 165
361
Index of Programs
362
DOT&E Activity and Oversight DoD Programs
Army Programs
Navy Programs
Air Force Programs
Ballistic Missile Defense Systems
Live Fire Test and Evaluation
Information Assurance and Interoperability
Test and Evaluation Resources
REC DI
NAL TES
T ALUATION EV
Index
TIO RA
&
Center for Countermeasures
TOR O PE
Joint Test and Evaluation
www.dote.osd.mil

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.

Suggested Citation:'10 Conclusions and Recommendations.' National Research Council. 2015. Reliability Growth: Enhancing Defense System Reliability. Washington, DC: The National Academies Press. doi: 10.17226/18987.