DODI 5000.02 Enclosure 4: Developmental Test and Evaluation

1. PURPOSE

This enclosure provides policy and procedure for developmental test and evaluation of defense acquisition programs.

2. OVERVIEW

a. Program managers use DT&E 1. Any testing used to assist in the development and maturation of products, product elements, or manufacturing or support processes. 2. Any engineering-type test used to verify status of technical progress, verify that design risks are minimized, substantiate achievement of contract technical performance, and certify readiness for initial operational testing. Developmental tests generally require instrumentation and measurements and are accomplished by engineers, technicians, or soldier operator-maintainer test personnel in a controlled environment to facilitate failure analysis. activities to manage and mitigate risks during development, to verify that products are compliant with contractual and operational requirements, and to inform decision makers throughout the program life cycle. DT&E provides program engineers and decision makers with knowledge to measure progress, identify problems, and to characterize system capabilities and limitations, and manage technical and programmatic risks. DT&E results are also used as exit criteria to ensure adequate progress prior to investment commitments or initiation of phases of the program, and as the basis for contract incentives.

b. DT&E starts with capability requirements and continues through product development, delivery, and acceptance; transition to operational test and evaluation (T&E); Production The process of converting raw materials by fabrication into required material. It includes the functions of production-scheduling, inspection, Quality Control, and related processes. ; and operations and support. Consideration of developmental test and evaluation in the requirements and systems engineering processes ensures that capability requirements are measurable, testable, and achievable. Identifying and correcting deficiencies early is less costly than discovering system deficiencies late in the acquisition process.

c. The Program Manager will use a Test and Evaluation Master Plan (TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. ) as the primary planning and management tool for the integrated test program. Whenever feasible, testing will be conducted in an integrated fashion to permit all stakeholders to use data in support of their respective functions. Integrated testing requires the collaborative planning and collaborative execution of test phases and events to provide shared data in support of independent analysis, evaluation, and reporting by all stakeholders, particularly the systems engineering, developmental (both contractor and government) and operational T&E communities. The Program Manager will establish an integrated test planning group consisting of empowered representatives of test data producers and consumers (to include all applicable stakeholders) to ensure collaboration and to develop a strategy for robust, efficient testing to support systems engineering, evaluations, and certifications throughout the acquisition life cycle.

d. The Program Manager will identify the test resources needed to execute the DT&E program to acquire the data that will be used to understand program progress, identify issues, verify compliance, and balance cost and performance. Test resource requirements will be included in the TEMP.

e. The Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) will monitor the development test and evaluation program activities of Major Defense Acquisition Programs (MDAP An acquisition program that is designated by the USD(AT&L) as an MDAP; or is estimated to require an eventual total expenditure for RDT&E including all planned increments, of more than $480 million in FY 2014 constant dollars or, for procurement, including all planned increments, of more than $2.79 billion in FY 2014 constant dollars.s), and review the DT&E plans for those programs in the TEMP. The DASD(DT&E) will provide a recommendation to approve or disapprove the MDAP DT&E plans as well as advise the relevant technical authorities for these programs on the incorporation of best practices for developmental test from across the Department. For ACAT IA, and ACAT II and below programs, the Component Acquisition Executive Secretaries of the military departments or heads of agencies with the power of redelegation. In the military departments, the officials delegated as CAEs (also called SAEs are respectively, the ASA(AL&T); the ASN(RD&A); and the ASAF(A)). The CAEs are responsible for all acquisition functions within their components. This includes both the SAEs for the military departments and acquisition executives in other DoD components, such as SOCOM and DLA, which also have acquisition management responsibilities. will designate a DT&E organization to monitor DT&E activities and recommend approval or disapproval of the DT&E plans in the TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. . For all programs, the MDA (or designee) will approve or disapprove the DT&E plans in the TEMP. DASD(DT&E) authorities, responsibilities, and functions are described in 10 U.S.C. 139b (Reference (h)).

3. T&E MANAGEMENT

a. Program managers for MDAP An acquisition program that is designated by the USD(AT&L) as an MDAP; or is estimated to require an eventual total expenditure for RDT&E including all planned increments, of more than $480 million in FY 2014 constant dollars or, for procurement, including all planned increments, of more than $2.79 billion in FY 2014 constant dollars.s and MAIS programs will designate a Chief Developmental Tester in accordance with 10 U.S.C. 139b and 1706 (Reference (h)). The Chief Developmental Tester will be responsible for coordinating the planning, management, and oversight of all DT&E 1. Any testing used to assist in the development and maturation of products, product elements, or manufacturing or support processes. 2. Any engineering-type test used to verify status of technical progress, verify that design risks are minimized, substantiate achievement of contract technical performance, and certify readiness for initial operational testing. Developmental tests generally require instrumentation and measurements and are accomplished by engineers, technicians, or soldier operator-maintainer test personnel in a controlled environment to facilitate failure analysis. activities; maintaining insight into contractor activities; overseeing the T&E activities of other participating government activities; and helping the Program Manager make technically informed, objective judgments about contractor and government T&E planning and results. The Chief Developmental Tester will chair the integrated test planning group.

b. Program managers for MDAPs will designate a government test agency to serve as the lead DT&E organization in accordance with 10 U.S.C. 139b. The lead DT&E organization will be responsible for providing technical expertise on T&E issues to the Chief Developmental Tester; conducting DT&E activities as directed by the Chief Developmental Tester or his or her designee; supporting certification and accreditation activities when feasible; assisting the Chief Developmental Tester in providing oversight of contractors; and assisting the Chief Developmental Tester in reaching technically informed, objective judgments about contractor and government T&E planning and results. For all other programs, a lead DT&E organization should be used, when feasible, and identified in the TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. .

c. The designation of a Chief Developmental Tester and lead DT&E organization will be made as soon as practicable after the program office is established.

d. The Program Manager will use the TEMP as the primary planning and management tool for all test activities starting at Milestone A. The Program Manager will prepare and update the TEMP as needed and to support acquisition milestones or decision points. For the Full-Rate Production Decision Review or the Full Deployment Decision and thereafter, the MDA may require TEMP updates or addendums to plan for additional testing. Section 5 in Enclosure 5 of this instruction has additional policy for the TEMP in the context of operational testing.

e. Program managers for programs under DASD(DT&E) oversight will designate a T&E Working-level Integrated Product Team (WIPT) (also known as an Integrated Test Team), as soon as practicable after the MDD A review that is the formal entry point into the acquisition process and is mandatory for all programs. A successful MDD may approve entry into the acquisition management system at any point consistent with phase-specific and statutory requirements but will normally be followed by a MSA phase. The principal documents at this decision point are the ICD and Analysis of Alternatives Study Guidance and Plan. A successful MDD normally does not mean that a new acquisition program has been initiated. may approve entry into the acquisition management system at any point consistent with phase-specific and statutory requirements but will normally be followed by a MSA phase. The principal documents at this decision point are the ICD and Analysis of Alternatives Study Guidance and Plan. A successful MDD normally does not mean that a new acquisition program has been initiated. “] Materiel Development Solution [/tooltip]. The T&E WIPT develops and tracks the T&E program in all phases. The T&E WIPT will include empowered representatives of test data stakeholders such as Systems Engineering, DT&E, Operational T&E, Live Fire T&E, Product Support, the user, the intelligence community, and applicable certification authorities.

f. The Program Manager will take full advantage of DoD ranges, labs, and other resources. Systems have become more complex and resource constraints often force tradeoffs in the type and scope of testing that can be performed. The DT&E budget and schedule 1. Series of things to be done in a specific sequence within a given period of time. 2. A timetable. 3. A listing of activities and events organized by time. must allow testing that adequately verifies performance to contractual requirements in a controlled environment and to operational requirements.

4. DT&E ACTIVITIES

a. a. DT&E 1. Any testing used to assist in the development and maturation of products, product elements, or manufacturing or support processes. 2. Any engineering-type test used to verify status of technical progress, verify that design risks are minimized, substantiate achievement of contract technical performance, and certify readiness for initial operational testing. Developmental tests generally require instrumentation and measurements and are accomplished by engineers, technicians, or soldier operator-maintainer test personnel in a controlled environment to facilitate failure analysis. activities will start when requirements are being developed to ensure that key technical requirements are measurable, testable, and achievable.

b. A robust DT&E program includes a number of key activities to provide the data and assessments for decision making. The DT&E program will:

  • (1) Verify achievement of critical technical parameters and the ability to achieve KPP Performance attribute of a system considered critical or essential to the development of an effective military capability. KPPs are contained in the Capability Development Document and the Capability Production Document and are included verbatim in the Acquisition Program Baseline. KPPs are expressed in term of parameters which reflect Measures of Performance using a threshold/objective format. KPPs must be measurable, testable, and support efficient and effective Test and Evaluation. Mandatory KPPs are specified in the JCIDS Manual.s, and assess progress toward achievement of critical operational issues.
  • (2) Assess the system’s ability to achieve the thresholds prescribed in the capabilities documents.
  • (3) Provide data to the Program Manager to enable root cause determination and to identify corrective actions.
  • (4) Validate system functionality.
  • (5) Provide information for cost, performance, and schedule tradeoffs.
  • (6) Assess system specification compliance.
  • (7) Report on program progress to plan for reliability May be expressed initially as a desired failure-free interval that can be converted to a failure frequency for use as a requirement. growth and to assess reliability and maintainability performance for use during key reviews.
  • (8) Identify system capabilities, limitations, and deficiencies.
  • (9) Include T&E activities to detect cyber vulnerabilities within custom and commodity hardware and software.
  • (10) Assess system safety.
  • (11) Assess compatibility with legacy systems.
  • (12) Stress the system within the intended operationally relevant mission environment.
  • (13) Support cybersecurity assessments and authorization, including Risk Management Framework security controls.
  • (14) Support the interoperability certification process.
  • (15) Document achievement of contractual technical performance, and verify incremental improvements and system corrective actions.
  • (16) Assess entry criteria for Initial Operational Test and Evaluation ( IOT&E Dedicated Operational Test and Evaluation conducted on production, or production representative articles, to determine whether systems are operationally effective and suitable to support a Full-Rate Production decision. The term IOT&E is normally associated with programs on the Director, Operational Test and Evaluation Oversight List. ) and Follow-On Operational Test and Evaluation.
  • (17) Provide DT&E data to validate parameters in models and simulations.
  • (18) Assess the maturity of the chosen integrated technologies.
5. DT&E PLANNING CONSIDERATIONS

a. The Program Manager will:

  • (1) Use the TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. as the primary test planning and management document.
  • (2) The TEMP will:

(a) Contain an integrated test program summary and master schedule 1. Series of things to be done in a specific sequence within a given period of time. 2. A timetable. 3. A listing of activities and events organized by time. of all major test events or test phases. (b) Include an event-driven testing schedule that will allow adequate time to support pre-test predictions; testing; post-test analysis, evaluation, and reporting; reconciliation of predictive models; and adequate time to support execution of corrective actions in response to discovered deficiencies. The schedule should allow sufficient time between DT&E 1. Any testing used to assist in the development and maturation of products, product elements, or manufacturing or support processes. 2. Any engineering-type test used to verify status of technical progress, verify that design risks are minimized, substantiate achievement of contract technical performance, and certify readiness for initial operational testing. Developmental tests generally require instrumentation and measurements and are accomplished by engineers, technicians, or soldier operator-maintainer test personnel in a controlled environment to facilitate failure analysis. and IOT&E for rework, reports, and analysis and developmental testing of critical design changes. (c) Be a source document when developing the RFP A document used in negotiated acquisitions to communicate Government requirements to prospective contractors and to solicit proposals. RFPs for competitive acquisitions describe the Government’s requirement; anticipated terms and conditions that will apply to the contract; information required to be in the offeror’s proposal; and factors and significant sub-factors that will be used to evaluate the proposal and their relative importance. s. (d) Guide how contractor proposals will address program test needs such as: test articles; T&E data rights; government access to the Failure Reporting, Analysis and Corrective Action System and other test outcome repositories; built-in test and embedded instrumentation data (including software log files); contractor verification requirements; government use of contractor-conducted T&E; government review and approval of contractor T&E plans; government witness of contractor test events; and government review of contractor evaluations. See section 5 in Enclosure 5 of this instruction for additional details. (e) Include identification of all contractor and government system level reliability testing needed to support initial reliability planning estimates. The Program Manager will include the reliability developmental evaluation methodology for reliability critical items. The military departments/program managers will collect and retain data from the T&E of the reliability and maintainability of major weapon systems Items that can be used directly by the Armed Forces to carry out combat missions. to inform system design decisions, provide insight into sustainment costs, and inform estimates of operating and support costs for such systems. (f) Starting at Milestone B, include one or more reliability growth curves (RGCs). 1. If a single curve is not adequate to describe the overall system reliability, curves for critical subsystems with rationale for their selection will be provided. 2. For software (in any system), the TEMP will include projected and observed software maturity metrics. For hardware acquisitions, Milestone B RGCs will consist of observed (when available) and projected reliability. 3. RGCs will be stated in a series of intermediate goals tracked through fully integrated, system-level T&E events until the reliability threshold is achieved.

  • (3) Use scientific test and analysis techniques to design an effective and efficient test program that will produce the required data to characterize system behavior across an appropriately selected set of factors and conditions.
  • (4) Identify each developmental test phase or major developmental test event as a contractor or government DT&E. All programs will plan for the conduct of DT&E and/or integrated testing to provide confidence in the system design solution. Each major developmental test phase or event (including Test Readiness Reviews) will have test entrance and exit criteria. The developmental test completion criteria (customer needs) will dictate what data are required from the test event. (5) Ensure that all test infrastructure and/or tools (e.g., models, simulations, automated tools, synthetic environments) to support acquisition decisions will be verified, validated, and accredited (VV&A) by the intended user or appropriate agency. Test infrastructure, tools, and/or the VV&A strategy including the VV&A authority for each tool or test infrastructure asset will be documented in the TEMP. Program Managers will plan for the application and accreditation of any modeling and simulation tools supporting DT&E.
  • (6) Develop complete resource estimates for T&E to include: test articles, test sites and instrumentation, test support equipment, threat representations and simulations, test targets and expendables, support for operational forces used in test (both friendly and threat), models and simulations, testbeds, joint mission environment, distributed test networks, funding, manpower and personnel, training, federal/state/local requirements, range requirements, and any special requirements (e.g., explosive ordnance disposal requirements or corrosion prevention and control). Resources will reflect the best estimate for conducting all test activities. Resources will be mapped against the developmental evaluation framework and schedule to ensure adequacy and availability.
  • (7) Ensure that resource estimates identified in the TEMP are matched against the schedule and justified by analysis.
  • (8) Resource and ensure threat-appropriate Red Team/Penetration testing to emulate the threat of hostile penetration of program information systems in the operational environment. Additional guidance on Red Team operations is included in Chairman of the Joint Chiefs of Staff Instruction 6510.01F (Reference (bf)).
  • (9) Develop a strategy and budget resources for cybersecurity testing. The test program will include, as much as possible, activities to test and evaluate a system in a mission environment with a representative cyber-threat capability.
  • (10) Ensure that each major developmental test phase or event in the planned test program has a well-defined description of the event, specific objectives, scope, appropriate use of modeling and simulation, and a developmental evaluation methodology.
  • (11) Describe a developmental evaluation methodology in the TEMP starting at Milestone A that will provide essential information on programmatic and technical risks as well as information for major programmatic decisions. Starting at Milestone B, the developmental evaluation methodology will include a developmental evaluation framework to identify key data that will contribute to assessing progress toward achieving: KPPs, critical technical parameters, key system attributes, interoperability requirements, cybesecurity requirements, reliability growth, maintainability attributes, developmental test objectives, and others as needed. In addition, the developmental evaluation framework will show the correlation and mapping between test events, key resources, and the decision supported. The developmental evaluation methodology will support a Milestone B assessment of planning, schedule, and resources and a Milestone C assessment of performance, reliability, interoperability, and cybersecurity.
  • (12) Develop a software test automation strategy to include when key test automation software components or services will be acquired and how those decisions will be made. b. Programs will use government T&E Process by which a system or components are exercised and results analyzed to provide performance-related information. The information has many uses including risk identification and risk mitigation and empirical data to validate models and simulations. T&E enables an assessment of the attainment of technical performance, specifications, and system maturity to determine whether systems are operationally effective, suitable and survivable for intended use, and/or lethal. There are various types of T&E defined in statute or regulation: DT&E, OT&E, LFT&E, and Interoperability Certification. capabilities unless an exception can be justified as cost-effective to the government. Program managers will conduct a cost-benefit analysis for exceptions to this policy and obtain approval through the TEMP approval process before acquiring or using non-government, program unique test facilities or resources.

c. In accordance with DoD Instruction 8510.01 (Reference (bg)), all programs must have security controls implemented consistent with their information and system categorization. Program managers will ensure appropriate testing to evaluate capability to protect information and information systems from unauthorized access, use, disclosure, disruption, modification, or destruction. The Defense Intelligence Agency (DIA), in coordination with the Program Manager, will determine the generation of the relevant operational threat environment based on the Validated On-line Life-cycle Threat Report, the Multi-Service Force Deployment, the Joint Country Forces Assessment and scenario support products in accordance with DIA Directive 5000.200 (Reference (t)) and DIA Instruction 5000.002 (Reference (u)). d. Systems that operate as part of a system of systems may require deployment of additional test assets to evaluate end-to-end capabilities. Program managers will ensure that adequate testing of total system of systems performance is conducted as part of the DT&E program. e. For accelerated acquisition and urgent need programs, the levels of developmental testing required will be highly tailored to emphasize schedule over other considerations. Required testing to verify safety, capabilities, and limitations will be performed consistent with the urgency of fielding the capability. Responsibility for determining developmental testing requirements will be delegated to the lowest practical level. Urgent need programs will generally not be on an OSD DT&E Engagement list. If an Accelerated Acquisition program is on the DT&E Engagement list, complete developmental testing may be deferred so as not to impede early fielding; however, an operational assessment will typically be conducted. See paragraph 6a in Enclosure 5 of this instruction for a discussion of operational assessments, and Enclosure 13 for the policy and procedure regarding acquisition programs that respond to urgent needs.

6. DT&E EXECUTION, EVALUATION, AND REPORTING

a. DT&E 1. Any testing used to assist in the development and maturation of products, product elements, or manufacturing or support processes. 2. Any engineering-type test used to verify status of technical progress, verify that design risks are minimized, substantiate achievement of contract technical performance, and certify readiness for initial operational testing. Developmental tests generally require instrumentation and measurements and are accomplished by engineers, technicians, or soldier operator-maintainer test personnel in a controlled environment to facilitate failure analysis. Execution. As the Program Manager executes the program’s strategy for the DT&E, the Program Manager and test team will develop detailed test plans for each developmental test event identified in the TEMP. Test plans must consider the potential impacts on personnel and the environment in accordance with 10 U.S.C. 4321-4347 (Reference (ag)) and Executive Order 12114 (Reference (ah)). The Program Manager, in concert with the user and T&E community, will provide safety releases (to include National Environmental Policy Act documentation, safety, and occupational health risk acceptance in accordance with section 16 in Enclosure 3 of this instruction) to testers prior to any test that may impact safety of personnel. A Test Readiness Review will be conducted for those events identified in the TEMP. b. DASD(DT&E) Program Assessments. For MDAPs, MAIS programs, and USD(AT&L)-designated special interest programs, the DASD(DT&E) will provide the MDA with a program assessment at the Development RFP Release Decision Point, Milestones B and C, and updated to support the Operational Test Readiness Review or as requested by the MDA or Program Manager. The program assessment will be based on the completed DT&E and any Operational T&E activities completed to date, and will address the adequacy of the program planning, the implications of testing results to date, and the risks to successfully meeting the goals of the remaining T&E events in the program. c. DT&E Reports and Data

  • (1) The DASD(DT&E) and the acquisition chain of command (including the Program Manager) and their designated representatives will have full and prompt access to all ongoing developmental testing, and all developmental test records and reports, including but not limited to: data from all tests, system logs, execution logs, test director notes, certifications, and user/operator assessments and surveys. This applies to all government accessible data including classified, unclassified, and competition sensitive or proprietary data. Data may be preliminary and will be identified as such.
  • (2) The Program Manager and test agencies for all programs will provide the Defense Technical Information Center (DTIC) with all reports and the supporting data for the test events in those reports. Paragraphs 11c(5) through 11c(7) in Enclosure 5 of this instruction include a more detailed discussion.
  • (3) The DoD Components will collect and retain data from developmental test and evaluation, integrated testing, and operational test and evaluation on the reliability and maintainability of Acquisition Category I and II programs.
  • (4) Tables 2 and 6 in Enclosure 1 identify statutory and regulatory reporting and notification requirements associated with the conduct of DT&E.