DODI 5000.02 Enclosure 5: Operational and Live Fire Test and Evaluation

1. OVERVIEW

a. The fundamental purpose of test and evaluation (T&E Process by which a system or components are exercised and results analyzed to provide performance-related information. The information has many uses including risk identification and risk mitigation and empirical data to validate models and simulations. T&E enables an assessment of the attainment of technical performance, specifications, and system maturity to determine whether systems are operationally effective, suitable and survivable for intended use, and/or lethal. There are various types of T&E defined in statute or regulation: DT&E, OT&E, LFT&E, and Interoperability Certification. ) is to enable the DoD to acquire systems that work. To that end, T&E Process by which a system or components are exercised and results analyzed to provide performance-related information. The information has many uses including risk identification and risk mitigation and empirical data to validate models and simulations. T&E enables an assessment of the attainment of technical performance, specifications, and system maturity to determine whether systems are operationally effective, suitable and survivable for intended use, and/or lethal. There are various types of T&E defined in statute or regulation: DT&E, OT&E, LFT&E, and Interoperability Certification. provides engineers and decision-makers with knowledge to assist in managing risks, to measure technical progress, and to characterize operational effectiveness, suitability, and survivability. This is done by planning and executing a robust and rigorous T&E program.

b. The Program Manager is responsible for resourcing and executing the system’s approved T&E program. The Program Manager assembles a test team of empowered representatives of the various test data consumers. The team starts early (i.e., pre-Milestone A) to develop a robust, rigorous, and efficient test program that will be conducted in support of systems engineering An interdisciplinary approach and process encompassing the entire technical effort to evolve, verify and sustain an integrated and total life cycle balanced set of system, people, and process solutions that satisfy customer needs. SE is the integrating mechanism for the technical and technical management efforts related to the concept analysis, materiel solution analysis, engineering and manufacturing development, production and deployment, operations and support, disposal of, and user training for systems and their life cycle processes. , evaluations, and certifications throughout the program life cycle. The Program Manager documents the test program planning in the Test and Evaluation Master Plan (TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. ). All TEMPs will require DoD Components The Office of the Secretary of Defense; the military departments; the Chairman, Joint Chiefs of Staff and the Joint Staff; the combatant commands; the Office of the Inspector General of the DoD; the defense agencies; DoD field activities; and all other organization entities within the DoD. approval; TEMPs for programs under DT&E, oversight will also require DT&E 1. Any testing used to assist in the development and maturation of products, product elements, or manufacturing or support processes. 2. Any engineering-type test used to verify status of technical progress, verify that design risks are minimized, substantiate achievement of contract technical performance, and certify readiness for initial operational testing. Developmental tests generally require instrumentation and measurements and are accomplished by engineers, technicians, or soldier operator-maintainer test personnel in a controlled environment to facilitate failure analysis.approval. The operational and select live fire test events in the TEMP must have approved test plans. Test plans are written and approved by the test organization responsible for the test. Operational test plans (OTPs) for programs under DT&E OT&E oversight and live fire test plans (LFTPs) for programs under DT&E LFT&E oversight will require DT&E approval.

c. For programs under DT&E OT&E or LFT&E oversight, the DT&E will provide the MDA with milestone assessments. DT&E will submit a report to the Secretary of Defense and the congressional defense committees before programs under DT&E OT&E or LFT&E oversight may proceed beyond Low-Rate Initial Production (LRIP), in accordance with 10 U.S.C. 2366 and 2399 (Reference (h)).

2. APPLICABILITY

This enclosure applies to all defense acquisition programs under OSD OT&E The field test, under realistic conditions, of any item (or key component) of weapons, equipment, or munitions for the purpose of determining the effectiveness and suitability of the weapons, equipment, or munitions for use in combat by typical military users; and the evaluation of the results of such tests. or LFT&E oversight. This enclosure is written to the Hardware Intensive Program model described in paragraph 5c(3)(b) of this instruction, with tailoring instructions for the software within those programs and the software-specific acquisition models. When there is no distinction between Defense Unique Software Intensive Programs (Model 2) and Incrementally Deployed Software Intensive Programs (Model 3), they are referenced herein as “Software Acquisitions.” Tailoring for any software, irrespective of acquisition model, is identified as being “for software in any system.” Tailoring for Accelerated Acquisition models will be considered on a case-by-case basis.

3. DOT&E OVERSIGHT LIST

a. DOT&E may place any program or system on the DOT&E Oversight List for OT&E The field test, under realistic conditions, of any item (or key component) of weapons, equipment, or munitions for the purpose of determining the effectiveness and suitability of the weapons, equipment, or munitions for use in combat by typical military users; and the evaluation of the results of such tests. or T&E Process by which a system or components are exercised and results analyzed to provide performance-related information. The information has many uses including risk identification and risk mitigation and empirical data to validate models and simulations. T&E enables an assessment of the attainment of technical performance, specifications, and system maturity to determine whether systems are operationally effective, suitable and survivable for intended use, and/or lethal. There are various types of T&E defined in statute or regulation: DT&E, OT&E, LFT&E, and Interoperability Certification. oversight at any time.

b. DOT&E maintains the DOT&E Oversight List continuously online at https://extranet.dote.osd.mil/oversight/ (requires login with a Common Access Card).

c. The DOT&E Oversight List is unclassified. Classified and sensitive programs that are placed on DOT&E oversight will be identified directly to their MDAs.

d. The DOT&E Oversight List is the list of Major Defense Acquisition Programs (MDAPs) under DOT&E oversight. MDAPs on DOT&E oversight include those programs that meet the statutory definition of 10 U.S.C. 2430 (Reference (h)), and those that are designated by the DOT&E as MDAPs An acquisition program that is designated by the USD(AT&L) as an MDAP; or is estimated to require an eventual total expenditure for RDT&E including all planned increments, of more than $480 million in FY 2014 constant dollars or, for procurement, including all planned increments, of more than $2.79 billion in FY 2014 constant dollars. for the purposes of OT&E under the authority of paragraph (a)(2)(B) of 10 U.S.C. 139 (Reference (h)). The latter programs are not MDAPs for any other purpose.

e. Unless specifically waived, the test-related documentation that is required for MDAP programs will be required for all programs on the DOT&E Oversight List, including submission of Defense Intelligence Agency or DoD Component Validated On-line Life-cycle Threat Reports, TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. , OTPs, Live Fire Test Plans (LFTPs), and reporting of test results.

f. Force protection equipment (including non-lethal weapons) will be subject to DOT&E oversight, as determined by DOT&E. The DOT&E will approve required LFTPs and/or live fire strategies for such systems.

g. Capability upgrades, other alterations that materially change system performance, and alterations that pose substantial risk of degrading fielded military capabilities (if they fail) will be tested operationally. Product improvements or upgrades to system survivability will also be tested and evaluated.

h. The DOT&E Oversight List will identify programs grouped for coordinated or synchronized testing.

4. T&E PROGRAM MANAGEMENT

a. Early Engagement. Program managers for programs under DOT&E oversight will designate a T&E Process by which a system or components are exercised and results analyzed to provide performance-related information. The information has many uses including risk identification and risk mitigation and empirical data to validate models and simulations. T&E enables an assessment of the attainment of technical performance, specifications, and system maturity to determine whether systems are operationally effective, suitable and survivable for intended use, and/or lethal. There are various types of T&E defined in statute or regulation: DT&E, OT&E, LFT&E, and Interoperability Certification. WIPT (also known as an Integrated Test Team), as soon as practicable after the Materiel Development Decision. The T&E WIPT develops and tracks the T&E program in all phases. The T&E WIPT will include empowered representatives of test data stakeholders such as Systems Engineering, Developmental Test and Evaluation ( DT&E 1. Any testing used to assist in the development and maturation of products, product elements, or manufacturing or support processes. 2. Any engineering-type test used to verify status of technical progress, verify that design risks are minimized, substantiate achievement of contract technical performance, and certify readiness for initial operational testing. Developmental tests generally require instrumentation and measurements and are accomplished by engineers, technicians, or soldier operator-maintainer test personnel in a controlled environment to facilitate failure analysis.), OT&E, LFT&E, the user, Product Support, the intelligence community, and applicable certification authorities.

b. Lead Operational Test Agency (OTA). The lead OTA is the responsible OTA for a program. When more than one OTA is responsible for a program, the responsible OTAs will jointly identify the lead OTA. c. Required Documentation. T&E Process by which a system or components are exercised and results analyzed to provide performance-related information. The information has many uses including risk identification and risk mitigation and empirical data to validate models and simulations. T&E enables an assessment of the attainment of technical performance, specifications, and system maturity to determine whether systems are operationally effective, suitable and survivable for intended use, and/or lethal. There are various types of T&E defined in statute or regulation: DT&E, OT&E, LFT&E, and Interoperability Certification. program documentation that already exists in other acquisition documents may be provided by working links. Documentation that directly impacts the OT&E or LFT&E program will be included or linked in the applicable T&E documentation or else the documentation in question will be approved by DOT&E in addition to any other applicable approvals. DOT&E approval or disapproval of a document incorporating links constitutes approval or disapproval of the content applicable to operational testing in all of the links. Specifically, although DOT&E does not approve all the content of linked documents, DOT&E may require changes to linked content dealing specifically with operational or live-fire testing.

5. T&E PROGRAM PLANNING

a. The TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. is a signed contract among DOT&E, senior DoD Component leadership, the lead OTA, the MDA, and the Program Manager.

b. The Program Manager and T&E Process by which a system or components are exercised and results analyzed to provide performance-related information. The information has many uses including risk identification and risk mitigation and empirical data to validate models and simulations. T&E enables an assessment of the attainment of technical performance, specifications, and system maturity to determine whether systems are operationally effective, suitable and survivable for intended use, and/or lethal. There are various types of T&E defined in statute or regulation: DT&E, OT&E, LFT&E, and Interoperability Certification. WIPT will prepare and then update the TEMP to support the acquisition milestones. For the Full-Rate Production Decision Review or the Full Deployment Decision and thereafter (for DOT&E OT&E or LFT&E Oversight programs), DOT&E, the MDA, or the senior DoD Component leadership may require TEMP updates or addendums to address additional testing.

c. Working through the T&E WIPT, program managers for DOT&E oversight programs will make draft TEMPs available to program stakeholders as early and as frequently as possible. DoD Component-approved TEMPs will be submitted to OSD for approval not later than 45 calendar days prior to the milestone decision. (1) A TEMP may be waived for select Accelerated or Urgent Acquisitions. In cases when DOT&E decides a TEMP is not needed, early briefings to DOT&E (in lieu of the TEMP) are recommended to facilitate subsequent DOT&E approval of the OTPs and LFTPs. DOT&E will approve the OTPs and LFTPs for accelerated acquisition (including capabilities acquired in response to an urgent need and acquisitions granted Rapid Acquisition Authority) if those acquisitions are under DOT&E OT&E or LFT&E oversight. If DOT&E has placed an Accelerated Acquisition on oversight, it is because DOT&E has determined that OT&E or LFT&E is required before fielding. Testing to verify safety, survivability, and operational performance will be conducted consistent with the urgency of deploying the capability. The Secretary of Defense may authorize the Rapid Acquisition Official to defer some testing until after fielding if he or she determines that the testing would unnecessarily impede the deployment of the needed capability. Testing should normally include user feedback to support design and operational use improvements. (2) Initial Operational Test and Evaluation (IOT&E) is required for all programs under DOT&E oversight in accordance with 10 U.S.C. 2399 (Reference (h)). The lead OTA will conduct an independent, dedicated phase of IOT&E before full-rate production or full deployment that provides objective test results free from potential conflicts of interest or bias. The primary purpose of IOT&E is to determine a system’s operational effectiveness and operational suitability. IOT&E can also be used to support system certification requirements and training requirements as long as the primary purpose is accomplished.

d. The lead OTA for the program and the Program Manager will initiate coordinated planning for IOT&E as early as possible so that developing activities will be aware of expectations at IOT&E:

  • (1) The lead OTA for the program will provide an assessment of the T&E implications of the initial concept of operations (CONOPS) provided by the user in the Milestone A TEMP.
  • (2) Beginning at Milestone A, the lead OTA will provide a working link in the TEMP to a living document in which the DoD Component’s operational rationale for the requirements in the draft Capability Development Document (CDD) or equivalent requirements document will be tracked.
  • (3) For software acquisitions, the lead OTA will conduct an analysis of operational risk to mission accomplishment covering all planned capabilities or features in the system (see paragraph 7d in this enclosure for additional details). The analysis will include commercial and non-developmental items. The initial analysis will be documented in the Milestone A TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. and updated thereafter.
  • (4) The TEMP will include evaluation of mission-level interoperability across key interfaces. Systems that provide capabilities for joint missions will be tested in the expected joint mission environment

. e. Scientific test and analysis techniques (also referred to as Design of Experiments methodologies) should be employed to design an effective and efficient T&E Process by which a system or components are exercised and results analyzed to provide performance-related information. The information has many uses including risk identification and risk mitigation and empirical data to validate models and simulations. T&E enables an assessment of the attainment of technical performance, specifications, and system maturity to determine whether systems are operationally effective, suitable and survivable for intended use, and/or lethal. There are various types of T&E defined in statute or regulation: DT&E, OT&E, LFT&E, and Interoperability Certification. program. The TEMP should document the test program that will produce the required data to characterize combat mission capability across an appropriately selected set of factors and conditions.

  • (1) Starting at Milestone A, the TEMP should document T&E for phase completion (major test events required for milestone exit and entrance criteria). In addition, each major test phase or event should have test entrance and test completion criteria.
  • (2) Each major test phase or event should have a synopsis of the intended analysis. A synopsis should indicate how the required data for test completion will contribute to one or more standard measures of program progress. These include the following terms: (a) Critical operational issues (also known as critical operational issues and criteria). (b) KPP Performance attribute of a system considered critical or essential to the development of an effective military capability. KPPs are contained in the Capability Development Document and the Capability Production Document and are included verbatim in the Acquisition Program Baseline. KPPs are expressed in term of parameters which reflect Measures of Performance using a threshold/objective format. KPPs must be measurable, testable, and support efficient and effective Test and Evaluation. Mandatory KPPs are specified in the JCIDS Manual.s (c) Critical technical parameters. (d) Key System Attributes Performance attribute of a system considered important to achieving a balanced solution/approach to a system, but not critical enough to be designated as a Key Performance Parameter. KSAs must be measurable, testable, and support efficient and effective Test and Evaluation. KSAs are expressed in terms of Measures of Performance..
  • (3) Every TEMP will include a table of independent variables (or “conditions,” “parameters,” “factors,” etc.) that may have a significant effect on operational performance. Starting at Milestone B, the updated table of variables will include the anticipated effects on operational performance, the range of applicable values (or “levels,” “settings,” etc.), the overall priority of understanding the effects of the variable, and the intended method of controlling the variable during test (uncontrolled variation, hold constant, or controlled systematic test design).
  • (4) Starting at Milestone B, every TEMP will include an evaluation overview. The overview will show how the major test events and test phases link together to form a systematic, rigorous, and structured approach to evaluating mission capability across the applicable values of the independent variables. Test resources will be derived from the evaluation overview (see section 10 in this enclosure).
6. OT&E ACTIVITIES

a. Operational Assessments (OAs)

  • (1) The lead OTA will prepare and report results of one or more early OAs (EOAs) as appropriate in support of one or more of the design phase life-cycle events (namely, the CDD Validation, the Development RFP Release Decision Point, or Milestone B). An EOA is typically an analysis, conducted in accordance with an approved test plan, of the program’s progress in identifying operational design constraints, developing system capabilities, and mitigating program risks. For programs that enter development at Milestone B, the lead OTA will (as appropriate) prepare and report EOA results after program initiation and prior to the Critical Design Review A multi-disciplined technical review to assess design maturity, design build-to or code-to documentation and remaining risks, and establish the initial product baseline. It is used to determine whether the system design is ready to to begin developmental prototype hardware fabrication and/or software coding with acceptable risk Generally, this review assesses the system's design as captured in product specifications for each configuration item (CI) in the system’s product baseline, and ensures that each CI in the product baseline has been captured in the detailed design documentation. Normally conducted during the Engineering and Manufacturing Development (EMD) phase..
  • (2) An OA is a test event that is conducted before initial production units are available and which incorporates substantial operational realism. An OA is conducted by the lead OTA in accordance with a test plan approved by DOT&E for programs that are under OSD OT&E oversight. As a general criterion for proceeding through Milestone C, the lead OTA will conduct and report results of at least one OA. For an acquisition program employing the Incrementally Deployed Software Intensive Program model, a risk-appropriate OA is usually required in support of every limited deployment (see Model 3 at paragraph 5c(3)(d) in this instruction). An operational test, usually an OA, is required prior to deployment of accelerated or urgent acquisition programs that are under OSD OT&E or LFT&E oversight. An OA may be combined with training events (see paragraph 11a(9) in this enclosure). An OA is not required for programs that enter the acquisition system at Milestone C.

b. RFPs. An up-to-date TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. will be provided prior to release of RFP A document used in negotiated acquisitions to communicate Government requirements to prospective contractors and to solicit proposals. RFPs for competitive acquisitions describe the Government’s requirement; anticipated terms and conditions that will apply to the contract; information required to be in the offeror’s proposal; and factors and significant sub-factors that will be used to evaluate the proposal and their relative importance. s for Milestone B and Milestone C. To the maximum extent feasible, RFPs should be consistent with the operational test program documented in the TEMP.

c. OT&E for Reliability May be expressed initially as a desired failure-free interval that can be converted to a failure frequency for use as a requirement. and Maintainability

  • (1) The TEMP will include a plan (typically via working link to the Systems Engineering Plan) to allocate top-level reliability requirements down to the components and sub-components. Reliability allocations will include hardware and software, and will include commercial and non-development items.
  • (2) Reliability Growth (a) Beginning at Milestone B, the TEMP will include T&E for reliability growth and reliability growth curves (RGCs) for the whole system and the reliability of critical systems, sub-systems, components, and sub-components. Reliability-critical items require test to mitigate risk resulting from the use of new technologies or from challenging operating environments. T&E for reliability growth will provide data on initial reliability (namely: identify the contractor and government reliability testing needed to achieve initial reliability) and reliability test events. RGCs will display planned initial reliability, the allocated reliability requirement, a curve showing reliability that is expected during each reliability test event, and points marking reliability test results to date. (b) For software (in any system) reliability growth will be measured by software maturity metrics (e.g., counts of high priority defects) at regular intervals. (c) Beginning at Milestone B, the TEMP will include a working link to the Failure Modes, Effects and Criticality Analysis (FMECA) of identified or anticipated system failure modes, the impacted components and sub-components, and the method of failure mode discovery. A software defect or failure tracking database(s) may replace the FMECA in software acquisitions.
  • (3) Updated TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. s at Milestone C will include updated RGCs that reflect test results to date, any updates to the planned T&E for reliability growth, and a working link to the updated FMECA.

d. Use of Modeling and Simulation. Models or simulations that utilize or portray threat characteristics or parameters must have that portrayal accredited by the Defense Intelligence Agency. Every distinct use of a model or simulation in support of an operational evaluation will be accredited by an OTA, and, for programs under DOT&E Oversight, its use for the operational evaluation will be approved by DOT&E.

7. OT&E FOR SOFTWARE

a. Acquisition of software for any system will normally be supported by specialized models and early user involvement:

  • (1) As feasible, testing of software for any system should be supported by a model (or emulated hardware or virtual machine) of the digital device(s) on which the software runs.
  • (2) To the extent feasible, program managers should test PrototypeAdd a Tooltip Text human interfaces with operational users.
  • (3) Program managers for software acquisitions should develop process models of the time and effort needed to perform critical tasks and functions. Such models support operational test design and analysis of results as well as managerial needs such as sustainment cost projections and analysis of impacts of process changes.
  • (4) Program managers must sustain an operationally realistic maintenance test environment in which software patches can be developed and upgrades of all kinds (developed or commercial) can be tested. The maintenance test environment is a model of the operational environment in that it should be able to replicate software defects found in the operational environment.

b. Program managers for software acquisitions will provide plans at Milestone B indicating how system logs and system status records will interface with operational command and control. At IOT&E Dedicated Operational Test and Evaluation conducted on production, or production representative articles, to determine whether systems are operationally effective and suitable to support a Full-Rate Production decision. The term IOT&E is normally associated with programs on the Director, Operational Test and Evaluation Oversight List. or a prior test event, program managers for software acquisitions will demonstrate performance monitoring of operational metrics to manage and operate each system capability (or the whole system, as appropriate).

c. For software in any system, the evaluation of operational suitability will include a demonstrated capability to maintain the software. IOT&E or a prior test event will include an end-to-end demonstration of regression test, preferably automated, in the maintenance test environment. The demonstration will show how changes in requirements or discovered defects are mapped to lines of software that must be modified, and how modifications in software are mapped to the regression test scripts that will verify correct functioning of the modified software.

d. Risk-Assessed Level of Operational Test for Software Acquisitions (Models 3, 4, and Hybrids)

  • (1) OT&E The field test, under realistic conditions, of any item (or key component) of weapons, equipment, or munitions for the purpose of determining the effectiveness and suitability of the weapons, equipment, or munitions for use in combat by typical military users; and the evaluation of the results of such tests. E for software acquisitions will be guided by the assessment of operational risks of mission failure. A significant operational risk of mission failure is a risk that is at least moderately likely to occur, and if the risk does occur then the impact will cause a degradation or elimination one or more operational capabilities.
  • (2) At any level of risk, the lead OTA will coordinate with DOT&E on the required level of test and then observe the agreed-upon testing. At the lowest risk level, the lead OTA will review plans and observe developmental testing or developmental testing and integrated testing. At the highest risk level, the lead OTA will execute a full OT&E in accordance with the DOT&E-approved OTP. For intermediate risks, the lead OTA will coordinate with the responsible developmental testing organization to observe and execute some integrated developmental testing/operational testing in accordance with a DOT&E-approved OTP.
  • (3) DOT&E will require an operational test or OA for every Limited Deployment in any acquisition model. The scope of the OT&E or OA will be guided by the risk of capability being fielded or deployed.
  • (4) IOT&E Dedicated Operational Test and Evaluation conducted on production, or production representative articles, to determine whether systems are operationally effective and suitable to support a Full-Rate Production decision. The term IOT&E is normally associated with programs on the Director, Operational Test and Evaluation Oversight List. is required for every increment, in any acquisition model (except as noted for urgent operational needs). IOT&E will normally occur prior to the Full Deployment Decision Decision made by the Milestone Decision Authority of a Major Automated Information System acquisition program authorizing an increment of the program to deploy software for operational use. . IOT&E will be guided by an updated assessment of the operational risks in the capabilities and system interactions that have not been successfully evaluated in previous operational testing
8. CYBERSECURITY

a. a. Beginning at Milestone A, the TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. will document a strategy and resources for cybersecurity OT&E The field test, under realistic conditions, of any item (or key component) of weapons, equipment, or munitions for the purpose of determining the effectiveness and suitability of the weapons, equipment, or munitions for use in combat by typical military users; and the evaluation of the results of such tests. , LFT&E, and Interoperability Certification. “] T&E [/tooltip]. At a minimum, software in all systems will be assessed for vulnerabilities. Mission critical systems or mission critical functions and components will also require penetration testing from an emulated threat in an operationally realistic environment during OT&E.

b. Beginning at Milestone B, appropriate measures will be included in the TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. and used to evaluate operational capability to protect, detect, react, and restore to sustain continuity of operation. The TEMP will document the threats to be used, which should be selected based on the best current information available from the intelligence community.

c. The Program Manager, T&E Process by which a system or components are exercised and results analyzed to provide performance-related information. The information has many uses including risk identification and risk mitigation and empirical data to validate models and simulations. T&E enables an assessment of the attainment of technical performance, specifications, and system maturity to determine whether systems are operationally effective, suitable and survivable for intended use, and/or lethal. There are various types of T&E defined in statute or regulation: DT&E, OT&E, LFT&E, and Interoperability Certification. subject matter experts, and applicable certification stakeholders will assist the user in writing testable measures for cybersecurity and interoperability. d. The Program Manager and OTA will conduct periodic cybersecurity risk assessments to determine the appropriate Blue/Green/Red Team, and operational impact test events in alignment with the overall test strategy for evaluating the program for real world effects. Defense business systems will undergo Theft/Fraud operational impact testing.

9. LFT&E

10 U.S.C. 2366 (Reference (h)) mandates the LFT&E and formal LFT&E reporting for all covered systems, as determined by DOT&E, including Accelerated Acquisitions, survivability improvement, and kit programs to address urgent needs. DOT&E will require approval of LFT&E strategies and LFT&E test plans (including survivability test plans) for covered systems as defined in section 2366. The DOT&E will determine the quantity of test articles procured for all LFT&E test events for any system under DOT&E LFT&E oversight.

10. RESOURCES AND SCHEDULE

All TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. s will identify the resources needed to execute the planned T&E Process by which a system or components are exercised and results analyzed to provide performance-related information. The information has many uses including risk identification and risk mitigation and empirical data to validate models and simulations. T&E enables an assessment of the attainment of technical performance, specifications, and system maturity to determine whether systems are operationally effective, suitable and survivable for intended use, and/or lethal. There are various types of T&E defined in statute or regulation: DT&E, OT&E, LFT&E, and Interoperability Certification. activities. Resource estimates will be matched against the schedule 1. Series of things to be done in a specific sequence within a given period of time. 2. A timetable. 3. A listing of activities and events organized by time. and justified by analysis in the TEMP. All TEMPs will contain an updated integrated test program summary and master schedule of all major test events or test phases, to include LFT&E events.

a. Resource estimates (including but not limited to quantities of test articles, targets, expendables, threat simulations, operational forces, etc.) will be derived from defensible statistical measures of merit (power and confidence) associated with quantification of the differences among the factors affecting operational performance as well as the risk to the government of accepting a poorly performing system or incorrectly rejecting a system with acceptable performance. Specifically, the TEMP must discuss and display, or provide a reference to, the calculations done to derive the content of testing and to develop the associated resource estimates.

b. The Program Manager and the Services or Defense Agencies will allocate the resources identified in the TEMP. Each TEMP update will include an updated and complete T&E resource estimate.

c. Test infrastructure, resources (including threat representations), and tools to be used in operational tests must undergo verification by the developer, validation by the DoD Component, and accreditation by the OTA. Test infrastructure, resources, and tools, and their associated verification, validation, and accreditation strategies will be documented in the TEMP.

d. In accordance with 10 U.S.C. 2399 (Reference (h)), DOT&E will approve the quantity of test articles required for all operational test events for any system under DOT&E oversight. The DoD Component OTA will determine the quantity for programs that are not under DOT&E oversight.

e. The T&E schedule will be event-driven and allow adequate time to support pre-test predictions; testing; post-test analysis, evaluation, and reporting; reconciliation of predictive models; and adequate time to support execution of corrective actions in response to discovered deficiencies.

f. For incremental software acquisitions employing limited deployments (see Model 3 at paragraph 5c(3)(d) in this instruction), the Milestone The point at which a recommendation is made and approval sought regarding starting or continuing an acquisition program, i.e., proceeding to the next phase. Milestones established by DoDI 5000.02 are: Milestone A that approves entry into the Technology Maturation and Risk Reduction phase; Milestone B that approves entry into the Engineering and Manufacturing Development phase; and Milestone C that approves entry into the Production and Deployment phase. B TEMP will show a general schedule for the routine test sequence (developmental tests, certifications, integrated and operational tests) that will occur with every limited deployment within the allotted time for each limited deployment.

11. OPERATIONAL AND LIVE FIRE T&E EXECUTION

The general process for planning, executing, and reporting on operational and major live fire test events is shown in Figure 9.

Figure 9. Operational or Major Live Fire Test Event: Planning, Approval, Execution, and Reporting

a. Planning Test Events

  • (1) For all programs under DOT&E oversight, including Accelerated Acquisitions, DOT&E will approve OTPs and LFTPs prior to the corresponding operational or major live fire test events in accordance with 10 U.S.C. 2399. DOT&E will approve any LFTP for a major test event such as Full-up System Level test, Total Ship Survivability Trial, or Full Ship Shock Trials. The major live fire test events will be identified in the TEMP (or LFT&E strategy or equivalent document). Test plans are developed by a lead test organization (LTO). The LTO is the lead OTA for OT&E. The LTO varies for LFT&E.
  • (2) For programs under DOT&E oversight, the appropriate LTO will brief the DOT&E on T&E concepts for the OTP or the major LFT&E as early as possible and not less than 180 calendar days prior to start of any such testing. DOT&E and DoD Component leads will be kept apprised of changes in test concept and progress on the OTP. The lead OTA will deliver the DoD Component-approved OTP for DOT&E review not later than 60 calendar days before test start. The LTO for major live fire events will deliver the DoD Component-approved LFTP for DOT&E review not later than 90 days before test start.
  • (3) OTPs and major LFTPs will include the plans for data collection and management.
  • (4) Integrated Testing (a) Integrated testing is the collaborative planning and collaborative execution of test phases and events to provide shared data in support of independent analysis, evaluation and reporting by all stakeholders particularly the developmental (both contractor and government) and operational test and evaluation communities. It requires the active participation of the lead OTA in planning the integrated tests with the program office so that the operational objectives are understood, the testing is conducted in an operationally realistic manner, and the resultant data is relevant for use in operational evaluations. (b) For integrated test results to count for operational testing, the lead OTA must develop a plan for the integrated test to be approved by DOT&E before the start of testing that, at a minimum, details the required test realism and conditions, operational test objectives, operational test metrics and data collection requirements. Data collected outside an approved OTP or major LFTP can be used for a DOT&E operational or live fire evaluation if the data is approved by DOT&E. Depending on circumstances, DOT&E approval will not necessarily be possible in the TEMP and may require some other documentation. Data approval will be based on understanding of the realism of the test scenario(s) used and the pedigree (test conditions and methodologies) of the data. The data in question will typically come from operational exercises, certification events, and developmental test events conducted in operationally relevant environments. Data approval should be coordinated with the LTO and DOT&E prior to the start of testing. When advance coordination is not possible, the LTO will facilitate data re-use (in a DOT&E assessment or evaluation) through independent documentation of the test data pedigree (test conditions and methodologies).
  • (5) In OT&E, typical users or units will operate and maintain the system or item under conditions simulating combat stress in accordance with 10 U.S.C. 139 (Reference (h)) and peacetime conditions, when applicable. The lead OTA, in consultation with the user and the Program Manager, will identify realistic operational scenarios based on the CONOPS (per paragraph 5d(1) in this enclosure) and mission threads derived from the Joint Mission Essential Task List or DoD Component-specific Mission Essential Task List. See paragraph 7d of this enclosure for risk-assessed OT&E of software acquisitions.
  • (6) In accordance with 10 U.S.C. 2399 (Reference (h)), persons employed by the contractor for the system being developed may only participate in OT&E of systems under OSD OT&E oversight to the extent they are planned to be involved in the operation, maintenance, and other support of the system when deployed in combat. (a) A contractor that has participated (or is participating) in the development, production, or testing of a system for a DoD Component (or for another contractor of the DoD) may not be involved in any way in establishing criteria for data collection, performance assessment, or evaluation activities for OT&E. (b) These limitations do not apply to a contractor that has participated in such development, production, or testing, solely in test or test support on behalf of the DoD.
  • (7) IOT&E Dedicated Operational Test and Evaluation conducted on production, or production representative articles, to determine whether systems are operationally effective and suitable to support a Full-Rate Production decision. The term IOT&E is normally associated with programs on the Director, Operational Test and Evaluation Oversight List. for all programs will use production or production-representative test articles that, at a minimum, will incorporate the same parts and software items to be used in LRIP articles. Production-representative systems meet the following criteria: (a) The hardware and software must be as defined by the system-level critical design review, functional configuration audit, and system verification review, including correction of appropriate major deficiencies identified during prior testing. (b) For hardware acquisitions, production-representative articles should be assembled using the parts, tools, and manufacturing processes intended for use in full-rate production; utilize the intended production versions of software; and the operational logistics systems including mature drafts of maintenance manuals intended for use on the fielded system should be in place. The manufacturing processes to be used in full-rate production should be adhered to as closely as possible, and program managers for programs under DOT&E OT&E oversight will provide DOT&E a detailed description of any major manufacturing process changes. (c) For software acquisitions, a production-representative system consists of typical users performing operational tasks with the hardware and software intended for deployment, in an operationally realistic computing environment, with representative DoD information network operations and supporting cybersecurity capabilities. All manuals, training, helpdesk, continuity of operations, system upgrade and other life-cycle system support should be in place.
  • (8) IOT&E Dedicated Operational Test and Evaluation conducted on production, or production representative articles, to determine whether systems are operationally effective and suitable to support a Full-Rate Production decision. The term IOT&E is normally associated with programs on the Director, Operational Test and Evaluation Oversight List. will require more than an evaluation that is based exclusively on computer modeling, simulation, or an analysis of system requirements, engineering proposals, design specifications, or any other information contained in program documents in accordance with 10 U.S.C. sections 2399 and 2366 (Reference (h)). IOT&E will feature end-to-end testing of system capabilities including all interrelated systems needed to employ and support those capabilities. (9) Program managers for all programs (and particularly Accelerated Acquisitions) may, in coordination with the lead OTA, elect to perform integrated testing in conjunction with training, joint and operational exercises, or synchronized test events. Such testing is efficient, but inherently increases the risk that a significant problem will not be discovered. If no subsequent operational testing is conducted prior to fielding, then additional testing will typically be required subsequent to initial fielding. When subsequent testing is required, the plan for the T&E and reporting of results will be included in the applicable TEMP or other planning documentation.

b. Conducting Test Events

  • (1) Test plans must consider the potential impacts on personnel and the environment, in accordance with 42 U.S.C. 4321-4347 (Reference (ag)) and Executive Order 12114 (Reference (ah)). The Program Manager, working with the user and the T&E community, will provide safety releases (to include formal environment, safety, and occupational health risk acceptance in accordance with section 16 of Enclosure 3 of this instruction) to the developmental and operational testers prior to any test that may impact safety of personnel.
  • (2) Barring significant unforeseen circumstances, all elements of an approved OTP or LFTP must be fully satisfied by the end of an operational or live fire test. If an approved plan cannot be fully executed, DOT&E concurrence with any changes must be obtained before revised test events are executed. Once testing has begun, deviations from approved elements of the test plan cannot be made prior to the beginning of their execution without consultation with the OTA commander (for OTP) or appropriate LTO (for LFTP) and the concurrence of DOT&E. DOT&E concurrence is not required when a need to change the execution of an element of the test plan arises in real time as its execution is underway. If DOT&E on-site representatives are not present and the test director concludes changes to the plan are warranted that would revise events yet to be conducted, the test director must contact the relevant DOT&E personnel to obtain concurrence with the proposed changes. If it is not possible to contact DOT&E personnel in a timely manner, the test director can proceed with execution of the revised test event but must inform DOT&E of the deviations from the test plan as soon as possible.
  • (3) When the order of execution is identified in the TEMP as affecting the analysis of the data, test plans should include details on the order of test event execution and/or test point data collection.
  • (4) Operating instructions (i.e., tactics, techniques and procedures, standard operating procedures, technical manuals, technical orders) should be considered for their impact on the test outcomes and included in OTPs when relevant.
  • (5) Test plans must include the criteria to be used to make routine changes (delays for weather, test halts, etc.).
  • (6) If required data for the test completion criteria are lost, corrupted, or not gathered, then the test is not complete unless the requirement is waived by DOT&E.

c. Data Management, Evaluation, and Reporting

  • (1) DOT&E, the Program Manager and their designated representatives who have been properly authorized access, will all have full and prompt access to all records, reports, and data, including but not limited to data from tests, system logs, execution logs, test director notes, and user and operator assessments and surveys. Data include but are not limited to classified, unclassified, and (when available) competition sensitive or proprietary data. Data may be preliminary and will be identified as such.
  • (2) OTAs and other T&E agencies will record every OT&E and LFT&E event in some written form. Full reports will often contain multiple test events and will be accomplished in the most timely manner practicable. Interim summaries or catalogues of individual events will be prepared as results become available.
  • (3) Significant problems will be reported promptly to senior DoD leadership when those problems are identified. OTAs will publish interim test event summaries as interim reports when the test events provide information of immediate importance to the program decision makers. This will occur particularly in support of accelerated acquisitions and time critical operational needs. Such reports should provide the most complete assessment possible based on the available data and should not be delayed. Such reports will be followed by the planned comprehensive reporting.
  • (4) For DOT&E OT&E and LFT&E oversight programs, DOT&E will be kept informed of available program assets, assessments, test results and anticipated timelines for reporting throughout report preparation.
  • (5) The Program Manager and test agencies for all programs will provide the Defense Technical Information Center (DTIC) with all reports, and the supporting data and metadata for the test events in those reports. If there are limitations in the data or metadata that can be provided to DTIC, those limitations will be documented in the TEMP starting at Milestone B.
  • (6) Test agencies will provide the DoD Modeling and Simulation Coordination Office with a descriptive summary and metadata for all accredited models or simulations that can potentially be reused by other programs.
  • (7) The Secretaries of the Military Departments, in coordination with the Defense Acquisition Executive, DOT&E, and the Under Secretary of Defense for Personnel and Readiness, will establish a common set of data for each major weapon system type to be collected on damage incurred during combat operations. This data will be stored in a single dedicated and accessible repository at DTIC. The lessons learned from analyzing this data will be included, as appropriate, in both the capability requirements process and the acquisition process for new acquisitions, modifications, and/or upgrades.
12. OPERATIONAL TEST READINESS

The DoD Components The Office of the Secretary of Defense; the military departments; the Chairman, Joint Chiefs of Staff and the Joint Staff; the combatant commands; the Office of the Inspector General of the DoD; the defense agencies; DoD field activities; and all other organization entities within the DoD. will each establish an Operational Test Readiness Review process to be executed for programs under DOT&E oversight prior to any Operational Test. Prior to IOT&E, the process will include a review of DT&E 1. Any testing used to assist in the development and maturation of products, product elements, or manufacturing or support processes. 2. Any engineering-type test used to verify status of technical progress, verify that design risks are minimized, substantiate achievement of contract technical performance, and certify readiness for initial operational testing. Developmental tests generally require instrumentation and measurements and are accomplished by engineers, technicians, or soldier operator-maintainer test personnel in a controlled environment to facilitate failure analysis. results, an assessment of the system’s progress against the KPP Performance attribute of a system considered critical or essential to the development of an effective military capability. KPPs are contained in the Capability Development Document and the Capability Production Document and are included verbatim in the Acquisition Program Baseline. KPPs are expressed in term of parameters which reflect Measures of Performance using a threshold/objective format. KPPs must be measurable, testable, and support efficient and effective Test and Evaluation. Mandatory KPPs are specified in the JCIDS Manual.s, key system attributes Performance attribute of a system considered important to achieving a balanced solution/approach to a system, but not critical enough to be designated as a Key Performance Parameter. KSAs must be measurable, testable, and support efficient and effective Test and Evaluation. KSAs are expressed in terms of Measures of Performance., and critical technical parameters in the TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. , an analysis of identified technical risks to verify that those risks have been retired or mitigated to the extent possible during DT&E and/or OOT&E The field test, under realistic conditions, of any item (or key component) of weapons, equipment, or munitions for the purpose of determining the effectiveness and suitability of the weapons, equipment, or munitions for use in combat by typical military users; and the evaluation of the results of such tests. , a review of system certifications, and a review of the IOT&E entrance criteria specified in the TEMP.

13. CERTIFICATIONS

Testing in support of certifications should be planned in conjunction with all other testing.

a. The Program Manager is responsible for determining what certifications are required; ensuring involvement of the representatives of applicable certifying authorities in the T&E WIPT; and satisfying the certification requirements.

b. The Program Manager will provide the MDA, DOT&E, and the lead OTA with all data on certifications as requested.

c. In accordance with DoD Instruction 8330.01 (Reference (ab)), the TEMP Documents the overall structure and objectives of the T&E program and articulates the necessary resources to accomplish each phase of test. It provides a framework within which to generate detailed T&E plans and documents schedule and resource implications associated with the T&E program. The TEMP also identifies the necessary DT&E, OT&E, and Live Fire Test & Evaluation activities, and provides a clear roadmap connecting evaluation objectives, test measures, requirements, test methodologies, decision points, test events, and resources. For multi-Service or joint programs, a single integrated TEMP is required. for all programs must reflect interoperability and supportability requirements, and serve as the basis for interoperability assessments and certifications.

14. TEMP EVOLUTION THROUGH THE ACQUISITION MILESTONES

The preceding policies are summarized together with associated DOT&E guidance and TEMP outlines at http://www.dote.osd.mil/temp-guidebook/index.html.