Technical Reviews and Audits

Purpose of Technical Reviews and Audits

For DoD systems development, a properly tailored series of technical reviews and audits provide key points throughout the system development to evaluate significant achievements and assess technical maturity and risk. DoDI 5000.85 and the Adaptive Acquisition Framework Document Identification (AAFDID) Tool identify the statutory and regulatory requirements for acquisition programs. Regardless of acquisition pathway, the PM, Systems Engineer, and Lead Software Engineer work to properly align the applicable technical reviews to support knowledge-based milestone decisions that streamline the acquisition life cycle and save precious taxpayer dollars. Technical reviews and audits allow the PM, Systems Engineer, and Lead Software Engineer to jointly define and control the program???s technical effort by establishing the success criteria for each review and audit. A well-defined program facilitates effective monitoring and control through increasingly mature points.

The Engineering of Defense Systems Guidebook provides guidance on selecting and tailoring technical reviews and audits for each of the AAF pathways. Underpinning most if not all of these technical reviews and audits is the need to conduct a wide range of program-related analyses. Regardless of acquisition pathway, the ability to conduct such analyses can be profoundly impacted by the extent to which the program adopts a Digital Engineering (DE) approach (as described more fully in the Systems Engineering (SE) Guidebook, Section 2.2.2, Digital Engineering). As mentioned there, DoD???s approach to implementing DE is to ???securely and safely connect people, processes, data, and capabilities across an end-to-end digital enterprise. This will enable the use of models throughout the lifecycle to digitally represent the system of interest (i.e., SoS, processes, equipment, products, parts) in the virtual world.???

The extent to which a program adopts a DE approach will not impact ???what??? technical reviews and audits need to be conducted, but it can have a profound and revolutionary impact upon ???how??? they are conducted. A well-defined digital ecosystem, instantiated or leveraged, with an associated authoritative source of truth and static and dynamic models of systems and the battlespace will enable timely and iterative analyses. In addition, by leveraging constructive, virtual, and live simulation tools, the ecosystem can open up the trade space to enable exploration of options not easily analyzed elsewise.

Event-Driven Technical Reviews and Audits

Technical reviews of program progress should be event driven and conducted when the system under development meets the review entrance criteria as documented in the program???s Systems Engineering Plan (SEP). An associated activity is to identify technical risks associated with achieving entrance criteria at each of these points (see the DoD Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs). Systems Engineering (SE) is an event-driven process based on successful completion of key events as opposed to arbitrary calendar dates. As such, the SEP should clarify the timing of events in relation to other SE and program events. While the initial SEP and Integrated Master Schedule (IMS) have the expected occurrence in the time of various milestones (such as overall system Critical Design Review (CDR)), the plan should be updated to reflect changes to the actual timing of SE activities, reviews and decisions. Figure 3-1 of the SE Guidebook provides the end-to-end perspective and the integration of SE technical reviews and audits across all AAF pathways. Technical reviews should be tailored appropriately for other acquisition pathways.

Figure 3-1. Technical Reviews and Audits for the Major Capability Acquisition Life Cycle

Image of the Technical Reviews and Audits for the Major Capability Acquisition Life Cycle

Support of the Defense Acquisition System

Properly structured, technical reviews and audits support the Defense Acquisition System by:

Providing a disciplined sequence of activities to define, assess, and control the maturity of the system???s design and technical baseline, reducing risk over time.
Facilitating an accurate technical assessment of the system???s ability to satisfy operational requirements established in capability requirements documents.
Providing a framework for interaction with the Joint Capabilities Integration and Development System (JCIDS) and Planning, Programming, Budgeting and Execution (PPBE) processes.
Providing a technical assessment and assurance that the end product fulfills the design and process requirements.

Knowledge-Based Approach

Successful development of a complex system requires a knowledge-based approach. Increasing levels of knowledge are a natural consequence of design maturation; however, successful programs establish a deliberate acquisition approach whereby major investment decision points are supported by requisite levels of knowledge. The GAO study on Assessments of Selected Weapons Programs (GAO-12-400SP) provides quantitative evidence to affirm this best practice.

Timing and Tailoring of Technical Reviews and Audits

Technical reviews should occur when the requisite knowledge is expected and required. This section provides guidance on entrance and exit criteria for the level of maturity expected at each technical review and audit. OSD established the expected reviews and audits for each phase of systems development in the outline for the SEP. These policy and guidance documents provide a starting point for the PM, Systems Engineer, and Lead Software Engineer to develop the program???s unique set of technical reviews and audits. Tailoring is expected to best suit the program objectives (see SE Guidebook, Section 1.4, Systems Engineering Policy and Guidance). The SEP captures the output of this tailoring and is reviewed and approved to solidify the program plan.

Programs that tailor the timing and scope of these technical reviews and audits to satisfy program objectives increase the probability of successfully delivering required capability to the warfighter. Technical reviews provide the forum to frame issues and assumptions. They define options necessary to balance risk in support of continued development.

Technical Baselines and Technical Reviews

The technical baseline (including the functional, allocated and product baselines) established at the conclusion of certain technical reviews inform all other program activity. Accurate baselines and disciplined reviews serve to integrate and synchronize the system as it matures, which facilitates more effective milestone decisions and ultimately provides better warfighting capability for less money. The technical baseline provides an accurate and controlled basis for:

Managing change.
Cost estimates, which inform the PPBE process and the APB.
Program technical plans and schedules, which also inform the APB.
Contracting activity.
Manufacturing and Quality (M&Q) efforts.
Test and Evaluation (T&E) efforts.
Risk analysis and risk balancing.
Reports to acquisition executives and Congress.

System Developers and Technical Reviews

The PM and the Systems Engineer need to keep in mind that technical reviews and audits provide visibility into the quality and completeness of the developer???s work products. These requirements should be captured in the contract specifications or Statement of Work (SOW). The program office should consider providing the SEP with the Request for Proposal (RFP) and requiring the contractor deliver a Systems Engineering Management Plan (SEMP) that is consistent with the SEP. As a best practice, the SEMP should include entrance criteria and associated design data requirements for each technical review and audit. The configuration and technical data management plans should clearly define the audit requirements.

Contract incentives are frequently tied to completion of technical reviews. Some stakeholders may have a strong incentive to call the review complete as soon as possible. The review chairperson and Systems Engineer should exercise best judgment in an objective, informed manner to ensure the reviews are not prematurely declared complete.

Complex Systems and Incremental Reviews and Audits

For complex systems, reviews and audits may be conducted for one or more system elements, depending on the interdependencies involved. These incremental system element-level reviews lead to an overall system-level review or audit. After all incremental reviews are complete, an overall summary review is conducted to provide an integrated system analysis and capability assessment that could not be conducted by a single incremental review. Each incremental review should complete a functional or physical area of design. This completed area of design may need to be reopened if other system elements drive additional changes in this area. If the schedule is being preserved through parallel design and build decisions, any system deficiency that leads to reopening design may result in rework and possible material scrap.

Program Protection Planning

To design for system security, the program protection planning and execution activities should be integrated into the systems engineering technical reviews and audits. See Technology and Program Protection (T&PP) Guidebook (forthcoming) Section 5 for system security engineering (SSE) criteria for each technical review and audit.

Roles and Responsibilities

For each technical review, a technical review chair is identified and is responsible for evaluating products and determining the criteria are met and action items are closed. The Service chooses the technical review chair, who could be the PM, Systems Engineer, or other subject matter expert selected according to the Service???s guidance. This guidance may identify roles and responsibilities associated with technical reviews and audits. It also may specify the types of design artifacts required for various technical reviews. In the absence of additional guidance, each program should develop and document its tailored design review plan in the SEP.

The following notional duties and responsibilities associated with the PM, Systems Engineer, and Lead Software Engineer should be considered in the absence of specific Service or lower level (e.g., System Command or PEO) guidance:

Typical PM Responsibilities for Technical Reviews and Audits:

The PM is typically responsible for:

Co-developing with the Systems Engineer the technical objectives of the program that guide the technical reviews and audits.
Co-developing with the Systems Engineer the earned value credit derived from the review.
Approving, funding, and staffing the planned technical reviews and audits; documenting this plan in the SEP and applicable contract documents.
Ensuring the plan for each review includes participants with sufficient objectivity with respect to satisfying the pre-established review criteria.
Ensuring the plan addresses the need for timely and sufficient data to satisfy the statutory and regulatory requirements of DoDI 5000.85.
Controlling the configuration of each baseline and convening configuration steering boards when user requirement changes are warranted. This can lead to an unscheduled gateway into the Functional Capabilities Board (FCB) and JCIDS process not identified in Figure 3-1 above.

Typical Systems Engineer Responsibilities for Technical Reviews and Audits:

The Systems Engineer is typically responsible for:

Co-developing with the PM the technical objectives of the program that guide the technical reviews and audits.
Developing and documenting the technical review and audit plan in the SEP, carefully tailoring each event to satisfy program objectives and SEP outline guidance associated with the minimum technical reviews and audits.
Ensuring the plan is event based with pre-established review criteria and entrance and exit criteria for each event, informed by the knowledge point objectives for the specific program and acquisition pathway.
Identifying the resources required to support the plan; ensuring the activities leading up to the official review and audit are integrated.
Ensuring technical reviews and audits are incorporated into the IMP and IMS.
Coordinating with Chief Development Tester to provide at each technical review: Developmental Test and Evaluation (DT&E) activities to-date, planned activities, assessments to-date and risk areas.
Ensuring a status of applicable design considerations are provided at each technical review. See SE Guidebook, Section 5, for 24 typical design considerations.
Establishing technical reviews and audits and their review criteria in the applicable contract documents (e.g., SOW, IMP).
Monitoring and controlling execution of the established plans.
Coordinating with the appointed technical review chairperson on the technical review plans and supporting execution of the technical reviews.
Assigning responsibilities for closure actions and recommending to the chairperson and PM when a system technical review should be considered complete (see SE Guidebook, Figure 3-2).

Figure 3-2: Technical Review Process

Image of the Technical Review Process

Key Stakeholders for Technical Reviews and Audits

The PM, Systems Engineer, and Lead Software Engineer should identify key stakeholders who have an interest or role in the review, which may include:

Technical review chairperson
PEO
Contracting Officer
Defense Contract Management Agency (DCMA) and Defense Contract Audit Agency (DCAA)
Product Support Manager (PSM)
Product Improvement Manager/Requirements Officer
End User Community
Chief Developmental Tester
Interdependent Acquisition Programs
Business Financial Manager
Deputy Director, Engineering
Service Technical Leadership such as chief engineers
Subject Matter Experts (SMEs)

Technical Review and Audit Criteria and Definitions

Specific review criteria are provided in each technical review and audit section below. These criteria should be achieved and all action items closed before a technical review is considered complete. The Systems Engineer may refer to IEEE 15288.2 "Standard for Technical Reviews and Audits on Defense Programs" as a resource. Instructions for how DoD military and civilian employees can access the IEEE 15288.2 via the Acquisition Streamlining and Standardization Information System (ASSIST) are located on the DDR&E(AC)/Engineering website. If a PMO chooses to use IEEE 15288.2, additional guidance for implementing the DoD-adopted systems engineering standard on acquisition programs contracts can be found in the Best Practices for Using Systems Engineering Standards (ISO/IEC/IEEE 15288, IEEE 15288.1, and IEEE 15288.2) on Contracts for Department of Defense Acquisition Programs guidance document. When comparing this section on technical reviews and audits to IEEE 15288.2 keep in mind:

The Alternative Systems Review (ASR) focuses on achieving a government-to-government understanding of the user???s needs and the preferred materiel solution. It occurs in the Materiel Solution Analysis phase of the Major Capability Acquisition pathway before a development contract is awarded.
The Test Readiness Review (TRR) is addressed in the T&E Enterprise Guidebook (forthcoming).
With the exception of TRR, this Section 3 of the SE Guidebook addresses all technical reviews and audits in clauses 5 and 6 of IEEE 15288.2. Annexes A-D of the standard address other reviews that may be useful, depending on program needs: Software requirements and architecture review (SAR); software specification review (SSR); integration readiness review (IRR); and flight readiness review (FRR).

Alternative Systems Review (ASR)

The Alternative Systems Review (ASR), typically held for the MCA pathway, is conducted to support a dialogue between the end user and acquisition community. It leads to a draft performance specification for the preferred materiel solution. The ASR typically occurs during the MCA Materiel Solution Analysis (MSA) phase, after completion of the Analysis of Alternatives (AoA) and before Milestone A. It focuses technical efforts on requirements analysis. See SE Guidebook, Section 3.1

System Requirements Review (SRR)

The System Requirements Review (SRR) is a multi-disciplined technical review to ensure that the developer understands the system requirements and is ready to proceed with the initial system design. This review assesses whether the system requirements are captured in the system performance specification (sometimes referred to as the System Requirements Document (SRD)). See SE Guidebook, Section 3.2

System Functional Review (SFR)

The System Functional Review (SFR) is held to evaluate whether the functional baseline satisfies the end-user requirements and capability needs and whether functional requirements and verification methods support achievement of performance requirements. At completion of the SFR, the functional baseline is normally taken under configuration control by the government. See SE Guidebook, Section 3.3

Preliminary Design Review (PDR)

The Preliminary Design Review (PDR) should provide sufficient confidence to proceed with detailed design. The PDR ensures the preliminary design and basic system architecture are complete, that there is technical confidence the capability need can be satisfied within cost and schedule goals and that risks have been identified and mitigation plans established. It also provides the acquisition community, end user and other stakeholders with an opportunity to understand the trade studies conducted during the preliminary design, and thus confirm that design decisions are consistent with the user???s performance and schedule needs and the validated JCIDS Capability Development Document (CDD), or CDD-like document for other pathways. The PDR establishes the allocated baseline. See SE Guidebook, Section 3.4

Critical Design Review (CDR)

The Critical Design Review (CDR) confirms the system design is stable and is expected to meet system performance requirements, and confirms the system is on track to achieve affordability and should-cost goals as evidenced by the detailed design documentation. The CDR establishes the initial product baseline. See SE Guidebook, Section 3.5

System Verification Review (SVR) / Functional Configuration Audit (FCA)

The System Verification Review (SVR) is the technical assessment point at which the actual system performance is verified to meet the requirements in the system performance specification and is documented in the functional baseline. The Functional Configuration Audit (FCA) is the technical audit during which the actual performance of a system element is verified and documented to meet the requirements in the system element performance specification in the allocated baseline. Further information on FCA can be found in MIL-HDBK-61, Configuration Management Guidance. SVR and FCA are sometimes used synonymously when the FCA is at the system level. See SE Guidebook, Section 3.6

Production Readiness Review (PRR)

The Production Readiness Review (PRR) for the system determines whether the system design is ready for production, and whether the developer has accomplished adequate production planning for entering Low-Rate Initial Production (LRIP) and Full-Rate Production (FRP) for the MCA pathway. Production readiness increases over time with incremental assessments accomplished at various points in the life cycle of a program. See SE Guidebook, Section 3.7

Physical Configuration Audit (PCA)

The Physical Configuration Audit (PCA) is a formal examination of the "as-built" configuration of the system or a configuration item against its technical documentation to establish or verify its product baseline. The objective of the PCA is to resolve any discrepancies between the production-representative item that has successfully passed Operational Test and Evaluation (OT&E) and the associated documentation currently under configuration control. A successful PCA provides the Milestone Decision Authority (MDA) with evidence that the product design is stable, the capability meets end-user needs and production risks are acceptably low. At the conclusion of the PCA, the final product baseline is established and all subsequent changes are processed by formal engineering change action. Further information can be found in MIL-HDBK-61, Configuration Management Guidance. SVR and FCA are sometimes used synonymously when the FCA is at the system level. See SE Guidebook, Section 3.8

Test Readiness Review (TRR)

The TRR is used to assess a contractor???s readiness for testing configuration items, including hardware and software. It typically involves a review of earlier or lower-level test products and test results from completed tests and a look forward to verify the test resources, test cases, test scenarios, test scripts, environment and test data have been prepared for the next test activity. For programs using the MCA pathway, TRRs typically occur in the EMD and P&D phases. A TRR provides the formal approval authority with a review showing that the system is ready to enter the test and that the funding and execution of a test executes the test and gathers the required information. TRRs assess test objectives, test methods and procedures, test scope, safety, and whether test resources have been properly identified and coordinated. TRRs are also intended to determine if any changes are required in planning, resources, training, equipment, or timing to successfully proceed with the test. If any of these items are not ready, senior leadership may decide to proceed with the test and accept the risk, or mitigate the risk in some manner.

Resources

Key Terms

Allocated Baseline
Alternative System Review (ASR)
Critical Design Review (CDR)
Functional Baseline
Functional Configuration Audit (FCA)
Preliminary Design Review (PDR)
Product Baseline
Production Readiness Review (PRR)
System Functional Review (SFR)
System Requirements Review (SRR)
System Verification Review (SVR)
Technical Baselines
Test Readiness Review (TRR)

Policy and Guidance

DoDI 5000.85, Major Capability Acquisition; Section 3, Major Capability Acquisition Procedures, provides information about life cycle phase activities and decision reviews that technical reviews and audits support.
DoDI 5000.88, Engineering of Defense Systems, Section 3.5. Program Technical Reviews and Assessments
Engineering of Defense Systems Guidebook, DDR&E(AC); this guidebook provides guidance on activities to be conducted in life cycle phases of the six Adaptive Acquisition Framework pathways.
Systems Engineering Guidebook, DDR&E(AC); this guidebook provides guidance on systems engineering in general (Section 1), system-level considerations (Section 2), technical reviews and audits (Section 3), SE technical and technical management processes (Section 4), and SE design considerations (Section 5).
GAO-12-400SP, Defense Acquisitions: Assessments of Selected Weapon Programs.
DI-SESS-81785A, Systems Engineering Management Plan (SEMP) Data Item Description (DID).
IEEE 15288.2, Standard for Technical Reviews and Audits on Defense Programs

Articles

ODASD(SE) Best Practices for Using Systems Engineering Standards on Contracts for DoD Acquisition Programs, April 2017

NDIA Guidance for Utilizing Systems Engineering Standards (IEEE 15288.1 and IEEE 15288.2) on Contracts for Defense Projects, 23 July 2015

Back to top

Training

CLE 003: Technical Reviews (access print files)