Standard: Test and Evaluation Master Plan
Phase: Concept and Technology Development
Activity: Program Planning/Management
Task: Establish Test and Evaluation Strategy
References: • Interim Defense Acquisition Guidebook
• DFAS 8430.1, Chapter 4
Effective Date: May 17, 2002
DEFENSE FINANCE AND ACCOUNTING SERVICE
Test and Evaluation Master Plan
(Program Title/System Name)
Date of Issue and Status: Date plan is acknowledged as acceptable (mm/dd/yyyy) and whether plan is draft
or approved. Also include version number.
Issuing organization: Identify the organization issuing this plan.
TEST AND EVALUATION MASTER PLAN
1. Purpose. This document identifies the high-level requirements and defines the objectives and overall
structure of the Test and Evaluation (T&E) for the (insert system name). It provides the framework within
which detailed T&E plans are generated. It details the test strategy, schedule, and resource requirements for
test and evaluation. It relates program schedule, test management strategy and structure, and required resources
to Critical Operational Issues (COIs), Key Performance Parameters (KPPs) and operational performance
parameters (threshold and objective criteria), and critical technical parameters (CTPs), derived from the
Operational Requirements Document (ORD), evaluation criteria, and major decisions. Specifically, this Test
and Evaluation Master Plan (TEMP):
• Defines operational and system requirements
• Establishes test levels and objectives
• Assigns all items to be tested (requirements, performance objectives, Federal Financial Management
Requirements (FFMRs), system interfaces, etc.) to testing levels
• Relates performance objectives to test level
• Assigns test management responsibility
• Assigns test execution and support responsibility
• Establishes high-level test schedule
• Defines how the system components will accomplish the planned testing and evaluation for each life
cycle phase in order to support major program decisions
• Identifies special T&E resources and requirements to facilitate long-range planning including the
rationale and schedule for planned tests, and
• Relates the T&E effort clearly to technical characteristics, technical risk, operational issues and
concepts, system performance, reliability, availability, maintainability, logistics requirements, and major
2. Scope. Identify the program/programs this TEMP applies to. Briefly describe the program in terms of
requirements, organization, and development.
3. PART I--SYSTEM INTRODUCTION.
a. Mission Description. In this section, summarize the business/mission need as described in the Mission
Need Statement (MNS). Reference the MNS, Command, Control, Communications, Computers, and
Intelligence (C4I) Support Plan (C4ISP) and ORD. Describe the mission in terms of objectives and general
capabilities. Include a description of the operational and support environment envisioned for the system.
Provide an operational view (OV-1).
b. System Description. Briefly describe the system design, to include the following items:
(1) Key features and subsystems, both hardware and software (such as architecture, interfaces, security
levels, etc.), for each block/configuration, which enables the system to perform its required
(2) Interfaces with existing or planned systems that are required for mission accomplishment. Address
relative maturity and integration and modification requirements for non-developmental items.
Include interoperability with existing and/or planned systems of other Department of Defense (DoD)
components or allies. Correlate this description with the ORD information exchange requirements
(IERs) of the program. The IER matrix required in the ORD is referenced or included here. Provide
a diagram of the system architecture command, control, communications, computer, intelligence,
surveillance, and reconnaisance (C4ISR) System View (SV-1, System Interface Diagram)).
(3) Critical system characteristics or unique support concepts resulting in special test and analysis
requirements (e.g., post deployment software support; hardness against nuclear effects; resistance to
countermeasures; resistance to reverse engineering/exploitation efforts (Anti-Tamper).
c. System Threat Assessment. Reference the System Threat Assessment and briefly summarize the threat
environment described in that assessment. Note: This section should directly correspond to Section 2 of the
program's ORD. For most Defense Finance and Accounting Service (DFAS) automated information systems
(AISs), the below threat statement is sufficient to complete this section of the TEMP.
System security requirements will be developed in accordance with the provisions of Office of Management
and Budget Circular A-130, Management of Federal Information Resources, and DoD Directive 5200.28,
Security Requirements for Automated Information Systems (AIS). System security risks will be assessed and
managed in accordance with DoD Instruction 5200.40, DoD Information Technology Security Certification and
Accreditation Process (DITSCAP). DFAS Policy contained in DFAS Information Management Regulation
8000.1-R, Part E, Information System Security Policy, will be used to implement the federal and DoD
requirements contained in the above referenced Circular, Directive, and Instruction.
d. Measures of Effectiveness and Suitability. List the performance (operational effectiveness and
suitability) capabilities and characteristics identified as required in the ORD. Organize them by COI. Limit the
list to critical measures that apply to mission essential capabilities. Include and clearly identify all KPPs .For
each listed required capability or characteristic, provide the minimum acceptable value and the objective from
the ORD. Include all ORD KPPs. Also include all ORD CTPs in Part I, Section e. The critical operational
effectiveness and suitability parameters and constraints will include manpower, personnel, training, software,
computer resources, transportation, compatibility, interoperability and integration, etc. For each listed
parameter, provide the threshold and objective values from the ORD and the ORD reference.
Note: If the Operational Test Agency (OTA) or the Director, Operational Test and Evaluation (DOT&E)
determines that the required capabilities and characteristics contained in the ORD provide insufficient
measures for an adequate operational test and evaluation (OT&E), the OTA or DOT&E shall propose
additional measures through the integrated product team (IPT) process. Upon receipt of such a proposal, the
ORD approval authority shall establish the level of required performance characteristics.
Note: The data that is inserted in this section should be taken from sections 4 and 5 of the ORD.
Note: The following definitions apply to measures presented in the TEMP.
COIs are measures of a system’s essential operational capabilities, risks, or uncertainties. They address the
user-identified critical tasks and areas of operational risk that most significantly affect mission
accomplishment. They are not parameters, thresholds, or objectives. Failure to achieve these would render the
system unresponsive to the user’s needs. COIs are included in the TEMP and are reviewed and updated at each
decision point as part of the TEMP review process. COIs will be phrased as questions that must be answered in
order to properly evaluate operational effectiveness and suitability. COIs are repeated in Part IV of this
Measures of Effectiveness (MOE) and Measures of Suitability (MOS). MOEs and MOSs are defined by the
operating command as measures of operational capabilities in terms of engagement or outcome. Key items of
suitability include availability, maintainability, reliability, usability/personnel, and training. An OT&E MOE
or MOS is a measure of a system’s task accomplishment. They should be developed to a level of specificity so
that a system’s effectiveness or suitability can consistently be evaluated with the set criteria. MOEs and MOSs
will be quantitatively linked to COIs if at all possible. MOEs and MOSs will be included in the TEMP and
updated as part of the review process.
DFAS Baseline COIs include: Mission Performance; Compatibility, Interoperability, and Integration (CII);
Usability; Information Assurance; and Supportability. The COIs and associated MOEs/MOSs are listed below:
(1) CIO - Mission Performance (Effectiveness). Does the system provide the functionality to enable
the users to complete financial, accounting, and budgeting transactions necessary to meet mission
requirements? See Attachment A for DFAS’s Performance Measurements.
• MOE: The system must comply with FFMRs.
• MOE: The system must record, report, and interpret required data to perform the mission.
• MOE: The system must provide for a transaction-driven general ledger to perform the mission.
• MOE: The system must be able to recognize liabilities and establish control over amounts to be
paid (accounts payable).
• MOE: The system must be able to record, account for, and provide for management control of
amounts due from others (revenue and accounts receivable).
• MOE: The system must be able to obligate or disburse funds within the limits of amounts
appropriated or authorized by the Congress, whichever is less.
• MOE: The system must provide management the capability to identify costs and implement
appropriate costing methodologies.
• MOE: The system must be able to transfer data in a timely manner.
• MOE: The system must provide adequate end-to-end response times and notify the user of the
disposition of the action.
• MOE: The system must be able to support defined user requirements during normal and peak
• MOE: The system must provide specified, pre-formatted information (reports) in a timely
• MOE: The system must be able to access information at the required level of detail, in the
required format, in a timely manner.
(2) CIO - Compatibility, Interoperability, and Integration (Effectiveness). Is the system
interoperable and compatible with the existing (non-migratory) accounting and finance systems with which it
must interface? See Attachment A for list of DFAS’s baseline CII performance measures.
• MOE: The system will be operationally standardized.
• MOE: The system will be technically standardized.
• MOE: The system will electronically transmit data successfully to the external interfacing
• MOE: The system will electronically receive data successfully from the external interfacing
• MOE: The system must be able to accept, process, and transmit information in electronic data
interchange (EDI) standard formats.
• MOE: The system must be capable of electronic transmission of payments and payment data.
(3) CIO - Y2K Compliancy (Effectiveness). Is the system hardware and software Year 2000
• MOE: The system hardware (basic input/output system (BIOS) and operating system (OS)) will
be Y2K compliant up to and including the year 2027.
• MOE: The system software will be Y2K compliant up to and including the year 2027 (or
beyond if so designed)
(4) COI - Usability (Suitability). Is the system usable by its intended users in its operational
environment? See Attachment A for list of DFAS’s baseline usability performance measures.
• MOS: The system will perform an automated process to permit single data entry.
• MOS: The system will provide ease of use features such as an intuitive graphical user interface.
• MOS: System training must adequately prepare users to perform system operations.
• MOS: The system user manuals and other documentation shall provide the necessary
information for the user to accomplish the mission.
• MOS: The system shall provide for event reporting (particularly with regard to failed processes)
in both the real-time and batch modes.
• MOS: The system shall provide alternative processing options when the requested process
cannot be performed or is terminated in error.
• MOS: The system shall provide on-line information (help) to assist and enable users to perform
• MOS: The system must provide to the user a centrally accessible hierarchy of professional
assistance (help desk) on a prioritized basis.
(5) COI - Information Assurance (Effectiveness and Security). Does the system protect financial
privacy and user information; provide adequate system protection; and possess the capability to survive
potential threat interference? See Attachment A, Information Assurance, for listing of DFAS Performance
• MOE: The system in its entirety shall meet or exceed the requirements for a system security
level of C2.
• MOE: The system must provide accurate and valid data to perform the mission.
• MOE: The system must be able to identify errors and reject incorrectly entered data.
• MOE: The system shall provide the features necessary to ensure data integrity by limiting and
verifying user access and by limiting data modification capabilities.
• MOE: The system must provide a chronological record of system activities that is sufficient to
enable the reconstruction, reviewing, and examination of the sequence of environment and
activities surrounding or leading to an operation, a procedure, or an event in a transaction from
its inception to final results.
• MOE: The system must have a Continuity of Operations Plan (COOP) established to ensure the
ability to recover from disasters and for the transition from peace to war activities.
• MOE: The system shall provide the features necessary to prevent data loss from deliberate
intrusion and viruses.
• MOE: The system must have the capability to return to a known operational state and rebuild
transactions up to the operational state preceding a restart.
(6) COI - Supportability (Suitability). Does the system operate consistently enough to support user
mission requirements? See Attachment A, Supportability, for appropriate DFAS Performance Measures.
• MOS: The system must be available as specified by the users.
• MOS: The system must be maintained in an operational environment.
• MOS: The system software will attain a high level of maturity.
• MOS: The system hardware, communications, and encryption will be maintainable and
repairable in a timely manner.
Note: Provided in Attachment A to this standard are typical performance parameters that are found in sections
4 and 5 of the ORD. They have been taken from the DFAS Systems Integration Directorate Baseline
Performance Parameter (BPP) database. BPPs have been annotated with a single * or ** to help determine
which are likely KPP candidates and which are likely to be considered CTPs. Parameters marked with a single
* are likely to be KPPs. Those marked with a ** are not likely to be KPPs but are likely to be CTPs. They are
arranged by COI and further break each COI. Use this listing as a starting point in developing program COIs,
MOEs, MOSs, KPPs, and CTPs. CTPS are to be presented immediately below.
e. Critical Technical Parameters. CTPs are technical capabilities or characteristics that outline
minimum baseline telecommunications, hardware, and software requirements. CTPs are broad, generic
statements derived from a review of the MNS, ORD, and other system documentation. CTPs do not replace
lower level technical requirements defined in other system segment specification requirements documentation.
Failure to achieve CTPs would render the system unresponsive to DoD or DFAS needs. A set of core (non-
system specific) CTPs are listed below. Additional CTPs may be added depending on program scope.
(1) List in a matrix format, the CTPs of the system (including software maturity and performance
measures) that have been evaluated or will be evaluated during the remaining phases of developmental testing.
CTPs are derived from the ORD, critical system characteristics and technical performance measures and shall
include the parameters in the Acquisition Program Baseline. Discuss the relationship between the CTPs and
the objectives and thresholds in the ORD. All ORD CTPs must be included in this section.
(2) Next to each technical parameter, list the accompanying objectives and thresholds.
(3) Highlight CTPs that must be demonstrated before entering the next acquisition or operational test
phase and ensure that the actual values that have been demonstrated to date are included in the last column.
(4) Compatibility, interoperability, and integration (CII) issues critical to the operational effectiveness of
suitability of the system must be addressed in the user's effectiveness COI and associated criteria. MOEs and
measures of performance (MOPs) developed for each interoperability COI must be stated. Evaluation criteria
and data requirements for each interoperability-related MOE/MOP must be clearly defined.
• CTP: The system must facilitate the preparation of financial statements and other financial
reports in accordance with federal accounting and reporting standards and applicable statutory
and regulatory requirements.
• CTP: The system must provide and/or collect accurate, complete, reliable and consistent
• CTP: The system must provide for a specified number of concurrent users.
• CTP: The system must provide adequate response times for data transactions.
• CTP: The system must provide for adequate agency management reporting.
• CTP: The system must provide a complete audit trail to facilitate audits and oversight.
• CTP: The system must provide interfaces to transmit and receive data from external data
• CTP: The system must survive identified security risks.
• CTP: The system must be Y2K compliant.
• CTP: The system must be designed to a high degree of reliability, availability, and
An example of a sample of a typical CTP matrix is provided below. Figure x lists the CTPs that will be tested
for Program X. The table also lists the ORD paragraph that is the source of the CTP requirements, the
objective and threshold, and the status of each CTP.
Critical Technical Parameter K ORD Threshold Objective MS Decision
(CTP) P Reference Value Value Supported
Data Currency X Par.184.108.40.206 99% IOCa 100% FOC MS IIIA
99.8% IOC b MS IIIB
Data Accuracy X Par 220.127.116.11 99% IOCa 100% FOC MS IIIA
99.8% FOC MS IIIE
Interoperability X Par. JITC JITC MS IIIA
18.104.22.168 Certified 99% Certified
MSIIIA 100% FOC
Figure x Sample List of CTPs
PART II--INTEGRATED TEST PROGRAM SUMMARY.
a. Introduction. In the introduction to this section, provide a high-level summary of the program's test
strategy from developmental through operational testing including all compliance certifications. Test
events should include unit testing (UT), system integration testing (SIT), functional validation testing
(FVT), enterprise integration testing (EIT), enterprise performance testing, and enterprise acceptance
testing, certification testing (CT), and OT&E. Test events must track to the integrated test schedule
provided below. In the summary, identify the test objectives, test methods, office of primary
responsibility (OPR), test location, test tools, and test documentation planned for each test event.
Provide a table cross-referencing test objectives to test events (see example in Table 1). Further
breakdown of developmental test (DT) and operational test (OT) events and requirements shall be
provided in Parts III and IV.
Table 1. Example - Test Objectives Matrix
Objective UT SIT FVT CT EIT EPT EAT OT&E
Module Logic • •
Internal Interfaces • • •
External Interfaces • • • • •
Functional Requirements • • • • •
End-to-end threads • • • •
FFMR Requirements • • • • •
Off-Nominal & Error Handling • • • • • •
Security Requirements • • • • •
User Interface Requirements • • • • •
Performance Requirements • • • • •
Performance Stress Test •
Operations & Maintenance • • • •
Continuity of Operations • • •
Operational Requirements • • • • •
C4ISR/Interoperability • •
Information Assurance • •
FFMR Certification • •
Joint Technical Architecture (JTA) • •
b. Integrated Test Program Schedule.
(1) Display on a chart the integrated time sequencing of the critical T&E phases and events, related
activities, and planned cumulative funding expenditures by appropriation. An example chart is provided in the
(2) Include event dates such as milestone decision points, operational assessments, test article
availability, software version releases, appropriate phases of developmental test and evaluation (DT&E),
OT&E, Initial Operational Capability (IOC), Full Operational Capability (FOC), and statutorily-required
reports All evolutionary and incremental phases and associated testing, milestones, and IOCs will be
(3) Provide a single master schedule for multi-service or Joint and Capstone TEMPs showing all DoD
component system event dates.
FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05
CY98 CY99 CY00 CY01 CY02 CY03 CY04 CY05
Definition & Risk
Phase II Engineering &
Mfg . Incremental
Development by Phases
(If Applicable) (a) (b) (c)
DT&E (b) (c)
Full Scale Deployment
By Incremental Phases
(0) (I/II) III IOCa IOCb IOCd/e FOC
(a) III III
(b/c) IOCc (d/e)
($M)(Current) (Sunk FY95-98)
DWCF Capital 35.1 18.9 23.6 29.5 39.5 38.5 30.3 26.6
DWCF Operations - 7.2 9.0 12.5 13.3 13.3 13.6 16.7
Total $35.1 $26.1 $32.6 $41.9 $52.8 $51.8 $43.9 $43.3
Source: Program X, Acquisition Program Baseline (APB), dated xx April 2000
(a) User Site 1, Increment Phase 1
(b) User Site 2, Increment Phase 2
(c) User Site 3, Increment Phase 3
(d) User Site 4, Increment Phase 4
(e) User Site 5, Increment Phase 5
Integrated Test Program Schedule
c. Management (Roles and Responsibilities).
(1) Discuss the T&E responsibility of all participating organizations (developers, testers, evaluators,
(2) Provide the date (fiscal quarter) when the decision to proceed beyond low-rate initial production is
planned. (Low-rate initial production quantities required for OT must be identified for approval by
the DOT&E prior to Milestone C (MS C) for acquisition category I programs and other acquisition
category programs designated for Office of the Secretary of Defense test and evaluation oversight.
(3) Provide the proposed or approved performance Exit Criteria to be assessed at the next major
decision point. For a TEMP update, generated by a program breach or significant change, provide
the Acquisition Decision Memorandum-approved Exit Criteria from the current phase’s beginning
milestone decision, or any revised ones generated by the breach or significant change.
PART III--DEVELOPMENTAL TEST AND EVALUATION OUTLINE.
a. DT&E Overview. Explain how DT&E will verify the status of engineering and manufacturing
development progress; verify that design risks have been minimized; and substantiate achievement of contract
technical performance requirements; and be used to certify readiness for dedicated OT. Specifically, identify:
(1) Technology/Subsystem. Any technology/subsystem that has not yet demonstrated its ability to
contribute to system performance and ultimately fulfill mission requirements.
(2) Hardware and Software Design. The degree to which system hardware and software design has
stabilized so as to reduce production decision uncertainties.
b. Future DT&E. Discuss all remaining DT&E that is planned, beginning with the date of the current
TEMP revision and extending through completion of production. Place emphasis on the next phase of testing.
For each phase, include:
(1) Configuration Description. Summarize the functional capabilities of the system's developmental
configuration and how they differ from the production release.
(2) DT&E Objectives. State the test objectives for this phase in terms of CTPs to be confirmed.
Identify any specific technical parameters that the milestone decision authority has designated as exit criteria
and/or directed to be demonstrated in a given phase of testing. Example exit criteria are provided below:
a) Written certification from the Program Manager that the system has successfully completed
DT&E and is ready for OT&E.
b) Published DT&E report
c) Compliance certifications issued for DFAS Corporate Information Infrastructure (DCII),
Defense Information Infrastructure Common Operating Environment (DIICOE), DoD Finance
and Accounting Model (DFAM), and FFMR as applicable.
d) Updated Requirements Traceability Matrix (RTM).
e) Indication of customer acceptance as a result of System/Enterprise Acceptance Test (SAT/EAT).
f) Operational Test Readiness Review (OTRR) checklist items (see Part IV-c-4 for list).
(3) DT&E Events, Scope of Testing, and Basic Scenarios. Summarize the test events, test scenarios,
and the test design concept. Quantify the testing (e.g., number of test hours, test events). List the specific threat
systems, surrogates, countermeasures, component or subsystem testing, and testbeds the use of which are
critical to determine whether DT objectives are achieved. As appropriate, particularly if an agency separate
from the test agency will be doing a significant part of the evaluation, describe the methods of evaluation. List
all models and simulations to be used, explain the rationale for their credible use and provide their source of
verification, validation and accreditation (VV&A). Describe how interoperability and compatibility with other
systems, as applicable, to include unplanned stimuli, will be tested.
(4) Limitations. Discuss the test limitations that may significantly affect the evaluator's ability to draw
conclusions, the impact of these limitations, and resolution approaches.
PART IV--OPERATIONAL TEST AND EVALUATION OUTLINE.
a. OT&E Overview.
(1) Effectiveness. The primary purpose of OT&E is to determine whether systems are operationally
effective and suitable for the intended use by representative users before production or deployment.
(2) Relationships. Discuss and relate program schedule, test management structure, and required
resources to operational requirements, COIs, test objectives, and milestone decision points. Describe how
testing will evaluate the system (operated by typical users) in an environment as operationally realistic as
possible, including threat representative hostile forces and the expected range of natural environmental
b. COIs. List in this section the COIs from Part I that will be tested during OT&E. Refer to Part I for
discussion of the MOEs and MOPs. Define the evaluation criteria and data requirements for each MOE/MOP.
Note: COIs are the operational effectiveness and operational suitability issues (not parameters, objectives or
thresholds) that must be examined in OT&E to evaluate/assess the system's capability to perform its mission.
COI: Mission Performance (Effectiveness). Does the system provide the functionality to
enable the users to complete financial, accounting, and budgeting transactions necessary to meet
COI: Compatibility, Interoperability, and Integration (Effectiveness). Is the system
nteroperable and compatible with the existing (non-migratory) accounting and finance systems
with which it must interface?
COI: Usability (Suitability). Is the system usable by its intended users in its operational
COI: Information Assurance (Effectiveness). Does the system protect financial privacy and
user information; provide adequate system protection; and possess the capability to survive
potential threat interference?
COI: Supportability (Suitability). Does the system operate consistently enough to support user
c. Future OT&E. For each remaining phase of OT&E, separately address the following:
(1) Configuration Description. Identify the system to be tested during each phase, and describe any
differences between the tested system and the system that will be fielded including, where applicable, software
maturity performance and criticality to mission performance and the extent of integration with other systems
with which it must be interoperable or compatible. Characterize the system (e.g., prototype, engineering
development model, production representative or production configuration).
(2) OT&E Objectives. State the test objectives, including the objectives and thresholds and COIs to be
addressed by each phase of OT&E and the milestone decision review(s) supported. OT&E that supports the
beyond low rate initial production decision shall have test objectives that examine all areas of operational
effectiveness and suitability.
(3) OT&E Events, Scope of Testing, and Scenarios. Summarize the scenarios and identify the events
to be conducted, type of resources to be used, the threat simulators and the simulation(s) to be employed, the
type of representative personnel who will operate and maintain the system, the status of the logistic support, the
operational and maintenance documentation that will be used, the environment under which the system is to be
employed and supported during testing, the plans for interoperability and compatibility testing with other
United States/Allied weapon and support systems as applicable, etc. Identify planned sources of information
(e.g., development testing, testing of related systems, modeling, simulation, etc.) that may be used by the
operational test agency to supplement this phase of OT&E. Whenever models and simulations are to be used:
identify the planned models and simulations; explain how they are proposed to be used; and provide the source
and methodology of the verification, validation, and accreditation underlying their credible application for the
proposed use. If OT&E cannot be conducted or completed in this phase of testing and the outcome will be an
operational assessment instead of an evaluation, this shall clearly be stated and the reason(s) explained.
(4) Operational Test Readiness Entrance Criteria. The program will use the following checklist to
determine the system/increment’s readiness for OT&E.
Project/Program Officer confirms: Y-N-N/A Comments
System baseline frozen and configuration
JTA Profile Certification (Final) received
User training, documentation, and logistics
Workarounds for all open priority 3 Test
Discrepancy Reports (TDR) documented and
disseminated to users
Full (or interim) security accreditation
Approved system-level COOP in place and a
Table-top exercise conducted
Y2K compliant system
Developmental Tester confirms:
DT exit criteria satisfied
System interfaces and interoperability
No Open priority 1 or 2 TDRs
Operational Tester confirms:
OT&E plans approved
Testers/data collectors ready to test
Test Site Commanders & Users confirm:
Understanding and support of testing at their
Users and system administrators trained, in
place, and ready
User documentation complete
Training material complete
Site configuration installed and functional
Figure X Sample Operational Test Readiness Review Checklist
(5) Limitations. Discuss the test limitations, including threat realism, resource availability, limited
operational (military, climatic, nuclear, etc.) environments, limited support environment, maturity of tested
system, safety, etc., that may impact the resolution of affected COIs. Indicate the impact of the test limitations
on the ability to resolve COIs and the ability to formulate conclusions regarding operational effectiveness and
operational suitability. Indicate the COIs affected in parenthesis after each limitation.
PART V--TEST AND EVALUATION RESOURCE SUMMARY.
a. Summary of Key T&E Resources. Provide a summary (preferably in a table or matrix format) of all
key T&E resources, both government and contractor, that will be used during the course of the acquisition
program. Specifically, identify the following test resources:
(1) Test Articles. Identify the actual number of and timing requirements for all test articles, including
key support equipment and technical information required for testing in each phase by major type of DT&E
and OT&E. If key subsystems (components, assemblies, subassemblies or software modules) are to be tested
individually, before being tested in the final system configuration, identify each subsystem in the TEMP and the
quantity required. Specifically identify when prototype, engineering development, preproduction, or
production models will be used.
(2) Test Sites and Instrumentation. Identify the specific test facilities to be used for each type of
testing. Compare the requirements for test facilities dictated by the scope and content of planned testing with
existing and programmed test facility capability, and highlight any major shortfalls, such as inability to test
under representative natural conditions. Identify instrumentation that must be acquired specifically to conduct
the planned test program. Describe how environment compliance requirements will be met.
(3) Test Support Equipment. Identify test support equipment that must be acquired specifically to
conduct the test program.
(4) Threat Representation. Identify the type, number, availability, and fidelity requirements for all
representations of the threat to be used in testing. Compare the requirements for threat representations with
available and projected assets and their capabilities. Highlight any major shortfalls. Each representation of
the threat (target, simulator, model, simulation or virtual simulation) shall be subjected to validation
procedures to establish and document a baseline comparison with its associated threat and to determine the
extent of the operational and technical performance differences between the two throughout the life cycle of the
(5) Simulations, Models and Testbeds. For each T&E phase, identify the system simulations to be
used, including computer-driven simulation models and hardware/software-in-the-loop testbeds. Identify the
resources required to validate and certify their credible usage or application before their use.
(6) Special Requirements. Discuss requirements for any significant non-instrumentation capabilities
and resources such as: special data processing/databases, unique mapping/charting/geodesy products, extreme
physical environmental conditions or restricted/special use air/sea/landscapes.
(7) T&E Funding Requirements. Estimate, by fiscal year and appropriation line number (program
element), the funding required to pay direct costs of planned testing. State, by fiscal year, the funding currently
appearing in those lines (program elements). Identify any major shortfalls.
(8) Manpower/Personnel Training. Identify manpower/personnel and training requirements and
limitations that affect T&E execution.
b. Project of Key Resources Demonstration and Validation Testing and Early Operational
Assessment. Project the key resources necessary to accomplish demonstration and validation testing and early
operational assessment. Estimate, to the degree known at Milestone B (MS B), the key resources necessary to
accomplish DT&E and OT&E. As system acquisition progresses, reassess and refine the preliminary test
resource requirements. Reflect in subsequent TEMP updates any changed system concepts, resource
requirements, or updated threat assessment. Discuss any resource shortfalls which introduce significant test
limitations. Provide planned corrective action for each shortfall and/or limitation..
a. Cite in this section all documents referred to in the TEMP.
b. Cite all reports documenting technical, live fire, and OT&E.
Annex B--ACRONYMS. List and define acronyms used in the TEMP.
Annex C--POINTS OF CONTACT. Provide a list of points of contact as illustrated below:
Name Organization Telephone
DTSE&E Action Officer OUSD(A&T)/DTSE&E
DOT&E Action Officer OSD, DOT&E
ATTACHMENT A - Performance Measures.
Additional Attachments. Provide as appropriate.
TEST AND EVALUATION MASTER PLAN
Program Manager/Functional Project Officer Date
Reviewed by (no signature required):
Quality Assurance Manager
Business Line Executive *
Director for Internal Review
J6 C4 Systems Directorate J8 Force Structure, Resources, and
Director for Systems Integration Date
DISA, Joint Interoperability Test Command(JITC) Date
For MAIS programs only:
Director for Information and Technology Date
For MAIS programs only:
OUSD (C) Date
* or delegated Deputy Business Line Executive or Product Line Executive
Attachment A- PERFORMANCE MEASURES
Examples of DFAS relevant performance measures are listed in the following paragraphs. Select those
applicable to the proposed system:
a. MISSION PERFORMANCE. Mission Performance is separated into two distinct categories: Financial
System Performance and Automated Information System (AIS) Performance.
(1) *Financial System Performance. Financial system performance is defined as the overall evaluation
of the extent to which the system meets Federal Financial Management Requirements (FFMR) in accordance
with the Federal Financial Management Improvement Act (FFMIA) of 1996.
Threshold: The _System Name will be certified as compliant with the FFMIA of 1996. Summary
requirements used to measure this compliance can be found in the DFAS "A Guide to Federal Requirements for
Financial Management Systems".
Financial Management System Performance includes the following six major areas listed below. These areas
align with the 15 DFAS Guide chapters, which also contain another nine minor areas, not listed here. The
definitions below are provided to help program managers determine which apply to the AIS under development
and whether performance parameters and associated threshold and objectives should be defined. The
definitions were developed in conjunction with, and approved by, the Systems Integration Assistant Deputy
Director for Accounting Standards and Requirements.
(a) Financial Reporting. Financial reporting is the process of recording, reporting, and
interpreting, in terms of money, an entity's financial transactions, and events with economic consequences for
the entity. Financial reporting represents the culmination of the various processes that initiate, record, classify,
and summarize an agency's financial transactions.
Financial reports should provide timely and useful information to support (1) management's fiduciary role, (2)
budget formulation and execution functions, and (3) fiscal management of program delivery and decision-
making. Financial reporting encompasses internal management and external reports, including annual
financial statements and other financial reports required by legislation and/or regulation.
(b) General Ledger. The general ledger is the highest level of financial control and summarization
of financial transactions and events within an entity. The general ledger is to be transaction-driven in that all
transactions (financial events with economic consequences to the agency) are to be recorded. The general
ledger maintains account balances by fund structure and individual general ledger accounts (both budgetary
and proprietary). The general ledger is required to be the basis for preparing internal and external financial
reports/statements. Subsidiary ledgers at various levels of detail support the general ledger and may be
maintained in the core financial system or in other systems.
(c) Accounts Payable. Accounts payable, as part of the payment management function, represents
amounts owed by an agency for goods and services it has received from contractors and vendors. Accounts
payable are used to recognize liabilities and establish control over amounts to be paid, including dates
payments should be made for Prompt Payment Act purposes.
(d) Revenue and Accounts Receivable. Agencies are to recognize revenue as it is earned and
financing sources (appropriations, etc.) as provided by the Congress/Office of Management and Budget. The
accounts receivable function is required to record, account for, and provide for management control of
amounts due from others as a result of the performance of services, delivery/sale of goods, passage of time (as
related to interest earned), loans made, and other actions.
(e) Funds Control and Budgetary Accounting. Funds control processes and systems are required to
ensure that funds are not obligated or disbursed in excess of amounts appropriated or authorized by the
Congress, whichever is less. Budgetary accounts (4000 series) are established within the general ledger to
account for budgetary resources and use/consumption.
(f) Managerial Cost Accounting and Other Requirements. Cost accounting is the process by which
costs (both direct and indirect) are assigned to cost objects/outputs, including goods and services provided or
activities carried out. Various costing methodologies--such as activity-based costing, job order costing,
process costing, and standard costing--may be used by an entity. Management is responsible for identifying
cost objects/outputs and systematically implementing the appropriate costing methodologies for determining
the full cost of the objects/outputs.
(2) AIS Performance. AIS performance is defined as the ability of the system, within time constraints, to
deliver specified outcomes to a specified number of users satisfactorily. AIS performance consists of the
(a) *Data Transmission. The act of transferring organized information from the originator or
repository to a receiving organization. (Note, for specific AIS, it may be necessary to define this parameter for
various types of data transmissions.)
Threshold: The system must transfer data completely and accurately from the originator or repository
to a receiving organization in __ (time) or less.
Objective: The system must transfer data completely and accurately from the originator or repository
to a receiving organization in __ (time) or less.
(b) *End-to-End Response Time. End-to-end response time is the elapsed time from user input (e.g.,
pressing enter key) to the notification by the system of the disposition of the action (e.g., notification of action
acceptance or rejection). (Note, for specific AIS, it may be necessary to define this parameter for various types
of transactions. Time to complete the processing and provide a product is defined in the "Information (Report)
Threshold: Users must be notified of the disposition of ___ (type) requested actions in __ (time) or
less ___ percent of the time. The disposition of the remaining ___ percent of ___ (type) requested actions must
be finished within ___ (time).
Objective: Users must be notified of the disposition of ___ (type) requested actions in __(time) or
(c) *Capacity Requirements. The data storage, communications and processor capability is
adequate to support defined user requirements during normal workload and peak load periods.
(Note, number of users, user demographics such as where they are located, normal workload time, and peak
workload time are possible dimensions for this parameter.)
Threshold: The data storage, communications and processor capability adequately support at least
___ percent of defined user requirements during normal workload; ___ percent during peak load periods. The
system supports at least ____ simultaneous on-line users under all workload conditions.
Objective: The data storage, communications and processor capability adequately support all
defined user requirements under peak workload conditions. The system supports ____ (all?) simultaneous on-
line users under peak workload conditions.
(d) *Information (Report) Delivery. Information Delivery, access, processing, and presentation of
specified, pre-formatted information (reports), will be accomplished in a timely manner.
(Note, types of reports and associated disposition should be defined and acceptable values for each given here.
These may include standard on-line transactions, ad hoc queries, batch reports, and so forth)
Threshold: _________ (defined type report or query) will be delivered to the user in __(time) or less
from the time of request.
Objective: _________ (defined type report or query) will be delivered to the user in __(time) from the
time of request.
(e) *Data Availability (Access). The ability of the AIS to access information at the required level of
detail, in the required format, within the required timeframe. The system must be able to access specific
categories of data as designed. For example: (1) the current year; (2) data from the previous 4 years; (3) data
over 5 years old. There should be a retrieval value defined for each legitimate category of data.
Threshold: (Tailor to each age category of data as necessary.) At least __ percent of ____ requests
for information available on-line (current) are answered in __ (time) or less.
b. COMPATIBILITY, INTEROPERABILITY, AND INTEGRATION (CII). Compatibility,
Interoperability, and Integration are defined as the utilization of standardized processes and data leading to
the ability of two or more systems or components to exchange information and to use the information that has
been exchanged. The long-term objective of CII is to establish an infrastructure that can accommodate the
widest possible range of missions and operational scenarios by allowing users to enter the infrastructure at
anytime, anyplace, in the execution of any mission. Compatibility. Compatibility is the capability of two or
more items or components of equipment or material to exist or function in the same system or environment
without mutual interference. Interoperability. Interoperability is the ability of systems, units, or forces to
provide services (exchange information) to or accept services from other systems, units, or forces and to use the
services so exchanged to operate effectively together. Integration. Integration is the arrangement of systems in
an architectural environment so that they function together in an efficient and logical way. [There are three
types of DFAS Corporate Information Infrastructure (DCII) applications as defined in the DCII Specification,
version 4.1, 23 February 1999. "Type III represents the DFAS end-state vision. Types I and II are identical
except for one feature. Type II has the ability to exchange standard transactional data, whereas Type I does
not. Otherwise, both Types I and II applications include pre-existing applications…" The project officer must
determine which type DCII application the AIS is before determining threshold and objective values in this
(1) **Operational Architectural Standardization. The establishment of consistent criteria for
information exchange requirements and mission area interactions, tasks, and interoperability.
Type I and II AIS:
Threshold: Type I and II systems will be evaluated against the DoD Finance and Accounting Data
Model (DFADM) and DFAS Finance and Accounting Activities (Process) Model (DFAPM); analysis and
deviations will be documented to support future modification or re-engineering decisions.
Type III AIS:
Threshold: A Type III system will be certified compliant with the DCII Operational Architecture by the
DCII Functional Architect as part of the DCD and DCII Configuration Control Board process.
(2) Technical Architectural Standardization. That portion of architectural standardization that focuses
upon conforming to Federal and DoD technical guidance as it pertains to the design and construction of
finance and accounting AIS. The minimal set of rules governing the arrangement, interaction, and
interdependence of the parts or elements whose purpose is to ensure that a conformant system satisfies a
specified set of requirements. The DCII Specification, Defense Information Infrastructure Common Operating
Environment (DII COE), and Joint Technical Architecture (JTA) outline technical architectural
standardization requirements. NOTE: If the system is to run on the DFAS Enterprise Local Area Network
(ELAN), then it must also be tested and certified by the DFAS ELAN Platform Engineering Team (EPET) and
an additional parameter for this test criteria established. If included, passing the EPET certification will be a
(a) **DCII. The DCII is a collection of common services, tools, procedures, hardware and software
platforms, standards, policies, communication facilities, and other integrated elements that enable DFAS to
provide shared, integrated access to corporate information assets in a compliant, maintainable, interoperable
Threshold: The system is certified as compliant with the appropriate DFAS technical architecture by
the DCII Technical Architecture Manager.
(b) **DII COE. The DII COE provides an approach for building interoperable systems, a reference
implementation containing a collection of reusable software components, a software infrastructure for
supporting mission-area applications, and a set of guidelines, standards, and specifications. The Project Office
must determine if the AIS will be operating in a Common Operating Environment (COE) or a Standard
Operating Environment (SOE). If COE, the Program Manager and the DFAS Technical Architect must
determine applicability along with the level of compliance. When applicable, JTA specifies a minimum of level
5 compliance with COE, so if the level is less, the threshold should mandate the requirement to include the plan
and time to attain level 5. If SOE, the system must operate in compliance as defined in the SOE Architecture
document, 30 April 1998.]
Threshold: System will be level ___ DII COE compliant. (Type III applications are to be level 7
compliant. If not SOE, Types I and II should be level 5 compliant. If these do not reflect initial compliance
parameters, then the plan and time to attain the appropriate level must be documented and available for review
to pass this parameter).
Objective: System will be level ___ DII COE compliant.
Threshold: System will be SOE compliant.
(c) **JTA. The JTA provides DFAS systems with the basis for the needed seamless interoperability,
both internal and external. The JTA outlines adopted commercial technical standards to which DFAS systems
must be designed. [The Project Office in conjunction with Defense Information Systems Agency (DISA), Center
for Standards (CFS) will identify applicable JTA standards. The DISA CFS will issue the JTA Standards
Profile certification for ___________ system. The mandated standards are incorporated into system design
and are verified (Standards Conformance testing) during developmental testing.]
Threshold: The DISA CFS issues a JTA Profile Certification for _________system prior to the
Interim Progress Review (IPR) held during the System Development and Demonstration Phase. The system will
comply with all identified mandated standards.
(d) **Open Systems Environment. An integrated open systems approach shall be applied to the
system's design that emphasizes commercially-supported practices, products, performance specifications, and
Threshold: Must be interoperable with system(s) interfacing with (Name your system), have
established life cycle supportability prior to MS C, and be capable of incremental system upgradability without
major redesign during initial procurement and re-procurement of systems, subsystems, components, spares, and
services, and during post-production support.
(e) **EPET/Development Platform Engineering Team (DPET) Certification. New software
applications must be tested to ensure there are no conflicts with other software programs within the intended
operating environment. For DOS, NOVELL, and Microsoft based applications, EPET in Pensacola conducts
this compliance testing prior to completion of DT&E. If the software is UNIX or Oracle based, the DPET in
Indianapolis conducts the testing.
Threshold: The appropriate team (EPET or DPET) issues a certification during Software Integration
Testing (Type I & II) or Enterprise Integration Testing (Type III) for ________system showing compliance.
(3) * Interoperability. The ability of the systems, units, or forces to provide services to and accept
services from other systems, units, or forces, and to use the services so exchanged to enable them to operate
effectively together. The conditions achieved among communication-electronics systems or items of
communications-electronics equipment when information or services can be exchanged directly and
satisfactorily between them and/or their users.
(a) **Threshold: The system will accurately process defined interfaces with other automated systems
(protocol, media, layout, content).
(b) **Threshold: The system will convert received non-standard data into a usable, standard DFAS
format as defined within data specifications.
(c) **Threshold: The system will be issued a Joint Interoperability Certificate of compliance by the
Joint Interoperability Test Command (JITC) prior to the Full Rate Production (FRP) Decision Review.
(4) Information Exchange Requirements (IERs). IERs characterize the information exchanges to be
performed by the proposed system. For ORDs, IERs are identified as those information exchanges that are
external to the system. IERs identify who exchanges what information with whom, why the information is
necessary, and how the information exchange must occur. The quality (i.e., frequency, timeliness, security) and
quantity (i.e., volume, speed, and type of information) are attributes of the information exchange included in
There is no threshold or objective required as IER is a mandatory section of the ORD (Part 4b). However, the
Program Office must ensure that the IER matrix fully documents and describes each external interface to the
system as specified in CJCSI 6212.01B and CJCSI 3170.01B. The IER compliance is tested as part of the
interoperability KPP in terms of interfaces being tested.
(5) **Electronic Commerce/Electronic Data Interchange (EC/EDI). The concept of conducting
business in an electronic environment in order to reduce the dependence on paper copies and files, streamline
business processes, and reduce data errors and transaction costs. EC/EDI encompasses such areas as Web
Invoicing (WInS), Wide Area WorkFlow (WAWF), Electronic Document Access (EDA), Electronic Document
Management (EDM), and Electronic Funds Transfer (EFT). The two distinct EC/EDI standards families that
apply to DFAS are ASC ANSI X12, which is nationally recognized, and UN/EDIFACT, which is observed and
used internationally. The format that any given system uses depends upon Electronic Commerce requirements
identified and documented for that system. [EC/EDI compliance is validated as part of the JTA standards
conformance certification process, as well as assessed as part of the Interoperability certification process.]
Threshold: EC/EDI compliance will be satisfied by the issuance of a JTA standards conformance
certification and joint interoperability certification.
c. USABILITY. Usability is the establishment of the basis for effective integration of human factors
engineering and the human computer interface into the acquisition of defense systems. The evaluation of
Usability will consider the following areas:
(1) **Single Data Entry. The system requires that once discrete financial and accounting data passes
edit checks and is entered into the system database; no further entry of the same data is required to perform an
automated process. This is system specific. One way to measure would be to trace a sample of data through
the system and to examine a sample of data entry screens. If the system prompts for entry of information
already identifiable as resident, single data entry is not achieved. Percentage for the threshold value could
thus be computed. The Requirements IPT (RIPT) must determine the best way to trace and measure single data
entry for the given AIS.
Threshold: Single data entry must be achieved ___ percent of the time.
(2) Human/Computer Interface. Human/Computer interface includes the interactions between the user
and the system. Included in this interaction are use of controls, computer displays, and environmental
concerns such as noise, lighting, workspace layout, procedures, and documentation. The system will comply
with applicable Human Computer Integration requirements as specified in the JTA. There are two dimensions
here. One, does the AIS comply with applicable graphical user interface (GUI) design guidelines? For major
modifications, it may not be cost effective to totally redesign the system user interface to meet such guidelines.
They have been developed to make the system easy for the user to understand and use; so another dimension is
how easily the user interacts with the system through the user interface. The RIPT should determine whether
the first dimension applies. The RIPT should determine how the second should be measured.
Threshold: GUI will comply with the DFAS GUI guideline.
Threshold: The _______ system will be judged easy to understand and use, based on human/computer
interaction, by ________% of users as determined by the independent tester based upon user feedback.
Threshold (for web-based applications only): The _______ system will comply with applicable
standards as outlined in the Americans with Disabilities Act, section 508.
(3) Training. Training must ensure users are certified as capable of operating and using the DFAS
systems. Training programs will be structured and organized to effectively support user needs and updated as
system changes occur. Through the use of initial and sustainment training, users and maintainers will be
certified to operate and maintain the system.
All training must be tailored to the appropriate level of expertise demonstrated by users at a given site. Course
and reference materials must be available to individual trainees.
Threshold: ____% of surveyed training graduates, users and maintainers, indicate training was
adequate or better to use the system. ____% of the users rate the course and reference materials as complete
Threshold: ____% of training graduates, users and maintainers, continue to indicate training was
adequate or better to use the system after using the system for _______days/weeks/months.
(4) Event Reporting. The act of generating situational notification, particularly with regard to failed
processes. The evaluation of event reporting will consider the following areas: The system needs to notify the
user or operator of the status of processing. The examples given here may not be appropriate for all AIS. The
RIPT should consider what processes or failures, within what time constraints, and with what level of accuracy
such notifications should occur and devise appropriate parameters.
(a) Job Failure Notification. Informing the user that something is incorrect or did not complete
Threshold: __ percent of notifications received from the system when a process terminates
abnormally are rated by the user as accurate and informative. At least ___ percent of job failure notifications
are generated and delivered within ___seconds of incurring a triggering error.
(b) Batch Feedback Message. Informative, written communications from the system to the user
detailing the outcome of off line processing.
Threshold: __ percent of batch feedback messages received from the system are rated by the user as
accurate and informative and in sufficient detail to satisfy the user in regards to understanding the disposition
of the process requested. At least __ percent of batch feedback messages are generated and delivered within
___ seconds of the termination of a batch process
(c) Exception Processing. Alternative processing performed as a decision aid when the requested
process cannot be performed or is terminated in error.
Threshold: ___percent of the exception processes are executed in ____ (time); the remaining
exception processes are executed in _____time. ___percent of the exception processes are rated by the user as
timely and informative and in sufficient detail to allow the formulation of a plan to redress the problem that
necessitated the exception process.
(5) On-line Information (Help). A collection of data, facts, news, advice or tips which serves as a
repository of general help which is quickly available to users having current access to the system. On-line
information is provided by the system to enable users to perform their duties, to include the successful
resolution of minor problems.
Threshold: ____% of the users rate the usefulness, completeness, and accuracy of on-line information as
adequate or better.
(6) Help Desk. The Help Desk is a centrally accessible hierarchy of professional assistance available to
the users on a prioritized basis. It must provide the capability to obtain context sensitive help within a
reasonable amount of time.
Threshold: ____% of the users surveyed rate the utility of the help desk as adequate or better.
d. INFORMATION ASSURANCE. Information Assurance includes having safeguards, policies, and
procedures established to ensure the required information is entered, processed and stored in a recoverable
fashion and that only authorized personnel will have access to it. The evaluation of Information Assurance will
consider the following areas:
(1) **AIS Security Accreditation. Security accreditation is a formal declaration by a Designated
Approving Authority that an AIS is approved to operate in a particular security mode using a prescribed set of
safeguards. The accreditation must be accomplished in accordance with the DoD Instruction 5200.40, "DoD
Information Technology Security Certification and Accreditation Process (DITSCAP)."
Threshold: Acquire Interim Authority to Operate (IATO) prior to MS C.
Objective: Acquire full certification and accreditation prior to FOC.
(2) Data Accuracy/Validity. Ensuring information is correct and complete by: (a) Maintaining the
mathematical and textual accuracy of data within the system and (b) Providing the ability to identify missing or
erroneous data and notifying the user to take corrective action or to implement a procedure to correct. The
evaluation of Data Accuracy/Validity will consider the following areas:
(a) **Data Integrity. In information processing, data integrity is the ability to safeguard the
condition of data to ensure accuracy, currency, consistency, and completeness.
Threshold: ____% of data errors are identified and corrected through reconciliations and edits
which maintain data integrity. The remaining percent are identified and can be corrected manually to maintain
(b) **Identification and Notification of Erroneous or Missing Information. This identification and
notification involves the process of evaluating data for accuracy by the application of validation rules and
alerting the proper individual or agency of the perceived problem and suggested remedy.
Threshold: ____% of the time the system generates and distributes informational messages
identifying problems and suggesting remedies.
(3) **Audit Trail. A chronological record of system activities that is sufficient to enable the
reconstruction, reviewing, and examination of the sequence of environment and activities surrounding or
leading to an operation, a procedure, or an event in a transaction from its inception to final results. A
qualitative assessment will be performed by the independent tester, based on input provided by the users, to
measure the sufficiency of the audit data.
Threshold: ____% of the transactions evaluated provide sufficient audit trail data to the users.
Objective: All of the transactions evaluated provide sufficient audit trail data to the users.
(4) Continuity of Operations Program (COOP) Plan. Written policies and procedures to ensure
continuity of operations without significant loss of operational capability. Such policies and procedures
include actions taken to safeguard personnel, protect DFAS assets, manage risk, avert or lessen the impact on
operations of an adverse situation, provide for timely resumption and continuation of operations, and maintain
adequate service levels for DFAS customers.
Threshold: Plans are certified and a table-top exercise completed prior to MS C.
Objective: Plans are certified and a full COOP test scheduled with DFAS Information and Technology
Directorate prior to Final Operational Capability (FOC).
(5) **System Restart (Recovery and Restoration). The ability to return to a known operational state and
rebuild transactions up to the operational state preceding a restart. (Recovery and restoration must be
available following an operating system, application, database, power, or communication failure.)
Catastrophic failures that physically prevent recovery or restoration are excluded.
Threshold: The system must be operational within ____hours of the failure.
The system will automatically return to a known operational state and recover transactions up to the
operational state preceding a restart within _____hours.
The system provides audit information to identify unrecoverable transactions requiring manual
Objective: The system will immediately return to a known operational state and rebuild transactions up
to the operational state preceding a restart.
e. SUPPORTABILITY. Supportability is the probability that the system will be available to begin a
mission, will not fail during a mission, and can be maintained with available maintenance support. System
Reliability will not be addressed as it more accurately applies to hardware components of the system as
opposed to the AIS. The evaluation of Supportability will consider the following areas:
(1) **Availability. A measure of the degree to which an item is in an operable state, at the start of a
mission, and when the mission is called for at a random point in time. A distinction between peak and non-
peak usage periods must be made and is system-unique, depending upon functionality and design. Generally
speaking, peak periods encompass end-of-month and end-of-year processing, cyclic reporting, and other tasks
performed during high volume periods.
Threshold: The system will be available___ percent of the time during peak usage periods.
The system will be available ___ percent of the time during non-peak usage periods.
Objective: The system will be available___ percent of the time during peak usage periods.
The system will be available ___ percent of the time during non-peak usage periods.
(2) Maintainability. (a) A characteristic of design and installation, expressed as the probability that an
item will be retained in, or restored to, a specified condition within the given period if prescribed procedures
are used. (b) The amount of maintenance support required per operating hour to restore the system to an
operational mode or to improve the system in accordance with evolving requirements. Further, maintainability
will be considered as two types, software and system maintainability.
(a) **Software Maintainability. The ability to quickly, efficiently and correctly adjust to changing
system requirements. One way to measure this is by the level of the system in the Capability Maturity Model
(CMM), as assessed by the Software Engineering Institute (SEI). This may not be an appropriate measure for
all AIS. If not, the RIPT should discuss and determine other measures by which software maintainability can
Threshold: The __________system will be certified SEI CMM Level 2 prior to MS C.
Objective: The system will be certified SEI CMM Level III prior to MS C.
(b) **System Maintainability. The component of maintainability that is concerned with all system
aspects including software, hardware, communications, and encryption. Dimensions of this parameter may
include types of failure. The RIPT may also want to consider defining allowable scheduled repair or upgrade
down times and remedial repair down times.
Threshold: Mean Time To Repair (MTTR). The MTTR will be __(time) or less for a __type of repair.
f. **YEAR 2000 (Y2K). The ability of information technology to successfully process data containing
dates before, on, and after January 1, 2000, separately or together.
Note: There is no threshold or objective provided as Y2K certification is no longer required. However, the
program office should still address compliance and implement tests as outlined in the DFAS Y2K Management
Plan and in the current version of the JTA for interfaces and standard date formats.