Defense Acquisitions: Factors Affecting Outcomes of Advanced
Concept Technology Demonstrations (02-DEC-02, GAO-03-52).
The Advanced Concept Technology Demonstration (ACTD) program was
started by the Department of Defense (DOD) as a way to get new
technologies that meet critical military needs into the hands of
users faster and less cost. GAO was asked to examine DOD's
process for structuring and executing ACTDs.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-03-52
ACCNO: A05644
TITLE: Defense Acquisitions: Factors Affecting Outcomes of
Advanced Concept Technology Demonstrations
DATE: 12/02/2002
SUBJECT: Military operations
Military research and development
Operational testing
DOD Advanced Concept Technology
Demonstration Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-03-52
Report to the Subcommittee on Readiness and Management Support, Committee
on Armed Services, U. S. Senate
United States General Accounting Office
GAO
December 2002 DEFENSE ACQUISITIONS
Factors Affecting Outcomes of Advanced Concept Technology Demonstrations
GAO- 03- 52
Since the ACTD program was started in 1994, a wide range of products have
been tested by technology experts and military operators in realistic
settings* from unmanned aerial vehicles, to friend- or- foe detection
systems, to biological agent detection systems, to advanced simulation
technology designed to enhance joint training. Many of these have
successfully delivered new technologies to users. In fact, 21 of 24
projects we examined that were found to have military utility delivered at
least some technologies to users that meet military needs.
Though the majority of the projects we examined transitioned technologies
to users, there are factors that hamper the ACTD process. For example:
Technology has been too immature to be tested in a realistic setting,
leading to cancellation of the demonstration. Military services and
defense agencies have been reluctant to fund
acquisition of ACTD- proven technologies, especially those focusing on
joint requirements, because of competing priorities. ACTDs* military
utility may not have been assessed consistently.
Some of the barriers we identified can be addressed through efforts DOD
now has underway, including an evaluation of how the ACTD process can be
improved; adoption of criteria to be used to ensure technology is
sufficiently mature; and placing of more attention on the end phase of the
ACTD process. Other barriers, however, will be much more difficult to
address in view of cultural resistance to joint initiatives and the
requirements of DOD*s planning and funding process.
Advanced Concept Technology Demonstrations Projects DEFENSE ACQUISITIONS
Factors Affecting Outcomes of Advanced Concept Technology Demonstrations
www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 52. To view the full report,
including the scope and methodology, click on the link above. For more
information, contact Katherine Schinasi at (202) 512- 4841 or schinasik@
gao. gov. Highlights of GAO- 03- 52, a report to the
Subcommittee on Readiness and Management Support, Committee on Armed
Services, U. S. Senate
December 2002
The Advanced Concept Technology Demonstration (ACTD) program was started
by the Department of Defense (DOD) as a way to get new technologies that
meet critical military needs into the hands of users faster and for less
cost. GAO was asked to examine DOD*s process for structuring and executing
ACTDs.
We are recommending that DOD strengthen its criteria for assessing the
military utility of ACTD projects; consider ways to ensure funding is
provided for acquisitions; and have the Secretary weigh in on decisions on
whether to transition technologies that are tested under the program.
DOD generally concurred with the recommendations on improving military
utility assessments and ensuring timely funding for the transition of
successful ACTD projects. DOD partially concurred with our recommendation
on obtaining high- level concurrence on any decision not to transition
ACTD projects addressing joint requirements.
Page i GAO- 03- 52 Defense Acquisitions Letter 1
Results in Brief 2 Background 2 Twenty- one of 24 Projects Transitioned at
Least Some
Technologies to Users 5 Some Factors Can Hamper the ACTD Process 9
Initiatives Are Underway to Improve ACTD Outcomes 17 Conclusions 19
Recommendations for Executive Action 19 Agency Comments and Our Evaluation
20 Scope and Methodology 20
Appendix I Technology Readiness Levels and Their Definitions 23
Appendix II Comments from the Department of Defense 24
Table
Table 1: Summary of Outcomes 8
Figures
Figure 1: ACTD Process 4 Figure 2: Technologies Tested in Military
Operations in Urban
Terrain ACTD 7 Figure 3: Illustration of Factors Influencing Outcomes 10
Abbreviations
ACTD Advanced Concept Technology Demonstration DOD Department of Defense
GCCS Global Command and Control System OSD Office of the Secretary of
Defense TRL Technology Readiness Level Contents
Page 1 GAO- 03- 52 Defense Acquisitions
December 2, 2002 The Honorable Daniel Akaka Chairman The Honorable James
Inhofe Ranking Minority Member Subcommittee on Readiness and Management
Support Committee on Armed Services United States Senate
The Advanced Concept Technology Demonstration (ACTD) program was initiated
by the Department of Defense (DOD) in 1994 as a way to get new
technologies that meet critical military needs into the hands of users
faster and at less cost than the traditional acquisition process. Under
its traditional process, which takes an average of 10 to 15 years to
develop a product, DOD explores various weapon concepts, defines what the
specific weapon system will look like, refines plans through systems
development and demonstration, and then produces the equipment in larger-
scale quantities. By contrast, under the ACTD process, which takes an
average of 2 to 6 years, military operators and developers test
prototypes, which have already been developed and matured, in realistic
settings. If they find these items to have military utility, DOD may
choose to buy additional quantities or just use items remaining after the
demonstration. If users find these items do not have utility, DOD may
reject them altogether* an outcome that enables DOD to save time and
money.
A key distinction between the traditional acquisition process and the ACTD
process is that the ACTD process is intentionally set up to be much more
flexible and streamlined. Decisions to move from stage- to- stage are less
formal and the process itself is managed by a set of guidelines, which
contain advice and suggestions, as opposed to formal directives and
regulations. This was done to encourage innovation and creativity as well
as participation from the services and the defense agencies on projects
that have joint applications.
You requested that we examine DOD*s process for structuring and executing
ACTDs, particularly with respect to DOD*s ability to transition promising
technologies to military users. In doing so, we reviewed 24 of the 99
projects that have been undertaken so far. Of the 24 projects reviewed, 21
had transitioned at least some technologies found to have military utility
to users as acquisition programs, residual items, or both.
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 03- 52 Defense Acquisitions
Among these were the Predator and Global Hawk unmanned aerial vehicles,
devices to combat weapons of mass destruction, weapons and equipment for
use in urban combat, and various information systems tools and decision
aides.
Though the majority of the projects we examined had transitioned
technologies to users, we found that there are opportunities for DOD to
improve the ACTD process. These include (1) ensuring candidate
technologies are mature enough to be tested in a realistic setting,
military services and defense agencies sustain their commitment to
projects, especially those focusing on joint requirements, and appropriate
expertise is employed for carrying out demonstrations and transitions; and
(2) developing specific criteria to evaluate demonstration results. Such
actions would enable the ACTD process to produce better candidates and
help DOD to prevent delays and funding gaps.
DOD recognizes that the ACTD process could be improved. In response, it
has adopted criteria that should help ensure technologies are sufficiently
mature for the demonstrations. It is strengthening guidance so that
projects can be planned and managed better. To maximize outcomes, DOD
still needs to strengthen assessments of military utility and ensure that
projects are adequately funded through the transition. We are making
recommendations to DOD to address both issues.
In commenting on a draft of this report, DOD generally concurred with our
recommendations on improving military utility assessments and on ensuring
timely funding for the transition of successful ACTD projects. DOD
partially concurred with our recommendation on obtaining high- level
concurrence on any decision not to transition ACTD projects addressing
joint requirements.
The ACTD process is intended to be much more flexible and streamlined than
DOD*s formal acquisition process and in turn to save time and money. Under
the ACTD program, prototypes are developed and provide users with the
opportunity to demonstrate and assess the prototypes* capabilities in
realistic operational scenarios. From these demonstrations, users can
refine operational requirements, develop an initial concept of operations,
and determine the military utility of the technology before deciding
whether additional units should be purchased. Not all projects are
selected for transition into the normal acquisition process. Specifically,
potential users can conclude that the technology (1) does not have Results
in Brief
Background
Page 3 GAO- 03- 52 Defense Acquisitions
sufficient military utility and that acquisition is not warranted or (2)
has sufficient utility but that only the residual assets of the
demonstration are needed and no additional procurement is necessary.
Separate technologies within one project may even have varied outcomes.
DOD*s traditional approach to developing and buying weapons* which takes
an average of 10 to 15 years* is marked by four phases: exploring various
weapon concepts, defining what the specific weapon system will look like,
refining plans through systems development and demonstration, and
producing the equipment in larger- scale quantities and operating and
supporting it in the field. Before a program can proceed to each phase,
defense officials review its progress to evaluate the ability to meet
performance goals and whether risk is under control.
The ACTD process is marked by three phases: selection of the projects,
demonstration of the technologies, and residual use of prototypes and/ or
the transition of them to acquisition programs if the services or defense
agencies decide to acquire more. The selection process begins via a data
call to both the research and development and warfighting communities. The
*Breakfast Club,* a panel of technology experts from various
organizations, reviews the potential candidates. Candidates selected by
this panel are submitted to the Joint Requirements Oversight Council for
prioritization and then to the Under Secretary of Defense for Acquisition,
Technology and Logistics for a final selection. Decisions to move from
stage- to- stage, are less formal than the formal acquisition process, and
the process is managed by a set of Office of the Secretary of Defense
(OSD) guidelines, which contain advice and suggestions, as opposed to
formal directives and regulations. While ACTD teams are to prepare
management plans for the projects that spell out roles and
responsibilities, objectives, and approaches, these plans are supposed to
be flexible, short (less than 25 pages), and high level. Figure 1
illustrates the major phases of the ACTD process.
Page 4 GAO- 03- 52 Defense Acquisitions
Figure 1: ACTD Process
a This phase had been shortened for fiscal year 2003 and 2004 candidates.
The ACTD demonstration phase typically lasts an average of 2 to 4 years,
with an added 2- year residual phase. According to OSD, this provides
ample time to develop fieldable prototypes and to allow users to evaluate
them. For less complex systems or systems that are available quickly (e.
g., commercial- off- the- shelf systems), the time line may be
significantly shorter. Similarly, for very complex systems that require
extensive integration and developmental testing, more time may be
required. A key to keeping the time frame short, according to DOD, is
beginning the demonstration with mature technology. This prevents delays
associated with additional development and rework.
Page 5 GAO- 03- 52 Defense Acquisitions
The ACTD process places the highest priority on addressing joint military
needs, although some ACTDs focus on service specific capabilities. For
example, DOD has found that combat identification systems across the
services needed to be enhanced to reduce fratricide so that systems
belonging to individual services and components, and even allies, could
work together more effectively. As a result, it undertook an ACTD project
that tested new technology designed to improve the capability of combat
forces to positively identify hostile, friendly, and neutral platforms
during air- to- surface and surface- to- surface operations. Another ACTD
project was designed to demonstrate the capability to conduct joint
amphibious mine countermeasure operations. Recently, some ACTD programs
have focused on enhancing homeland security with domestic agencies. For
example, DOD is now testing a command and control system that will allow
emergency personnel first responding to the scene of an attack to talk to
each other and have a better situational awareness.
ACTDs are funded by a variety of sources, including the office within OSD
with the oversight responsibility for the ACTD program and the military
services or defense agencies responsible for conducting the demonstrations
and/ or the transitions. In fiscal year 2001, a total of $546 million was
budgeted for ACTDs*$ 120 million from OSD and $426 million from the
services and agency partners. Participating combatant commands provide
additional resources through their support of training, military
exercises, and other resources. Funding to acquire and maintain additional
units comes from service and agency budgets.
Of the 24 projects we reviewed, 21 transitioned at least some technologies
to users, meaning that users found that these had some level of military
utility and that a military service or a defense agency chose to accept
and fund their transition in the form of residual assets or as an
acquisition.
For 13 of these projects, the services or agencies decided to acquire
more of the items tested, and as a result, transitioned the items into
formal acquisition programs. Two of the 13 had no residual assets in use.
For 8 projects, the services/ agencies decided not to acquire additional
items, but to continue using the residual assets. Three projects had no
residual assets and no acquisition planned.
However, some of these projects experienced mixed outcomes* e. g., some
technologies may have ended up in residual use while others were acquired
or rejected altogether or the lead military service may have Twenty- one
of 24
Projects Transitioned at Least Some Technologies to Users
Page 6 GAO- 03- 52 Defense Acquisitions
rejected the technology while other components decided to acquire it. For
example:
The Counterproliferation I project consisted of a variety of
technologies, including sensors, targeting systems, and advanced weapons,
designed to find and destroy nuclear, biological, and chemical facilities.
The technologies were used in military operations in Kosovo. For example,
an improved infrared sensor that can assess bomb damage to facilities was
accepted by the Air Force as an upgrade to its standard targeting pod. Two
other technologies* a hard targetpenetrating bomb and a fuzing 1 system*
have transitioned to production and are expected to achieve initial
operational capability in fiscal year 2003. However, the project*s weapon
borne sensor technology did not prove to be mature enough and was dropped
from the ACTD prior to any demonstrations.
The Link- 16 project demonstrated an interoperability between the Link16
communications link and other variable message format systems to improve
situational awareness, interdiction, surveillance, and close air support.
No service has adopted it for formal acquisition, but some regional
combatant commanders and lower- level commands have purchased additional
systems. Since the system was not adopted across DOD, its utility could
not be optimized.
The Military Operations in Urban Terrain project field- tested 128 items
designed to enhance operations in urban environments* such as attacking
and clearing buildings of enemy troops. Of these, 32 technologies were
determined to have merit and were kept as residual items to be further
evaluated. Some of these have already transitioned or are planned for
transition to acquisition programs, including a doorbreaching round, a
man- portable unmanned aerial vehicle, elbow and kneepads, explosive
cutting tape, ladders, body armor, and flexible restraining devices.
1 These systems typically recognize or detect targets, initiate
detonation, and determine the direction of detonation.
Page 7 GAO- 03- 52 Defense Acquisitions
Figure 2: Technologies Tested in Military Operations in Urban Terrain ACTD
Notes: SOF Personal Equipment Advanced Requirements (SPEAR), Unmanned
Aerial Vehicle (UAV).
Page 8 GAO- 03- 52 Defense Acquisitions
Table 1: Summary of Outcomes Moved into acquisition (in whole or in part)
Used residuals No residual or acquisition Battlefield Awareness and Data
Dissemination
Technologies to enhance sharing of intelligence and other data
Adaptive Course of Action
Technologies to facilitate crisis planning (e. g., enabling simultaneous
viewing of battle plans as they develop)
Consequence Management
Technologies to detect and model biological warfare agents.
Unattended Ground Sensors
Sensors to enhance capabilities to detect, locate, identify, and report
timecritical targets
Common Spectral Measurement and Signature Exploitation
Technologies to show tactical utility of measurement and signature
intelligence
Joint Modular Lighter
Modular causeway system
Counterproliferation I
Technologies to help detect and respond to nuclear, biological, and
chemical threats
Information Assurance: Automated Intrusion Detection Environment
Technologies to assess attacks on computer networks
Miniature Air Launched Decoy
Small air- launched decoy system to suppress enemy air defense systems
Small Unit Logistics
Software for logistics mission planning
Joint Logistics
Software to support logistics planning
Human Intelligence and Counterintelligence Support Tools
Off- the- shelf technology to support intelligence operations
Precision/ Rapid Counter Multiple Rocket Launcher
Technologies designed to facilitate strikes against North Korean long-
range artillery
Joint Countermine a
Technologies to facilitate amphibious mine countermeasure operations
Navigation Warfare
Jamming , antijamming, and other electronic technologies
Military Operations in Urban Terrain
Technologies to assist operations in urban environments
Personnel Recovery Mission Software
Software to facilitate personnel recovery operations
Predator
Medium altitude endurance unmanned aerial vehicle
Link 16
Software to facilitate sharing of tactical information across military
services
Portal Shield
Technologies to detect and identify biological attacks on air bases or
ports
Rapid Force Projection Initiative a
Long- range precision sensors, weapon systems, munitions, and digital
communications systems designed to defeat an enemy armored force
Global Hawk
High- altitude, long endurance unmanned aerial vehicle
Synthetic Theater of War
Simulation technologies to support joint training and mission rehearsals
Combat Identification a Technologies to identify friendly and hostile
forces
a One of three projects that did not also have residual assets in use.
Source: GAO*s analysis.
Page 9 GAO- 03- 52 Defense Acquisitions
Though the majority of the projects we examined transitioned technologies
to users, we identified a range of factors that hampered this process.
Specifically:
The technology has been too immature to be tested in a realistic
setting, leading to possible cancellation of the demonstration. The
military services and defense agencies have been reluctant to fund
acquisition of ACTD- proven technologies, especially those focusing on
joint requirements, because of competing priorities. Appropriate
expertise has not been employed for demonstrations and
transitions. Transition for software projects has not been adequately
planned. DOD lacks specific criteria to evaluate demonstration results,
which
may cause acquisition decisions to be based on too little knowledge. At
times, top- level support can overcome these barriers. But more systemic
improvements focused on transition planning and funding commitment could
reduce the need for high- level intervention. Figure 3 highlights the
specific factors we identified. Some Factors Can
Hamper the ACTD Process
Page 10 GAO- 03- 52 Defense Acquisitions
Figure 3: Illustration of Factors Influencing Outcomes
Because ACTDs are often conducted during large- scale, force- on- force
military exercises, any new systems being tested must be dependable, able
to perform as intended, and available on schedule in order not to
negatively affect the exercises. As such, DOD has stressed that new
technologies proposed for ACTDs should be *mature,* that is, they should
have already been demonstrated to perform successfully at the subsystem or
component level.
The technology of the ACTDs in our sample was not always mature. In some
cases, problems were fairly basic, such as a technology having inadequate
power supply or being too heavy and bulky to carry out its intended
operation. In other cases, technologies had not reached a point where they
could be tested in a realistic setting, forcing users to forego certain
parts of a test. For example:
The Joint Countermine project tested 15 technologies, including
detection systems and clearance/ breaching systems. During demonstration,
users found that detection technologies had unacceptably high false alarm
rates and a mine and heavy obstacle clearing device was simply too heavy,
bulky, slow and difficult to Technology Maturity
Page 11 GAO- 03- 52 Defense Acquisitions
operate remotely. Moreover, several systems could not be demonstrated on
their intended platforms, or even associated with a suitable substitute
platform. Further, a number of critical operational sequences, such as
launch/ recovery, ordnance handling, and system reconfiguration, had not
been demonstrated. As a result, only some technologies in this project
have transitioned.
The Consequence Management project examined 15 technologies designed to
identify and respond to a biological warfare threat. During demonstration,
users found that some of the items used to collect samples failed to
operate and did not have sufficient battery capability and that switches
broke. None of the other technologies performed flawlessly, and
limitations such as size and weight made it apparent that they were not
field ready. None of the technologies from this project entered into the
acquisition process, nor did DOD continue to use any of the residual
assets. 2
Technologies supporting the Joint Modular Lighter System, a project
testing a modular causeway system, failed during the demonstration because
they had not been properly designed to withstand real world sea
conditions. Consequently, the ACTD was concluded without a demonstration.
The Navigation Warfare project, which focused on validating technologies
for electronic warfare countermeasures, was terminated after DOD found
that some of the technologies for the project could not be demonstrated.
Some of the jamming technologies associated with this project are still
being evaluated.
The technical maturity of software is also vital to successful
demonstrations. If software is not able to work as intended, a project*s
demonstration may be limited as a consequence. For this reason, one ACTD
operations manager stressed that software technologies should be as mature
as possible at the start of the ACTD. One ACTD included in our review
experienced problems with software immaturity going into demonstration.
Because software technologies in the Battlefield
2 However, this ACTD did produce a published concept of operations for
both units involved in the demonstrations, the Technical Escort Unit and
the Chemical- Biological Incident Response Force. In addition, this ACTD
provided the first opportunity for these units to work together and
demonstrated the ability of DOD units to integrate with other federal,
state, and local agencies.
Page 12 GAO- 03- 52 Defense Acquisitions
Awareness and Data Dissemination ACTD were not mature, certain planned
exercises could not be concluded.
Before fiscal year 2002, OSD*s guidance only generally described the
expectations for technology maturity and OSD did not use a consistent,
knowledge- based method for measuring technology maturity of either
hardware or software technologies. Specifically, OSD officials selecting
the ACTDs used simple ranking schemes to capture the degree of technical
risk after consulting with subject area experts. The results of these
efforts were not usually documented. Studies conducted by the
Congressional Budget Office in 1998 and DOD*s Inspector General in 1997
also found that without guidelines on how to assess maturity, DOD
officials defined mature technology in widely contrasting ways.
In the last year, OSD has changed its guidance to address this problem.
Specifically, it now requires technology maturity to be assessed using the
same criteria* technology readiness levels (TRLs)* that DOD uses to assess
technical risk in its formal acquisition programs. 3 This change is
discussed in more detail later in this report.
Although OSD provides start- up funding for ACTDs, the military services
and defense agencies are ultimately responsible for financing the
acquisition and support of equipment or other items that may result from
an ACTD. At times, however, the military services did not want to fund the
transition process. This action either slowed down the acquisition process
or resulted in no additional procurements. Projects that were particularly
affected by this reluctance included those that tested unmanned aerial
vehicles and software applications for enhancing the performance of a
system to defeat enemy artillery. In other cases, DOD leaders stepped in
to support the projects since there was a strong need for the technology
and/ or an extremely successful demonstration.
For example: The Predator is a medium- altitude unmanned aerial vehicle
used for
reconnaissance that progressed from a concept to a three- system 3 See U.
S. General Accounting Office, Best Practices: Better Management of
Technology Development Can Improve Weapon System Outcomes, GAO/ NSIAD- 99-
162 (Washington, D. C.: July 30, 1999). Sustaining Commitment
Page 13 GAO- 03- 52 Defense Acquisitions
operational capability in less than 30 months. The Predator ACTD was
initiated in 1995. Since then, the Predator has been deployed in a range
of military operations, most recently in the war in Afghanistan. Twelve
systems, each containing four air vehicles, are being procured. The Air
Force was designated as the lead service for the ACTD, even though it had
shown no interest in this or other unmanned aerial vehicle programs. A
transition manager was never assigned to this project. The Defense
Airborne Reconnaissance Office was also reluctant to field and support the
system beyond the test- bed phase. Further, at one point, the project
almost ran out of funds before its end. Nevertheless, the Joint Staff
directed the Air Force to accept the system from the Army and the Navy,
which had acted as co- lead services throughout the demonstration phase.
The Global Hawk is a high- altitude unmanned aerial vehicle designed for
broad- area and long- endurance reconnaissance and intelligence missions.
It has also been successfully used in recent military missions. The Air
Force was also reluctant to fund this program. Nevertheless, eventually
the Air Force had to accept the system since the system answered a
critical need identified during the Gulf War, was considered to be a
success in demonstration, and received support from the President, the
Secretary of Defense, and the Congress.
In at least one case, the Precision/ Rapid Counter Multiple Rocket
Launcher ACTD, DOD did not overcome reluctance and, in turn, missed out on
an opportunity to acquire important warfighting capabilities with joint
applications. This project successfully demonstrated improved capability
in rocket launch detection, command and control, and counterfire necessary
for countering the threat from North Korean multiple rocket artillery with
a system called the Automated Deep Operations Coordination System (ADOCS).
Following the demonstration, the Army* the lead service for the project*
decided not to formally acquire technologies since it was pursuing a
similar development program. Moreover, the Navy, the Air Force, and the
United States Forces, Korea, have acquired and deployed their own unique
versions of the software.
The military services may not want to fund technologies focusing on
meeting joint requirements either because they do not directly affect
their individual missions and/ or because there are other service-
specific projects that the services would prefer to fund. At the same
time, OSD officials told us that they lack a mechanism for ensuring that
decisions on whether to acquire items with proven military utility are
made at the joint level, and not merely by the gaining organizations, and
that these
Page 14 GAO- 03- 52 Defense Acquisitions
acquisitions receive the proper priority. DOD*s Joint Requirements
Oversight Council, which is responsible for validating and prioritizing
joint requirements, plays a role in deciding which ACTD nominees are
selected for demonstration, but it does not have a role in the transition
decision process, and is not currently concerned with transition outcomes.
Moreover, no other DOD organization appears to have been given authority
and responsibility for decisions regarding joint acquisition, integration,
and support issues.
Another factor hindering transition funding has been the lack of alignment
of the ACTD transition process with the DOD planning process. The planning
process requires the services/ agencies to program funds for technology
transition long before the services/ agencies assuming transition
responsibilities know whether a candidate technology is useful to them.
Consequently, at times, the services/ agencies had to find funds within
their own budgets to fund the transition.
The problem of not involving the staff with the appropriate expertise to
carry out demonstrations and transition planning *in all phases of the
ACTD process* may also affect ACTD outcomes. OSD*s guidance recommends
that ACTDs use Integrated Product Teams to organize and conduct ACTDs.
Integrated Product Teams bring together different skill areas (such as
engineering, purchasing, and finance). By combining these areas of
expertise into one team, there is no need to have separate groups of
experts work on a product sequentially. We have reported in the past that
this practice improved both the speed and quality of the decisionmaking
process in developing weapon systems. 4 Conversely, not involving the
acquisition, test, and sustainment communities precludes the opportunity
for OSD to understand during the demonstrations the significant issues
that will arise after transition. In some cases, ACTD projects did not
employ a *transition manager* as called for by OSD*s guidance. This
manager, working for the service or the agency leading the demonstration,
is to prepare the transition plan and coordinate its execution. When a
manager was not designated, these duties often fell to a technical
manager, who was primarily responsible for planning, coordinating, and
directing all development activities through the demonstration. One ACTD*
the Human Intelligence and
4 See U. S. General Accounting Office, Best Practices: DOD Teaming
Practices Not Achieving Potential Results, GAO- 01- 510 (Washington, D.
C.: Apr. 10, 2001). ACTD Management
Page 15 GAO- 03- 52 Defense Acquisitions
Counterintelligence Support Tools* experienced high turnover in the
*operational manager* position. Specifically, it had five different
operational managers over its life. The operational manager, who
represents the ACTD sponsoring command, is responsible for planning and
organizing demonstration scenarios and exercises, defining a concept of
operations for the ACTD, assessing whether the project has military
utility, and making recommendations based on that assessment.
In addition to not involving the right people, at times ACTDs simply did
not anticipate issues important to a successful transition early in the
process. OSD*s guidance calls on teams to prepare a transition strategy
that includes a contracting strategy and addresses issues such as
interoperability, supportability, test and evaluation, affordability,
funding, requirements, and acquisition program documentation. The guidance
also suggests that the transition strategy anticipate where in the formal
acquisition process the item would enter (e. g., low rate initial
production or system development and demonstration) or even whether the
item could be acquired informally, for example, through small purchases of
commercially available products. Specifically, the lead service has the
responsibility to determine the transition timing, nature, and funding
methodology. In two ACTDs, a transition strategy was never developed. Both
of these projects ended up transitioning only as residual assets.
The 1998 Congressional Budget Office study identified similar problems
with transition planning. The study specifically noted that while DOD
calls for each management plan to include some discussion of possible
acquisition costs, few plans did so. The Congressional Budget Office
asserted that this was probably because so little was known about a
project*s future at its start. Even when more was known later in the
demonstration, however, plans remained sketchy.
Software technologies present special planning challenges for transition.
Because of the fast- paced nature of advanced technology, it is critical
to move software ACTD projects through the demonstration and transition
phases quickly so that they are not outdated by the time they are acquired
or integrated into existing software programs and databases. At the same
time, transition might be slowed by incompatibilities between the
operating systems and/ or language of the technologies of the ACTD
candidate( s) and those of the intended host. This can be difficult since
newer applications, particularly commercial- off- the- shelf systems, may
be built to different technical standards or use different languages or
supporting programs. Software Challenges
Page 16 GAO- 03- 52 Defense Acquisitions
It was apparent in several ACTDs that there were technical difficulties in
integrating the new technologies into their intended platforms. For
example, the Adaptive Course of Action project tested software tools
intended to enhance DOD*s Global Command and Control System (GCCS)
specifically by facilitating near real- time collaborative joint planning
by multiple participants during crisis action planning. In this case,
transition has been slowed and may possibly not occur because the software
module cannot be easily integrated into GCCS (partially due to its use of
a different database program) and DOD has not analyzed other functionality
and security issues associated with adding the new module. In another
project, Battlefield Awareness and Data Dissemination, which focused on
providing a synchronized, consistent battlespace description to
warfighters, the transition had a mixed outcome. One collection of
software applications was successfully transitioned to GCCS, but the
transition of others was not as successful. The software application that
was successfully integrated was an update of existing GCCS applications
and the developers of the software had good working relationships with
GCCS managers. The software that experienced problems was not as
compatible.
Another factor potentially affecting the outcomes of ACTDs is the lack of
specific criteria for making assessments of military utility. These
assessments evaluate the technologies of ACTD projects after the
demonstrations. It is important that OSD have some assurance that the
assessments are fact- based, thorough, and consistent, because they
provide the basis upon which the military users can base their transition
recommendations. OSD*s guidance calls for measures of effectiveness and
performance to help gauge whether an item has military utility. It defines
measures of effectiveness as high- level indicators of operational
effectiveness or suitability and measures of performance as technical
characteristics that determine a particular aspect of effectiveness or
suitability. But the guidance does not suggest how detailed the measures
should be, what their scope should be, or what format they should take.
Consequently, we found that the scope, content, and quality of military
utility assessments varied widely. For some of the ACTDs we reviewed, no
documentation on military utility could be found. Without more specific
criteria, customized for each ACTD, there is a risk that decisions on
whether to acquire an item will be based on unsound data. Military Utility
Assessments
Page 17 GAO- 03- 52 Defense Acquisitions
DOD has undertaken several initiatives to improve the ACTD process,
including adopting criteria to ensure technology is sufficiently mature;
evaluating how the ACTD process can be improved; and placing more
attention on transition planning and management (rather than on simply the
selection and demonstration phases) through additional guidance, training,
and staffing. These initiatives target many of the problems that can
hinder success; however, DOD has not addressed the need to establish
specific criteria for assessing the military utility of each of the
candidate technologies and to establish a mechanism to ensure funding is
made available for the transition.
Specifically, DOD headquarters, commands, military services, and a defense
agency have undertaken the following efforts.
OSD has adopted the same TRL criteria for fiscal year 2003 ACTD projects
that DOD uses for assessing technical risks in its formal acquisition
programs. These criteria apply to hardware as well as software. Adhering
to this standard should help DOD to determine whether a gap exists between
a technology*s maturity and the maturity demanded for the ACTD. TRLs
measure readiness on a scale of one to nine, starting with paper studies
of the basic concept, proceeding with laboratory demonstrations, and
ending with a technology that has proven itself on the intended item.
According to a senior OSD official, projects must be rated at least at TRL
5 when they enter the demonstration phase. This means that the basic
technological components of the item being demonstrated have been
integrated with reasonably realistic supporting elements so that the
technology can be tested in a simulated environment. An example would be
when initial hand- built versions of a new radio*s basic elements are
connected and tested together. We reviewed submissions for the final 16
fiscal year 2003 ACTD candidates and found that actual and projected TRLs
of each technology ranged from 4 to 9. 5 According to a senior OSD
official, during the review of fiscal year 2003 candidates, there were
some technologies with a TRL rating of 4 were accepted for demonstration
because the need for them was compelling.
5 See appendix I for a description of TRLs. A single ACTD candidate could
be comprised of multiple technologies assessed at different readiness
levels. We have found that a TRL of 7 at the state of product development
indicates a low risk for cost and schedule increases. Initiatives Are
Underway to Improve ACTD Outcomes
Page 18 GAO- 03- 52 Defense Acquisitions
In early 2002, OSD reviewed the ACTD process to examine current ACTDs
for relevancy in a changing military environment and identify ways to make
sure projects are value- added as well as to enhance transition. The
results of this review included recommendations for additional discipline
and informational requirements in the ACTD candidate selection phase,
increased program management focus on the execution phase, and more
emphasis on management oversight.
OSD has also designated a staff member to manage transition issues and
initiated a training program for future ACTD managers. This training will
emphasize technology transition planning and execution.
To enhance future technology transitions, OSD has taken action to better
align the ACTD selection and the DOD planning and programming process.
Moreover, OSD has issued new guidance for the fiscal year 2004 ACTD
candidates that calls on the gaining military or defense agencies to
identify funds specifically for the demonstration and the transition,
appoint a dedicated transition manager, and develop a transition plan
before it will approve future ACTD candidates.
The combatant commanders, military services, and a defense agency are
also strengthening their guidance for conducting ACTDs. For example, the
U. S. European Command has updated its guidance and the U. S. Joint Forces
Command has developed detailed guidance for selecting and managing ACTDs.
Additionally, the U. S. Pacific Command has developed definitive policies,
procedures, and responsibilities for sponsoring and co- sponsoring ACTD
programs. The U. S. Special Operations Command issued a policy memorandum
for ACTD participation. The Army has begun development of an ACTD tracking
system. It is also requiring ACTD candidate submissions to include TRL and
other quantitative information. The Air Force has drafted both a policy
directive and an instruction regarding ACTDs. The four services have begun
meetings amongst themselves to discuss and review their future ACTD
candidates. The Defense Information Systems Agency is also engaged in an
effort to improve the transition of software technologies to users of
systems such as GCCS.
Collectively, these efforts target many of the factors that can impede the
ACTD process. However, OSD has not yet taken steps to develop specific
criteria for assessing whether each of the ACTD candidates meet military
needs. More guidance in this regard, particularly with respect to the
scope and depth of these assessments and the need to document their
results, can help to make sure (1) decisions are based on sound
information and
Page 19 GAO- 03- 52 Defense Acquisitions
(2) items that could substantially enhance military operations are
acquired. Moreover, while OSD is requiring services and agencies to
identify funds for demonstration and acquisition early in the process, it
does not have a mechanism for ensuring that this funding will be provided.
As a result, it may continue to experience difficulty in getting the
services to fund projects that meet joint needs but do not necessarily fit
in with their own unique plans.
The ACTD process has achieved some important, positive results in terms of
developing and fielding new technologies to meet critical military needs
quickly and more cost- effectively. DOD recognizes that further
improvements are needed to increase opportunities for success. Its efforts
to strengthen assessments of technology readiness and management controls*
combined with more consistent, fact- based assessments of military
utility* should help ensure that the ACTD program will produce better
candidates. However, DOD*s initiatives will be challenging to implement
since they require decision makers to balance the need to preserve
creativity and flexibility within the ACTD process against the need for
structure and management control. Moreover, to fully capitalize on the
improvements being made, DOD needs to ensure that the services sustain
their commitment to projects, especially those shown to meet critical
joint military needs. This will also be a challenge because it will
require DOD to overcome the services and agencies* cultural resistance to
joint initiatives and its lack of a programming and funding process for
joint acquisitions. A place to make a good start in this regard may be to
require the services and agencies to designate funding for ACTD transition
activities and to have the Secretary of Defense weigh in on decisions on
whether to continue to acquire technologies that are tested and proven
under the ACTD program.
To ensure that transition decisions are based on sufficient knowledge, we
recommend that the Secretary of Defense develop and require the use of
specific criteria for assessing the military utility of each of the
technologies and concepts that are to be demonstrated within each ACTD.
The criteria should at a minimum identify measurement standards for
performance effectiveness and address how results should be reported in
terms of scope, format, and desired level of detail.
To ensure funding of the transition and its aftermath, we recommend that
the Secretary of Defense explore the option of requiring the services or
defense agencies to develop a category within their budgets specifically
Conclusions
Recommendations for Executive Action
Page 20 GAO- 03- 52 Defense Acquisitions
for ACTD transition activities, including procurement and follow- on
support.
To ensure that transition decisions reflect DOD*s priorities, we recommend
that the Secretary of Defense require that the lead service or defense
agency obtain the concurrence of the Secretary*s designated representative
on any decision not to transition an ACTD that is based on joint
requirements and determined to be militarily useful.
In commenting on a draft of this report, DOD generally concurred with the
first two recommendations and outlined the actions to be taken to (1)
define ACTD measurement standards and reporting formats for military
utility assessments, and (2) work with the services to enhance their
ability to enable follow- on transition and support of ACTD products. DOD
partially concurred with our recommendation on the transition of
militarily useful technology intended to address joint requirements. DOD
stated that it would work to provide more information to the Joint Staff
on specific ACTD results and evaluate quarterly meetings between the
service acquisition executives and the Under Secretary of Defense for
Acquisition, Technology and Logistics as a possible forum to raise issues
on specific ACTDs. These actions may not address the intent of the
recommendation, which is to provide the joint warfighter the opportunity
to influence the DOD*s investment decisions. The ACTD program offers a
good opportunity in the DOD acquisition system to evaluate equipment and
concepts in the joint warfighting environment. However, while ACTDs often
start based on a joint requirement, that perspective and priority may
change when it comes to transition issues. For the DOD actions to
effectively address this condition, the joint perspective should be more
effectively represented in ACTD transition issues. DOD*s comments are
reprinted in appendix II.
Between fiscal year 1995 and 2002, DOD initiated 99 ACTDs. As we began our
review, 46 of these had completed their demonstration phase or had been
canceled. We reviewed 24 of these in detail. We could not review the
remainder to the same level of detail because their military utility
assessments were incomplete or not available and because we did not choose
to present information on those projects that were highly classified. To
assess the results of the completed ACTDs, we examined each project*s
military utility assessment documents, final program reports, lessons
learned reports, and other pertinent ACTD documents, such as the program
acquisition strategies. We interviewed operational Agency Comments
and Our Evaluation Scope and Methodology
Page 21 GAO- 03- 52 Defense Acquisitions
and technical managers and other knowledgeable program officials at the
unified combatant commanders, defense agencies, and the services to
discuss the phases of each ACTD project and its transition status.
Specifically, we interviewed officials at the Science and Technology
Office of the United States Pacific Command, Camp Smith, Hawaii; the
European Command, Stuttgart, Germany; the Central Command, Tampa, Florida;
the Special Operations Command, Tampa, Florida; the Joint Forces Command,
Norfolk, Virginia; the Air Combat Command, Hampton, Virginia; the Army
Training and Doctrine Command, Hampton, Virginia; and the Marine Corps
Warfighting Lab, Quantico, Virginia. We also contacted ACTD officials at
the Program Executive Office of the Air Base and Port Biological Program
Office, Falls Church, Virginia; the Defense Information Systems Agency,
Falls Church, Virginia; the Defense Advanced Research Projects Agency,
Arlington, Virginia; the Defense Threat Reduction Agency, Fort Belvoir,
Virginia; and the Defense Intelligence Agency, Arlington, Virginia.
To determine the factors that affected the transition outcomes of
completed ACTD projects, we met with the operational and technical
managers for each ACTD as well as other knowledgeable program officials
and the designated ACTD representatives from each of the services. We
compared information gathered on the individual ACTDs to discern those
factors that were salient in a majority of the cases. In order to better
understand ACTD program guidance, funding, and management that can affect
transition outcomes, we spoke with relevant officials within the office of
the Deputy Undersecretary of Defense, Advanced Systems and Concepts (DUSD
(AS& C)), including staff responsible for funding and transition issues,
and the Executive Oversight Manager for each ACTD. We also discussed ACTD
management and transition issues with representatives of the DUSD (AS& C),
Comptroller; the Joint Staff; and the Director, Defense Research and
Engineering; the Defense Advanced Research Projects Agency; and the
Defense Information Systems Agency. We did not conduct a detailed review
of the users* acceptance or satisfaction with the items of the ACTD
process.
We conducted our review between October 2001 and October 2002 in
accordance with generally accepted government auditing standards.
We are sending copies of this report to the Chairmen and Ranking Minority
Members of the Subcommittee on Defense, Senate Committee on
Appropriations; the House Committee on Armed Services; and the
Page 22 GAO- 03- 52 Defense Acquisitions
Subcommittee on Defense, House Committee on Appropriations; and the
Secretaries of Defense, the Army, the Navy, and the Air Force. We are also
sending copies to the Director, Office of Management and Budget. In
addition, this report will be made available at no charge on the GAO Web
site at http:// www. gao. gov.
If you or your staff have questions concerning this report, please contact
me at (202) 512- 4841. Others who made key contributions to this report
include William Graveline, Tony Blieberger, Cristina Chaplain, Martha Dey,
Leon Gill, and Nancy Rothlisberger.
Katherine V. Schinasi Director, Acquisition and Sourcing Management
Appendix I: Technology Readiness Levels and Their Definitions
Page 23 GAO- 03- 52 Defense Acquisitions
Technology readiness level Description
1. Basic principles observed and reported. Lowest level of technology
readiness. Scientific research begins to be translated into
applied research and development. Examples might include paper studies of
a technology*s basic properties 2. Technology concept and/ or application
formulated. Invention begins. Once basic principles are observed,
practical applications can be
invented. The application is speculative and there is no proof or detailed
analysis to support the assumption. Examples are still limited to paper
studies. 3. Analytical and experimental critical function and/ or
characteristic proof of concept.
Active research and development is initiated. This includes analytical
studies and laboratory studies to physically validate analytical
predictions of separate elements of the technology. Examples include
components that are not yet integrated or representative. 4. Component
and/ or breadboard. Validation in laboratory environment.
Basic technological components are integrated to establish that the pieces
will work together. This is relatively *low fidelity* compared to the
eventual system. Examples include integration of *ad hoc* hardware in a
laboratory. 5. Component and/ or breadboard validation in relevant
environment. Fidelity of breadboard technology increases significantly.
The basic technological
components are integrated with reasonably realistic supporting elements so
that the technology can be tested in a simulated environment. Examples
include *high fidelity* laboratory integration of components. 6. System/
subsystem model or prototype demonstration in a relevant environment.
Representative model or prototype system, which is well beyond the
breadboard tested
for technology readiness level (TRL) 5, is tested in a relevant
environment. Represents a major step up in a technology*s demonstrated
readiness. Examples include testing a prototype in a high fidelity
laboratory environment or in a simulated operational environment. 7.
System prototype demonstration in an operational environment. Prototype
near or at planned operational system. Represents a major step up from TRL
6,
requiring the demonstration of an actual system prototype in an
operational environment, such as in an aircraft, vehicle or space.
Examples include testing the prototype in a test bed aircraft. 8. Actual
system completed and *flight qualified* through test and demonstration.
Technology has been proven to work in its final form and under expected
conditions. In almost all cases, this TRL represents the end of true
system development. Examples include developmental test and evaluation of
the system in its intended weapon system to determine if it meets design
specifications. 9. Actual system *flight proven* through successful
mission operations. Actual application of the technology in its final form
and under mission conditions, such as
those encountered in operational test and evaluation. In almost all cases,
this is the end of the last *bug fixing* aspects of true system
development. Examples include using the system under operational mission
conditions.
Appendix I: Technology Readiness Levels and Their Definitions
Appendix II: Comments from the Department of Defense
Page 24 GAO- 03- 52 Defense Acquisitions
Appendix II: Comments from the Department of Defense
The Department of Defense provided written comments on a draft of our
report. In a November 27, 2002, letter DOD modified its comments from
*partially concur* to *concur* with our recommendation 1.
Appendix II: Comments from the Department of Defense
Page 25 GAO- 03- 52 Defense Acquisitions
Appendix II: Comments from the Department of Defense
Page 26 GAO- 03- 52 Defense Acquisitions
Appendix II: Comments from the Department of Defense
Page 27 GAO- 03- 52 Defense Acquisitions (120105)
The General Accounting Office, the investigative arm of Congress, exists
to support Congress in meeting its constitutional responsibilities and to
help improve the performance and accountability of the federal government
for the American people. GAO examines the use of public funds; evaluates
federal programs and policies; and provides analyses, recommendations, and
other assistance to help Congress make informed oversight, policy, and
funding decisions. GAO*s commitment to good government is reflected in its
core values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail this list to you every afternoon, go to www. gao. gov
and select *Subscribe to daily E- mail alert for newly released products*
under the GAO Reports heading.
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548
To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax: (202)
512- 6061
Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm E- mail:
fraudnet@ gao. gov Automated answering system: (800) 424- 5454 or (202)
512- 7470
Jeff Nelligan, managing director, NelliganJ@ gao. gov (202) 512- 4800 U.
S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D. C.
20548 GAO*s Mission
Obtaining Copies of GAO Reports and Testimony
Order by Mail or Phone To Report Fraud, Waste, and Abuse in Federal
Programs
Public Affairs
*** End of document. ***