[House Hearing, 106 Congress]
[From the U.S. Government Publishing Office]





         JOINT STRIKE FIGHTER ACQUISITION REFORM: WILL IT FLY?

=======================================================================

                                HEARING

                               before the

                   SUBCOMMITTEE ON NATIONAL SECURITY,
                  VETERANS AFFAIRS, AND INTERNATIONAL
                               RELATIONS

                                 of the

                              COMMITTEE ON
                           GOVERNMENT REFORM

                        HOUSE OF REPRESENTATIVES

                       ONE HUNDRED SIXTH CONGRESS

                             SECOND SESSION

                               __________

                              MAY 10, 2000

                               __________

                           Serial No. 106-202

                               __________

       Printed for the use of the Committee on Government Reform


  Available via the World Wide Web: http://www.gpo.gov/congress/house
                      http://www.house.gov/reform


                               __________

                    U.S. GOVERNMENT PRINTING OFFICE
70-782                     WASHINGTON : 2001

_______________________________________________________________________
 For sale by the Superintendent of Documents, U.S. Government Printing 
                                 Office
Internet: bookstore.gpo.gov  Phone: (202) 512-1800  Fax: (202) 512-2250
               Mail: Stop SSOP, Washington, DC 20402-0001


                     COMMITTEE ON GOVERNMENT REFORM

                     DAN BURTON, Indiana, Chairman
BENJAMIN A. GILMAN, New York         HENRY A. WAXMAN, California
CONSTANCE A. MORELLA, Maryland       TOM LANTOS, California
CHRISTOPHER SHAYS, Connecticut       ROBERT E. WISE, Jr., West Virginia
ILEANA ROS-LEHTINEN, Florida         MAJOR R. OWENS, New York
JOHN M. McHUGH, New York             EDOLPHUS TOWNS, New York
STEPHEN HORN, California             PAUL E. KANJORSKI, Pennsylvania
JOHN L. MICA, Florida                PATSY T. MINK, Hawaii
THOMAS M. DAVIS, Virginia            CAROLYN B. MALONEY, New York
DAVID M. McINTOSH, Indiana           ELEANOR HOLMES NORTON, Washington, 
MARK E. SOUDER, Indiana                  DC
JOE SCARBOROUGH, Florida             CHAKA FATTAH, Pennsylvania
STEVEN C. LaTOURETTE, Ohio           ELIJAH E. CUMMINGS, Maryland
MARSHALL ``MARK'' SANFORD, South     DENNIS J. KUCINICH, Ohio
    Carolina                         ROD R. BLAGOJEVICH, Illinois
BOB BARR, Georgia                    DANNY K. DAVIS, Illinois
DAN MILLER, Florida                  JOHN F. TIERNEY, Massachusetts
ASA HUTCHINSON, Arkansas             JIM TURNER, Texas
LEE TERRY, Nebraska                  THOMAS H. ALLEN, Maine
JUDY BIGGERT, Illinois               HAROLD E. FORD, Jr., Tennessee
GREG WALDEN, Oregon                  JANICE D. SCHAKOWSKY, Illinois
DOUG OSE, California                             ------
PAUL RYAN, Wisconsin                 BERNARD SANDERS, Vermont 
HELEN CHENOWETH-HAGE, Idaho              (Independent)
DAVID VITTER, Louisiana


                      Kevin Binger, Staff Director
                 Daniel R. Moll, Deputy Staff Director
           David A. Kass, Deputy Counsel and Parliamentarian
                    Lisa Smith Arafune, Chief Clerk
                 Phil Schiliro, Minority Staff Director
                                 ------                                

Subcommittee on National Security, Veterans Affairs, and International 
                               Relations

                CHRISTOPHER SHAYS, Connecticut, Chairman
MARK E. SOUDER, Indiana              ROD R. BLAGOJEVICH, Illinois
ILEANA ROS-LEHTINEN, Florida         TOM LANTOS, California
JOHN M. McHUGH, New York             ROBERT E. WISE, Jr., West Virginia
JOHN L. MICA, Florida                JOHN F. TIERNEY, Massachusetts
DAVID M. McINTOSH, Indiana           THOMAS H. ALLEN, Maine
MARSHALL ``MARK'' SANFORD, South     EDOLPHUS TOWNS, New York
    Carolina                         BERNARD SANDERS, Vermont 
LEE TERRY, Nebraska                      (Independent)
JUDY BIGGERT, Illinois               JANICE D. SCHAKOWSKY, Illinois
HELEN CHENOWETH-HAGE, Idaho

                               Ex Officio

DAN BURTON, Indiana                  HENRY A. WAXMAN, California
            Lawrence J. Halloran, Staff Director and Counsel
                  J. Vincent Chase, Chief Investigator
                           Jason Chung, Clerk
                    David Rapallo, Minority Counsel

                            C O N T E N T S

                              ----------                              
                                                                   Page
Hearing held on May 10, 2000.....................................     1
Statement of:
    McNaugher, Dr. Thomas L., deputy director, Arroyo Center, 
      Rand; Rodney Larkins, business development manager, 3M 
      Corp.; and Dr. Wesley Harris, professor, Department of 
      Aeronautics and Astronautics...............................   100
    Rodrigues, Louis J., Director, National Security and 
      International Affairs Division, U.S. General Accounting 
      Office.....................................................     5
    Soloway, Stan Z., Deputy Secretary of Defense, Acquisition 
      Reform, Department of Defense; and Major General Raymond 
      Huot, U.S. Air Force, Acquisition Programs, Department of 
      Defense....................................................    50
Letters, statements, etc., submitted for the record by:
    Blagojevich, Hon. Rod R., a Representative in Congress from 
      the State of Illinios, prepared statement of...............    37
    Harris, Dr. Wesley, professor, Department of Aeronautics and 
      Astronautics, prepared statement of........................   120
    Huot, Major General Raymond, U.S. Air Force, Acquisition 
      Programs, Department of Defense:
        Information concerning TRLs..............................    96
        Prepared statement of....................................    72
    Larkins, Rodney, business development manager, 3M Corp., 
      prepared statement of......................................   112
    McNaugher, Dr. Thomas L., deputy director, Arroyo Center, 
      Rand, prepared statement of................................   104
    Rodrigues, Louis J., Director, National Security and 
      International Affairs Division, U.S. General Accounting 
      Office, prepared statement of..............................    10
    Shays, Hon. Christopher, a Representative in Congress from 
      the State of Connecticut, prepared statement of............     3
    Soloway, Stan Z., Deputy Secretary of Defense, Acquisition 
      Reform, Department of Defense:
        Information concerning TRL levels........................    98
        Prepared statement of....................................    56

 
         JOINT STRIKE FIGHTER ACQUISITION REFORM: WILL IT FLY?

                              ----------                              


                        WEDNESDAY, MAY 10, 2000

                  House of Representatives,
       Subcommittee on National Security, Veterans 
              Affairs, and International Relations,
                            Committee on Government Reform,
                                                    Washington, DC.
    The subcommittee met, pursuant to notice, at 10 a.m., in 
room 2247, Rayburn House Office Building, Hon. Christopher 
Shays (chairman of the subcommittee) presiding.
    Present: Representatives Shays, Terry, Biggert, and 
Blagojevich.
    Staff present: Lawrence J. Halloran, staff director and 
counsel; J. Vincent Chase, chief investigator; Robert Newman, 
professional staff member; Jason M. Chung, clerk; David 
Rapallo, minority counsel; and Earley Green, minority assistant 
clerk.
    Mr. Shays. I would like to call this hearing to order, to 
welcome our witnesses and our guests and again to invite anyone 
who wants to, to take off their coats. It's a hot room and we 
have asked it to be cooled down, but feel free to take off your 
jackets.
    The military procurement holiday is about to end. Defense 
budgets being debated today on both sides of the Capitol 
reflect a bi-cameral and bipartisan consensus on the need to 
modernize the aging planes, ships, weapons and equipment used 
to win the cold war. Today we discuss the need to modernize the 
acquisition systems the Department of Defense [DOD], will use 
to procure post-cold war weapons systems.
    Just as the weaponry of the last century won't win the 
peace in the next, the acquisition practices of the past will 
not efficiently or affordably meet future defense needs. 
Fifteen-year development cycles enshrine old technologies now 
rendered obsolete in 15 months. Massive cost overruns and 
schedule slippages are fueled in part by the launch of 
engineering and design work before hoped-for technologies have 
been refined. Extraneous, often pervasive incentives, push 
program officials toward artificial deadlines and premature 
production commitments.
    Various iterations of acquisition reform at DOD have 
attempted to address these problems and reinvigorate a 
hidebound acquisition culture inside and outside the Pentagon. 
In launching the $200 billion Joint Strike Fighter [JSF], 
aircraft acquisition, DOD promised the prgram would be a model 
of reform driven by affordability and the technical knowledge 
base, not by the disingenuous optimism and defense budget 
politics that proved so costly in the past.
    At the subcommittee's request, the General Accounting 
Office [GAO], analyzed the JSF acquisition strategy to 
determine if the promise of reform is being fulfilled in 
practice. Their report released today finds the Joint Strike 
Fighter program straying from commercial best practices and 
knowledge-driven benchmarks. As the date approaches to select a 
prime JSF contractor and begin engineering on the final system 
concepts, DOD appears ready to abandon quantitative measures of 
technological maturity and revert to the business as usual of 
concurrent technology development and product development.
    GAO recommends DOD focus on risk reduction efforts by 
maturing critical technologies prior entering the next phase of 
the JSF program, even if that means delaying contractor 
selection and contract awards beyond the planned March 2001 
date. The program should be permitted to pursue the original 
low-risk acquisition strategy according to GAO, without the 
penalty of withdrawal of funding support.
    DOD disagrees, claiming critical technologies will be 
mature enough to proceed in final design and engineering next 
year.
    As the debate unfolds, the choice should not be between a 
fully funded Joint Strike Fighter and a commitment to 
acquisition reform. We can have both. If the program succumbs 
to cold war acquisition habits, costs will skyrocket, the 
development cycle will stretch over the horizon, and the next 
generation fighter needed by the Air Force, the Navy, and the 
Marines might never fly.
    We welcome the testimony of all our witnesses on this 
important subject.
    [The prepared statement of Hon. Christopher Shays follows:]

    [GRAPHIC] [TIFF OMITTED] T0782.001
    
    [GRAPHIC] [TIFF OMITTED] T0782.002
    
    Mr. Shays. And if I could, I would like to welcome my 
colleague and ask if he has any comments to make.
    Mr. Terry. Thank you, no. I don't have any opening 
statement. I'll submit one in a few days for the record.
    I do want to compliment you on holding this hearing. When 
we look through the issues concerning the future of the 
military and the confidence that people have in our military 
and as we make a renewed commitment as a Congress to our 
military, we have got to ensure that the protocol is in place, 
the system is in place to ensure that we use our money wisely, 
that we're looking toward the future. And when I'm speaking at 
veterans organizations or just simply people that are 
interested in my home of Omaha, NE and home of Offutt Air Force 
Base, that's what they want to know.
    That's why this hearing is so important today. Since it is 
so important, let me not continue to use the time, let's hear 
from our witnesses.
    Mr. Shays. Thank you Mr. Terry.
    Just some housekeeping, if we could do that now. I ask 
unanimous consent that all members of the subcommittee be 
permitted to place an opening statement in the record and that 
the record remain open for 3 days for that purpose. Without 
objection, so ordered.
    I ask further unanimous consent that all witnesses be 
permitted to include their written statements in the record. 
Without objection, so ordered.
    Our first panel, we have three, is Mr. Louis Rodrigues, 
Director, National Security and International Affairs Division, 
U.S. General Accounting Office [GAO]. In fact, I think that's 
referred to more as GAO than the full title.
    Now Mr. Rodrigues, if you could stand up, we'll swear you 
in, as we swear in all our witnesses. Raise your right hand, 
please.
    [Witness sworn.]
    Mr. Shays. Note for the record our witness has responded in 
the affirmative. What we're going to do is we're going to run 
the clock for 5 minutes then we will flip it again for another 
5. And then if you could conclude within the 10-minute period.

 STATEMENT OF LOUIS J. RODRIGUES, DIRECTOR, NATIONAL SECURITY 
  AND INTERNATIONAL AFFAIRS DIVISION, U.S. GENERAL ACCOUNTING 
                             OFFICE

    Mr. Rodrigues. Thank you, Mr. Chairman. Mr. Chairman, 
Congressman Terry, I am pleased to be here today to discuss the 
application of best commercial practices to DOD weapons systems 
in general, the Joint Strike Fighter in particular. Before 
getting into details, I would like to emphasize the importance 
of the Joint Strike Fighter decision to reforming DOD's weapons 
acquisition process.
    As you know, the Department is in the process of rewriting 
its directives governing systems acquisition, referred to as 
DOD 5000 series. At the Department's request, we have been 
participating in this effort through input to its working 
group. The objective of the rewrite is to bring about better, 
cheaper, faster outcomes in weapons programs. It is acquisition 
reform. Our contributions or inputs to this effort are based on 
our reports for the Senate Armed Services Committee on using 
best commercial practices to improve weapons program outcomes.
    The DOD draft rewrite embodies critical features documented 
in our work. Two of these features are critical to the upcoming 
Joint Strike Fighter decision. First, the technology 
development must be separated from product development. That 
is, before entering engineering manufacturing development, we 
must have a match between proven technologies and requirements. 
And second, metrics to accurately measure technology must be 
used.
    In the 5000 series rewrite, they are adopting a measurement 
system we used in our Joint Strike Fighter assessment, referred 
to as technology readiness levels. The commitment to this 
knowledge-based versus the current schedule- and funding-driven 
process is reflected in the testimony of the Deputy Under 
Secretary of Defense for Acquisition Reform on March 16th 
before the House Government Reform Subcommittee on Government 
Management, Information, and Technology, ``In the new systems 
acquisition environment, key acquisition and long-term funding 
commitments will not be made until technology is mature.''
    I have a lot of respect for the people in DOD who are 
leading acquisition reform. Philosophically we are in agreement 
on the best practices and the changes that are needed in the 
DOD environment to make such practices work on weapons systems. 
We at GAO are extremely encouraged by the commitment of DOD 
acquisition leaders to improving the weapons acquisition 
outcomes through the use of a knowledge-based commercial 
business practices. At the same time, however, we are concerned 
that the written directives and oral commitments will have 
little impact if not reflected in key decisions.
    In that sense, the key acquisition decision of entering 
engineering and manufacturing development on the Joint Strike 
Fighter stands out as the flagship for weapons acquisition 
reform. To apply anything less than the standards in the 
directives will send a clear message that while the 
instructions and rhetoric are changing, it's business as usual.
    I will now briefly describe what we have learned from our 
best practices work and how we have applied that to assessing 
the Joint Strike Fighter program. Our best practices work has 
shown that a knowledge-based process is essential to getting 
better cost schedule and performance outcomes. This means that 
the decisionmakers must have virtual certainty about critical 
facets of the product under the development when needed. This 
knowledge can be measured in three junctures that we refer to 
as knowledge points, as shown in the chart to my right.
    Knowledge point 1 is when a match is made between the 
customer's requirements and the available technology. This 
occurs prior to entering product development.
    Knowledge point 2 is when the product's design is 
determined to be capable of meeting performance requirements. 
This occurs about midway through the product development phase.
    And knowledge point 3 is when the product is determined to 
be producible within cost, schedule, and quality targets, which 
occurs prior to entering production.
    Today I'll focus on only knowledge point 1 because it is 
the biggest contributor to a successful product development, 
achieving subsequent knowledge points depends on it, and it is 
the point where JSF should be as it enters EMD. As a technology 
is developed, it moves from a concept to a feasible invention 
to a component that must fit onto a product and function as 
expected. In between, there are increasing levels of 
demonstration that can be measured.
    In our review of best practices for including new 
technology and products, we applied a scale of technology 
readiness levels from 1 to 9, pioneered by NASA and adapted by 
the Air Force Research Laboratory. Without going into the 
details of each level, a level 4 equates to a laboratory 
demonstration of technology that is not in a usable form.
    Imagine an advanced radio technology that can be 
demonstrated with components that take up a table top. When 
initial hand-built versions of all the radio's basic elements 
are hand wired and tested together in a laboratory, the radio 
reaches a readiness level of 5.
    A technology readiness level of 7 is the demonstration of 
the technology that approximates its final form and occurs in 
an environment outside the laboratory. That same radio at level 
7 would be installed and demonstrated in a platform such as an 
existing fighter aircraft. The lower the level of the 
technology at the time it is included in product development, 
the higher the risk that it will cause problems in product 
development.
    According to the Air Force Research Laboratory, level 7 
enables the technology to be included on a product development 
with acceptable risk.
    When we asked leading commercial firms to apply these 
standards to their own methods of assessing technology 
maturity, we found that most insisted on even higher levels of 
readiness before they allowed a new technology into product 
development.
    Regarding the Joint Strike Fighter, in conjunction with the 
program office and the two competing contractors, we determined 
the readiness levels of critical technologies. The table to my 
right shows the technology readiness levels of eight critical 
technology areas identified by the Joint Strike Fighter program 
office.
    Let me try to explain this a little bit. What we had them 
do is score the technologies at three points. The blue line 
reflects the readiness level for each of those technology areas 
where they were at program launch. The yellow line reflects 
where they were at the time they did the scoring. And the red 
extension reflects where they plan to be based on what they're 
planning to do between now and the down select to the 
engineering and manufacturing development phase. So it's the 
totality of the line that shows where they plan to be when they 
enter EMD.
    In terms of engineering and manufacturing development, 
which is reflected by the second diamond on the right, none of 
the critical technology areas are projected to be at readiness 
level 7, which the Air Force Research Laboratory considers 
acceptable for entry into engineering and manufacturing 
development. Should any of these technologies be delayed, or 
worse, not available for incorporation into the final JSF 
design, the impact on the program would be dramatic. For 
example, if one of the critical technologies needed to be 
replaced with its planned backup, DOD could expect an increase 
of about 10 percent in unit costs. The backup technology would 
also significantly increase aircraft weight which can 
negatively impact aircraft performance. This technology is 
projected to be at a technology readiness level of 5 at the 
beginning of the engineering manufacturing development phase, 
substantially below the criteria of 7.
    As noted earlier, at the policy level, DOD officials have 
agreed that technology development should be kept separate from 
product development and that technology readiness levels are a 
valid way to assess technology maturity. However, in response 
to our report on the Joint Strike Fighter, DOD balked at the 
use of technology readiness levels and their implications for 
keeping technology development out of the fighter's engineering 
and manufacturing development phase. One of the reasons DOD 
cited for its unwillingness to accept the technology readiness 
levels assessed was that the levels were based on integration 
in the Joint Strike Fighter aircraft.
    On the contrary, the technology readiness levels assessed 
by the program office and the contractors were based on a clear 
understanding that a level 7 could be reached by demonstrating 
the technology in a relevant environment. It was further made 
clear that a relevant environment would include demonstrating a 
technology in an existing aircraft like an F-16, not a Joint 
Strike Fighter. There is no misunderstanding.
    A second reason DOD disagreed with the readiness levels 
assessed was that its own risk mitigation plans and judgment 
were more meaningful and that they showed the technology risk 
to be acceptable. Risk mitigation plans and judgment are 
necessary to managing any major development effort. However, 
without an underpinning such as technology readiness levels 
that allows transparency into program decisions, these methods 
allow significant technical unknowns to be judged acceptable 
risks because a plan exists for resolving them in the future.
    Experience on previous programs has shown that such methods 
have rarely assessed technical unknowns as a high or 
unacceptable risk. Consequently, they fail to guide programs to 
meet promised outcomes. Technology readiness levels are based 
on demonstrations of how well technology has actually 
performed. Their strength lies in the fact that they 
characterize knowledge that exists rather than plans to gain 
knowledge in the future.
    In conclusion, we believe that separating technology 
development from product development can create conditions for 
a successful Joint Strike Fighter program. To proceed as 
planned, entering the engineering manufacturing development 
phase of the program with immature technologies, is to risk 
scheduled delays and cost growth. Instead, the program has an 
opportunity to mature technologies in a more risk-tolerant 
environment by making the right decisions now.
    In our report, we recommend that the Joint Strike Fighter 
program continue in its current program definition and risk 
reduction phase, delaying the decision to move into engineering 
and manufacturing development until technologies are 
demonstrated to acceptable levels. Taking the additional time 
to mature the technologies will then allow the program manager 
to focus on design and manufacturing risks during engineering 
and manufacturing development. It also increases the 
possibility of completing product development in a more timely 
and predictable manner. Such a delay does not necessarily 
lengthen the total product development cycle. In fact, the 
knowledge gained from time spent developing technologies in the 
beginning can often shorten the time it takes to get the 
product to market.
    Similarly, a delay should not be misinterpreted as a 
lessening of support for the Joint Strike Fighter program. 
Rather, it would demonstrate decisionmakers' willingness to 
make the up-front investment necessary to mature key 
technologies before committing the Joint Strike Fighter team to 
deliver a product. Such a commitment is more likely to put the 
program on a better footing to succeed than placing the burden 
on the engineering and manufacturing development phase.
    Mr. Chairman, that concludes my statement and I would be 
glad to answer any questions you or the other members may have.
    [The prepared statement of Mr. Rodrigues follows:]

    [GRAPHIC] [TIFF OMITTED] T0782.003
    
    [GRAPHIC] [TIFF OMITTED] T0782.004
    
    [GRAPHIC] [TIFF OMITTED] T0782.005
    
    [GRAPHIC] [TIFF OMITTED] T0782.006
    
    [GRAPHIC] [TIFF OMITTED] T0782.007
    
    [GRAPHIC] [TIFF OMITTED] T0782.008
    
    [GRAPHIC] [TIFF OMITTED] T0782.009
    
    [GRAPHIC] [TIFF OMITTED] T0782.010
    
    [GRAPHIC] [TIFF OMITTED] T0782.011
    
    [GRAPHIC] [TIFF OMITTED] T0782.012
    
    [GRAPHIC] [TIFF OMITTED] T0782.013
    
    [GRAPHIC] [TIFF OMITTED] T0782.014
    
    [GRAPHIC] [TIFF OMITTED] T0782.015
    
    [GRAPHIC] [TIFF OMITTED] T0782.016
    
    [GRAPHIC] [TIFF OMITTED] T0782.017
    
    [GRAPHIC] [TIFF OMITTED] T0782.018
    
    [GRAPHIC] [TIFF OMITTED] T0782.019
    
    [GRAPHIC] [TIFF OMITTED] T0782.020
    
    [GRAPHIC] [TIFF OMITTED] T0782.021
    
    [GRAPHIC] [TIFF OMITTED] T0782.022
    
    [GRAPHIC] [TIFF OMITTED] T0782.023
    
    [GRAPHIC] [TIFF OMITTED] T0782.024
    
    Mr. Shays. Thank the gentleman very much. Thank you very 
much, Mr. Rodrigues. The bottom line is we're scheduled to 
build the Air Force F-22 and the F/A-18, the F Super Hornet, 
and then we're scheduled in the future to see this project go 
forward. You're not suggesting in any way that we end this 
program and not do it; correct?
    Mr. Rodrigues. Correct, Mr. Chairman.
    Mr. Shays. The bottom line to the debate we're going to 
have today is you just want them to follow their game plan.
    Mr. Rodrigues. We want them to----
    Mr. Shays. By ``they,'' the DOD to follow the game plan.
    Mr. Rodrigues. Yes, sir. Mr. Chairman, what we're talking 
about is that the Joint Strike Fighter decision is such a major 
decision it is the signal of what happens in acquisition 
reform. It will underscore the implementation of the changes 
that are being made today in the regulations to guide this 
process. And if we make the same types of decisions that we 
have made in the past, that is, to allow unknowns to creep into 
product development, to allow technology development to be done 
concurrently with product development, we'll continue to see 
very long development cycles like we have seen on the F-22, 
like we saw on the B-2, and like we have seen on programs 
historically.
    And that situation undermines our modernization effort. It 
leads to cost growth and schedule delays. It leads to the 
problems that we have seen with the programs in the past where 
we end up cutting the quantities in half and never do meet our 
modernization goals. In fact, it exacerbates the problems that 
we're trying to resolve, one of which is aging of the fleet. It 
is a real issue. And to continue to move into programs where we 
can't deliver on cost and schedule, that are fit into very 
fight funding wedges, puts us in a position where we end up in 
the future cutting the quantities because of cost problems and 
ending up without the modernization that we absolutely need.
    Mr. Shays. The bottom line, I just want to establish the 
point that GAO is not saying that we scrap this program. You 
want us to follow the game plan. And so what this committee has 
to understand is what you're saying and what DOD is saying and 
where those disagreements occur.
    I'm going to expose my ignorance a little bit here. If we 
could go to the first chart, I find that's the best way to 
learn. I sometimes make my staff nervous when I do this.
    Mr. Rodrigues. What this depicts, Mr. Chairman is, in 
effect, the commercial model.
    Mr. Shays. What I need to do is just incorporate this with 
your nine product requirements. How does that--this isn't the 
first three of the nine. Can you incorporate this to the--to 
TRLs?
    Mr. Rodrigues. TRLs are a metric. They are a metric. 
They're the metric that allows you to make the determination of 
the technology match to requirements. So that the first 
knowledge point in any program is assuring that you have a 
match between the technology and the requirement.
    Mr. Shays. Is point 3 here point 7 on TRLs?
    Mr. Rodrigues. No. The 7 on the TRL is that that match has 
occurred in that technology-to-requirements match. It is the 
first knowledge point. It is where we are approaching on the 
Joint Strike Fighter.
    Mr. Shays. And the debate, it seems to me, is whether DOD 
thinks your match point at 7 is the key point. They're willing 
to move forward before they've reached 7. They dispute that 
they haven't reached point 7.
    Mr. Rodrigues. I'm sure that will become a confusing issue.
    Mr. Shays. I just want to know what you think.
    Mr. Rodrigues. To me it's absolutely clear. Applying 
technology readiness levels, which is the demonstration of 
these technologies, puts those technologies where they are on 
the second chart.
    Would you put that chart back up, please?
    And we're accepting their projections of where they're 
going to be. Once again as I said, the diamond to the right is 
where you should have to be. That is the level 7. That is the 
acceptable risk for entry into engineering and manufacturing 
development.
    Mr. Shays. What I'm asking you is do they dispute what you 
have up here and are they--believe that the acceptable risk is 
at point 6 and you believe the acceptable risk is at point 7? 
Where is the dispute as far as you would articulate it?
    Mr. Rodrigues. They argue that there was a misunderstanding 
in the application by the contractors. We did not score these. 
The contractors applied the criteria. The DOD argument, as I 
understand it, is that there was a misunderstanding in the 
application. To get to a level 7 requires the demonstration of 
the technology or the hardware involved in this technology in 
the form fit and function. In other words, what it's going to 
really look like when it has to go onto the Joint Strike 
Fighter in a relevant environment.
    And the Department's position, as I understand it, is that 
there was a misunderstanding and that what people were thinking 
when they scored this was that we were saying it had to 
actually be on the Joint Strike Fighter. Well, that obviously 
cannot happen before the program enters engineering and 
manufacturing development because there is no Joint Strike 
Fighter.
    We worked very closely with the contractors and program 
office people who were at those meetings to explain exactly how 
to do this. I have the sheets that were provided to them. We 
spent days. And we made sure that they understood that we are 
talking about surrogates. There is physically no way to do the 
kind of demonstration they're contending these people were 
thinking we were talking about absent a Joint Strike Fighter. 
What we were talking about was demonstrated on surrogates.
    Mr. Shays. To give some life to these technologies, you've 
listed under numbers because of proprietary issues, but--so I 
won't debate which is which. But what we're talking about is 
propulsion, we're talking about flight systems, we're talking 
about weapons. Each of these is a technology. We're talking 
about structures and materials. In avionics we're talking about 
radar and the mission systems and supportability and training 
and producibility. Those are the things that we're talking 
about.
    Now when we started out, they were at level technology 1, 
whichever one it is, was at level 3. And you're then saying 
that they have gotten to level 5 and expect to get in 
technology 1 to level 6. And then make a commitment. They're 
wanting to commit at level 6, correct?
    Mr. Rodrigues. That's what we're saying. They're not going 
to agree with that.
    Mr. Shays. But that's what you think they're saying.
    Mr. Rodrigues. Yes.
    Mr. Shays. And they're willing to commit on technology 8 at 
level 6. You're not saying they're willing to commit on 
technology 4 at level 5, are you; or are you?
    Mr. Rodrigues. Right now our understanding is they plan to 
move ahead----
    Mr. Shays. And every one of the levels I mentioned----
    Mr. Rodrigues. Unless they trade these off. But right now 
they're in the program--these are critical path technologies. 
These things are essential. Not technology 1 necessarily--
technology areas 2 through 7 are critical to meeting the 
affordability goal. And if affordability is the primary factor 
driving this program, which is my understanding of what 
everybody is signed up to do, an affordable next generation 
aircraft, then being able to launch or have a program without 
these becomes really problematic.
    The one example I used was if one of these areas was 
excluded or you had to go to its fallback, you would add almost 
10 percent cost to the conventional variant which represents 
1,700 plus of the aircraft they're going to build. That is a 
significant cost difference. And that's only one area.
    So these all deal with affordability. As long as 
affordability remains paramount, the question becomes if you 
don't use these, what does it do to your cost projections? If 
you go do a block approach which--talking about using an 
iterative process going with the block one that doesn't have 
everything and moving on--I think we have to understand what is 
the implications of that. What you're doing then is launching 
into the development of a product and the production of a 
product that I'm not sure it even comes close to meeting cost 
goals, because I don't know what the effect of deleting these 
from the first blocks are. And once you have that production 
line started, if you're betting on technology in some 
subsequent block in order to get the affordability into the 
program that you need in the long run, I don't think that we 
solved anything by going to a block approach.
    Mr. Shays. I would agree with that. Let me just go back to 
your first chart again on the best practices model. There are 
three knowledge points. And I appreciate the indulgence of the 
committee just to go a little further on this. Knowledge 1, 
this is basically the character of best practices act for any 
industry in this modern day and age.
    Mr. Rodrigues. That's the best commercial practice.
    Mr. Shays. The concept is matches made between the 
customer's requirements and the available technology. So 
whatever technology is available, how can we meet the 
customer's requirements? We try to match those two.
    Mr. Rodrigues. Right.
    Mr. Shays. And then the second point is when the product's 
design is determined to be capable of meeting performance 
requirements.
    Mr. Rodrigues. Yes. And there's a metric associated with 
that as well. The Joint Strike Fighter right now would be in 
the equivalent of what's called the technology development 
phase. It is in risk reduction. There are concept 
demonstrations in the combined phase. So it's really before 
that point of technology match.
    Mr. Shays. We're not at point 2 yet.
    Mr. Rodrigues. No, the product development is the 
equivalent of the engineering and manufacturing development 
phase. Knowledge point 2 is a standard knowledge point that 
occurs both in industry and in the Department. The Department 
just didn't adhere to it. They actually have the standard and 
they have the metrics for measuring it. It's something called 
CDR, critical design review, and the standard is that 90 
percent of the drawings are released to manufacturing at that 
point. Now, unfortunately, the Department doesn't adhere to 
that. The standard exists, they don't apply it. In commercial 
industry, when we were doing the best commercial practice work, 
they exceed the 90 percent.
    Mr. Shays. I have two more basic questions. When we get to 
knowledge point 3, the product is determined to be producible 
within cost schedule and quality targets. Now, is the desire on 
the part of the military to move forward, in your judgment, 
with production before we have reached point 2, is that because 
they want the product to do more than right now the technology 
allows?
    Mr. Rodrigues. Let me try to clarify something. This isn't 
about moving forward at this point with production. This is 
about entry into product development.
    Mr. Shays. Right.
    Mr. Rodrigues. And I don't know----
    Mr. Shays. But even the point of--the question still 
stands, and thank you for clarifying. The bottom line, though, 
is I'm just trying to understand the tension. In other words, 
they want this airplane to do more than the technology 
presently allows; correct?
    Mr. Rodrigues. Using technology readiness levels, that's an 
absolutely true statement.
    Mr. Shays. So the issue is if they had to accept the 
technology that existed today, they wouldn't be able to have 
this plane do what they want it to do. So that's the tension. 
But they want the project to keep moving forward. The message 
that I'm hearing from GAO is saying, relax, we're at the 
cutting edge, we're not talking about the cold war where we 
have to rush this to the marketplace.
    Mr. Rodrigues. Absolutely.
    Mr. Shays. So we need to slow down, develop the technology, 
before we start to do the development.
    Mr. Rodrigues. Yes. And the other thing they're trying to 
get across in here is that when we make those decisions to 
accept these unproven technologies into a product development--
and maybe I need to explain product development better. Product 
development is the engineering and manufacturing development 
phase in this context. In the engineering and manufacturing 
development phase, we should be focusing on developing and 
manufacturing the final product. In this case it's the full-up 
plane with everything on it.
    What we do, or what we have done historically in the 
Department of Defense, is we go into that phase, which is 
difficult in and of itself if you're using all proven 
technologies, and to try to integrate a bunch of new 
technologies into a final product is still not an easy process. 
What we do is we allow immature technologies that are pacing 
items. These are defined, the ones we had up here, critical 
technology areas. Critical means they're in the critical path 
to success. We allow that immature technology, unproven 
technology, to enter into the product development phase. What 
you end up with then are long development cycles because now 
you have to bring those technologies along. You have people 
having to focus on technology development when we should be 
focusing on engineering and manufacturing development of a 
product, not the subtechnologies that go into it. As I said, 
that's a challenge by itself.
    And in industry they create a job that's doable by a 
program manager. His job is to bring proven technologies 
together into a form that gives you the product that meets the 
customer's requirements. We expect our program managers to 
manage technology development and product development 
concurrently. And technology development is invention. 
Invention cannot be scheduled. And we set up tight schedules 
where these things have to fit in those schedules. The money is 
all lined up. Money becomes the driver, schedule becomes the 
driver. And what we have actually accomplished tends to fall by 
the wayside.
    Mr. Shays. Let me recognize Mr. Blagojevich, the ranking 
member. I didn't--if you have an opening statement you want to 
make I'm happy to have you do it, or we can get right into 
questions. But he's your witness and you have him for as long 
as you need him.
    Mr. Blagojevich. Great. Thank you, Mr. Chairman I'll just 
dispense with the opening statement and ask a couple of 
questions of Mr. Rodrigues.
    [The prepared statement of Hon. Rod R. Blagojevich 
follows:]

[GRAPHIC] [TIFF OMITTED] T0782.025

[GRAPHIC] [TIFF OMITTED] T0782.026

[GRAPHIC] [TIFF OMITTED] T0782.027

    Mr. Blagojevich. Mr. Rodrigues, the chairman briefly 
mentioned that if we slow down the effort to get into the 
engineering and manufacturing phase--that's essentially the 
gist of what your argument is--that does not necessarily slow 
down the entire program. And it's conceivable, is it not, that 
if we slow down in the technological development phase, that in 
the long run the time saved early on to get it right could 
actually shorten the process? Can you speak to that.
    Mr. Rodrigues. That's exactly the fundamentals behind 
everything in the practices that we're laying out from the work 
that we have done on best commercial practices: that putting 
the time in to get the technology match up front makes the 
product development cycle doable. Instead of having programs 
that extend for 10 and 12 years, that are absolutely 
unmanageable--if I could share with you--would you put up the 
chart on the 10 years? This is what happens to you as programs 
stretch and grow--and this is DOD data.
    Now, I would contend that it's very optimistic, but what 
their data show is that on average you'd have an 11-year 
program cycle. In 11 years you go through 4 program managers, 5 
program executive officers, 8 service acquisition executives, 8 
defense acquisition executives, 5 chairmen of the Joint Chiefs 
of Staff, 7 Secretaries of Defense, 3 Presidents, and 11 cycles 
of coming up here to get money and going through the Department 
to get money. And we wonder why programs drag on and take 
forever?
    One of the problems we have is our cycle times becoming so 
long in the product development because we're doing technology 
development and you should expect problems. We see them. I 
mean, this body approved entry into production on the F-22 with 
a single flight hour. And we expect that to stay on schedule? 
And we expect that to stay on cost? We knew virtually nothing 
about the capability of that plane in terms of demonstration.
    And what we have are a lot of hopes pinned on judgment 
about an ability to deliver those things. Judgment is not good 
enough in a best commercial practice. Demonstration is what 
counts. If you don't demonstrate it, you end up with 12-year 
cycles and you end up with these problems and you end up with 
absolutely no accountability.
    We have to match cycle time to program managers' tenures to 
be able to hold people accountable. Give them a job that's 
doable, which is product development when we're in product 
development. And the only way you can do that is separating 
technology development from product development. And the only 
way you can legitimately do that is through a metric that 
allows you to absolutely measure demonstration.
    Mr. Blagojevich. I know that chart--three Presidents, seven 
Secretaries of Defense, four program managers, the only ones 
that last 11 years are generally the Members of Congress.
    Mr. Rodrigues. And, unfortunately, me.
    Mr. Blagojevich. And you. Can you talk about the importance 
of the continuity of keeping a program manager involved in a 
program like this, how it works in the commercial sector and 
how successful that is to have that continuity?
    Mr. Rodrigues. Clearly, as I look at these and try to 
figure out what can you control here, the only one you could 
really control and would really have a direct effect is the 
program manager. Program executive officers have a whole 
portfolio; some of the others are political appointees, you 
can't control that. But the program manager tenure can be 
controlled. And if you don't have that, what you have is a 
situation we're in now. The Joint Strike Fighter is on its 
third program manager. It's only been around for 3\1/2\ years. 
It's on its third. The third one is there now. His job is to 
deliver this program into engineering and manufacturing 
development as scheduled. The money is there. They're ready to 
go. Everything is lined up. It's in all their outyear budgets.
    We talk about wonderful plans for modernization and the 
need to be able to get a new plane. And I agree with that. We 
need to get new planes out there. My problem with the way we're 
going to do this one is it won't get there in time. It won't 
get there in quantity and we won't reach modernization. This 
isn't what's going to get you there.
    In terms of matching that program manager's tenure--and it 
is what gives you accountability, we have to give these people 
doable jobs. Put them in there for the tenure of the program to 
deliver the product so we can get focused on the product. Our 
focus now is on the next increment of funding, the next 
milestone. How can people stay focused on delivery of a product 
that's 12 years away? I can't think of anybody. To tell you the 
truth I have been doing this, leading this work for over 12 
years. I could not have focused 12 years ago on where I am 
today. There is no way. If you told me this is what I had to 
do, I would have told you you were crazy. It just doesn't 
happen. You focus in shorter increments. People don't think 
that way.
    Industry has gone from basically an 18-month to a 4\1/2\ 
month cycle. The reason for that is to get that focus on 
product, give people doable jobs, get the stuff out there 
quickly because stuff turns over fast in terms of technologies. 
And rather than have these huge cycle times that lead to 
obsolescence--I mean, we're spending hundreds of millions of 
dollars right now, on the F-22, to buy parts that are already 
obsolete. And we haven't even built the plane. When you're 
looking at 12- to 15-year cycle times, that's what's going to 
happen to you. We have to shorten that.
    I would submit as we did during a hearing before the Senate 
Arms Services Committee, Subcommittee on Readiness and 
Management--that in developing a model that would deal with 
these issues. One of the things that you have to do is you have 
to put a strict limit on the engineering and manufacturing 
development or the product development phase. And 5 years, I 
think Dr. Gansler has said 67 months at this point, is their 
goal. Five years is a doable thing. It allows you to match 
tenure. It brings accountability. It brings focus to the 
process. And it forces you to do trades on the front end. If 
you have a limit that you're going to be held accountable to, 
then you will have to do the trades of the technology and costs 
on the front end that allows you to do a deliverable product in 
a 5-year cycle.
    Mr. Blagojevich. One final question, Mr. Rodrigues. We like 
the Joint Strike Fighter, you and I. You certainly--you have 
said you do. I certainly think it's a good idea. We're for 
this. We want to see this get done sooner rather than later. 
It's good for our military, good for our national defense. We 
agree to that. Right?
    Mr. Rodrigues. Right.
    Mr. Blagojevich. What do you say to the argument that if we 
adopt what you're recommending as opposed to what the 
Department of Defense is recommending, this can hamper the 
ability to keep the support here in the Congress to be able to 
fund this program?
    Mr. Rodrigues. Unfortunately, I think that's a real risk. I 
mean, it requires an understanding on the part of the Members. 
The fact of the matter is if we launch with immature 
technologies, you will not get this program on the schedule or 
on the cost. We will end up with the problems that we have had 
historically by going into product development with immature 
technology. You cannot schedule invention. It will create 
problems. And if they're critical path items, they become the 
pacing item. And the cost of that is phenomenal.
    Let me give you some idea of this. Right now on the Joint 
Strike Fighter if you were to annualize numbers and take a look 
at it on a single contractor--because that's where we're going 
to go in EMD, down to one contractor--right now in the risk 
reduction phase, we're spending the equivalent of about $265 
million a year to do risk reduction. That includes the demo and 
all the other risk reduction activities on these critical 
technologies. To fail at $265 million a year on something is 
kind of OK. When we go into EMD, our costs for a single 
contractor immediately jumps to an annualized rate of $1.2 
billion and within 2 years of that we'll be at $4 billion a 
year.
    Now, you have a problem with a critical path technology at 
that point you are carrying a $4 billion program on your back. 
Not $265 million program, where you can tolerate some problems 
in invention. And you know the basic cost tradeoffs are for 
every dollar mistake I make in the risk reduction or technology 
development phase, if I correct that mistake in the engineering 
and manufacturing phase it's going to cost me $10, and if I 
wait until the production phase to correct that same problem--
which the Department is doing on the F-22s without having not 
completed the development phase--here that $1 mistake now costs 
you $100 to correct. So this has to do with bringing 
affordability and discipline to the process to get what we 
need, which is a modernized fleet. And I just don't see where 
following the practices of the past are going to get us to 
where we need to be.
    Mr. Blagojevich. Thank you, Mr. Rodrigues.
    Mr. Shays. Thank you. Mr. Terry.
    Mr. Terry. This is a very interesting discussion. Before I 
ask a couple of my questions I'll say that I took a tour here a 
couple weeks ago of a company in my hometown that's one of the 
leading high-tech support for our private sector industries 
across the Nation. I mean--and they made a comment during their 
presentation when I was being wowed by their technology. They 
said, ``Our clients expect us to be cutting edge but not 
bleeding edge.''
    And as I've listened to your testimony and thoughts about 
immature technology, that's really what we're talking about is 
getting ahead of ourselves in the technology and perhaps 
slowing down.
    I've got several questions but I'll narrow it down to a 
couple here. And one is, just so I can put it in kind of a 
economical or simple terms, I see a catch-22 here where we in 
Congress expect out of our Department of Defense, and I think 
the people of this country expect if anyone is going to be on 
the bleeding edge or that's acceptable, it would be our 
Department of Defense. They want that type of technology in 
there.
    Now, assuming that they have those type of political 
pressures, how do they develop a more acceptable best 
commercial practice time line? It seems to me that they never 
can. How do we do that?
    Mr. Rodrigues. Right.
    Mr. Terry. Taking in the desire to assure the public that 
we're using best commercial practices, but then also accepting 
the role that we want them to be on the far beyond cutting edge 
to maybe the bleeding edge of technology that's always moving.
    Mr. Rodrigues. Right. I don't see a disparity between these 
two things. What we're talking about is how you go about 
getting a better product. It's really about how do you get 
those technologies, those more advanced technologies in the 
field faster. Do you do it through a more iterative process or 
do you go for everything that's out there. The latter takes us 
20 years to get there and, by the time you get there, you can't 
support it. It's obsolete. You have got to buy out stuff 
because nobody builds this anymore. It really is about getting 
down to the basics of changing the culture and the incentives.
    But the other part of it is, you can be on the bleeding 
edge but be on it in technology development. Go chase those 
technologies. You need parameters around what you're going to 
invest in, how you're going to manage that technology base and 
how you're going to graduate that technology to a product, and 
when is the right time to bring it into a product. I'm not 
saying don't chase technologies. I'm saying don't chase it as 
part of a product development. That has cost way too much 
money.
    Mr. Terry. And hence, then, you're doing your job in 
showing us or talking about the best commercial practices and 
how to institute that in the system. But then how do we 
justify, as Congressmen or the President or Department of 
Defense, you know, all right, we identify what we need in this 
plane, let's manufacture it today in our Joint Strike Fighter 
force. Then the technology changes 2 or 4 years after 
production. That makes it obsolete and we end up spending more 
money in a production of the next generation, sooner than we 
probably should have. How do we justify that? I'm kind of 
asking you a political question. That's not your realm here.
    Mr. Rodrigues. I think the Department is moving to--and 
probably Mr. Soloway will talk about it--is the problem with 
the rapid changes in technology. It's a reality. You can't 
change that. People have to recognize that that's the 
situation. And then what we really need to do is go to a more 
iterative process in how we build things. In other words, we 
don't decide today on a firm design that we're going to build 
for 20 years. I mean, that isn't how the world works. In fact, 
that isn't how we really build planes. If you look at the F-16, 
we built A's and B's and block 10 and block 20, block 30, block 
40, block 50. Now we're at block 60. Now we're getting ready to 
build block 60-I for somebody else. I mean, in reality that's 
what you end up doing. Things change and you make those changes 
in the subsystems. But that basic design that you commit to has 
to last longer than just a few planes so you can get your money 
back out of it.
    So there are ways to do this. To me, it mostly rests with 
the Department. It's what they come up here and sell. 
Unfortunately, we have a system now that incentivizes everybody 
to come up here and promise you the world and promise you all 
these wonderful things and lay out way early in a process an 
exact commitment to schedules and costs, build all that funding 
in. And you know when they're doing that? They're doing that 
before they even get into that technology development phase. 
They don't have the slightest idea of whether those things are 
going to work or what they're going to cost. But we lock into a 
commitment early on.
    And one of the things that needs to be done is to delay the 
commitment until you have the technology match. We shouldn't 
allow people, to sell a product based on a technology that's 
absolutely unproven. We're raising everyone's expectations to 
levels that are going to----
    Mr. Terry. And during your presentation when you were 
talking about that and answering the chairman's questions, I 
wrote a note down: So what responsibility does Congress have? 
Because I think we encourage that. So maybe we should ask you 
to do a GAO report on what type of congressional or political 
reforms we need in house so we don't encourage that type. I 
agree that in trying to abide businesslike procedures, we don't 
encourage them from our side to do it either.
    Mr. Rodrigues. We have done some pieces on this whole issue 
of what incentivizes the culture and the process at DOD, and 
Congress is a big player on that.
    Mr. Terry. Before we run and vote, let me ask this one 
simple question of you. In your testimony you had mentioned and 
use kind of colloquial language here, but they said--they're 
talking about adopting the best commercial practices, they're 
talking the talk but they aren't walking it yet. How do we 
ensure that they are implementing these type of best commercial 
practices at the highest level instead of just talking about 
it?
    Mr. Rodrigues. I think you have to do it program by 
program. And the Joint Strike Fighter is the lead program and 
it is the one to do it on. And in the report I believe that 
you're going to release here at the hearing, we have matters 
for congressional consideration. And those matters for 
consideration spell out what we think you should require the 
Secretary of Defense to lay out for you--and we need a metric 
that is understandable.
    Let me talk about technology readiness levels. They are 
actually very simple. I think that's why people don't like them 
because you'll actually--you will understand them. And to 
simply say what they are, there are nine levels. What we're 
talking about is progressing to the level of 7 for entry into 
product development or engineering and manufacturing 
development. The first three levels are basically paper 
studies.
    Now, I don't know how many people want to launch a whole 
big aircraft program based on a paper study but I'm not too 
fond of it--paper study of a new technology. Not a good idea. 
The next two levels, 4 and 5 are basically laboratory hardware 
demonstrated in a laboratory environment. These are pretty easy 
things. I don't know how many labs you've been out to, but I've 
been to laser labs and I was very disappointed. I thought I 
would see a laser. All I see is these parts spread all over 
tables. They're hand wired. They can do the stuff you have to 
do in a lab. It's only in that lab environment. It's not going 
to be flying on a plane like we want it to, or space aircraft.
    And then to get to level 6 and 7, you actually have to get 
that big thing that's spread out in a lab down to the form and 
the fit that it will have to be in in the intended product. You 
have to have actually taken it out into the environment you're 
going to use it in. That doesn't mean you put it on the plane 
it's going to be on, because the plane doesn't exist. That 
means you put it on a plane, if its something that has to fly, 
and you actually fly it.
    So now you have the form and fit that you're going to need 
for the product, you put it into the environment--we're not 
talking about operational testing. We're saying understand the 
environment. That's level 6 and 7. These things are 
understandable. They are things that you can measure fairly 
readily. They are not subject to engineering judgment or 
consensus, which is what we have tended to use or what we have 
used in the past that gets us into so much trouble. And they 
are things that you as a board of directors need to begin to 
understand and to focus on and to hold the Joint Strike Fighter 
to.
    Mr. Terry. What is the timeframe for the product 
development timeframe?
    Mr. Rodrigues. Right now they're planning on going into----
    Mr. Terry. That you would recommend.
    Mr. Rodrigues. Oh, I would recommend that we limit it to 5 
years so we can bring accountability to the process. It would 
be hard with an 8 or 10-year cycle to say we're going to put a 
program manager in there and we expect him to stay there for 10 
years. I mean, life just doesn't happen that way.
    Mr. Terry. Thank you.
    Mr. Shays. I'm sorry, we're going to have to keep you here. 
We'll have just a few more questions afterwards. We have to 
vote. We stand in recess. We'll try and be back in 10 minutes.
    [Recess.]
    Mr. Shays. Mr. Rodrigues, the hearing is reopened. I'm 
curious, that took me 15 minutes? So next time I'll say 15. I'm 
sorry.
    I want to just establish some basic points and I'm not 
looking for long answers. One of the points I want to establish 
is to understand what I think I understand but to just have you 
describe it. It's not an uncommon practice for a building to be 
built before it's been fully designed. And the argument is you 
can build it faster and you can save yourself from increased 
costs, inflation rate and so on.
    I make the assumption that when you're building a building 
and you haven't designed it, you still have all the technology 
there. So that kind of argument isn't compatible where you kind 
of build before you've totally designed. Here we're trying to 
make sure that we know that technology exists, obviously, 
before production, but even before development; is that 
correct?
    Mr. Rodrigues. Yes. Before product development, yes.
    Mr. Shays. And what I want to understand as well is that 
with the F-22, that basically is a new airplane, we're not 
following this practice that we're following with Joint Strike 
Fighter. The F-18 which we are--which the--E and F, and F is a 
modification, a larger F-18, but a modification of existing 
technology. It's not a new plane, correct?
    Mr. Rodrigues. There's very little commonality between the 
F-18C/D and the F-18E/F. The Department sold that as an 
ungraded F-18, but there's nothing the same about it.
    Mr. Shays. So I didn't get the answer I expected. But 
basically you consider it a pretty new plane.
    Mr. Rodrigues. Yeah.
    Mr. Shays. We're building the B-22 but we made a decision 
that we would not build the AFX which was to replace the A-12 
Navy attack plane. We dropped the multi-role fighter. So we are 
now going to be developing the JSF and it is going to have to--
the JSF is going to have to fulfill a lot of very different 
roles. It's going to be used by the Air Force in conventional 
takeoff and landing. It's going to be used by the Navy for 
shorter takeoffs on carriers, and my understanding is it has to 
be a tougher plane and it's going to weigh a little more. And 
then we are going to be using it for the Marine Corps and the 
U.K. Royal Navy as a vertical takeoff, like the Harrier jet. 
And yet it's all coming from one program. So there are a lot of 
technologies that are in play here.
    Your bottom line point is that we should not develop and 
produce this plane until we follow the game plan, which is to 
make sure the technology exists and that we use best commercial 
practices. The first question that I want to ask in regards to 
best commercial practices, what best practices would not be 
appropriate in the DOD acquisition process? Can you think of 
any?
    Mr. Rodrigues. No I can't think of any at all. Really what 
it then comes down to--and I think it was something that came 
out in an earlier question--it really comes down to how do you 
incentivize the process to put these things in place. What do 
we do to make it so that bringing knowledge to the table rather 
than judgment, to really be focused on setting a product 
development up for absolute success, how do you create the 
incentives to do that, to not oversell, to not overpromise, to 
not overcommit, but do what technology allows?
    Mr. Shays. How much cheaper is it to wait now to develop 
the technology than to begin to develop or produce this plane 
without the technology?
    Mr. Rodrigues. As I said, the rules of thumb are if you run 
into problems when you're in technology development, $1 problem 
there becomes a $10 problem when you enter into product 
development. And if you actually get all the way to 
manufacturing or production and you're still doing technology 
and you're having problems, that same dollar problem now 
becomes a $100 problem. So----
    Mr. Shays. So a 10-to-1 ratio and then a 100-to-1 ratio.
    Mr. Rodrigues. Order of magnitude increases, yes. And some 
estimates are much higher than that. Commercial industry, when 
you're doing huge runs of production items where you have to do 
recalls, those numbers increase dramatically.
    Mr. Shays. Now, to the best of your knowledge--you're 
speculating--what would motivate, in your judgment, the 
military, the DOD to move forward? And if--I always believe 
there are logical reasons why they want to move forward. Do you 
think it's the potential that they think Congress might 
withdraw if they're not heavily committed, and therefore they'd 
rather get the plane this way, even if it costs more and they 
get less planes? Because if it costs more they're going to get 
less planes.
    Mr. Rodrigues. Right.
    Mr. Shays. Or is there another reason that I'm not 
thinking?
    Mr. Rodrigues. No, I think that is it. It is the 
incentives. The money is in place. It's very difficult to get 
money in wedges and approval and get something going. I mean, 
picture a program manager that comes in now. His job at this 
point in time on something like the Joint Strike Fighter, is to 
get the program into engineering and manufacturing development 
on schedule. The money is all lined up. In fact, the program 
manager leaves as the down select is made, and the process is 
handed over to somebody else who is going to have to worry 
about this.
    But let's imagine that he takes a look at it and he says, 
well, wait a second, you know, some of these technologies we 
really don't know all that much about it, so what I want to do 
is propose a delay. Well, now he has to go through the process 
of the Department of Defense where there are all kinds of 
people competing for these limited dollars waiting to find a 
place where they can take it from. And the Department does it 
itself all the time. Then it comes up here to the Congress and 
there are a lot of people looking to fund other things and 
there's only so much money to go around. That risk is 
absolutely real.
    Mr. Shays. It's a real risk but it's even more of a risk if 
the F-22 and the F/A-18E and F are going to be much more costly 
than we anticipated because they're not following these kind of 
practices. So in a sense their Joint Strike Fighter is going to 
be following a process that should save money in the long run 
or certainly not add to cost, competing with 2 other weapons 
systems that may gobble up costs. So I have sympathy for that 
if that's it.
    Then I'll just ask this last question. What is magical 
about a date, these time--I mean, is there--you haven't 
developed any logic that says that a cold war enemy or a non-
cold war enemy is going to be able to beat us and supercede us. 
So from that standpoint we don't have a rush; correct?
    Mr. Rodrigues. Right. The objective was to get an 
affordable family of next generation aircraft. The key was 
affordable. That's not basically threat driven at that point.
    Mr. Shays. I'm going to just say it can be refuted by DOD. 
But the bottom line is one of the potential luxuries of the 
cold war ending--I consider the world a more dangerous place 
but for other reasons--but the cold war has ended, we have some 
ability to develop the technology or a lot of ability to 
develop the technology before we go into development and 
production.
    Mr. Rodrigues. Absolutely. But going back, if you don't 
have the technology match, the fact that there is some threat 
out there you need to deal with doesn't justify the mismatch of 
technology because you're going to run into problems when you 
try to build that thing. It is going to take you longer and 
longer and cost you more and more.
    Mr. Shays. So even if the cold war hadn't ended, you would 
be arguing the same thing.
    Mr. Rodrigues. I would be arguing for some constraint in 
what you do as you move forward. There are ways to deal with 
that. There are some things where we don't have capability--and 
I can't talk about those where there are real threats out there 
that we can't deal with, particularly in electronics--where we 
would go ahead with a very limited scale of something to get 
some kind of capability, even if we know it won't meet the 
whole thing. It gives us something in the short run to be able 
to deal with part of the problem. There are cases where you can 
make a case for why you would want to take those risks. I don't 
see those in the Joint Strike Fighter.
    Mr. Shays. Thank you. Mrs. Biggert has joined us. I yield 
to her--not yield to her but give her the floor.
    Mrs. Biggert. Thank you, Mr. Chairman. I just have one 
question. I'm sorry I missed the testimony. But yesterday the 
Senate Armed Forces Committee deleted the engineering and 
preliminary manufacturing money but said that the Pentagon 
could get the $424 million back if the plane proved it was 
ready to take the next step. Do you think that this will be a 
possibility?
    Mr. Rodrigues. That they'll take the next step?
    Mrs. Biggert. Yes, that----
    Mr. Rodrigues. That they'll be able to accomplish that, 
take the next step within this year? Not really; no, I don't. 
Not based on where they are in technologies, provided those 
technologies remain untradable from meeting the affordability 
goal. As long as those technologies are there--I don't believe 
you could get it done in a year. Could they get lucky and do 
it? I suppose. I don't think they have the demonstrations laid 
out at this point that would get them to the appropriate 
technology level to have that assurance. I just don't see how 
that would happen.
    Mrs. Biggert. Thank you. Thank you, Mr. Chairman.
    Mr. Shays. Thank you. Mr. Terry.
    Mr. Terry. No.
    Mr. Shays. Is there any last--is there a question we should 
have asked that you would have liked us to ask or any point 
that you want to make?
    Mr. Rodrigues. There is one point that I skipped, actually 
two things. One is when we talked about the differences as far 
as we're concerned in the scoring of the technologies and the 
Department's position on that. I do want to make you aware that 
we had a closed hearing and we went through those technologies.
    And could you put that back up again?
    I want to point one thing out. One of the basic arguments 
was that there was a misunderstanding that when we scored the 
technologies they thought we were saying in order for it to get 
to a level 7, the level needed to enter EMD, that it had to be 
on the Joint Strike Fighter aircraft. As I said, that would be 
an impossibility, obviously not the standard. We spent a great 
deal of time working with the people who apply this at the 
contractor's plants and the program office to get this clear 
understanding that we were talking about demonstrating 
something in close form and fit in the environment; not on the 
Joint Strike Fighter, on some kind of surrogate platform.
    But if you look at this, you can see a number of those 
technologies are projected only to be at readiness level 4; 
technology area 7, technology area 3, and technology area 2. As 
I said earlier, level 4 is not form and fit, it is laboratory 
hardware in a laboratory environment. So this argument about, 
oh, there was confusion about scoring this because we thought 
you meant you had to have it on a Joint Strike Fighter doesn't 
even come into play in the scoring of those technologies.
    And I can tell you we made that absolutely clear. So there 
is no misunderstanding.
    The other thing is what can Congress do or what should you 
do? I think we should require them to be held to a level 7 of 
demonstration. And in those cases where they can or believe 
that they should move without that, they should have to provide 
a very discreet--first of all, that they score it properly and 
own up to the scoring. Let's not play games. These are pretty 
clear things to score. There is no confusion.
    And once we score it, if the scoring comes out less than a 
level 7, the only way they should be able to move forward is 
make it clear to you where they are on those critical 
technologies and then explain to you why it is that we need to 
take on that additional risk of moving forward without having 
demonstrated the technologies that are going to be critical to 
the building of that final product, taking those technologies 
and having the very difficult task of trying to integrate a 
whole bunch of technologies into a final product.
    Mr. Shays. You wanted to make that point, but it does raise 
a question now. And the question it raises for me is, is this a 
package deal? Do all have to be there in order to go to that 
next step of development or do we isolate each one of these 
technologies as a separate issue before we move forward?
    Mr. Rodrigues. As long as they're on the critical path, the 
pacing item in any area, whatever it is--the long pole in the 
tent is what you have to worry about.
    Mr. Shays. Listen to the question I'm asking. Are they 
independent or do I have to take them all as a package?
    Mr. Rodrigues. No, they're independent. You do each one.
    Mr. Shays. Right. I anticipated that would be the answer 
but I just wanted to make sure.
    Are you at liberty to tell us which areas have the greatest 
challenge right now? Is this a proprietary issue or not? I 
can't imagine it would be. Without identifying which is one and 
which is two and so on, can you tell us the area where we're 
doing the best and where we're potentially doing the worst?
    Mr. Rodrigues. Clearly, the best----
    Mr. Shays. Let me ask a question. Is that an uncomfortable 
question?
    Mr. Rodrigues. I can say clearly from this, the best are 
the ones that would be technology area 1 which I think you have 
a thing that tells you what that is. I can't--I don't want to 
say.
    Mr. Shays. Fair enough.
    Mr. Rodrigues. But clearly one and eight.
    Mr. Shays. Someone will have to explain to me, and I should 
have clarified before but I'm not going to push the point, why 
it even matters, why we can't have that dialog. But, fair 
enough. I guess because we haven't made a choice on who gets 
what.
    Mr. Rodrigues. Both contractors don't work the same 
technologies.
    Mr. Shays. That's fair. You've been wonderful. You've put 
the ball in play and you've given an opportunity for those that 
follow to answer your points. I don't know if you or someone 
else can stay to hear the other presentations because we might 
seek to have your office respond. You preferably, but someone 
else.
    Mr. Rodrigues. I'll stay.
    Mr. Shays. We'll go to our second panel.
    Is that OK?
    We'll go to panel two. And I thank you, Mr. Rodrigues.
    We have Mr. Stan Soloway who is the Deputy Secretary of 
Defense Acquisition Reform, Department of Defense, and Major 
General Raymond Huot, U.S. Air Force Acquisition Program, 
Department of Defense. If you both would stand, we'll swear you 
in. What I think I will do is just slide you over a little bit. 
Mr. Soloway, if you could just move your chair a little bit so 
we give Mr. Huot--General Huot, I'm sorry General. If you raise 
your right hands, please.
    [Witnesses sworn.]
    Mr. Shays. Thank you. Gentlemen, it's great to have you 
here. I appreciate your spending the time with us. And I want 
to make sure that, Mr. Soloway, we have you first. Am I 
breaking protocol or am I keeping protocol?
    Mr. Soloway. No, that's correct.
    Mr. Shays. I'm in charge. Well, Mr. Soloway, we'll start 
with you. Thank you for being here.

  STATEMENTS OF STAN Z. SOLOWAY, DEPUTY SECRETARY OF DEFENSE, 
 ACQUISITION REFORM, DEPARTMENT OF DEFENSE; AND MAJOR GENERAL 
RAYMOND HUOT, U.S. AIR FORCE, ACQUISITION PROGRAMS, DEPARTMENT 
                           OF DEFENSE

    Mr. Soloway. Thank you, Mr. Chairman and members of the 
committee. It's a pleasure for me to be here today. If I could 
just divert for a moment from my prepared text and say that I 
do believe that there is extraordinary commonality between what 
the GAO is recommending and what we are looking at in terms of 
a strategy of how to deal with the issues before us, and 
hopefully we'll have a robust discussion of that as we move 
forward.
    But I am pleased to be here to have this opportunity to 
discuss with you our continued progress with acquisition reform 
and particularly how it relates and applies to the Department's 
Joint Strike Fighter program.
    As you know, acquisition reform has been a top priority for 
the Department for the last several years and encompasses a 
wide range of initiatives and has had many real successes. Let 
me list just a few. One excellent example is the Joint Direct 
Attack Munition [JDAM], which performed so flawlessly in 
Kosovo. Designated as an acquisition pilot program, JDAM was 
originally expected to cost in excess of $40,000 per unit, but 
through a combination of acquisition reforms and focused 
innovative program management we can now purchase JDAM for less 
than half that amount.
    Mr. Shays. That's the unmanned plane.
    Mr. Soloway. No, JDAM is essentially a guidance, is the 
best way to describe it.
    General Huot. Probably the best way to say it, it's a 
strap-on kit for a general purpose bomb. It gives it all 
weather capability. INS, Inertial Navigational System, with 
global positioning system guidance.
    Mr. Shays. Gotcha. I've seen it and I appreciate having it 
clarified.
    Mr. Soloway. Then there is the precision location GPS 
receiver or PLGR. This receiver, purchased largely through the 
new commercial buying authorities contained in the Federal 
Acquisition Streamlining Act, replaces a previous field version 
built to extensive military specifications that weighed over 30 
pounds, required two operators, had only one channel and cost 
us thousands of dollars per unit. PLGR, on the other hand, 
requires only one operator, has five channels, weighs just over 
2 pounds and costs less than $1,000 per unit.
    The All Ordnance Destruct System [AODS], is a flight 
termination system used on the rocket system launch program 
vehicles at the Space and Missile System Center. The majority 
of these vehicles are used as targets in support of the 
ballistic missile defense programs. By utilizing the commercial 
buying authority known as FAR, Part 12, unit costs were reduced 
from the previous purchase price of $900,000 a kit down to 
$55,000 a kit, but more importantly, the new kits are also 
technically superior.
    Today, circuit cards for the avionics in the F-22 are being 
produced largely on a commercial line at TRW, thus saving the 
Department significant resources that would have had to be 
devoted to unique development and production facilities. 
Moreover, the reliability testing on those circuit cards have 
demonstrated excellent results and costs appear to be 
significantly lower than expected, 55 percent to 70 percent 
less than their military standardized counterpart.
    Acquisition reform is also central to the development of 
the Navy's new Virginia Class Attack Submarine. Key to the 
success of the program has been the use of integrated product 
and process development teams, the use of open systems 
architecture, and the insertion of commercial off-the-shelf 
technologies. The Navy will benefit there with a cost avoidance 
of $30 million per ship but, more importantly, the Virginia 
Class will operate at a 32 percent lower total ownership cost 
than the comparable Seawolf.
    Indeed, much has changed and for the better. Given the 
complexity of our business practices and the entrenched 
cultures we have inside and outside of the government, I 
believe we have made substantial progress.
    But clearly we must do more. The security environment we 
face is unpredictable and unstable. And our success in meeting 
the challenges of the battlefield of the near and more distant 
future will hinge in large part on our ability to access and 
integrate true cutting-edge technologies that provide us the 
dominance, speed, and scope of information that we need.
    One of the means for accomplishing these goals is through a 
restructuring of how we develop, manufacture, and maintain our 
weapon systems. It is no secret that cycle times for new weapon 
systems from concept to fielding remain unacceptably high and 
that such long cycle times too often result in the fielding of 
already obsolete technologies. Since some technology decisions 
must be made early in a program, it is clear that our history 
of taking 15 or more years to field new systems is not at all 
consonant with the torrid pace of technology change we see 
today.
    There are, of course, many reasons for these long cycle 
times. Key among them is often the very nature of the 
requirements set forth for any individual program. 
Traditionally, our requirements have been both inflexible and 
involved extraordinary technology challenges that can take many 
years to meet. This is beginning to change. Today both our 
operational and acquisition and technology communities 
recognize that to optimize support for our men and women in the 
field and to most responsibly steward the public's tax dollars, 
we need to institute new requirements and acquisition 
strategies. Indeed, we are confident we can significantly 
reduce cycle times and costs and provide items for the 
warfighter faster, for a more flexible evolutionary approach.
    How we do this, how we develop, manufacture, and maintain 
systems is based on the Department's 5000 Series documents, our 
bible for systems acquisition, which we are currently 
rewriting. The DOD 5000 rewrite will drive the Department 
further toward evolutionary acquisition and increase our focus 
on flexibility and requirements. Additionally, the new 5000 
will require greater technology maturity prior to entering the 
manufacturing phase of a program.
    In the new systems acquisition environment, key acquisition 
decisions and long-term funding commitments may not be made 
until technology shows the required maturity and risks are 
better understood and mitigated than has traditionally been the 
case. And the JSF program is a forerunner of this new approach. 
Indeed, since its inception, the JSF program has been 
recognized for utilizing and actually pioneering many 
acquisition reform concepts and applying them to the actual 
business processes and contract vehicles being utilized, 
including but not limited to the critical precept of the new 
5000 series.
    For instance, modeling and simulation has been proven in 
both industry and government to help reduce the time resources, 
and ultimately risk, associated with systems development. 
Representations of proposed systems, basically virtual 
prototypes, are embedded in realistic, synthetic environments 
to support the various phases of the acquisition process. The 
JSF program has made extensive use of M&S in the requirements 
development process and is continuing the use of M&S with 
mission level virtual pilot-in-the-loop simulation to support a 
more thorough evaluation of required avionics capability.
    In keeping with the best proven practice in the commercial 
technology world, virtual manufacturing and virtual maintenance 
are also being pursued to facilitate planning and drive down 
the associated cost.
    The JSF is also incorporating an evolutionary acquisition 
strategy. In this process, the warfighter and the buyer work 
side by side to facilitate a better understanding of the 
requirements and decisions on tradeoffs between performance and 
cost. Specifying operational requirements in an incremental 
manner phased over time and matching them against the projected 
threat and available technologies have allowed the JSF program 
to exercise thoughtful judgment in balancing cost, schedule and 
performance.
    The use of evolutionary acquisition further mitigates risks 
by allowing technologies to be inserted as they mature.
    As I mentioned before, probably the most important change 
in the new 5000 rewrite is the emphasis on technology maturity 
before entering into system integration, or what is today known 
as engineering and manufacturing development. Of course, 
transition into EMD is a challenge in every program. And in the 
JSF program, as in others, it will be up to the design teams 
and the program office along with the Service Acquisition 
Executives and the Defense Acquisition Executives to determine 
the acceptable level of technology readiness prior to EMD 
decision.
    Among the many factors that can help us make overall 
technology readiness assessment are technology readiness levels 
[TRLs], which are used sometimes in the DOD. The minimum TRL 
rating of 1 begins at paper studies, as Mr. Rodrigues said, of 
a technology's basic properties and rises to the maximum rating 
of 9 for a system in its final form, operating under mission 
conditions.
    However, there is no hard and fast rule in DOD and NASA or 
elsewhere in government as to specific threshold TRLs for any 
given decision. Additionally, it should be noted that there are 
more comprehensive methodologies that we do use which can and 
do provide more value as risk management as opposed to risk 
measurement tools. Indeed, the DOD 5000 rewrite does not 
prescribe a required technology readiness level but does 
recommend using TRLs as a tool to help measure the maturity 
level of the technology.
    What the 5000 does prescribe is that technologies be 
demonstrated in a relevant environment with a fallback plan at 
a higher maturity level. In other words, if a far-reaching, 
newer technology does not pass relevant testing, a lesser 
proven technology could be utilized as long as it enables the 
system to still achieve its critical performance requirements.
    Recently the GAO provided a draft report to the Department 
on the JSF that recommends extending the JSF development 
schedule to allow for further maturation of technologies. Their 
recommendation is based on their understanding that critical 
technologies will have inadequate levels of technology maturity 
based on TRLs at the time the EMD contract is to be awarded and 
the decision made in the spring of 2001.
    The GAO report clearly articulates a strategy for systems 
development that we embrace, as evidenced by the revisions to 
the DOD 5000 I mentioned earlier. Indeed, for the most part, we 
are in violent agreement. Where we differ is on the definition 
and applicability of TRLs in the decisionmaking process. GAO's 
position does appear to be that achieving a TRL level of 7 
should be required prior to entering EMD, again as Mr. 
Rodrigues pointed out. Our view, however, is that a level 7 
requires the very kind of systems integration that takes place 
during EMD and that it is infeasible to produce full-scale 
testing of this type prior to that phase.
    Moreover, as noted earlier, we, like NASA and others, see 
TRLs as but one input to the decision process. In fact, as the 
NASA Director of Programs told us, ``NASA does not formally use 
nor rigorously apply the definition of technology readiness 
level, TRL, to its systems development. We generally commit to 
development of operational systems when all critical 
technologies have achieved TRLs of 5 or 6.''
    In assessing a technology prior to EMD, we do seek to 
assess whether the individual technology has been proven or is 
close to being proven in appropriate developmental 
environments. We do not agree that it is necessary or even 
feasible to demonstrate the full integration of the technology 
prior to EMD. Indeed, the GAO report does seem to require that 
all technology must be flown on an actual or prototype JSF 
platform in order to demonstrate adequate maturity for EMD. 
However, much of the JSF avionics and software, for example, 
can and should be demonstrated on the ground at a lower cost. 
Technologies that must be flown for adequate demonstration will 
in fact be or have been flying on the concept demonstrator 
aircraft, commercial aircraft, the F-16, the F-22, the F/A-18E/
F and the Eurofighter.
    So, in short, although the technology will not be 
demonstrated on the actual JSF or a JSF prototype per se, it 
will be tested in a relevant environment. And the success of 
those demonstrations is critical to our confidence in our 
ability to successfully integrate the technologies during EMD. 
And we are confident we will have all critical technologies at 
an adequate and appropriate technology readiness level by the 
time an EMD decision must be made next year. I emphasize the 
decision is not to be made for another year.
    GAO has also expressed concern that if we make a premature 
production decision we could be locked into manufacturing 
processes based on an expected technology capability, thus 
creating the risk that if the integration fails we will face 
exorbitant cost and time in redevelopment.
    We agree there is risk. There is always risk in the 
integration of complex systems. And how the JSF program 
mitigates those risks is the key. As I think you will hear from 
Major General Huot, the risk mitigation initiatives associated 
with JSF have probably been the most comprehensive and 
aggressive of any DOD program ever.
    Further, as I noted before, the rapid maturation of 
modeling and simulation capabilities has enabled the 
development and testing of a wide range of critical 
technologies that might not otherwise have been possible and 
has played a key role in our ability to effectively assess the 
risks on the JSF program. Thus, any decision to proceed into 
EMD and production of the JSF will carefully assess all risks 
against the fallback alternatives.
    Finally, let me be very clear. The strategy of revised 
acquisition process that will be prescribed in the DOD 5000 
rewrite and which is largely being implemented on the JSF does 
represent a very real departure from our traditional approach 
to systems development. In the past, we would indeed consider 
technology development as part of the engineering and 
manufacturing phase. Now we have technology development before 
we make a commitment to--proceed into system development and 
demonstration, and eventually production.
    Mr. Chairman, acquisition reform has been made possible by 
a strong partnership between the Congress and the executive 
branch, and reflects our mutual commitment to ensuring that the 
government operates far more efficiently and effectively than 
has historically been the case. I appreciate having had this 
opportunity to be here today, because as we continue on the 
successful path we have forged, we need the continued support 
and commitment of the Congress.
    We also appreciate the continued support and encouragement 
the GAO has provided as we continue to change the worlds's 
largest buying institution. Our disagreement on the specific 
role of TRLs notwithstanding, we are in close agreement on 
where the DOD's acquisition process must go. We also believe it 
is important to let programs like JSF that are demonstrating 
the acquisition and management reforms of recent years to have 
the flexibility to manage their programs and make decisions 
based on weighing the risks against costs, schedule, and 
technology maturity.
    That concludes my statement. I would be happy to answer any 
of your questions now and certainly stand ready to provide any 
additional information the committee believes would be helpful 
in fostering a clearer understanding of this or other DOD 
programs.
    Mr. Shays. Thank you Mr. Soloway. That was a very helpful 
statement. We did allow you to go over the 10 minutes because I 
think it's important for you and DOD to really state your case 
on the record. Then we can examine your statement.
    [The prepared statement of Mr. Soloway follows:]

    [GRAPHIC] [TIFF OMITTED] T0782.028
    
    [GRAPHIC] [TIFF OMITTED] T0782.029
    
    [GRAPHIC] [TIFF OMITTED] T0782.030
    
    [GRAPHIC] [TIFF OMITTED] T0782.031
    
    [GRAPHIC] [TIFF OMITTED] T0782.032
    
    [GRAPHIC] [TIFF OMITTED] T0782.033
    
    [GRAPHIC] [TIFF OMITTED] T0782.034
    
    [GRAPHIC] [TIFF OMITTED] T0782.035
    
    [GRAPHIC] [TIFF OMITTED] T0782.036
    
    [GRAPHIC] [TIFF OMITTED] T0782.037
    
    [GRAPHIC] [TIFF OMITTED] T0782.038
    
    [GRAPHIC] [TIFF OMITTED] T0782.039
    
    Mr. Shays. I don't think I'll be as generous with the third 
panel. We'll stick with the 10 minutes.
    General, now you only have 2 minutes.
    Mr. Soloway. That's the way it usually works between OSD 
and the services.
    Mr. Shays. General, you have your time. And we welcome your 
testimony.
    General Huot. Thank you, Mr. Chairman and members of the 
committee. I appreciate the opportunity to talk about the Joint 
Strike Fighter program and the significant accomplishments it 
has demonstrated in the area of acquisition reform.
    By way of background, for those of you on the committee who 
I haven't had the opportunity to meet, I have been in the 
operational end of the warfighting business for most of my 
career. I flew the F-105 Thunderchief in combat in Southeast 
Asia during the Vietnam conflict. Since then, I have flown 
several other attack aircraft, the A-7, A-10, F-18, and almost 
every model of the F-16. I was a wing commander during the 
Persian Gulf war. And since then, in my headquarters 
experience, I've been the Deputy Director of Operational 
Requirements and almost 2 years in this acquisition position as 
the mission area director responsible for fighters, bombers and 
munitions, and the JSF program falls into that category.
    Now, the JSF program has been a leader in the 
implementation of acquisition reform since its inception as the 
Joint Advanced Strike Technology Program or the JAST program. 
You may recall that we combined Air Force and Navy requirements 
after we cancelled the multi-role fighter program and the AX, 
helped them meet joint service requirements.
    Now, the need to produce an affordable aircraft drove 
several key decisions early in the program, including a single 
engine, a single seat, and a common family of airplanes for all 
three services. Specific advanced technologies were selected 
and then prioritized based on their contribution to not only 
warfighting benefits but also in terms of life cycle costs.
    We established aggressive cost goals for average unit 
recurring flyaway costs and for engineering and manufacturing 
development. And cost goals are being established even now for 
life cycle costs. These cost goals continue to serve as 
baseline independent variables for requirements and technology 
affordability trades. And I'll talk about that regard in a 
minute because this is a new process that we have gone through. 
It is a robust and highly successful cost and operational 
performance trade process has been implemented by the Joint 
Strike Fighter on a continuous basis in order to help achieve 
those cost goals.
    Participation by the three major stakeholders in this 
process--our warfighters, industry, and the government has been 
key to the success of this cost and operational performance 
trade process. The warfighters, represented by both operators 
and maintainers, continue to provide a clear and unambiguous 
view on fighting concepts and needs as well as expected threats 
in combat conditions.
    And industry provides us with detailed weapons system 
concept and cost information and the program office adds an 
understanding of cost schedule performance supportability and 
resource constraints.
    More importantly, government engineers and analysts provide 
assessments on the cost and performance of contractor concepts. 
The result is that every requirement on this airplane has to 
earn its way onto the airplane based on cost effectiveness or 
cost-benefit analysis.
    Now, in contrast to some programs which are not initiated 
until a formal validated requirements document existed, the 
Joint Strike Fighter program was established so that specific 
weapons system requirements would not be frozen until the 
leveraging cost and operational performance trades had been 
performed and key technologies and concepts had been matured. 
This just-in-time approach to requirements avoided premature 
commitment to requirements that would be excessively costly to 
meet, fail to take advantage of available technology or, 
conversely, depend upon immature technology. In other words, it 
allowed time to work down the cost of the weapons system and 
insures that the requirements are consistent with the available 
technology.
    I would like to spend just a few minutes discussing the 
current phase of the JSF Program Concept Demonstration Phase 
we're in. This phase began in November 1996 and it's scheduled 
to be complete in the spring of next year. The major activities 
during this phase are propulsion system development, 
requirements analysis and definition, the technology maturation 
programs and, of course, building and flying two concept 
demonstrator aircraft per contractor.
    In regards to the concept demonstrator aircraft, I want to 
emphasize a couple points to the committee. These aircraft are 
concept demonstrator aircraft. They are not and were never 
intended to be prototypes. These demonstrator aircraft are 
required in the concept demonstration phase to accomplish three 
very specific objectives: One, to demonstrate a high degree of 
commonality across all three common service variants; to 
demonstrate short takeoff, the vertical landing, hover and 
transition to and from forward flight; and then demonstrate 
satisfactory low-speed carrier approach flying and handling 
qualities.
    In fact, there are other items to be demonstrated by the 
contractors on these airplanes but these are for competitive 
advantage purposes. But the hard requirements are the three 
things I listed above.
    We are totally convinced the JSF program office and our 
contractor teams have done a great job in ensuring that the 
concept development phase of this program will demonstrate a 
low level of technology risk for critical enabling technology 
and processes prior to entering EMD in the early part of next 
year on the current schedule.
    This has been accomplished through a very rigorous and 
disciplined methodology for risk assessment, risk management, 
and risk reduction. Now, Pratt & Whitney is developing engines 
for both the JSF contractors' demonstrator aircraft based on 
their highly successful development program for the F-22. The 
result here is a high degree of commonality not only among the 
JSF contractors but also between the Joint Strike Fighter and 
the F-22. The F-119 engine core is essentially the same for 
both the F-22 and the Joint Strike Fighter. This commonality 
lowers JSF risk, development time, life cycle cost, and 
accelerates that propulsion system maturity so we can provide a 
safer airplane for our warfighters.
    Under the current schedule, in fact, we expect to have over 
500,000 hours of operational F-22 engine time before we have 
that engine at initial operational capability in the JSF. This 
is more than double what we had on the F-100 engine in the F-15 
when we transitioned to the single engine F-16.
    I would like to just talk a little bit about the last two 
concept demonstrator--concept development goals--technology 
maturation and concept demonstrator aircraft, a bit further. 
The JSF program identified numerous technology maturation 
efforts to ensure low risk into engineering and manufacturing 
development. Now these technology maturation efforts aim to 
fulfill two key recommendations of the 1985-1986 Packard 
Commission: to apply advanced technology to reduce costs, not 
just to increase performance, and to demonstrate advanced 
technologies prior to the start of the EMD.
    In the Single Acquisition Management Plan [SAMP], that was 
signed by Dr. Kaminski in November 1996 it clearly states that 
the goal of ``tech mat'' program is to evolve the most 
promising leading-edge technologies to a low level of risk 
prior to integration into the JSF EMD program.
    It's important to emphasize that integration and its 
corresponding risk is, and always was, to be addressed in EMD. 
Exit criteria from the concept development phase states the 
Joint Strike Fighter will demonstrate to a low level of 
technical risk those critical enabling technologies, processes, 
and system characteristics. Integration on the JSF program is 
the focus of EMD.
    Now the Joint Strike Fighter Program Office, in conjunction 
with each competing contractor, identified critical 
technologies processes and system characteristics required for 
the program tailored to each contractor design. Robust risk 
management processes were established by each competing 
contractor and validated by the program office. The government 
did not specify to the contractors which techniques must be 
used to track risk. The contractors selected what they felt 
were the best methodologies to accomplish that task.
    Now the government has been provided real-time access to 
those systems, actually on a computer data base, for oversight 
and review during the entire phase of the program. Both 
contractors utilize what is known as ``waterfall charts'' using 
a Willoughby template. This is a common and accepted 
methodology in industry and government. In fact, it's taught at 
our Defense Systems Management College where risks have been 
identified, baselined, and tracked to document the very 
specific events required to reduce the risk of these critical 
technologies, processes, and system characteristics to a low 
level of risk prior to EMD initiation.
    Implementation of that risk management strategy has not 
changed since the program entered the concept development phase 
in 1996. And, most significantly, all of the critical 
technologies have achieved or are on track to achieve a low 
level of risk prior to the start of EMD.
    I want to assure the committee that DOD is convinced that 
our JSF weapons system contractors are appropriately reducing 
the risk of these technologies through an affordable mix of 
flight and ground demonstrations, component demonstrations, and 
modeling and simulation and analysis. Based on a request from 
the Subcommittee on Military Procurement of the House Armed 
Services Committee, both our JSF prime contractors and Pratt & 
Whitney recently reaffirmed in detailed written responses, 
which were also shared with the GAO, that sufficient testing 
and demonstration is in place for the Joint Strike Fighter 
program to enter EMD at low risk.
    In summary, clearly since the program's inception almost 6 
years ago, the program office has been following a rigorous 
risk reduction plan. The risk reduction plan is on track to 
reduce the risk of each technology to low, by entry into EMD, 
and leave the integration of those technologies to the EMD 
phase where it belongs. This risk reduction effort has been an 
important part in the program's overall goals to implement 
acquisition reform.
    This Joint Strike Fighter is vital to the modernization of 
all of our services' air forces and many of our closest allies. 
Any significant delay to the program would result in increased 
costs and also have serious impacts on our force structure and 
readiness. This was highlighted in detail in a recent May 2 
Deputy Secretary of Defense letter to our services' most senior 
leadership. That letter was addressed to each of the service 
Secretaries, to the Chief of Staff of the Air Force, the 
Commandant of the Marine Corps, and the Chief of Naval 
Operations.
    The JSF was chartered to do business differently to 
demonstrate leadership in acquisition reform, and it has done 
this. Having embraced these concepts, it was rewarding to those 
who have worked so hard on this program to be presented the DOD 
David Packard Excellence in Acquisition Award in March 1997. 
All of this has been accomplished under the twin goals of 
developing an affordable weapons system that can meet the 
warfighters' needs well into the 21st century while helping to 
reform the acquisition process.
    So, Mr. Chairman, when you ask, ``Joint Strike Fighter, 
acquisition reform: Will it fly?'', my answer to you is that 
JSF has already demonstrated acquisition reforms as called for 
by Congress, and it will continue to write the book that future 
programs will follow. Thank you.
    Mr. Shays. Thank you.
    [The prepared statement of General Huot follows:]

    [GRAPHIC] [TIFF OMITTED] T0782.040
    
    [GRAPHIC] [TIFF OMITTED] T0782.041
    
    [GRAPHIC] [TIFF OMITTED] T0782.042
    
    [GRAPHIC] [TIFF OMITTED] T0782.043
    
    [GRAPHIC] [TIFF OMITTED] T0782.044
    
    [GRAPHIC] [TIFF OMITTED] T0782.045
    
    [GRAPHIC] [TIFF OMITTED] T0782.046
    
    [GRAPHIC] [TIFF OMITTED] T0782.047
    
    [GRAPHIC] [TIFF OMITTED] T0782.048
    
    [GRAPHIC] [TIFF OMITTED] T0782.049
    
    Mr. Shays. It's great to have you both here and your 
statements were helpful. And I would like to ask a number of 
questions, stating first that I think we on both sides of the 
table agree that JSF is a necessary program. We're going to see 
this plane built and all its variations. And so it's just 
really a question of how to proceed. And--but I do think there 
are some substantive differences between GAO and your position, 
DOD's position. And I'd like to investigate that a little bit. 
But I would also like to acknowledge, you know, where we may 
agree, so then we can just really focus on where we need to 
focus on.
    The bottom line is that the new attack fighter, the AFX for 
the Navy and the multi-role fighter, MRF, for the Air Force, is 
dropped and we don't see any hope or need to resurrect that. 
That's pretty much off the table. So it makes JSF even more 
important.
    I think we both can agree--so you nodded your heads. I take 
that as a yes. You're going to have to speak. Actually it's 
very hard for her to take a nodding of a head. Since I didn't 
address it to one, I understand. So I'll start with you, Mr. 
Soloway. And General Huot, if you disagree, you certainly would 
step in. So I assume if Mr. Soloway says--gives the answer, you 
are in agreement with him and vice versa. Is that fair enough?
    Mr. Soloway. OK.
    Mr. Shays. So the AFX and MRF are off the table and we 
agree that JSF, Joint Strike Fighter, is our next plane, in 
addition to the F-22 and the F/A-18E and F. Would you agree 
with GAO that the F/A-18E and F is a significantly different 
plane, almost a brand new plane, or would you contend that it 
is a variation on a plane that we already have? I'll ask both 
of you.
    Mr. Soloway. Actually, the general has the expertise.
    General Huot. Mr. Chairman, I really don't feel qualified 
to provide a specific answer there. I don't know that much 
about the specific differences between those two aircraft.
    Mr. Shays. I've always known the nonmilitary side to be 
willing to give an answer on this. So, Mr. Soloway.
    Mr. Soloway. But I've already passed to the general.
    Mr. Shays. So that's going to be--it's going to stand 
basically, that comment, that it is a significantly different 
plane by the GAO, if it's not refuted.
    Mr. Soloway. That would be my understanding, but we can 
certainly get you more detail on what the differences are.
    General Huot. It may be appropriate, Mr. Chairman, to go to 
the Navy and answer that question.
    Mr. Shays. OK. Fair enough. We would agree that the Joint--
JSF is intended to do three basic tasks: for the Air Force, a 
conventional airplane, tactical airplane; for the Navy, slow-
speed structure that has to be a little more durable to take 
the harder landings and the shorter landings; and for the 
Marines, a vertical takeoff. And those are, General--I make an 
assumption--three very different tasks.
    General Huot. Yes, Mr. Chairman. Probably the best way to 
clarify that, for the Air Force the Joint Strike Fighter will 
replace F-16s and A-10's. For the Marine Corps they will 
replace the AV-8s and some of their F-18s. And for the Navy 
they will complement their F-18E/F force. And in the three 
variants of the airplane we achieve--we plan to achieve a high 
degree of commonality, and right now that appears to be 
somewhere between 70 and 80 percent commonality between each of 
the three variants. There is the conventional takeoff-and-
landing variant for the Air Force. There is the carrier-suited 
variant for the Navy, and the STOVL variant, the short-takeoff-
and-vertical-landing variant, that will satisfy the needs not 
only for our Marine Corps, but also for some of our foreign 
participants, notably the United Kingdom.
    Mr. Shays. Things like the avionics would be similar in all 
three?
    General Huot. Right now the avionics are planned to be 
common in all three variants.
    Mr. Shays. So some will be the same?
    General Huot. Significant savings there in terms of 
commonality.
    Mr. Shays. But admittedly we tried this in the past where 
we tried to have one plane meet the needs of more than one 
branch--Navy, Marines, Air Force--and we haven't always had--it 
hasn't been all that successful, correct? I know we have----
    General Huot. I would say that the F-4 is probably an 
example that you might refer to where it didn't meet everyone's 
requirements.
    I would point to the fact in this program and as a 
warfighter I would say this has been a great process. This Cost 
and Operational Performance Trades process that I talked about 
in my oral testimony, this is where we had what we call an ops 
advisory group, an OAG, a group of warfighters that got 
together and worked with the contractor and the SPO, the 
systems program office; and we worked this requirements process 
to evolve the requirements for the services over a 5-year 
period. And in fact we finally got their final Operational 
Requirements Document approved by the Joint Requirements 
Oversight Council, that's a group chaired by the vice chairman 
of the Joint Chiefs and with membership of the vice chiefs of 
each of the services, approved on March 13th.
    And as I said, what we did in there is, we went carefully 
through to make sure that each of the services gets the 
requirements that they need to do their individual job. So, you 
see, in each of these variants there will be some unique 
requirements to meet unique service requirements, but at the 
same time achieve the greatest degree of commonality that we 
can. And we think----
    Mr. Shays. But the bottom line is--and I accept what you're 
saying, but the F-4 was an example of a plane that didn't do 
quite enough for any of the branches. It's not to say we 
shouldn't do it again to try to make it work, but this is a 
significant undertaking. Because it is not just for the Navy 
and it's not just for the Air Force, it's not just for the 
Marines; it's for all three with a variation of basically 
between 20 and 30 percent from one to the other.
    And I would like to know if you agree or disagree with GAO 
when they said that if the technology development is $1, if 
they haven't developed the technology by production 
development--product development, excuse me--that it's $10, and 
that if they've gotten to the point of production, that $1 
becomes $100.
    General Huot. Mr. Chairman I have never heard those numbers 
before.
    Mr. Shays. So you would want us to substantiate those 
numbers?
    General Huot. I would have no way of commenting on those.
    Mr. Shays. Fine. If you would speak just a little louder. 
But the bottom line is you have no way. Fair enough.
    Intuitively, to me, that is not an unrealistic number. In 
other words I wouldn't be surprised if that were the case, but 
I accept for the time being that it hasn't been documented. But 
it is now part of the record as a statement that we need to 
address.
    Mr. Soloway. I think that we can--we will take the question 
for the record also, but I am not aware of any program that we 
have ever done that has had a 100 fold increase in cost from--
--
    Mr. Shays. But it may just be that aspect.
    Mr. Soloway. Or even that aspect, I'm just not aware of it.
    Mr. Shays. Let me clarify. Not that the whole program goes 
up, but that particular failure of technology has resulted in a 
10 to 1 in product development and a 100 to 1 in production. So 
on the table--and your point is you're not aware of that kind 
of--but intuitively we would agree that costs would go up 
significantly, and we have a heck of a lot of past history to 
document that that's the case.
    Now, I see a difference between the logic when you're 
building a school building your technology is there, but you 
haven't designed it. So you say, let's do it all at once, and 
in many cases it's proved to be cheaper and you get the product 
sooner. You don't wait to design everything; some things happen 
in process. But this is a bit different, correct? I can't use 
that same analogy, and the DOD would be--wouldn't use that same 
analogy.
    Mr. Soloway. What we would say, sir, on that question is 
that we are in fact demonstrating the critical-path 
technologies prior to going into EMD. The difference really 
comes down to how one defines technology readiness levels as a 
means of measuring where you are in that process. We are not 
entering into EMD where we have critical-path technologies that 
either don't exist or we haven't demonstrated in what we call a 
relevant environment. But the difference really comes down to--
and if you look, for instance, at the way in which the 
technology readiness levels are defined, even in the GAO report 
where at level 6 they talk in terms of system or subsystem 
models or prototypes demonstration in a relevant environment, 
we will have done that, I believe, in every technology, every 
critical-path technology prior to EMD. Level 7 speaks to a 
system prototype.
    Mr. Shays. Could someone put up that chart that GAO had. It 
was figure 4 in their report. That's the technologies 1 through 
8.
    Mr. Soloway. My only point was going to be----
    Mr. Shays. I know, but you were making reference to it, so 
let's just leave it. Thank you very much.
    Mr. Soloway. When you talk in terms of technology readiness 
level 7, which Mr. Rodrigues said they believe should be the 
threshold test before you go into engineering and manufacturing 
development, our view on that is that definition of TRL 7 
speaks to a system prototype demonstration in an operational 
environment. That suggests to us that you have had to do a 
great deal about both systems, engineering and systems 
integration, because you're dealing here with a system of 
systems.
    Mr. Shays. What does it mean, though, when they say in a 
``test bed aircraft.''
    Mr. Soloway. A test bed aircraft can be any aircraft you're 
flying to test a given--it might not be one aircraft which--I 
said in my testimony, we have the CDA, we're using commercial 
aircraft, the F-16s, F-18s, F-22, Eurofighter; various 
different technologies that need to fly will fly in an 
aircraft----
    Mr. Shays. But it isn't a prototype then?
    Mr. Soloway. No, it's not a prototype, though; that's 
correct. That's the key difference.
    Mr. Shays. The question that my counsel is asking is, who 
said it was? I mean, are you putting up a strong----
    Mr. Soloway. I'm sorry?
    Mr. Shays. That it was a prototype?
    Mr. Soloway. It says under the definition of ``technology 
readiness levels'' in the GAO report of last July the 
definition of a level 7 is a system prototype demonstration, 
and I think if--without trying to get----
    Mr. Shays. Then it says, examples include testing the 
prototype.
    Mr. Soloway. Testing the prototype.
    Mr. Shays. So we have eight basic technologies and they 
don't technically have to be tested on a prototype plane; they 
can be tested on another plane, correct? They could--so----
    Mr. Soloway. Well, you get into various discussions. For 
instance, on some of the--we can't get into specific----
    Mr. Shays. I think this is a key point.
    Mr. Soloway. It is a very key point, I agree.
    Mr. Shays. And it would be wrong to suggest that somehow 
you have to have a prototype. The question is, can you test it 
in other ways to know it works before we move forward with 
development?
    Mr. Soloway. Sir, actually--and I'm going to give you how I 
read their chart, and then General Huot may have a different 
sort of interpretation.
    I think there's actually a great deal of grey area about 
the TRLs. They're not quite as clear as has been defined, and I 
think that's one of the reasons NASA and others have said you 
have to have some flexibility.
    Let me use the language out of the report that says, 
examples include testing the prototype in a test bed aircraft. 
So you have to have a prototype of the system. Now, does that 
mean in full form, fit and function where, for instance, the 
radar array has to be the full size, full scale that you're 
going to use; or if technology allows you to, would you be 
testing a variant of the radar where the basic technology 
exists on a--for instance, an F-18 or some other aircraft.
    Others would be in the avionics. As I noted and I believe 
General Huot has talked to, the avionics don't necessarily need 
to be flown on a test bed aircraft. They can be tested in an 
environment which is tremendously--has great integrity with 
regard to the operational requirements.
    So it's not an automatic, in our view, threshold that says 
you have to test everything in a prototype which involves, 
essentially full scale, for each individual technology?
    General Huot. Mr. Chairman, could I make just----
    Mr. Shays. Let me just ask this point. The laboratory goes 
from one to three, correct, and then it gets out of the 
laboratory?
    General Huot. Those definitions become fuzzy in the sense 
that in order to demonstrate some technologies on a low level, 
you don't necessarily have to fly them at all. You may be able 
to do that in an environment that never involves putting that 
technology in an airplane. And so what the program did--let me 
talk about what the program did a little bit.
    The program did not use technology readiness levels, 
primarily because they didn't include a risk management 
process. What they did do is, they used these waterfall charts, 
or Willoughby templates where they identified all of the key 
technologies that needed to be reduced to low risk before going 
into EMD.
    Now, these--this process identifies the technology and then 
lays out over a schedule the critical events that have to 
happen to reduce that specific technology to a low level of 
readiness. In addition to that, it provides off ramps, if you 
will, where if that technology is not going to work, they would 
revert to an alternate technology.
    This was chosen by the contractors as a way of not only 
assessing risk, which the TRL process does, but also managing 
risk and being sure that you get to that low level of risk when 
you're ready to go into EMD. And that's the process that they 
used and that's the process that the program office monitored 
each of the contractors on a----
    Mr. Shays. Where you both agree, and then we'll define it: 
You both agree you shouldn't go into EMD before those critical 
technology--before you come to low-level risk.
    General Huot. Exactly.
    Mr. Shays. So the debate now is whether what you described 
as low-level risk is low-level risk. I mean, that's going to 
be--because you buy into the point, though, that you don't move 
forward if you're at high-level risk. And high-level risk is 
where you don't have the technology.
    Mr. Soloway. There's always--as I said in my testimony, 
when you're dealing particularly in this kind of complexity of 
systems, the risk of integration, which is what you do in EMD, 
is always there. I mean, to say that we can whittle that risk 
down to a very low level in this level of complexity is not 
realistic. What we are looking to do, just to make sure we're 
clear on the terminology, is to arrive at a very low level of 
risk relative to the individual technologies.
    Mr. Shays. Let me just tell you what makes me nervous. You 
have described--you basically have described the technical 
readiness levels as tools rather than requirements. And then 
you're doing one thing more; as I hear it, you're also in a 
sense redefining low-level risk. And I thought at least we 
could get to the point where we agree you don't move forward 
unless you're at low-level risk.
    You're trying to qualify low-level risk in a way that makes 
me uncomfortable.
    Mr. Soloway. I just don't want to leave on the table a 
suggestion that when you go to the integration and EMD that 
your risks are automatically low. What you have done is 
substantially reduce your risk by proving the individual 
technologies. But you still have a high-risk element because 
this is a complex undertaking.
    Mr. Shays. Well, I'm just going to tell you, that leaves me 
a little uneasy because that allows you to define it in a way 
that allows to you move forward, no matter, as I hear it.
    Mr. Soloway. In the past, we would not necessarily in all 
of our systems development bring individual technologies to the 
level of maturity that the JSF is doing, that we will now be 
requiring in the new 5000 rewrite; that's the individual 
technology levels. To get further than that into and integrate 
the system is what you do in engineering and manufacturing 
development. That is always--there is always risk there, as I 
said in my testimony.
    We have reduced that risk by proving out the individual 
technologies. That's the big step forward that we have taken 
over previous practice.
    Mr. Shays. I feel like I'm dealing with a moving target 
here. I know you're trying to be helpful and you're trying to 
be responsive. What I want to say, and then we'll just have our 
disagreements--what I want to say to you is, you view technical 
readiness levels as a tool. I view them as something more than 
a tool. I view them as more a requirement, and you view them 
more as a tool.
    I view low-level risk as--obviously, when you're dealing 
with military hardware, there are risks, but I think that low-
level and high-level are pretty clear. And then there are going 
to be different levels of low risk.
    Now, where I may have some question is, if you've leveled--
if so much of one technology, say the structures and materials, 
you've got almost everything licked, but one thing, and that 
one thing is still there, but you don't want to hold up the 
rest of the project because everything else is ready, I would 
think that you may still have something that needs to be 
resolved; and you wouldn't want to wait another year for the 
whole project if you want to start to integrate these 
technologies together.
    So I may be a little off with that and I'll, you know, ask 
GAO to help me out on this later as well, and other witnesses. 
All I'm saying to you is, I'm uncomfortable by your referring 
to this as a tool and your redefining low-level risk. That's 
where I'm uncomfortable.
    Let me do this: I'm going to come back. I'm going to let 
counsel and minority staff ask questions as well, because in 
the process of their asking questions it helps to define what I 
want to continue to ask.
    Mr. Halloran.
    Mr. Halloran. You want him to go first?
    Mr. Rapallo. Sure.
    Mr. Shays. This is David Rapallo.
    Mr. Rapallo. Could you just describe what effects you 
mentioned briefly in your statements--what effects a delay 
would have, a delay of say 6 months?
    General Huot. I wrote down some specific notes to talk to 
that. I think probably the best source of information there 
now--there was an attachment to Mr. DeLeon's letter to our 
senior service leadership that talked about the impacts of 
delays.
    Obviously, if you delay the program any period of time, you 
are going to incur some costs of--increased costs in the 
program itself. But more importantly, in this particular case, 
you're looking at a delay in development that could roll into a 
delay in production that could be more than 6 months. And if 
you will go back and look at the history of programs, some 
delays in development of a program have led to just that.
    If you were to incur any kind of a significant delay here, 
we would have some significant force structure impacts and some 
impacts on our readiness. The DEPSECDEF letter attachment 
looked at a scenario that went out as much as 3 years and said, 
if you delayed the program procurement phase by 3 years, that 
we would end up with about a three fighter wing equivalent 
force structure shortfall that we'd have to deal with in a 
couple of different ways. You could buy gap-filler aircraft, F-
16s, more F-16s, or you could do service life extensions to 
those aircraft if that were possible.
    Now, we have done service life extensions to F-16s, so in 
general, we think that's probably a ``doable'' do. How much you 
would have to do would have to be determined.
    In the case of the Marines, though, it appears that they 
would not have that kind of an option where there really isn't 
felt to be, in my view or my understanding, a viable way to 
carry the AV-8 further into the future. So they would end up 
with a force structure deficit in order to meet their military 
commitments worldwide.
    And without getting into the details of what that would 
mean, in terms of their ability to do their mission, that 
attachment in the DEPSECDEF letter talked about some of those 
impacts to our CINCs worldwide. So we think in the long run 
there are some very significant impacts if you would slip this 
in terms of force structure and readiness impacts.
    Mr. Soloway. I might add one other to that in terms of 
cost. The longer we delay a decision--and I don't want to 
suggest that the decision--Mr. Chairman, you asked the 
question, one of the members asked the question about what is 
sacrosanct about a certain timeframe for a decision.
    We set out a timeframe in which we thought we could make a 
decision, and we still believe that within a year we're going 
to be able to make an EMD decision. But it is driven by our 
requirement of having the technology maturity of where we think 
it needs to be to make that decision, not by any arbitrary line 
in the sand.
    But presuming for a moment that we have what we believe are 
adequate levels of technology maturity, you have to realize 
that we would be delaying that decision for a longer period of 
time. Carrying both contractors, for instance, which is going 
to continue costs and so forth, would have an impact on our 
international partners who are participating in the program and 
the potential of either pullout or other reopeners, if you 
will, in terms of the international relationship.
    So there are a number of sort of follow-on impacts that 
could transpire as well.
    Mr. Shays. I want Mr. Rapallo to continue to develop this 
line, but I just wanted to ask you the question again: You say 
``adequate levels of technology and maturity.'' That's just a 
different term than ``low level,'' and ``adequate'' leaves me 
really----
    Mr. Soloway. I apologize if I seem to be obfuscating, 
because I really wasn't, sir.
    When you made the comment that we would be at low risk, we 
will have low risk relative to the individual technologies, 
yes, that is a requirement. And that is a requirement. What is 
not a requirement necessarily is that TRLs be the measure of 
that risk.
    We are recommending TRLs, by the way, in the new 5000 just 
as a common language, but there are, as General Huot said, 
other ways in which one assesses, measures and manages to risk 
mitigation and risk management other than TRLs. They themselves 
are not the only.
    Mr. Shays. I'm going to go back. You've made the point that 
the Harrier--I call it the Harrier jet; is that the British 
term for it? But the vertical takeoff jet is deteriorating 
significantly. It needs to be replaced soon. The question mark 
is, do you speed up everything because of that? And one of the 
questions--I have total acceptance that if you're in 
production, and then we in Congress do what we sometimes do: We 
say, instead of 30 planes, it's 20 planes; we add costs, I 
understand that. I understand how we slow things in production, 
but frankly sometimes the technology isn't there or is being 
revised. That's one of the costs of slowing it down.
    But I have a harder time understanding how slowing down the 
decision before you go into development and production adds 
significantly to costs. And I'll just have you come to that 
question afterwards, but you had mentioned it and I just want 
to--I would love you to continue.
    Mr. Rapallo. That would be--my only followup is, GAO has 
sort of stated or described a process where if you delay the 
EMD process, you can develop the technologies and ultimately 
save money and save time. Do you disagree with that?
    Mr. Soloway. No, I don't disagree with that, but on this 
given program, I think what we're saying is, we would not go 
into EMD if we did not have the technologies developed, so that 
the delay in time is not necessarily going to help. We believe 
the technologies will be at a level that allows us to make a 
responsible decision to move to EMD.
    Now, whether we agree that, based on a TRL chart, if one 
were to use that measure, one would need to be at TRL 7 or 6, 
that's--that is actually the crux of the debate, and what does 
it require to achieve a TRL 7 under the construct. But clearly 
if the technologies are not there, you cannot--particularly the 
critical-path technologies that we're talking about, then you 
wouldn't make that decision.
    So in this case, all we're saying is, to say today that we 
ought to delay the program, when we believe the technologies 
are either there or close to there and will be there in the 
timeframe, there would be no reason to delay. Otherwise, if the 
technologies aren't there, it's correct, you would not want to 
go into EMD; you would want to take advantage of further 
development time.
    Mr. Rapallo. Thank you.
    Mr. Shays. If you want to come back, we will.
    Mr. Halloran.
    Mr. Halloran. Well, I guess the followup that occurs then 
is, you believe that technology--to be ready, how will you 
know? If not with some more quantitative or objective measure 
than TRLs, how will you know that, say, the vertical-takeoff-
and-landing technology--for example, speaking hypothetically, 
how will you know whether it's ready or not? How will you 
measure that?
    Mr. Soloway. First of all, as General Huot said, that there 
are things some people call the Willoughby templates, and so 
there are other methodologies that are used. And you may, I 
think, hear from other witnesses in terms of probability, as 
well as objective measures and so forth.
    But the other reality is, and I don't mean to be glib when 
I say this, it's either going to be working or it's not. It's 
either--the capability to do what you are talking about has 
either been demonstrated or not demonstrated on a concept 
aircraft or what have you. The radar or whatever technology 
you're talking about will have or will not have been 
demonstrated on another plane. The avionics will or will not 
have been demonstrated in a suitable laboratory environment.
    Actually, when the chairman mentioned laboratories at 
levels 1 to 3, there is a category in level 6 that talks about 
a high-fidelity laboratory, different than sort of the low 
grade. So we will actually have demonstrated the individual 
technologies in a variety of different ways most relevant to 
what we have to prove and see.
    General Huot. I think the short answer there is that we 
will continue this rigorous and disciplined approach that we 
have had using these waterfalls to track each one of these 
technologies and assure that it's a low level of risk. In fact, 
in the input that we got from Boeing and Lockheed and Pratt & 
Whitney in the inputs requested by House Armed Services, the 
Subcommittee on Military Procurement, they went through, in 
great detail, each of the technologies describing the process 
that they were using and reaffirming that we would be at low 
level of risk prior to entry into EMD. That is really the tool 
that we have been using. And it is a risk management tool that 
tracks, is event based, and it also again allows you to make 
decisions if you need to go to an alternate technology to be at 
low risk prior to EMD.
    Mr. Halloran. Stay with that for a second and describe the 
waterfall again. What I heard and I think what the chairman 
heard is that high risk plus risk mitigation plan equals low 
risk.
    General Huot. No, not a plan. You have to execute the plan 
and then you monitor the plan to assure you've gotten there. So 
the----
    Mr. Halloran. That's a quantitative way?
    General Huot. In a quantitative way.
    The program office goes through, and as the contractor 
completes each of the events that they've agreed are to track 
this from high risk to medium risk to low risk, they check them 
off. And so you end up essentially with a checklist that says 
you are at low risk in that technology area--you have done all 
the things that you said you needed to do to get there.
    Mr. Soloway. Let me just add one other thing, if I could, 
on that point. Once you've done that, you also have, and 
General Huot mentioned a couple of times, fallbacks. I think 
it's important to understand that as you go through this 
process, let's say for instance that we have a technology that 
was demonstrated on an F-16 and that we thought this is ready 
to go. For some reason, as we go forward, it doesn't work. What 
do we have to have done prior to EMD to address that 
potentiality? That is where the fallback is. You have to have 
an assessment of what lower-level capabilities you can access 
without affecting the critical performance parameters of the 
plan.
    Mr. Halloran. And cost?
    Mr. Soloway. Certainly.
    Mr. Halloran. Because I was going there next anyway. I'm 
glad you raised that. Because GAO made the point that in a 
couple of these critical areas the fallback was quite expensive 
and did affect performance or the requirements anyway, for 
example, and yet you testified that the 5000 series calls for a 
fallback that it would be at a higher----
    Mr. Soloway. The fallback itself, if it's measured, a TRL 
would be higher, but it might not be as high technology.
    Mr. Halloran. But it involves a tradeoff of some kind.
    Mr. Soloway. There's a tradeoff certainly, and you have to 
assess all that in this process. And you have critical 
performance parameters that you have to meet and you can't go 
below that.
    Mr. Halloran. Let me change the subject slightly and talk 
about the definition of threshold between technology maturation 
and integration.
    It struck me that in rescoring the original contractors, 
TRL scoring, the Department kind of expanded the integration to 
get itself out from under some low scores here and to push 
things up to 6s and 7s, to where you could define your risks 
more clearly, or at least as lower. So talk about that 
threshold some more.
    I mean, integration, there's an example in some testimony 
that comes in in a minute of a DOD system that looked fine in 
pieces, but when they came together they interfered with each 
other and didn't work. And that's clearly an integration issue, 
and it's a separate set of risks, as you testify. But is 
getting to demonstration of form, fit and function in a proper 
environment, is that integration or is that maturization, 
maturity?
    Mr. Soloway. I would suggest that the way--if you're taking 
that comment and weighing it against the way TRLs are laid out 
and what levels you hit, because it's sort of the context of 
the question--that you have to do a fair amount of integration 
to do that. So my answer would be, yes, we think that--let me 
go back to your initial question, because what you really were 
asking is, did we try to jimmy numbers to get out from under a 
bad score. That was the first question.
    And my answer is, and I can say this as one who was not 
involved in that initial process, but have gone back over the 
last month or two and have met with all of the players, have 
had extensive discussions, services and so forth. I genuinely 
believe that there was, despite what Mr. Rodrigues said, a lack 
of understanding of how these were being applied, because what 
was being done was assessing levels relative to the risk of 
systems integration. This is for the JSF. And so when you do 
that, you are--your relative risk is higher.
    But when you're weighing it against a question of, have you 
proven out the individual technology, which is what we believe 
and the 5000 really speaks to--and the new 5000 needs to be 
matured in a way we have not traditionally done before, before 
you go to EMD--when you do that, that is--the result is the 
rescored numbers.
    Mr. Halloran. What was the source of the misunderstanding? 
As you say, the 5000 draft had been out a while. It had 
reached--it had received some resistance.
    Mr. Soloway. This was not relative to the 5000 rewrite. 
This was relative to the GAO----
    Mr. Halloran. But the issue and the proposed role of TRLs, 
as you state in your testimony, at some point push would come 
to shove and a low TRL would prevent a technology from moving 
into EMD; that was pretty well known.
    What was the source of the misunderstanding?
    General Huot. How to grade TRLs? Our program office had 
never used TRLs. When they graded them to come up with these 
ratings that you see on the chart, they rated those, including 
the risk of integration on the Joint Strike Fighter. Of course, 
that's going to give you a much higher risk; that's why the 
lower ratings.
    They did not rate the individual technology area by itself 
without consideration of the risk of integration. When they 
went back and did that, in fact, that was spelled out in Dr. 
Schneiter's letter that responded to the GAO report, he made it 
very clear that the Department position was that the Department 
does not agree with the conclusion, which is based on 
misinterpretation of a process for determining the readiness of 
technologies for incorporation in major systems; and went on to 
say later that the GAO ground rules for scoring technology 
rating levels included the risk of integrating the technology 
onto the JSF platform.
    The JSF program office used those ground rules to arrive at 
the ratings contained in the draft report, and upon review and 
discussion with other users of technology readiness levels, 
that program office determined that only the maturity of the 
technology, not its integration, should be rated to determine 
the readiness under EMD.
    One finds that technology risk is expected to be an 
acceptable level of EMD start.
    Mr. Halloran. Let's go back to my original question. Define 
``integration''--``integration'' meaning form, fit and function 
down to the size it will have to perform next to other systems, 
or ``integration'' meaning actually working with other systems 
on an airplane?
    If you define it that broadly, then of course you'll, I 
think, diminish the use effectiveness of TRLs as a threshold 
judgment to get into EMD.
    General Huot. Let me say just a couple things here. First 
of all, it was clear that some of these things have to fly on 
aircraft in order to get to the low level of risk. In fact, if 
you would bear with me just a moment if you look at those eight 
technology areas without addressing the eight technologies 
specifically--for example, technology No. 4 is flying on an F-
16, technology No. 1 is flying on a concept demonstrator 
aircraft--in fact, a lot of these technology areas will fly on 
aircraft. But when the program office made their initial 
assessment using those, the TRL scoring, they included the risk 
of integration on the JSF platform to score those.
    Mr. Halloran. Let me finally, Mr. Soloway, in your 
testimony, in talking about the 5000 series, and you say--which 
is largely being implemented in JSF, speaking more broadly 
about acquisition reform, now, what--in what way is it not 
largely--gives you some room out--in what way is it not--what 
challenges does the Department face in more fully applying--was 
this referring to this program and others that follow it?
    Mr. Soloway. You've got two separate questions. Let me 
start with the JSF specific and then move to the broader 
question if I could.
    On the JSF, when you--we complete the 5000 rewrite, which 
is now out for comment, we expect to have it finalized in the 
next 30 to 60 days; it's been a long process; it's a fairly 
complex set of documents--there will be actually a new systems 
acquisition model, if you will, with different kinds of entry 
and exit criteria based on much of the kind of model that GAO 
talked about this morning. And this is precisely where we are 
going in the Department, and the kinds of criteria that you 
have to meet to go into various phases of the process, and they 
may have different names, and it's really a different model.
    JSF has, in effect, mirrored that model, but not precisely 
because the model didn't even exist until the last several 
months, so it's not a precise marry-up in that way. But I think 
JSF has very much reflected what we are seeking to do in the 
5000.
    On the broader question, what DOD or the Congress needs to 
do, I think there are--the list is quite long in terms of the 
challenges we face. Part of it is the sort of entrenched 
cultures that we both have in terms of how we view programs and 
systems development, and some of the issues that you raised 
with Mr. Rodrigues earlier today. And that's both an internal, 
DOD problems with the Congress, and the external world. Part of 
it has to do with--and I'll give you one example.
    I think one of the real challenges with the new 5000 
rewrite--and this is not relative to JSF, this is a broader 
question--is if you are in this front-end process where you 
want to, and have to, demonstrate technologies, you want to 
also--and thus not make your commitment to EMD and, really, 
program commitment--until you've gotten to that point, you will 
also hopefully, as we go down this path, be looking at more 
options than you might otherwise have been looking at. And what 
that leads you to is having to avoid, or wanting to avoid, a 
program commitment either in the building or from the political 
environment before you're ready to say, I've got a set of 
technologies that can get me this capability, and that's when 
I'm ready to make that commitment.
    You know, traditionally we'd say, we're going to do a new 
jet or we're going to do a new ship, and then we sort of work 
it through. But the ship never left the table or the plane. It 
was fairly rare once you got that far down the path.
    What we really want to do is both for ourselves inject a 
great deal more discipline. As I said, we have tremendous 
agreement, and GAO has been a partner at the table as we have 
built this model, but we also have to have discipline in the 
process to realize that we may want to be looking at different 
kinds of capabilities and different kinds of mission options 
before you make those kinds of commitments. And that's 
traditionally been very difficult, I think, for all of us.
    So there are a number of challenges that we all face in 
moving this forward. And I think that they're very--I think 
they're challenges that we can overcome. As I said, I do 
believe that JSF is really the forerunner coming out and 
applying many of these principles.
    The General was talking about the process they went through 
with the various stakeholders, the different services. A really 
flexible requirements process driven by what the customer 
really needed and what we believed as we went through the 
process technology was going to be able to provide in a 
reasonable period of time, so that we didn't end up waiting 18 
years or 20 years, as has been the case in the past.
    Mr. Halloran. Finally, let me ask a followup where Mr. 
Rapallo began.
    Would you agree that all of the impacts you cited that 
might flow from a delay at the demonstration evaluation phase 
can potentially be more serious, more costly and lengthier if 
they're incurred in the next or subsequent phases of the 
program, that, as is described--we may disagree on the $1 to 
$100, but there is an almost inevitable escalating effect in 
delaying problems in this process.
    Mr. Soloway. That would certainly be the conceivable 
output--outcome. However, I think our view at this point is 
that the delay--to arbitrarily decide today to delay, to really 
do, in essence--and I don't want to get into nitpicking, but in 
essence what we're talking about here when you get to level 7 
is the early phases of EMD.
    That's what we do in engineering. We focus on the 
manufacturing part and the production part of it. We don't 
focus on the ``E'', which is the engineering. That's a critical 
next step in bringing these technologies together. We believe, 
and we could be proven wrong next year and the decision would 
not thus be made, but we believe the technologies will be at 
that proven level that enable you now to move into that 
engineering of EMD phase.
    So given that, the delay to us would be costly, 
unnecessarily costly, and would be just unnecessary on the face 
of it. But in a hypothetical world having nothing to do with 
JSF, certainly if you have tremendous technology issues and you 
go into EMD and lock yourself in, you could be asking for an 
escalated problem. I wouldn't disagree with that; I just don't 
think that applies to the JSF.
    Mr. Halloran. Thank you.
    Mr. Shays. Let me say, then, I would like David to come 
back with answering a question related to what you were saying 
up here.
    I've heard you both say, you know, this is something, you 
know, we have never done before. You're very proud of the 
program, the fact that you're moving toward a best commercial 
practices. But what I feel like, as I hear you, is that it's 
going to be so significant to DOD that you've moved in this 
area if you do it halfway, a three-quarters way, this is 
monumental. And I might agree with that, that it would be 
monumental, but I have to hear an argument that says you can't 
go all the way with the program. And so for you to say you're 
doing three-quarters of it--and you haven't used those numbers, 
but that's the feeling I get--wouldn't satisfy me unless you 
say we can't do best commercial practices here, here and here 
because, and then let us evaluate it.
    So I am going to stay on the table that really there is no 
argument against using best commercial practices to the nth 
degree, unless, and I just kind of want to say that to you.
    Mr. Soloway. If we gave that impression, I think both of us 
would regret it. How do you define doing it all the way versus 
the three-quarters?
    Mr. Shays. Well, when you start to talk about it as a tool 
and not a requirement. And maybe in the private sector it's a 
tool and not a requirement, and then that's fine.
    Yes?
    General Huot. I would just say that--I would emphasize that 
we use a different tool. And again the reason we did it is 
because we wanted something that would allow us to manage risk, 
not just assess it. And our intent is not to go three-quarters 
of the way; we intend to reduce every critical technology that 
we identified to a low level of risk before we go into EMD--
very important point.
    Mr. Soloway. And the requirement that is there is the 
technology maturation requirement. That is a requirement; that 
is not a negotiable.
    What is negotiable is the TRL becomes like the yardstick, 
so it's versus using a yardstick or a ruler or another method 
of measuring what the outcome is going to be. But the key here, 
the outcome we all seek is technology maturation. And that is 
not a negotiable.
    Mr. Shays. Does the DOD 5000 rewrite follow this practice, 
best commercial practices?
    Mr. Soloway. Yes.
    Mr. Shays. Because when you say it, you say in your 
statement on page 10, ``Finally, let me be very clear. The 
strategy I have articulated for the revised acquisition process 
that will be prescribed in the DOD 5000 rewrite, and which is 
largely being implemented on the JSF, represents a real 
departure from our traditional approach to system 
development.''
    That's the basis for my making that comment. I buy into the 
fact that it represents a real departure from our traditional 
approach but it's still the word ``largely,'' you know. So just 
so you have a sense of why I get the feeling I get.
    Mr. Rapallo, you had a question.
    Mr. Rapallo. I just had one followup on Mr. Halloran's 
questions about the TRL levels.
    You said--I'm just trying to understand. The information 
that was provided to GAO was from the contractors, their 
evaluation, or from the program office?
    General Huot. You know, I wasn't involved in that process, 
but I know the program office is involved with the contractors.
    Mr. Shays. I didn't hear the question. I would like to ask 
you to ask it again.
    Mr. Rapallo. It was just a question of where the 
information came from, the TRL levels, from the contractor or 
the joint program office, or probably some combination of both.
    Mr. Shays. So what is the answer?
    [The information referred to follows:]

    The input for the TRLs reflects a consolidated answer from 
the JSF Program Office, incorporating input from the Boeing 
Company and Lockheed Martin Corporation.

    General Huot. I believe it is, Mr. Chairman--I wasn't 
involved in the exact process. I know the program office was 
involved, but I think the contractors actually did the work 
either with the program office or the oversight of the program 
office.
    Mr. Rapallo. And your position was that the GAO asked for 
information based on TRL levels, asking also, including the 
risk of integration?
    General Huot. Yes.
    Mr. Rapallo. That information was provided?
    General Huot. Yes.
    Mr. Rapallo. And that would be incorrect, you're saying? 
Basically, I'm trying to figure out if these numbers are too 
low because of that.
    Mr. Soloway. That's the crux issue here, that if you 
measure individual technologies relative to the risk of being 
able to integrate, or to integration, you come up with one 
answer. If you measure them as individual technologies relative 
to their technology maturity, you come up with the second. And 
that's what the rescoring was.
    So in our view, the rescoring actually does raise the 
numbers, not to level 7, but they certainly raise it.
    Mr. Rapallo. Are these the numbers that represent the 
rescoring or the initial?
    Mr. Soloway. These numbers are the initial.
    Mr. Rapallo. So the rescore numbers would be higher than 
this? The question is what are they I guess.
    Mr. Soloway. I think all of them would be at five and six.
    Mr. Shays. I would like you to tell us what would they be 
at.
    Mr. Soloway. I don't know if I have the specifics on each 
of them.
    Why don't we take that for the record and give you--I'm 
sorry. OK. These are our estimated ratings.
    Mr. Shays. Based on not having integrated?
    Mr. Soloway. Right. These are what was referred to as ``the 
rescored numbers.''
    Mr. Shays. Without the risk of integration?
    Mr. Soloway. Right, they were not scored. We have--and I'll 
just take them in order--7, 6, 6, 7, 6, 7, 6, 7.
    Mr. Shays. OK.
    Mr. Soloway. And I have the rationale, if you just--I mean, 
I can tell you what the technology--but in order of these 
numbers, here again, as--I think General Huot referred to this 
a moment ago. One, flying on the concept demonstrator aircraft, 
which is we believe consistent, technologies already flown in 
aircraft helicopters or spacecraft, another one, such systems 
are similar or being used in similar commercial environments, 
including the Boeing 777, Federal Express Caterpillar, etc.; 
another one flying on the F-16; another technology flown on the 
JSF flying test bed; another one, the same, flying test bed; 
another one, flown on the JSF flying test bed; and another one, 
flying on the CDA as well as the F-22, the F/A-18 and the 
Eurofighter. So in each of these cases, if you look at are we 
using the technology in a relevant environment, per se, they 
are all being demonstrated where--in that means.
    Now, that is the rescored numbers.
    Mr. Shays. But that's based on your assessment, correct?
    Mr. Soloway. It's a DOD assessment.
    Mr. Shays. Right. And so in technology 7 you have a leap of 
two levels just by not having to integrate. And technology 1 
you leaped up one, in technology 2 you went up two, technology 
3 you went up two. In technology 4 you went up two. In 
technology 6 you went up two. In technology 7 you went up two. 
That's a big jump.
    General Huot. Again, because you're trying to mature the 
technology without considering the risk of the integration on 
the JSF platform.
    Mr. Shays. So we can basically accept GAO's assessment that 
they got--if we have a footnote that says it's not integration, 
based on--excuse me, based on that, and we could accept your 
numbers based--excuse me, the other way around.
    They're saying it's not integrated; therefore, that's the 
score. And you're saying, if you don't consider--I'm going to 
say it right here.
    Mr. Soloway. I think I understand where you're going.
    Mr. Shays. For my own sense of self-esteem, I need to say 
this again.
    The bottom line is, GAO gave you a lower score because of a 
failure to integrate. And you're saying integration shouldn't 
have been a factor in your score, and your score is higher 
without integration.
    Mr. Soloway. You're asking if we can accept this as--under 
that other definition, if you will, and I think I would have to 
say I don't believe we can because there are other issues 
associated with it.
    Mr. Shays. You want a little conference here?
    Mr. Soloway. We might want to take it for the record. But 
the table itself is confusing, and I'm not sure that we would 
precisely accept every level that was there.
    Mr. Shays. I welcome sometimes a little bit of 
qualification by two people in the same table.
    If that's all right, General, do you want to just say what 
your sense is?
    General Huot. I think once again that you go back to where 
these things came from. They were provided to the GAO, but 
those scores were based on considering the risk of integration 
on the JSF. When those scores were rescored again and provided 
to the GAO, they were without integration of JSF, the 
integration on the JSF, that's when they go up to the 
appropriate level.
    Mr. Shays. But I just want to know if you accept their 
score when there's--by--and look at the score that you have 
provided us, would they accept your score, in your judgment? I 
mean, in your talks with each other, was this really the big 
debate whether it was integration or not and you could pretty 
much agree on your numbers?
    OK. The answer is hard to figure out what they were saying 
when they did this, right?
    Mr. Soloway. Let us take that for the record, sir.
    [The information referred to follows:]

    [GRAPHIC] [TIFF OMITTED] T0782.050
    
    Mr. Shays. Fair enough. You all have been very responsive. 
And is there a question you would have liked me to ask that you 
prepared all night for that you want to--I'm just going to say 
my staff gave me so much to read I stayed up all night. The 
statements have been very interesting on the third panel, as 
well as your presentation has been very, very helpful and, I 
think, candid.
    Mr. Soloway. I'd just add one point, it's not a question, 
and that is to go back to what we said earlier.
    The debate here is not over whether or not we do need to 
demonstrate and whether or not we're committed to demonstrating 
technology maturity before we move into EMD. And that is a very 
important step in a critical discipline that we are instituting 
in our process with the JSF and in future programs. And I 
believe it's very important, as we do that, to recognize what 
it takes and to recognize the ``E'' engineering part of EMD, 
which is the next step.
    So I would just reiterate that point again.
    Mr. Shays. I don't know if I'll be chairman next March but 
it will be very interesting. One of the things we do is, we put 
every statement on the record; then we have to live with it. 
I'm just--the problem is, when it comes to this kind of 
program, the same people aren't always the same people that 
have to answer for the decisions that were made 2 years ago--I 
mean on the congressional side as well as your side.
    Mr. Soloway. Sir, I just point out I won't be here next 
March; that's a guarantee. I think you saw the news this 
morning where both the House and Senate Armed Services 
Committees have included language requiring us to be able to 
demonstrate to the Congress that we have achieved technology 
maturity before we move into EMD. And we can live with that and 
have no problem with coming back to demonstrate that or 
document that for the Congress next year.
    Mr. Shays. That means low-level risk.
    Mr. Soloway. Thank you.
    Mr. Shays. General, any comment you'd like to make?
    General Huot. No, I think that covered it.
    Mr. Shays. We'll end on that note. Very nice to have you 
both here. Thank you.
    We'll get to our third panel. I appreciate the patience of 
our third panel. I have three panelists, if you would remain 
standing so I can swear you in: Dr. Thomas McNaugher, deputy 
director, Arroyo Center, RAND; Mr. Rodney Larkins, business 
development manager of 3M Corp.; Dr. Wesley Harris, Department 
of Aeronautics and Astronautics, Massachusetts Institute of 
Technology.
    If the three of you would--we'll make sure we get you 
there. If you would raise you right hands, please. Thank you.
    [Witnesses sworn.]
    Mr. Shays. Note for the record that all three have 
responded in the affirmative, and we'll just go down the row.
    Doctor, we'll start with you first. And actually we have 
two doctors here. I'm sorry.
    Dr. McNaugher.

STATEMENTS OF DR. THOMAS L. McNAUGHER, DEPUTY DIRECTOR, ARROYO 
CENTER, RAND; RODNEY LARKINS, BUSINESS DEVELOPMENT MANAGER, 3M 
    CORP.; AND DR. WESLEY HARRIS, PROFESSOR, DEPARTMENT OF 
                  AERONAUTICS AND ASTRONAUTICS

    Mr. McNaugher. Thank you, Mr. Chairman. It is a pleasure 
indeed and an honor to be here. At the risk----
    Mr. Shays. I will ask you to speak a little louder or move 
the mic a little closer. Can you pull it a little closer?
    Mr. McNaugher. Somewhere between last night and this 
morning I have acquired a cold, so I am about an octave lower.
    Mr. Shays. I have absolutely no sympathy for you.
    Mr. McNaugher. Let me start by saying I don't know much 
about the JSF program per se. I was asked to speak here because 
I think I know a great deal about the weapons acquisition 
process, at least as it functioned during the cold war. I wrote 
a book about that. It came out in 1989 from the Brookings 
Institution. And in that book, called ``New Weapons, Old 
Politics,'' I examined what might be called some of the 
perverse patterns of cold war weapons acquisition, one of which 
was the tendency to rush weapons through development and into 
production.
    In particular, I focused less on the move from what's 
called the ``demonstration-validation phase'' and more on the 
move out of what is now called EMD and into production, so-
called ``production concurrency.'' So it's not exactly the same 
issue that you're dealing with today although I think you'll 
find a lot of what I have to say about the incentive structure 
and the way we measure----
    Mr. Shays. Could you just suspend for 1 second. I'm sorry. 
Thank you.
    Mr. McNaugher [continuing]. Production concurrency or the 
tendency to substantially overlap the latter stages of 
development with the early stages of a move to high-rate 
production was a fairly common practice during the cold war. It 
was justified on two grounds. Technically it was justified on 
what might be called the ``theory of declining uncertainty.'' 
back in the ``dem-val'' phase. In a somewhat linear fashion, we 
reduce those uncertainties during development until, by the end 
or nearing the end of EMD, we should be able to move into 
production.
    On the other hand, on strategic grounds we were confronting 
a numerically superior enemy, so we justified the rush to 
production as a way of getting the jump on the Soviet Union, 
getting ahead. The faster you could get these new technologies 
out, the farther ahead of the Soviet Union you were, everything 
being equal.
    Now, when I started my book, I more or less agreed with 
those premises. When I finally started writing, I had come to 
disagree, especially with the first one, the theory of 
declining uncertainty. I would substitute what I call the ``J-
curve theory of program uncertainty.'' True, at the beginning 
of a development project you have enormous risks, and you do, 
through ``dem-val'' and in the early part of EMD, neck those 
down to reasonable levels. But I encountered in almost all 
programs a rather sharp upturn in risks as the program actually 
went into production.
    Three reasons for that: One was systems integration. And I 
like the fact that you're talking about systems integration 
this early in the program. During the cold war systems 
integration often occurred very late in the development 
program, sometimes after production systems were already out in 
the field. And what you find, looking back at the history, is 
that even simple components stuck together in a new way 
produced new problems that you didn't know were there, these 
problems often force a design iteration.
    The second problem that drove risks up was the move to 
production tooling itself. Again, even with simple systems, 
somehow the move to a production model from a preproduction 
prototype introduced risks, some of them great, some of them 
small, which almost always led to a subtle interaction between 
the producing engineers and the developing engineers.
    And finally, when you got this thing out into the field 
where real soldiers and sailors and pilots could use it, they 
invariably discovered new ways to use it that it wasn't 
designed to do, and it started to break, and then you would 
have to go back and redesign it and fix those. This is, in a 
sense, a tribute to the ingenuity of developers and also users 
in the military, but it could be pretty embarrassing sometimes, 
the kind of breakage you could get. So all three of those then 
would appear very late in the process.
    To the extent that production concurrency ignored those 
risks, and to use the jargon of today's hearings, to the extent 
that, in a sense, it wasn't ``knowledge-based,'' it didn't take 
account of these late-arising risks, we had chosen a fairly 
expensive way of developing weapons during the cold war. If 
you'd geared up for production, capitalized, and hired the 
labor, every delay, every change, cost a lot of money. If you 
already had systems in the field you were faced with this 
vexing question, do we retrofit the fixes to the fielded 
systems, which was always expensive, or do we just let those 
go? This led to what I would call the ABC approach to 
development.
    The A model of a lot of airplanes, and other things, too, 
often had serious design anomalies in it; the B model got it 
roughly right; and the C model is where you really wanted to be 
all along, but it took maybe even 100 or 200 production models 
to get there. So it was expensive. And when you did that, the 
average effectiveness of our overall force actually was lower 
than it would have been had you waited and gotten all of the 
production run, or most of it, to the C level.
    Now, we accepted those costs ostensibly in a desire to 
confront the Soviet Union. And you would think that now with 
the cold war over, as you say, Congressman Shays, we could 
relax and we could do this better. Let me make clear that the 
way you do it better is not by having a sharp divide between 
development and production.
    The point of my research was that the early stages of 
production are part of the development process. Again, to turn 
to the jargon of today, knowledge points 2 and 3 tend to fudge 
together and you don't want to have a sharp gap. Rather, you 
had to treat the early stages of production in a way that with 
respect to late-arising uncertainties, perhaps starting very 
low, with a low rate of production, getting the systems out 
into the field, flying or shooting or driving the dickens out 
of them for awhile, and taking all of that information back and 
imparting it to your design before you really went to high-rate 
production.
    One of my colleagues at RAND years ago remodeled some of 
the Air Force aircraft programs of the 1960's, 1970's and 
1980's on that basis and concluded that you could probably get 
a more effective fleet overall for a little less money if you 
did it that way. Now, you added a little bit to the development 
cycle, which everybody thought was too long, but with the cold 
war over, that shouldn't be much of a concern.
    So we might be in a position to move now to a more relaxed 
approach. I'm skeptical of our ability to do this, however, 
because I think that production concurrency was as much a 
political as a military strategy; that is, it was rooted in the 
politics that we see here today--in the politics of the 
acquisition process.
    Surely in the development of a new system, the move into 
late development and early production has to be the most 
vulnerable stage. You now have prototypes of the system, you 
fly them, drive them; the data almost always is going to 
contradict the optimism of early assessments. Sometimes you get 
tragic accidents, a helicopter crashes or a plane crashes. 
Sometimes you get funny accidents, you know--the gun that's 
supposed to shoot a helicopter shoots the fan on a nearby 
latrine, as was the case with the Armys DIVAD.
    If this were purely a technical environment, everybody 
would have a good laugh, and they'd go back and redesign that 
system and fix it. But in the very charged political 
environment that can come to surround expensive programs, you 
know, it's hard to desensitize that evidence. Knowledge and 
information become very dangerous. And delay--I'm thinking of 
your earlier discussion--you cannot convince a program manager 
that a 6-month delay isn't the beginning of a mortal wound.
    How do you handle that? Well, in a sense you stack the 
deck. You raise the costs of slowing down. And you may even 
have production versions out there before you actually get that 
information.
    What I'm describing, in a sense, is the dilemma of weapon 
acquisition in this country as a political as well as a 
technical process. From a technical point of view, you really 
want to have a great deal of flexibility late in the process. 
From a political point of view, flexibility can be downright 
dangerous. So we tend to structure a certain amount of 
inflexibility in there.
    This is not a dilemma we have ever resolved very well in 
our history. If you step back from the cold war and look at the 
200 years in which this Nation has bought weapons, you would 
argue that the problem wasn't slowing the acquisition process 
down, it was getting it to produce anything at all. We would 
test endlessly and then not buy. Our technology generally 
lagged the Europeans, and as a rule, we had trouble getting 
things through the juggernaut of the political process. That 
ended with World War II and with the cold war.
    So while talking about political strategies may sound like 
I'm being critical, we ought to understand that for 50 years we 
produced and systematically modernized the best force posture 
in the world.? So it's not clear to me that we can slow this 
thing down. And this is why I refuse to pass judgment on the 
patterns of acquisition I studied in the cold war.
    We are now in a unique period in the Nation's history. We 
are not confronting a great Soviet threat, as you've said, 
although the world remains dangerous; moreover, we remain 
engaged, and there are threats out there. So we're sort of at 
the Goldilocks situation, you know, it's not no threat, not a 
big threat, but sort of right in between. And the question is, 
can we come to the Goldilocks solution for weapons acquisition? 
That is a process that is relaxed enough to take account of 
late-arising uncertainties, but not a process that goes to 
sleep and doesn't produce anything.
    And that returns me really to the first paragraph of your 
statement, Mr. Chairman, which talks about the emergence of a 
consensus about how we're going to handle this.
    This hearing focuses on the JSF specifically, but the JSF, 
as you recognize, is the beginning of a wave of new 
modernizations and recapitalizations after a 10-year 
procurement holiday. So this hearing is also part of the way in 
which we construct this consensus, and it remains to be seen 
how that will work out.
    Thank you.
    Mr. Shays. Thank you very much. Your book was the reason 
why you're here and your testimony justifies your presence.
    [The prepared statement of Mr. McNaugher follows:]

    [GRAPHIC] [TIFF OMITTED] T0782.051
    
    [GRAPHIC] [TIFF OMITTED] T0782.052
    
    [GRAPHIC] [TIFF OMITTED] T0782.053
    
    [GRAPHIC] [TIFF OMITTED] T0782.054
    
    [GRAPHIC] [TIFF OMITTED] T0782.055
    
    [GRAPHIC] [TIFF OMITTED] T0782.056
    
    Mr. Shays. Mr. Larkins.
    Mr. Larkins. Good morning, or I guess I should say good 
afternoon.
    Mr. Shays. Good afternoon. You've been wonderful to be so 
patient. But I learn as much from the third panel, and 
sometimes more. So from my standpoint, I'm very happy you are 
here.
    Mr. Larkins. Thank you. As business development manager for 
government research programs at 3M, I'm here to discuss 3M's 
``best practices'' associated with our new product 
commercialization process and the timely integration of 
breakthrough technology into ``change the basis of 
competition'' products.
    3M is a diversified manufacturing company with sales of 
more than $15.5 billion. We manufacture a broad range of 
products directed at six distinct commercial markets. 3M has 
grown by pioneering innovative technologies and creating new 
products with these technologies, thereby creating new markets 
and revolutionizing existing ones.
    3M has a written policy objective which states that 30 
percent of all product sales for a given year come from 
products that were introduced in the 4 preceding years. Thirty-
four percent of 3M's total sales in 1999 came from products in 
this category. It is these new products which help sustain the 
profitable growth of our corporation. To maintain this rapid 
pace of technology development and new product introduction, 3M 
has evolved a well-defined technology development and 
commercialization process.
    A basic tenet to successfully introducing new products is 
the discipline to make the up-front investment in research and 
development. 3M has invested more than $1 billion per year over 
the past 3 years in research and development. This investment 
is directed both at broadening and strengthening 3M's existing 
technology portfolio, as well as moving products rapidly 
through the commercialization process.
    Currently, approximately 20 percent of 3M's research and 
development budget is directed at technology development and 
enhancement. Approximately 80 percent is directed at product 
scale-up and commercialization. This investment ensures the 
availability of critical technologies to turn into products, 
and the product development resources to focus on the 
corporation's priority programs.
    In our quest to maintain technical and market leadership in 
the markets we serve, we have evolved into a laboratory 
structure which substantially segregates technology development 
from product development. Our experience has been, that the key 
technologies are used broadly across the corporation's market 
groups. By focusing on technology development in a technology 
center, a critical mass of technology experts is created, and 
costly redundancy is eliminated.
    3M has formally established 14 technology centers for the 
corporation's most pervasive technologies. The responsibility 
of these centers is to establish and maintain world-class 
capabilities in these critical technologies. The second 
responsibility is to work with product development teams in the 
business units to integrate this technology into the product 
development programs to assure successful and timely product 
introduction. This step is critical, and a number of management 
tools have been established to ensure that it takes place. 3M 
is acutely aware that even our best technologies, when not 
applied to timely commercial product development, are of no 
value, and we work hard to maximize that application.
    Critical to the commercial success of our company is our 
ability to select and to focus on programs with the best 
possibility for changing the basis of competition in large 
market opportunities. Despite 3M's large investment in research 
and development, our resources are finite and there are always 
more opportunities than there are resources. Important steps in 
program initiation are opportunity identification, development 
of a comprehensive product or system description, a business 
assessment of projected results once success has been realized, 
and a technology assessment which indicates that technologies 
are in place to meet performance requirements. All of these 
elements must be in place before a program is approved for 
scale-up and commercialization.
    At the initiation of the product development process, a 
formal review is held to assure that all elements are in place 
for success. Hence, success on these key programs is critical 
to the success of the corporation and our goal is to put these 
programs in the very best possible position to succeed.
    Once under way, the development team follows a detailed new 
product introduction plan which outlines all elements which 
must be accomplished during the product development process. 
Periodic program reviews are routinely conducted with the 
emphasis placed on encouraging and rewarding candor on the part 
of the product development teams and identifying and 
eliminating impediments to the success of the team.
    In addition, corporate review teams are employed. These 
review teams bring together experts in product development from 
across the corporation to identify potential roadblocks to 
successful commercialization and to identify resources to 
eliminate those roadblocks.
    In summary, 3M's approach to technology integration and 
product development has proven to be very successful. This 
process, however, is far from being perfect. It's a process 
that we will continue to evolve, and our goal, of course, is 
100 percent success on all of our priority programs in the 
corporation.
    Thank you, Mr. Chairman. I'll be happy to answer any 
questions.
    [The prepared statement of Mr. Larkins follows:]

    [GRAPHIC] [TIFF OMITTED] T0782.057
    
    [GRAPHIC] [TIFF OMITTED] T0782.058
    
    [GRAPHIC] [TIFF OMITTED] T0782.059
    
    [GRAPHIC] [TIFF OMITTED] T0782.060
    
    [GRAPHIC] [TIFF OMITTED] T0782.061
    
    Mr. Shays. Thank you, Mr. Larkins. You work for, I think, a 
pretty amazing company.
    When I think of your company I think of it as--though very 
large, it has tremendous innovation as a small company might 
have. And it's great to have you here. I would love to know how 
we can seek comparables between what you do and what we need to 
do in government in general, and specifically with defense.
    Dr. Harris. I just want you to know, sir, I'm always awed 
when I see a ``doctor,'' and I see MIT next to it.
    Mr. Harris. Thank you. Good afternoon, Mr. Chairman. I very 
much appreciate the opportunity to address this distinguished 
subcommittee on----
    Mr. Shays. I would ask you to move the mic a little closer 
to you. If you can turn it over the paper, as long as it 
doesn't get in the way. Does it get in the way? OK. That's 
perfect.
    Mr. Harris. I'm fine. Thank you.
    Again, I very much appreciate the opportunity to address 
this distinguished subcommittee on certain issues related to 
technology maturity and acquisition reform. I wish to state at 
the outset that I approach this important topic from the 
perspective of both an academic researcher and as a former 
government manager, not as an engineer practicing within the 
defense industry.
    It is my view that our defense acquisition policy and 
practices are complex. This complexity is a result of several 
factors, including a dynamic or shifting defense industrial 
base, a declining acquisition budget, a constantly evolving 
threat environment, a diverse force structure, and most 
importantly, an increasing rate of change of technology.
    The impact of the last factor, namely the increasing rate 
of change of technology, on technology maturity from an 
acquisition policy perspective, is difficult to overstate. The 
impact is made more profound when the global nature of much of 
this technology is considered in the acquisition of new weapons 
systems.
    I would like to add to this list the following elements 
that have impact on the acquisition of defensive systems: the 
current focus on greater life cycle value, the emphasis on more 
rapid deployment, the emphasis on upgradability, sustainment 
and maintenance.
    Mr. Chairman, while noting that today this subcommittee is 
addressing a specific program within the defense acquisition 
arena, I wish to state for the record that the impact of 
acquisition reform reaches beyond the procurement of defense 
systems and its related technology. Through our defense 
industry base, acquisition reform drives our national economy 
and impacts world peace, in short, our success in developing an 
effective and efficient acquisition strategy that captures 
mature technology, exposes the risk to control our future, to 
produce wealth and to continue to contribute to the advancement 
of humanity.
    Based on my research and government experiences, I wish to 
share with this subcommittee today several national successes. 
First, there exist several case studies of successful 
acquisition of defense systems in production. Second, there 
exist case studies of successful parallel development of 
advanced technology to high maturity levels where the 
government is customer. These two successes have many things in 
common, and I believe are related to today's issues of 
acquisition reform in the environment of technology maturity.
    My success or my research on economically incentivized 
contracts focused on several important programs including the 
Sensor Fused Weapons system, the Joint Direct Attack Munitions 
program, the C-17 program, C130J, the F-414 engine, development 
that goes within the F-18E/F airplane and the F-117 engine that 
goes in the C-17 airplane, as well as the Boeing 757 airplane. 
These programs were in production, were able to develop an 
economically incentivized contract or a win-win solution for 
both government and contractor.
    A few comments now on the development of a critical complex 
technology in advance of full system acquisition: In the early 
1990's, NASA's Office of Aeronautics did develop and manage two 
technology development programs. These were the high-speed 
research program and the advanced subsonic technology program. 
During the same time period, NASA also worked jointly with 
industry and DOD to develop advanced gas turbine components 
within the integrated high-performance turbine engine 
technology program.
    These three programs of technology development were 
successful, and as stated in the prepared statement, over 12 
reasons why they were successful.
    Mr. Chairman, the subcommittee may wish to note the very 
strong commonality between the factors leading to successful 
development of technology parallel to full-system acquisition 
and the factors that enable a win-win solution for programs 
already in production. At the most fundamental level, the 
environment for favorable development of advanced technology is 
very similar to the environment for acquisition of defense 
systems in production where technology risk is low, 
corresponding to technology at a high maturity level.
    These programs, both advanced technology development 
programs and acquisition of full defense systems, strongly 
suggest that advanced technology at a high maturity level is 
essential to the acquisition of affordable systems with 
requirements for superior performance. The importance of 
advanced technology at a high maturity level is so great, in my 
opinion, that the government must incentivize the contractor to 
develop advanced technology.
    This means that the government must place a premium on the 
development of technology to a high maturity level. The premium 
must compare favorably with other awards available to the 
contractor. The economic realities of a high premium will, or 
at least should, drive the government to a lean portfolio 
management condition. Advanced technologies selected for 
development to high maturity level must be, or should be, based 
on realistic projections of need.
    Mr. Chairman before concluding, I would like to add a few 
additional comments. I wish to note that to improve the chances 
of successfully developing advanced technology to a high level 
of maturity, one or we, those involved, should adopt common, 
quantitative-based language and assessment tools. Qualitative 
prescription of technology readiness, such as the technology 
readiness levels, are insufficient and inconsistent with 
realistic projections of need. The desired quantitative-based 
language and assessment tools are recommended to be derived 
from probability in each technology readiness level. The TRL 
would be expressed or defined by probability bands.
    In conclusion, we, as a Nation, have demonstrated the 
capability to produce weapons systems driven by advanced 
technology. My colleagues have clearly confirmed that. To 
become more efficient at the acquisition process, we must 
continue to develop advanced technology in parallel with 
acquisition of full, complete systems. Key elements again in 
this efficiency are to incentivize industry with favorable 
premiums, a realistic projection of advanced technology needs, 
and the use of quantitative-based language and assessment 
tools.
    The bottom line is that government and industry know enough 
about each other and about advanced technology development to 
make affordable acquisition of full systems and technology 
insertion work, to make it work efficiently. The health and 
wealth of our Nation depends upon this working very 
efficiently.
    I would entertain questions, if there are any.
    [The prepared statement of Mr. Harris follows:]

    [GRAPHIC] [TIFF OMITTED] T0782.062
    
    [GRAPHIC] [TIFF OMITTED] T0782.063
    
    [GRAPHIC] [TIFF OMITTED] T0782.064
    
    [GRAPHIC] [TIFF OMITTED] T0782.065
    
    [GRAPHIC] [TIFF OMITTED] T0782.066
    
    [GRAPHIC] [TIFF OMITTED] T0782.067
    
    [GRAPHIC] [TIFF OMITTED] T0782.068
    
    Mr. Shays. Thank you. There will be questions. And I'll 
just throw it out to all of you.
    First off, you were all here for the entire hearing, and I 
thank you for that, because the value is that you can comment 
without me having to repeat the questions.
    Conceptually, a ``best practice'' model for acquisition 
that makes sense in the private sector for private development 
also makes sense for the private sector for a public customer. 
Would you all agree with that?
    And if not, if you would, qualify it where you would--I 
mean conceptually. I make the assumption nobody would disagree.
    Mr. McNaugher. You're saying that the commercial best 
practices is transferable to the public process of----
    Mr. Shays. Yes, and that we should seek to use commercial 
best practices.
    Mr. McNaugher. The only caveat I would raise, Mr. Chairman, 
I mean, obviously no commercial firm operates in the kind of 
political environment----
    Mr. Shays. Just a little louder. I know you're not feeling 
well, and I said I wasn't sympathetic; but I didn't want to 
give you sympathy because I didn't want it to get worse.
    Mr. McNaugher. There's a lot of differences, obviously, 
between buying weapons as a public good and developing a 
private sector item. The only difference, though, that leads me 
to question the use, the extensive use of best practice models 
is I don't think in the commercial world very often you 
systematically take these huge leaps out into the unknown, as 
we did during the cold war and, I think, still do; that is, not 
only the components of a system, but the integration of a 
system really represents a dramatic jump.
    Mr. Shays. But you don't have any fear that we're doing 
extensive use of this now. I mean, we're not, this is the--I'm 
not aware that a commercial best practices model is running 
rampant through our government.
    Mr. McNaugher. No. What I'm saying is, trying to apply it 
to the government, to this process, has limits. Most commercial 
firms insofar as they're doing marginal improvements are aware 
of risks and costs at a fairly good level of certainty. You 
know, most of the discussions of risks in the early phases of 
cold war development programs proved in the long run to have 
been wrong. You don't know what the numbers really are. You 
have an idea of a cost performance curve----
    Mr. Shays. And they were almost always understated.
    Mr. McNaugher. Yes. That's not an indictment of anybody's 
behavior. Technology just doesn't give up its secrets very 
easily. So we have these very precise discussions during ``dem-
val,'' but, the numbers and curves are going to shift. That's 
again why you need more flexibility down the road.
    I think commercial practice may not have to deal with that 
kind of uncertainty.
    Mr. Shays. OK. We'll touch on that a little bit more in a 
second.
    Mr. Larkins, when I think of 3M, I would think that you use 
a knowledge-based process, a best practices model for 
acquisition as defined by GAO when they talked about knowledge, 
point one, match is made between the customer's requirements 
and the available technology.
    Mr. Larkins. Yes, sir.
    Mr. Shays. Knowledge, two, when the product design is 
determined to be capable of meeting performance requirements. 
And three, knowledge point three, when the product is 
determined to be producible with cost schedule and quality 
targets. I would think 3M would just be right in the center of 
that kind of philosophy.
    Mr. Larkins. Yes, sir, that's correct.
    Mr. Shays. Where it might differ is that I think of 3M, 
with no disrespect, but just in terms of putting it in 
perspective, that you are seeing customer needs and then you're 
seeing how you can meet those needs with products that you have 
potentially available in technology that you have available.
    I make an assumption in most cases that technology is 
pretty developed, or sometimes do you have to--do you see a 
need and you just start out from scratch with a technology?
    Mr. Larkins. That happens, sir.
    Mr. Shays. What happens?
    Mr. Larkins. Where we will, in fact, start from scratch and 
develop the technology. A lot of it depends on how broad your 
window of opportunity is in the market.
    What we are finding today is that if you don't meet a 
market need quickly, then the market need is met by someone 
else. So what we try to do is do a rapid technology assessment, 
determine whether in fact we have technologies in house to meet 
that need. If we don't, then we will go out and make a 
technology partnership or acquire a technology.
    Mr. Shays. And admittedly the--so time is of the essence?
    Mr. Larkins. Absolutely.
    Mr. Shays. But your technology requirements may not and 
certainly wouldn't be as complex as developing an aircraft or--
--
    Mr. Larkins. That's very fair to say.
    Mr. Shays. So we need to have some empathy for the task at 
hand, that applying best commercial practices is a very 
important effort, but certainly we can carry the commercial to 
the defense; and we may be--it may be unfair to say that you--
DOD needs to do it just like we do it in the private sector.
    Mr. Larkins. Yes, I agree that it would be unfair to say 
that their technology integration problems are as easy as ours, 
because they certainly are not. And--but I would like to agree 
with my colleague, Dr. Harris.
    I think one of the reasons we are successful and are able 
to achieve the type of record in product integration scale-up 
that we do is because we do have a significant investment in 
technology. We have a broad base of technology platforms that 
we can tap into, and they're ready to be integrated.
    Mr. Shays. But it would be unlikely for 3M to go into 
development and production without the technology there to back 
it up?
    Mr. Larkins. We absolutely would not.
    Mr. Shays. You would not. So the general principle that 
there is significant logic in developing the technology before 
you went into development and production still holds for the 
defense as well.
    Mr. Larkins. Well, all I can say is that we do not go into 
a product scale-up mode until we have the technologies in hand.
    Mr. Shays. So, then, if you can't meet the deadline, you 
just don't produce the product?
    Mr. Larkins. That's correct.
    Mr. Shays. Dr. Harris, do you want to comment on the 
questions I have asked so far?
    Mr. Harris. Yes, I would like to comment, sir.
    The use of the words ``commercial best practices'' and our 
consideration of the--this environment of acquisition of 
defense systems, that phrase is a very loaded one. It has many 
meanings and usually it differs, depending upon the speaker.
    Mr. Shays. Is that--if I could interrupt--is that because 
there is no, one, so-called ``commercial best practice''; there 
are--different companies have their version of best practices?
    Mr. Harris. Well, that's somewhat it, sir, but I think--my 
statement is based primarily on the fact that there's no 
commonly agreed-upon body of knowledge as to what it really is. 
It's a sort of catchall phrase that has caught many up into 
some more political stance than real substance.
    For example, the question of scaling along technology 
complexity, the question of scaling in terms of number of 
products, the questions of availability of markets all would 
impact commercial practices. And you don't have a serious 
discussion of commercial best practices being transferable 
along what is in fact scalable.
    So I'm somewhat personally concerned as to whether--what we 
really mean when we say that DOD or the government ought to 
rush immediately into commercial practices. It just doesn't 
have the substance that I think it must have in order to hold 
those who practice within government accountable when we say 
that they ought to use or should be using best practices.
    Mr. Shays. So you would be more sympathetic to the view 
that--that when we use TRL, technical readiness levels, that it 
be more a tool than a requirement. And you heard them refer----
    Mr. Harris. Yes, I did hear that discussion, sir. Yes, Mr. 
Chairman. I think we're wrapping ourselves around an axle with 
the qualitative discussions of technology readiness levels. 
Until we can move to a position where we can quantify what we 
mean by technology readiness, we can't hold those who practice, 
who acquire the weapons systems, accountable. Because they and 
their contractors will always be at each other's throats as to 
what they mean when they present an argument or position on 
technology readiness levels.
    So I'm calling for more quantitative language and 
assessment tools to enable all of us to understand precisely 
where we are within this arena called ``technology readiness 
levels.''
    Mr. Shays. That's spoken like a true academician. You're 
fulfilling your role perfectly.
    Well, one of the things is--and I'm not burdened by being a 
lawyer, and I don't have any perceived--and the reason why I 
love this committee and love the work as a legislator, I'm not 
trying to prove a point. I'm trying to understand what we as a 
committee need to recommend to the full Congress and to the 
executive branch.
    But intuitively, I have accepted as a fact--and, Dr. 
McNaugher, I'm going to try to have you separate a little bit 
the so-called politics side of this, and then we'll get back 
into that. But I accept as just very logical, without a lot of 
empirical knowledge, but just your own life experiences, that 
if you are going to be developing technology and development, 
and fail to come to grips with some technology; and then you've 
got a point where the technology hasn't been resolved and yet 
you're still moving things along, that your costs are going to 
go up significantly. You're going to begin to slow down your 
development after you started, particularly in production.
    And I accept the point that you made, Dr. McNaugher, that 
sometimes we haven't perfected the product until we have 
produced 100 planes. And that's a real negative.
    So, Dr. Harris, tell me why intuitively I shouldn't accept 
the fact that we should develop the technology before we go 
into development, but particularly into production.
    Mr. Harris. Well, I can't tell you that. I want my 
testimony to read very clearly that we----
    Mr. Shays. Don't put words in your mouth.
    Mr. Harris. We as a Nation have, in my opinion, been 
successful in developing technology in advance of acquisition 
of the full system. My experiences at NASA with the high-speed 
civil transport, the new 700,000-pound Mach 2.5 airplane, we 
developed that technology with industry as partners. The 
airplane was never built because it was a business decision, 
not because the technology was not there, or was designed to be 
there, in advance of building the airplane. So we know how to 
build or how to develop technology in advance of acquisition.
    Mr. Shays. But the question is, is it going to be cost 
effective?
    Mr. Harris. Oh, I would say it definitely is cost 
effective. And the effectiveness goes up, I believe, with the 
complexity of the final product. If you're talking about very 
complicated systems that require a real stretch in technology, 
then clearly the effectiveness is going to be there, the cost 
effectiveness.
    Mr. Shays. I'll allow the other two to jump in in a second, 
but where I'm not with you is, I would think that waiting to 
develop the technology before you did the development, but 
particularly production, would be to your advantage as long as 
you don't have the time restraint of having to get into the 
marketplace because someone else is going to build the product.
    Mr. Harris. Well, I agree, as I know the threat, at least 
the military threat, that this Nation currently faces, we have 
that time.
    Mr. Shays. Right. OK.
    Any other responses to any points that either of you, Mr. 
Larkins or Dr. McNaugher, any comments you want to make?
    When you heard the debate between--the dialog between the 
first panel and the second panel, I want each of you to tell me 
your reaction not in--as concise as you can; and tell me how 
you reacted to the two different testimonies, because I think 
both testimonies were quite excellent. I think GAO gave us a 
wonderful vehicle in which to have this debate and dialog.
    But I would love to know how you reacted as you heard it.
    No criticism of the committee allowed, just the other side.
    Mr. Harris. Well, I'll go first.
    Mr. McNaugher. Yeah, feel free. I'm happy to defer.
    Mr. Harris. I thought the first panel was again a very 
excellent presentation. I think there was an honest mistake in 
interpreting what technology readiness levels were meant. Was 
it to be on a full-up system or a comparable system? And that 
led, I think, to a downgrading of the so-called ``technology 
readiness levels.''
    What I thought was missing in the first presentation was a 
normalization of the two possible streams to the same system, 
one stream as to what we currently have, or at least what the 
first panel proposes that we have, that we buy this complex 
weapons system and develop the technology while we buy it. 
Another stream would be to develop the critical technology off 
stream, then insert it when we are ready to move forward with 
the full system.
    Now, my question is this: What is the cost going along the 
first stream and what is the total cost going along the second 
stream? And what are the time lines involved? When does the 
clock start?
    Mr. Shays. How would we know that if we don't know if we 
can actually develop the technology?
    Mr. Harris. As far as I know, sir, no one has ever done a 
comparative analysis of that sort--unless Dr. McNaugher has 
done it. Yet we get this constant bombardment of how to develop 
systems and how to develop technologies that support systems 
off line.
    But the answer is, Mr. Chairman, no one really knows.
    Mr. Shays. What I heard, and then I'll open it to the other 
two of you, when I found myself wondering if moving forward 
with development represents--where you still have high risk on 
technology, whether that represents a significant negative. But 
I clearly feel that if you went into production without the 
technology, then we are really opening up potential costs--I 
mean, that ratio. So I found myself more tolerant of DOD kind 
of wanting to go to that development stage.
    I would become very concerned if they wanted to go that one 
step further. In other words, I think there's still this high 
risk in some of their technology.
    Mr. Harris. OK. I don't know the specific technologies 
involved with this weapons system, as I am not a consultant to 
any of the primes. But there are two different ways of getting 
to the same end goal; a highly complex system is what we don't 
have a comparative analysis to.
    Mr. Shays. Fair enough.
    Dr. McNaugher.
    Mr. McNaugher. First of all, I don't know enough about the 
TRL methodology to comment on it explicitly. I think it's 
absolutely appropriate to be hammering on the readiness of 
these technologies. The disagreement between the two prior 
panels--and not everybody agreed that there was a 
disagreement--seemed to circle around the question of whether 
you're assessing risk in the context of systems integration or 
the risk of specific technologies. I can at least understand 
why you get a different number in each of those cases, given 
what I said earlier.
    And so the one upshot I would have, and I think I'm echoing 
something Dr. Harris said, is whenever you move or this project 
moves from ``dem-val'' to EMD, don't think that just because 
you have reached TRL 7 for components, you're out of the woods. 
There's a lot of development work and there's going to be a lot 
of sometimes nasty surprises up ahead.
    Mr. Shays. And I don't think GAO was suggesting that.
    Mr. McNaugher. Agreed. So I can understand why there might 
be a different measure, depending upon whether you're thinking 
of the specific technology or that technology in the context of 
an overall system, because the system does pose its own 
uncertainties.
    Mr. Shays. Mr. Larkins.
    Mr. Larkins. There seemed to me to be two basic issues that 
came up. I think----
    Mr. Shays. Can you move the mic just a little closer.
    Mr. Larkins. There seem to be two basic issues of 
disagreement. I think there was an awful lot of agreement, but 
there were two basic issues that I got that there was 
disagreement. One was that--one was the TRL as--the definition 
used in each individual category. And again I have to agree 
with Dr. Harris here; I think that if you're going to use 
categories to evaluate a project, you should have general 
agreement by everybody, the people who are being evaluated and 
the evaluators, as to what those categories are and where they 
fit. And there obviously wasn't any there.
    The other disagreement was that there is a substantial 
difference between a fully integrated system with eight 
apparent critical technologies versus evaluating each of the 
individual critical technologies.
    And frankly, at 3M, when we look at integrating several new 
technologies into a program, if we consider, for example, that 
each technology has a 90 percent chance of success, then you 
have to do a multiple of those chances of success to arrive at 
a final chance of success for your program. So if each of your 
programs has a 2 percent chance of failure and you have--or 
each of your technologies has a 2 percent chance of failure and 
there are four technologies, you're looking at a 16 percent 
chance of failure right there.
    So I am completely sympathetic with what the Department of 
Defense is saying here.
    Mr. Shays. And yet you would be sympathetic with the fact 
this integration sometimes may mean that the technology doesn't 
work?
    Mr. Larkins. That's correct. And that's why we frankly take 
a very conservative view of that approach. We--and believe me, 
we do take risks, but they are calculated risks; and the 
greater the risk on a commercialization program, the tighter we 
monitor it.
    Mr. Shays. Do you have any questions?
    Mr. Chase would like to ask questions.
    Mr. Chase. Mr. Larkins, what kind of incentives do you 
provide your teams, your project teams, to ensure that a new 
product that you're trying to develop comes in on time and 
within budget?
    Mr. Larkins. Well, there are, in my view anyway, three 
different types of incentives, peer recognition and reward--and 
we have programs in place which recognize individuals and teams 
for outstanding success in product introduction--and, of 
course, promotion. And when a person is a program manager and 
they're involved in a successful product introduction, there 
are certainly promotions that are involved in that type of an 
activity. And finally, financial incentives as well, which go 
along with the promotion and separate from that.
    So recognition and reward and promotion.
    Mr. Shays. Thank you. I was going to ask that question 
after you did. Because I just wonder if in the--the Federal 
Government, when they are utilizing the services of a private 
company, whether those companies themselves have their own 
internal programs to encourage as much innovation and extra 
effort and so on.
    Let me say this to you. I've really asked the questions I 
wanted to ask. I'm going to allow all of you to make some 
closing comments or ask yourself a question you wish I had 
asked, that you think I should have asked.
    I would also like to just ask if Mr. Rodrigues would mind 
coming to the corner here, just to kind of share your 
observations.
    Is there anyone from DOD that was here that would like to 
or--if they hear a comment--do we have a representative from 
DOD who would be?
    We had let them know they would be invited if they were 
able to stay. I realize they couldn't necessarily stay. If you 
could, just sit on the corner here, if you could do that.
    Are there any comments that you would like to make or 
observations? If so, I would move the mic over to you.
    This is not--the purpose of this is not a debate, but just 
to say things that you would want us to focus in on.
    Mr. Rodrigues. I guess I would like to say a couple of 
things to try to add a little clarification.
    The work that we did on the Joint Strike Fighter and the 
application of the TRLs, being able to do that was the 
culmination of years of work. It was not easy to define a model 
that reflected what we defined as, at this point, ``commercial 
best practices'' that can be applied in DOD. That was gleaned 
from a lot of work with a lot of companies, and working with 
the Department looking at successes and failures and trying to 
categorize what people do to come up with knowledge points that 
are critical to being able to move forward successfully.
    When we put that over at the Department of Defense for 
comment, the original comments that we were getting back were 
very, very negative because it would require a significant 
change in the way they do things.
    Dr. Gansler, coming on board, he took that on and rewrote 
the comments; and I can tell you I never get comments like 
this. At the end, he writes a personal note, on the formal 
comments coming over, ``A very thoughtful and helpful report, 
Jack.'' Believe me, I never get these.
    What we then did is, we said, OK, if you agree with--if you 
could put that chart up, please.
    Mr. Shays. You know, it's interesting, just a nice little 
comment like that goes a long way. We all need to do it, all of 
us.
    Mr. Rodrigues. That's the only one I've ever gotten.
    So if you look at this chart, then we said, OK, if 
everybody can agree with this--and we had to come up with a 
model, run it back by all the commercial companies we did, 
because they didn't know how to define what they did.
    They don't think about it in this term. Nobody had a model 
for us to use. We had to work and find it. Find it, define it, 
get it understood by both parties, DOD and the commercial guys; 
and let's agree to some things.
    Once we got agreement on that, we went back and said, if 
this is true, let's peel the onion. If technology development, 
separating technology development from product development is 
so critical, how do you do it?
    Mr. Shays. I need to know what your bottom line is.
    Mr. Rodrigues. My bottom line is, we did a whole study to 
come up with the technology readiness levels. We didn't get 
there by accident. We did a whole thing focusing on knowledge 
point one. And we worked with a lot of individual technologies, 
both commercial and defense. What we found was that they work, 
that they can predict outcome, that if you meet the right 
technology level, you can get success. If you don't, you are 
virtually guaranteed failure.
    Mr. Shays. You're a brave guy to say that right next to Dr. 
Harris.
    Mr. Rodrigues. I'm not guaranteeing there aren't better 
ways to do it. I'm just saying they are a great indicator.
    Now, where I was going with this was or trying----
    Mr. Shays. Just don't get too deep here.
    Mr. Rodrigues. In that report, before we applied TRLs to 
the JSF Program--see, we applied it in the abstract and didn't 
criticize anything--we never said anything.
    We said, here they are. Here's a tool. Isn't it wonderful? 
Can't you guys use this? Doesn't it make sense? When we defined 
it in there--and this went to the Department for comment, and 
I'll read you their comments. We said--we defined TRL 7. We 
were talking about where does integration fit. There are two 
types of integration, there is subsystem integration and there 
is product integration.
    Mr. Shays. OK.
    Mr. Rodrigues. We got a whole bunch of technologies that 
are at subsystems. Integrating proven technologies into a 
product is a real challenge, and it is something that has to be 
done. Matching technology to requirement up front doesn't stop 
that from happening. You still have to do it. It's a challenge.
    What we're trying to do is isolate that, because what 
should be done in product development--isolate that from the 
technology development which is proving the technologies in the 
subsystem forms in which they have to be, so you're not trying 
to do that concurrently, especially on pacing items. So we said 
when radio components--this is in the TRL, report--are 
assembled inside a case that resembles a final radio design and 
then demonstrated aboard a surrogate of the intended aircraft, 
the radio reads TRL 7 now when they comment on this report, 
they say that the Department agrees--this is their writing, 
it's not mine, reprinted in the report--the Department agrees 
that TRL is an important input and is necessary, but adds 
they're not sufficient alone to decide when and where to insert 
new technologies in a weapons system programs.
    Military system development decisions require a total 
ownership cost approach through the entire life cycle system. 
Now I totally agree with that. We weren't trying to imply that 
once you reach 7 you have to put it on a product. We were 
saying if you don't reach 7, you've got problems and you 
shouldn't put it on a product. They went on to say in their 
comments----
    Mr. Shays. I want you to bring this to a close.
    Mr. Rodrigues. This is their comment. The Department 
concurs with GAO that the weapons system program manager should 
assure that technology is matured to TRL 7--I just read how we 
defined it in the report and they agreed--before insertion into 
a new system. They agreed with that here.
    The difference in the JSF study was we then cross-walked 
that, and said, hey, you know what? When you apply this on the 
program you're doing, you're not there and you shouldn't move 
ahead. It wasn't until we made that cross-walk in a draft 
report for comment that all of a sudden this issue of a 
misunderstanding, came up. It came up once they saw how it was 
used. If they had read this report in detail, which I had 
provided to them, they would have seen how it would be used.
    Mr. Shays. Let me just say I think they did read the 
report. The advantage that GAO or Members of Congress have is 
that we can look at something that someone else is building or 
doing, and we can sit with hindsight and all the other things, 
and really be pretty analytical about it.
    I think your report was done very tastefully. I think it 
served a tremendous use here and will be used by other 
committees because, as you know, your report was used by the 
Armed Services Committee even before we released it today.
    I think DOD was fairly respectful of your analysis. I think 
there are--I have a sense of where our disagreements are. And a 
lot of good is going to come from the report. And--but I'm 
excited that this is a program being used by Defense. I want to 
make sure that they're using it to the fullest extent possible.
    Let me just--I'll let you say another comment here, but let 
me just ask Dr. Harris or Dr. Larkins or Dr. McNaugher: Any 
question you want to ask yourself and then answer brilliantly?
    Mr. Harris. Mr. Chairman, no question, but I would like to 
conclude by simply repeating myself.
    We, as a Nation, have demonstrated the capability to 
produce weapons systems driven by advanced technology. To 
become more efficient at the acquisition process, we must 
continue to develop advanced technology in parallel to 
acquisition of full, complete systems. Key elements in this 
efficiency are, No. 1, incentivize industry with favorable 
premiums, develop this advanced technology; No. 2, realistic 
projections of advanced technology needs, meaning a management 
of our DOD technology advancement portfolio; third, use of 
quantitative-based language and assessment tools.
    Mr. Shays. Thank you very much.
    Mr. McNaugher. Three points, the first taking off from Dr. 
Harris' last remark; and it's a subject of a different hearing, 
but one that should be kept in mind.
    We have never found how to make R&D per se profitable in 
the defense industry. The historic tendency is, firms sink 
their own money into development and get well in production. If 
you want to relax this process and maybe even cancel the 
occasional system, you probably want to have a way, a 
mechanism, for making R&D profitable. We've never found that, 
and that's the way you encourage industry. It's a profound 
problem in the defense industrial base, precisely in this area 
we're moving into.
    My second point has to do with the earlier discussion of 
the tenure of program managers. I must say I'm fascinated with 
the idea of a program manager who actually has to live with the 
results of his or her decisions 5 or 6 years later, I think it 
would be a profound change. Again, though, stepping back and 
looking at the full sweep of American history, remember that 
before World War II the Army had an Ordnance Department which 
was a bunch of full-time program managers and technocrats, if 
you will. They wore uniforms, but they didn't rotate.
    I think the Army's conclusion, as an institution coming out 
of World War II, was that the Ordnance Department was not 
attentive enough to the user, the actual operator. And it was 
more fixated on the technologies than on military capability. 
What the Army did in the 1950's and 1960's was begin to 
substitute line users as program managers, and but also as the 
chief of R&D, for example.
    Mr. Shays. Wasn't there also another danger that they 
developed too cozy a relationship with the organization they 
were buying from?
    Mr. McNaugher. Well----
    Mr. Shays. Over time you just began to become almost their 
advocate, not necessarily their----
    Mr. McNaugher. My sense of the ordnance department, to the 
extent that I understand the history, is that it had a somewhat 
thorny relationship with commercial firms because it was doing 
its own R&D, and it was always being besieged by Members of 
Congress pushing Samuel Colt's pistol or rifle. So it was a 
more combative relationship.
    Mr. Shays. So--I didn't realize, they did the R&D; I just 
thought they were----
    Mr. McNaugher. Springfield Arsenal. The arsenals did R&D, 
so they were competing with industry to some extent. But the 
point is, a program manager tenured for 15 years or 10 years 
may become very attentive to sort of the permanent features in 
the environment, of which the political structure is one, and 
not as attentive to the military user. So maybe there's a 
length of tenure that sort of balances those needs. I don't 
think it's 3 years either. I think it's longer than that.
    And finally, at the risk of complicating things, I would 
just separate technologies into two bundles, slow-moving and 
fast-moving technologies. If you look at an airplane, the 
engine airframe combination had its fast-moving days back in 
World War I and the 1950's, and that's slow-moving technology. 
You can afford to stop, assess that technology, test it.
    If you look at the electronics in the avionics----
    Mr. Shays. Let me just interrupt you a second so we can ask 
a question of the recorder.
    Thank you.
    Mr. McNaugher. If we look at avionics in the cockpit of 
that airplane, which may come to 40 or 45 percent of the cost 
of the airplane, we're looking at technologies where if you 
stop and test and decide, you know, they're already obsolete--I 
mean, they're turning over in a year, 18 months, 2 years--and 
I've always felt that we really need--now that electronics 
technologies are seen to be the key to the revolution in 
military affairs we really need to handle these by different 
commercial practices. One can be much slower and more judicious 
because you're way out on the flat of the cost performance 
curve; you're pushing for that marginal improvement in jet 
engines or airframes. The other is just so fast-moving that you 
almost have to be turning your design and your tests over 
constantly. The DOD needs a different approach entirely.
    Mr. Shays. I thank you.
    Mr. Larkins, I knew that would bring you out.
    Mr. Larkins. If I could just make one.
    Mr. Shays. Would you move the mic, please, toward you.
    Mr. Larkins. If I could just make one additional comment, I 
would say that our experience in industry with the importance 
of focusing on fewer larger programs, that having trained 
people who are--whom we call program managers or program 
development managers, whose expertise is scaling up new 
programs. We are beginning to see within industry a focus on 
that as a discipline.
    3M traditionally has what we call a dual-ladder system 
where we have technology people on one side of the promotion 
scale and management people on the other side of the 
professional scale. So you have research and management moving 
up together, the same opportunities for advancement, the same 
opportunities for pay, this kind of thing. But we are now in 
industry beginning to see a third leg or a third ladder in this 
process, which is a focus on program management.
    Mr. Shays. Any other comment?
    Mr. Rodrigues. I would like to make one, because this 
becomes a real area of concern; it is related to this. I have 
started looking at some data really hard-looking at this whole 
issue, this issue of profitability for research and 
development. In private industry, research and development is 
investment, the return is on sales. You don't make money in 
R&D. You make money building a product and selling it to 
somebody. The same principles should apply in the Department of 
Defense and should apply in the defense industry. The reward 
for doing the right thing is you sell us something that you 
make a profit on.
    When I looked at this issue and was trying to study this, 
what I find is when I look at tactical aircraft, from 1973 to 
1991 we built anywhere, annually, from 350 to 500 aircraft. 
From 1991 to today, we're building handsful, 40, 50 aircraft, 
sometimes much, much less than that.
    And then we have an industry saying, we're not profitable; 
Wall Street is beating us up. There are a lot of other factors, 
but one of them is, we're not producing a lot.
    We got into a cycle where we're heavily into R&D and so now 
the issue becomes, how do we make R&D profitable? The issue 
becomes, we have a modernization strategy--I think this is a 
real issue that we need to take a hard look at--that drives us 
to cycles that make the industry unprofitable at times.
    Is the solution to pay them for R&D, pay profit rates, 
increase profit rates on something that is normally an 
investment account; or is the solution to better plan on 
modernization so that you equalize production, so people can 
make money and we can modernize?
    Mr. Shays. I think it's a very fine point.
    I saw Dr. Harris start to waiver a little bit here. I am at 
overload. So whatever you guys are, I'm at overload; I see a 
new recorder, and we're done.
    Julie Thomas, thank you for all your work. There's no 
question you were working hard today.
    And I enjoyed this hearing. I learned a lot. I think our 
witnesses and all three panels were terrific. And I also 
appreciate the staff for both the majority and minority for all 
their good work. So, everyone, have a beautiful day. This 
hearing is over.
    [Whereupon, at 2 p.m., the subcommittee was adjourned.]