J. Michael Gilmore, the Pentagon’s director of operational test and evaluation, has recently urged that “production-representative articles should be used for test and evaluation,” according to an October 18 memo by him. by In other words, the Defense Department should be testing weapons that come off a contractor’s production line, or are similar to the ones that will. These are the weapons that will be most like the weapons that eventually may make their way to the battlefield. Weapons produced for the development phase of a program, particularly in its earlier stages, are often substantially different from the production-representative versions.
The Joint Strike Fighter (JSF) is perhaps the most notable target of his frustration.
In December 2009, Gilmore’s annual report on test and evaluation noted “the [military] Services and operational test agencies need to monitor the production-representative quality of” the first batches of JSF aircraft and support systems. “Given the concurrency of development, production, and test, shortfalls in capability must be recognized early to ensure resources are available to modify these aircraft and support systems so they are production-representative and ready for a successful” initial operational test and evaluation (IOT&E).
IOT&E is the phase of testing that occurs before a full rate production decision and is supposed to utilize production-representative systems. This phase’s importance in a weapons program cannot be understated. In theory, the director of operational test and evaluation can decide to hold up full rate production or give it the thumbs up – although there have been numerous successful attempts to circumvent this power. Thomas Christie, a former director of test and evaluation (DOT&E), in a speech before the Annual International Test and Evaluation Association Symposium in September of last year, said:
Speaking from my own experience as the DOT&E from 2001 to early 2005, my office was responsible for producing roughly 30 Beyond Low-Rate Initial Production, or BLRIP, reports to the Secretary of Defense and Congress. By law, these reports are a prerequisite for any full-rate production decision. These reports assessed over half of these systems to be either not operationally effective or not operationally suitable, or both. In not one case was one of these programs stopped as a result of the information available in the reports or presented at the production DAB [Defense Acquisition Board meeting].
Leaving aside the question of whether decisionmakers have the stomach to make hard choices, it’s difficult for the DOT&E to make a decision on whether production of a system should go full speed ahead until they’ve had a chance to fully test a system like the one that will be in production in massive quantities.
This is not a new concept; Gilmore’s recent memo simply reiterated longstanding Defense Department policies. He fleshed out some of the criteria his office will use to determine whether a weapon system is production representative. “Wherever practicable, IOT&E will be conducted using low-rate initial production (LRIP) systems assembled using the parts, tools, and manufacturing processes intended for use in full-rate production,” Gilmore wrote. “The system will also utilize the intended production versions of software. In addition, the logistics system and maintenance manuals intended for use on the fielded system should be in place.”
However, he gives weapons programs somewhat of an out, allowing that sometimes it's “impractical” to expect LRIP systems will be available. In those cases, “at a minimum,” the system being tested should “incorporate the same parts and software to be used in LRIP articles.” Any differences between non-production articles and the production representative systems need to be pointed out to the test office, according to the memo.
The bottomline, at least according to Gilmore’s memo: His office will have the final call on whether non-production articles will be acceptable.