The Hidden Cost of Reusability

All space flight providers want to control costs. From the traditional government and commercial enterprises to the new entrants, all want to keep costs as low as reasonably possible. Gone a long time ago—and forever—are the days of unlimited resources to get the job done. Bottom lines matter, and matter a lot.

It has long been argued that one way to keep costs reasonable is to reuse launch hardware. Even when adding the required refurbishment/reflight checks, repairs, component replacements, etc., to the equation it can be shown on paper to save money in the long run. The Shuttle was sold to Congress in part on this philosophy. And being the first reusable spacecraft they could know no different.

Nor could NASA fully know the long-term implications of reusability.

An additional fact is this: Adding astronauts to the equation requires changing ‘reasonably’ to safely. No argument there, right? But what are the implications of this change? Simply put, based on Shuttle experience, it requires lots more of the checks, repairs, component replacements, etc., in an attempt to make the re-flight as technically close to the first as possible. The newer the hardware, the less risk of failure. No argument there either.

These costs are easy to understand and accept. What about the posting’s title? Hidden? I have alluded to it in earlier postings and it’s not just a financial issue. It can cost a lot more than money.

Here it is:

With the desire to refly hardware that has (theoretically) performed well before, a tendency to rely on past performance when approving current flight rationale is given more weight than it should.

That tendency can either be seen overtly, or it can be latent. If obvious, it can be dealt with through open discussion and full debate of the issue at hand. If hidden, it can fester, grow, become a ‘new normal’—or worse, lead to disaster.

Examples in the Shuttle program are numerous. From those we recognized and dealt with—wire chafing, flow liner cracks, hydrogen leaks—to those that dealt us the most severe of consequences—foam shedding, O-ring scorching—we were challenged by the need to re-fly hardware, all with the overarching tenet of doing so safely.

Tragically, we didn’t always balance that correctly.

I sincerely hope today’s and tomorrow’s astronaut launch providers succeed 100% of the time. Recognizing when a ‘new normal’ is kicking in is part of the success criteria.

spacex-falcon-9-crs-9-landing
SpaceX Falcon 9 first stage returns to Earth after the CRS-9 launch. (Photo courtesy SpaceX)

Launch Countdown

Why was a Shuttle launch countdown three days long? The answer to this question has roots back to the ‘elders’.

Those of us that were part of the final years of the thirty-year operational life of the Space Transportation System inherited some truly thoughtful processes from those who came before us. Some processes were born as early as the Mercury program, nourished through the Gemini and Apollo days, and adapted to the Shuttle needs.

In architectural design, it’s called “form follows function.” Ditto for many of the ways we processed hardware and launched our friends into space.

Granted, some early concepts for a new adventure like this didn’t pan out. The good ones did, and they live on today in virtually all launch system designs. I was asked to design the launch team for the Constellation program in the 2005 timeframe. As part of that effort a small group was formed and we benchmarked numerous high-power teams looking for best practices (and worst!). It amazed us how closely other successful teams resembled ours. Why? Because it worked, and worked well. Period. Nuclear subs, unmanned rockets, ESA and Russian space programs, emergency response authorities, aircraft flight testing, and others, had teams very, very similar to ours. But I stray…

Three-day long launch countdowns? The basic reason was it was easier and more effective to control the myriad of tasks in those three days under one governing, integrated procedure than had they all been conducted as individual, stove-piped operations. Integrating all the tasks under one orchestra leader (the NASA Test Director) allowed for better command and control, and visibility for managers like me. The NTD would integrate and lead all the element Test Conductors, ensuring no conflicts, or overlaps, or omissions.

What tasks? In the simplest form they were these. On the first two days:

  1. Pad closeout and securing
  2. Fuel Cell cryogenic reactant loading
  3. Communication system activation
  4. Flight Crew equipment stowage
  5. Rotating Service Structure retraction to its launch position.

On day 3—Launch countdown day:

  1. External Tank fuel load
  2. Astronaut boarding
  3. Launch.

That’s basically it. Within these major tasks were hundreds of steps , but that was about it.

This basic design was created by my predecessors and not changed much at all for the 135 Shuttle flights. Norm Carlson, Apollo Launch Vehicle Test Conductor (the forerunner to the Shuttle NTD position) knew it would work. His compatriots knew it would work. We inherited it and it still worked.

Like many of the other processes we (the “late bloomers”) inherited, we enjoyed ones already tested and proven successful.

To Norm, Tharpe, Tribe, Horace, Page, Sieck, Fuller, Breakfield, and so many, many others—Thanks.

shuttle_countdown_sign
Launch countdown sign at Kennedy Space Center. (source: The Register)

Getting Ready for Launch Countdown

Fourteen years ago this week we were doing our final preparations on Columbia and the ground systems, getting ready to enter launch countdown (LCD).

With launch scheduled for January 16, the 3-day countdown was to begin Monday, 1/13. After dusting off vehicle and ground systems (and ourselves!) last week following our 9-day holiday period vacation, we were conducting several final tests that weren’t in the launch procedure but needed to be done as close to it as possible. On the schedule was installing the ordnance items, pressurizing the hypergolic system, checking out the space suits the crew would use if a spacewalk became necessary, doing some crew equipment and Spacehab installations, and closing the aft compartment of the Orbiter. Assuming this all went well, we’d have the 2-day weekend off just before LCD.

Ordnance installation and hypergolic fuel system pressurization were major hazardous operations requiring the pad to be cleared of non-essential personnel. And once done, they put the vehicle in a somewhat higher hazard level so we did them as late as practical. Access to the pad was more tightly controlled afterward. Dozens of explosives were used on the Shuttle and ground, mainly in separation sequences. The solid rocket boosters needed to be separated from the mobile launch platform, the SRBs from the external tank, the Orbiter from the ET, etc. All were done with ordnance firings. Installing them and verifying proper connections was an extremely precise skill. Our guys were the best at it.

Pressurizing the hypergolic propellants was necessarily done late in the pad flow to expose the least number of folks to that state following the pressurization. Hyper propellants—ones that ignite spontaneously when combined vs. needing an ignition source—were actually loaded well before the holidays. Hypers are stored at room temperatures, unlike the cryogenic liquid hydrogen and oxygen in the ET, but need to pressurized with nitrogen and helium gasses to get proper flow rates. Hypers were used in the thrusters, the auxiliary power units in the hydraulic system, and OMS engines.

Closing the aft compartment may not sound like a big deal, but it took several days to inspect each and every system, remove protective covers and access platforms, clean things up, and do other detailed work. It was an arduous process. It usually turned up a few items that needed to be addressed before we could have full confidence we were ready to go. We allotted 4½ days to fully inspect and prepare the multi-level aft compartment.

Checking out the space suits was fairly straightforward. Communications, power, and breathing systems were checked, as were all parts of the suits themselves. There were no scheduled EVAs in Columbia’s research mission, but we always provided two suits in case it became necessary. Every crew practiced an EVA to close and latch the payload bay doors if they wouldn’t close automatically. The Orbiter couldn’t reenter with those doors ajar!

We also did some “early stow” inside the Spacehab—experiments that weren’t time critical, different supplies, etc. Sounds easy—but remember the vehicle is vertical at the pad, and so is the Spacehab. Getting into it from the Crew Module was both tricky and ingenious. It involved a Rube Goldberg contraption consisting of tripods, hoists, wire rope, bosun’s chair, and nerve. The technician would be sitting in the chair, lowered down into the module via the wire and hoist system, dangling in free air all while installing stuff in lockers on the sidewalls. Easy, huh? In our jargon, the MVAK, the Module Vertical Access Kit, was neat to watch but a nightmare for the guys. But it was also the best thing we had to get the job done vertically. By the way, it would be left in place after using it this week. It would be used again during launch countdown to stow the late, time critical experiments – live worms, frozen samples, etc.

All this work went as scheduled so we had Saturday and Sunday off. Monday morning, however, we’d enter our Launch Countdown procedure and really act like launch was coming!

Next time, why does it take three days to launch the Shuttle?

mvak-diagram
Diagram of Module Vertical Access Kit (MVAK) operation used for a Spacelab module on STS-51B. KSC had two teams trained in MVAK operations. (NASA diagram)

 

Security and STS-107

Having contingency plans for many of the more troublesome possible events during launch was a hallmark of the Shuttle program. Developing these plans and training for them was ‘business as usual’ for us. So was hoping they would never be needed.

On-Pad abort (shutdown of the main engines just prior to T-0), the slidewire emergency egress system, Return to Launch Site Abort, Transoceanic Abort Landing, are four we had. Do you know which one was actually used?

One of the major changes in my job happened following the attacks on 9/11/01. Security for launches was never the same after that day.

Let’s go back to that time. Columbia’s launch was a mere sixteen months after 9/11, and only the seventh one after those attacks. Security around the country was transformed for special events and everyday life. Security for Shuttle launches went from an important but local event to one of national significance and regional impact. Declared a National Asset, the Shuttle “enjoyed” the same level of attention and support as the Super Bowl and presidential trips. For those of us trying to get real work done (launching it!), “enjoy” became the euphemism for enduring it while necessarily becoming part of it. The Launch Director became part of the security system, big time.

Obviously, I can’t say a lot about the specifics, but rest assured we were well protected. The posture for launches before 9/11 was probably about 10% of what we had after in terms of complexity and impact. Before, we cleared certain areas of KSC and CCAFS of visitors and employees to protect them from a launch accident. Afterward, we cleared much more area to also protect us from a possible attack. Ocean exclusion zones expanded, and we had many more assets to clear and secure the area. Lockdown of the land areas became more important and started earlier in the launch campaign. Clearing local and regional airspace was the most significant change, and the most difficult to plan and execute. Airports and airline routes were affected like never before.

The DOD stepped in, rightfully, and thankfully so. They had the knowledge, experience, and assets to pull it off. Integrating their operation and responsibilities into the launch countdown process was a tremendous amount of work, but was required. It was a contingency plan we hoped would never be needed.

Fourteen years ago this week we conducted our final launch security simulation with well over a dozen local, state, and federal agencies participating. We had a special area for their reps in the LCC with links back to their supporting staff. Dedicated communications capability from me to the DOD was tested one final time. Ground, sea, and air assets participated. A subset of the launch team was on hand as were a few shuttle program managers. A simulated airspace violation was declared and we reacted, and reacted well. The sim ended with Columbia in orbit. The plans were tweaked just a little then finalized. This greater team was ready for the real thing.

It was the fourth sim we held in preparation for STS-107. Why so much attention for this mission? The prior seven had less intrusive and expensive “tabletop sims” – usually no real air assets flying for instance. Well, in addition to the (now normal) practice of protecting the shuttle, Ilan Ramon was flying as the first Israeli astronaut, and our intelligence community friends were “concerned.” Security was expanded for STS-107 like no other launch before or since. Without details, we postulated all sorts of possibilities that could affect launch, and protected for all of them. It was a tremendously successful exercise of cooperation between so many agencies that shared a common goal.

Lots of work and expense? You bet. But it was most thorough protection against the unlikely that we ever did.

And it would pay off.

Next week: Launch Day

ksc-02pd-1938-crew-and-m113-tcdt
The STS-107 crew trains for an emergency evacuation with the M-113 armored personnel carrier during TCDT week. (NASA photo KSC-02PD-1938)

Columbia’s final flow

Last time I talked a little bit about Columbia’s final time at the launch pad and why it had to extend over the annual year-end holiday period. This time I’d like to take you back to those days 14 years ago and recall it in more detail.

The final Launch Team training exercise for STS-107 happened on Friday, December 6, 2002. It was the last chance for the whole team to practice together in an environment that simulated launch countdown conditions as closely as possible. It was also the ‘cert run’ for all members to be declared certified to conduct the actual launch in mid January. It went very well with the team solving numerous diabolical problems crafted by the training team to test our knowledge and ability to work under time constraints imposed by a countdown clock. And though the whole thing was conducted against a math model instead of the actual flight and ground hardware, it always felt like launch day itself, less the TV cameras and ties! We were ready. We took the weekend off.

We came back to work Monday morning December 9 and promptly rolled the vehicle out to the Pad. This eight-hour job was normally conducted at night to avoid daytime storms and lightning, but this time of year they weren’t a big threat so we rolled on first shift. I gave the GO to roll and the team did what they did best, getting the Columbia stack (Orbiter, ET, SRBs) on the Mobile Launch Platform to Pad A in a little less than 8 hours on top of the massive Crawler Transporter. Once there, the CT lowered the MLP onto six mounts at the pad and backed away, its job done.

The rest of that week was spent performing a ‘pad validation’ test wherein all systems between the stack and the pad are connected and wrung out. These include all power, communications, data, water, gasses, etc. During this two-day test we performed a confidence run on all three Auxiliary Power Units, as they were replaced in the Orbiter Processing Facility and needed to be tested “outside” and prior to launch day when they’d be needed to power the hydraulic system pumps. Also done this first week were all the preparations for loading the hypergolic fuels for the thrusters, Orbiter Maneuvering System engines, and topping off the APU tanks. In parallel with the hyper preps we conducted a test of the main propulsion system by pressurizing it with helium and looking for leaks. This test was developed after we experienced a series of troublesome leaks in the late 80s that delayed several missions.

The second full week at the pad was dominated by two tasks – loading the hypergolic fuels and conducting the practice launch with the astronauts participating. Hyper load took two full days as did the Terminal Countdown Demonstration Test, with the simulated T-0 at 1100 EST, Friday, December 20. Both went very well and it was great seeing Rick Husband and his crew one last time before launch itself. (I’ve described the traditional crew dinner during TCDT in another post, by the way.) We took both days of the weekend off.

Monday, December 23 was a day of preparing the vehicle and ground systems for the nine days we’d be home with our families before returning to work Thursday, January 2, 2003. But even though most of us were off those days a very dedicated skeleton crew cared for Columbia every day. Walkdowns, visual inspections, security sweeps were all done as a matter of course. We were all glad to see her in such great shape January 2, now just fourteen days away from launch.

columbia-rollout-ksc-02pd-1883-web-version
Columbia passes the Launch Control Center (right) on the way to Pad A for what would be her final launch. (NASA photo KSC-02PD-1883)

Launch fever

In an earlier posting, I talked about ‘schedule pressure’ and how it should have more correctly been referred to as ‘schedule awareness’. Awareness of the schedule contributed to good, productive decision-making. Pressure-based decisions could be shortsighted or worse, destructive.

What about decisions on launch day itself? Hundreds of subordinate schedules and decisions are about to pay off. Pressure? Awareness? Something else?

There’s a phenomenon in the launch business called “Launch Fever.” It, too, is very real. It is never productive and can be destructive. It’s the #1 thing on a Launch Director’s list to recognize and deal with. What is it? Why is it a big deal? What does a LD do about it?

Put yourself in the Shuttle Firing Room on launch day. A tremendous amount of work by thousands of people across the country is about to pay off. The vehicle is full of fuel. The astronauts are all strapped in, their hard work about to pay off big time. TV cameras are rolling. VIPs fill viewing areas, and tens of thousands of tourists line the causeways around the Cape area. Crew families await their loved ones’ achievement. Blue jeans have been replaced by suits and dresses on launch team members. Teams at other NASA and ISS centers are listening. KSC is in full launch mode with security tight on the ground, sea, and in the air. The final schedule achievement is about to be realized. We are about to launch the Space Shuttle. Everyone wants to launch. Our business is to launch. That’s Launch Awareness.

Enter a technical problem, weather issue, fouled Range, or any one of a thousand reasons to question whether we should launch or not. There’s a definite feeling, a palpable mood that sweeps over the collective team. For a while, good decision-making becomes secondary to the desire to launch.

Imagine that: Good decision-making becomes secondary to the desire to launch. Real? It absolutely can be. That’s Launch Fever.

What to do about it? After recognizing that it’s happening, which sometimes can be tricky in the first place, the #1 job of the Launch Director kicks in: Keep that Shuttle on the ground. As the last person to say “GO” it’s incumbent on the LD to have absolutely no reservations to launch. Zero. So even in the face of everyone else wanting to launch by saying GO, my job was to NOT launch the Shuttle unless and until I was ready.

Pressure? Only if you allowed it. Awareness of it was mandatory.

I was asked once what would have happened if on launch day I said NOGO in the face of everyone else being GO. Part of my training, and the training I gave each of my assistant LDs, goes like this. “That’s easy to answer. On that day we would not have launched. The astronauts and the vehicle would be on the ground and and they would be safe. NASA might have a new Launch Director the next day, but on that day we would have been safe.”

I never had to do put that training to use, as good decision-making always prevailed.

exp-nr-endeavor-launch-delayed-cnn-640x360
The final launch of shuttle Endeavour, STS-134, is scrubbed on April 29, 2011 due to problems with heaters on the vehicle’s auxiliary power units. President Obama and his family were on hand but unable to see the launch because of the scrub. (CNN photo)

Schedule pressure

I was asked in a press conference once if ‘schedule pressure’ had affected my decision for a launch. I wanted to throw the reporter out of the room, but I liked my job. Schedule pressure affecting a decision? Instead of a tirade, it was clear the reporter needed a little refresher in the basics of Project Management 101. Here it is:

Work scope. Budget. Schedule. Magic, huh?

Numerous times in the Shuttle program we were accused of letting the schedule affect some decision made. News flash—of course it did! What the reporter was really asking was is if the schedule made me make an unsafe decision. If he had actually asked me that way, then I would have thrown him out.

My job, and indeed the jobs of ALL the great people working so hard to put our astronauts into orbit, was to “get it done, safely.” No exceptions, no excuses for anything less. I found that the suggestion that it was otherwise to be appalling and personally offensive. But we heard it from time to time, and not just from the press. Indeed, to their defense, they had to ask it. Doesn’t make it any more friendly, but it was their job to push for answers. I much preferred the phrase ‘schedule awareness’, and that was how I answered the question. Schedules were just as real as money and scope. But none of the three trumped safety.

Of course we got the “how much did it cost?” question, and much more frequently than the safety question, fortunately. I always liked the cost question because that never drove my decisions, much to the dismay of our program office. My own view was that we had enough money to get the work done to the agreed upon schedule, and a little more. After all, the vast majority of our costs at KSC were labor, so we met the schedule based on the size of the workforce driven by allotted budget. Magic again. More money would have permitted more people working more shifts no doubt, but absent that, we got by with what we had.

Work scope was driven by essentially three factors. There were the normal systems tests, checkouts, maintenance, etc., that were required just to maintain a healthy, functioning program. There was scope, driven by failures and therefore some needed repair or replacement. And then there were system upgrades over the life of the program to modernize and improve safety margins. These could range from relatively minor enhancements, such as upgrades to the slidewire emergency egress system, to major flight hardware improvements, such as upgrades to the Space Shuttle Main Engines aimed at reducing the chances of a shutdown during ascent.

It was during these debates when scope growth was weighed against cost and schedule. And—virtually every upgrade decision made was based on an improvement to some safety margin, or it wouldn’t have made it to the table for discussion in the first place. Some ‘operability’ upgrades were accepted, but I would argue that these had their roots in improving safety by allowing better and/or easier user-to-system interaction.

Project Management 101. Simple. But in manned spaceflight, Safety was the overarching requirement—inescapable and thoroughly embraced. It wasn’t ‘the fourth factor’—it was the paramount factor, not often taught in the typical PM 101 course.

sts133-s-065
Mike Leinbach gives the Go for launch of Discovery  on mission STS-133, February 24, 2011 (NASA photo STS133-S-065)