Returning to a New Normal

As the work in the reconstruction hangar wound down and people gradually returned to their pre-accident jobs, we found ourselves being re-integrated back into a sort of ‘new normal’.

The atmosphere was different, the work itself was different, and the Shuttle was likely on borrowed time. Combine this new normal with the still-present emotional response to the Columbia accident, and you get a workforce with more questions than we could answer, more concern for their futures than confidence—people more in need of direction than ever.

Those of us in leadership and management positions had lots to do dealing with the ongoing CAIB investigation. We were concerned about what it was going to take to get us ready to fly again, debating changes to the External Tank, Orbiter, and other systems. But by far, the most important thing we had to do was to lay out the future for the workforce. The difficulty was that the future was anything but clear for months to come.

Many months.

We needed to stay together as a team despite having no firm game plan. And while everyone understood the uncertainty, it was still an extremely unusual feeling. It would clear up after a couple more months. We would fly again to fulfill international agreements and finish the International Space Station (ISS). But when would we fly again? Would layoffs be coming in the interim? And then once we got back in business, how long would the Shuttle continue in operation? We had originally envisioned flying until 2020, but that was likely to be cut short once ISS assembly was completed.

Open and honest communication throughout all organizations and at all levels became even more important than usual. While we were short on answers, we acknowledged it—and the folks appreciated the candor.

Personally, I thought it was very important to begin to look forward as soon as practical. Not as soon as possible, but as soon as it made sense to do so. In May, 2003 I asked a few close team members what they thought of getting back into launch countdown simulations soon. The responses were split about 50-50. I really wanted to do it to accomplish two main objectives. First, we needed to maintain our proficiency for the inevitable return to flight. Secondly, it would demonstrate to the launch team and to the rest of the processing team that we really were going to fly again. People knew when the team went into training for the day. It was obvious.

So I asked the simulation team to begin to develop a series of training sessions to begin as soon as they could. And on June 1, exactly 4 months after the accident, the Shuttle Launch Team was back together, doing what we did best.

The feeling in the Firing Room that day was unusual to be sure. It was a mix of somber and joy. Reflection and anticipation. But it felt right, too. The “rust” was virtually non-existent, and the team performed exceptionally well.

firing room console
Firing Room 4 launch console, with an open countdown procedure manual from the STS-135 mission. (Photo by Jonathan Ward)

It turned out to be exactly the right thing to do and at the right time. We held sims approximately every six weeks thereafter.

 

As the return to flight plan firmed up, numerous other training sessions were held—Mission Management Team sims, NASA HQ contingency sims, launch sims, landing sims, etc. Everyone got to participate, and rightfully so—because we were going to fly the Shuttle again.

Sometime.

The Rescue Scenario

Shortly after the accident, during the third week in February 2003, a few of us contemplated if a rescue mission of Columbia’s crew could have been conducted. If it could, what were the chances of success?

Under the guidance of Shuttle Program managers we were asked to quietly study it. We were to conduct our studies in part to satisfy our own curiosity and in part knowing the Columbia Accident Investigation Board (CAIB) would no doubt ask us one day. The Flight Directors at Johnson Space Center (JSC) would do the on-orbit assessment, and I would do the Kennedy Space Center (KSC) operations assessment. The two would combine to answer the unfriendly—but necessary—question.

My part concluded that from a pure timing perspective, a rescue was theoretically possible. The result from JSC was the same: theoretically possible. But both required unrealistic assumptions and actions that were not consistent with the mission being flown, or usual program priorities or objectives.

Rescue would have involved having us launch Atlantis—next in line to fly—as soon as possible, rendezvous with Columbia, transfer the astronauts via some sort of tether to Atlantis, and come home. The crew of seven from Columbia would be aboard Atlantis with her rescue crew of four. Four of the crew members would have to ride home strapped to the deck; there were only seven seats on the orbiter. Columbia herself would then be guided to a ditching in the ocean.

At the time of the accident, Atlantis was almost ready to roll out of the Orbiter Processing Facility to the VAB. A full-court press to expedite that and get to the launch pad would be required. The Pad “flow” would be truncated to only those tasks required, the rest omitted to save time. Things like the Terminal Countdown Demonstration Test and cryogenic loading simulation would be eliminated. Other required tasks would be done three shifts per day, seven days per week. Meanwhile, the rescue scenario flight plan would be developed at JSC.

Assuming no significant glitches, launch could have been as early as February 11. This also assumed no significant processing or launch delays occurred, including weather. That also assumed that Atlantis would not have her remote manipulator arm installed, which was almost certainly needed for a rescue mission. Installing the arm would have pushed the earliest launch date to February 13.

If everything went according to plan—and that was a BIG if—the rescue would have happened two days before Columbia‘s consumables ran out. Columbia would have been in orbit for almost a full month by then, two weeks longer than any previous Shuttle mission.

The key to the entire study was that consumables on board Columbia needed to preserved as much as possible, extending Columbia’s time on orbit awaiting Atlantis’ arrival. Food, water, etc. all needed to be stretched to the max. The limiting commodity however were the lithium hydroxide (LiOH) canisters needed to scrub carbon dioxide from the cabin air. Not food, not water, not power, but the ability to provide breathable air for Columbia’s crew.

The assumption made for the study was that we needed to put the crew on alert for extending LiOH no later than Day 4 of the mission. The crew would have had to go into a very low activity mode to keep their respiration as low as possible. This would have had the effect of terminating the mission’s objectives, effectively ending the reason for the mission. To do this would have been one of the unrealistic moves required. AND, to even get to this posture would have required either proof that the Orbiter was fatally damaged by that day, or assuming so. That was another unrealistic assumption, since the request for additional imagery didn’t occur until Day 6 of the mission, by which time it would already have been too late to conserve the consumables.

But when the two studies were combined, we saw that it would have been technically possible to rescue the crew. That’s the cold, data-driven answer. The truth is that the assumptions I mentioned above, and a few others, would have required extraordinary efforts in both ground and mission operations AND management decision making while we were lacking definitive damage information. All this would have been far outside the normal Shuttle practices at the time.

It should also be noted that the decision to actually launch the rescue mission would have been an extraordinary thing in and of itself. Would we commit a crew of four on Atlantis to rescue Columbia’s, crew possibly facing the same damaging foam loss during its launch? A tough decision to say the least, bigger than NASA alone could make. I believe the President would have had a role in that decision.

But it never got to the point that we’d find out.

No rescue mission was ever contemplated during Columbia’s time on orbit, let alone one early enough to give it a fighting chance of success. We just didn’t have the evidence to support making such a decision, and there was no realistic way in which we could have had that evidence by the time that decision needed to be made.

The CAIB asked us about the scenario in early May 2003. Admiral Gehman, a superior leader, intentionally waited to ask the question until some of the raw emotions had time to subside a little.

When we saw the analyses, there was no grumbling, but there was grief. We couldn’t save the ship. Columbia was doomed, no matter what. Maybe we could have saved the crew. But there were so many what-ifs and assumptions, so many things that had to go completely differently from the very first hours of the mission. Would it have been successful? I don’t know. But we never even had the chance to try.

As much as it hurt people to think about the remote possibility of saving Columbia’s crew, the study helped prompt discussions on how to save a future crew of a damaged shuttle. The studies led to the safe-haven scenario, in which damaged Orbiters could dock at the International Space Station to enable the crews to wait there for a later rescue mission.

KSC and JSC used the Columbia rescue scenario to design a one-time rescue mission that could back up the final Hubble servicing mission. After the successful completion of STS-121 in July 2006, proving that we’d finally solved the foam-shedding problem, NASA Administrator Mike Griffin formally approved the Hubble servicing mission.

On May 11, 2009 Atlantis was poised for launch to the Hubble from Pad 39A at Kennedy. Standing on Pad 39B two miles to the north was Endeavour, ready to go into orbit if there were any problems with Atlantis. For the first and only time, NASA had two shuttles in launch countdown simultaneously. We were ready to launch Endeavour one day after Atlantis if necessary. Tremendous dedication and work went into getting us to this dual launch posture. Fortunately—like many other things in the space business—this contingency capability was assured but never needed.

Atlantis’s flight went flawlessly, so the rescue mission never flew. Atlantis’ crew successfully prolonged Hubble’s life and upgraded its instrument package.

In a roundabout way, Columbia had once again contributed to the advancement of scientific discovery.

STS-125 and sts-400
On Pad A (foreground), Atlantis awaits launch for the STS-125 Hubble servicing mission, while Endeavour (STS-400) sits on Pad B for a possible rescue mission. (NASA/Troy Cryder)

The Day We Launched Early

This posting doesn’t relate to the Columbia accident, but it addresses a question I get on my monthly KSC tours offered through the Kennedy Space Center Visitor’s Center.

Why were the launch windows for flights to the International Space Station 10 minutes long, while others—especially early in the program—were 2.5 hours long? The answer is Rendezvous.

Without getting into a mini-course on orbital mechanics, for ISS missions we had to get to a specific point in orbit while the station was there. That’s pretty clear. The trick is that the Shuttle had to do it with the available fuels on board—those in the Solid Rocket Boosters, the Shuttle Main Engines, and the Orbital Maneuvering System and thrusters. Simplistically, that total is called the ‘performance’ of the entire system. The launch window was in large part defined by it. It said we had to launch at a time that when after the Shuttle reached orbit it was close enough to ‘catch’ the ISS with the least amount of maneuvering fuels necessary. If too far away, more fuel would have been needed than was available. Rendezvous would have been impossible. Mission failure. So that permissible distance in orbit defined exactly when launch had to occur. And the least fuel would be used if we launched at that precise time, called the ‘preferred time’. The available maneuvering distance once in orbit said rendezvous could still happen if we launched about 5 minutes before or 5 minutes after the preferred time, again, all based on fuel usage. Thus the 10 minute launch window.

In contrast, for missions not requiring rendezvous, like deploying a satellite or STS-107’s Spacehab mission, just getting to the proper orbit was the main objective. It didn’t matter quite as much when—just that the Shuttle got there. The length of the window was defined by other parameters. Usually the most constraining one was the length of time the astronauts could spend on their backs strapped in the Orbiter awaiting launch. This was 5 hours 15 minutes. Given the time the first astronaut (the Commander) got into the shuttle before launch was usually 2 hours 45 minutes before the scheduled T-0, it resulted in a 2.5 hour launch window. There you go!

But what about ‘launching early’? How can that be?

Flash back to August 10, 2001, Discovery’s launch day for STS-105. The mission was to take the Leonardo Multi Purpose Logistics Module (MPLM) to the International Space Station. Our launch attempt on the previous day was scrubbed due to weather. Today, all was going well in the Firing Room for launch, but a mid-summer storm was pounding Lake Okeechobee, approximately 110 miles south of the launch pad, and the storm was moving north. Heavy rain and lightning were heading at us with no expectation to dissipate. Time of arrival in our area was calculated to be about T-0. Weather launch commit criteria were going to be violated—no way around it. If we scrubbed today, we would have a several-day turnaround before the next launch attempt, as we would have to top off the cryogenics aboard the Shuttle. The only hope was to launch as soon as possible and beat the storm’s arrival.

By rule and practice, how the available launch window was used was at the Launch Director’s discretion (within reason of course!). I called the Flight Director in Houston and discussed my idea to launch at the beginning of the 10-minute window rather than in the middle as we had planned. He liked the idea, so we accelerated countdown beginning about 2 hours prior to liftoff, to save the 5 minutes.

All worked out well. We launched at the opening of the window, 5 minutes early, about 20 minutes before the lightning sensors went out of limits. We beat the storm!

Launching early worked. It was never done before or after. STS-105 holds the distinction as the only manned mission ever to ‘launch early’.

STS-105 launch ksc-01pp-1467
Whew! Discovery beats the storm, as STS-105 becomes the only manned mission to launch ahead of the announced launch time. (NASA photo)

What to Do with the Debris?

Fourteen years ago, in early April 2003, we were about 2/3 of the way through recovering Columbia’s debris from Texas, although we didn’t know it at the time. But the number of debris trucks arriving at the reconstruction hangar at Kennedy Space Center had begun to tail off in the preceding weeks, so we knew at some point they’d stop altogether.

Two initiatives were being worked at that time. First, what to do with the debris, and secondly, how would debris found after operations ended in Texas find its way to us? What were the people finding items after the recovery operations ended to do with the material they found?

I’ll briefly address both now, with the intent to more fully discuss them in subsequent postings.

As stated in a previous entry in this blog, Administrator O’Keefe was instrumental in the decision to learn from Columbia’s accident and in particular, from the debris. Having gotten his unofficial “go” to develop the concept to study the debris, the task to actually put the concept into practice fell on a few of us in the hangar. I asked Scott Thurston, Columbia’s NASA Vehicle Manager, to develop the necessary plans. He did an outstanding job. He and a very small group debated where to store the debris, how to “advertise” that it even existed for study, the requirements for organizations to obtain select pieces, the logistics of lending it to them (it’s not easy lending government property to private organizations), and the proper approval authorities and documents. And, by the way, how to do this for many years to come – also not easy.

The results of their labor and Scott’s leadership are clear. The debris loan program is very much still alive, with several hundred pieces either actively out for study or with studies already concluded. The material is stored in a climate-controlled room in the Vehicle Assembly Building, also allowing easy access for employees to view it. It has a full-time NASA curator—Mike Ciannilli—who also developed and runs NASA’s Apollo, Challenger, Columbia Lessons Learned Program. Mike was very active in the debris recovery as an aerial searcher in Texas, and his passion for sharing the lessons makes him the perfect person for the job.

As with debris from Challenger, some pieces of Columbia continue to be found. To deal with this in Texas, a program involving local authorities is charged with taking calls from anyone finding pieces that may be from Columbia. They in turn call Ciannilli, who is responsible for determining the authenticity of the find and returning the material to KSC to join the other 84,000+ pieces of the ship’s debris.

The most “famous” piece found in this manner was a cryogenic tank from Columbia‘s fuel cell system that had been submerged in Lake Nacogdoches since February, 2003. A severe drought in the summer of 2011 lowered the lake level to the point that the tank was high and dry.

tank lake nacogdoches
An aluminum cryogenic tank from Columbia’s fuel cell system, uncovered in Lake Nacogdoches in August 2011, more than eight years after the accident. (NASA photo)

Numerous other pieces have been found by farmers, ranchers, hikers, etc. I suspect debris will continue to be found from time to time. We know for certain that three of the six main engine turbine pumps are still out there somewhere. But like the three that we recovered, they are no doubt buried deep in the East Texas or Louisiana dirt or at the bottom of a body of water. They will probably never be found.

powerhead
One of Columbia’s powerheads—found buried fourteen feet under the Louisiana mud. (NASA photo)

We officially wrapped up recovery operations in early May, 2003. The vast majority of Columbia that we will ever find is already home. And some of it is being used to advance our understanding in materials and structures subjected to extreme conditions. The goal is to design future spacecraft that can better withstand such conditions. One such example is a seat design capable of withstanding very high torsional forces.

Columbia continues her scientific and research missions, well after her last space flight. That legacy would have made her final crew proud.

Attitudes toward Challenger and Columbia

There are so many aspects of the recovery and reconstruction of Columbia to discuss no one book could possibly tell them all. And this observation is just from my perspective—one person out of the many, many people who contributed one way or the other to the overall effort. Other accounts would add many more personal stories and technical information. There are at least a dozen books on the accident from varying perspectives. One thing we all agree on is that the response of NASA to the loss of Columbia and her crew was vastly different from the loss of Challenger and her crew.

What are the factors that made this true? What evidence is there supporting it? And, most significantly, why is important? Briefly recalling the two accidents begins to tell the story.

Challenger was lost January 28, 1986. It was the 25th Shuttle flight, not yet five years into the program designed to be America’s single launch system. (Recall that at the time, NASA needed to fly as many commercial and military payloads as possible to cost-justify the shuttle.) Although the public was already starting to tune out the Shuttle program, this was a high-profile mission, with “Teacher in Space” Christa McAuliffe on board. The explosion was seen in person and on TV by millions of people. It was horrific. Most of us will never forget the images of the expanding plume of vapor, the solid rocket boosters careening out of control, and then the innumerable vapor trails of objects plummeting from the vapor cloud into the ocean.

Columbia was lost February 1, 2003. It was the 113th shuttle mission, and NASA had been flying space shuttles for more than twenty years. Most Americans probably didn’t even know the mission was being flown. People were blasé about the program and this purely research mission. If you hadn’t gotten up early on that Saturday morning in Dallas or along the debris path in sparsely-populated East Texas, all you saw later that day was a few videos of some trails in the sky. That accident was equally horrific as Challenger‘s. Just as many astronauts died. But precious few people outside of NASA or Texas remember the accident at all. (As crazy as it sounds, there aren’t even any public domain, NASA-taken photographs of the accident itself, since it happened far from the nearest NASA facility.)

The responses to the two by NASA leadership were as different as the missions themselves. The reason is most often described as “the mood of the agency was different.” In 1986 NASA wanted to move on after Challenger, put the accident behind us— writing it off as a one time horrible event—and get back to flying the Shuttle. In 2003 we knew very early after the loss it would probably end the program, but we still wanted to “find the cause, fix it, and fly again.” (We were bound by international treaty to finish the International Space Station.) But we also wanted to learn from the loss.

That’s the most significant difference between Columbia and Challenger. Learn from it. Don’t pretend it didn’t happen.

Bury challenger
Part of Challenger’s fuselage sidewall being lowered into the silo at LC 31/32. (NASA photo)

So much could be said about how this was manifested, but none of it could have happened without the strong leadership of Sean O’Keefe, NASA’s Administrator in 2003. His treatment of the loss, the crew families, the Columbia Accident Investigation Board, and the ‘NASA family’ was nothing short of inspirational. It was exactly what we all needed.

 

The evidence of his leadership is easy to see. Columbia is preserved and used in research into the effects of hypersonic re-entry on materials and structures. It is available for study by any organization with appropriate research goals. It is stored in the VAB. It is also used to educate NASA and contractor employees about the risks of spaceflight and the need for everyone to be vigilant in doing their job the best they can at all times.

By comparison, Challenger is buried in an abandoned Minuteman missile silo at launch complex 31/32 on the Cape Canaveral Air Force Station. The debris fills the deep silo and the side rooms and is sealed with a concrete cap. It is locked away, only accessible in the most extreme cases (once for a Shuttle study on an issue with fuel system flowliner cracks). Not even a sign or marker exists to commemorate it.

This one example of the comparison of the response to the two is easy. To fully describe how it was so completely different would take another book on the organizational, political, and social influences existing at the two times.

And I’m certainly not the right person to do that or one to point fingers at the Challenger leaders and their decisions. But I am one to celebrate the response NASA had to Columbia. It was the right thing to do, lead by the right person to do it.

In later blog entries, we’ll dive deeper into the decision to preserve Columbia and the benefits that have already accrued from that decision.

Tables with debris for loan to JSC 3
Columbia Research and Preservation Office. Wing materials at right are packaged for loan to Johnson Space Center, where they were used to develop on-orbit heat shield repair techniques. (NASA photo, November 2003)

It Bears Repeating

 

I am pleased to report that Jonathan and I submitted the manuscript for Bringing Columbia Home to our editor yesterday.

We’ve been working on this book for nearly two years. In fact, it was two years ago today that we first met, at the funeral of our mutual friend, Norm Carlson.

The past two months felt very much like “Press to MECO” as we went through multiple reviews and revisions to meet our submission date. And just like after a successful Shuttle launch, now I can catch my breath and take the luxury of a little time to pause and reflect on the process that got us to this point and what it means to me.

What was the most significant learning I had in the process of helping Jonathan research and write Bringing Columbia Home? By far, it was how so many American citizens came together so willingly to help us when we needed it most.

Call it what you like. I like ‘the American Spirit.’

We were all hurting from the loss of Columbia. Most of all, the crew families were devastated. No more needs to be said about them.

Those of us in the NASA community were stunned and hurting.

The folks in East Texas were shocked and felt the loss deeply from the very beginning.

The 25,000 people from across America that came together over the course of three months to recover the astronauts and debris came to feel the loss just as much, and as soon as they joined the effort. There was no ‘ramp up’ in emotions.

I’m certain other people around the world felt an emotional connection to the accident as well.

What those of us involved in the recovery and reconstruction shared was something very special. It was the NEED to help. I know the same happens in war, though I have never personally experienced it. It is a need to help your country and comrades. Unique to America? No, but certainly true about us. It is something to be proud of, and to share.

This is precisely why the book will shortly exist.

ALL Americans should know this story of our country’s spirit at its best. They deserve to know it. I believe it’s especially important now when it seems like bickering and divisiveness have become a sort of new norm in our country.

If there’s a message of hope in a story about the aftermath of a terrible national tragedy, it is that Americans are at their very core a compassionate, caring, and committed people who will rise to a challenge and accomplishing incredible things.

The Hidden Cost of Reusability

All space flight providers want to control costs. From the traditional government and commercial enterprises to the new entrants, all want to keep costs as low as reasonably possible. Gone a long time ago—and forever—are the days of unlimited resources to get the job done. Bottom lines matter, and matter a lot.

It has long been argued that one way to keep costs reasonable is to reuse launch hardware. Even when adding the required refurbishment/reflight checks, repairs, component replacements, etc., to the equation it can be shown on paper to save money in the long run. The Shuttle was sold to Congress in part on this philosophy. And being the first reusable spacecraft they could know no different.

Nor could NASA fully know the long-term implications of reusability.

An additional fact is this: Adding astronauts to the equation requires changing ‘reasonably’ to safely. No argument there, right? But what are the implications of this change? Simply put, based on Shuttle experience, it requires lots more of the checks, repairs, component replacements, etc., in an attempt to make the re-flight as technically close to the first as possible. The newer the hardware, the less risk of failure. No argument there either.

These costs are easy to understand and accept. What about the posting’s title? Hidden? I have alluded to it in earlier postings and it’s not just a financial issue. It can cost a lot more than money.

Here it is:

With the desire to refly hardware that has (theoretically) performed well before, a tendency to rely on past performance when approving current flight rationale is given more weight than it should.

That tendency can either be seen overtly, or it can be latent. If obvious, it can be dealt with through open discussion and full debate of the issue at hand. If hidden, it can fester, grow, become a ‘new normal’—or worse, lead to disaster.

Examples in the Shuttle program are numerous. From those we recognized and dealt with—wire chafing, flow liner cracks, hydrogen leaks—to those that dealt us the most severe of consequences—foam shedding, O-ring scorching—we were challenged by the need to re-fly hardware, all with the overarching tenet of doing so safely.

Tragically, we didn’t always balance that correctly.

I sincerely hope today’s and tomorrow’s astronaut launch providers succeed 100% of the time. Recognizing when a ‘new normal’ is kicking in is part of the success criteria.

spacex-falcon-9-crs-9-landing
SpaceX Falcon 9 first stage returns to Earth after the CRS-9 launch. (Photo courtesy SpaceX)