Hurricane Season and Rollbacks

June 1 thru November 30. The Atlantic Hurricane Season. As June 1 passed I was reminded how much ‘fun’ it was to experience.

From the earliest days of our manned spaceflight programs, the Launch Director was responsible for two things that outweighed all others, including launch itself. These were to protect the safety of the workforce, and to protect the flight and ground hardware. Obviously, these are everyone’s responsibility, and everyone took them very seriously. But when major processing decisions were required that had significant safety and/or processing implications, the LD made the final call.

Some of those decisions are well-known, the final decision to launch being the most obvious. Others were also the responsibility of the LD but not as visible. The decision to roll the vehicle out to the launch pad, establishing and enforcing the employee work time rules, and approving personnel exposure to launch pad hazards after external tank (ET) fueling are a few. The decision to roll the vehicle back to the Vehicle Assembly Building (VAB) from the launch pad was also given to the LD (along with a Shuttle Program rep that I’ll explain later).

What could drive us to roll back? Remember the woodpeckers and the damage they did to the ET that couldn’t be fully repaired at the pad? (That was STS-70, in June 1995.) How about the hydrogen leaks in the 1990s?

There was another event that could force us to bring the stack back to the VAB for safekeeping – threatening hurricanes.

If you recall, the vehicle spent about a month at the pad in preparation for launch. Unlike expendable rockets that can spend as little as one day at the pad, the Shuttle needed quite a bit more preparation time before launch. Payload installation, ordnance installations, hypergolic fueling, TCDT—those are just four of the numerous pre-flight jobs that had to be done at the pad. They were required for every mission regardless of the calendar.

Enter Hurricane Season. The Space Shuttle would spend that month of prep time on its seaside launch pad, less than ½ mile from the usually tranquil Atlantic Ocean. On occasion, that tranquility would be broken by the effects from tropical storms and hurricanes. If the Shuttle’s presence on the pad and tropical weather coincided, tough decisions would be required.

Protect the hardware. Protect the people.

The Shuttle program benefitted from the weather forecasts and advice from some of the best meteorologists in the world. The 45th Space Wing of the US Air Force provided weather support for us. Every day—not just during hurricane season, but every day—I would hold a 10-minute call with them for the daily forecast. Numerous processing groups would tie in to benefit from the information and how it might affect their work plans that day. If tropical systems began to form, the intensity on the calls would increase. If there were also a Shuttle at the pad, it would take on an additional air of importance and urgency. Daily calls would increase to twice per day, then every six hours, or even more as the threat got closer.

Ultimately, if the storm track, intensity, and speed combined to actually threaten KSC in the near future we would need to roll the Shuttle back to the VAB to ride out the storm. We could hunker down at the pad if the winds didn’t exceed 60 knots. But if the forecast had higher winds, we needed to get to the safety of the VAB, and this needed to be done early enough that allowed the workers time to get home to deal with their own, final storm preparations.

A lot went into those discussions, but for this entry it can be summarized as balancing the desire to stay at the pad and launch on time vs taking the protection option and delaying launch at least two weeks. But remember: the top priority was safety for the people and hardware. Launch schedules were secondary. But the balance needed to be struck, and it was the responsibility of the Launch Director and the Shuttle Program Manager (delegated to his KSC rep) to do it.

Why would it delay launch at least two weeks if we rolled back to the VAB? The three sets of required tasks, when combined, needed about two weeks to complete—rollback preparations including the roll itself, stay time in the VAB as the storm passed by, and then roll back out and re-perform pad operations undone in Step 1.

To get ready to roll back was at least two full days of activity—usually three—and was done with the storm bearing down on us and our families and homes. How long it took to prepare for rollback depended how close to launch we were when the decision to roll was made. The closer to launch meant more work had been performed that needed to be undone. The Payload Bay Doors needed to be closed after securing the payload. The aft compartment of the Orbiter needed to be closed with special doors, the side hatch closed, etc. Add to these relatively obvious preps things like hypergolic and ordnance systems securing, Pad-to-MLP disconnections (power, comm, gases, data lines, etc.), and you get the idea. The “final” disconnection was this contraption called the ‘9099 interface’ – a large bank of data and power lines on the side of the MLP. Once disconnected, all the work done to verify those systems were launch-ready became history—work that would have to be completely re-done once the Shuttle came back to the launch pad.

We had to roll the crawler out to the launch pad. The rollback itself was typically eight hours or so followed by rudimentary connections of the MLP to the VAB shore systems. Then the final workers could go home and shutter their houses.

Stay time on the VAB was storm-dependent of course, but let’s say three to four days until the “all clear” was declared to reopen KSC. If storm damage existed that would prevent normal work, add that repair time.

The workforce returned to work and got ready to roll back out to the pad. Two days minimum to get ready. Roll out was the same eight hours, and usually at night in the summer to avoid the afternoon thunderstorms. Once at the pad, you have to reconnect everything (reconnecting each interface means going through the complete set of checks to ensure that the connectors are properly mated again) and get back into the same posture as before the decision was made to ride out the storm in the VAB. Add back those three days or so. Then pick a new launch day, verifying that the new launch date doesn’t impact other planned launches at the Cape, other vehicles arriving at or departing the International Space Station, and so on.

All told, the decision to ride out a storm in the VAB meant an impact to the launch schedule of about two weeks.

No magic, just a lot of (necessary) work to protect the flight hardware.

Next week: Tropical Storm Ernesto and the “Half-Rollback”

The Stafford-Covey Return-to-Flight Task Group

International treaty required the United States to complete the core assembly of the International Space Station, up through the installation of Node 2 (later called the Harmony module) as soon as possible. NASA had previously committed to the US Congress that Node 2—onto which the European Space Agency’s Columbus module and the Japanese Kibo module would be berthed—would be launched by February 2004.

While meeting that date was clearly impossible after the Columbia accident, NASA was still compelled to complete its share of the work on the ISS as soon as possible. There were still many flights needed to complete the ISS’s central truss and expand its solar power system before Node 2 could be installed. None of that work was possible without the shuttle. The modules were already built, but there was no other way to get them into space and support the spacewalks necessary to install them. NASA therefore had to get the shuttle flying again.

In May 2003, three months after the accident and before the Columbia Accident Investigation Board (CAIB) had completed its investigation, NASA expected to resume shuttle operations by the end of 2003 or early 2004. NASA wanted to be sure that it was not letting schedule and political pressure force the agency into taking undue risks.

In early May, NASA Deputy Administrator Fred Gregory announced that former astronaut Lt. General Thomas Stafford had been requested to head a group to provide an independent assessment of NASA’s return-to-flight plans. On May 22, 2003, NASA named former shuttle astronaut Richard Covey to report to Stafford and lead a working group to oversee and test NASA’s compliance with the CAIB’s findings and recommendations. Some of the members of the panel included former Secretary of the Navy Richard Danzig, Apollo 8 astronaut Bill Anders (who was also the retired CEO of General Dynamics), and former NASA Launch Director Bob Sieck, among a host of other government and industry executives and technical experts.

NASA Administrator Sean O’Keefe said that NASA would only decide that it was safe to fly the shuttle again when the Administrator had the Stafford-Covey Task Group’s independent confirmation that NASA had fully complied with the CAIB’s recommendations.

Stafford Covey group jsc2003e56782
NASA’s Joy Huff shows a space shuttle leading edge subsystems panel to members of the Stafford-Covey Task Group in August 2003. From left: Dr. Amy Donahue, David Lengyel, Dr. Katherine Clark, Richard Covey, and William Wegner. (NASA photo)

The Task Group went into full operation once the CAIB’s report was issued in August 2003. The CAIB made 15 specific recommendations that NASA needed to address before the shuttle could return to flight. Many of those findings required extensive changes to hardware, procedures, and management practices.

NASA’s hopes of flying again in 2003 or 2004 quickly were overtaken by the realization that there was a long and difficult road ahead. By December 2003, the planned launch date had moved to September 2004. However, an interim report by the Stafford-Covey Task Group that month said that “progress on the many recommendations is uneven” and that it was too soon to say whether that new launch date was possible. The Task Group’s interim report also chided NASA for not being timely in responding to some requests for information.

It was not comfortable information for O’Keefe to hear. However, it meant that the Task Group was doing its job of being “an umpire calling balls and strikes in a zone defined by the CAIB recommendations.”

The Task Group issued additional interim reports in April 2004 and January 2005, noting progress as well as areas that still required attention.

By June 2005, NASA had closed out all but three of the CAIB’s recommendations. The Task Group believed that the three remaining recommendations were so challenging that NASA could not comply completely with the intent of the CAIB. For example, the most contentious open item was a vaguely-worded recommendation that NASA have the ability to repair the “widest possible range of damage” when the shuttle was on orbit.

In July 2005, the Task Group was satisfied that NASA had done everything in its power to make the shuttle as safe as possible to fly again, and they told the Administrator that NASA had met the intent of the CAIB’s requirements for returning to flight. The Task Group’s final report made it clear, however, that it was up to the NASA Administrator and his staff—not the CAIB or the Task Group—to determine if the remaining risk was low enough to allow the shuttle to fly.

Shuttle Discovery launched on the STS-114 mission on July 26, 2005. Although the external tank unexpectedly (alarmingly) shed foam again, the safety inspection and repair techniques that NASA developed in the wake of the CAIB report ensured that the crew was able to complete their mission and return safely to Earth.

Returning to a New Normal

As the work in the reconstruction hangar wound down and people gradually returned to their pre-accident jobs, we found ourselves being re-integrated back into a sort of ‘new normal’.

The atmosphere was different, the work itself was different, and the Shuttle was likely on borrowed time. Combine this new normal with the still-present emotional response to the Columbia accident, and you get a workforce with more questions than we could answer, more concern for their futures than confidence—people more in need of direction than ever.

Those of us in leadership and management positions had lots to do dealing with the ongoing CAIB investigation. We were concerned about what it was going to take to get us ready to fly again, debating changes to the External Tank, Orbiter, and other systems. But by far, the most important thing we had to do was to lay out the future for the workforce. The difficulty was that the future was anything but clear for months to come.

Many months.

We needed to stay together as a team despite having no firm game plan. And while everyone understood the uncertainty, it was still an extremely unusual feeling. It would clear up after a couple more months. We would fly again to fulfill international agreements and finish the International Space Station (ISS). But when would we fly again? Would layoffs be coming in the interim? And then once we got back in business, how long would the Shuttle continue in operation? We had originally envisioned flying until 2020, but that was likely to be cut short once ISS assembly was completed.

Open and honest communication throughout all organizations and at all levels became even more important than usual. While we were short on answers, we acknowledged it—and the folks appreciated the candor.

Personally, I thought it was very important to begin to look forward as soon as practical. Not as soon as possible, but as soon as it made sense to do so. In May, 2003 I asked a few close team members what they thought of getting back into launch countdown simulations soon. The responses were split about 50-50. I really wanted to do it to accomplish two main objectives. First, we needed to maintain our proficiency for the inevitable return to flight. Secondly, it would demonstrate to the launch team and to the rest of the processing team that we really were going to fly again. People knew when the team went into training for the day. It was obvious.

So I asked the simulation team to begin to develop a series of training sessions to begin as soon as they could. And on June 1, exactly 4 months after the accident, the Shuttle Launch Team was back together, doing what we did best.

The feeling in the Firing Room that day was unusual to be sure. It was a mix of somber and joy. Reflection and anticipation. But it felt right, too. The “rust” was virtually non-existent, and the team performed exceptionally well.

firing room console
Firing Room 4 launch console, with an open countdown procedure manual from the STS-135 mission. (Photo by Jonathan Ward)

It turned out to be exactly the right thing to do and at the right time. We held sims approximately every six weeks thereafter.

 

As the return to flight plan firmed up, numerous other training sessions were held—Mission Management Team sims, NASA HQ contingency sims, launch sims, landing sims, etc. Everyone got to participate, and rightfully so—because we were going to fly the Shuttle again.

Sometime.

Columbia’s Reconstructed Debris Tells a Compelling Story

Throughout February, March, and April of 2003, truckloads of Columbia’s debris retrieved in East Texas arrived at the reconstruction hangar at Kennedy Space Center. In the first month or so, several truckloads arrived every week. By April, those deliveries were beginning to tail off as searchers completed clearing the debris field.

Workers in the field knew that NASA was conducting important analyses on the material they’d found and shipped to KSC, but they had no concept of the scope of the operation until they saw the reconstruction hangar themselves. NASA’s Mike Ciannilli (who now runs NASA’s Columbia Recovery Office and the Apollo 1, Challenger, and Columbia Lessons Learned Program) worked as an aerial spotter in East Texas during the recovery period. He said, “We had no clue what was going on at KSC. The first time I went to the hangar, I was blown out of the water. I was just mesmerized by it.”

By the end of April, the debris had begun to tell a very clear story about Columbia’s demise. Almost all of the debris showed the effects of extreme heat, oxidation, and mechanical stress to varying degrees. But what damage happened before the ship broke apart, and what happened afterward? Painstaking forensic analysis was required to gradually tease apart the sequence of the accident.

Tile Tables

Reconstruction workers initially used a computer graphics package (originally developed to track the process of waterproofing the tiles during launch processing) to plot which of the insulating tiles had been recovered from Columbia’s wings. It quickly became obvious that the software was inadequate to provide the information needed for the forensic analysis. Which tiles were damaged? How were they damaged? Were there larger-scale patterns of burns, pitting, or other damage?

The solution was to lay the tiles out on a large, elevated table on which was drawn the outline of the wing and the location and part number of every single one of the 2,800 silica tiles on the left wing. As tiles were recovered and identified (sometimes based only on a few millimeters of thickness), engineers lay them in place on the table.

Patterns quickly emerged.

Micklos tile table
Thermal protection system lead engineer Ann Micklos places a tile on the left wing tile table. (Photo courtesy Ann Micklos)

The carrier panel tiles, which formed the closeout between the leading edge and the rest of the wing, showed where the breach in the wing occurred. The carrier panel tiles behind Panel 9 on the leading edge were slumped and coated with metallic deposits from structures in the interior of the wing.

Burn patterns showed that plasma entering the breach in the wing at thousands of miles per hour pressurized the wing cavity and blew out vents in the upper and lower surfaces of the wing.

As the hot plasma entered and melted its way through the wing’s interior, its heat caused the adhesive that held the tiles onto the wing surface to fail. Those tiles—designed to take heat from the outside surfaces, not from where they were glued onto the ship—peeled off of the wing.

The wing was clearly baking from the inside out.

Finally, the tiles found farthest west in the debris field—those which had come off the earliest in the moments leading to the breakup, as the shuttle was flying from west to east across Texas—were all from the left wing, and all from the area behind leading edge panels 8 and 9.

Leading Edge Frames

Boeing’s Mike Gordon and NASA structures engineer Lyle Davis spent many long hours examining the recovered pieces of the reinforced carbon-carbon (RCC) panels on Columbia’s wing’s leading edges. These pieces ranged in size from larger than one square foot to smaller than a thumbnail.

Since the RCC panels were U-shaped and several feet long, it was impractical to lay them out two-dimensionally. Kennedy Space Center’s shops constructed a set of clear Lexan frames that held the pieces of each RCC panel and its underlying structure, allowing people to see in three dimensions what the front edge of the wing looked like.

KSC-03pd-1556 05-14-2003 Leinbach and RCC Lexan media briefing
Columbia reconstruction director Mike Leinbach briefs the media about the wing leading edge displays, May 14, 2003. (NASA photo)

The RCC frames also told a compelling story about the accident. Panels 1 though 7, and from Panel 10 outboard, were fractured from forces after the shuttle disintegrated. Their metallic attachment fittings were still partially intact.

However, at Panels 8 and 9, no metallic fittings were recovered. The stainless steel support structure—which melts at 2,500°F—had melted away and was deposited as slag onto the interior of adjacent panels on the wing. The retrieved pieces of RCC from Panels 8 and 9 were knife-edged and heavily eroded, providing evidence that plasma at over 3,000°F was acting as a blowtorch at high pressure on those panels for a prolonged period. This was the only place on the wing where this pattern was observed.

Materials scientists analyzed the cross-section of the slag deposits inside the left wing panels. This analysis showed what the metals were (insulating blankets, aluminum from wing spars, Inconel and steel from fittings, etc.) and how the deposits were laid down. It clearly told the story of the wing’s failure in time sequence.

As with the tiles, the location of the leading edge components in the debris field also gave the sequence of the accident. The leading edge components from the middle to the tip of the left wing (Panels 8 to 22) were found farthest west, between Dallas and Palestine. Pieces of the leading edge of the left wing closest to the shuttle’s fuselage (Panels 1 to 7) were found farther east, between Palestine and Nacogdoches. The right wing leading edge pieces were all found farther east. This provided evidence that the left wing failed before the right wing, with the most likely point of failure being at Panels 8 and 9.

Proof Positive

NASA now had the evidence to prove how Columbia broke up.

NASA announced in early May that only one single failure scenario explained all of the evidence contained in the debris, the OEX recorder, and the telemetry received in Houston. Something—most likely the piece of foam from the external tank during ascent to orbit—caused a breach in the leading edge of the left wing, allowing plasma to penetrate the wing during reentry and erode it from the inside out.

Whether the foam actually punctured the RCC at the leading edge, or whether it pushed the spacers apart and created a gap in the panels, will never be conclusively known.

The last truckload of debris arrived at the reconstruction hangar on May 6. Wreckage that had been strewn over thousands of square miles of forest and field—pieces that had been carefully collected by tens of thousands of searchers working in tough conditions for three months—was all here in this one place. The twisted, burned, and shattered debris had told an important story, one that would have been impossible to write without the collective efforts of so many dedicated people in East Texas and Louisiana and of the nation’s wildland firefighting teams.

“Each piece was evidence of how hard Columbia fought to come home to us,” astronaut Pam Melroy said. “We saw every recovered piece as a victory.” Every piece of debris moved the reconstruction team closer to their goal: We will find the problem, fix it, and move forward in their honor.

[Portions excerpted from “Bringing Columbia Home,” © 2017 by Mike Leinbach and Jonathan Ward.]

The Columbia Recovery Phase Ends

Thanks to the tireless and incredibly efficient efforts of the Texas Forest Service, the US Forest Service, FEMA, EPA, and NASA, recovery operations wrapped up at the end of April and beginning of May in 2003.

From the middle of February through the end of April, the Type 1 and Type 2 wildland fire crews from the US Forest Service walked every square foot of an area larger than the state of Rhode Island in their search for debris from Columbia. They painstakingly searched forests, fields, briar patches, farms, ranches, swamps. They dodged bulls, avoided snakes, endured heat and cold, suffered through hailstorms, scratched bug bites, steered clear of suspected meth labs, missed their families, and slept in tents during their two to three weeks in the field. Grid searches turned up thousands of pieces of shuttle material that on average was about one square inch—and in many cases, smaller than a fingernail.

Air crews  logged over 5,000 flight hours in their search efforts. Divers from the Navy, FBI, Houston Police Department, EPA, Texas Department of Public Safety, and the Galveston Police Department conducted more than 3,000 dives and spent more than 800 hours on the bottom of lakes searching for debris from Columbia. The overall water search effort covered twenty-three square miles of lake bed.

Nearly 25,000 men and women from almost every US state participated in the search operations. The combined effort was over 1.5 million man hours. Searchers recovered more than 84,700 pounds of material from Columbia, equal to about 38 percent of the vehicle’s landing weight. Most emotionally important, the remains of Columbia‘s crew had been recovered and returned to their loved ones.

It was the largest land search and recovery operation in United States history, and the first major incident under the jurisdiction of the new Department of Homeland Security.

Animated graphic showing the progress of the grid searches for Columbia‘s material along the 250-mile-long debris path from February through April 2003. (Courtesy Mark Stanford, Texas Forest Service)

The Palestine, Texas camp closed on April 18. A few days later, the Hemphill collection center closed. Search operations at the western end of the debris field continued for a few more weeks, until the number of pieces being recovered was less than one per grid. All ground operations in Texas ended on April 30, with the Nacogdoches camp closing on May 3 and the Corsicana camp closing the following day. Search operations moved to Utah on May 2 for eight days, as radar had tracked some objects falling off the shuttle during its flight over the state. However, no shuttle debris was ever recovered west of Littlefield, Texas.

 

NASA’s Space Flight Awareness organization sponsored a huge dinner at the Lufkin Civic Center on April 29 to celebrate the end of the search operations and to thank the local communities and agencies for their help. The scale of the event was impressive. NASA’s Ed Mango likened it to the celebration scene in the movie, The Right StuffJan Amen from the Texas Forest Service reported, “Dinner was steak and chicken, green beans, rice, rolls, salad, pie, all prepared by the Diboll Country Club. Free drinks flowed freely.”

It was the kind of party that only Texans know how to throw.

Banquet - The Crowd
Part of the crowd at the banquet in the Lufkin Civic Center on April 29, 2003. (Photo by Jan Amen)

Administrator Sean O’Keefe hosted the event for NASA, and Scott Wells spoke on behalf of FEMA. County judges and civic leaders from every county in East Texas were on hand. County Judge Jack Leath, Tom Maddox, Greg Cohrs, Roger and Belinda Gay, Marsha Cooper, and a host of other people represented Sabine County. Dignitaries from the various Native American Tribes and Nations attended. An astronaut sat at every table.

Astronaut Dom Gorie opened the ceremony with a heartfelt invocation that brought tears to the eyes of nearly everyone present. The Expedition Six crew sent a live video message from the International Space Station. A video about the Columbia crew followed.

O’Keefe and NASA’s Dave King, who was in charge of the recovery operation, presented plaques recognizing the nation’s appreciation for the contributions of the people at every table in the hall. The spouses of Columbia’s crew spoke of their gratitude to the people of East Texas for bringing their loved ones home again. Eileen Collins closed the ceremony on behalf of the next shuttle crew scheduled to fly in space.

Banquet - Evelyn Husband
Evelyn Husband, wife of Columbia’s Commander Rick Husband, conveys her thanks to the people of Texas. (Photo by Jan Amen)

It was a fitting and emotional close to a tumultuous three months. The people of East Texas had provided the nation and the world with an enduring lesson in how to handle a crisis with dignity, compassion, and competence. They met and worked side by side with astronauts, rocket scientists, engineers, technicians, and fire crews from across the country. In return, they had the satisfaction of knowing that they had gone far beyond the call of duty in the hopes of returning the American shuttle fleet to flight again.

At the end of the evening, after Jan Amen dropped off her last load of astronauts and families at their hotels in Lufkin, she wrote to a friend, “I absolutely lost it. I squalled all the way back to Cudlipp like a big fat crybaby. I’m whooped!”

[Portions of this blog post are excerpted from Bringing Columbia Home, (c) 2017 by Mike Leinbach and Jonathan Ward.]

The Rescue Scenario

Shortly after the accident, during the third week in February 2003, a few of us contemplated if a rescue mission of Columbia’s crew could have been conducted. If it could, what were the chances of success?

Under the guidance of Shuttle Program managers we were asked to quietly study it. We were to conduct our studies in part to satisfy our own curiosity and in part knowing the Columbia Accident Investigation Board (CAIB) would no doubt ask us one day. The Flight Directors at Johnson Space Center (JSC) would do the on-orbit assessment, and I would do the Kennedy Space Center (KSC) operations assessment. The two would combine to answer the unfriendly—but necessary—question.

My part concluded that from a pure timing perspective, a rescue was theoretically possible. The result from JSC was the same: theoretically possible. But both required unrealistic assumptions and actions that were not consistent with the mission being flown, or usual program priorities or objectives.

Rescue would have involved having us launch Atlantis—next in line to fly—as soon as possible, rendezvous with Columbia, transfer the astronauts via some sort of tether to Atlantis, and come home. The crew of seven from Columbia would be aboard Atlantis with her rescue crew of four. Four of the crew members would have to ride home strapped to the deck; there were only seven seats on the orbiter. Columbia herself would then be guided to a ditching in the ocean.

At the time of the accident, Atlantis was almost ready to roll out of the Orbiter Processing Facility to the VAB. A full-court press to expedite that and get to the launch pad would be required. The Pad “flow” would be truncated to only those tasks required, the rest omitted to save time. Things like the Terminal Countdown Demonstration Test and cryogenic loading simulation would be eliminated. Other required tasks would be done three shifts per day, seven days per week. Meanwhile, the rescue scenario flight plan would be developed at JSC.

Assuming no significant glitches, launch could have been as early as February 11. This also assumed no significant processing or launch delays occurred, including weather. That also assumed that Atlantis would not have her remote manipulator arm installed, which was almost certainly needed for a rescue mission. Installing the arm would have pushed the earliest launch date to February 13.

If everything went according to plan—and that was a BIG if—the rescue would have happened two days before Columbia‘s consumables ran out. Columbia would have been in orbit for almost a full month by then, two weeks longer than any previous Shuttle mission.

The key to the entire study was that consumables on board Columbia needed to preserved as much as possible, extending Columbia’s time on orbit awaiting Atlantis’ arrival. Food, water, etc. all needed to be stretched to the max. The limiting commodity however were the lithium hydroxide (LiOH) canisters needed to scrub carbon dioxide from the cabin air. Not food, not water, not power, but the ability to provide breathable air for Columbia’s crew.

The assumption made for the study was that we needed to put the crew on alert for extending LiOH no later than Day 4 of the mission. The crew would have had to go into a very low activity mode to keep their respiration as low as possible. This would have had the effect of terminating the mission’s objectives, effectively ending the reason for the mission. To do this would have been one of the unrealistic moves required. AND, to even get to this posture would have required either proof that the Orbiter was fatally damaged by that day, or assuming so. That was another unrealistic assumption, since the request for additional imagery didn’t occur until Day 6 of the mission, by which time it would already have been too late to conserve the consumables.

But when the two studies were combined, we saw that it would have been technically possible to rescue the crew. That’s the cold, data-driven answer. The truth is that the assumptions I mentioned above, and a few others, would have required extraordinary efforts in both ground and mission operations AND management decision making while we were lacking definitive damage information. All this would have been far outside the normal Shuttle practices at the time.

It should also be noted that the decision to actually launch the rescue mission would have been an extraordinary thing in and of itself. Would we commit a crew of four on Atlantis to rescue Columbia’s, crew possibly facing the same damaging foam loss during its launch? A tough decision to say the least, bigger than NASA alone could make. I believe the President would have had a role in that decision.

But it never got to the point that we’d find out.

No rescue mission was ever contemplated during Columbia’s time on orbit, let alone one early enough to give it a fighting chance of success. We just didn’t have the evidence to support making such a decision, and there was no realistic way in which we could have had that evidence by the time that decision needed to be made.

The CAIB asked us about the scenario in early May 2003. Admiral Gehman, a superior leader, intentionally waited to ask the question until some of the raw emotions had time to subside a little.

When we saw the analyses, there was no grumbling, but there was grief. We couldn’t save the ship. Columbia was doomed, no matter what. Maybe we could have saved the crew. But there were so many what-ifs and assumptions, so many things that had to go completely differently from the very first hours of the mission. Would it have been successful? I don’t know. But we never even had the chance to try.

As much as it hurt people to think about the remote possibility of saving Columbia’s crew, the study helped prompt discussions on how to save a future crew of a damaged shuttle. The studies led to the safe-haven scenario, in which damaged Orbiters could dock at the International Space Station to enable the crews to wait there for a later rescue mission.

KSC and JSC used the Columbia rescue scenario to design a one-time rescue mission that could back up the final Hubble servicing mission. After the successful completion of STS-121 in July 2006, proving that we’d finally solved the foam-shedding problem, NASA Administrator Mike Griffin formally approved the Hubble servicing mission.

On May 11, 2009 Atlantis was poised for launch to the Hubble from Pad 39A at Kennedy. Standing on Pad 39B two miles to the north was Endeavour, ready to go into orbit if there were any problems with Atlantis. For the first and only time, NASA had two shuttles in launch countdown simultaneously. We were ready to launch Endeavour one day after Atlantis if necessary. Tremendous dedication and work went into getting us to this dual launch posture. Fortunately—like many other things in the space business—this contingency capability was assured but never needed.

Atlantis’s flight went flawlessly, so the rescue mission never flew. Atlantis’ crew successfully prolonged Hubble’s life and upgraded its instrument package.

In a roundabout way, Columbia had once again contributed to the advancement of scientific discovery.

STS-125 and sts-400
On Pad A (foreground), Atlantis awaits launch for the STS-125 Hubble servicing mission, while Endeavour (STS-400) sits on Pad B for a possible rescue mission. (NASA/Troy Cryder)

The Day We Launched Early

This posting doesn’t relate to the Columbia accident, but it addresses a question I get on my monthly KSC tours offered through the Kennedy Space Center Visitor’s Center.

Why were the launch windows for flights to the International Space Station 10 minutes long, while others—especially early in the program—were 2.5 hours long? The answer is Rendezvous.

Without getting into a mini-course on orbital mechanics, for ISS missions we had to get to a specific point in orbit while the station was there. That’s pretty clear. The trick is that the Shuttle had to do it with the available fuels on board—those in the Solid Rocket Boosters, the Shuttle Main Engines, and the Orbital Maneuvering System and thrusters. Simplistically, that total is called the ‘performance’ of the entire system. The launch window was in large part defined by it. It said we had to launch at a time that when after the Shuttle reached orbit it was close enough to ‘catch’ the ISS with the least amount of maneuvering fuels necessary. If too far away, more fuel would have been needed than was available. Rendezvous would have been impossible. Mission failure. So that permissible distance in orbit defined exactly when launch had to occur. And the least fuel would be used if we launched at that precise time, called the ‘preferred time’. The available maneuvering distance once in orbit said rendezvous could still happen if we launched about 5 minutes before or 5 minutes after the preferred time, again, all based on fuel usage. Thus the 10 minute launch window.

In contrast, for missions not requiring rendezvous, like deploying a satellite or STS-107’s Spacehab mission, just getting to the proper orbit was the main objective. It didn’t matter quite as much when—just that the Shuttle got there. The length of the window was defined by other parameters. Usually the most constraining one was the length of time the astronauts could spend on their backs strapped in the Orbiter awaiting launch. This was 5 hours 15 minutes. Given the time the first astronaut (the Commander) got into the shuttle before launch was usually 2 hours 45 minutes before the scheduled T-0, it resulted in a 2.5 hour launch window. There you go!

But what about ‘launching early’? How can that be?

Flash back to August 10, 2001, Discovery’s launch day for STS-105. The mission was to take the Leonardo Multi Purpose Logistics Module (MPLM) to the International Space Station. Our launch attempt on the previous day was scrubbed due to weather. Today, all was going well in the Firing Room for launch, but a mid-summer storm was pounding Lake Okeechobee, approximately 110 miles south of the launch pad, and the storm was moving north. Heavy rain and lightning were heading at us with no expectation to dissipate. Time of arrival in our area was calculated to be about T-0. Weather launch commit criteria were going to be violated—no way around it. If we scrubbed today, we would have a several-day turnaround before the next launch attempt, as we would have to top off the cryogenics aboard the Shuttle. The only hope was to launch as soon as possible and beat the storm’s arrival.

By rule and practice, how the available launch window was used was at the Launch Director’s discretion (within reason of course!). I called the Flight Director in Houston and discussed my idea to launch at the beginning of the 10-minute window rather than in the middle as we had planned. He liked the idea, so we accelerated countdown beginning about 2 hours prior to liftoff, to save the 5 minutes.

All worked out well. We launched at the opening of the window, 5 minutes early, about 20 minutes before the lightning sensors went out of limits. We beat the storm!

Launching early worked. It was never done before or after. STS-105 holds the distinction as the only manned mission ever to ‘launch early’.

STS-105 launch ksc-01pp-1467
Whew! Discovery beats the storm, as STS-105 becomes the only manned mission to launch ahead of the announced launch time. (NASA photo)

What to Do with the Debris?

Fourteen years ago, in early April 2003, we were about 2/3 of the way through recovering Columbia’s debris from Texas, although we didn’t know it at the time. But the number of debris trucks arriving at the reconstruction hangar at Kennedy Space Center had begun to tail off in the preceding weeks, so we knew at some point they’d stop altogether.

Two initiatives were being worked at that time. First, what to do with the debris, and secondly, how would debris found after operations ended in Texas find its way to us? What were the people finding items after the recovery operations ended to do with the material they found?

I’ll briefly address both now, with the intent to more fully discuss them in subsequent postings.

As stated in a previous entry in this blog, Administrator O’Keefe was instrumental in the decision to learn from Columbia’s accident and in particular, from the debris. Having gotten his unofficial “go” to develop the concept to study the debris, the task to actually put the concept into practice fell on a few of us in the hangar. I asked Scott Thurston, Columbia’s NASA Vehicle Manager, to develop the necessary plans. He did an outstanding job. He and a very small group debated where to store the debris, how to “advertise” that it even existed for study, the requirements for organizations to obtain select pieces, the logistics of lending it to them (it’s not easy lending government property to private organizations), and the proper approval authorities and documents. And, by the way, how to do this for many years to come – also not easy.

The results of their labor and Scott’s leadership are clear. The debris loan program is very much still alive, with several hundred pieces either actively out for study or with studies already concluded. The material is stored in a climate-controlled room in the Vehicle Assembly Building, also allowing easy access for employees to view it. It has a full-time NASA curator—Mike Ciannilli—who also developed and runs NASA’s Apollo, Challenger, Columbia Lessons Learned Program. Mike was very active in the debris recovery as an aerial searcher in Texas, and his passion for sharing the lessons makes him the perfect person for the job.

As with debris from Challenger, some pieces of Columbia continue to be found. To deal with this in Texas, a program involving local authorities is charged with taking calls from anyone finding pieces that may be from Columbia. They in turn call Ciannilli, who is responsible for determining the authenticity of the find and returning the material to KSC to join the other 84,000+ pieces of the ship’s debris.

The most “famous” piece found in this manner was a cryogenic tank from Columbia‘s fuel cell system that had been submerged in Lake Nacogdoches since February, 2003. A severe drought in the summer of 2011 lowered the lake level to the point that the tank was high and dry.

tank lake nacogdoches
An aluminum cryogenic tank from Columbia’s fuel cell system, uncovered in Lake Nacogdoches in August 2011, more than eight years after the accident. (NASA photo)

Numerous other pieces have been found by farmers, ranchers, hikers, etc. I suspect debris will continue to be found from time to time. We know for certain that three of the six main engine turbine pumps are still out there somewhere. But like the three that we recovered, they are no doubt buried deep in the East Texas or Louisiana dirt or at the bottom of a body of water. They will probably never be found.

powerhead
One of Columbia’s powerheads—found buried fourteen feet under the Louisiana mud. (NASA photo)

We officially wrapped up recovery operations in early May, 2003. The vast majority of Columbia that we will ever find is already home. And some of it is being used to advance our understanding in materials and structures subjected to extreme conditions. The goal is to design future spacecraft that can better withstand such conditions. One such example is a seat design capable of withstanding very high torsional forces.

Columbia continues her scientific and research missions, well after her last space flight. That legacy would have made her final crew proud.

The Recovery Passes the Halfway Mark

At the beginning of April 2003, the search efforts for recovery of Columbia‘s debris passed the halfway mark.

From the time operations went into full swing at the end of the third week of February, the Texas Forest Service had overseen the mobilization of more than 12,200 men and women from the National Wildfire Coordinating Group. These firefighters came from more than forty states and Puerto Rico.

April 1 ground search status
Firefighters came from nearly every US state and Puerto Rico to search for Columbia’s wreckage. This map shows the number of wild land firefighters and support staff deployed by each state as of April 1, 2003. 

They entered the recovery zone through a processing facility set up by the Texas Forest Service in Longview, Texas. After an orientation on what they were looking for and the hazards they might encounter, the fire crews and their supervisory Incident Management Teams were deployed to camps in Corsicana, Palestine, Nacogdoches, and Hemphill. These towns were spaced roughly fifty miles apart along the debris field. The crews then spent two to three weeks conducting grid searches in their assigned areas. Then they were rotated out and replaced by fresh crews.

Longview cots
Cots for transiting fire crews in the Longview staging camp. (Photo by Jan Amen)

Their efforts produced astonishing results. As of April 2, 2003, the crews had searched every foot of an area of 426,844 acres (667 square miles). They had recovered 65,730 pounds of material from Columbia, equal to about 29% of the vehicle’s weight. Their efforts were also being supplemented by 37 helicopters, 8 fixed-wing aircraft, and salvage divers and surface boats in Lake Nacogdoches and the Toledo Bend Reservoir.

In this first incident response by the newly-created Department of Homeland Security, FEMA coordinated the federal agencies and funded the operations. NASA managed the overall search and provided technical assistance. The Environmental Protection Agency identified and handled hazardous materials, and transported all materials recovered during the search. The Texas Forest Service coordinated the air and ground searches. The US Forest Service, the Bureau of Land Management, the Bureau of Indian Affairs, and the National Park Service provided the majority of the search crews and equipment. The US Navy and Coast Guard conducted the water searches.

Those are just some of the lead agencies. More than ninety federal, state, and local government agencies assisted in some way with the aftermath of the Columbia accident.

It was on the surface a collaboration of unlikely partners, but each agency brought its core expertise to bear in the largest and most remarkable inter-agency operation ever conducted.

Astronaut Jerry Ross told me during an interview for the book that “people first and foremost need to understand the greatness of the United States and its citizens. The United Sates has an incredible wealth of capabilities. To see the energy and expertise and materials and technical capabilities that descended on Lufkin within hours of the accident was so reassuring.”

In times when we hear people complaining about “government incompetence,” it’s helpful to remember that agencies are made up of people. People—not faceless agencies—get the work done. And we need to know that our federal and state agencies are made up of a lot of motivated and dedicated people who want to see our country be successful.

“Their Mission Became Our Mission”

In East Texas, people make sure you are aware that nine people lost their lives as a result of the Columbia accident—the seven astronauts and two searchers.

Supplementing the massive ground search for Columbia‘s debris was an aerial search of the area on several miles on each side of the ground search corridor. The Texas Forest Service managed more than thirty contracted helicopters that were operating in the air over East Texas every day.

Pilots flew just above treetop level, as spotters searched for debris on the ground. The spotters generally included representatives from the Texas Forest Service, the US Forest Service, and NASA or its contractor United Space Alliance (USA).

The air searches were not without risk. Flying just above the treetops, there was often little room for a pilot to make an emergency landing if something went wrong. Makeshift power lines cropped up in strange places in the rural areas, posing hazards to air crews who attempted to land to pick up debris.

On the afternoon March 27, 2003, luck ran out for one of the air search crews. Pilot Jules “Buzz” Mier was flying his Bell 407 helicopter over the Angelina National Forest that day, accompanied by Charles Krenek of the Texas Forest Service in the front seat. In the rear of the helicopter were Matt Tschacher of the US Forest Service, Richard Lange from USA, and Ronnie Dale from NASA’s Safety and Process Assurance Branch at Kennedy Space Center.

Mier was an Army Vietnam veteran with thousands of hours of flight experience. He had a business operating air tours in the Grand Canyon. When he heard that the country needed experienced pilots for the Columbia search, he answered the call to service.

Krenek was a resident of Lufkin, Texas. He had twenty-six years of experience as an aviation specialist and a woodland firefighter. He was well-known and well-liked throughout East Texas.

Krenek-Mier crew
The helicopter crew on the day of the accident. From left: Matt Tschacher, Buzz Mier, Richard Lange, Ronnie Dale, and Charles Krenek. (Photo courtesy “Boo” Walker)

After a stop for lunch and refueling, Mier and his crew took off at 3:15 on Thursday, March 27 for their second search mission of the day. About an hour into the flight, the engine suddenly quit. Mier, flying just above the treetops over Ayish Bayou, had no options for autorotating to a safe landing. The helicopter crashed nose-first into the crown of an oak tree and plummeted to the ground. Mier and Krenek were killed instantly when the cockpit was crushed. The three men in the rear seat were all badly injured, but they all survived.

Local resident William Dickerson and his nephew were fishing in the area when they heard the helicopter go down. They helped the injured men from the helicopter and to the nearest road, and then went to call for help. US Forest Service law enforcement officer Doug Hamilton and Sabine County Sheriff Tom Maddox were among the first responders. After wading back through the swampy area to the crash site, they realized they could not bring out the bodies and Mier and Krenek without assistance.

Felix Holmes and Marsha Cooper of the US Forest Service brought a bulldozer and several all-terrain vehicles to the site. Holmes knocked down pine trees with his dozer to create a makeshift path for the ATVs.

East Texas was in shock after the accident, losing a “favorite son” in the search for Columbia. Despite the grief, no one voiced regrets or blame. Rather, the community regarded it as a sacrifice to an important undertaking. It proved that the people of East Texas were doing their utmost to help NASA return to flight, no matter the personal cost.

The motto that had been circulating since the early days of the recovery effort now seemed even more poignant. Their mission became our mission took on a much deeper meaning for the citizens of East Texas. They were now inextricably part of the Columbia story—their own blood mixed with the blood of NASA’s astronauts.

The crash caused an immediate stand-down in air searches while the NTSB and the participating agencies investigated the accident and reviewed search procedures. The cause was traced to a faulty fuel control unit. Search procedures were revised to improve safety and eliminate unnecessary risks. Searches would have to be from a higher altitude. There would be no more landing to pick up debris; searchers would call in GPS coordinates for ground teams to make recoveries. Finally, air operations would be concentrated farther west in the debris field, which as less heavily forested than the eastern portion.

Now that they had given the lives of one of their own men to the cause, it was more critical than ever to ensure that the task was worthwhile—that these two men and the crew of Columbia had not died in vain. And indeed, the air searches were very productive, eventually accounting for 65 percent of the shuttle debris that ended up on the grid in the reconstruction hangar. They helped solve the mystery of the accident.

[Please see this excellent article by Christopher Freeze for more technical information about the accident.]