It Bears Repeating

 

I am pleased to report that Jonathan and I submitted the manuscript for Bringing Columbia Home to our editor yesterday.

We’ve been working on this book for nearly two years. In fact, it was two years ago today that we first met, at the funeral of our mutual friend, Norm Carlson.

The past two months felt very much like “Press to MECO” as we went through multiple reviews and revisions to meet our submission date. And just like after a successful Shuttle launch, now I can catch my breath and take the luxury of a little time to pause and reflect on the process that got us to this point and what it means to me.

What was the most significant learning I had in the process of helping Jonathan research and write Bringing Columbia Home? By far, it was how so many American citizens came together so willingly to help us when we needed it most.

Call it what you like. I like ‘the American Spirit.’

We were all hurting from the loss of Columbia. Most of all, the crew families were devastated. No more needs to be said about them.

Those of us in the NASA community were stunned and hurting.

The folks in East Texas were shocked and felt the loss deeply from the very beginning.

The 25,000 people from across America that came together over the course of three months to recover the astronauts and debris came to feel the loss just as much, and as soon as they joined the effort. There was no ‘ramp up’ in emotions.

I’m certain other people around the world felt an emotional connection to the accident as well.

What those of us involved in the recovery and reconstruction shared was something very special. It was the NEED to help. I know the same happens in war, though I have never personally experienced it. It is a need to help your country and comrades. Unique to America? No, but certainly true about us. It is something to be proud of, and to share.

This is precisely why the book will shortly exist.

ALL Americans should know this story of our country’s spirit at its best. They deserve to know it. I believe it’s especially important now when it seems like bickering and divisiveness have become a sort of new norm in our country.

If there’s a message of hope in a story about the aftermath of a terrible national tragedy, it is that Americans are at their very core a compassionate, caring, and committed people who will rise to a challenge and accomplishing incredible things.

Columbia Accident Investigation Board Public Hearings Begin, March 6, 2003

One of NASA Administrator Sean O’Keefe’s first actions after Columbia disintegrated on February 1, 2003 was to activate an independent investigative board. NASA had at least fourteen formal internal task forces and teams designated to respond to the accident and investigate technical issues. However, policy mandated the formation of an independent board after a significant incident and loss of the crew. This board named itself the Columbia Accident Investigation Board, or CAIB (pronounced kabe) soon after its first meeting.

O’Keefe asked Admiral Harold “Hal” Gehman to chair the board. Gehman had recently completed an investigation into the terrorist bombing of the USS Cole. Other members of the Board included leaders from the military, NASA, FAA, research institutions, and other experts.

One of the truisms of a catastrophic accident such as Columbia‘s destruction is that it is almost never due to a single piece of hardware failing. Complex systems like the space shuttle and commercial airplanes are designed and built with all sorts of fail-safe mechanisms and components. For example, if a guidance computer fails, there is a backup (and sometimes several) that can take over. History has shown repeatedly that it usually takes a cascading series of failures to cause the destruction of something like the space shuttle.

Those failures are not always mechanical. They are often compounded by breakdown of  processes and how the organizations and people work within the system.

Therefore, the CAIB was empowered to examine not only the physical causes of the accident, but also to look into any organizational, cultural, procedural, policy, design, or other factors that may have played a role.

So while NASA’s internal teams and task forces collected and analyzed the debris from the accident, looked into the telemetry that was received from Columbia in its final minutes, and ran checks on the history of the ship’s hardware, the CAIB interviewed personnel who were involved in various discussions and decisions prior to the accident.

The CAIB held the first of its public hearings on March 6, 2003. Space Shuttle Program Manager Ron Dittemore was one of the first to testify, and the CAIB questioned him at length about subject such as recent changes to program staffing that might have affected operational safety. Johnson Space Center director Jefferson Howell, like Dittemore, disagreed with any suggestion that NASA underestimated shuttle risks or that safety did not receive the highest possible emphasis. However, former Ames Research Center director Harry McDonald suggested that he believed NASA had moved too far toward considering the shuttles as “operational” vehicles rather than complex research and development spacecraft.

Testimony later in the week focused on issues such as whether the insulating foam on the shuttle’s external tank could absorb water and potentially freeze, making the foam heavier and thus more of a threat to the orbiter. Other witnesses spoke about the decision to fly two shuttle missions after Atlantis was struck by insulation on STS-112 in October 2002.

Although many of the points that were raised were difficult for NASA to hear, Administrator O’Keefe had made it abundantly clear that he wanted full and open cooperation with the investigation. The only chance NASA had of finding and fixing the issues that doomed Columbia was to bring everything out into the spotlight for intense scrutiny.

BoardPortrait

Members of the Columbia Accident Investigation Board. From left to right, seated: Scott Hubbard, Dr. James N. Hallock, Dr. Sally Ride, Board Chairman Admiral (retired) Hal Gehman, Steven Wallace, Dr. John Logsdon, Dr. Sheila Widnall. Standing from left to right: Dr. Douglas Osheroff, Maj. General John Barry, Rear Admiral Stephen Turcotte, Brig. General Duane Deal,  Maj. General Kenneth W. Hess, and Roger Tetrault.

First Advance Review for the Book

Mike and I have been working on our book for nearly two years now. The manuscript goes to our editor at Skyhorse Publishing on March 15. Everything is on track for the book to be released around Christmas this year.

We sent a courtesy copy of the manuscript recently to Sean O’Keefe, who was NASA Administrator at the time of the Columbia accident. Sean was an early and enthusiastic supporter of our project when we kicked it off, and he is very pleased with the results:

Mike and Jonathan have done a brilliant job capturing the depth of emotion and human engagement of what has been covered by others only as a technical investigative treatment. In doing so, they have made the story very personal for the thousands of people who invested themselves in this critical chapter of space exploration history. This is a valuable contribution about a defining moment that demonstrates NASA’s resolve and the selfless generosity of the American spirit.

—Sean O’Keefe, former NASA Administrator

We can’t wait to share this story with you!

The Hidden Cost of Reusability

All space flight providers want to control costs. From the traditional government and commercial enterprises to the new entrants, all want to keep costs as low as reasonably possible. Gone a long time ago—and forever—are the days of unlimited resources to get the job done. Bottom lines matter, and matter a lot.

It has long been argued that one way to keep costs reasonable is to reuse launch hardware. Even when adding the required refurbishment/reflight checks, repairs, component replacements, etc., to the equation it can be shown on paper to save money in the long run. The Shuttle was sold to Congress in part on this philosophy. And being the first reusable spacecraft they could know no different.

Nor could NASA fully know the long-term implications of reusability.

An additional fact is this: Adding astronauts to the equation requires changing ‘reasonably’ to safely. No argument there, right? But what are the implications of this change? Simply put, based on Shuttle experience, it requires lots more of the checks, repairs, component replacements, etc., in an attempt to make the re-flight as technically close to the first as possible. The newer the hardware, the less risk of failure. No argument there either.

These costs are easy to understand and accept. What about the posting’s title? Hidden? I have alluded to it in earlier postings and it’s not just a financial issue. It can cost a lot more than money.

Here it is:

With the desire to refly hardware that has (theoretically) performed well before, a tendency to rely on past performance when approving current flight rationale is given more weight than it should.

That tendency can either be seen overtly, or it can be latent. If obvious, it can be dealt with through open discussion and full debate of the issue at hand. If hidden, it can fester, grow, become a ‘new normal’—or worse, lead to disaster.

Examples in the Shuttle program are numerous. From those we recognized and dealt with—wire chafing, flow liner cracks, hydrogen leaks—to those that dealt us the most severe of consequences—foam shedding, O-ring scorching—we were challenged by the need to re-fly hardware, all with the overarching tenet of doing so safely.

Tragically, we didn’t always balance that correctly.

I sincerely hope today’s and tomorrow’s astronaut launch providers succeed 100% of the time. Recognizing when a ‘new normal’ is kicking in is part of the success criteria.

spacex-falcon-9-crs-9-landing
SpaceX Falcon 9 first stage returns to Earth after the CRS-9 launch. (Photo courtesy SpaceX)

Wildland Fire Crews to the Rescue

Within two weeks of the Columbia accident, an intense search effort by NASA, the US Forest Service, the Texas Forest Service, the FBI, the National Guard, the Texas Department of Public Safety, local law enforcement officers, and hundreds of volunteers located the remains of Columbia‘s crew in Sabine County, Texas.

Meanwhile, in the Persian Gulf, seven thousand miles away from the pine thickets of East Texas, the US was gearing up for the opening of another war front in Operation Iraqi Freedom. The National Guard was pulled within a few days of the last crew member being located.

The immediate urgency of the search operation was significantly reduced. NASA and the EPA had located and cleared many of the hazardous materials that had fallen to the ground from the shuttle—tanks with hypergolic propellants, pyrotechnic charges, and the like.

And yet, tens of thousands of pieces of debris still remained on the ground in a path more than 250 miles long and 10 miles wide, stretching from near Dallas to Ft. Polk, Louisiana. NASA still had no firm evidence about what had caused the accident; there were still many possible failure scenarios on the fault tree.

The spring “green up” would be coming within a month or two. Once that occurred, the underbrush would become impassable to ground search teams. And it was unreasonable to expect the small towns along the debris corridor to endure the disruption and hardship of supporting  a large number of searchers for a prolonged time. Although there were many volunteers who would gladly have continued working in the search, the NASA and FEMA leaders in Lufkin resolved not to put civilians in harm’s way for what was now a debris salvage operation.

So how could such a huge expanse of land be cleared quickly?

Mark Stanford of the Texas Forest Service and Marc Rounsaville of the US Forest Service made a proposal to NASA and FEMA: Use US Forest Service Incident Management Teams (IMTs) supported by wild land fire crews to run the search. Stanford and Rounsaville had actually proposed this more than a week earlier, but idea hadn’t taken hold. Now, it seemed intriguing.

National IMTs come from all over the United States. They are trained to deal with many types of complex “all-hazard” incidents, ranging from wildfires to hurricanes to the response to the World Trade Center attacks. IMTs deploy as completely self-contained units, with their own logistical support, and can be on site within days. They would be supported by hundreds of fire crews, each composed of twenty able-bodied men and women who were already skilled in the techniques of grid searching. They would only need to be trained in what to look for.

Astronauts Dom Gorie and Jerry Ross, who were helping to manage the debris search for NASA, asked how many people could be brought in. Stanford said he could have 1,000 people on site within a matter of days. Gorie and Ross immediately asked, “How about a thousand more?” Stanford said it could be done. “How about another thousand?” It was possible, but might take a few extra days.

Gorie was impressed. He said, “It was miraculous. I had no idea that anything like this could be generated just for this effort.”

Calculations showed that search lines of people stretched five to ten feet apart were likely to find at least 75 percent of all objects six inches square or larger in the debris field. Ground searchers would cover every square foot of an area three miles on either side of the center line of the debris corridor. Air crews in government-contracted helicopters would search another two miles on each side of the ground search corridor. This would theoretically enable NASA to recover an amazing 95 percent of the debris on the ground.

And the operation could be completed by April 15.

NASA and FEMA gave the approval to get the operation underway by the last week of February, 2003. The Texas Forest Service would manage the overall effort. There would be two months of an all-out push to find and recover the rest of Columbia‘s debris.

Ultimately, nearly 22,000 men and women were involved in this ground search.

We will hear much more about this operation in the coming weeks.

nac-arena
The Nacagoches Rodeo Arena becomes a “tent city” of wild land fire crew searchers.

Reconstruction Begins

Just a few days after the Columbia accident, NASA’s management discussed where and how to reconstruct the debris from the ship that was being found in Texas and Louisiana. Telemetry from Columbia cut off more than 30 seconds before the accident, and the ship’s Orbital Experiments (OEX) recorder—akin to an airplane’s black box—might not have made it to the ground. There was a chance that the debris could provide clues about the cause of the accident and how the shuttle broke up.

On February 3, NASA Administrator Sean O’Keefe and his deputy, Bill Readdy, announced that Kennedy Space Center would be the site for reconstructing Columbia. Steve Altemus from the launch team volunteered immediately to set up and lead the day-to-day operations. Mike Leinbach was named to head the overall reconstruction effort, as the “up and out manager” a few days later.

The Reusable Launch Vehicle (RLV) hangar at the southeast end of the Shuttle Landing Facility runway was a perfect location—convenient, secure, and big enough for the task. NASA quickly negotiated a lease for the hangar—which is owned by Space Florida, even though it’s on KSC grounds—and got to work setting it up.

rlv-reconstruction-hangar
The RLV hangar (foreground) at the southeast end of the Shuttle Landing Facility. (NASA photo)

The NTSB and Boeing Air Safety provided guidance in laying out a grid. NASA would “peel open” the orbiter so that its external surfaces were lying face-up on the hangar floor. The dimensions on the floor grid were 10% larger than real life, to give examiners room to walk around between the pieces of debris and examine them from all angles. NASA set up the outline of the orbiter as if the shuttle had been towed into the hangar nose-first.

 

Placing items relative to each other on the grid as they would have been on the orbiter enabled researchers to find patterns that might point to the cause of the accident, as opposed to damage that occurred after the vehicle broke up.

reconstruction-hangar-layout
The floor plan of the RLV Hangar.

Even though EPA and NASA were decontaminating the debris as it was collected in the field, NASA needed to plan for the possibility that some pieces of debris might come back with toxic hypergolic propellant residue. A decontamination area was set up near the receiving area. NASA also eventually used a second hangar—adjacent to the Apollo/Saturn V Visitors Center—to store large components made of “friable” materials that could cause respiratory problems if their particles or fibers were inhaled. This included sections of the shuttle’s payload bay doors and composite-overwrapped pressure vessels.

 

NASA set up a walled-off room at one side of the hangar. Here, astronaut Pam Melroy and her team of engineers and technicians would be responsible for reconstructing Columbia‘s crew module in three dimensions.

Materials for internal systems (power, hydraulics, miscellaneous plumbing, etc.) and the Spacehab experiments were stored in bread racks on the sides of the hangar.

Everything would be photographed, barcoded, and logged into a database when it entered the hangar. GPS coordinates had been taken for every piece recovered in the field. As the reconstruction team positively identified various components, they could use this information to target searchers to areas where critical items might be found.

NASA set up the hangar, established the operating procedures, and staffed the reconstruction positions within 10 days.

The first two truckloads of material arrived from Barksdale Air Force Base on February 14, the same day Mike Leinbach returned to KSC from leading the Rapid Response Team at Barksdale. Trucks brought more debris every other day for the next several weeks, and would continue arriving regularly until the last shipment on May 6.

KSC security special agents escorted every shipment, accompanied by representatives of other NASA centers, who wished to participate in the honor of bringing Columbia home one final time.

The hangar operated with two shifts per day, six days a week, with about 150 personnel per shift, as people began the massive job of identifying and studying tens of thousands of pieces of Columbia‘s debris.

debris-and-tire-03-11-2003-ksc-03pd-0638
By March 11, 2003—less than a month after hangar operations began—nearly 34,000 pieces of debris were in the hangar, totaling 43,200 pounds, or roughly 19% of Columbia‘s dry weight. (NASA photo KSC-03pd-0638)

This is the first of many articles in the coming months about the reconstruction activities.

The Second Week of the Recovery

NASA and the country held memorial services for Columbia‘s crew beginning with a service at Johnson Space Center on February 4. The next day, a service was held at Washington DC’s National Cathedral. On February 7, NASA leaders gathered on the Shuttle Landing Facility runway at Kennedy Space Center. Columbia‘s first pilot, astronaut Robert Crippen, delivered a moving eulogy to Columbia and her crew.

NASA’s press release of February 5 stated that remains of all seven Columbia crew members had been returned to Dover Air Force Base. That was factually correct, but not complete information. What the release did not say was that two of the crew were still missing. Up to 1,600 Forest Service personnel, National Guard troops, employees of the Texas Department of Public Safety, and volunteers combed the woods of Sabine County over the weekend of February 8 and 9 in a desperate search for the crew. Cold rain, sleet, and miserable conditions plagued the searchers.

On February 10, one of the National Guard units located the remains of the sixth of Columbia‘s crew near Toledo Bend Reservoir. Remains of the seventh and final crew member were found the next day. The sense of elation and relief felt by the searchers can’t be adequately described. Local resident Mike Alexander, a volunteer searcher since the day after the accident, broke into tears when he heard the announcement. “I just started crying out there in the woods,” he said. “I couldn’t help it. I thought, ‘We got closure now.'”

Volunteers in Sabine County had provided almost all of the food and shelter for the searchers. It was impossible to determine precisely how much food had been donated or how many thousand meals had been served at Hemphill’s VFW hall. Best estimates were that this relatively small community and a handful of volunteers had prepared and served 30,000 to 50,000 meals in two weeks. The community had donated over $620,000 in services to the recovery—at no cost to the federal or state government.

“It’s just people helping people—that’s what this small town is about,” said Roger Gay, Hemphill VFW’s commander. “Everybody likes to help everybody else, and they don’t expect anything from it. It was an occurrence that happened, and we dealt with it the best way we knew how.”

NASA’s Dave King summed up, “The people of East Texas made you proud to be an American, because they sacrificed and gave everything they had to try to help us. It was unbelievable what they did for us.”

Elsewhere in the 250-mile-long corridor where Columbia‘s wreckage came to Earth, hundreds of United Space Alliance and NASA personnel from Kennedy Space Center, along with representatives of the EPA, were working with local authorities to investigate debris sightings, check the pieces for contamination with hypergolic propellants, and then bring the items back to collection centers. The debris being found ranged from pieces of the wing to items from the crew module.

On February 7, searchers near Palestine, Texas found a videotape cassette, one of many that had at least partially survived the accident. Astronaut Ron Garan collected the tapes and flew them on a T-38 to Washington, DC, where he inspected them at the NTSB’s headquarters. Most were blank or were data tapes. However, one caused him to freeze when he started playing it.

It was the cockpit video of Columbia‘s reentry.

“The hair stood up on the back of my neck,” Garan said, “because we didn’t know how long the video was going to last or what it was going to show.” To his relief, the tape ended several minutes before the first sign of trouble.

NASA publicly released the video on February 28. The tape showed Columbia’s crew happy, being professional, and enjoying the ride. They were passing the camera around, smiling at each other, and remarking on the sight of the glowing plasma surrounding the orbiter. They were obviously unaware that anything was wrong with their ship.