The Apollo Challenger Columbia Lessons Learned Program

One of the most positive outgrowths from the Columbia accident remains the Debris Loan Program. I have sung its praise in earlier posts, and for good reason. Another initiative underway has equal or greater positive potential.

Something more meaningful than studying the debris in order to design safer spacecraft? Yes. What about something to deal with some of the common root causes of America’s manned spaceflight accidents? To deal with them BEFORE they become big problems.

This is NASA’s Apollo Challenger Columbia Lessons Learned Program. (ACCLLP)

Never heard of it? That’s understandable, given its newness. But I hope that as it matures and gains traction within NASA, its value will be realized and its lessons made available even outside NASA organizations. The lessons NASA learned the hard way are widely applicable. Think about the value of open communications vs. stifled debate as just one example.

So what is the ACCLLP?

It’s a fully-funded and staffed NASA HQ initiative to teach all NASA and supporting organizations about the causes—and especially the common causes—of the three fatal accidents. And obviously, as the causes are discussed, the more important topic of their lessons learned are emphasized.

Back to open communications: Challenger and her crew were victims of stifled debate as surely as flawed hardware. Improvements to the hardware proved easier to implement than the organizational and cultural corrections based on that tough lesson learned. NASA improved its communications and management practices, but they eroded over the 17 years between Challenger and Columbia. We fell back into some bad habits: over-confidence, less hunger for fully understanding the potential for minor issues becoming major ones, and to a degree stifled debate. Columbia was victim to those practices just as surely as Challenger.

How does ACCLLP work and how can it help?

Mike Ciannilli (who I fondly call “Chachi”) is a former member of the shuttle launch team and one of the people who conducted aerial searches for Columbia’s debris. He created, developed, and now manages the ACCLLP for NASA and is 100% dedicated to its success. Mike has developed lessons learned programs and “teaches” them to the workforce at KSC and other centers. He routinely conducts tours of the Columbia Preservation room in the Vehicle Assembly Building each week. These reach a wide array of folks across all NASA Centers and missions, including civil servants, contractors along with the brand new commercial crew workforce—even NASA Space Act partners such as NASCAR. Mike ensures that these tours powerfully emphasize the lessons of the past and present, and their direct impact to the future.

Mike Ciannilli with his beloved Columbia (photo courtesy Mike Ciannilli)

Mike is also creating the Agency’s largest lessons learned library with teaching materials from NASA and other organizations. It is the agency’s best advocate to not ignore or forget the past, but rather to learn from it.

A traveling road show took some key pieces of Columbia’s debris to the NASA Centers over a decade ago. Word has it that Chachi is creating a brand new and really cool concept to bring Columbia herself to the masses as we speak. Fireside chats featuring ‘graybeards’ talking about Apollo 1, Challenger, Columbia have been held to full houses and garnering outstanding reviews. And other concepts to discuss lessons learned are being developed to help spread the word, including videos that would be made widely available. Mike is overloaded with requests, and loves it!

So if you’re currently involved in manned spaceflight (or unmanned spaceflight for that matter) and want to know more about this extremely positive outcome of NASA’s three fatal accidents, look up Mike. He’s based at KSC and would love to have you and your organization exposed to his work. I promise you’ll not only be impressed with his enthusiasm, but the lessons learned will be applicable and valuable for your people. There’s no doubt.

The Columbia Debris Loan Program

It’s been a while since I posted a new blog. No excuses, just apologies. Hurricane Irma certainly did a number on Florida, and all us residents are still dealing with the aftermath.

On a recent visit to the Columbia Preservation Room in the VAB (to show the debris to some ULA employees), Mike Ciannilli updated us on the continuing success of the debris loan program with some truly amazing statistics. I want to share his success with you here, but first please allow me some personal reflections.

From left, Mike Leinbach, KSC Center Director Jim Kennedy, and Scott Thurston unveil the plaque commemorating the Columbia PReservation Office on January 29, 2004. (NASA photo)

When the reconstruction team proposed studying Columbia vs burying it (à la Challenger), we could not possibly have dreamed how successful and inspiring the outcome would be. With Administrator O’Keefe’s full support and encouragement, Scott Thurston crafted an in-depth concept for the program, wrote and released a Request for Information to industry to judge interest—and the rest, as they say, is history. THANKS, Scott.

Astronaut Pam Melroy, who led the crew module reconstruction effort, with Amy Mangiacapra at the dedication of the Columbia Preservation Office on January 29, 2004. (NASA photo)

Once we knew definitively studying the debris was a good idea, the Columbia Preservation Office was officially dedicated (on the first anniversary of the accident) in its new home on the 16th floor of A Tower of the VAB. United Space Alliance’s Amy Mangiacapra was the first curator and held the position for 10 years. Alone, she cared for the debris, collected pieces requested for study, managed the room, escorted visitors, dealt with ceiling leaks, swept the floors, and did everything/anything required. She was Columbia’s caretaker, and she did it in a manner beyond what she (and we) considered a job. Columbia was her “work child”. Amazing. THANKS, Amy.

NASA created a full-time position for the office in 2014. Mike Ciannilli became the first NASA curator, and by any measure, is the perfect person to take care of Columbia following Amy. (As with the astronauts, we like to give each other nicknames. Because Mike bears a resemblance to “Chachi” Happy Days, we lovingly refer to him as “Chachi” or “Chach.”)

Mike Ciannilli at the entrance to the Columbia Preservation Office in the VAB. (Jonathan Ward photo)

Not only does Chach administer the debris loan program, but under his leadership and that of KSC Center Director Bob Cabana, the Forever Remembered memorial was opened in the Atlantis building at the KSC Visitor Complex. Columbia’s forward window frames are in a display case for everyone to see and contemplate. They also got the Challenger families to embrace the concept, with the large sidewall piece of Challenger now occupying an adjacent case. Moreover, with the vision of the greater good that can come from studying past failures, in 2016 Mike created the “Apollo Challenger Columbia Lessons Learned Program,” bringing those hard-taught lessons to the workforce and others in hopes they won’t be repeated ever again. GREAT JOB, Chachi.

Finally, the statistics Chach and all of us are so proud to share: To date some 260 pieces of Columbia have been lent to academia and industry for study. As I write this, 12 pieces weighing over 1000 pounds are being studied to advance the understanding in how different materials and structures behave when subjected to the extreme conditions of hypersonic re-entry. And of particular pride, there are three individuals who used Columbia in their PhD dissertations in achieving the highest scholastic degrees in engineering and material science. That’s what I call success!!!!!!

In a very real sense Columbia continues her mission….

KSC “Launch Director” Tours

About 18 months ago I was asked to support the Kennedy Space Center Visitor Complex in their overarching goal for their guests of being educated while being entertained. I was honored to be considered for it, given the other major ‘attractions’ that further that goal. Heroes and Legends—the re-envisioned Astronaut Hall of Fame—recently opened in the original Debus Center. A new Mars exhibit will soon begin construction. Honoring the fallen astronauts in Forever Remembered and the tribute to the Apollo 1 crew are moving reminders of the risks associated with spaceflight. By far, however, the KSCVC celebrates the successes and contributions to mankind that the space programs have provided for over 50 years. Attendance promises a full day of activities and memories unlike the typical theme parks in Orlando.

My small part is called, cleverly, the Launch Director Tour! A few times each month, I take a group on a personal tour of the Atlantis attraction, the Launch Control Center, launch pads, and conclude at the Apollo Saturn V facility. In the four-hour tour we discuss not only those specifics but get into anything the group wishes to know more about. As I tell them at the outset, “If you leave here today and wish you had asked me a question but didn’t, bad on you! My job is to make your time here as full as possible. Answering your questions is an integral part of that.” And the groups aren’t shy!

At my Launch Director console in Firing Room 4. (Photo by Jonathan Ward)

Since they are truly ‘avid space fans’ the questions asked are just terrific. Technical specifics of the Shuttle, the early manned spaceflight programs and the current and future ones, most memorable moments, most difficult launch, and the questions from kids are special! We also always have foreign guests on the tours and their perspectives on America’s programs offer unique and memorable interactions.

I could write a book in itself on the questions I get on my tours—they are that good. My personal favorites generally fall into three categories. 1: What is the future of manned spaceflight? 2: How I made the final launch decision. 3: My favorite launch.

Discussing the future always comes up. As do the politics behind decisions. This discussion can always take unexpected turns! But briefly, I express my view that Mars should not be the next goal of our manned efforts, but establishing a permanent base on the Moon. Why? We need to learn to live on another body before we take off on the exponentially more difficult trip to Mars. The Moon is the next logical step. Not as sexy as Mars, but vastly more logical in our progression off Earth.

I made the final launch decision by thinking about my seven friends on the rocket and asking myself if I’m ready to commit them to the most risky thing they have ever done. The astronauts are real people with real lives, real families, real children, spouses, parents. How can I give a “go” without considering their families? This was always part of my final decision, but became even more paramount, if that’s possible, following Columbia.

My favorite launch is answered two ways: the most difficult and the most unexpected. The most difficult was STS-107, Columbia’s final mission. Why? The book has more detail, but we had a security scare just before liftoff that had me holding onto my console to steady my hands, literally. Just sixteen months after the attacks of 9/11, and with an Israeli astronaut on board, Columbia was recognized as a prime terrorist target. Liftoff was fine, but foam damage during ascent doomed the mission and crew. The launch with the most unexpected event was STS-105 and the need to launch earlier than planned due to impending bad weather. I addressed this in an earlier posting. Check it out.

My all-time favorite question, however, came from a kid from England, maybe ten years old. She asked, “How does an orbit work?” WOW! From a kid came a question demonstrating great thought, curiosity, and desire to learn. The goal of the whole tour concept had come true in spades. Fortunately, luckily, I could work my way through it, but first I asked the group how many people knew the answer. Of the 35 people, 2 knew. I asked them to help me answer. With my pen as a prop we demonstrated the balance between gravity and speed and the little girl’s eyes lit up. She got it. That’s why we do the tours. It was perfect.

I invite all of you to join us if you ever find yourself at the KSC Visitor’s Complex. Advance reservations for the tour are recommended—and can be made at this link. The tour frequently sells out weeks in advance. If not on my tour, the overall KSCVC experience promises not to disappoint in any way.

Required Changes

A lot has been written about the recovery from the Columbia accident in terms of changes we needed to make to get back to flying the Shuttle again. In general, the changes fell into two categories. One bucket contained changes to hardware, the other were changes to management practices.

In the early summer of 2003, we didn’t know how much time we’d eventually have to make these changes—just that we’d take whatever time was necessary to get them done, and with the confidence we did them right. But based on the recovery from the Challenger accident of 1986, we figured we wouldn’t be flying again for a couple of years. Could be longer; could be a little quicker. But the charge to all of us was to get the work done correctly, first and foremost. Sort of like resolving a problem in the final throes of launch countdown – solve the problem first, then look up at the clock and see if you have any time left in which to launch.

That’s not to say we were lackadaisical about it. Hardly. We were well aware of the need to get flying again to the ISS. But once again, it was ‘schedule awareness’ vs ‘schedule pressure’. There was a difference from the time critical launch environment of course where technical problems were solved based solely on data, and bad decisions couldn’t be recalled. In the recovery period, lengthy, philosophical debates were fairly common. But decisions needed to be made and progress in the improvements needed to be real.

The foam loss problem on the external tank needed to be fixed. Adding the capabilities to inspect the Orbiter’s tiles and effect some level of repair prior to re-entry was also necessary. These were obviously the top two flight hardware upgrades undertaken. But each Project (Orbiter, ET, SRB, Ground Processing, etc.) was asked to essentially recertify their existing system as flight-worthy, or suggest upgrades aimed at improving safety margins. These suggestions would be debated at the Program-level change boards and either accepted for implementation (and funded) or not.

Changes weren’t too widespread for us at KSC and the Ground Processing directorate. For the most part, our work practices on the flight hardware were mature and adequate. Extra care was to be taken when working on the External Tank’s foam to avoid damage, but nothing too onerous.

One significant finding in the accident review that we were responsible for correcting was the inadequate ascent imagery. As you may recall, on Columbia‘s final launch one ground tracking camera was inoperable, another was out of focus, and the just sheer number of assets documenting the critical portion of ascent couldn’t guarantee the full suite of images necessary to help resolve issues. As a result, we undertook a complete review of the ‘imagery system’ composed of tracking video cameras, still photography, high-speed engineering film assets, and the Operational Television System (pad cameras). We needed to be sure we had enough visual documentation to address issues, and have confidence on launch day the assets were working and could ‘see’ the vehicle. The Columbia Accident Investigation Board (CAIB) even recommended we have Launch Commit Criteria (LCC) for the system. More on that in a moment.

In addition to improving the visible launch documentation we needed some sort of long-range tracking system that could detect issues long after ground-based cameras effectively lost sight of the vehicle. Later – the C-band radar system. Likewise, on-orbit imagery needed to be understood and policies firmed up to enlist help from the intelligence community if needed.

This 50-ft. C-band radar dish was installed near Haulover Canal north of the KSC launch complex, as one of three radar dishes used in the new Debris Radar System. The other two were on ships. (NASA photo)

For the sake of brevity, the final ground-based system we installed was one of guaranteeing adequate views at least through SRB separation, from three independent positions, and from both north of the pad and south of the pad. We needed close-in views, mid-length (2-5 miles), and longer-range views from 10 miles or beyond. No distance requirement was set, just that we had these three ‘zones’ covered. Obviously, siting the individual assets would be case-dependent. At least two cameras at each location added to the certainty of coverage. The status of each would be reported to the responsible system engineer on the launch team and relayed to us. They would be committed for launch during the hold at T-9 minutes.

What about the CAIB launch commit criteria requirement? What about clouds obstructing one or more views? What about night launches? Good questions.

The CAIB did not specify what type LCC they wanted, although in informal talks they were going after specific camera views and operability. Given the uncertainty of guaranteeing views, I opted to enact an LCC based solely on the system operating properly. The issue of adequate views (cloud coverage, one or more specific cameras being down, etc.) was left to judgment on launch day. That decision would be made jointly by me (as Launch Director) and the Mission Management Team chairperson. The CAIB accepted the idea, so we pressed on with buying and installing an elaborate collection of video and still cameras located north and south of the pad. And we installed a control system for the cameras close to or at the pad. It was that control system that had the LCC. On launch day, the pre-launch MMT chair and I would get information on the views we would get during ascent and would decide if we’d launch with anything less than the full complement.

We had a requirement to launch during the light of day for the Return to Flight mission. That mandate remained in place until we had confidence the foam loss issue was resolved, AND that the radar system could detect debris issues regardless of daylight. We relaxed the lighted-launch requirement starting with STS-116 in December 2006, the first night launch of a Shuttle since the Columbia accident.

The system proved to be a great addition to the safety for the astronauts and the vehicle. Never again would the vehicle be hidden from view during ascent. We had enough cameras to make up for one or two not working as designed and had all angles covered. Ground-based imagery never caused a scrub and always provided clear views of the vehicle – and plenty of them.

Tropical Storm Ernesto and the Half Rollback

Every now and then, a little creativity goes a long way…….

As noted in my previous post, we were charged with protecting the Kennedy Space Center workforce and the hardware. But of course, we were in the business to launch the Shuttle, not stay on the ground indefinitely while maximizing the safety statistics. The basic tenet for all of us was “Get work done, safely.” Apply that to all aspects of Shuttle processing and launch. Meeting the manifest was important. Launching was important. Doing it safely was paramount.

Decisions that could affect the manifest required me to consult with the Shuttle Program managers at Johnson Space Center (JSC)—and for good reason. The “Program” was “theirs”, not “mine”, or KSC’s. So for major decisions like rolling back for storms two people needed to agree: the Launch Director (me, at KSC) and the Operations Manager (the JSC manager stationed at KSC). We conferred and jointly made those major manifest-impacting decisions.

Flashback to late August, 2006. Atlantis was on the pad preparing for STS-115’s assembly mission to the ISS. Atlantis was originally scheduled to launch on August 27. Our ground team made an unprecedented replacement of some parts on the shuttle with a week to go until launch, and we maintained the schedule. Then on August 25, one of the most powerful lightning bolts ever recorded at Kennedy Space Center hit the lightning mast at the top of the launch pad’s Fixed Service Structure. We needed at least 24 hours to assess the damage from that strike, moving the launch date to August 29.

That was all trouble enough. But meanwhile, a minor tropical weather disturbance had been forming in the Caribbean. Unimpressive—but like all such systems, we would monitor it in case the unexpected happened and it became a threat to us.

True to form, after days of watching the storm, now named Ernesto, its track and forward speed appeared likely to affect Florida’s Space Coast while Atlantis was still preparing for launch.

Decisions were coming, no doubt. We began the initial work to roll the shuttle back to the Vehicle Assembly Building as a precaution. This work could easily be reversed and still hold the launch date if the storm took an unexpected turn away from us. Discussions with the 45^th Space Wing meteorologists were now being held every six hours to coincide with the official forecasts from the National Hurricane Center.

As Ernesto crossed western Cuba and headed into the Florida Straits, it grew to hurricane strength. The frequency of our calls went up to the rare every three hours, and then almost continuously. Ernesto was on a track to enter extreme south Florida and head up the eastern side of the peninsula, essentially right at us.

Meanwhile, Atlantis had a launch date that was sandwiched in between several other arriving and departing vehicles at the International Space Station, other Eastern Test Range operations, and other constraints that were beyond our control. If we didn’t launch before about the 12^th of September (as I recall), we would have to stand down for another two weeks or so. Schedule awareness (not schedule pressure) was real.

The decision was coming. Stay at the pad and risk exceeding our wind limits and possible Shuttle damage? Or roll back to the safety of the VAB and miss the near term launch opportunity?

After numerous tense calls with the Launch Weather Officer, LeRoy Cain (the JSC Ops Manager) and I made the decision to play it safe and roll back. It was the morning of the day before the storm’s predicted arrival, approximately 40-44 hours hence. Rolling early enough to beat the winds was the game we needed to play.

But what if the storm changed course and became less of a threat? That became a real possibility after the wheels were already in motion.

As the ground operations team was rolling Atlantis off the pad for its eight-hour trip back to the VAB, the question came to me: Could we reverse the roll and return to the pad IF the storm really did weaken and veer off course, permitting us to stay at the pad?

The Crawler-Transporter (CT), built for Apollo, went in only one direction: forward. Forward toward the pad, and forward toward the VAB. What? The CT had two control stations—one on the east face, one on the west face. It was from either of these “cockpits” that the crew drove the vehicle. There was no way to turn that mammoth vehicle around on the crawlerway. To go the other way, you’d just stop, get out of the western control cab en route to the VAB, get into the eastern cab, and then start driving again, this time toward the pad.

I asked the Support Test Manager if his guys could do this. He went pale. Never before even contemplated, no procedures allowing it, the time required, etc., etc. all indicated a negative response was likely. But Bobby Briggs, being the best STM at KSC, said he’d “look into it.” His eventual answer was that if we decided before the CT was halfway to the VAB, his guys could do it.

Perfect answer.

Armed with that, I went to LeRoy Cain to see what he thought of the idea. He liked it immediately. If the storm veered away in the next four hours or so, we could stop and return to the pad.

For the sake of brevity, here’s the punchline. We did stop the roll and went back to the pad following the “final” call with the LWO, Kathy Winters. The storm would start to weaken coming up on its overland track, AND the track had it going a bit farther to our east. Atlantis would be on the best side of the storm.

As all of us watched the storm progress up the state from the safety of our homes. The favorable track was verified. It passed to the east of the pad by approximately 40 miles, as I recall. Winds remained within limits and no damage resulted.

We launched successfully Atlantis on September 9, 2006. Astronaut Brent Jett commanded the mission. He had been an integral part of the crew recovery effort after the Columbia accident.

As I look back on this achievement, I can’t help but think about this being just one example of team creativity and their can-do, will-do approach to all operational challenges. I can’t put into words how proud of them I was, and am to this day.

This ‘partial rollback’ was needed to do two things – protect the vehicle, and, because of the way things turned out, preserve a launch opportunity. Get work done, safely.

Damn, it was fun!

Hurricane Season and Rollbacks

June 1 thru November 30. The Atlantic Hurricane Season. As June 1 passed I was reminded how much ‘fun’ it was to experience.

From the earliest days of our manned spaceflight programs, the Launch Director was responsible for two things that outweighed all others, including launch itself. These were to protect the safety of the workforce, and to protect the flight and ground hardware. Obviously, these are everyone’s responsibility, and everyone took them very seriously. But when major processing decisions were required that had significant safety and/or processing implications, the LD made the final call.

Some of those decisions are well-known, the final decision to launch being the most obvious. Others were also the responsibility of the LD but not as visible. The decision to roll the vehicle out to the launch pad, establishing and enforcing the employee work time rules, and approving personnel exposure to launch pad hazards after external tank (ET) fueling are a few. The decision to roll the vehicle back to the Vehicle Assembly Building (VAB) from the launch pad was also given to the LD (along with a Shuttle Program rep that I’ll explain later).

What could drive us to roll back? Remember the woodpeckers and the damage they did to the ET that couldn’t be fully repaired at the pad? (That was STS-70, in June 1995.) How about the hydrogen leaks in the 1990s?

There was another event that could force us to bring the stack back to the VAB for safekeeping – threatening hurricanes.

If you recall, the vehicle spent about a month at the pad in preparation for launch. Unlike expendable rockets that can spend as little as one day at the pad, the Shuttle needed quite a bit more preparation time before launch. Payload installation, ordnance installations, hypergolic fueling, TCDT—those are just four of the numerous pre-flight jobs that had to be done at the pad. They were required for every mission regardless of the calendar.

Enter Hurricane Season. The Space Shuttle would spend that month of prep time on its seaside launch pad, less than ½ mile from the usually tranquil Atlantic Ocean. On occasion, that tranquility would be broken by the effects from tropical storms and hurricanes. If the Shuttle’s presence on the pad and tropical weather coincided, tough decisions would be required.

Protect the hardware. Protect the people.

The Shuttle program benefitted from the weather forecasts and advice from some of the best meteorologists in the world. The 45^th Space Wing of the US Air Force provided weather support for us. Every day—not just during hurricane season, but every day—I would hold a 10-minute call with them for the daily forecast. Numerous processing groups would tie in to benefit from the information and how it might affect their work plans that day. If tropical systems began to form, the intensity on the calls would increase. If there were also a Shuttle at the pad, it would take on an additional air of importance and urgency. Daily calls would increase to twice per day, then every six hours, or even more as the threat got closer.

Ultimately, if the storm track, intensity, and speed combined to actually threaten KSC in the near future we would need to roll the Shuttle back to the VAB to ride out the storm. We could hunker down at the pad if the winds didn’t exceed 60 knots. But if the forecast had higher winds, we needed to get to the safety of the VAB, and this needed to be done early enough that allowed the workers time to get home to deal with their own, final storm preparations.

A lot went into those discussions, but for this entry it can be summarized as balancing the desire to stay at the pad and launch on time vs taking the protection option and delaying launch at least two weeks. But remember: the top priority was safety for the people and hardware. Launch schedules were secondary. But the balance needed to be struck, and it was the responsibility of the Launch Director and the Shuttle Program Manager (delegated to his KSC rep) to do it.

Why would it delay launch at least two weeks if we rolled back to the VAB? The three sets of required tasks, when combined, needed about two weeks to complete—rollback preparations including the roll itself, stay time in the VAB as the storm passed by, and then roll back out and re-perform pad operations undone in Step 1.

To get ready to roll back was at least two full days of activity—usually three—and was done with the storm bearing down on us and our families and homes. How long it took to prepare for rollback depended how close to launch we were when the decision to roll was made. The closer to launch meant more work had been performed that needed to be undone. The Payload Bay Doors needed to be closed after securing the payload. The aft compartment of the Orbiter needed to be closed with special doors, the side hatch closed, etc. Add to these relatively obvious preps things like hypergolic and ordnance systems securing, Pad-to-MLP disconnections (power, comm, gases, data lines, etc.), and you get the idea. The “final” disconnection was this contraption called the ‘9099 interface’ – a large bank of data and power lines on the side of the MLP. Once disconnected, all the work done to verify those systems were launch-ready became history—work that would have to be completely re-done once the Shuttle came back to the launch pad.

We had to roll the crawler out to the launch pad. The rollback itself was typically eight hours or so followed by rudimentary connections of the MLP to the VAB shore systems. Then the final workers could go home and shutter their houses.

Stay time on the VAB was storm-dependent of course, but let’s say three to four days until the “all clear” was declared to reopen KSC. If storm damage existed that would prevent normal work, add that repair time.

The workforce returned to work and got ready to roll back out to the pad. Two days minimum to get ready. Roll out was the same eight hours, and usually at night in the summer to avoid the afternoon thunderstorms. Once at the pad, you have to reconnect everything (reconnecting each interface means going through the complete set of checks to ensure that the connectors are properly mated again) and get back into the same posture as before the decision was made to ride out the storm in the VAB. Add back those three days or so. Then pick a new launch day, verifying that the new launch date doesn’t impact other planned launches at the Cape, other vehicles arriving at or departing the International Space Station, and so on.

All told, the decision to ride out a storm in the VAB meant an impact to the launch schedule of about two weeks.

No magic, just a lot of (necessary) work to protect the flight hardware.

Next week: Tropical Storm Ernesto and the “Half-Rollback”

Returning to a New Normal

As the work in the reconstruction hangar wound down and people gradually returned to their pre-accident jobs, we found ourselves being re-integrated back into a sort of ‘new normal’.

The atmosphere was different, the work itself was different, and the Shuttle was likely on borrowed time. Combine this new normal with the still-present emotional response to the Columbia accident, and you get a workforce with more questions than we could answer, more concern for their futures than confidence—people more in need of direction than ever.

Those of us in leadership and management positions had lots to do dealing with the ongoing CAIB investigation. We were concerned about what it was going to take to get us ready to fly again, debating changes to the External Tank, Orbiter, and other systems. But by far, the most important thing we had to do was to lay out the future for the workforce. The difficulty was that the future was anything but clear for months to come.

Many months.

We needed to stay together as a team despite having no firm game plan. And while everyone understood the uncertainty, it was still an extremely unusual feeling. It would clear up after a couple more months. We would fly again to fulfill international agreements and finish the International Space Station (ISS). But when would we fly again? Would layoffs be coming in the interim? And then once we got back in business, how long would the Shuttle continue in operation? We had originally envisioned flying until 2020, but that was likely to be cut short once ISS assembly was completed.

Open and honest communication throughout all organizations and at all levels became even more important than usual. While we were short on answers, we acknowledged it—and the folks appreciated the candor.

Personally, I thought it was very important to begin to look forward as soon as practical. Not as soon as possible, but as soon as it made sense to do so. In May, 2003 I asked a few close team members what they thought of getting back into launch countdown simulations soon. The responses were split about 50-50. I really wanted to do it to accomplish two main objectives. First, we needed to maintain our proficiency for the inevitable return to flight. Secondly, it would demonstrate to the launch team and to the rest of the processing team that we really were going to fly again. People knew when the team went into training for the day. It was obvious.

So I asked the simulation team to begin to develop a series of training sessions to begin as soon as they could. And on June 1, exactly 4 months after the accident, the Shuttle Launch Team was back together, doing what we did best.

The feeling in the Firing Room that day was unusual to be sure. It was a mix of somber and joy. Reflection and anticipation. But it felt right, too. The “rust” was virtually non-existent, and the team performed exceptionally well.

Firing Room 4 launch console, with an open countdown procedure manual from the STS-135 mission. (Photo by Jonathan Ward)

It turned out to be exactly the right thing to do and at the right time. We held sims approximately every six weeks thereafter.

 

As the return to flight plan firmed up, numerous other training sessions were held—Mission Management Team sims, NASA HQ contingency sims, launch sims, landing sims, etc. Everyone got to participate, and rightfully so—because we were going to fly the Shuttle again.

Sometime.

The Rescue Scenario

Shortly after the accident, during the third week in February 2003, a few of us contemplated if a rescue mission of Columbia’s crew could have been conducted. If it could, what were the chances of success?

Under the guidance of Shuttle Program managers we were asked to quietly study it. We were to conduct our studies in part to satisfy our own curiosity and in part knowing the Columbia Accident Investigation Board (CAIB) would no doubt ask us one day. The Flight Directors at Johnson Space Center (JSC) would do the on-orbit assessment, and I would do the Kennedy Space Center (KSC) operations assessment. The two would combine to answer the unfriendly—but necessary—question.

My part concluded that from a pure timing perspective, a rescue was theoretically possible. The result from JSC was the same: theoretically possible. But both required unrealistic assumptions and actions that were not consistent with the mission being flown, or usual program priorities or objectives.

Rescue would have involved having us launch Atlantis—next in line to fly—as soon as possible, rendezvous with Columbia, transfer the astronauts via some sort of tether to Atlantis, and come home. The crew of seven from Columbia would be aboard Atlantis with her rescue crew of four. Four of the crew members would have to ride home strapped to the deck; there were only seven seats on the orbiter. Columbia herself would then be guided to a ditching in the ocean.

At the time of the accident, Atlantis was almost ready to roll out of the Orbiter Processing Facility to the VAB. A full-court press to expedite that and get to the launch pad would be required. The Pad “flow” would be truncated to only those tasks required, the rest omitted to save time. Things like the Terminal Countdown Demonstration Test and cryogenic loading simulation would be eliminated. Other required tasks would be done three shifts per day, seven days per week. Meanwhile, the rescue scenario flight plan would be developed at JSC.

Assuming no significant glitches, launch could have been as early as February 11. This also assumed no significant processing or launch delays occurred, including weather. That also assumed that Atlantis would not have her remote manipulator arm installed, which was almost certainly needed for a rescue mission. Installing the arm would have pushed the earliest launch date to February 13.

If everything went according to plan—and that was a BIG if—the rescue would have happened two days before Columbia‘s consumables ran out. Columbia would have been in orbit for almost a full month by then, two weeks longer than any previous Shuttle mission.

The key to the entire study was that consumables on board Columbia needed to preserved as much as possible, extending Columbia’s time on orbit awaiting Atlantis’ arrival. Food, water, etc. all needed to be stretched to the max. The limiting commodity however were the lithium hydroxide (LiOH) canisters needed to scrub carbon dioxide from the cabin air. Not food, not water, not power, but the ability to provide breathable air for Columbia’s crew.

The assumption made for the study was that we needed to put the crew on alert for extending LiOH no later than Day 4 of the mission. The crew would have had to go into a very low activity mode to keep their respiration as low as possible. This would have had the effect of terminating the mission’s objectives, effectively ending the reason for the mission. To do this would have been one of the unrealistic moves required. AND, to even get to this posture would have required either proof that the Orbiter was fatally damaged by that day, or assuming so. That was another unrealistic assumption, since the request for additional imagery didn’t occur until Day 6 of the mission, by which time it would already have been too late to conserve the consumables.

But when the two studies were combined, we saw that it would have been technically possible to rescue the crew. That’s the cold, data-driven answer. The truth is that the assumptions I mentioned above, and a few others, would have required extraordinary efforts in both ground and mission operations AND management decision making while we were lacking definitive damage information. All this would have been far outside the normal Shuttle practices at the time.

It should also be noted that the decision to actually launch the rescue mission would have been an extraordinary thing in and of itself. Would we commit a crew of four on Atlantis to rescue Columbia’s, crew possibly facing the same damaging foam loss during its launch? A tough decision to say the least, bigger than NASA alone could make. I believe the President would have had a role in that decision.

But it never got to the point that we’d find out.

No rescue mission was ever contemplated during Columbia’s time on orbit, let alone one early enough to give it a fighting chance of success. We just didn’t have the evidence to support making such a decision, and there was no realistic way in which we could have had that evidence by the time that decision needed to be made.

The CAIB asked us about the scenario in early May 2003. Admiral Gehman, a superior leader, intentionally waited to ask the question until some of the raw emotions had time to subside a little.

When we saw the analyses, there was no grumbling, but there was grief. We couldn’t save the ship. Columbia was doomed, no matter what. Maybe we could have saved the crew. But there were so many what-ifs and assumptions, so many things that had to go completely differently from the very first hours of the mission. Would it have been successful? I don’t know. But we never even had the chance to try.

As much as it hurt people to think about the remote possibility of saving Columbia’s crew, the study helped prompt discussions on how to save a future crew of a damaged shuttle. The studies led to the safe-haven scenario, in which damaged Orbiters could dock at the International Space Station to enable the crews to wait there for a later rescue mission.

KSC and JSC used the Columbia rescue scenario to design a one-time rescue mission that could back up the final Hubble servicing mission. After the successful completion of STS-121 in July 2006, proving that we’d finally solved the foam-shedding problem, NASA Administrator Mike Griffin formally approved the Hubble servicing mission.

On May 11, 2009 Atlantis was poised for launch to the Hubble from Pad 39A at Kennedy. Standing on Pad 39B two miles to the north was Endeavour, ready to go into orbit if there were any problems with Atlantis. For the first and only time, NASA had two shuttles in launch countdown simultaneously. We were ready to launch Endeavour one day after Atlantis if necessary. Tremendous dedication and work went into getting us to this dual launch posture. Fortunately—like many other things in the space business—this contingency capability was assured but never needed.

Atlantis’s flight went flawlessly, so the rescue mission never flew. Atlantis’ crew successfully prolonged Hubble’s life and upgraded its instrument package.

In a roundabout way, Columbia had once again contributed to the advancement of scientific discovery.

On Pad A (foreground), Atlantis awaits launch for the STS-125 Hubble servicing mission, while Endeavour (STS-400) sits on Pad B for a possible rescue mission. (NASA/Troy Cryder)

The Day We Launched Early

This posting doesn’t relate to the Columbia accident, but it addresses a question I get on my monthly KSC tours offered through the Kennedy Space Center Visitor’s Center.

Why were the launch windows for flights to the International Space Station 10 minutes long, while others—especially early in the program—were 2.5 hours long? The answer is Rendezvous.

Without getting into a mini-course on orbital mechanics, for ISS missions we had to get to a specific point in orbit while the station was there. That’s pretty clear. The trick is that the Shuttle had to do it with the available fuels on board—those in the Solid Rocket Boosters, the Shuttle Main Engines, and the Orbital Maneuvering System and thrusters. Simplistically, that total is called the ‘performance’ of the entire system. The launch window was in large part defined by it. It said we had to launch at a time that when after the Shuttle reached orbit it was close enough to ‘catch’ the ISS with the least amount of maneuvering fuels necessary. If too far away, more fuel would have been needed than was available. Rendezvous would have been impossible. Mission failure. So that permissible distance in orbit defined exactly when launch had to occur. And the least fuel would be used if we launched at that precise time, called the ‘preferred time’. The available maneuvering distance once in orbit said rendezvous could still happen if we launched about 5 minutes before or 5 minutes after the preferred time, again, all based on fuel usage. Thus the 10 minute launch window.

In contrast, for missions not requiring rendezvous, like deploying a satellite or STS-107’s Spacehab mission, just getting to the proper orbit was the main objective. It didn’t matter quite as much when—just that the Shuttle got there. The length of the window was defined by other parameters. Usually the most constraining one was the length of time the astronauts could spend on their backs strapped in the Orbiter awaiting launch. This was 5 hours 15 minutes. Given the time the first astronaut (the Commander) got into the shuttle before launch was usually 2 hours 45 minutes before the scheduled T-0, it resulted in a 2.5 hour launch window. There you go!

But what about ‘launching early’? How can that be?

Flash back to August 10, 2001, Discovery’s launch day for STS-105. The mission was to take the Leonardo Multi Purpose Logistics Module (MPLM) to the International Space Station. Our launch attempt on the previous day was scrubbed due to weather. Today, all was going well in the Firing Room for launch, but a mid-summer storm was pounding Lake Okeechobee, approximately 110 miles south of the launch pad, and the storm was moving north. Heavy rain and lightning were heading at us with no expectation to dissipate. Time of arrival in our area was calculated to be about T-0. Weather launch commit criteria were going to be violated—no way around it. If we scrubbed today, we would have a several-day turnaround before the next launch attempt, as we would have to top off the cryogenics aboard the Shuttle. The only hope was to launch as soon as possible and beat the storm’s arrival.

By rule and practice, how the available launch window was used was at the Launch Director’s discretion (within reason of course!). I called the Flight Director in Houston and discussed my idea to launch at the beginning of the 10-minute window rather than in the middle as we had planned. He liked the idea, so we accelerated countdown beginning about 2 hours prior to liftoff, to save the 5 minutes.

All worked out well. We launched at the opening of the window, 5 minutes early, about 20 minutes before the lightning sensors went out of limits. We beat the storm!

Launching early worked. It was never done before or after. STS-105 holds the distinction as the only manned mission ever to ‘launch early’.

Whew! Discovery beats the storm, as STS-105 becomes the only manned mission to launch ahead of the announced launch time. (NASA photo)

What to Do with the Debris?

Fourteen years ago, in early April 2003, we were about 2/3 of the way through recovering Columbia’s debris from Texas, although we didn’t know it at the time. But the number of debris trucks arriving at the reconstruction hangar at Kennedy Space Center had begun to tail off in the preceding weeks, so we knew at some point they’d stop altogether.

Two initiatives were being worked at that time. First, what to do with the debris, and secondly, how would debris found after operations ended in Texas find its way to us? What were the people finding items after the recovery operations ended to do with the material they found?

I’ll briefly address both now, with the intent to more fully discuss them in subsequent postings.

As stated in a previous entry in this blog, Administrator O’Keefe was instrumental in the decision to learn from Columbia’s accident and in particular, from the debris. Having gotten his unofficial “go” to develop the concept to study the debris, the task to actually put the concept into practice fell on a few of us in the hangar. I asked Scott Thurston, Columbia’s NASA Vehicle Manager, to develop the necessary plans. He did an outstanding job. He and a very small group debated where to store the debris, how to “advertise” that it even existed for study, the requirements for organizations to obtain select pieces, the logistics of lending it to them (it’s not easy lending government property to private organizations), and the proper approval authorities and documents. And, by the way, how to do this for many years to come – also not easy.

The results of their labor and Scott’s leadership are clear. The debris loan program is very much still alive, with several hundred pieces either actively out for study or with studies already concluded. The material is stored in a climate-controlled room in the Vehicle Assembly Building, also allowing easy access for employees to view it. It has a full-time NASA curator—Mike Ciannilli—who also developed and runs NASA’s Apollo, Challenger, Columbia Lessons Learned Program. Mike was very active in the debris recovery as an aerial searcher in Texas, and his passion for sharing the lessons makes him the perfect person for the job.

As with debris from Challenger, some pieces of Columbia continue to be found. To deal with this in Texas, a program involving local authorities is charged with taking calls from anyone finding pieces that may be from Columbia. They in turn call Ciannilli, who is responsible for determining the authenticity of the find and returning the material to KSC to join the other 84,000+ pieces of the ship’s debris.

The most “famous” piece found in this manner was a cryogenic tank from Columbia‘s fuel cell system that had been submerged in Lake Nacogdoches since February, 2003. A severe drought in the summer of 2011 lowered the lake level to the point that the tank was high and dry.

An aluminum cryogenic tank from Columbia’s fuel cell system, uncovered in Lake Nacogdoches in August 2011, more than eight years after the accident. (NASA photo)

Numerous other pieces have been found by farmers, ranchers, hikers, etc. I suspect debris will continue to be found from time to time. We know for certain that three of the six main engine turbine pumps are still out there somewhere. But like the three that we recovered, they are no doubt buried deep in the East Texas or Louisiana dirt or at the bottom of a body of water. They will probably never be found.

One of Columbia’s powerheads—found buried fourteen feet under the Louisiana mud. (NASA photo)

We officially wrapped up recovery operations in early May, 2003. The vast majority of Columbia that we will ever find is already home. And some of it is being used to advance our understanding in materials and structures subjected to extreme conditions. The goal is to design future spacecraft that can better withstand such conditions. One such example is a seat design capable of withstanding very high torsional forces.

Columbia continues her scientific and research missions, well after her last space flight. That legacy would have made her final crew proud.