Landing on the Moon

Part 2: Debrief - Apollo 11 post-mission notes

20 May 2017, Gary

Post-mission debriefings are important. Evaluating what happened, what went right, and what went wrong, is the best way to improve future missions. Only after the fact, when there's time to study the details, do some subtle yet critical facts reveal themselves. Sometimes there was a problem that nobody had noticed. Other times, what seemed like a big problem, turns out to be no big deal.

Sure, you could skip the debrief, or you could look for improvements by going over the details again. That's the thing about knowledge, you'll never know what you don't know unless you stop and look for it.

descent — From Powered Descent Initiation (PDI) to the lunar surface took about 12 minutes. ^[4]

Down-range position error

The Apollo 11 LM landed about 3 nautical-miles off target. The guidance system was actually a combination of three separate systems: primary (PGNCS), alternate (AGS) and tracking data from Earth (MSFN). Back at mission control, all three systems were monitored constantly to see if any one of them disagreed with the others. All three systems performed as planned and data deviations stayed within acceptable limits.

The navigation error was caused by several small ΔV inputs to the spacecraft during coasting flight before the descent engine was started. In other words, something unexpectedly pushed the spacecraft off course. It wasn't an invisible space monster, it was small RCS attitude maneuvers and venting from the cooling system. These tiny little nudges during orbit were enough to cause the spacecraft to miss the landing site by 3 nautical miles.

The pre-selected landing area was nearly 10 nautical miles across. During mission planning, engineers had performed Monte Carlo analyses that predicted a 99% probability of landing within 3.6 nautical miles of the exact center. Saying they were off course isn't really accurate. The landing wasn't dead center but it was still within the permission mapped area. Mission planners knew there might be some last minute obstacles and when there were, it was not a problem.

Bingo fuel

The Saturn V rocket used to go to the moon is still (as of 2017) the biggest rocket ever built. This is because it had to lift all the fuel and other equipment needed to get out of Earth orbit, into lunar orbit, down do the Moon's surface, back up again, then back to Earth. Every pound lifted into orbit requires an exponentially larger rocket. The Saturn V was at the limits of what a rocket could do. Because of this, there was no room for extra weight which meant no room for extra fuel while landing on the moon.

Still, not wanting to kill astronauts, NASA brought enough fuel to be safe, but no more. If everything went perfect, it would require approximately 12 minutes of fuel to land. Since things sometimes go wrong, there were a couple minutes of extra fuel so the astronauts could hover over to a new landing site if necessary. The fuel valve might not work perfectly so there was a contingency for that. Also, the fuel indicators also aren't perfect, especially when the fuel sloshes about, so there was a margin for that too.

NASA, following fighter pilot tradition, used a concept called Bingo fuel. Basically, it means "Almost out of fuel, land now or crash." For the Apollo missions, Bingo fuel officially meant 20 seconds of fuel remaining before needing to abort. So when CapCom called out "60 seconds" what they were really saying is "60 seconds until the 20 second Bingo fuel margin."

In addition to that, NASA was using the most conservative estimate for the amount of remaining fuel. If you had two fuel gauges, and one showed emptier than the other, which one would you bet your life on? NASA made the safe bet but after the landing, further investigation revealed that there was indeed more fuel remaining than indicated.

With all the hovering Neil Armstrong did looking for a good landing spot, he got right down to the last drop of fuel. That may seem reckless because with no tow trucks or ambulances nearby to help if something went wrong, he had no room for mistakes. However, the deeper story reveals how much fuel he really had left. Not only did he have the 20 seconds of Bingo fuel, he also knew there was probably a safety margin not shown on the gauges. Indeed, post analysis revealed incorrect readings due to fuel sloshing so there was actually about 45 seconds of fuel remaining. Most importantly though, Armstrong already had the LM down to about 50 feet above the surface. At that height, even with Bingo fuel, he knew he could still get on the ground safely. Even if they fell the last few feet, things fall six times slower on the moon than they do here on Earth.

Armstrong probably never had any intention of aborting. Imagine getting that close then giving up. For these guys, failure was not an option. Indeed, in a post mission statement, Armstrong said ^[3]:

I guess that, at that altitude, running out of fuel wasn't a consideration. Because we would have let it just quit on us, probably, and let it fall on in.

1202 & 1201 errors

Within 10 seconds of landing on the moon, the lab that had designed the guidance computer received an urgent phone call from NASA. They needed an explanation for those 1201 and 1202 program alarms. The mission would not be allowed to continue as planned without an explanation and reassurance that the computer was still functioning properly. They had less than 24 hours to figure it out.

The design team immediately ran to their simulation facility and worked all night, receiving another phone call from NASA about once every 15-30 minutes. The team went over old ground, new ground, brainstorms, crazy ideas, anything. They had to find an answer.

Armstrong and Aldrin practicing in the simulator.

The 1202 and 1201 alarms meant the computer was running out of memory. This was probably caused by something running too slow, exhausting available resources, and causing lower priority tasks to run out of memory and fail. But why was the computer running too slow?

Then someone remembered seeing a similar problem. Once, during a test with the Rendezvous Radar Switch on, the I/O system started stealing too many cycles while looking for radar data which then caused the other tasks to slow down. But why would the Rendezvous Radar Switch be on? That was only used for rendezvous during ascent, it shouldn't be on during descent. Had the astronauts made a mistake?

Either way, the first step was to see if the radar really was the problem. They sifted through the piles of telemetry data sent back to Earth from the LM, found the correct 16-bit word, found the correct bit, and... yikes!!! it showed that the radar was indeed on. Maybe an astronaut had made a mistake.

Before accusing anyone, the engineers sifted through the 4-inch thick book of operating procedures and there it was, the checklist said to turn the radar switch on prior to descent. It wasn't the fault of the astronauts, they had followed procedure correctly.

So why hadn't this error ever been seen before? If this was standard procedure, why had none of the simulation tests ever shown this problem? As it turns out, since the simulator didn't use an active radar, that particular switch didn't really do anything. That's the danger of testing different systems in isolation, the problem only revealed itself when the systems were all operated together for the first time while descending to the moon.

Finally, an answer, and just in time. You can see it in the official transcript. Shortly before ascent back up to orbit, CAPCOM calmly gave Buzz Aldrin a couple checklist changes.

MET 05:01:41:44, CapCom: Got a couple changes to your surface checklist here. ... The main one being that we do not want the rendezvous radar on during the ascent, and we think that this will take care of some of the overflow of program alarms which we were getting during descent.

The astronauts hadn't made a mistake and neither had the software engineers. The guidance computer had been designed in such a way that it could handle unexpected errors just like this one. With no way to do a live test of landing on the moon without actually going to the moon, unexpected errors were inevitable. The entire world watched the first moon landing. Far fewer people understand how big of an achievement it really was.

PID Autopilot

Descent Computer Modes
P-63	Braking Phase Guidance
P-64	Approach Phase Guidance
P-65	Velocity Nulling Guidance
P-66	Rate of Descent
P-67	Manual Guidance

The Apollo LM did have an autopilot. Several actually, each one designed for a specific task. There was one for controlling the sink rate and others for controlling the descent trajectory, ascent trajectory, orbit injection burns, and so on. Like the cruise control in a car, they weren't designed to be fully autonomous but rather to do the precision grunt work while a human made the high level decisions.

Apollo-era computers were quite primitive by modern standards but still plenty powerful for a well designed autopilot. The basic theory they used is called PID and was originally developed in the 19th century by observing ship helmsmen. PID stands for Proportional, Integral, Derivative. That sounds fancy and the in-depth theory can fill textbooks and justify PhD's but the basics are easy to understand. Don't let the complicated looking formula intimidate you.

Let's start with three definitions, two inputs and one output:

PV = Process Variable = The value being measured. On a car's cruise control this would be the current speed.
SP = Set Point = The target value. For example, the desired speed.
OP = OutPut = The controlled value, computed by the above formula. Think of this as the throttle.

Looking at the PID formula, you can see that it is simply the sum of three terms: Proportional, Integral, Derivative. Each term is basically an error (e) scaled by a constant (K). The error is nothing but the Set Point minus the Process Variable (e = SP - PV) so the three terms are:

Proportional (present) - This is the current error (e = SP - PV) scaled by a constant (K_p). For a car's cruise control, it's proportional to the difference between the target speed and the current speed. So if the target speed is 55 mph, and the current speed is 40 mph, the error would be 15 mph. If the current speed is 65mph, the error would be -10 mph. The constant (K_p) acts as a gain. Adjusting the gain will adjust how sensitive the controller is. Too much gain will cause an overshoot (constantly swinging back and forth across the target) while too little gain will cause an undershoot (never quite reaching the target).

Integral (past) - This is the sum of all past errors (integral) scaled by a constant (K_i). In a sense, the integral remembers the past. It creates more output when there is a large cumulative error (i.e. we're way off) and less when it's close. As the output approaches the desired target, the proportional value (present) gets smaller and smaller, never quite getting there. The integral (past) helps push the output all the way to the target.

Derivative (future) - This is the error's rate of change (derivative) scaled by a constant (K_d). Since the set point (SP) rarely changes, the rate of change is simply the current measured value (PV) minus the previous value. By adding in the rate of change, the derivative helps to predict the future. For example "We're coming in hot, start slowing down now" or the opposite "This is taking forever, hurry up and get there." It helps avoid overshooting the target and helps smooth the system.

When you look at it this way, you can see how even a very primitive computer could handle this math. Put enough of these formulas together and you can land on the moon. Things do get a bit more complex but those are the amazingly powerful basics. If you're interested, here is some C++ code: raw pid.h & pid.cpp or Github.

Gary.land

Landing on the Moon

Part 2: Debrief - Apollo 11 post-mission notes

Down-range position error

Bingo fuel

1202 & 1201 errors

References

Down-range position error

Bingo fuel

1202 & 1201 errors

References

Sign In