Somehow, we need to mathematically describe the interaction of light with surfaces. Step one is to describe surfaces. The standard description is to describe the surface orientation with a "normal"--a vector sticking out 90 degrees from the surface (your Calc III book probably described normals as "perpendicular to the surface's tangent plane"). There are actually two perpendicular-to-the-surface vectors that can be used as normals--one pointing inside the surface, and the other pointing outside, away from the surface. We'll always make our normals point away from the surface.

How can you calculate normals? For a sphere centered at the origin, it's easy--the normal at a point p is just p! For a more complicated surface, you'll have to compute the normals, often by taking the cross product of the edge vectors along your triangles (cross product takes two input vectors, and returns one vector perpendicular to both input vectors). In C++, if you call "glNormal3f" before glVertex, you're setting the "vertex normal". You can then read this in your vertex shader as "gl_Normal", which is a builtin vec3. The standard thing to do is then copy this vec3 into a "varying" parameter you pass to your fragment shader, where you compute the final object color.

So say light is arriving at a surface from direction L (another 3D vector, which by convention points from the surface towards the light source!). If the object's surface is tilted to the incoming light, only a fraction of the incoming light actually arrives at the surface. A little math shows that fraction is actually equal to the cosine of the angle (call it a) between the incoming light and the surface normal. For unit vectors N and L, we can compute cos(a) as dot(N,L), because dot product of unit vectors gives you the cosine of the angle between the vectors (another property of dot product that's easy to verify given the vectors (cos(a),sin(a)) and (1,0)!). If you don't have unit vectors, call "normalize" to make them have unit length. But be careful of the w coordinate--GLSL's normalize (and dot product) commands stupidly include the w coordinate, so vectors intended for lighting should either have w=0 or else just be vec3's (which amounts to the same thing!).

Overall, given a light of color C, coming from direction vector L, arriving at a surface with normal N, the light arriving at the surface is then just:

C * dot(N,L)

But if N and L face in opposite directions (e.g., when the surface is facing directly away from the light source, where a is 180 degrees), the dot product (like the cosine) goes negative. This is bad, because light sources don't actually *remove* light when you're facing away from them--they just fail to add any. So usually we calculate the arriving light as

C * clamp (dot(N,L), 0.0, 1.0);

You can actually compute the lighting in either your vertex or fragment shader. It's usually a bit faster at the vertex level (since there are usually fewer vertices than fragments), but it's more accurate at the fragment level (since interpolating normals is better than interpolating finished colors).