Storing Normals in a Texture

In one line

Lighting decides brightness by looking only at the normal, so if you read the normal from a texture, even a flat surface looks bumpy. Which coordinate system to store that normal in is the question of tangent space.

Why this was needed

If you built the bumps of a brick wall as real geometry, you would need hundreds of thousands of triangles. But most of the cue people use to recognize bumps is the change in brightness. A protruding part receives light head on and is bright, and a recessed part receives it at a slant and is dark.

As you saw in the previous course, brightness is decided by the dot product of the normal and the light direction. Then, instead of changing the geometry, if you swap out only the normal per pixel, you can produce the same brightness changes. The bumps of a brick appear on a plane made of two triangles. The silhouette is still flat, but seen from the front it is hard to tell.

How it works

A normal is a unit vector whose components are each between -1 and 1, while a texture channel is between 0 and 255. So you need a rule for carrying it across.

저장할 때  c = (n * 0.5 + 0.5) * 255
읽을 때    n = c / 255 * 2 - 1     (그리고 다시 정규화)

The normal of a flat area, (0, 0, 1), becomes (128, 128, 255). This is why a normal map looks light purple overall. That color simply means "it is flat here".

Which coordinate system to store the normal in is the next problem. If you store normals in world space as they are, that texture is right only when the object is in that orientation. If the object rotates, everything is wrong.

That is why you use tangent space. It sets up, at each point of the surface, a coordinate system relative to that surface. The three axes are these.

N (normal)    표면에 수직인 방향
T (tangent)   표면 위에서 텍스처의 u 가 늘어나는 방향
B (bitangent) 표면 위에서 텍스처의 v 가 늘어나는 방향

In this coordinate system, "flat" is always (0, 0, 1). It is the same however the object is placed. So you can attach the same normal map to several objects, and it stays correct when the object moves.

T and B are computed from the vertex positions and the UVs. You get them by solving together the two edges of the triangle and the corresponding UV changes as simultaneous equations.

e1 = p1 - p0,  e2 = p2 - p0
d1 = uv1 - uv0,  d2 = uv2 - uv0
r  = 1 / (d1.x * d2.y - d2.x * d1.y)
T  = (e1 * d2.y - e2 * d1.y) * r
B  = (e2 * d1.x - e1 * d2.x) * r

There are cases where the denominator becomes 0. It is when a triangle's UVs are lumped into a single point or lie on a single line, and tangent space is not defined for such a triangle. It is a common problem that arises when unwrapping UVs, so tools give a warning.

Normal maps are usually baked from a high-resolution model. You build a model with millions of triangles, and at each point of the low-resolution model, you find the normal of the high-resolution surface, convert it to tangent space, and write it into the texture. That is why a game character has a few tens of thousands of triangles but carries the surface wrinkles of the multi-million-triangle model as they are.

There are also a few related techniques. A height map stores only a single height instead of the normal and computes the slope in the shader to make the normal. The storage is one third but more computation is needed. Parallax mapping imitates a sense of depth by shifting UVs a little according to the viewing direction, and it expresses, to a degree, the "occlusion" that a normal map alone cannot. Displacement mapping actually pushes vertices, so even the silhouette changes, but it needs that much geometry.

What it looks like in the field

Most problems where you attach a normal map and the lighting looks reversed come from the direction of the green channel. Some tools bake assuming the v axis points up, and some assume it points down. If the two conventions are mixed, the bumps look reversed. The fix is to flip the green channel in the shader or to rebake the texture.

Another is compression. If you use ordinary image compression on a normal map as is, it is visibly damaged. This is because small errors in color are not noticed by people, but errors in a normal are amplified into errors in brightness. So you use a compression format dedicated to normal maps, and then you sometimes do not store the z component and restore it with z = sqrt(1 - x² - y²).

And a normal map cannot change the silhouette. The edge of the object still looks cut flat, and when viewed at a slant, the bumps disappear. So where the silhouette matters, you use real geometry or another technique such as parallax mapping.

Finally, let me sort out where a normal map and the lighting calculation meet. There are two methods. One is to convert the tangent-space normal read from the texture to world space with the TBN matrix and take its dot product with the world-space light, and the other is the reverse: convert the light direction to tangent space and take the dot product there. The results are the same but the cost differs. If there is only one light, moving the light is cheaper, and you can even move it in advance in the vertex shader and interpolate. Conversely, if there are several lights or you also use an environment map, it is better to move the normal to world space.

What you will do in the next lab

You make the rule for carrying a normal into a color, and generate a normal map of hemispheres arranged in a grid yourself. Next, you compute the three tangent-space axes from vertex positions and UVs, and compare pictures of the same plane lit with a constant normal and with the normal map. Finally, you move the light from left to right to produce six frames and confirm in numbers that the bright side follows the light.