A Coordinate System on the Surface

In one line

UV is a coordinate on the texture image, and if you attach it to each vertex, the rasterizer interpolates it per pixel. Pulling a color out at that coordinate is sampling, and the method you need differs between magnification and minification.

Why this was needed

If you attach a color to each vertex, a triangle has only three colors. To express a brick pattern or letters, you would have to cut the vertices as finely as the pattern, and that is unmanageable.

Textures turn that problem around. You take the color off the geometry and put it in a separate image, and only decide where in that image each point of the surface points. That "where" is the UV coordinate. Then you can attach an arbitrarily complex picture to a rectangle made of two triangles.

How it works

UV is usually set between 0 and 1, with (0,0) at one corner of the image and (1,1) at the diagonally opposite corner. Which corner is (0,0) differs by API — it is the bottom left in OpenGL and the top left in Direct3D. Half of the bugs where a texture comes out flipped come from here.

How to handle UV outside the range is the wrap mode. Repeat (repeat) takes only the fractional part and repeats the pattern, clamp (clamp) stretches the edge value, and mirror (mirror) flips it each time as it tiles. Repeat is right for tiled patterns, and clamp is right when pasting a single picture as is.

There are two basic ways to pull out a color.

최근접(nearest)   i = floor(u * 폭), j = floor(v * 높이) 자리의 텍셀 하나
쌍선형(bilinear)  이웃한 텍셀 네 개를 가로세로 비율로 섞는다

What matters in bilinear sampling is that the center of a texel is at (i+0.5)/width. If you leave out this half cell, the picture shifts by half a texel. So you shift the coordinate to u * 폭 - 0.5 (the placeholder is the width) and then split it into the integer part and the fractional part.

Perspective-correct interpolation is especially important for textures. If you simply blend UV with barycentric coordinates computed in screen coordinates, the pattern is warped on a surface with perspective. This is because intervals divided evenly on the screen are not even in 3D space. The correct method is to interpolate u/w, v/w and 1/w and then divide the first two by the last.

iw = Σ bary_k * (1 / w_k)
u  = Σ bary_k * (u_k / w_k)  /  iw

The floor patterns wobbling in early console games was because this correction was not done. Hardware now does it for you, but it is worth knowing why it is needed.

Minification is a different problem. If one pixel covers dozens of texels, picking just one of them cannot represent the original pattern. Even if the camera moves slightly, the texel you pick changes and the screen shimmers. The fix is the mipmap — you build in advance a chain of the texture shrunk by half each time, and pick the level that matches the number of texels the pixel covers. In exchange for using one third more memory, the shimmering disappears, and when drawing small, the cache hit rate actually goes up and it gets faster.

Anisotropic filtering makes up for the limit of mipmaps. A mipmap assumes that one pixel covers texels equally in width and height, but on a surface seen at a slant, like a floor, it covers a long stretch in only one direction. If you then match the mip level to the long side, the short side gets blurred more than needed. So the anisotropic filtering in a game's graphics settings draws several samples along the long direction to undo that blur. The effect of this option is that a road stretching into the distance becomes sharp.

How a texture is laid out in memory also affects performance. Neighboring pixels on screen usually read neighboring texels of the texture, but if you lay it out row-major as is, vertically neighboring texels are far apart in memory. So the GPU rearranges and stores textures in small square blocks. This is called swizzling or a tiled layout, and it is one of the reasons a texture upload is slower than a plain copy.

What it looks like in the field

A common problem is a thin line showing at the boundary of a tiled pattern. It is because bilinear sampling in repeat mode blends the right edge and the left edge of the image, but this is in fact the repeat mode working as intended. If you use clamp mode instead, it does not blend the opposite edge.

Another is atlas bleeding. It is a method of gathering several pictures into one sheet and cutting them out with UV, but at the lower levels of the mipmap, the colors of neighboring pictures get mixed in. You must leave padding between pictures or limit the mip level.

Finally, let me point out who decides the UV. The UV of a vertex is decided by a person or a tool at the modeling stage. It is the job of unfolding a 3D surface onto a 2D plane, so you have to decide where to put the cutting lines and where to concentrate the distortion. This work is called UV unwrapping, and if done badly the pattern stretches or the seams stand out. On the shader side you only receive and use the UV that is already decided, but it is good to know that when a pattern looks wrong, the cause may be in the model rather than the shader.

What you will do in the next lab

You make a checker pattern texture and implement nearest-neighbor and bilinear sampling. Next, you paste the pattern on a quadrilateral with perspective, view the uncorrected and corrected versions side by side, and produce a difference image. Finally, you build one mip level from a finely flickering pattern and confirm in numbers that the shimmering disappears.