Build the oracle before you start shrinking

In one line

Half of building a minimal reproducible case is not the skill of shrinking but having a machine judge "is it still the same failure?" Without a judge, shrinking turns into guesswork.

Why this was needed

The reproduction steps a customer sends are usually everything that person did that day. They come with a 12-step procedure and a 200-line input file. If you try to find the cause while carrying all of that, the combinations you have to look at explode, you start from an arbitrary place, and two days later you are still in the same spot.

Everyone knows it has to be shrunk. The problem is getting lost while shrinking. You remove one line and the program dies with a different error. Since the other error is also an error, you mistake it for "it still reproduces" and keep shrinking. The three-line input that comes out of that reproduces some other defect that has nothing to do with the defect we were chasing. Take those three lines to the customer's development team and a day is gone.

Why the judge comes first

So there is an order. Build the judge before you shrink. The judge (the oracle) is a program that takes one candidate input and answers only one question: "does this produce the very failure we are chasing right now?" A judgment a person makes by eye cannot be used for shrinking. Shrinking means testing candidates dozens to hundreds of times, and a person cannot cope with that many.

The judge looks at about three things.

The exit code — that it is nonzero is not enough. You look at which code it is.
The signature string — an error name or exception name that appears only in that failure. The narrower, like ERR-QTY-COMMA, the better.
The location — if needed, even which step it occurred in. There are programs where the same exception arises in two places.

Conversely, there are things the judge must not look at. Values that differ from run to run, such as the execution time, temporary file names, and the number of log lines, do not go into the judgment. If they do, the same candidate reproduces on some days and not on others.

How it works

Once you have a judge, a machine does the shrinking. The simplest method is to remove one line at a time. For 200 lines you call the judge 200 times, and each pass leaves fewer lines, so you go around again. It is correct but slow.

Zeller and Hildebrandt's ddmin does this in chunks. It splits the candidate into n pieces and first checks whether one piece alone reproduces it, and if not, checks whether the rest with one piece removed reproduces it. If neither does, it splits more finely and repeats the same thing. If only two lines are the cause, the number of calls falls close to the logarithm of the number of lines — finding two lines among 256 takes only a little over a hundred calls to the judge.

[a b c d e f g h]        전체가 재현된다
  갈라 본다 → [a b c d] [e f g h]
  [a b c d] 재현 안 됨
  [e f g h] 재현 안 됨          ← 조각 하나만으로는 안 된다
  나머지로 본다 → [e f g h](= a b c d 를 뺀 것) 안 됨, [a b c d] 안 됨
  더 잘게 → [a b] [c d] [e f] [g h]
  [a b] 뺀 나머지 재현됨 → 후보가 [c d e f g h] 로 줄었다
  … 반복 …
결과: [c g]  — 여기서 c 나 g 를 빼면 재현되지 않는다

Saying the result is 1-minimal means that removing any one of what is left stops the reproduction. It does not mean "the smallest" — a completely different, smaller combination may exist. So after you finish shrinking, you try removing each remaining line one by one and record separately the proof that it stops. Without this proof, you cannot answer the question "why is this line needed?"

What it looks like in the field

First, shrink the procedure the same way as the input. It is common that only four of the 12 steps are actually needed. The other eight steps are the person's habit. When you shrink the procedure, the vague explanation of "it's the environment" goes away.

Second, run it in a clean place. If there are files left by the previous run, it still reproduces even when you remove the "step that creates the configuration file". Then you reach the wrong conclusion that the step is unnecessary. Each time you test a procedure, take a fresh empty directory.

Third, change the values to check that the shrunk case is real. If the remaining two lines are locale=de and qty=1,5, see whether it still reproduces when you change the number to 9,25. If it does, the cause is not that number but the combination of the comma and the locale. Only when you have gone this far is it a report that the customer's development team can fix right away.

Fourth, send the minimal case together with the judge. If you send only the two input lines, the receiving side starts by asking "why is this a problem?" If you send the judge along, the receiving side runs it once at their own end and immediately stands in the same place.

What really matters in practice

Build the judge first. If you change this order, shrinking becomes guesswork.
Any failure is not the same failure. Look at the exit code and the signature together.
Prove 1-minimality and write it down. Leave as a record that removing each line stops it.
Check whether it is structure, not values. If it reproduces even when you change the values, the cause is structure.

What you will do in the next lab

You unpack the incident bundle sent by (fictional) Daerim Trading. It is one processor, a 200-line feed, and a 12-step reproduction procedure. First you build an oracle that judges the "same failure", shrink the feed with that oracle, and try removing each remaining line one by one to prove it is 1-minimal. Then you implement ddmin yourself to automate the shrinking — the grader runs your shrinker on a 256-line input with the answer hidden in a different place each time, and looks both at whether the result is right and at how many times you called the judge. Finally you shrink the 12-step procedure, check whether it still reproduces when you change the values, and deliver it as a report.